Glassvision/IfcOpenShell
1
0
Fork 0
mirror of https://github.com/IfcOpenShell/IfcOpenShell.git synced 2026-08-05 23:41:44 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
ifcviewer-wgpu
IfcOpenShell/nix/build-all.py
T
Andrej730 d1027c5877 build-all: fix missing CXXFLAGS in manual build (f8f4725) 
In the mentioned commit I've dropped workaround, because issue upstream was resolved, but forgot to load flags from `pyodide` fro manual build.
2026-08-05 18:37:46 +05:00

1677 lines
66 KiB
Python
Raw Permalink Blame History

#!/usr/bin/python
# /// script
# ///
###############################################################################
# #
# This file is part of IfcOpenShell. #
# #
# IfcOpenShell is free software: you can redistribute it and/or modify #
# it under the terms of the Lesser GNU General Public License as published by #
# the Free Software Foundation, either version 3.0 of the License, or #
# (at your option) any later version. #
# #
# IfcOpenShell is distributed in the hope that it will be useful, #
# but WITHOUT ANY WARRANTY; without even the implied warranty of #
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the #
# Lesser GNU General Public License for more details. #
# #
# You should have received a copy of the Lesser GNU General Public License #
# along with this program. If not, see <http://www.gnu.org/licenses/>. #
# #
###############################################################################
"""
Example usage:
# Build all targets by default, except BonsaiViewer (it's set explicitly).
python build-all.py
# Build just the provided targets.
python build-all.py IfcParse IfcOpenShell-Python
Available arguments:
``-py-313`` - build for specific Python version
(building for all supported Python version by default).
``-wasm`` - compile for wasm
``-without-xxx`` - do not build dependency ``xxx`` (e.g. ``--without-swig``)
``-mac-cross-compile-intel`` - cross compile for Intel Mac on Apple Silicon host
``-shared`` - build shared libraries. By default will build static.
``-ifcopenshell-shared`` - build only IfcOpenShell's own libraries as shared
(dependencies stay static). Redundant if ``-shared`` is also passed.
``-diskcleanup`` - clean up build directories after finishing building dependencies
``-v`` - enable verbose logs
Used environment variables:
Boolean-like env variables accept the following values (case-insensitive):
`opt-in` - `1`, `on`, `true`, `yes`; ``opt-out`` - `0`, `off`, `false`, `no`.
- ``CXXFLAGS``, ``CPPFLAGS``, ``CFLAGS``, ``LDFLAGS``
- ``BUILD_DIR`` - build directory. By default will use "build" folder in IfcOpenShell repository.
- ``DEPS_DIR`` - dependencies directory. By default will create automatic folder in build directory.
- ``BUILD_CFG`` - build configuration, 'RelWithDebInfo' by default.
- ``USE_CURRENT_PYTHON_VERSION`` - use current python config instead of compile from source
`off` by default.
- ``IFCOS_NUM_BUILD_PROCS`` - number of concurrent processes defaults to available cores + 1
- ``NO_CLEAN`` - do not clean `ifcopenshell` build directories but continue working on current build
(installed dependencies are never cleared).
By default option is disabled, to enable pass any value from `1`, `on`, `true`.
- ``IFCOS_SCHEMAS`` - schemas to be built; defaults to cmake default (8 schemas), to be supplied as `2x3;4;4x3_add2`
- ``USE_OCCT`` - whether to use official Open CASCADE instead of Community Edition
`on` by default
- ``WASM_PYTHON_PATH`` - path to WASM Python installation,
used to deduce `PYVERSION` (e.g. '3.13.2'), `PYTHONINCLUDE`,
`SIDE_MODULE_CFLAGS`, `SIDE_MODULE_LDFLAGS`.
Allows to build wasm without pyodide build environment, which can be useful for debugging build issues.
Example value: 'pyodide/cpython/installs/python-3.13.2'
- ``ADD_COMMIT_SHA`` - `off` by default. If enabled
`ADD_COMMIT_SHA` and `VERSION_OVERRIDE` will be set to `ON` while configuring IfcOpenShell
- ``BUILD_BONSAIVIEWER`` - enable building BonsaiViewer, `off` by default.
- ``IFCOS_BUILD_PYTHON_WRAPPER`` - enable building the Python wrapper, `on` by default.
# This script builds IfcOpenShell and its dependencies #
# #
# Prerequisites for this script to function correctly: #
# * cmake * git * bzip2 * tar * c(++) compilers * autoconf #
# #
# if building with USE_OCCT additionally: #
# * glx.h #
# #
# if building with OCCT 7.4.0 additionally: #
# * libfontconfig1-dev #
# #
# if building with -shared #
# * libgl1-mesa-dev libxext-dev libxmu-dev libxmu-headers libxi-dev #
# #
# for python37 to install correctly additionally: #
# * libffi(-dev[el]) #
# for Python build we also needs ssl and zlib #
# (since we do `pip install numpy` at the end) #
# * libssl-dev #
# #
# on debian 7.8 these can be obtained with: #
# $ apt-get install git gcc g++ autoconf bison bzip2 cmake #
# mesa-common-dev libffi-dev libfontconfig1-dev #
# libssl-dev xz zlib1g-dev #
# #
# on ubuntu 14.04: #
# $ apt-get install git gcc g++ autoconf bison make cmake #
# mesa-common-dev libffi-dev libfontconfig1-dev #
# libssl-dev xz-utils zlib1g-dev #
# #
# on OS X El Capitan with homebrew: #
# $ brew install git bison autoconf automake libffi cmake #
# $ # `bison` shipped with Mac is too old for swig build, #
# $ # so we use `brew`. #
# $ export PATH=$(brew --prefix bison)/bin:$PATH #
# #
# on RHEL-related distros: #
# $ dnf install git gcc gcc-c++ autoconf bison make cmake #
# mesa-libGL-devel libffi-devel fontconfig-devel bzip2 #
# automake patch byacc xz zlib-devel openssl-devel #
"""
import glob
import logging
import multiprocessing
import os
import platform
import shutil
import ssl
import subprocess as sp
import sys
import sysconfig
import tarfile
import threading
import time
from collections.abc import Generator, Sequence
from datetime import datetime
from pathlib import Path
from typing import Literal
from urllib.request import urlretrieve
logger = logging.getLogger(__name__)
logger.setLevel(logging.INFO)
ch = logging.StreamHandler()
logger.addHandler(ch)
def is_on_off(value: str | None, *, default: bool) -> bool:
if value is None:
return default
lowered = value.lower()
if lowered in {"1", "on", "true", "yes"}:
return True
if lowered in {"0", "off", "false", "no"}:
return False
return default
PROJECT_NAME = "IfcOpenShell"
USE_CURRENT_PYTHON_VERSION = is_on_off(os.getenv("USE_CURRENT_PYTHON_VERSION"), default=False)
ADD_COMMIT_SHA = is_on_off(os.getenv("ADD_COMMIT_SHA"), default=False)
IFCOS_BUILD_PYTHON_WRAPPER = is_on_off(os.getenv("IFCOS_BUILD_PYTHON_WRAPPER"), default=True)
BUILD_BONSAIVIEWER = is_on_off(os.getenv("BUILD_BONSAIVIEWER"), default=False)
USE_OCCT = is_on_off(os.getenv("USE_OCCT"), default=True)
PYTHON_VERSIONS = ["3.10.3", "3.11.8", "3.12.1", "3.13.6", "3.14.0"]
JSON_VERSION = "3.11.3"
OCE_VERSION = "0.18.3"
OCCT_VERSION = "7.8.1"
BOOST_VERSION = "1.86.0"
EIGEN_VERSION = "3.4.0"
PCRE_VERSION = "8.41"
LIBXML2_VERSION = "2.13.8"
SWIG_VERSION = "4.2.1"
OPENCOLLADA_VERSION = "v1.6.68"
GMP_VERSION = "6.3.0"
MPFR_VERSION = "3.1.6" # latest is 4.1.0
CGAL_VERSION = "v5.6.3"
USD_VERSION = "23.05"
TBB_VERSION = "2021.9.0"
ROCKSDB_VERSION = "10.4.2"
ZSTD_VERSION = "1.5.7"
MANIFOLD_VERSION = "3.2.1"
QT6_VERSION = os.getenv("QT6_VERSION", "6.8.3")
# binaries
cp = "cp"
bash = "bash"
git = "git"
bunzip2 = "bunzip2"
tar = "tar"
cc = "cc"
cplusplus = "c++"
autoconf = "autoconf"
automake = "automake"
make = "make"
date = "date"
curl = "curl"
wget = "wget"
strip = "strip"
xz = "xz" # Used implicitly for `tar -xf *.tar.xz`.
brew = "brew"
explicit_targets = [s for s in sys.argv[1:] if not s.startswith("-")]
"""Targets provided by CLI."""
flags = set(s.lstrip("-") for s in sys.argv[1:] if s.startswith("-"))
"""CLI flags."""
# Helper function for coloured printing
NO_COLOR = (
"\033[0m" # <ref>http://stackoverflow.com/questions/5947742/how-to-change-the-output-color-of-echo-in-linux</ref>
)
BLACK_ON_WHITE = "\033[0;30;107m"
RED = "\033[31m"
GREEN = "\033[32m"
YELLOW = "\033[33m"
MAGENTA = "\033[35m"
def cecho(message, color=NO_COLOR):
"""Logs message `message` in color `color`."""
logger.info(f"{color}{message}\033[0m")
# Flags.
APPLE = platform.system() == "Darwin"
MAC_CROSS_COMPILE_INTEL = "mac-cross-compile-intel" in flags
assert platform.system() == "Darwin" or not MAC_CROSS_COMPILE_INTEL
WASM = "wasm" in flags
"""Build WASM outside pyodide build environment."""
WASM_CMAKE_IS_USING_INIT_VARS = False
if WASM:
def get_pyodide_config_var(var_name: str) -> str:
output = sp.check_output(["pyodide", "config", "get", var_name], encoding="utf-8").strip()
return output
if "PYODIDE_ROOT" not in os.environ:
cecho("WARNING. Couldn't find 'PYODIDE_ROOT' in environment variables.", YELLOW)
cecho("Assuming building wasm outside pyodide build environment and resetting necessary variables.", YELLOW)
os.environ["SIDE_MODULE_CFLAGS"] = get_pyodide_config_var("cflags")
os.environ["SIDE_MODULE_CXXFLAGS"] = get_pyodide_config_var("cxxflags")
os.environ["SIDE_MODULE_LDFLAGS"] = get_pyodide_config_var("ldflags")
# Override cmake toolchain for all `emcmake` calls,
# needed for shared libraries (resulting .so wrapper)
# and to ensure compilation is pyodide compatible (e.g. `-fwasm-exceptions` is used in compilation flags).
os.environ["CMAKE_TOOLCHAIN_FILE"] = get_pyodide_config_var("cmake_toolchain_file")
required_vars = (
"SIDE_MODULE_CFLAGS",
"SIDE_MODULE_CXXFLAGS",
"SIDE_MODULE_LDFLAGS",
"CMAKE_TOOLCHAIN_FILE",
)
missing_vars = [v for v in required_vars if v not in os.environ]
assert not missing_vars, f"Some variables required for WASM compilation are missing: {', '.join(missing_vars)}"
def get_pyodide_build_version() -> tuple[int, ...]:
pyodide_build_suffix = "pyodide-build version:"
output = sp.check_output(["pyodide", "--version"], encoding="utf-8").strip()
assert pyodide_build_suffix in output, output
version_line = next(l for l in output.splitlines() if l.startswith(pyodide_build_suffix))
version = version_line.partition(":")[2].strip()
return tuple(map(int, version.split(".")))
# Pyodide still in transition from `FLAGS` to `FLAGS_INIT`.
# `FLAGS_INIT` allow us to provide flags using environment variables
# and providing `FLAGS` directly would break pyodide toolchain.
# NOTE: changes in pyodide-build currently got postponed:
# https://github.com/pyodide/pyodide-build/pull/249
WASM_CMAKE_IS_USING_INIT_VARS = get_pyodide_build_version() >= (99, 0, 0)
# 0.31 is required for SIDE_MODULE_CXXFLAGS to be provided.
assert get_pyodide_build_version() >= (0, 31)
TOOLSET = None
if platform.system() == "Darwin":
# C++11 features used in OCCT 7+ need a more recent stdlib
# TOOLSET = "10.9" if USE_OCCT else "10.6"
# /Users/runner/work/IfcOpenShell/IfcOpenShell/build/Darwin/arm64/10.9/build/rocksdb/cache/clock_cache.cc:732:14: error: aligned allocation function of type 'void *(std::size_t, std::align_val_t)' is only available on macOS 10.13 or newer
# /Users/runner/work/IfcOpenShell/IfcOpenShell/src/ifcparse/IfcFile.cpp:539:14: error: 'exists' is unavailable: introduced in macOS 10.15
TOOLSET = "10.15"
IFCOS_NUM_BUILD_PROCS = os.getenv("IFCOS_NUM_BUILD_PROCS", multiprocessing.cpu_count() + 1)
SCRIPT_PATH = Path(__file__).parent
REPO_PATH = SCRIPT_PATH.parent
CMAKE_DIR = (REPO_PATH / "cmake").resolve().__str__()
BUILD_DIR = os.environ.get("BUILD_DIR", (REPO_PATH / "build").__str__())
if WASM:
arch = "wasm"
elif MAC_CROSS_COMPILE_INTEL:
arch = "x86_64"
else:
arch = platform.machine()
DEFAULT_DEPS_DIR = Path(BUILD_DIR) / platform.system() / arch
if TOOLSET:
DEFAULT_DEPS_DIR = DEFAULT_DEPS_DIR / TOOLSET
DEFAULT_DEPS_DIR = os.path.realpath(DEFAULT_DEPS_DIR)
DEPS_DIR = os.getenv("DEPS_DIR", DEFAULT_DEPS_DIR)
if not os.path.exists(DEPS_DIR):
os.makedirs(DEPS_DIR)
INSTALL_DIR = Path(DEPS_DIR) / "install"
BUILD_CFG = os.getenv("BUILD_CFG", "RelWithDebInfo")
# Print build configuration information
cecho(
f"""This script fetches and builds {PROJECT_NAME} and its dependencies
""",
BLACK_ON_WHITE,
)
cecho(
"""Script configuration:
""",
GREEN,
)
cecho(f"""* USE_OCCT = {USE_OCCT}""", MAGENTA)
if USE_OCCT:
cecho(" - Compiling against official Open Cascade")
else:
cecho(" - Compiling against Open Cascade Community Edition")
cecho(f"* Build Directory = {BUILD_DIR}", MAGENTA)
cecho(f"* Dependency Directory = {DEPS_DIR}", MAGENTA)
cecho(f" - The directory where {PROJECT_NAME} dependencies are installed.")
cecho(f"* Build Config Type = {BUILD_CFG}", MAGENTA)
cecho(""" - The used build configuration type for the dependencies.
Defaults to RelWithDebInfo if not specified.""")
if BUILD_CFG == "MinSizeRel":
cecho(" WARNING: MinSizeRel build can suffer from a significant performance loss.", RED)
cecho(f"* IFCOS_NUM_BUILD_PROCS = {IFCOS_NUM_BUILD_PROCS}", MAGENTA)
cecho(""" - How many compiler processes may be run in parallel.""")
cecho(f"* IFCOS_SCHEMAS = '{os.environ.get('IFCOS_SCHEMAS')}'", MAGENTA)
cecho(""" - IFC Schemas to compile. If not provided, fallback to default provided in cmake.
""")
dependency_tree: dict[str, tuple[str, ...]] = {
"IfcParse": ("boost", "libxml2", "rocksdb"),
"IfcGeom": ("IfcParse", "occ", "manifold", "json", "cgal", "eigen", "OpenCOLLADA"),
"IfcConvert": ("IfcGeom",),
"OpenCOLLADA": ("libxml2", "pcre"),
"IfcGeomServer": ("IfcGeom",),
"IfcOpenShell-Python": ("python", "swig", "IfcGeom"),
"BonsaiViewer": ("IfcGeom", "qt6"),
"swig": (),
"boost": (),
"libxml2": (),
"python": (),
"occ": (),
"pcre": (),
"json": (),
"cgal": (),
"eigen": (),
"rocksdb": ("zstd",),
"zstd": (),
"manifold": (),
"qt6": (),
# 'usd': ('boost', 'oneTBB')
}
def gather_dependencies(dep: str) -> Generator[str]:
yield dep
for d in dependency_tree[dep]:
if f"without-{d.lower()}" not in flags:
for x in gather_dependencies(d):
yield x
if "v" in flags:
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s")
ch.setFormatter(formatter)
else:
logger.setLevel(logging.INFO)
if MAC_CROSS_COMPILE_INTEL:
MAC_CROSS_COMPILE_INTEL_ARGS = ["-DCMAKE_OSX_ARCHITECTURES=x86_64"]
MAC_CROSS_COMPILE_INTEL_BJAM_ARGS = ["architecture=x86"]
MAC_CROSS_COMPILE_INTEL_CXX = "clang++ -arch x86_64"
MAC_CROSS_COMPILE_INTEL_CC = "clang -arch x86_64"
MAC_CROSS_COMPILE_INTEL_AUTOCONF_HOST_ARGS = ["--host=x86_64-apple-darwin"]
else:
MAC_CROSS_COMPILE_INTEL_ARGS = []
MAC_CROSS_COMPILE_INTEL_BJAM_ARGS = []
MAC_CROSS_COMPILE_INTEL_CXX = ""
MAC_CROSS_COMPILE_INTEL_CC = ""
MAC_CROSS_COMPILE_INTEL_AUTOCONF_HOST_ARGS = []
OFF_ON = ["OFF", "ON"]
BUILD_STATIC = "shared" not in flags
"""Whether dependencies are built static."""
IFCOPENSHELL_STATIC = BUILD_STATIC and "ifcopenshell-shared" not in flags
"""Whether IfcOpenShell's own libraries are built static."""
ENABLE_FLAG = "--enable-static" if BUILD_STATIC else "--enable-shared"
DISABLE_FLAG = "--disable-shared" if BUILD_STATIC else "--disable-static"
LINK_TYPE = "static" if BUILD_STATIC else "shared"
LINK_TYPE_UCFIRST = LINK_TYPE.capitalize()
LIBRARY_EXT = "a" if BUILD_STATIC else "so"
PIC = "-fPIC" if BUILD_STATIC else ""
if any(f.startswith("py-") for f in flags):
PYTHON_VERSIONS = [pyv for pyv in PYTHON_VERSIONS if f"py-{''.join(pyv.split('.')[:2])}" in flags]
if any(f.startswith("occt-") for f in flags):
OCCT_VERSION = next(f.split("-", 1)[1] for f in flags if f.startswith("occt-"))
if explicit_targets:
targets = {dep for target in explicit_targets for dep in gather_dependencies(target)}
else:
targets = set(dependency_tree.keys())
targets = set(t for t in targets if "without-%s" % t.lower() not in flags)
if not explicit_targets and not BUILD_BONSAIVIEWER:
targets.difference_update({"BonsaiViewer", "qt6"})
if BUILD_BONSAIVIEWER:
targets.update(gather_dependencies("BonsaiViewer"))
# Opt-out for the Python wrapper. Currently used by the bonsai CI workflow
# on macOS, where the post-plug-in-refactor wrapper hard-links
# ifcopenshell.document.rdb.dylib but the CREATE_BUNDLE install rule does
# not actually drop the dylib next to the wrapper in site-packages — see
# the commit message that introduced this gate.
if not IFCOS_BUILD_PYTHON_WRAPPER:
targets.discard("IfcOpenShell-Python")
if WASM:
SKIP_TARGETS_FOR_WASM = {
"rocksdb",
"opencollada",
"swig",
"pcre",
"IfcGeom",
"IfcConvert",
"IfcGeomServer",
"BonsaiViewer",
"qt6",
}
SKIP_TARGETS_FOR_WASM = {t.lower() for t in SKIP_TARGETS_FOR_WASM}
skip_targets = {t for t in targets if t.lower() in SKIP_TARGETS_FOR_WASM}
if skip_targets:
cecho(f"Skipping targets for wasm build: {', '.join(sorted(skip_targets))}", YELLOW)
targets.difference_update(skip_targets)
print("Building:", *sorted(targets, key=lambda t: len(list(gather_dependencies(t)))))
# Check that required tools are in PATH
yacc = "yacc" # Used during swig building process, installed with `bison` on Debian / `byacc` on Red Hat.
bison = "bison"
missing_commands: list[str] = []
required_commands = [git, bunzip2, tar, cc, cplusplus, autoconf, automake, make, "patch", "cmake", yacc, xz, bison]
if "wasm" in flags:
# Skip swig build for WASM.
required_commands.append("swig")
required_commands.append("pyodide")
required_commands.remove(yacc)
required_commands.remove(bison)
if platform.system() == "Linux" and "BonsaiViewer" in targets:
required_commands.append("patchelf")
for cmd in required_commands:
if shutil.which(cmd) is None:
missing_commands.append(cmd)
if missing_commands:
raise ValueError(f"Required tools not installed or not added to PATH: {', '.join(missing_commands)}")
MAC_INTEL_BIN_PATH = "/usr/local/bin"
if MAC_CROSS_COMPILE_INTEL:
brew = f"{MAC_INTEL_BIN_PATH}/brew"
assert os.path.exists(brew), f"For intel cross compilation the brew path is expected to be '{brew}'."
# identifiers for the download tool (could be less memory consuming as ints, but are more verbose as strings)
download_tool_default = download_tool_py = "py"
download_tool_git = "git"
# Create log directory and file
log_dir = os.path.join(DEPS_DIR, "logs")
if not os.path.exists(log_dir):
os.makedirs(log_dir)
LOG_FILE = os.path.join(log_dir, sp.check_output([date, "+%Y%m%d"], encoding="utf-8").strip()) + ".log"
if not os.path.exists(LOG_FILE):
open(LOG_FILE, "w").close()
logger.info(f"using command log file '{LOG_FILE}'")
# Causing havoc in python 3.11 build
try:
del os.environ["__PYVENV_LAUNCHER__"]
except:
pass
def restore_env(var_name: str, old_value: str | None) -> None:
if old_value is None:
del os.environ[var_name]
else:
os.environ[var_name] = old_value
def run(cmds: Sequence[str], cwd: str | None = None, can_fail: bool = False) -> str:
"""
Wraps `subprocess.Popen.communicate()` and logs the command being executed,
sets up logging `stderr` to `LOG_FILE` (in append mode) and returns stdout
with leading and trailing whitespace removed.
"""
def timestamp() -> str:
return datetime.now().strftime("%Y-%m-%d %H:%M:%S,%f")[:-3] # same format as logging
def stream_reader(pipe, collector: list[str], log_file) -> None:
for line in iter(pipe.readline, ""):
log_file.write(f"{timestamp()} {line}")
log_file.flush()
collector.append(line)
pipe.close()
logger.debug(f"running command `{' '.join(cmds)}` in directory '{cwd}'")
stdout: list[str] = []
stderr: list[str] = []
# Ensure both live logs available in the log file
# and the putput.
with open(LOG_FILE, "a", encoding="utf-8") as log_file_handle:
proc = sp.Popen(cmds, cwd=cwd, stdout=sp.PIPE, stderr=sp.PIPE, encoding="utf-8")
assert proc.stdout and proc.stderr
t_out = threading.Thread(target=stream_reader, args=(proc.stdout, stdout, log_file_handle))
t_err = threading.Thread(target=stream_reader, args=(proc.stderr, stderr, log_file_handle))
t_out.start()
t_err.start()
t_out.join()
t_err.join()
proc.wait()
logger.debug(f"command returned {proc.returncode}")
if proc.returncode != 0 and not can_fail:
print("-" * 70)
print("".join(stderr))
print("-" * 70)
raise RuntimeError(f"Command `{' '.join(cmds)}` returned exit code {proc.returncode}")
return "".join(stdout).strip()
if platform.system() == "Darwin":
if run(["sw_vers", "-productVersion"]) >= "11.":
# Apparently not supported
PYTHON_VERSIONS = [pv for pv in PYTHON_VERSIONS if tuple(map(int, pv.split("."))) >= (3, 7)]
if run(["sw_vers", "-productVersion"]) < "10.16":
# This is now solved with the '__PYVENV_LAUNCHER__' hack
# PYTHON_VERSIONS = [pv for pv in PYTHON_VERSIONS if tuple(map(int, pv.split("."))) < (3, 11)]
pass
BOOST_VERSION_UNDERSCORE = BOOST_VERSION.replace(".", "_")
OCE_LOCATION = f"https://github.com/tpaviot/oce/archive/OCE-{OCE_VERSION}.tar.gz"
BOOST_LOCATION = f"https://github.com/boostorg/boost/releases/download/boost-{BOOST_VERSION}/"
# Helper functions
def run_autoconf(dependency_name: str, configure_args: list[str], cwd: str) -> None:
configure_path = os.path.realpath(os.path.join(cwd, "..", "configure"))
if not os.path.exists(configure_path):
run(
[bash, "./autogen.sh"], cwd=os.path.realpath(os.path.join(cwd, ".."))
) # only run autogen.sh in the directory it is located and use cwd to achieve that in order to not mess up things
# Using `sh` over `bash` fixes issues with building swig
prefix = os.path.realpath(f"{DEPS_DIR}/install/{dependency_name}")
wasm = []
if "wasm" in flags:
wasm.append("emconfigure")
run(
[
*wasm,
"/bin/sh",
"../configure",
*(["--host=wasm32"] if "wasm" in flags and not any(s.startswith("--host") for s in configure_args) else []),
*configure_args,
f"--prefix={prefix}",
],
cwd=cwd,
)
def run_cmake(arg1, cmake_args: list[str], cmake_dir: str | None = None, cwd: str | None = None):
if cmake_dir is None:
P = ".."
else:
P = cmake_dir
wasm = []
if "wasm" in flags:
wasm.append("emcmake")
cmake_flags: list[str] = []
if not WASM or not WASM_CMAKE_IS_USING_INIT_VARS:
# For WASM we provide flags using just environment variables.
# If we provide them using cmake vars, it will override emscripten toolchain flags.
# Unsure if we need this in general even for non-WASM builds.
cmake_flags.extend(
[
f"-DCMAKE_CXX_FLAGS='{os.environ['CXXFLAGS']}'",
f"-DCMAKE_C_FLAGS='{os.environ['CFLAGS']}'",
]
)
if not any("BUILD_SHARED_LIBS" in f for f in cmake_args):
cmake_flags.append(
f"-DBUILD_SHARED_LIBS={OFF_ON[not BUILD_STATIC]}",
)
run(
[
*wasm,
"cmake",
P,
*cmake_flags,
*cmake_args,
f"-DCMAKE_BUILD_TYPE={BUILD_CFG}",
f"-DCMAKE_SHARED_LINKER_FLAGS={os.environ['LDFLAGS']}",
],
cwd=cwd,
)
def git_clone_or_pull_repository(clone_url: str, target_dir: str, revision: str | None = None) -> None:
"""Lazily clones the `git` repository denoted by `clone_url` into
the `target_dir` or pulls latest changes if the `target_dir` exists (naively assumes
that a working clone exists there) and optionally checks out a revision
`revision` after cloning or in the existing clone if `revision` is not
`None`."""
if not os.path.exists(target_dir):
logger.info(f"cloning '{clone_url}' into '{target_dir}'")
run([git, "clone", "--recursive", clone_url, target_dir])
else:
logger.info(f"directory '{target_dir}' already cloned. Pulling latest changes.")
run([git, "-C", target_dir, "fetch", "--all", "--tags", "--force"])
# detect whether we are on a branch and pull
if run([git, "rev-parse", "--abbrev-ref", "HEAD"], cwd=target_dir) != "HEAD":
run([git, "pull", clone_url], cwd=target_dir)
if revision != None:
run([git, "reset", "--hard"], cwd=target_dir)
run([git, "fetch", "--all"], cwd=target_dir)
run([git, "checkout", revision], cwd=target_dir)
def build_dependency(
name: str,
mode: Literal[
"cmake",
"autoconf",
"bjam",
],
build_tool_args: list[str],
download_url: str,
download_name: str,
*,
download_tool: Literal["py", "git"] = download_tool_default,
revision: str | None = None,
patch: list[str] | None = None,
shell=None,
pre_compile_subs: Sequence[tuple[str, str, str]] = (),
additional_files: dict[str, str] | None = None,
no_append_name=False,
cmake_dir=None,
**kwargs,
) -> None:
"""Handles building of dependencies with different tools (which are
distinguished with the `mode` argument. `build_tool_args` is expected to be
a list which is necessary in order to not mess up quoting of compiler and
linker flags.
:param pre_compile_subs: A sequence of ``(fn, before, after)``
:param additional_files: Mapping path->url.
:param kwargs: Additional ``mode`` related kwargs.
"""
check_dir = os.path.join(DEPS_DIR, "install", name)
if os.path.exists(check_dir):
logger.info(f"Found existing {name}, skipping")
return
build_dir = os.path.join(DEPS_DIR, "build")
if not os.path.exists(build_dir):
os.makedirs(build_dir)
logger.info(f"\rFetching {name}... ")
if download_tool == download_tool_py:
if no_append_name:
url = download_url
else:
url = os.path.join(download_url, download_name)
download_path = os.path.join(build_dir, download_name)
if not os.path.exists(download_path):
for _ in range(3):
try:
urlretrieve(url, os.path.join(build_dir, download_path))
break
except ConnectionError as e:
print(e, "... retrying...")
time.sleep(30.0)
continue
else:
logger.info(
f"Download '{download_path}' already exists, assuming it's an undamaged download and that it has been extracted if possible, skipping"
)
elif download_tool == download_tool_git:
logger.info(f"\rChecking {name}... ")
git_clone_or_pull_repository(download_url, target_dir=os.path.join(build_dir, download_name), revision=revision)
else:
raise ValueError(f"download tool '{download_tool}' is not supported")
download_dir = os.path.join(build_dir, download_name)
if os.path.isdir(download_dir):
extract_dir_name = download_name
extract_dir = os.path.join(build_dir, extract_dir_name)
else:
download_tarfile_path = os.path.join(build_dir, download_name)
if download_name.endswith(".tar.gz") or download_name.endswith(".tgz"):
compr = "gz"
elif download_name.endswith(".tar.bz2"):
compr = "bz2"
elif download_name.endswith(".tar.xz"):
compr = "xz"
else:
raise RuntimeError("fix source for new download type")
# ty: false positive bug upstream.
download_tarfile = tarfile.open(
name=download_tarfile_path, mode=f"r:{compr}"
) # ty:ignore[no-matching-overload]
# tarfile seriously doesn't have a function to retrieve the root directory more easily
extract_dir_name = os.path.commonprefix([x for x in download_tarfile.getnames() if x != "."])
# run([tar, "--exclude=\"*/*\"", "-tf", download_name], cwd=build_dir).strip() no longer works
if extract_dir_name is None:
extract_dir_name = run(
[bash, "-c", f"tar -tf {download_name} 2> /dev/null | head -n 1 | cut -f1 -d /"], cwd=build_dir
)
extract_dir = os.path.join(build_dir, extract_dir_name)
if not os.path.exists(extract_dir):
run([tar, "-xf", download_name], cwd=build_dir)
if additional_files:
for path, url in additional_files.items():
if not os.path.exists(path):
urlretrieve(url, os.path.join(extract_dir, path))
if patch is not None:
for p in patch:
patch_abs = (SCRIPT_PATH / p).absolute().__str__()
if os.path.exists(patch_abs):
try:
run(["patch", "-p1", "--batch", "--forward", "-i", patch_abs], cwd=extract_dir)
except Exception as e:
# Assert that the patch has already been applied
run(["patch", "-p1", "--batch", "--reverse", "--dry-run", "-i", patch_abs], cwd=extract_dir)
else:
raise FileNotFoundError(patch_abs)
if shell is not None:
sp.run(shell, shell=True, check=True, cwd=extract_dir)
if mode != "bjam":
extract_build_dir = os.path.join(extract_dir, *([cmake_dir] if cmake_dir else []), "build")
if os.path.exists(extract_build_dir):
shutil.rmtree(extract_build_dir)
os.makedirs(extract_build_dir)
logger.info(f"\rConfiguring {name}...")
if mode == "autoconf":
run_autoconf(name, build_tool_args, cwd=extract_build_dir)
elif mode == "cmake":
run_cmake(name, build_tool_args, cwd=extract_build_dir)
else:
raise ValueError()
for fn, before, after in pre_compile_subs:
with open(os.path.join(extract_dir, fn), "r") as f:
s = f.read()
s = s.replace(before, after)
with open(os.path.join(extract_dir, fn), "w") as f:
f.write(s)
logger.info(f"\rBuilding {name}... ")
run([make, f"-j{IFCOS_NUM_BUILD_PROCS}", "VERBOSE=1"], cwd=extract_build_dir)
logger.info(f"\rInstalling {name}... ")
run([make, "install"], cwd=extract_build_dir)
logger.info(f"\rInstalled {name} \n")
else: # bjam
logger.info(f"\rConfiguring {name}...")
run([bash, "./bootstrap.sh"], cwd=extract_dir)
logger.info(f"\rBuilding {name}... ")
run(["./b2", f"-j{IFCOS_NUM_BUILD_PROCS}"] + build_tool_args, cwd=extract_dir, can_fail="wasm" in flags)
logger.info(f"\rInstalling {name}... ")
shutil.copytree(
os.path.join(extract_dir, "boost"), os.path.join(DEPS_DIR, "install", f"boost-{BOOST_VERSION}", "boost")
)
logger.info(f"\rInstalled {name} \n")
if "diskcleanup" in flags:
shutil.rmtree(build_dir, ignore_errors=True)
def get_qt6_aqt_config() -> tuple[str, str, str]:
if platform.system() != "Linux":
raise ValueError("Automatic Qt6 installation with aqtinstall is only configured for Linux builds.")
machine = platform.machine().lower()
if machine in {"x86_64", "amd64"}:
return "linux", "linux_gcc_64", "gcc_64"
if machine in {"aarch64", "arm64"}:
return "linux_arm64", "linux_gcc_arm64", "gcc_arm64"
raise ValueError(f"Automatic Qt6 installation is not configured for architecture '{platform.machine()}'.")
def install_qt6() -> str:
# If the caller pre-set QT_DIR (e.g. macOS CI using Homebrew-installed
# Qt6), validate it points at a real Qt6 install and skip aqtinstall
# entirely. The aqt download is only wired for Linux; on macOS/Windows
# the supported flow is a pre-installed Qt6 advertised via QT_DIR.
preset_qt_dir = os.environ.get("QT_DIR", "").strip()
if preset_qt_dir:
preset_qt_config = Path(preset_qt_dir) / "lib" / "cmake" / "Qt6" / "Qt6Config.cmake"
if preset_qt_config.exists():
logger.info(f"Using pre-set QT_DIR={preset_qt_dir}, skipping aqt install")
return preset_qt_dir
logger.warning(
f"QT_DIR={preset_qt_dir} is set but {preset_qt_config} not found; " f"falling through to aqtinstall"
)
host, qt_arch, install_suffix = get_qt6_aqt_config()
qt_install_root = INSTALL_DIR / f"qt6-{QT6_VERSION}-{install_suffix}"
qt_dir = qt_install_root / QT6_VERSION / install_suffix
os.environ["QT_DIR"] = str(qt_dir)
qt_config = qt_dir / "lib" / "cmake" / "Qt6" / "Qt6Config.cmake"
qt_core = qt_dir / "lib" / "libQt6Core.so.6"
qt_svg = qt_dir / "lib" / "cmake" / "Qt6Svg" / "Qt6SvgConfig.cmake"
if qt_config.exists() and qt_core.exists() and qt_svg.exists():
logger.info(f"Found existing Qt6 at {qt_dir}, skipping")
return str(qt_dir)
os.makedirs(qt_install_root, exist_ok=True)
try:
import aqt # ty:ignore[unresolved-import]
except ModuleNotFoundError:
logger.error(
"Could not find an existing Qt6 install, so aqtinstall is needed to fetch it automatically. "
"Install the `aqtinstall` PyPI package or set QT_DIR."
)
exit(1)
run(
[
sys.executable,
"-m",
"aqt",
"install-qt",
host,
"desktop",
QT6_VERSION,
qt_arch,
"-O",
str(qt_install_root),
# Keep the install lean by filtering archives: qtbase provides
# Core/Gui/Widgets (and the Qt6::CorePrivate target), qtsvg provides
# Qt6::Svg. Both are base-Qt archives, not add-on modules.
"--archives",
"icu",
"qtbase",
"qtsvg",
]
)
if not (qt_config.exists() and qt_core.exists() and qt_svg.exists()):
raise RuntimeError(f"Qt6 installation did not produce a usable Qt at {qt_dir}.")
return str(qt_dir)
cecho("Collecting dependencies:", GREEN)
# Set compiler flags for 32bit builds on 64bit system
# TODO: This is untested
ADDITIONAL_ARGS = []
if platform.system() == "Darwin":
ADDITIONAL_ARGS = [f"-mmacosx-version-min={TOOLSET}"] + ADDITIONAL_ARGS
# If the linker supports GC sections, set it up to reduce binary file size
# -fPIC is required for the shared libraries to work
compiler_flags = "CFLAGS", "CXXFLAGS", "LDFLAGS"
CXXFLAGS = os.environ.get("CXXFLAGS", "")
CFLAGS = os.environ.get("CFLAGS", "")
LDFLAGS = os.environ.get("LDFLAGS", "")
ADDITIONAL_ARGS_STR = " ".join(ADDITIONAL_ARGS)
CXXFLAGS_MINIMAL = f"{CXXFLAGS} {PIC} {ADDITIONAL_ARGS_STR}"
CFLAGS_MINIMAL = f"{CFLAGS} {PIC} {ADDITIONAL_ARGS_STR}"
if "wasm" in flags:
# WASM `SIDE_MODULE_` are absorbed by `emcmake` automatically.
CXXFLAGS = CXXFLAGS_MINIMAL
CFLAGS = CFLAGS_MINIMAL
elif sp.call([bash, "-c", "ld --gc-sections 2>&1 | grep -- --gc-sections &> /dev/null"]) != 0:
if BUILD_STATIC:
CXXFLAGS = f"{CXXFLAGS} {PIC} -fdata-sections -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden {ADDITIONAL_ARGS_STR}"
CFLAGS = f"{CFLAGS} {PIC} -fdata-sections -ffunction-sections -fvisibility=hidden {ADDITIONAL_ARGS_STR}"
else:
CXXFLAGS = CXXFLAGS_MINIMAL
CFLAGS = CFLAGS_MINIMAL
LDFLAGS = f"{LDFLAGS} -Wl,--gc-sections {ADDITIONAL_ARGS_STR}"
else:
if BUILD_STATIC:
CXXFLAGS = f"{CXXFLAGS} {PIC} -fvisibility=hidden -fvisibility-inlines-hidden {ADDITIONAL_ARGS_STR}"
CFLAGS = f"{CFLAGS} {PIC} -fvisibility=hidden -fvisibility-inlines-hidden {ADDITIONAL_ARGS_STR}"
else:
CXXFLAGS = CXXFLAGS_MINIMAL
CFLAGS = CFLAGS_MINIMAL
LDFLAGS = f"{LDFLAGS} {ADDITIONAL_ARGS_STR}"
if "lto" in flags:
for f in compiler_flags:
locals()[f] += f" -flto={IFCOS_NUM_BUILD_PROCS}"
os.environ["CXXFLAGS"] = CXXFLAGS
os.environ["CPPFLAGS"] = CXXFLAGS
os.environ["CFLAGS"] = CFLAGS
os.environ["LDFLAGS"] = LDFLAGS
# Use colors in logs to make them more readable locally.
# Output from compiler used by cmake.
os.environ["CMAKE_COLOR_DIAGNOSTICS"] = "ON"
# Output from cmake itself.
os.environ["CLICOLOR_FORCE"] = "1"
# Some dependencies need a more recent CMake version than most distros provide
# @tfk: this is no longer needed
# build_dependency(name="cmake-%s" % (CMAKE_VERSION,), mode="autoconf", build_tool_args=[], download_url="https://cmake.org/files/v%s" % (CMAKE_VERSION_2,), download_name="cmake-%s.tar.gz" % (CMAKE_VERSION,))
if "json" in targets:
dependency_name = f"json-{JSON_VERSION}"
build_dependency(
name=dependency_name,
mode="cmake",
build_tool_args=[
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/{dependency_name}",
"-DJSON_BuildTests=OFF",
],
download_url=f"https://github.com/nlohmann/json/releases/download/v{JSON_VERSION}",
download_name="json.tar.xz",
)
if "eigen" in targets:
dependency_name = f"eigen-install-{EIGEN_VERSION}"
build_dependency(
name=f"{dependency_name}",
mode="cmake",
# We add '-install-' in the middle, so it won't be confused with git repo we used previously.
build_tool_args=[
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/{dependency_name}",
],
download_url=f"https://gitlab.com/libeigen/eigen/-/archive/{EIGEN_VERSION}/",
download_name=f"eigen-{EIGEN_VERSION}.tar.gz",
)
if "pcre" in targets:
OLD_CC, OLD_CXX = None, None
if MAC_CROSS_COMPILE_INTEL:
OLD_CC, OLD_CXX = os.environ.get("CC"), os.environ.get("CXX")
os.environ["CC"] = MAC_CROSS_COMPILE_INTEL_CC
os.environ["CXX"] = MAC_CROSS_COMPILE_INTEL_CXX
# Keep it autoconf as OpenCOLLADA is pretty old and might break
# if we update it's dependencies for mmore modern cmake.
build_dependency(
name=f"pcre-{PCRE_VERSION}",
mode="autoconf",
build_tool_args=[DISABLE_FLAG],
download_url=f"https://downloads.sourceforge.net/project/pcre/pcre/{PCRE_VERSION}/",
download_name=f"pcre-{PCRE_VERSION}.tar.bz2",
)
if MAC_CROSS_COMPILE_INTEL:
restore_env("CC", OLD_CC)
restore_env("CXX", OLD_CXX)
if "swig" in targets:
dependency_name = f"swig-{SWIG_VERSION}"
build_dependency(
name=dependency_name,
mode="cmake",
build_tool_args=[
"-DWITH_PCRE=OFF",
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/{dependency_name}",
],
download_url="https://github.com/swig/swig.git",
download_name="swig",
download_tool=download_tool_git,
revision=f"v{SWIG_VERSION}",
)
if USE_OCCT and "occ" in targets:
occt_args: list[str] = []
patches: list[str] = []
if OCCT_VERSION < "7.4":
patches.append("./patches/occt/enable-exception-handling.patch")
# Skip ExpToCasExe as we don't need it and it requires additional dependencies.
# Before 7.7.2 ExpToCasExe is part of DataExchange, DETools doesn't exist yet.
# Since we do need DataExchange (used for IgesSerializer), we use a patch to skip only ExpToCasExe.
if "7.7.2" > OCCT_VERSION >= "7.7":
patches.append("./patches/occt/no_ExpToCasExe.patch")
elif OCCT_VERSION >= "7.7.2":
occt_args.append("-DBUILD_MODULE_DETools=OFF")
if "wasm" in flags:
patches.append("./patches/occt/no_em_js.patch")
build_dependency(
name=f"occt-{OCCT_VERSION}",
mode="cmake",
build_tool_args=[
f"-DINSTALL_DIR={DEPS_DIR}/install/occt-{OCCT_VERSION}",
f"-DBUILD_LIBRARY_TYPE={LINK_TYPE_UCFIRST}",
f"-DBUILD_MODULE_Draw=0",
f"-DBUILD_RELEASE_DISABLE_EXCEPTIONS=Off",
# Disable xlib explicitly, as it tries to use it on Desktop Ubuntu, adding unnecessary dependency.
f"-DUSE_XLIB=OFF",
# Avoid building 3D Viewer.
f"-DUSE_FREETYPE=OFF",
f"-DUSE_OPENGL=OFF",
f"-DUSE_GLES2=OFF",
f"-DCMAKE_POLICY_VERSION_MINIMUM=3.5",
*MAC_CROSS_COMPILE_INTEL_ARGS,
*occt_args,
],
download_url="https://github.com/Open-Cascade-SAS/OCCT",
download_name="occt",
download_tool=download_tool_git,
patch=patches,
revision="V" + OCCT_VERSION.replace(".", "_"),
)
elif "occ" in targets:
build_dependency(
name=f"oce-{OCE_VERSION}",
mode="cmake",
build_tool_args=[
f"-DOCE_DISABLE_TKSERVICE_FONT=ON",
f"-DOCE_TESTING=OFF",
f"-DOCE_BUILD_SHARED_LIB=OFF",
f"-DOCE_DISABLE_X11=ON",
f"-DOCE_VISUALISATION=OFF",
f"-DOCE_OCAF=OFF",
f"-DOCE_INSTALL_PREFIX={DEPS_DIR}/install/oce-{OCE_VERSION}",
],
download_url="https://github.com/tpaviot/oce/archive/",
download_name=f"OCE-{OCE_VERSION}.tar.gz",
)
if "manifold" in targets:
dependency_name = f"manifold-{MANIFOLD_VERSION}"
patches = []
if WASM:
patches.append("./patches/manifold/install-metadata-for-emscripten.patch")
build_dependency(
name=dependency_name,
mode="cmake",
build_tool_args=[
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/{dependency_name}",
"-DMANIFOLD_PAR=OFF",
"-DMANIFOLD_CROSS_SECTION=OFF",
"-DMANIFOLD_PYBIND=OFF",
"-DMANIFOLD_JSBIND=OFF",
"-DMANIFOLD_CBIND=OFF",
"-DMANIFOLD_TEST=OFF",
"-DMANIFOLD_EXPORT=OFF",
"-DMANIFOLD_DOWNLOADS=OFF",
*MAC_CROSS_COMPILE_INTEL_ARGS,
],
download_url="https://github.com/elalish/manifold.git",
download_name="manifold",
download_tool=download_tool_git,
revision=f"v{MANIFOLD_VERSION}",
patch=patches,
)
if "libxml2" in targets:
OLD_CC = ""
if MAC_CROSS_COMPILE_INTEL:
OLD_CC = os.environ.get("CC")
os.environ["CC"] = MAC_CROSS_COMPILE_INTEL_CC
build_tool_args = [
"--without-python",
ENABLE_FLAG,
DISABLE_FLAG,
"--without-zlib",
"--without-iconv",
"--without-lzma",
]
if "wasm" in flags:
build_tool_args.append("--without-threads")
build_dependency(
f"libxml2-{LIBXML2_VERSION}",
"autoconf",
build_tool_args=build_tool_args,
download_url=f"https://download.gnome.org/sources/libxml2/{'.'.join(LIBXML2_VERSION.split('.')[0:2])}/",
download_name=f"libxml2-{LIBXML2_VERSION}.tar.xz",
)
if MAC_CROSS_COMPILE_INTEL:
if OLD_CC is None:
del os.environ["CC"]
else:
os.environ["CC"] = OLD_CC
if "OpenCOLLADA" in targets:
patches = ["./patches/opencollada/pr622_and_disable_subdirs.patch"]
# OpenCOLLADAConfig.cmake.in hardcodes shared-lib targets on Unix regardless of
# whether shared libs were actually built. We make it follow `USE_SHARED` instead.
patches.append("./patches/opencollada/config_select_libs_by_use_shared.patch")
if "wasm" in flags:
# This is necessary for the WASM build, because recent versions of
# clang don't have the tr1:: namespace anymore. However, it breaks
# some versions of gcc (9.4.0 at least) due to specializing std::hash
# outside of the std:: namespace.
patches.append("./patches/opencollada/remove_tr1.patch")
build_dependency(
"OpenCOLLADA",
"cmake",
build_tool_args=[
f"-DLIBXML2_INCLUDE_DIR={DEPS_DIR}/install/libxml2-{LIBXML2_VERSION}/include/libxml2",
f"-DLIBXML2_LIBRARIES={DEPS_DIR}/install/libxml2-{LIBXML2_VERSION}/lib/libxml2.{LIBRARY_EXT}",
f"-DPCRE_INCLUDE_DIR={DEPS_DIR}/install/pcre-{PCRE_VERSION}/include",
f"-DPCRE_PCREPOSIX_LIBRARY={DEPS_DIR}/install/pcre-{PCRE_VERSION}/lib/libpcreposix.{LIBRARY_EXT}",
f"-DPCRE_PCRE_LIBRARY={DEPS_DIR}/install/pcre-{PCRE_VERSION}/lib/libpcre.{LIBRARY_EXT}",
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/OpenCOLLADA/",
# OpenCOLLADA is ancient at this point and allows cmake 2.6+, which results in error in cmake 4.
f"-DCMAKE_POLICY_VERSION_MINIMUM=3.5",
f"-DUSE_STATIC={OFF_ON[BUILD_STATIC]}",
f"-DUSE_SHARED={OFF_ON[not BUILD_STATIC]}",
*MAC_CROSS_COMPILE_INTEL_ARGS,
],
download_url="https://github.com/KhronosGroup/OpenCOLLADA.git",
download_name="OpenCOLLADA",
download_tool=download_tool_git,
patch=patches,
revision=OPENCOLLADA_VERSION,
)
if "python" in targets and not USE_CURRENT_PYTHON_VERSION and "wasm" not in flags:
# Python should not be built with -fvisibility=hidden, from experience that introduces segfaults
OLD_CPP_FLAGS = os.environ["CPPFLAGS"]
OLD_CXX_FLAGS = os.environ["CXXFLAGS"]
OLD_C_FLAGS = os.environ["CFLAGS"]
os.environ["CXXFLAGS"] = CXXFLAGS_MINIMAL
os.environ["CPPFLAGS"] = CXXFLAGS_MINIMAL
os.environ["CFLAGS"] = CFLAGS_MINIMAL
# On OSX a dynamic python library is built or it would not be compatible
# with the system python because of some threading initialization
PYTHON_CONFIGURE_ARGS: list[str] = []
original_path = ""
if platform.system() == "Darwin":
PYTHON_CONFIGURE_ARGS = ["--enable-shared"]
open_ssl_prefix = run([brew, "--prefix", "openssl@3"]).strip()
# I'm not sure why, but if I do `"{open_ssl_prefix}"` (keep the quotes),
# autconf fails to find ssl.
PYTHON_CONFIGURE_ARGS.append(f"--with-openssl={open_ssl_prefix}")
if MAC_CROSS_COMPILE_INTEL:
original_path = os.environ["PATH"]
# Need to ensure python will pick up intel's `pkg-config`,
# otherwise it might attempt to use ARM libraries (e.g. `zstd`) and fail.
os.environ["PATH"] = f"{MAC_INTEL_BIN_PATH}{os.pathsep}{original_path}"
PYTHON_CONFIGURE_ARGS.extend(["--with-universal-archs=intel-64", "--enable-universalsdk"])
for PYTHON_VERSION in PYTHON_VERSIONS:
# Don't fail silently on missing Python dependencies (e.g. openssl or zlib),
# because later ifcopenshell-python build will fail too but in a more confusing way.
build_dependency(
f"python-{PYTHON_VERSION}",
"autoconf",
PYTHON_CONFIGURE_ARGS,
f"http://www.python.org/ftp/python/{PYTHON_VERSION}/",
f"Python-{PYTHON_VERSION}.tgz",
)
python_install = INSTALL_DIR / f"python-{PYTHON_VERSION}"
python_bin = python_install / "bin" / "python3"
# `_ssl` module is present -> we will be able to install `numpy` later
# to verify IfcOpenShell installation
try:
run([str(python_bin), "-c", "import _ssl"])
except RuntimeError:
print(
"ERROR: Python was built without SSL support (_ssl module is missing). "
f"To fix this: remove the installed Python at {python_install}; "
"install OpenSSL development libraries and re-run."
)
raise
if MAC_CROSS_COMPILE_INTEL:
assert original_path
os.environ["PATH"] = original_path
os.environ["CPPFLAGS"] = OLD_CPP_FLAGS
os.environ["CXXFLAGS"] = OLD_CXX_FLAGS
os.environ["CFLAGS"] = OLD_C_FLAGS
if "boost" in targets:
str_concat = lambda prefix: lambda postfix: "" if postfix.strip() == "" else "=".join((prefix, postfix.strip()))
toolset = []
if "wasm" in flags:
toolset.append("toolset=emscripten")
build_dependency(
f"boost-{BOOST_VERSION}",
mode="bjam",
build_tool_args=[
f"--stagedir={DEPS_DIR}/install/boost-{BOOST_VERSION}",
"--with-system",
"--with-program_options",
"--with-regex",
"--with-thread",
"--with-date_time",
"--with-iostreams",
"--with-filesystem",
f"link={LINK_TYPE}",
*toolset,
*map(str_concat("cxxflags"), CXXFLAGS.strip().split(" ")),
*map(str_concat("linkflags"), LDFLAGS.strip().split(" ")),
"stage",
"-s",
"NO_BZIP2=1",
*MAC_CROSS_COMPILE_INTEL_BJAM_ARGS,
],
download_url=BOOST_LOCATION,
# don't remember what this is, but fail on 1.86
# patch="./patches/boost/boostorg_regex_62.patch",
download_name=f"boost-{BOOST_VERSION}-b2-nodocs.tar.gz",
)
if "wasm" in flags:
# only supported on nix for now
run(
("find", ".", "-name", "*.bc", "-exec", "bash", "-c", "emar q ${1%.bc}.a $1", "bash", "{}", ";"),
cwd=f"{DEPS_DIR}/install/boost-{BOOST_VERSION}/lib",
)
if "cgal" in targets:
gmp_args: list[str] = []
mpfr_args: list[str] = []
OLD_HOST_CC = None
if WASM:
if APPLE:
# Override `HOST_CC`, otherwise `emcc` will try to use it's own `clang` which can only build
# wasm executables and build will fail.
os.environ["HOST_CC"] = "clang"
# Disable assembly, otherwise `emcc -c conftest.s` will crash due to assembly mismatch.
gmp_args.extend(("--disable-assembly", "--enable-cxx"))
mpfr_args.extend(("--host", "none"))
elif "x86" in arch:
gmp_args.append("--enable-fat") # See issues #7458 #7556
OLD_CC = None
if MAC_CROSS_COMPILE_INTEL:
OLD_CC = os.environ.get("CC")
# Otherwise it's using arm64 `gcc` and fails to build gmp.
os.environ["CC"] = MAC_CROSS_COMPILE_INTEL_CC
gmp_args.extend(MAC_CROSS_COMPILE_INTEL_AUTOCONF_HOST_ARGS)
# Fixes configure failing to find a working compiler under GCC 15's default -std=gnu23.
# Issue presumably will be resolved in any next gmp version, but currently the last one is 6.3.0.
# Patch is just applying fix from upstream meantion below:
# https://gmplib.org/list-archives/gmp-bugs/2025-February/005561.html
gmp_patches = ["./patches/gmp/001-fix-std23.patch"]
if GMP_VERSION != "6.3.0":
raise Exception(f"GMP_VERSION changed to {GMP_VERSION}, check whether {gmp_patches} is still needed.")
build_dependency(
name=f"gmp-{GMP_VERSION}",
mode="autoconf",
build_tool_args=[ENABLE_FLAG, DISABLE_FLAG, "--with-pic", *gmp_args],
pre_compile_subs=(
[("build/config.h", "HAVE_OBSTACK_VPRINTF 1", "HAVE_OBSTACK_VPRINTF 0")] if "wasm" in flags else []
),
patch=gmp_patches,
# Sometimes ftp.gnu.org is very slow, use ftpmirror.gnu.org as a workaround.
download_url="https://ftpmirror.gnu.org/gnu/gmp/",
download_name=f"gmp-{GMP_VERSION}.tar.bz2",
)
if WASM and APPLE:
restore_env("HOST_CC", OLD_HOST_CC)
build_dependency(
name=f"mpfr-{MPFR_VERSION}",
mode="autoconf",
build_tool_args=[ENABLE_FLAG, DISABLE_FLAG, *mpfr_args, f"--with-gmp={DEPS_DIR}/install/gmp-{GMP_VERSION}"],
download_url=f"http://www.mpfr.org/mpfr-{MPFR_VERSION}/",
download_name=f"mpfr-{MPFR_VERSION}.tar.bz2",
)
if MAC_CROSS_COMPILE_INTEL:
restore_env("CC", OLD_CC)
build_dependency(
name=f"cgal-{CGAL_VERSION}",
mode="cmake",
build_tool_args=[
f"-DGMP_LIBRARIES={DEPS_DIR}/install/gmp-{GMP_VERSION}/lib/libgmp.{LIBRARY_EXT}",
f"-DGMP_INCLUDE_DIR={DEPS_DIR}/install/gmp-{GMP_VERSION}/include",
f"-DMPFR_LIBRARIES={DEPS_DIR}/install/mpfr-{MPFR_VERSION}/lib/libmpfr.{LIBRARY_EXT}",
f"-DMPFR_INCLUDE_DIR={DEPS_DIR}/install/mpfr-{MPFR_VERSION}/include",
f"-DBoost_INCLUDE_DIR={DEPS_DIR}/install/boost-{BOOST_VERSION}",
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/cgal-{CGAL_VERSION}/",
f"-DCGAL_HEADER_ONLY=On",
f"-DBUILD_SHARED_LIBS=Off",
],
download_url="https://github.com/CGAL/cgal.git",
download_name="cgal",
download_tool=download_tool_git,
revision=CGAL_VERSION,
)
if "usd" in targets:
build_dependency(
name=f"oneTBB-{TBB_VERSION}",
mode="cmake",
build_tool_args=[f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/tbb-{TBB_VERSION}", f"-DTBB_TEST=OFF"],
download_url="https://github.com/oneapi-src/oneTBB",
download_name="oneTBB",
download_tool=download_tool_git,
revision=f"v{TBB_VERSION}",
)
build_dependency(
name=f"usd-{USD_VERSION}",
mode="cmake",
build_tool_args=[
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/usd-{USD_VERSION}",
f"-DBOOST_ROOT={DEPS_DIR}/install/boost-{BOOST_VERSION}",
f"-DTBB_ROOT_DIR={DEPS_DIR}/install/tbb-{TBB_VERSION}",
f"-DPXR_ENABLE_PYTHON_SUPPORT=FALSE",
f"-DPXR_ENABLE_GL_SUPPORT=FALSE",
f"-DPXR_BUILD_IMAGING=FALSE",
f"-DPXR_BUILD_TUTORIALS=FALSE",
f"-DPXR_BUILD_EXAMPLES=FALSE",
f"-DPXR_BUILD_USD_TOOLS=FALSE",
f"-DPXR_BUILD_TESTS=FALSE",
],
download_url="https://github.com/PixarAnimationStudios/USD",
download_name="USD",
download_tool=download_tool_git,
revision=f"v{USD_VERSION}",
)
if "zstd" in targets:
build_dependency(
name=f"zstd-{ZSTD_VERSION}",
mode="cmake",
build_tool_args=[
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/zstd-{ZSTD_VERSION}",
f"-DZSTD_BUILD_STATIC=ON",
f"-DZSTD_BUILD_SHARED=OFF",
f"-DCMAKE_INSTALL_LIBDIR=lib",
*MAC_CROSS_COMPILE_INTEL_ARGS,
],
cmake_dir="build/cmake/",
download_url="https://github.com/facebook/zstd",
download_name="zstd",
download_tool=download_tool_git,
revision=f"v{ZSTD_VERSION}",
)
if "rocksdb" in targets:
build_dependency(
name=f"rocksdb-{ROCKSDB_VERSION}",
mode="cmake",
build_tool_args=[
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/rocksdb-{ROCKSDB_VERSION}",
f"-DFAIL_ON_WARNINGS=Off",
f"-DWITH_TESTS=OFF",
f"-DWITH_TOOLS=OFF",
f"-DWITH_GFLAGS=OFF",
f"-DWITH_BENCHMARK_TOOLS=OFF",
f"-DWITH_CORE_TOOLS=OFF",
f"-DROCKSDB_BUILD_SHARED=Off",
f"-DCMAKE_POSITION_INDEPENDENT_CODE=On",
f"-DUSE_RTTI=On",
f"-DWITH_ZSTD=On",
f"-DPORTABLE=1",
f"-DCMAKE_PREFIX_PATH={DEPS_DIR}/install/zstd-{ZSTD_VERSION}",
*MAC_CROSS_COMPILE_INTEL_ARGS,
],
download_url="https://github.com/facebook/rocksdb",
download_name="rocksdb",
download_tool=download_tool_git,
revision=f"v{ROCKSDB_VERSION}",
)
if "qt6" in targets:
install_qt6()
cecho("Building IfcOpenShell:", GREEN)
IFCOS_DIR = os.path.join(DEPS_DIR, "build", "ifcopenshell")
if os.environ.get("NO_CLEAN", "").lower() not in {"1", "on", "true"}:
if os.path.exists(IFCOS_DIR):
shutil.rmtree(IFCOS_DIR)
os.makedirs(IFCOS_DIR, exist_ok=True)
ifcos_build_dir = os.path.join(IFCOS_DIR, "build")
os.makedirs(ifcos_build_dir, exist_ok=True)
cmake_args = [
"-DUSE_MMAP=OFF",
"-DBUILD_EXAMPLES=OFF",
"-DBUILD_SHARED_LIBS=" + OFF_ON[not IFCOPENSHELL_STATIC],
"-DGLTF_SUPPORT=ON",
"-DBoost_NO_BOOST_CMAKE=On",
"-DCREATE_BUNDLE=On",
"-DADD_COMMIT_SHA=" + OFF_ON[ADD_COMMIT_SHA],
"-DVERSION_OVERRIDE=" + OFF_ON[ADD_COMMIT_SHA],
*MAC_CROSS_COMPILE_INTEL_ARGS,
]
"""Default CMake args to use for all CMake configs."""
cmake_args_prefix_path: list[str] = [
f"{DEPS_DIR}/install/boost-{BOOST_VERSION}",
f"{DEPS_DIR}/install/eigen-install-{EIGEN_VERSION}",
f"{DEPS_DIR}/install/json-{JSON_VERSION}",
]
def get_cmake_args_prefix_path(additional_paths: Sequence[str] = ()) -> list[str]:
args_prefix_path = cmake_args_prefix_path.copy()
args_prefix_path.extend(additional_paths)
prefix_path = ";".join(args_prefix_path)
if WASM:
# `emcmake` is disabling search in PATH, so we provide root paths instead.
# Provide '/' to PATH, so it will be combined with provided root paths,
# otherwise, depending on environment, it might not search the root path itself.
return [f"-DCMAKE_FIND_ROOT_PATH={prefix_path}", "-DCMAKE_PREFIX_PATH=//"]
else:
return [f"-DCMAKE_PREFIX_PATH={prefix_path}"]
if "wasm" in flags:
# Boost is built by the build script so should not be found
# inside of the sysroot set by the emscriptem toolchain
cmake_args.append("-DWASM_BUILD=On")
schemas = os.environ.get("IFCOS_SCHEMAS")
if schemas:
cmake_args.append(f"-DSCHEMA_VERSIONS={schemas}")
if "cgal" in targets:
cmake_args_prefix_path.append(f"{DEPS_DIR}/install/cgal-{CGAL_VERSION}")
cmake_args_prefix_path.append(f"{DEPS_DIR}/install/gmp-{GMP_VERSION}")
cmake_args_prefix_path.append(f"{DEPS_DIR}/install/mpfr-{MPFR_VERSION}")
cmake_args.append(f"-DCGAL_WITH_GMPXX=Off")
if "occ" in targets and USE_OCCT:
cmake_args_prefix_path.append(f"{DEPS_DIR}/install/occt-{OCCT_VERSION}")
elif "occ" in targets:
# We don't support find_package for OCE.
occ_include_dir = f"{DEPS_DIR}/install/oce-{OCE_VERSION}/include/oce"
occ_library_dir = f"{DEPS_DIR}/install/oce-{OCE_VERSION}/lib"
cmake_args.extend(["-DOCC_INCLUDE_DIR=" + occ_include_dir, "-DOCC_LIBRARY_DIR=" + occ_library_dir])
if "manifold" in targets:
cmake_args_prefix_path.append(f"{DEPS_DIR}/install/manifold-{MANIFOLD_VERSION}")
cmake_args.append("-DWITH_MANIFOLD=On")
if "OpenCOLLADA" in targets:
# pcre is a dependency of OpenCOLLADA, but since we `find_package`,
# we don't need to add it explicitly here as cmake will find it from the config.
cmake_args_prefix_path.append(f"{DEPS_DIR}/install/OpenCOLLADA")
else:
cmake_args.extend(
[
f"-DCOLLADA_SUPPORT=Off",
]
)
if "libxml2" in targets:
cmake_args_prefix_path.append(f"{DEPS_DIR}/install/libxml2-{LIBXML2_VERSION}")
if "usd" in targets:
cmake_args.append("-DUSD_SUPPORT=ON")
cmake_args_prefix_path.extend(
[
f"{DEPS_DIR}/install/tbb-{TBB_VERSION}",
f"{DEPS_DIR}/install/usd-{USD_VERSION}",
]
)
if "rocksdb" in targets:
cmake_args.extend(
[
f"-DWITH_ROCKSDB=On",
f"-DWITH_ZSTD=On",
]
)
cmake_args_prefix_path.extend(
[
f"{DEPS_DIR}/install/rocksdb-{ROCKSDB_VERSION}",
f"{DEPS_DIR}/install/zstd-{ZSTD_VERSION}",
]
)
if "swig" in targets:
cmake_args_prefix_path.append(f"{DEPS_DIR}/install/swig-{SWIG_VERSION}")
if os.environ.get("QT_DIR"):
cmake_args_prefix_path.append(os.environ["QT_DIR"])
cmake_args.append(f"-DQT_DIR={os.environ['QT_DIR']}")
ifcos_build_args = [
f"-DBUILD_IFCGEOM={OFF_ON['IfcGeom' in targets]}",
f"-DBUILD_GEOMSERVER={OFF_ON['IfcGeomServer' in targets]}",
f"-DBUILD_CONVERT={OFF_ON['IfcConvert' in targets]}",
f"-DBUILD_BONSAIVIEWER={OFF_ON['BonsaiViewer' in targets]}",
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/ifcopenshell",
]
if not WASM and (
"BonsaiViewer" in targets
or not explicit_targets
or {"IfcGeom", "IfcConvert", "IfcGeomServer", "BonsaiViewer"} & set(explicit_targets)
):
logger.info("\rConfiguring executables...")
exec_args = [
*ifcos_build_args,
f"-DBUILD_IFCPYTHON=OFF",
]
run_cmake("", exec_args + cmake_args + get_cmake_args_prefix_path(), cmake_dir=CMAKE_DIR, cwd=ifcos_build_dir)
logger.info("\rBuilding executables... ")
run([make, f"-j{IFCOS_NUM_BUILD_PROCS}", "VERBOSE=1"], cwd=ifcos_build_dir)
run([make, "install/strip" if BUILD_CFG == "Release" else "install"], cwd=ifcos_build_dir)
if "IfcOpenShell-Python" in targets:
wrapper_ldflags = ""
if platform.system() == "Darwin":
# On OSX the actual Python library is not linked against.
wrapper_ldflags = "-Wl,-undefined,dynamic_lookup"
def compile_python_wrapper(
python_version: str,
python_include: str | None = None,
python_executable: str | None = None,
python_path: Path | None = None,
) -> str | None:
"""
:return: Path to module dir if ``python_executable`` was provided, otherwise ``None``.
"""
assert bool(python_path) ^ bool(python_include)
logger.info(f"\rConfiguring python {python_version} wrapper...")
if python_path:
# We couldn't just prefix PATH and have to provide all variables explicitly,
# see ifcwrap/cmake for the details.
python_executable = (Path(python_path) / "bin" / "python3").__str__()
python_include = run(
[
python_executable,
"-c",
"import sysconfig; print(sysconfig.get_config_var('INCLUDEPY'))",
]
)
assert python_include
old_ldflags = os.environ["LDFLAGS"]
if wrapper_ldflags:
os.environ["LDFLAGS"] = f"{old_ldflags} {wrapper_ldflags}"
try:
run_cmake(
"",
ifcos_build_args
+ [
"-DBUILD_IFCPYTHON=ON",
]
+ cmake_args
+ get_cmake_args_prefix_path()
+ [
*([f"-DPYTHON_EXECUTABLE={python_executable}"] if python_executable else []),
# Needed because pyodide is expecting setup.py to be in the root.
*([f"-DPYTHON_MODULE_INSTALL_DIR={REPO_PATH}"] * WASM),
f"-DPYTHON_INCLUDE_DIR={python_include}",
f"-DCMAKE_INSTALL_PREFIX={DEPS_DIR}/install/ifcopenshell/tmp",
"-DUSERSPACE_PYTHON_PREFIX="
+ ["Off", "On"][os.environ.get("PYTHON_USER_SITE", "").lower() in {"1", "on", "true"}],
],
cmake_dir=CMAKE_DIR,
cwd=ifcos_build_dir,
)
finally:
os.environ["LDFLAGS"] = old_ldflags
logger.info(f"\rBuilding python {python_version} wrapper... ")
run([make, f"-j{IFCOS_NUM_BUILD_PROCS}", "ifcopenshell_wrapper", "VERBOSE=1"], cwd=ifcos_build_dir)
run([make, "install/local"], cwd=os.path.join(ifcos_build_dir, "ifcwrap"))
if python_executable:
run([python_executable, "-m", "ensurepip"])
run([python_executable, "-m", "pip", "install", "--user", "numpy", "typing_extensions"])
module_dir = run(
[python_executable, "-c", "import inspect, ifcopenshell; print(inspect.getfile(ifcopenshell))"]
)
# Use just the last line is used,
# because output might contain warning like `No stream support: No module named 'lark'`.
module_dir = module_dir.strip().splitlines()[-1]
module_dir = os.path.dirname(module_dir)
if platform.system() != "Darwin":
if BUILD_CFG == "Release":
for so in glob.glob(os.path.join(module_dir, "*.so")):
if "wasm" in flags:
run(["wasm-strip", so, "-k", "dylink.0"])
elif os.path.basename(so).startswith("_ifcopenshell_wrapper"):
# TODO: This symbol name depends on the Python version?
run([strip, "-s", "-K", "PyInit__ifcopenshell_wrapper", so], cwd=module_dir)
else:
run([strip, "--strip-unneeded", so], cwd=module_dir)
return module_dir
if "wasm" in flags:
compile_python_wrapper(
run(["pyodide", "config", "get", "python_version"]),
run(["pyodide", "config", "get", "python_include_dir"]),
)
# Copy setup.py where pyodide build system expects it.
shutil.copy(REPO_PATH / "pyodide" / "setup.py", REPO_PATH)
# Empty pyproject so it's contents won't affect the resulting wheel
# otherwise the wheel will use version and dependencies from toml, not setup.py.
(REPO_PATH / "pyproject.toml").write_text("")
elif USE_CURRENT_PYTHON_VERSION:
python_info = sysconfig.get_paths()
compile_python_wrapper(platform.python_version(), python_info["include"], sys.executable)
else:
for python_version in PYTHON_VERSIONS:
python_path = INSTALL_DIR / f"python-{python_version}"
module_dir = compile_python_wrapper(python_version, python_path=python_path)
assert module_dir
# Not sure why, but added after reading this in the logs
# cp: /Users/runner/work/IfcOpenShell/IfcOpenShell/build/Darwin/x86_64/10.15/install/ifcopenshell/python-3.9.11: No such file or directory
# D'oh this was just due to a missing f-string f but doesn't hurt to keep it in.
run(["mkdir", "-p", os.path.join(DEPS_DIR, "install", "ifcopenshell")])
dest = os.path.join(DEPS_DIR, "install", "ifcopenshell", f"python-{python_version}")
if os.path.exists(dest):
shutil.rmtree(dest)
run([cp, "-R", module_dir, dest])
logger.info("\rBuilt IfcOpenShell...\n\n")
Reference in New Issue View Git Blame Copy Permalink