file::paged_reading(true), set before initialize(), runs the full parse
(serial or parallel) through the paged reader with 64 KB pages and a
4 MB cache instead of reading the whole file into memory; the whole
file is then never held. Every stage already reads through the reader,
so nothing else changes. The equality test now runs the same file paged,
serially and with five workers each holding its own page cache.
TXG 58 MB / 210_King 147 MB / OKgate22 231 MB: one thread 1.09 / 2.89 /
5.25 s against 1.07 / 2.75 / 5.12 s in memory, twelve threads 0.48 /
1.27 / 1.99 s against 0.44 / 1.24 / 2.01 s; peak memory 311 / 727 /
1119 MB against 365 / 871 / 1347 MB, that is, down by the size of the
file. Whether this should become the default is a decision the numbers
on the PR are meant to inform.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
The DATA section is split into one chunk per thread and each worker runs
the same per-instance reader as the serial parse over its own reader,
storage, inverse records and simple-type list; the results are merged in
file order, so instance order, GlobalId precedence and inverse records
are identical to the serial parse. Reference resolution then splits over
the same threads: each instance's slots are its own and the name table
is complete and read-only by then. The default is one thread per core,
capped at 16; IFCOPENSHELL_PARSE_THREADS or file::parse_threads()
overrides it, and 1 parses as before.
The instance headers are read by one loop, for_each_instance_header(),
shared with the lazy index: it looks declarations up once per keyword,
passes over a bypassed instance's attribute list and slides past a stray
keyword the way the serial reader does (the lazy index therefore no
longer falls back on one).
Finding the split points is the one place that looks at raw bytes rather
than tokens, because tokenizing the file serially first would leave
nothing to parallelise. It applies three rules: a string starts and ends
at a quote and cannot span a line, and a comment runs from /* to */; a
split is a '#' that starts a line outside both. Getting a string's end
wrong can only lose a candidate, never accept a wrong one, since no
string contains a newline. The equality test puts a comment holding a
fake instance and a string holding "/*" between the chunks.
file_reader gains for_each_span(), which hands a byte range out span by
span (one span for a buffer, one per page for the paged reader), and
reopen(), a reader over the same file for another thread.
TXG 58 MB / 210_King 147 MB / OKgate22 231 MB, 12 threads: 0.44 / 1.24 /
2.01 s against 1.07 / 2.75 / 5.12 s on one thread; memory after the parse
within 1–4%, peak +5–4%.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
A lazy open reads the DATA section once with the tokenizer's index
policy and builds what indexes the file: a shell per instance (name and
declaration, no attribute array), the complete inverse index with
attribute indices, the GlobalId map and the by-type lists. No attribute
value is decoded. The first time an instance's attributes are touched,
ensure_loaded() seeks the retained paged reader to the instance and runs
the same load_attributes() the full parse runs, with inverse registration
off, then resolves that instance's references from its own slots. A
modified instance is materialised first, so writing works.
There is no scanner of its own: the index pass consumes next<index_tokens>()
and counts parentheses and commas on the operator tokens; a keyword where
an instance should start, or a token the tokenizer rejects, stops the
index and the file is parsed in full. The offset of each instance's
attribute list is kept in one sorted vector that exists only in lazy
mode, so a full parse pays nothing for it. Materialising from several
threads at once is not safe.
TXG 58 MB / 210_King 147 MB / OKgate22 231 MB, single thread: lazy open
0.61 / 1.73 / 2.86 s against the full parse's 1.05 / 2.69 / 4.99 s, at
141 / 374 / 534 MB against 274 / 654 / 1036 MB; reading one attribute of
every instance afterwards costs a further 0.56 / 1.44 / 4.86 s.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
The parser could not resolve a #name when it read it, because the
instance may be defined further down the file, so it left the slot empty
and appended (owner, attribute, name) to a side table that a second pass
walked. The table held one entry per reference for the whole read: 64 MB
on a 58 MB model, the high-water mark of opening.
Now the reference stays where the tokenizer put it: the attribute slot
holds the instance_reference, or the reference_or_simple_type aggregate
for a list (mixed with inline typed values or not), until every instance
has been read, and resolve_instance_references() walks each instance's
slots and swaps names for instances. Ordering is what the tokenizer
produced; nothing is re-derived. A missing name becomes null in a scalar
and is dropped from an aggregate, as before; the error keeps its offset.
The three transient alternatives are appended to the attribute pack and
to argument_type in lock step and are never visible once a file is
loaded. Simple type instances read inline (IfcPropertySetDefinitionSet)
have their own slots, so their references need no diversion.
The table remains for the header entities and for streaming consumers of
instance_streamer::references(), which leave resolve_references_in_place
off.
TXG 58 MB / 210_King 147 MB / OKgate22 231 MB, single thread: time
unchanged (1.05 / 2.69 / 4.99 s), memory after the parse 287 -> 274,
698 -> 654, 1086 -> 1036 MB, peak 400 -> 365, 965 -> 871, 1471 -> 1347 MB.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
spf_lexer::next() becomes next<Policy>(). full_tokens, the default, is
what the parser has always had. index_tokens is what the lazy index
needs: a string is ended but not decoded, and a number, enumeration or
binary comes back as Token_LITERAL with only its position; names,
keywords and operators are read as before. Each policy compiles to its
own loop from the one implementation, so there is no second tokenizer.
character_decoder gains skip(): the same state machine as the
conversion with the collection compiled out, so an escape such as \S\'
(an apostrophe as the page character) ends the string at the same byte
under both policies. A byte-level scan would have ended it early.
Also fixes a comment that follows a token without whitespace, ",/* x */",
which skip_comment() never saw because the slash had been consumed.
TXG (58 MB), single thread: tokenizing the whole file 194 MB/s with
full_tokens, 249 MB/s with index_tokens; through 64 KB pages 196 and
205 MB/s. The parse itself is unchanged.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
callgrind on the tokenizer showed SWAR::has_special_char and eq_mask
compiled as calls, one per eight bytes; paged_file_impl::size() out of line
behind every eof() and remaining(); and the cursor's page-cache check not
inlined into peek() because it shared a function with the page fetch. The
SWAR helpers are forced inline, size() is defined in the class, and
cached_() is split into an inline check and an out-of-line refresh.
TXG (58 MB), single thread: tokenizer 196 -> 204 MB/s in memory and
136 -> 193 MB/s through 64 KB pages; strict parse through pages
1.15 -> 0.97 s against 0.93 s in memory; lazy index pass over pages
218 -> 311 MB/s; lazy open 0.52 -> 0.43 s.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
Every peek(), get() and SWAR word read on a paged reader went through the
page cache's hash map and LRU list, so tokenizing through pages ran at a
fifth of the speed of the in-memory buffer. The reader now remembers the
page its cursor was last on and serves reads that fall inside it from the
pointer, revalidated against an eviction counter on the implementation
so a page that left the cache is never read through a stale pointer.
Reads that straddle a page boundary take the existing paths.
The whole tokenizer over each file, 64 KB pages, 64 cached (4 MB):
in-memory buffer paged before paged after
TXG 58 MB 200 MB/s 42 MB/s 142 MB/s
210_King 148 MB 181 MB/s 34 MB/s 118 MB/s
OKgate22 232 MB 197 MB/s 38 MB/s 128 MB/s
This is the step that makes reading in pages a candidate for the default
path rather than a fallback; the remaining gap is the page fetch itself.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
Each instance was four separate allocations: the instance_data record, the
attribute array object it pointed at, that array's index bytes, and its
slot storage. A 58 MB model made 10.4 million mallocs to load 918k
instances, and massif attributed 99 MB of its 450 MB peak to malloc
bookkeeping alone.
variant_array now allocates the size byte, the per-slot type indices and
the slots as one block, and instance_data holds the array in a
std::optional instead of behind a pointer (an empty optional keeps the
meaning the null pointer had: attribute storage constructed on the fly
from the RocksDB backend). No ownership or lifetime changes; the same
object owns the same data.
Parse, C++ file constructor, on top of the previous commits:
TXG 58 MB 0.96 -> 0.94 s steady 310 -> 262 MB peak 383 -> 335 MB
210_King 148 MB 2.74 -> 2.63 s steady 758 -> 630 MB peak 948 -> 820 MB
OKgate22 232 MB 3.70 -> 3.61 s steady 1165 -> 971 MB peak 1447 -> 1252 MB
mallocs while loading TXG: 10.39M -> 7.56M.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
byguid_ was a std::map<std::string, ...>: a red-black node plus a
heap-allocated 22-character string per rooted instance, and a lookup that
walks ~18 levels of string comparisons on a 200k-entry file.
guid_map keeps keys of up to 23 characters inline in an unordered_map node
(every valid GlobalId is 22), and routes anything longer to an ordered map
so invalid files still work. Same std::string-keyed interface as before.
Parse, C++ file constructor, on top of the previous commits:
TXG 58 MB 1.08 s -> 1.03 s 341 -> 335 MB
210_King 148 MB 2.80 s -> 2.72 s 836 -> 830 MB
OKgate22 232 MB 4.25 s -> 3.94 s 1271 -> 1253 MB
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
full_buffer_impl::size(), get(), get_u32() and get_u64() (and their mmap_impl
twins) were defined out of line, so every character the lexer read crossed a
call boundary with its own bounds check. Callgrind put the three at 5.6% of
parse self time; inlining them lets the compiler hoist the checks out of the
scanning loops, which is worth more than their own cost.
Also reserve the streamer's inverse vector from the file size (about one
record per 32 bytes of SPF on real models) and shrink it once the bulk load
is sorted, so the doubling copies and the capacity slack go away.
Parse time, C++ file constructor, 12-core Linux box:
TXG 58 MB 1.35 s -> 1.15 s
210_King 148 MB 3.70 s -> 3.09 s
OKgate22 232 MB 6.18 s -> 5.30 s
Python ifcopenshell.open(): 1.40 -> 1.22, 3.70 -> 3.24, 6.27 -> 5.52 s.
Memory unchanged. The removed exported symbols mean the Python wrapper must
be rebuilt against this library.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
Also drop-in replacement mostly, except extra cmake args need to be provided now after `--` - `python run-cmake.py vs2022-x64 -- -DGLTF_SUPPORT=ON`.
Internally, script relies on env variables much much less.
Mainly drop-in replacement for `build-ifcopenshell.bat` with `--help`, kw args and args validation, but there's a small caveat.
Previously it was possible to pass args to the underlying build tool as simple positionals - e.g. `build-ifcopenshell vs2022-x64 Release /p:Foo=bar`.
This behaviour is disabled now, because it doesn't allow validating provided args - it's impossible to tell whether `--config Release` is meant to be passed to msbuild or was meant as `--build-cfg Release` for `build-ifcopenshell`.
But it's still possible to pass args to msbuild by using `--` - `python build-ifcopenshell.py vs2022-x64 Release -- /p:Foo=bar`
`install-ifcopenshell.py` is now just a small wrapper passing `--target INSTALL` arg.
* Rewrite to_string() to use switch{} and handle Token_NONE and identifier without as_string()
* Add regression tests for to_string() on tokens without a string form
Cover both halves of the recursion that made a whitespace-only file
segfault: token::to_string() on the EOF marker and on an instance name,
and a parse of input that lexes to zero tokens, which is how the header
parser reaches token::as_string() on the EOF marker.
Generated with the assistance of an AI coding tool.
---------
Co-authored-by: Bruno Postle <bruno@postle.net>
process_deletion_inverse() called inverse_index::remove_source(), which
walked every record in the file's inverse index to find the ones whose
source is the deleted instance: O(R) per deletion, the dominant cost of
file.remove() on large files now that the lookup side no longer re-sorts.
The records a deleted instance contributed are exactly the entity
references in its own attributes, so walk those with the same visitor
build_inverses_() uses for registration and remove each record with a
targeted binary search instead. remove_source() has no callers left and
is deleted.
Also use the ordered view of batch_deletion_ids_ (a boost multi_index
that already had one) for the is-this-referencer-also-being-deleted
check in process_deletion_(), which was a linear std::find over the
sequenced view: O(b) per referencing instance made batch deletion of b
instances quadratic.
file.remove on 300 IfcPropertySet of a 155 MB IFC4 model (201k IfcRoot)
drops from 3.15 ms to 0.17 ms per call, batched removal of 2000 from
3.34 ms to 0.17 ms per call, root.remove_product on 100 walls from
332 ms to 131 ms per call.
Claude-Session: https://claude.ai/code/session_01HNrXDmR88wKPCYwGE21SyH
(cherry picked from commit 938442303f)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Requiring quotes around a decimal was a surprise with no way to predict it:
Foobar.Baz>1 works, Foobar.Baz>1.5 is a syntax error, and > >= < <= are
almost always used on numbers. Writing Width>"0.2" also reads like a text
comparison even though it is not.
The "." is excluded from unquoted_string because it separates a pset from a
property, but that separator has already been consumed by the time a value is
read, so a number is unambiguous in value position. Add a decimal_string
alternative to the value rule only. Signed and leading dot forms are accepted,
1. is not, and pset, prop and query keys are unchanged - they still need
quoting for a ".".
The transformer returns the same string a quoted value would, so =1.5 and
="1.5" are the same query and compare() does the existing type coercion.
Nothing that parses today changes meaning; only inputs that used to be a
syntax error now work.
Update the quoting docs accordingly, and assert the Bay property that the
filter test set up but never checked.
Generated with the assistance of an AI coding tool.
An unquoted pset, prop, key or value may not contain any of , . = > < * !
or whitespace. This was never stated, and the only motivation the docs gave
for quoting was the "Level 3" example having a space, so there was nothing
to suggest that Pset_WallCommon.ThermalTransmittance=1.5 is a syntax error
while FireRating=2HR is fine. Decimal numbers are the common way to hit it.
State the rule where the reader meets values, in a new subsection under each
of the two value tables. The two grammars exclude different characters, so
the element value section states its own set and only the delta. Note that
all of these characters are a syntax error except the comma, which is read
as the filter separator instead, so Name=Foo,IfcWall silently means "named
Foo and an IfcWall" rather than matching the literal value.
Also fix the Query filter example, which had the same problem in the file
itself: query:types.count=0 does not error, it is silently parsed as a
property filter for a count property in a property set named query:types,
and matches nothing. Quote the keys, as the other query examples already do.
Generated with the assistance of an AI coding tool.
Moving to Python to make Windows build scripts more maintainable.
It's intended to be a drop-in replacement, so it should be possible to just switch `.\build-deps.cmd` to `python build-deps.py`, keeping exactly the same arguments and behaviour will be the same.
`build-deps.cmd` is deprecated, but not yet removed, but will be shortly after more testing.
Other batch files will be migrated to Python shortly after too.
Apparently `ty` is being too strict here and warning about `Any` possibly being `PathLike` which is not supported on older Pythons.
```
error[deprecated]: The overload of `which` is deprecated
--> src\bonsai\bonsai\bim\module\drawing\operator.py:2267:34
|
2267 | command[0] = shutil.which(command[0]) or command[0]
| ^^^^^^^^^^^^ On Windows before Python 3.12, using a PathLike as `cmd` would always fail or return `None`.
error[deprecated]: The overload of `which` is deprecated
--> src\bonsai\bonsai\tool\drawing.py:1324:30
|
1324 | command[0] = shutil.which(command[0]) or command[0]
| ^^^^^^^^^^^^ On Windows before Python 3.12, using a PathLike as `cmd` would always fail or return `None`.
```
I've found this commit 4400a6e that was introducing this option to the main cmake, but I'm not sure if it ever was propagated to the main branch or was reverted inside one of the merges.
But anyway, `USE_STATIC_MSVC_RUNTIME` is currently used only in svgfill, not used in the main cmake and works out of sync with main cmake (it sets `Boost_USE_STATIC_RUNTIME` to `OFF` instead of guard+setting it to `ON`).
So removing this option and syncing the code.
There was an error building a standalone ifcwrap when previously built IfcOpenShell had rocksdb - `_ifcopenshell_wrapper.cpython-314-x86_64-linux-gnu.so: undefined symbol: _ZTV17RocksDbSerializer`.
`IFOPSH_WITH_ROCKSDB` propagated to the build, requiring rocksdb symbols, but `WITH_ROCKSDB` wasn't set and `document_serializer_rdb` target wasn't provided.
When fixing warnings, noticed that was working a bit inproperly - `bnode` always end up being an empty list (because there's no `brick, A, REF.IFCReference` triple), so then passing empty list to `triples` resulted in selecting all nodes isntead of just the expected `bnode` (that's the behaviour `sqlachemy` was sending the warnings about - when empty lists unexpectedly selected everything).
`None, None` assumed by default.
Though this was introduced in Python 3.13, before 3.13 it only breaks if we'd do `typing.Generator[T]` (which is deprecated) -`collections.abc.Generator[T]` works fine, it seems it never had an arity check.
Raising `AttributeError` is definitely wasn't correct here, since it
might push the code to assume it's a wrong entity type. Returning some stub value like `None` also could suggest incorrect derived attribute
value, leading to unexpected behaviour. So adding an error, so it would
propagate and code would need to be adjusted not to rely on derived
attributes, if it actually interacts with sql/stream.
`test_unit` was asserting that derived attr will return `None`, though
it was actually raising `AttributeError`.
Facet.filter() implementations broad-phase query with
ifc_file.by_type(), then check isinstance(elements, list) to decide
whether a previous facet already narrowed the candidate set. In
v0.9.0, file.by_type() returns a tuple instead of a list, so that
check silently failed and every facet after the first re-scanned the
whole model instead of the already-narrowed (possibly empty) set.
This let a prohibited Entity+Attribute applicability match instances
of the wrong class, e.g. an IfcSlab satisfying an "IFCWALL" Entity
facet's chain. Accept tuples too, matching how by_type() results are
actually returned now.
get_attribute_category() returns 3 for a derived attribute, but
sqlite_entity.__getattr__() only branched on FORWARD (1) and INVERSE
(2), so any derived attribute (e.g. IfcSIUnit.Dimensions) fell
through to the final AttributeError instead of returning None, which
is what SQLite-linked files are documented to do since derived
attributes are not computed for them. Mirrors the DERIVED handling
already present in entity_instance.py's __getattr__.
diff() looked up common elements with self.old.by_id(global_id) /
self.new.by_id(global_id), passing a GlobalId string into a method
that expects a STEP integer id. On v0.8.0 file.by_id() was a Python
wrapper that transparently dispatched strings to by_guid(), so the
bug was silent. v0.9.0's file class binds by_id directly to the
C++ instance_by_id(int), so it now raises
TypeError: in method 'file_by_id', argument 2 of type 'int'.
* Honour IfcAxis2PlacementLinear Axis/RefDirection in the loft builder
make_loft() (src/ifcgeom/infra_sweep_helper.cpp), shared by
IfcSectionedSolidHorizontal and IfcSectionedSurface, mishandled a cross
section's IfcAxis2PlacementLinear in two ways:
1. A placement carrying Axis but no RefDirection was placed with a fixed
[e_y | e_z | e_x] world-axis permutation that ignored the directrix.
On any directrix not running along +X (e.g. a north-south road
pavement, or anywhere along a curve) the profile came out mis-oriented
or collapsed to a sliver.
2. When two adjacent CrossSectionPositions used direction vectors
inconsistently (a raked RefDirection at one, a plain Axis at the
other) make_loft() logged GEO 42, dropped the rotation for the whole
segment and squared every cap -- and in one configuration left the
sweep frame flipped, so OpenCASCADE failed to build the solid at all.
Now a small profile_basis() helper builds every cross section's frame the
same way: profile Y = Axis, profile normal = RefDirection, and -- when
RefDirection is absent -- profile normal = the directrix tangent, so the
section stays perpendicular to the path (buildingSMART IFC4.x-IF #147).
When the two bracketing placements ask for the same orientation the sweep
frame carries it, built against the curve. When they disagree the sweep
frame stays on the shared Axis (continuous with the neighbouring
consistent segments, so nothing flips) and each end's own authored
orientation is folded into its profile points via a change of basis, so
each end cap still lands exactly as authored while the body in between
keeps following the directrix. The all-equal and no-direction-vector
paths are unchanged.
The two mappings now also carry the raw RefDirection through on
cross_section, alongside the existing rotation matrix.
Adds C++ tests (a raked end logs no GEO 42; a directrix that does not run
along +X still lofts a full-size solid) and Python tests (uniform prism
raked at one end and square at the other; a north-south directrix keeps
its width; OffsetLateral/OffsetVertical are scaled by the model length
unit).
* Renames profile_rotations to profile_axis for consistency with profile_ref_directions
Dion Moult