A lazy open reads the DATA section once with the tokenizer's index
policy and builds what indexes the file: a shell per instance (name and
declaration, no attribute array), the complete inverse index with
attribute indices, the GlobalId map and the by-type lists. No attribute
value is decoded. The first time an instance's attributes are touched,
ensure_loaded() seeks the retained paged reader to the instance and runs
the same load_attributes() the full parse runs, with inverse registration
off, then resolves that instance's references from its own slots. A
modified instance is materialised first, so writing works.
There is no scanner of its own: the index pass consumes next<index_tokens>()
and counts parentheses and commas on the operator tokens; a keyword where
an instance should start, or a token the tokenizer rejects, stops the
index and the file is parsed in full. The offset of each instance's
attribute list is kept in one sorted vector that exists only in lazy
mode, so a full parse pays nothing for it. Materialising from several
threads at once is not safe.
TXG 58 MB / 210_King 147 MB / OKgate22 231 MB, single thread: lazy open
0.61 / 1.73 / 2.86 s against the full parse's 1.05 / 2.69 / 4.99 s, at
141 / 374 / 534 MB against 274 / 654 / 1036 MB; reading one attribute of
every instance afterwards costs a further 0.56 / 1.44 / 4.86 s.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
The parser could not resolve a #name when it read it, because the
instance may be defined further down the file, so it left the slot empty
and appended (owner, attribute, name) to a side table that a second pass
walked. The table held one entry per reference for the whole read: 64 MB
on a 58 MB model, the high-water mark of opening.
Now the reference stays where the tokenizer put it: the attribute slot
holds the instance_reference, or the reference_or_simple_type aggregate
for a list (mixed with inline typed values or not), until every instance
has been read, and resolve_instance_references() walks each instance's
slots and swaps names for instances. Ordering is what the tokenizer
produced; nothing is re-derived. A missing name becomes null in a scalar
and is dropped from an aggregate, as before; the error keeps its offset.
The three transient alternatives are appended to the attribute pack and
to argument_type in lock step and are never visible once a file is
loaded. Simple type instances read inline (IfcPropertySetDefinitionSet)
have their own slots, so their references need no diversion.
The table remains for the header entities and for streaming consumers of
instance_streamer::references(), which leave resolve_references_in_place
off.
TXG 58 MB / 210_King 147 MB / OKgate22 231 MB, single thread: time
unchanged (1.05 / 2.69 / 4.99 s), memory after the parse 287 -> 274,
698 -> 654, 1086 -> 1036 MB, peak 400 -> 365, 965 -> 871, 1471 -> 1347 MB.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
spf_lexer::next() becomes next<Policy>(). full_tokens, the default, is
what the parser has always had. index_tokens is what the lazy index
needs: a string is ended but not decoded, and a number, enumeration or
binary comes back as Token_LITERAL with only its position; names,
keywords and operators are read as before. Each policy compiles to its
own loop from the one implementation, so there is no second tokenizer.
character_decoder gains skip(): the same state machine as the
conversion with the collection compiled out, so an escape such as \S\'
(an apostrophe as the page character) ends the string at the same byte
under both policies. A byte-level scan would have ended it early.
Also fixes a comment that follows a token without whitespace, ",/* x */",
which skip_comment() never saw because the slash had been consumed.
TXG (58 MB), single thread: tokenizing the whole file 194 MB/s with
full_tokens, 249 MB/s with index_tokens; through 64 KB pages 196 and
205 MB/s. The parse itself is unchanged.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
callgrind on the tokenizer showed SWAR::has_special_char and eq_mask
compiled as calls, one per eight bytes; paged_file_impl::size() out of line
behind every eof() and remaining(); and the cursor's page-cache check not
inlined into peek() because it shared a function with the page fetch. The
SWAR helpers are forced inline, size() is defined in the class, and
cached_() is split into an inline check and an out-of-line refresh.
TXG (58 MB), single thread: tokenizer 196 -> 204 MB/s in memory and
136 -> 193 MB/s through 64 KB pages; strict parse through pages
1.15 -> 0.97 s against 0.93 s in memory; lazy index pass over pages
218 -> 311 MB/s; lazy open 0.52 -> 0.43 s.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
Every peek(), get() and SWAR word read on a paged reader went through the
page cache's hash map and LRU list, so tokenizing through pages ran at a
fifth of the speed of the in-memory buffer. The reader now remembers the
page its cursor was last on and serves reads that fall inside it from the
pointer, revalidated against an eviction counter on the implementation
so a page that left the cache is never read through a stale pointer.
Reads that straddle a page boundary take the existing paths.
The whole tokenizer over each file, 64 KB pages, 64 cached (4 MB):
in-memory buffer paged before paged after
TXG 58 MB 200 MB/s 42 MB/s 142 MB/s
210_King 148 MB 181 MB/s 34 MB/s 118 MB/s
OKgate22 232 MB 197 MB/s 38 MB/s 128 MB/s
This is the step that makes reading in pages a candidate for the default
path rather than a fallback; the remaining gap is the page fetch itself.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
Each instance was four separate allocations: the instance_data record, the
attribute array object it pointed at, that array's index bytes, and its
slot storage. A 58 MB model made 10.4 million mallocs to load 918k
instances, and massif attributed 99 MB of its 450 MB peak to malloc
bookkeeping alone.
variant_array now allocates the size byte, the per-slot type indices and
the slots as one block, and instance_data holds the array in a
std::optional instead of behind a pointer (an empty optional keeps the
meaning the null pointer had: attribute storage constructed on the fly
from the RocksDB backend). No ownership or lifetime changes; the same
object owns the same data.
Parse, C++ file constructor, on top of the previous commits:
TXG 58 MB 0.96 -> 0.94 s steady 310 -> 262 MB peak 383 -> 335 MB
210_King 148 MB 2.74 -> 2.63 s steady 758 -> 630 MB peak 948 -> 820 MB
OKgate22 232 MB 3.70 -> 3.61 s steady 1165 -> 971 MB peak 1447 -> 1252 MB
mallocs while loading TXG: 10.39M -> 7.56M.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
byguid_ was a std::map<std::string, ...>: a red-black node plus a
heap-allocated 22-character string per rooted instance, and a lookup that
walks ~18 levels of string comparisons on a 200k-entry file.
guid_map keeps keys of up to 23 characters inline in an unordered_map node
(every valid GlobalId is 22), and routes anything longer to an ordered map
so invalid files still work. Same std::string-keyed interface as before.
Parse, C++ file constructor, on top of the previous commits:
TXG 58 MB 1.08 s -> 1.03 s 341 -> 335 MB
210_King 148 MB 2.80 s -> 2.72 s 836 -> 830 MB
OKgate22 232 MB 4.25 s -> 3.94 s 1271 -> 1253 MB
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
full_buffer_impl::size(), get(), get_u32() and get_u64() (and their mmap_impl
twins) were defined out of line, so every character the lexer read crossed a
call boundary with its own bounds check. Callgrind put the three at 5.6% of
parse self time; inlining them lets the compiler hoist the checks out of the
scanning loops, which is worth more than their own cost.
Also reserve the streamer's inverse vector from the file size (about one
record per 32 bytes of SPF on real models) and shrink it once the bulk load
is sorted, so the doubling copies and the capacity slack go away.
Parse time, C++ file constructor, 12-core Linux box:
TXG 58 MB 1.35 s -> 1.15 s
210_King 148 MB 3.70 s -> 3.09 s
OKgate22 232 MB 6.18 s -> 5.30 s
Python ifcopenshell.open(): 1.40 -> 1.22, 3.70 -> 3.24, 6.27 -> 5.52 s.
Memory unchanged. The removed exported symbols mean the Python wrapper must
be rebuilt against this library.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
* Rewrite to_string() to use switch{} and handle Token_NONE and identifier without as_string()
* Add regression tests for to_string() on tokens without a string form
Cover both halves of the recursion that made a whitespace-only file
segfault: token::to_string() on the EOF marker and on an instance name,
and a parse of input that lexes to zero tokens, which is how the header
parser reaches token::as_string() on the EOF marker.
Generated with the assistance of an AI coding tool.
---------
Co-authored-by: Bruno Postle <bruno@postle.net>
process_deletion_inverse() called inverse_index::remove_source(), which
walked every record in the file's inverse index to find the ones whose
source is the deleted instance: O(R) per deletion, the dominant cost of
file.remove() on large files now that the lookup side no longer re-sorts.
The records a deleted instance contributed are exactly the entity
references in its own attributes, so walk those with the same visitor
build_inverses_() uses for registration and remove each record with a
targeted binary search instead. remove_source() has no callers left and
is deleted.
Also use the ordered view of batch_deletion_ids_ (a boost multi_index
that already had one) for the is-this-referencer-also-being-deleted
check in process_deletion_(), which was a linear std::find over the
sequenced view: O(b) per referencing instance made batch deletion of b
instances quadratic.
file.remove on 300 IfcPropertySet of a 155 MB IFC4 model (201k IfcRoot)
drops from 3.15 ms to 0.17 ms per call, batched removal of 2000 from
3.34 ms to 0.17 ms per call, root.remove_product on 100 walls from
332 ms to 131 ms per call.
Claude-Session: https://claude.ai/code/session_01HNrXDmR88wKPCYwGE21SyH
(cherry picked from commit 938442303f)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Modifications to build C++ with Visual Studio 2026 and the v145 toolset.
* Fixes linker settings for rocksdb for Debug and Release builds
* module is a C++ 20 keyword. Explicitly stating namespace allows cpp20 projects to build against the library
* Fixes crash when initializing an object with the initialize function when some of the attributes are empty, {}, or omitted, std::nullopt
* cleanup for vs2026 v145 toolset per @aothms review
* Fixes bug, IfcCurveSegment.setStartLength was setSegmentLength in alignment_helper.cpp
* Bumps boost to 1.92
E.g. IfcAdvancedHouse:
```
/usr/bin/x86_64-linux-gnu-ld.bfd: CMakeFiles/IfcAdvancedHouse.dir/IfcAdvancedHouse.cpp.o: in function `main':
IfcAdvancedHouse.cpp:(.text.startup.main+0x137): undefined reference to `hierarchy_helper<Ifc4x3_add2>::addBuilding(Ifc4x3_add2::IfcSite, Ifc4x3_add2::IfcOwnerHistory)'
/usr/bin/x86_64-linux-gnu-ld.bfd: IfcAdvancedHouse.cpp:(.text.startup.main+0x7c7): undefined reference to `hierarchy_helper<Ifc4x3_add2>::getRepresentationContext(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
/usr/bin/x86_64-linux-gnu-ld.bfd: IfcAdvancedHouse.cpp:(.text.startup.main+0x931): undefined reference to `hierarchy_helper<Ifc4x3_add2>::getRepresentationContext(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)'
```
Noticed by addressing gcc warning gcc warning that attribute order is incorrect:
```
//src/ifcparse/hierarchy_helper.i:721:31: warning: attribute ignored in explicit instantiation ‘class hierarchy_helper<Ifc2x3>’ [-Wattributes]
721 | template IFC_SCHEMA_API class hierarchy_helper<IfcSchema>;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
//src/ifcparse/hierarchy_helper.i:721:31: note: no attribute can be applied to an explicit instantiation
```
Rename header-scope aliases, enums, and helper types while retaining descriptive names where dropping the suffix would create a collision.
Generated with the assistance of an AI coding tool.
Create file-owned headers after storage is selected but before streaming starts. Let owner-backed streamers use that header directly, and keep owned_header_ exclusively for ownerless streamers.
Generated with the assistance of an AI coding tool.
Both surfaced on the first Windows/WASM CI run of this branch:
- XmlSerializer.cpp: the IfcPropertySetDefinitionSet block used a C-style
cast to convert the set to std::vector<IfcPropertySetDefinition>. GCC
invokes the non-explicit conversion operator; MSVC rejects the cast to a
template type (C2440/C3536/C2661). Use copy-initialisation instead, which
invokes the same implicit conversion portably. (This block was dead until
the SCHEMAS_->SCHEMA_HAS_ typo fix enabled it, so it had never hit MSVC.)
- parse.cpp: the floating-point parse path falls back to strtod_l because
libc++ =deletes the float from_chars overload. That fallback was guarded
for __APPLE__ only; Emscripten uses the same libc++, so WASM hit the
deleted from_chars. Extend the guard to __EMSCRIPTEN__ (its musl provides
strtod_l/newlocale, treating all locales as C).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both fixes were made on v0.8.0's IfcParse.cpp AFTER the datamodel branch
had already renamed it to parse.cpp. When main was later merged into the
branch, the modify/delete was resolved toward the deletion (merge
8c9c3cde2), so the changes never reached the live parse.cpp. They are
invisible to `git log v0.8.0...HEAD` because the originating commits sit
in the merged-in shared ancestry; only a content sweep of the renamed
files surfaces them.
- format_double now uses the shortest decimal representation that
round-trips exactly (Mac-safe manual implementation, no std::to_chars),
instead of setprecision(digits10) which padded clean REALs with noise
digits (0.0174532925199433 -> 0.017453292519943299) and rewrote every
REAL on re-save. Recovers ee2b357d7 + fa597536e + 821cf7b67, #7696.
- The [SYN004] non-entity-type parse-error branch now resets current_id
to 0 before advancing, matching its sibling error branches, so a
malformed non-entity instance no longer erroneously terminates parsing.
Recovers 7c9df9f98.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two related bugs in read_from_stream's reference-resolution
loop, both reachable from malformed input:
- has_attribute_value<IfcBaseClass*> only checks the stored
slot's type, not that it's non-null (e.g. an explicit $
value), so the following get_attribute_value() call could
return null and inst->declaration() crashed on it.
- byid_[ref] default-inserts (and returns) a null pointer
when the owning instance id isn't present, which was then
dereferenced unconditionally via ->data().
Added regression tests using the two minimized crash inputs
that found these.
Generated with the assistance of an AI coding tool.
(cherry picked from commit 56121ca061)
escape_xml escaped the five XML metacharacters but passed control
characters (0x00 to 0x1F other than tab, newline and carriage return)
through unchanged. Those bytes are illegal in XML 1.0 and cannot be
represented even as numeric character references, so any IFC string
containing them produced non-well-formed XML and SVG output.
Strip those illegal control characters before escaping. Bytes belonging to
a valid UTF-8 multibyte sequence are always >= 0x80, so filtering on the low
control range leaves real text intact. This is the shared helper used by the
SVG serializer text and attribute sites (audited: all route through it) and
by the XML/Collada paths, so both reports are resolved at one place.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
(cherry picked from commit 380675e214)
Follow-up to the scalar-only fix in #8754, per aothms's direct request on
that PR ("Please do make all int types consistent") and his own original
2023 design intent on issue #3058 ("make all integers (incl. schema
namespaces) an int64_t"). Widens the remaining inconsistent spots now that
compatibility isn't a constraint on this v0.9 branch:
- Integer aggregates (IfcTriangulatedFaceSet.CoordIndex and similar
List<int> attributes), including the SWIG to_vec_int/to_vec_vec_int
helpers, which previously silently truncated via static_cast<int> on the
Python-set path - the same bug class as the original scalar issue.
- The schema code generator (express/mapping.py's integer type mapping),
and all 12 generated schema header/source pairs regenerated to match, so
every schema-typed getter/setter (e.g. IfcOwnerHistory::CreationDate) is
int64_t end to end, not just the dynamic attribute-value path.
Instance/reference identifiers (STEP #123 ids) are deliberately left at
32-bit: they're a file-local index into internal maps, not an EXPRESS
domain value an application chooses, and no realistic STEP file has
billions of entities. The lexer's Token_IDENTIFIER parsing still funnels
through a 32-bit int for this reason - flagged as a known, low-risk gap
rather than fixed, since fixing it would mean touching indexing/hashing
code for no realistic benefit.
Verified: original PR's round-trip tests extended with aggregate cases
(IfcTriangulatedFaceSet.CoordIndex, InnerCoordIndices) at 64-bit boundary
values, in memory and through STEP text, IFC2X3 and IFC4. A standalone C++
program exercising the generated schema API directly (Ifc4::IfcOwnerHistory
::setCreationDate/CreationDate, IfcTriangulatedFaceSet::setCoordIndex/
CoordIndex) confirms int64_t end to end, bypassing SWIG. Full build
(BUILD_IFCGEOM, WITH_OPENCASCADE, BUILD_IFCPYTHON, IFC2X3+IFC4) clean.
test/util/test_attribute.py and test_file.py pass unchanged.
This contribution was produced with the assistance of an AI coding tool.
Dion Moult