A lazy open reads the DATA section once with the tokenizer's index
policy and builds what indexes the file: a shell per instance (name and
declaration, no attribute array), the complete inverse index with
attribute indices, the GlobalId map and the by-type lists. No attribute
value is decoded. The first time an instance's attributes are touched,
ensure_loaded() seeks the retained paged reader to the instance and runs
the same load_attributes() the full parse runs, with inverse registration
off, then resolves that instance's references from its own slots. A
modified instance is materialised first, so writing works.
There is no scanner of its own: the index pass consumes next<index_tokens>()
and counts parentheses and commas on the operator tokens; a keyword where
an instance should start, or a token the tokenizer rejects, stops the
index and the file is parsed in full. The offset of each instance's
attribute list is kept in one sorted vector that exists only in lazy
mode, so a full parse pays nothing for it. Materialising from several
threads at once is not safe.
TXG 58 MB / 210_King 147 MB / OKgate22 231 MB, single thread: lazy open
0.61 / 1.73 / 2.86 s against the full parse's 1.05 / 2.69 / 4.99 s, at
141 / 374 / 534 MB against 274 / 654 / 1036 MB; reading one attribute of
every instance afterwards costs a further 0.56 / 1.44 / 4.86 s.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
The parser could not resolve a #name when it read it, because the
instance may be defined further down the file, so it left the slot empty
and appended (owner, attribute, name) to a side table that a second pass
walked. The table held one entry per reference for the whole read: 64 MB
on a 58 MB model, the high-water mark of opening.
Now the reference stays where the tokenizer put it: the attribute slot
holds the instance_reference, or the reference_or_simple_type aggregate
for a list (mixed with inline typed values or not), until every instance
has been read, and resolve_instance_references() walks each instance's
slots and swaps names for instances. Ordering is what the tokenizer
produced; nothing is re-derived. A missing name becomes null in a scalar
and is dropped from an aggregate, as before; the error keeps its offset.
The three transient alternatives are appended to the attribute pack and
to argument_type in lock step and are never visible once a file is
loaded. Simple type instances read inline (IfcPropertySetDefinitionSet)
have their own slots, so their references need no diversion.
The table remains for the header entities and for streaming consumers of
instance_streamer::references(), which leave resolve_references_in_place
off.
TXG 58 MB / 210_King 147 MB / OKgate22 231 MB, single thread: time
unchanged (1.05 / 2.69 / 4.99 s), memory after the parse 287 -> 274,
698 -> 654, 1086 -> 1036 MB, peak 400 -> 365, 965 -> 871, 1471 -> 1347 MB.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
spf_lexer::next() becomes next<Policy>(). full_tokens, the default, is
what the parser has always had. index_tokens is what the lazy index
needs: a string is ended but not decoded, and a number, enumeration or
binary comes back as Token_LITERAL with only its position; names,
keywords and operators are read as before. Each policy compiles to its
own loop from the one implementation, so there is no second tokenizer.
character_decoder gains skip(): the same state machine as the
conversion with the collection compiled out, so an escape such as \S\'
(an apostrophe as the page character) ends the string at the same byte
under both policies. A byte-level scan would have ended it early.
Also fixes a comment that follows a token without whitespace, ",/* x */",
which skip_comment() never saw because the slash had been consumed.
TXG (58 MB), single thread: tokenizing the whole file 194 MB/s with
full_tokens, 249 MB/s with index_tokens; through 64 KB pages 196 and
205 MB/s. The parse itself is unchanged.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
byguid_ was a std::map<std::string, ...>: a red-black node plus a
heap-allocated 22-character string per rooted instance, and a lookup that
walks ~18 levels of string comparisons on a 200k-entry file.
guid_map keeps keys of up to 23 characters inline in an unordered_map node
(every valid GlobalId is 22), and routes anything longer to an ordered map
so invalid files still work. Same std::string-keyed interface as before.
Parse, C++ file constructor, on top of the previous commits:
TXG 58 MB 1.08 s -> 1.03 s 341 -> 335 MB
210_King 148 MB 2.80 s -> 2.72 s 836 -> 830 MB
OKgate22 232 MB 4.25 s -> 3.94 s 1271 -> 1253 MB
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
full_buffer_impl::size(), get(), get_u32() and get_u64() (and their mmap_impl
twins) were defined out of line, so every character the lexer read crossed a
call boundary with its own bounds check. Callgrind put the three at 5.6% of
parse self time; inlining them lets the compiler hoist the checks out of the
scanning loops, which is worth more than their own cost.
Also reserve the streamer's inverse vector from the file size (about one
record per 32 bytes of SPF on real models) and shrink it once the bulk load
is sorted, so the doubling copies and the capacity slack go away.
Parse time, C++ file constructor, 12-core Linux box:
TXG 58 MB 1.35 s -> 1.15 s
210_King 148 MB 3.70 s -> 3.09 s
OKgate22 232 MB 6.18 s -> 5.30 s
Python ifcopenshell.open(): 1.40 -> 1.22, 3.70 -> 3.24, 6.27 -> 5.52 s.
Memory unchanged. The removed exported symbols mean the Python wrapper must
be rebuilt against this library.
This commit was written by an AI coding tool and has not been verified by
a human.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
process_deletion_inverse() called inverse_index::remove_source(), which
walked every record in the file's inverse index to find the ones whose
source is the deleted instance: O(R) per deletion, the dominant cost of
file.remove() on large files now that the lookup side no longer re-sorts.
The records a deleted instance contributed are exactly the entity
references in its own attributes, so walk those with the same visitor
build_inverses_() uses for registration and remove each record with a
targeted binary search instead. remove_source() has no callers left and
is deleted.
Also use the ordered view of batch_deletion_ids_ (a boost multi_index
that already had one) for the is-this-referencer-also-being-deleted
check in process_deletion_(), which was a linear std::find over the
sequenced view: O(b) per referencing instance made batch deletion of b
instances quadratic.
file.remove on 300 IfcPropertySet of a 155 MB IFC4 model (201k IfcRoot)
drops from 3.15 ms to 0.17 ms per call, batched removal of 2000 from
3.34 ms to 0.17 ms per call, root.remove_product on 100 walls from
332 ms to 131 ms per call.
Claude-Session: https://claude.ai/code/session_01HNrXDmR88wKPCYwGE21SyH
(cherry picked from commit 938442303f)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Rename header-scope aliases, enums, and helper types while retaining descriptive names where dropping the suffix would create a collision.
Generated with the assistance of an AI coding tool.
Setting an IfcInteger/IfcTimeStamp typed attribute (e.g. IfcOwnerHistory.CreationDate)
outside the signed 32-bit range corrupted the value instead of raising, since the
Python wrapper's set_attribute_value_py() truncated it with a plain static_cast<int>
before handing it to the C++ storage. Unix timestamps before 1901-12-13 or after
2038-01-19 silently wrapped around (e.g. 3000000000 became -1294967296) rather than
being rejected or stored correctly. Fixes#3058, equivalent to PR #8683 but ported to
this branch's rewritten ifcparse (snake_case files, variant_array/instance_data
storage, SWIG PyObject-based attribute setter) instead of the old IfcEntityInstanceData
sources, which no longer exist here.
The scalar slot of the attribute variant (Argument_INT) becomes int64_t. Integer
aggregates (Argument_AGGREGATE_OF_INT, e.g. CoordIndex) and instance/reference
identifiers stay 32-bit, since neither is the value that overflows here; this narrow
scope is kept on its own technical merits (aggregates and identifiers were never the
source of the bug, and widening them would be a much larger, riskier change for no
benefit) even though aothms said compatibility isn't a concern on this v0.9-track
branch. express::Base::set_attribute_value promotes the schema-generated int to
int64_t at a single choke point, so the generated setters keep compiling unchanged.
The STEP lexer, writer, and SWIG wrapper (set_attribute_value_py, pythonize) are all
widened together, since widening only the Python-facing setter would have silently
wrapped the value on file write instead of raising.
Verified in a build (IFC2X3 and IFC4, BUILD_IFCGEOM off, no kernels): pre-1901,
post-2038, both 32-bit boundaries, and a 9e12 value all round trip exactly both in
memory and through STEP text serialization (write then reopen). A value outside the
64-bit range now raises a clean exception instead of corrupting data. Ordinary
in-range integers and integer aggregates (e.g. IfcTriangulatedFaceSet.CoordIndex) are
unaffected. The existing util/test_attribute.py and test_file.py suites pass
unchanged; test_entity_instance.py has 5 pre-existing failures unrelated to this
change (confirmed identical on an unfixed build of this branch, caused by a missing
get_info_2 binding and _patch_swig_comparisons never being implemented here).
Generated with the assistance of an AI coding tool.
Fixes builds with newer GCC/libstdc++ that no longer provide <cstdint>,
<cstring>, <cfloat>, <memory>, <algorithm> etc. transitively. Also
disambiguates visit<> calls in taxonomy.h with the full namespace and
casts the character value in IfcCharacterDecoder to uint32_t to silence
ambiguous overload warnings.
Fixes builds with newer GCC/libstdc++ that no longer provide <cstdint>,
<cstring>, <cfloat>, <memory>, <algorithm> etc. transitively. Also
disambiguates visit<> calls in taxonomy.h with the full namespace and
casts the character value in IfcCharacterDecoder to uint32_t to silence
ambiguous overload warnings.
Read-only handles reject Flush/CompactRange, so the destructor's status
assertion always fired on shutdown when the streamer's sidecar was
opened with read_only=true. Track the flag and skip the write path; also
guard against a null db when the initial open failed.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Dion Moult