Glassvision/IfcOpenShell
1
0
Fork 0
mirror of https://github.com/IfcOpenShell/IfcOpenShell.git synced 2026-09-21 14:23:53 +00:00
Code Issues Packages Projects Releases Wiki Activity

ifcparse: build the lazy index in parallel

Browse Source 
On a file large enough the DATA section is split at the same boundaries
the parallel parse uses (chunk_bounds(), now shared) and each chunk is
indexed by its own worker with its own paged reader, lexer, shells,
offsets, GlobalIds and inverse records; the results are merged in file
order, so instance order, GlobalId precedence and inverse records are
identical to the serial index. The serial index is the same code run on
one chunk. The name table is reserved before the merge, which also
helps the serial case. The default thread count is the one the full
parse uses.

TXG 58 MB / 210_King 147 MB / OKgate22 231 MB, lazy open: 12 threads
0.37 / 1.01 / 1.34 s against 0.55 / 1.59 / 2.11 s on one thread and
0.44 / 1.17 / 1.91 s for the default (parallel full) open; memory after
the open +15 to +50 MB at 12 threads for the workers' page caches. The
equality test now opens the 12 MB replicated fixture lazily with five
workers and compares it instance by instance with the serial parse.

This commit was written by an AI coding tool and has not been verified by
a human.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
This commit is contained in:
Dion Moult
2026-09-14 18:47:06 +10:00
parent 97c83f0292
commit f24e9a6bca
2 changed files with 250 additions and 162 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ifcparse/tests/test_ifcopenshell_parse.cpp
+8 -1
View File
@@ -570,6 +570,12 @@ TEST_CASE("Parallel and paged parsing yield the same instances, attributes, inve
paged_parallel.paged_reading(true);
paged_parallel.parse_threads(5);
REQUIRE(paged_parallel.initialize(path.string()));
// And the lazy index built by 5 workers.
ifcopenshell::file lazy_parallel(ifcopenshell::uninitialized_tag{});
lazy_parallel.lazy_loading(true);
lazy_parallel.parse_threads(5);
REQUIRE(lazy_parallel.initialize(path.string()));
REQUIRE(lazy_parallel.lazy_loading());
std::filesystem::remove(path);
size_t count = 0;
@@ -583,7 +589,7 @@ TEST_CASE("Parallel and paged parsing yield the same instances, attributes, inve
a.to_string(sa);
b.to_string(sb);
REQUIRE(sb.str() == sa.str());
for (ifcopenshell::file* other : {&paged, &paged_parallel}) {
for (ifcopenshell::file* other : {&paged, &paged_parallel, &lazy_parallel}) {
const express::base c = other->instance_by_id((int)a.id());
REQUIRE(c);
std::ostringstream sc;
@@ -603,5 +609,6 @@ TEST_CASE("Parallel and paged parsing yield the same instances, attributes, inve
for (const auto& rooted : serial.instances_by_type("IfcRoot")) {
const std::string guid = rooted.get_attribute_value(0);
REQUIRE(parallel.instance_by_guid(guid).id() == serial.instance_by_guid(guid).id());
REQUIRE(lazy_parallel.instance_by_guid(guid).id() == serial.instance_by_guid(guid).id());
}
}