mirror of
https://github.com/IfcOpenShell/IfcOpenShell.git
synced 2026-09-19 14:41:25 +00:00
ifcviewer: v16 zstd-compressed sidecars (~10x smaller over the wire)
The .ifcview data is hugely redundant (repeated double instance matrices, patterned indices) — measured 12x zstd whole-file. Server Content-Encoding can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format. Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames — each independently Range-fetchable, so streaming is intact — and the critical + deferred metadata blocks are single zstd frames. SidecarChunk carries the compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED — decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs by decompress+scatter. zstd: desktop links libzstd (also compresses at bake); the web build (Emscripten has no zstd port) FetchContent's the pinned zstd source and compiles its decompress-only subset for wasm — no vendored blob, same version as desktop. New SidecarCompress wraps it (compress guarded off under Emscripten). Both stream paths — desktop StreamingThread worker + sync fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress; readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a 752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB, loads 13/13 chunks on web, 0 errors. Three fixes found while testing big federations on a real server: - Web-streamed race: streaming_from_web was set in the deferred-header callback (a round-trip after the model+chunks exist), so driveStreamingLoads could take the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set immediately after applyCachedModel. - OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on web that's an uncatchable bad_alloc abort. Cap total pool capacity (setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom. - Web never evicted (grow-or-block only). At the hard budget, fall through to the LRU/priority evictor so a big federation stays navigable (highest-contribution chunks win) instead of freezing with holes. 113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars (desktop bakes v16; scratch conv tool migrates v15→v16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -109,7 +109,18 @@ public:
|
||||
// false the first time addSubBuffer is refused even at the floor
|
||||
// size — eviction callers need this to know whether a future alloc
|
||||
// could rescue them, or whether eviction is the only path.
|
||||
bool can_grow() const { return !growth_disabled_ && per_sub_buffer_capacity_ > 0; }
|
||||
bool can_grow() const {
|
||||
return !growth_disabled_ && per_sub_buffer_capacity_ > 0
|
||||
&& (max_total_capacity_bytes_ == 0
|
||||
|| total_capacity_bytes() < max_total_capacity_bytes_);
|
||||
}
|
||||
|
||||
// Hard ceiling on total pool capacity (0 = unlimited). Once total capacity
|
||||
// reaches this, can_grow() returns false so the streaming driver EVICTS
|
||||
// instead of growing. Critical on web: a growth past the wasm heap ceiling
|
||||
// is a bad_alloc that -fno-exceptions turns into an uncatchable abort, so
|
||||
// the async grow-OOM detection can't save us — we must stop first.
|
||||
void setMaxTotalCapacity(uint64_t max_bytes) { max_total_capacity_bytes_ = max_bytes; }
|
||||
|
||||
// Proactively add a sub-buffer (no allocation). On web this kicks off the
|
||||
// async provisional-validation cycle so validated free space appears a
|
||||
@@ -175,6 +186,7 @@ private:
|
||||
WGPUDevice device_ = nullptr;
|
||||
WGPUBufferUsage usage_ = 0;
|
||||
uint64_t per_sub_buffer_capacity_ = 0;
|
||||
uint64_t max_total_capacity_bytes_ = 0; // 0 = unlimited (see can_grow)
|
||||
// The largest size addSubBuffer last *succeeded* at, in bytes.
|
||||
// Starts at per_sub_buffer_capacity_ (the probe's discovered max)
|
||||
// and decays as the driver refuses larger allocations. Future grow
|
||||
|
||||
Reference in New Issue
Block a user