mirror of
https://github.com/IfcOpenShell/IfcOpenShell.git
synced 2026-09-10 06:00:51 +00:00
ifcviewer: v16 zstd-compressed sidecars (~10x smaller over the wire)
The .ifcview data is hugely redundant (repeated double instance matrices, patterned indices) — measured 12x zstd whole-file. Server Content-Encoding can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format. Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames — each independently Range-fetchable, so streaming is intact — and the critical + deferred metadata blocks are single zstd frames. SidecarChunk carries the compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED — decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs by decompress+scatter. zstd: desktop links libzstd (also compresses at bake); the web build (Emscripten has no zstd port) FetchContent's the pinned zstd source and compiles its decompress-only subset for wasm — no vendored blob, same version as desktop. New SidecarCompress wraps it (compress guarded off under Emscripten). Both stream paths — desktop StreamingThread worker + sync fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress; readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a 752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB, loads 13/13 chunks on web, 0 errors. Three fixes found while testing big federations on a real server: - Web-streamed race: streaming_from_web was set in the deferred-header callback (a round-trip after the model+chunks exist), so driveStreamingLoads could take the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set immediately after applyCachedModel. - OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on web that's an uncatchable bad_alloc abort. Cap total pool capacity (setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom. - Web never evicted (grow-or-block only). At the hard budget, fall through to the LRU/priority evictor so a big federation stays navigable (highest-contribution chunks win) instead of freezing with holes. 113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars (desktop bakes v16; scratch conv tool migrates v15→v16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -177,6 +177,14 @@ struct ModelGpuData {
|
||||
// is recovered by walking mesh_ids and the model's MeshInfo[].
|
||||
uint64_t vertex_byte_size = 0;
|
||||
uint64_t index_count = 0;
|
||||
// v16: where this chunk's two zstd frames live in the file's geometry
|
||||
// section (offsets relative to model.geometry_section_offset) and their
|
||||
// compressed sizes. The raw sizes are vertex_byte_size / index_count*4.
|
||||
// A per-chunk load fetches [off, +comp) and decompresses.
|
||||
uint64_t v_comp_off = 0;
|
||||
uint64_t v_comp_size = 0;
|
||||
uint64_t i_comp_off = 0;
|
||||
uint64_t i_comp_size = 0;
|
||||
// Of `index_count`, how many are LOD1 indices. LOD0 indices occupy
|
||||
// chunk-local u32 offsets [0, index_count - lod1_index_count); LOD1
|
||||
// indices occupy [index_count - lod1_index_count, index_count). 0
|
||||
@@ -286,8 +294,9 @@ struct ModelGpuData {
|
||||
// streaming path: chunks may be non-resident and need byte-range reads
|
||||
// from this file. Empty path = legacy non-streaming load.
|
||||
std::string streaming_file_path;
|
||||
uint64_t streaming_vertex_section_offset = 0;
|
||||
uint64_t streaming_index_section_offset = 0;
|
||||
// v16: file offset of the compressed geometry section. A chunk's blobs are
|
||||
// at geometry_section_offset + chunk.{v_comp_off,i_comp_off}.
|
||||
uint64_t geometry_section_offset = 0;
|
||||
// Web only: chunk byte ranges come from the JS-side source — a picked File
|
||||
// (Blob.slice) or a remote URL (HTTP Range) — read asynchronously, not via
|
||||
// a synchronous fopen on streaming_file_path. Set by loadSidecarMetadataWeb
|
||||
@@ -308,8 +317,11 @@ struct ModelGpuData {
|
||||
// it; deferred_meta_loaded latches so it fetches at most once.
|
||||
std::vector<PackedElementInfo> elements;
|
||||
std::string string_table;
|
||||
uint64_t deferred_meta_offset = 0;
|
||||
uint64_t deferred_meta_bytes = 0;
|
||||
// v16: the deferred block is a single zstd frame at deferred_comp_offset of
|
||||
// deferred_comp_size bytes, expanding to deferred_raw_size.
|
||||
uint64_t deferred_comp_offset = 0;
|
||||
uint64_t deferred_comp_size = 0;
|
||||
uint64_t deferred_raw_size = 0;
|
||||
bool deferred_meta_loaded = false;
|
||||
// applyCachedModel rebases instance object_ids by this base to keep them
|
||||
// globally unique across models; deferred elements carry the sidecar's
|
||||
|
||||
Reference in New Issue
Block a user