mirror of
https://github.com/IfcOpenShell/IfcOpenShell.git
synced 2026-09-17 14:02:27 +00:00
ifcviewer: v16 zstd-compressed sidecars (~10x smaller over the wire)
The .ifcview data is hugely redundant (repeated double instance matrices, patterned indices) — measured 12x zstd whole-file. Server Content-Encoding can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format. Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames — each independently Range-fetchable, so streaming is intact — and the critical + deferred metadata blocks are single zstd frames. SidecarChunk carries the compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED — decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs by decompress+scatter. zstd: desktop links libzstd (also compresses at bake); the web build (Emscripten has no zstd port) FetchContent's the pinned zstd source and compiles its decompress-only subset for wasm — no vendored blob, same version as desktop. New SidecarCompress wraps it (compress guarded off under Emscripten). Both stream paths — desktop StreamingThread worker + sync fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress; readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a 752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB, loads 13/13 chunks on web, 0 errors. Three fixes found while testing big federations on a real server: - Web-streamed race: streaming_from_web was set in the deferred-header callback (a round-trip after the model+chunks exist), so driveStreamingLoads could take the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set immediately after applyCachedModel. - OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on web that's an uncatchable bad_alloc abort. Cap total pool capacity (setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom. - Web never evicted (grow-or-block only). At the hard budget, fall through to the LRU/priority evictor so a big federation stays navigable (highest-contribution chunks win) instead of freezing with holes. 113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars (desktop bakes v16; scratch conv tool migrates v15→v16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -81,16 +81,32 @@ static constexpr uint32_t SIDECAR_MAGIC = 0x49465657; // "IFVW"
|
||||
// painting, so first geometry no longer waits on the property data; the
|
||||
// deferred block is fetched lazily (or skipped where unused). Desktop
|
||||
// reads both. No back-compat: regenerate sidecars.
|
||||
static constexpr uint32_t SIDECAR_VERSION = 15;
|
||||
// v16 = Geometry + metadata are zstd-COMPRESSED. Each chunk's vertex bytes and
|
||||
// index bytes are stored as two independent zstd frames (so per-chunk
|
||||
// Range streaming still works — you fetch + decompress just one chunk),
|
||||
// and the critical + deferred metadata blocks are single zstd frames.
|
||||
// The chunk TOC records each chunk's compressed blob offsets/sizes plus
|
||||
// the raw (decompressed) sizes. ~3-5x fewer bytes over the wire while
|
||||
// keeping HTTP Range intact (unlike server Content-Encoding). No
|
||||
// back-compat: regenerate sidecars.
|
||||
static constexpr uint32_t SIDECAR_VERSION = 16;
|
||||
static constexpr uint32_t SIDECAR_ENDIAN = 0x01020304;
|
||||
|
||||
// Chunk table-of-contents entry (v14+). A chunk is a CONTIGUOUS range of
|
||||
// meshes in the (reordered) meshes array — and therefore a contiguous span of
|
||||
// vertex + index bytes, since the geometry is laid out in chunk order. The
|
||||
// loader builds chunk `i` from meshes [first_mesh, first_mesh + mesh_count).
|
||||
// Chunk table-of-contents entry (v16). A chunk is a CONTIGUOUS range of meshes
|
||||
// [first_mesh, first_mesh + mesh_count). Its vertex + index bytes are stored as
|
||||
// two zstd frames in the geometry section; the loader fetches [v_comp_off,
|
||||
// +v_comp_size) / [i_comp_off, +i_comp_size) (offsets relative to the geometry
|
||||
// section start) and decompresses them to v_raw_size / i_raw_size bytes — the
|
||||
// chunk-local (vbytes, idx) applyStreamedChunk consumes.
|
||||
struct SidecarChunk {
|
||||
uint32_t first_mesh;
|
||||
uint32_t mesh_count;
|
||||
uint64_t v_comp_off;
|
||||
uint64_t v_comp_size;
|
||||
uint64_t v_raw_size;
|
||||
uint64_t i_comp_off;
|
||||
uint64_t i_comp_size;
|
||||
uint64_t i_raw_size;
|
||||
};
|
||||
|
||||
// Fixed-size element record. Strings are stored as (offset, length) pairs
|
||||
|
||||
Reference in New Issue
Block a user