ifcviewer: v16 zstd-compressed sidecars (~10x smaller over the wire)

The .ifcview data is hugely redundant (repeated double instance matrices,
patterned indices) — measured 12x zstd whole-file. Server Content-Encoding
can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format.

Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames —
each independently Range-fetchable, so streaming is intact — and the critical +
deferred metadata blocks are single zstd frames. SidecarChunk carries the
compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED —
decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs
by decompress+scatter. zstd: desktop links libzstd (also compresses at bake);
the web build (Emscripten has no zstd port) FetchContent's the pinned zstd
source and compiles its decompress-only subset for wasm — no vendored blob,
same version as desktop. New SidecarCompress wraps it (compress guarded off
under Emscripten). Both stream paths — desktop StreamingThread worker + sync
fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress;
readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress
the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a
752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB,
loads 13/13 chunks on web, 0 errors.

Three fixes found while testing big federations on a real server:
- Web-streamed race: streaming_from_web was set in the deferred-header callback
  (a round-trip after the model+chunks exist), so driveStreamingLoads could take
  the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set
  immediately after applyCachedModel.
- OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on
  web that's an uncatchable bad_alloc abort. Cap total pool capacity
  (setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise
  MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom.
- Web never evicted (grow-or-block only). At the hard budget, fall through to the
  LRU/priority evictor so a big federation stays navigable (highest-contribution
  chunks win) instead of freezing with holes.

113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars
(desktop bakes v16; scratch conv tool migrates v15→v16).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dion Moult
2026-07-02 10:24:59 +10:00
parent 5299f6c13c
commit 0b8c787ac0
19 changed files with 862 additions and 442 deletions
+2 -2
View File
@@ -38,7 +38,7 @@
# Better to keep the web path's machinery local to this directory.
cmake_minimum_required(VERSION 3.21)
project(IfcViewerWeb LANGUAGES CXX)
project(IfcViewerWeb LANGUAGES C CXX) # C for the vendored zstd decoder
if(NOT EMSCRIPTEN)
message(FATAL_ERROR
@@ -114,7 +114,7 @@ target_link_options(IfcViewerWeb PRIVATE
# cap; --shared64 / MEMORY64 would lift this later if we need it).
"-sALLOW_MEMORY_GROWTH=1"
"-sINITIAL_MEMORY=268435456" # 256 MB
"-sMAXIMUM_MEMORY=2147483648" # 2 GB
"-sMAXIMUM_MEMORY=4294967296" # 4 GB (wasm32 max) — big federations need the headroom
# FETCH lets emscripten_fetch issue HTTP Range requests. The local
# file path (#88) reads byte ranges via Blob.slice and does NOT need
# this; it's retained for the remote-URL Range backend (follow-up).
Binary file not shown.