ifcviewer: v16 zstd-compressed sidecars (~10x smaller over the wire)

The .ifcview data is hugely redundant (repeated double instance matrices,
patterned indices) — measured 12x zstd whole-file. Server Content-Encoding
can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format.

Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames —
each independently Range-fetchable, so streaming is intact — and the critical +
deferred metadata blocks are single zstd frames. SidecarChunk carries the
compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED —
decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs
by decompress+scatter. zstd: desktop links libzstd (also compresses at bake);
the web build (Emscripten has no zstd port) FetchContent's the pinned zstd
source and compiles its decompress-only subset for wasm — no vendored blob,
same version as desktop. New SidecarCompress wraps it (compress guarded off
under Emscripten). Both stream paths — desktop StreamingThread worker + sync
fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress;
readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress
the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a
752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB,
loads 13/13 chunks on web, 0 errors.

Three fixes found while testing big federations on a real server:
- Web-streamed race: streaming_from_web was set in the deferred-header callback
  (a round-trip after the model+chunks exist), so driveStreamingLoads could take
  the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set
  immediately after applyCachedModel.
- OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on
  web that's an uncatchable bad_alloc abort. Cap total pool capacity
  (setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise
  MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom.
- Web never evicted (grow-or-block only). At the hard budget, fall through to the
  LRU/priority evictor so a big federation stays navigable (highest-contribution
  chunks win) instead of freezing with holes.

113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars
(desktop bakes v16; scratch conv tool migrates v15→v16).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dion Moult
2026-07-02 10:24:59 +10:00
parent 5299f6c13c
commit 0b8c787ac0
19 changed files with 862 additions and 442 deletions
+32 -23
View File
@@ -46,25 +46,20 @@
struct StreamingSidecar {
// Everything except vertices + indices — same shape as SidecarData but
// with empty vertices / indices vectors. The renderer uses meshes /
// instances / georef / elements immediately.
// instances / georef / chunks immediately (elements/strings deferred).
SidecarData meta;
// Byte offsets in the on-disk file where the vertex and index sections
// start (after their 4-byte count headers). Pair with vertex_total_bytes
// / index_total_bytes for the section length; per-chunk reads slice
// arbitrary ranges within these.
uint64_t vertex_section_offset = 0;
uint64_t vertex_total_bytes = 0;
uint64_t index_section_offset = 0;
uint64_t index_total_count = 0; // u32 indices, NOT bytes
// v16: the compressed geometry section starts here. Each chunk's two zstd
// blobs live at geometry_section_offset + SidecarChunk.{v_comp_off,i_comp_off};
// a per-chunk load fetches [that, +*_comp_size) and decompresses to *_raw_size.
uint64_t geometry_section_offset = 0;
// v15 deferred-metadata locator. The render-critical metadata block starts
// at critical_meta_offset and is critical_meta_bytes long; the deferred
// block (elements + string_table) runs from there to EOF. The web loader
// reads only the critical block before painting and fetches the deferred
// block on demand from [critical_meta_offset + critical_meta_bytes, EOF).
uint64_t critical_meta_offset = 0;
uint64_t critical_meta_bytes = 0;
// v16 deferred (property) block locator: a single zstd frame at
// deferred_comp_offset of deferred_comp_size bytes → deferred_raw_size. The
// web loader fetches it on demand (elements/strings); desktop reads it up front.
uint64_t deferred_comp_offset = 0;
uint64_t deferred_comp_size = 0;
uint64_t deferred_raw_size = 0;
// Resolved on-disk path so subsequent chunk reads can re-open / seek.
std::string file_path;
@@ -75,6 +70,20 @@ struct StreamingSidecar {
// before return — callers re-open for per-chunk reads.
std::optional<StreamingSidecar> readSidecarMetadataOnly(const std::string& ifc_path);
// Read + decompress one chunk's geometry (v16) from disk: the vertex zstd frame
// at [geometry_section_offset + v_comp_off, +v_comp_size) → out_vbytes (v_raw
// bytes) and the index frame → out_idx (i_raw/4 u32s). Returns false on I/O or
// decompress failure. Used by the desktop StreamingThread worker + the sync
// first-frame fallback; the web path decompresses in beginWebChunkLoad instead.
bool readChunkGeometryCompressed(const std::string& ifc_path,
std::uint64_t geometry_section_offset,
std::uint64_t v_comp_off, std::uint64_t v_comp_size,
std::uint64_t v_raw_size,
std::uint64_t i_comp_off, std::uint64_t i_comp_size,
std::uint64_t i_raw_size,
std::vector<std::uint8_t>& out_vbytes,
std::vector<std::uint32_t>& out_idx);
// --- Pure, buffer-based building blocks ------------------------------------
//
// The metadata lives in two disjoint regions of the file: a small fixed
@@ -85,15 +94,15 @@ std::optional<StreamingSidecar> readSidecarMetadataOnly(const std::string& ifc_p
// source and hand the bytes to these parsers, so the wire-format knowledge
// lives in exactly one place and is unit-testable without touching a file.
// Bytes the head spans: SidecarHeader (12) + uint32 num_vertex_bytes (4).
inline constexpr std::size_t SIDECAR_HEAD_BYTES = 16;
// Bytes the head spans: SidecarHeader (12) + uint64 geometry-section length (8).
inline constexpr std::size_t SIDECAR_HEAD_BYTES = 20;
// Parse the 16-byte head. Validates magic / version / endian and, on success,
// writes the vertex-section byte count (which locates the index-count field at
// SIDECAR_HEAD_BYTES + out_num_vertex_bytes). Returns false if `n` is short or
// the header is wrong. `data` must point at the start of the file.
// Parse the 20-byte head (v16). Validates magic / version / endian and, on
// success, writes the compressed-geometry-section byte length (the metadata
// blocks follow at SIDECAR_HEAD_BYTES + out_geom_bytes). Returns false if `n`
// is short or the header is wrong. `data` must point at the start of the file.
bool parseSidecarHead(const std::uint8_t* data, std::size_t n,
std::uint32_t& out_num_vertex_bytes);
std::uint64_t& out_geom_bytes);
// Parse the v15 render-CRITICAL metadata block (mesh dict, instance dict,
// georef, chunk TOC) — everything needed to set up + draw the scene. `data`