The .ifcview data is hugely redundant (repeated double instance matrices,
patterned indices) — measured 12x zstd whole-file. Server Content-Encoding
can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format.
Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames —
each independently Range-fetchable, so streaming is intact — and the critical +
deferred metadata blocks are single zstd frames. SidecarChunk carries the
compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED —
decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs
by decompress+scatter. zstd: desktop links libzstd (also compresses at bake);
the web build (Emscripten has no zstd port) FetchContent's the pinned zstd
source and compiles its decompress-only subset for wasm — no vendored blob,
same version as desktop. New SidecarCompress wraps it (compress guarded off
under Emscripten). Both stream paths — desktop StreamingThread worker + sync
fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress;
readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress
the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a
752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB,
loads 13/13 chunks on web, 0 errors.
Three fixes found while testing big federations on a real server:
- Web-streamed race: streaming_from_web was set in the deferred-header callback
(a round-trip after the model+chunks exist), so driveStreamingLoads could take
the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set
immediately after applyCachedModel.
- OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on
web that's an uncatchable bad_alloc abort. Cap total pool capacity
(setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise
MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom.
- Web never evicted (grow-or-block only). At the hard budget, fall through to the
LRU/priority evictor so a big federation stays navigable (highest-contribution
chunks win) instead of freezing with holes.
113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars
(desktop bakes v16; scratch conv tool migrates v15→v16).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
First-paint over a network is metadata-bound: the whole post-index metadata
(~10 MB on a 118 MB model) had to download before any geometry. But ~25% of
it — elements + string_table, the IFC element tree (names/GUIDs/hierarchy) —
is used only for UI/picking, never for rendering (ViewportCore never touches
it).
v15 splits the post-index metadata into a render-CRITICAL block (meshes,
instances, georef, chunk TOC) preceded by its byte length, then a DEFERRED
block (elements + string_table). The web loader reads only the critical block
before painting; the deferred block sits at a known, self-describing offset
([critical end, EOF)) and is fetched on demand. Desktop reads both (local).
Web on-demand path is wired and complete (not yet called — no UI consumer):
loadDeferredMetadataWeb(model_id) range-fetches + parses the deferred block
into ModelGpuData.elements/string_table, at most once; the first consumer
will be "show the selected object's name" on pick. No background prefetch —
view-only sessions never download the property data (saves 2.64 MB on this
model).
parseSidecarTail split into parseSidecarCritical + parseSidecarDeferred (pure,
unit-tested); StreamingSidecar gains the critical-block locator. Measured
(118 MB model): critical metadata 10.35 -> 7.71 MB, deferred 2.64 MB off the
path; first paint 10.5 -> 9.6 s @ 24 Mbps. (Instances still dominate the
critical block — the next metadata lever.) Format -> v15, no back-compat;
regenerate sidecars. 111/111 unit + 6/6 web smoke pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Makes large-model streaming over a network actually good — fixing read
amplification, then first-paint latency — building on the byte-range work.
v14 layout + TOC (SidecarLayout, pure + unit-tested)
The loader chunks meshes by spatial Morton order, but the sidecar stored
geometry in mesh-id order, so a chunk's meshes were scattered through the
file: streaming one chunk meant either hundreds of tiny range requests or
reading (and discarding) everything between them — a 113 MB model fetched
~340 MB, a 531 MB model 2.25 GB (4.2x). Fix: at bake, reorder meshes into
the loader's chunk order and rebuild vertex/index(LOD0+LOD1)/instance
sections so each chunk is one CONTIGUOUS byte range, and bake a chunk TOC
({first_mesh, mesh_count}). The loader builds chunks straight from the TOC
rather than re-deriving the plan — the float Morton quantisation isn't
bit-identical across toolchains (x86 baker vs wasm loader), so a re-derived
plan scatters the chunks. Format bumped to v14 (regenerate sidecars). The
reorder buckets instances by per-instance mesh_id (the baker never sets
MeshInfo.first_instance — trusting it scrambled every transform → geometry
at the origin). Multiset-verified on a 28,900-instance model: every
instance's placement + geometry preserved. Result: 531 MB fetches 531 MB
(1.0x) in 72 requests (was 2036).
Progressive streaming (concurrency cap + small chunks)
Even at 1x, geometry appeared only after ~the whole model arrived: the
browser multiplexes every in-flight Range request over one HTTP/2 conn, so
unbounded concurrency (9 in flight) split the bandwidth and nothing finished
until the end (measured: first paint after 113 of 118 MB / 35 s @ 24 Mbps).
Cap concurrent chunk loads (kMaxWebInflightChunks=2): the priority-sorted
top chunks finish and paint first, then the next → first paint 9 s. Chunk
size dropped 16->4 MB (cheap now that each chunk is one read; matches Cesium
3D Tiles / xeokit / SVF2) for smoother progression. First-paint is now
metadata-bound (~10 MB tail) — the next lever.
111/111 unit (new test_sidecar_layout: geometry preserved, contiguous layout,
Morton-identity) + 6/6 web smoke pass; desktop bake (SceneLoader) reorders
before writeSidecar; embedded web sample regenerated to v14.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Wire up wgpu init + scene load + RAF render so the embedded sample
sidecar paints on the canvas in both browsers.
Root-cause fix: BufferPool::addSubBuffer spin-waited on
PopErrorScope, which resolves via JS microtask on Dawn-web. The
spin blocked the JS event loop, so the microtask never fired and
the first allocation hung the page indefinitely. Skip the
error-scope dance on Emscripten; trust the buffer pointer.
ViewportCore: add initWgpuAsyncWeb (nested-callback adapter→device
chain with AllowSpontaneous mode, no spin) and loadSidecarFromPath
(Qt-free entry point). waitTickInstance becomes a no-op shim on
web; cull-threads / streaming_thread_ / wgpuSurfacePresent gated
off; chunk I/O runs inline.
main_web.cpp: AppState + initWgpuAsyncWeb → buildPipelines (+
HiZ/edge/pick) → loadSidecarFromPath → ready flag → Module._app_ptr
handoff. The RAF loop lives in shell.html (NOT here) because any
RAF helper called from inside Dawn-web's wgpu Promise.then chain
stalls the device callback.
CMakeLists.txt: EXIT_RUNTIME=0 + Module.noExitRuntime=true (shell)
keeps wasm alive past main() so the device promise lands; no
Asyncify; export _raf_tick_c so shell.html's RAF can call it.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Dion Moult