The .ifcview data is hugely redundant (repeated double instance matrices,
patterned indices) — measured 12x zstd whole-file. Server Content-Encoding
can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format.
Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames —
each independently Range-fetchable, so streaming is intact — and the critical +
deferred metadata blocks are single zstd frames. SidecarChunk carries the
compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED —
decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs
by decompress+scatter. zstd: desktop links libzstd (also compresses at bake);
the web build (Emscripten has no zstd port) FetchContent's the pinned zstd
source and compiles its decompress-only subset for wasm — no vendored blob,
same version as desktop. New SidecarCompress wraps it (compress guarded off
under Emscripten). Both stream paths — desktop StreamingThread worker + sync
fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress;
readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress
the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a
752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB,
loads 13/13 chunks on web, 0 errors.
Three fixes found while testing big federations on a real server:
- Web-streamed race: streaming_from_web was set in the deferred-header callback
(a round-trip after the model+chunks exist), so driveStreamingLoads could take
the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set
immediately after applyCachedModel.
- OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on
web that's an uncatchable bad_alloc abort. Cap total pool capacity
(setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise
MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom.
- Web never evicted (grow-or-block only). At the hard budget, fall through to the
LRU/priority evictor so a big federation stays navigable (highest-contribution
chunks win) instead of freezing with holes.
113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars
(desktop bakes v16; scratch conv tool migrates v15→v16).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Battle-testing real sidecars over HTTP Range exposed severe thrash: a
531 MB model re-fetched 2.25 GB (4×) and never converged — viewAll puts
the whole model in frustum, so every chunk wants to be resident, and the
web async path made it worse two ways:
- A web load only consumes pool space when it COMPLETES (async), so the
per-frame issuance over-committed the pool; completions then failed
applyStreamedChunk on a full pool, the chunk re-candidated with no
cooldown, and re-fetched every frame.
- Pool growth is itself async on web (provisional sub-buffers validated
off the JS event loop), so even fetched chunks failed to alloc until
the pool caught up, and re-fetched.
Fix: gate web chunk issuance on VALIDATED free space + in-flight
reservation, and grow the pool BEFORE fetching:
- streaming_web_inflight_bytes_ reserves each in-flight load's footprint
so we never have more bytes in flight than the pool can place.
- When a visible chunk doesn't fit validated free, don't fetch — call
pool_.requestGrowth() (BufferPool: drives the async provisional grow
without allocating) and short-back-off; the chunk is fetched once,
after space exists. When the pool is saturated (model > GPU memory),
long-cooldown so a never-fitting chunk isn't re-fetched. Gating before
the evictor also kills phase-2 visible↔visible swap thrash.
- On async load failure, cool down (short if the pool can still grow,
long if saturated) instead of re-candidating next frame.
Result (manual battle tool, host.mjs + real files): 531 MB now loads
23/23 chunks, 322 MB loads 14/14 — resident climbs monotonically with
ZERO thrash warnings and a stable resident set, vs the old re-fetch loop.
The whole model resides on the GPU and stays. (Remaining ~3× ramp
over-fetch — per-chunk re-loads during the async-growth ramp + read
amplification from chunk byte-locality — is a separate efficiency
follow-up, not thrash.) 6/6 web smoke + 107/107 unit pass; desktop
unaffected (the gate is web-only; requestGrowth is a no-op wrapper there).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
On web we skip the desktop error-scope spin-wait (it blocks the JS event
loop and hangs the page). The old web addSubBuffer then judged success by
`buf != nullptr` — but Dawn-web returns a NON-NULL error buffer on OOM,
so the pool committed an invalid sub-buffer, alloc handed out slices in
it, and every chunk_bind_group built against it failed ("BindGroup is
invalid" spam + a wgpuQueueSubmit panic). Loading a model larger than the
browser's WebGPU budget triggered exactly this.
Add the grown sub-buffer as *provisional* (alloc and the capacity/free
tallies skip it) and validate it through a non-blocking AllowSpontaneous
PopErrorScope. resolveProvisionalGrowth() clears the flag when it's good,
or drops the sub-buffer and latches growth_disabled_ on a real OOM — at
which point the streaming evictor bounds the working set to what fits
instead of cascading. Only one provisional grow is in flight at a time
(growth_pending_). Desktop keeps its synchronous halve-retry path
unchanged. All 100 unit tests still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The chunk planner (Morton sort + greedy pack) and instance composition
(federation × placement matrix chain + world-AABB derive) were inline
helpers in ViewportWindow.cpp. Pulled both out as free-function modules
so the math + lookup logic can be exercised without a Qt window or a
wgpu device. ViewportWindow now delegates; InstanceLookup is a using-
alias to InstanceCompose::InstanceLookup.
Also added an addSubBufferForTesting / clearSubPoolsForTesting seam to
BufferPool so the sub-allocator invariants can be pinned with fake
WGPUBuffer handles. The fakes are never dereferenced; the guard drops
the sub-pools before destructor would call wgpuBufferRelease.
Three new test binaries under src/ifcviewer/tests/, 33 cases / 173
assertions: BufferPool first-fit + alignment + coalescing + multi-
sub-pool isolation; ChunkPlanner Morton split / interleave / stable
sort / greedy-pack monotonicity and single-mesh-oversize; InstanceCompose
identity / translation / order-of-multiplication / large-placement
cancellation against federation false origin / column-major writeback /
findInstance lookup paths.
The GL backend is gone (task #53). The wgpu/non-wgpu folder split and
the Wgpu* class prefix were both disambiguation artefacts from the
overlap period — now pure dead weight.
## Folder + library merge
* `src/ifcviewer-wgpu/` → folded into `src/ifcviewer/` (git mv tracks
every file as a rename so blame/log history survives).
* `src/ifcviewer-wgpu-minimal/` → `src/ifcviewer-minimal/` (the exe was
already named `IfcViewerMinimal`; this just brings the folder + CMake
target name into line).
* `src/ifcviewer-wgpu/tests/test_wgpu_{selection,visibility}.cpp` →
`src/ifcviewer/tests/test_{selection,visibility}.cpp`, folded into
the existing `add_ifcviewer_unit_test(...)` helper.
* The `IfcViewerWgpu` static library is dissolved — its sources become
part of the unified `IfcViewer` static library, which now bundles
scene/loader + renderer in one target. The pre-merge circular
dependency (IfcViewer linking IfcViewerWgpu just to get the
ViewportWindow.h include path that SceneLoader.h needs) goes away.
* The wgpu-native FetchContent block, the Cocoa/QuartzCore link on
Apple, the OBJCXX-enabled `.mm` source, and the wgpu-native runtime
install all move into `src/ifcviewer/CMakeLists.txt` unchanged.
## Type renames (Wgpu prefix dropped from every Wgpu* identifier)
WgpuAreaMeasurement → AreaMeasurement
WgpuBufferPool → BufferPool
WgpuLengthMeasurement → LengthMeasurement
WgpuMetalSurface → MetalSurface
WgpuModelGpuData → ModelGpuData
WgpuOverlayFrame → OverlayFrame
WgpuOverlayRenderer → OverlayRenderer
WgpuSectionPlane → SectionPlane
WgpuSelectionState → SelectionState
WgpuStreamingLoader → StreamingLoader
WgpuStreamingThread → StreamingThread
WgpuViewportWindow → ViewportWindow
WgpuVisibilityState → VisibilityState
CMake target IfcViewerWgpuMinimal → IfcViewerMinimal (exe name was
already this since wgpu shipped as default).
Deliberately kept: `onWgpuLog` (wgpu-native log callback — names a
binding to an external API, not one of *our* types), and the WGPU*
enum/struct prefixes from wgpu-native's own headers. `WgpuMemProbe`
lives in the separate `src/wgpu-mem-probe/` standalone diagnostic
project and isn't touched.
## Include-path updates
Every `#include "../ifcviewer-wgpu/Wgpu<X>.h"` → `"../ifcviewer/<X>.h"`,
every in-directory `#include "Wgpu<X>.h"` → `"<X>.h"`. Includes from
sibling subdirectories (modules/, etc.) are updated to point at
`../../../ifcviewer/` instead of `../../../ifcviewer-wgpu/`.
## cmake/CMakeLists.txt simplification
The redundant `add_subdirectory(ifcviewer-wgpu)` blocks (one inside
the BUILD_BONSAIVIEWER fan-in, one in the BONSAIVIEWER-less standalone
block) collapse into a single unconditional
`add_subdirectory(../src/ifcviewer ifcviewer)`. The standalone block
keeps only `wgpu-mem-probe` (the diagnostic tool, unrelated to the
viewer lib).
## Verification
* Full build green: `IfcViewer` static lib, `IfcViewerMinimal` exe,
`BonsaiViewer` exe, all four pre-existing ifcviewer unit tests, and
the two new-location tests (`test_selection`, `test_visibility`).
* No stray `Wgpu<X>` identifier remains across `src/ifcviewer/`,
`src/bonsaiviewer/`, `src/ifcviewer-minimal/` (verified by grep).
* Renames tracked by git as `R` entries — `git log --follow` on
ViewportWindow.cpp etc. continues to show history through the move.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Dion Moult