mirror of
https://github.com/IfcOpenShell/IfcOpenShell.git
synced 2026-08-12 02:23:34 +00:00
f1d97ac5aaeebfb211a995e3c57c7620c7c71e02
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Dion Moult
|
0b8c787ac0 |
ifcviewer: v16 zstd-compressed sidecars (~10x smaller over the wire)
The .ifcview data is hugely redundant (repeated double instance matrices, patterned indices) — measured 12x zstd whole-file. Server Content-Encoding can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format. Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames — each independently Range-fetchable, so streaming is intact — and the critical + deferred metadata blocks are single zstd frames. SidecarChunk carries the compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED — decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs by decompress+scatter. zstd: desktop links libzstd (also compresses at bake); the web build (Emscripten has no zstd port) FetchContent's the pinned zstd source and compiles its decompress-only subset for wasm — no vendored blob, same version as desktop. New SidecarCompress wraps it (compress guarded off under Emscripten). Both stream paths — desktop StreamingThread worker + sync fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress; readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a 752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB, loads 13/13 chunks on web, 0 errors. Three fixes found while testing big federations on a real server: - Web-streamed race: streaming_from_web was set in the deferred-header callback (a round-trip after the model+chunks exist), so driveStreamingLoads could take the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set immediately after applyCachedModel. - OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on web that's an uncatchable bad_alloc abort. Cap total pool capacity (setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom. - Web never evicted (grow-or-block only). At the hard budget, fall through to the LRU/priority evictor so a big federation stays navigable (highest-contribution chunks win) instead of freezing with holes. 113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars (desktop bakes v16; scratch conv tool migrates v15→v16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Dion Moult
|
41a85a70ba |
ifcviewer: v15 — defer property metadata off the first-paint path
First-paint over a network is metadata-bound: the whole post-index metadata (~10 MB on a 118 MB model) had to download before any geometry. But ~25% of it — elements + string_table, the IFC element tree (names/GUIDs/hierarchy) — is used only for UI/picking, never for rendering (ViewportCore never touches it). v15 splits the post-index metadata into a render-CRITICAL block (meshes, instances, georef, chunk TOC) preceded by its byte length, then a DEFERRED block (elements + string_table). The web loader reads only the critical block before painting; the deferred block sits at a known, self-describing offset ([critical end, EOF)) and is fetched on demand. Desktop reads both (local). Web on-demand path is wired and complete (not yet called — no UI consumer): loadDeferredMetadataWeb(model_id) range-fetches + parses the deferred block into ModelGpuData.elements/string_table, at most once; the first consumer will be "show the selected object's name" on pick. No background prefetch — view-only sessions never download the property data (saves 2.64 MB on this model). parseSidecarTail split into parseSidecarCritical + parseSidecarDeferred (pure, unit-tested); StreamingSidecar gains the critical-block locator. Measured (118 MB model): critical metadata 10.35 -> 7.71 MB, deferred 2.64 MB off the path; first paint 10.5 -> 9.6 s @ 24 Mbps. (Instances still dominate the critical block — the next metadata lever.) Format -> v15, no back-compat; regenerate sidecars. 111/111 unit + 6/6 web smoke pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Dion Moult
|
e1be2f208c |
ifcviewer: v14 chunk-contiguous sidecar + progressive network streaming
Makes large-model streaming over a network actually good — fixing read
amplification, then first-paint latency — building on the byte-range work.
v14 layout + TOC (SidecarLayout, pure + unit-tested)
The loader chunks meshes by spatial Morton order, but the sidecar stored
geometry in mesh-id order, so a chunk's meshes were scattered through the
file: streaming one chunk meant either hundreds of tiny range requests or
reading (and discarding) everything between them — a 113 MB model fetched
~340 MB, a 531 MB model 2.25 GB (4.2x). Fix: at bake, reorder meshes into
the loader's chunk order and rebuild vertex/index(LOD0+LOD1)/instance
sections so each chunk is one CONTIGUOUS byte range, and bake a chunk TOC
({first_mesh, mesh_count}). The loader builds chunks straight from the TOC
rather than re-deriving the plan — the float Morton quantisation isn't
bit-identical across toolchains (x86 baker vs wasm loader), so a re-derived
plan scatters the chunks. Format bumped to v14 (regenerate sidecars). The
reorder buckets instances by per-instance mesh_id (the baker never sets
MeshInfo.first_instance — trusting it scrambled every transform → geometry
at the origin). Multiset-verified on a 28,900-instance model: every
instance's placement + geometry preserved. Result: 531 MB fetches 531 MB
(1.0x) in 72 requests (was 2036).
Progressive streaming (concurrency cap + small chunks)
Even at 1x, geometry appeared only after ~the whole model arrived: the
browser multiplexes every in-flight Range request over one HTTP/2 conn, so
unbounded concurrency (9 in flight) split the bandwidth and nothing finished
until the end (measured: first paint after 113 of 118 MB / 35 s @ 24 Mbps).
Cap concurrent chunk loads (kMaxWebInflightChunks=2): the priority-sorted
top chunks finish and paint first, then the next → first paint 9 s. Chunk
size dropped 16->4 MB (cheap now that each chunk is one read; matches Cesium
3D Tiles / xeokit / SVF2) for smoother progression. First-paint is now
metadata-bound (~10 MB tail) — the next lever.
111/111 unit (new test_sidecar_layout: geometry preserved, contiguous layout,
Morton-identity) + 6/6 web smoke pass; desktop bake (SceneLoader) reorders
before writeSidecar; embedded web sample regenerated to v14.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Dion Moult
|
a0ba3c0b98 |
ifcviewer: extract pure buffer-based sidecar parse + read-plan helpers
Splits the v13 metadata wire-format knowledge out of the FILE*-bound streaming reader into pure, buffer-based functions so the web byte-range path (#88) can reuse it without loading the whole sidecar into the wasm heap: - parseSidecarHead — validates the 16-byte head, yields num_vertex_bytes - parseSidecarTail — parses meshes/instances/georef/elements/strings from an in-memory tail buffer, bounds-checked - planSidecarReadRanges + SidecarReadPlan — the range-coalescing / scatter planner, promoted out of the anonymous namespace readSidecarMetadataOnly and the range readers now call these; desktop behaviour is unchanged (head + tail are small, the bulk is still skipped via seek). The metadata tail is split from the head around the bulk sections, so a blob-backed loader just slices those two regions and hands the bytes to the same parsers. Closes a coverage gap: StreamingLoader had no unit tests. Adds test_streaming_loader.cpp (7 cases: metadata round-trip, corrupt/truncated rejection, vertex+index range scatter, head validation, tail truncation, read-plan coalescing). 107/107 unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Dion Moult
|
8ab5c31e75 |
refactor: merge ifcviewer-wgpu into ifcviewer, drop Wgpu prefix
The GL backend is gone (task #53). The wgpu/non-wgpu folder split and the Wgpu* class prefix were both disambiguation artefacts from the overlap period — now pure dead weight. ## Folder + library merge * `src/ifcviewer-wgpu/` → folded into `src/ifcviewer/` (git mv tracks every file as a rename so blame/log history survives). * `src/ifcviewer-wgpu-minimal/` → `src/ifcviewer-minimal/` (the exe was already named `IfcViewerMinimal`; this just brings the folder + CMake target name into line). * `src/ifcviewer-wgpu/tests/test_wgpu_{selection,visibility}.cpp` → `src/ifcviewer/tests/test_{selection,visibility}.cpp`, folded into the existing `add_ifcviewer_unit_test(...)` helper. * The `IfcViewerWgpu` static library is dissolved — its sources become part of the unified `IfcViewer` static library, which now bundles scene/loader + renderer in one target. The pre-merge circular dependency (IfcViewer linking IfcViewerWgpu just to get the ViewportWindow.h include path that SceneLoader.h needs) goes away. * The wgpu-native FetchContent block, the Cocoa/QuartzCore link on Apple, the OBJCXX-enabled `.mm` source, and the wgpu-native runtime install all move into `src/ifcviewer/CMakeLists.txt` unchanged. ## Type renames (Wgpu prefix dropped from every Wgpu* identifier) WgpuAreaMeasurement → AreaMeasurement WgpuBufferPool → BufferPool WgpuLengthMeasurement → LengthMeasurement WgpuMetalSurface → MetalSurface WgpuModelGpuData → ModelGpuData WgpuOverlayFrame → OverlayFrame WgpuOverlayRenderer → OverlayRenderer WgpuSectionPlane → SectionPlane WgpuSelectionState → SelectionState WgpuStreamingLoader → StreamingLoader WgpuStreamingThread → StreamingThread WgpuViewportWindow → ViewportWindow WgpuVisibilityState → VisibilityState CMake target IfcViewerWgpuMinimal → IfcViewerMinimal (exe name was already this since wgpu shipped as default). Deliberately kept: `onWgpuLog` (wgpu-native log callback — names a binding to an external API, not one of *our* types), and the WGPU* enum/struct prefixes from wgpu-native's own headers. `WgpuMemProbe` lives in the separate `src/wgpu-mem-probe/` standalone diagnostic project and isn't touched. ## Include-path updates Every `#include "../ifcviewer-wgpu/Wgpu<X>.h"` → `"../ifcviewer/<X>.h"`, every in-directory `#include "Wgpu<X>.h"` → `"<X>.h"`. Includes from sibling subdirectories (modules/, etc.) are updated to point at `../../../ifcviewer/` instead of `../../../ifcviewer-wgpu/`. ## cmake/CMakeLists.txt simplification The redundant `add_subdirectory(ifcviewer-wgpu)` blocks (one inside the BUILD_BONSAIVIEWER fan-in, one in the BONSAIVIEWER-less standalone block) collapse into a single unconditional `add_subdirectory(../src/ifcviewer ifcviewer)`. The standalone block keeps only `wgpu-mem-probe` (the diagnostic tool, unrelated to the viewer lib). ## Verification * Full build green: `IfcViewer` static lib, `IfcViewerMinimal` exe, `BonsaiViewer` exe, all four pre-existing ifcviewer unit tests, and the two new-location tests (`test_selection`, `test_visibility`). * No stray `Wgpu<X>` identifier remains across `src/ifcviewer/`, `src/bonsaiviewer/`, `src/ifcviewer-minimal/` (verified by grep). * Renames tracked by git as `R` entries — `git log --follow` on ViewportWindow.cpp etc. continues to show history through the move. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |