mirror of
https://github.com/IfcOpenShell/IfcOpenShell.git
synced 2026-09-10 14:07:43 +00:00
ifcviewer: v16 zstd-compressed sidecars (~10x smaller over the wire)
The .ifcview data is hugely redundant (repeated double instance matrices, patterned indices) — measured 12x zstd whole-file. Server Content-Encoding can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format. Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames — each independently Range-fetchable, so streaming is intact — and the critical + deferred metadata blocks are single zstd frames. SidecarChunk carries the compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED — decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs by decompress+scatter. zstd: desktop links libzstd (also compresses at bake); the web build (Emscripten has no zstd port) FetchContent's the pinned zstd source and compiles its decompress-only subset for wasm — no vendored blob, same version as desktop. New SidecarCompress wraps it (compress guarded off under Emscripten). Both stream paths — desktop StreamingThread worker + sync fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress; readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a 752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB, loads 13/13 chunks on web, 0 errors. Three fixes found while testing big federations on a real server: - Web-streamed race: streaming_from_web was set in the deferred-header callback (a round-trip after the model+chunks exist), so driveStreamingLoads could take the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set immediately after applyCachedModel. - OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on web that's an uncatchable bad_alloc abort. Cap total pool capacity (setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom. - Web never evicted (grow-or-block only). At the hard budget, fall through to the LRU/priority evictor so a big federation stays navigable (highest-contribution chunks win) instead of freezing with holes. 113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars (desktop bakes v16; scratch conv tool migrates v15→v16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -46,25 +46,20 @@
|
||||
struct StreamingSidecar {
|
||||
// Everything except vertices + indices — same shape as SidecarData but
|
||||
// with empty vertices / indices vectors. The renderer uses meshes /
|
||||
// instances / georef / elements immediately.
|
||||
// instances / georef / chunks immediately (elements/strings deferred).
|
||||
SidecarData meta;
|
||||
|
||||
// Byte offsets in the on-disk file where the vertex and index sections
|
||||
// start (after their 4-byte count headers). Pair with vertex_total_bytes
|
||||
// / index_total_bytes for the section length; per-chunk reads slice
|
||||
// arbitrary ranges within these.
|
||||
uint64_t vertex_section_offset = 0;
|
||||
uint64_t vertex_total_bytes = 0;
|
||||
uint64_t index_section_offset = 0;
|
||||
uint64_t index_total_count = 0; // u32 indices, NOT bytes
|
||||
// v16: the compressed geometry section starts here. Each chunk's two zstd
|
||||
// blobs live at geometry_section_offset + SidecarChunk.{v_comp_off,i_comp_off};
|
||||
// a per-chunk load fetches [that, +*_comp_size) and decompresses to *_raw_size.
|
||||
uint64_t geometry_section_offset = 0;
|
||||
|
||||
// v15 deferred-metadata locator. The render-critical metadata block starts
|
||||
// at critical_meta_offset and is critical_meta_bytes long; the deferred
|
||||
// block (elements + string_table) runs from there to EOF. The web loader
|
||||
// reads only the critical block before painting and fetches the deferred
|
||||
// block on demand from [critical_meta_offset + critical_meta_bytes, EOF).
|
||||
uint64_t critical_meta_offset = 0;
|
||||
uint64_t critical_meta_bytes = 0;
|
||||
// v16 deferred (property) block locator: a single zstd frame at
|
||||
// deferred_comp_offset of deferred_comp_size bytes → deferred_raw_size. The
|
||||
// web loader fetches it on demand (elements/strings); desktop reads it up front.
|
||||
uint64_t deferred_comp_offset = 0;
|
||||
uint64_t deferred_comp_size = 0;
|
||||
uint64_t deferred_raw_size = 0;
|
||||
|
||||
// Resolved on-disk path so subsequent chunk reads can re-open / seek.
|
||||
std::string file_path;
|
||||
@@ -75,6 +70,20 @@ struct StreamingSidecar {
|
||||
// before return — callers re-open for per-chunk reads.
|
||||
std::optional<StreamingSidecar> readSidecarMetadataOnly(const std::string& ifc_path);
|
||||
|
||||
// Read + decompress one chunk's geometry (v16) from disk: the vertex zstd frame
|
||||
// at [geometry_section_offset + v_comp_off, +v_comp_size) → out_vbytes (v_raw
|
||||
// bytes) and the index frame → out_idx (i_raw/4 u32s). Returns false on I/O or
|
||||
// decompress failure. Used by the desktop StreamingThread worker + the sync
|
||||
// first-frame fallback; the web path decompresses in beginWebChunkLoad instead.
|
||||
bool readChunkGeometryCompressed(const std::string& ifc_path,
|
||||
std::uint64_t geometry_section_offset,
|
||||
std::uint64_t v_comp_off, std::uint64_t v_comp_size,
|
||||
std::uint64_t v_raw_size,
|
||||
std::uint64_t i_comp_off, std::uint64_t i_comp_size,
|
||||
std::uint64_t i_raw_size,
|
||||
std::vector<std::uint8_t>& out_vbytes,
|
||||
std::vector<std::uint32_t>& out_idx);
|
||||
|
||||
// --- Pure, buffer-based building blocks ------------------------------------
|
||||
//
|
||||
// The metadata lives in two disjoint regions of the file: a small fixed
|
||||
@@ -85,15 +94,15 @@ std::optional<StreamingSidecar> readSidecarMetadataOnly(const std::string& ifc_p
|
||||
// source and hand the bytes to these parsers, so the wire-format knowledge
|
||||
// lives in exactly one place and is unit-testable without touching a file.
|
||||
|
||||
// Bytes the head spans: SidecarHeader (12) + uint32 num_vertex_bytes (4).
|
||||
inline constexpr std::size_t SIDECAR_HEAD_BYTES = 16;
|
||||
// Bytes the head spans: SidecarHeader (12) + uint64 geometry-section length (8).
|
||||
inline constexpr std::size_t SIDECAR_HEAD_BYTES = 20;
|
||||
|
||||
// Parse the 16-byte head. Validates magic / version / endian and, on success,
|
||||
// writes the vertex-section byte count (which locates the index-count field at
|
||||
// SIDECAR_HEAD_BYTES + out_num_vertex_bytes). Returns false if `n` is short or
|
||||
// the header is wrong. `data` must point at the start of the file.
|
||||
// Parse the 20-byte head (v16). Validates magic / version / endian and, on
|
||||
// success, writes the compressed-geometry-section byte length (the metadata
|
||||
// blocks follow at SIDECAR_HEAD_BYTES + out_geom_bytes). Returns false if `n`
|
||||
// is short or the header is wrong. `data` must point at the start of the file.
|
||||
bool parseSidecarHead(const std::uint8_t* data, std::size_t n,
|
||||
std::uint32_t& out_num_vertex_bytes);
|
||||
std::uint64_t& out_geom_bytes);
|
||||
|
||||
// Parse the v15 render-CRITICAL metadata block (mesh dict, instance dict,
|
||||
// georef, chunk TOC) — everything needed to set up + draw the scene. `data`
|
||||
|
||||
Reference in New Issue
Block a user