mirror of
https://github.com/IfcOpenShell/IfcOpenShell.git
synced 2026-08-19 19:54:07 +00:00
Dion Moult
0b8c787ac0
The .ifcview data is hugely redundant (repeated double instance matrices, patterned indices) — measured 12x zstd whole-file. Server Content-Encoding can't be used (it breaks HTTP Range), so compress PER-CHUNK into the format. Format (v16): geometry becomes per-chunk zstd(vertices)+zstd(indices) frames — each independently Range-fetchable, so streaming is intact — and the critical + deferred metadata blocks are single zstd frames. SidecarChunk carries the compressed blob offsets/sizes; applyStreamedChunk (render/upload) is UNCHANGED — decompression slots into the fetch. Full readSidecar (test/tooling) reconstructs by decompress+scatter. zstd: desktop links libzstd (also compresses at bake); the web build (Emscripten has no zstd port) FetchContent's the pinned zstd source and compiles its decompress-only subset for wasm — no vendored blob, same version as desktop. New SidecarCompress wraps it (compress guarded off under Emscripten). Both stream paths — desktop StreamingThread worker + sync fallback (readChunkGeometryCompressed) and web beginWebChunkLoad — decompress; readSidecarMetadataOnly / the web bootstrap / loadDeferredMetadataWeb decompress the metadata blocks. streamingByteProgress reports COMPRESSED bytes. MEASURED: a 752 MB v15 federation → 75 MB v16 (10x; per-file 6.7-15.3x); PP-PLP 118→15 MB, loads 13/13 chunks on web, 0 errors. Three fixes found while testing big federations on a real server: - Web-streamed race: streaming_from_web was set in the deferred-header callback (a round-trip after the model+chunks exist), so driveStreamingLoads could take the sync fopen path meanwhile → "failed to read/decompress chunk 0". Now set immediately after applyCachedModel. - OOM abort on 18 models: the pool grew unbounded until an alloc failed, but on web that's an uncatchable bad_alloc abort. Cap total pool capacity (setMaxTotalCapacity, 3 GB) so it stops before the heap ceiling, and raise MAXIMUM_MEMORY 2→4 GB (wasm32 max) for headroom. - Web never evicted (grow-or-block only). At the hard budget, fall through to the LRU/priority evictor so a big federation stays navigable (highest-contribution chunks win) instead of freezing with holes. 113/113 desktop + 6/6 web smoke pass. No back-compat: regenerate sidecars (desktop bakes v16; scratch conv tool migrates v15→v16). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
52 lines
2.8 KiB
C++
52 lines
2.8 KiB
C++
/********************************************************************************
|
|
* *
|
|
* This file is part of IfcOpenShell. *
|
|
* *
|
|
* IfcOpenShell is free software: you can redistribute it and/or modify *
|
|
* it under the terms of the Lesser GNU General Public License as published by *
|
|
* the Free Software Foundation, either version 3.0 of the License, or *
|
|
* (at your option) any later version. *
|
|
* *
|
|
* IfcOpenShell is distributed in the hope that it will be useful, *
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of *
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the *
|
|
* Lesser GNU General Public License for more details. *
|
|
* *
|
|
* You should have received a copy of the Lesser GNU General Public License *
|
|
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
|
|
* *
|
|
********************************************************************************/
|
|
|
|
#ifndef SIDECARCOMPRESS_H
|
|
#define SIDECARCOMPRESS_H
|
|
|
|
// zstd wrappers for the .ifcview format (v16+). Geometry chunks and metadata
|
|
// blocks are stored zstd-compressed so the network pulls far fewer bytes while
|
|
// keeping HTTP Range streaming intact (unlike server Content-Encoding, which
|
|
// can't be byte-ranged). The baker compresses (desktop only); every loader —
|
|
// desktop and web — decompresses. The web build links the vendored single-file
|
|
// zstd DECODER (third_party/zstddeclib.c); desktop links the full libzstd.
|
|
|
|
#include <cstddef>
|
|
#include <cstdint>
|
|
#include <vector>
|
|
|
|
namespace SidecarCompress {
|
|
|
|
// Decompress a zstd frame in [src, src+src_size) into dst, which must have room
|
|
// for exactly raw_size bytes. Returns false on any zstd error or if the frame
|
|
// doesn't expand to exactly raw_size. Available on all platforms.
|
|
bool decompress(const std::uint8_t* src, std::size_t src_size,
|
|
std::uint8_t* dst, std::size_t raw_size);
|
|
|
|
#if !defined(__EMSCRIPTEN__)
|
|
// Compress [src, src+n) with zstd at `level`. Returns the compressed frame, or
|
|
// an empty vector on error. Bake/desktop only — the web build never compresses.
|
|
std::vector<std::uint8_t> compress(const std::uint8_t* src, std::size_t n,
|
|
int level);
|
|
#endif
|
|
|
|
} // namespace SidecarCompress
|
|
|
|
#endif // SIDECARCOMPRESS_H
|