Glassvision/IfcOpenShell
1
0
Fork 0
mirror of https://github.com/IfcOpenShell/IfcOpenShell.git synced 2026-08-09 09:21:46 +00:00
Code Issues Packages Projects Releases Wiki Activity

ifcviewer-web: stream user sidecars via Blob.slice byte ranges (#88)

Browse Source 
Picked files are no longer copied whole into the wasm heap. The browser
File object stays in JS (Module.__ifcvFile) and is read lazily through
Blob.slice byte ranges, so a 200-500 MB sidecar never enters wasm linear
memory — only chunk-sized slices do.

Mechanism (web-only, #if __EMSCRIPTEN__):

  - JS glue (EM_JS): ifcvFileSize + ifcvReadRangeInto — slice [off,off+n)
    of the File and copy it into a caller-provided heap pointer, then call
    back _ifcv_on_range_done. No malloc across the boundary; C pre-sizes
    the destination from the read plan.
  - webReadRangesAsync: reuses planSidecarReadRanges to coalesce a range
    set into Blob.slice reads (1 MB gap — each slice is an async hop),
    scatters them into a destination laid out in input order, and fires a
    continuation when the whole set lands. An in-flight map keyed by id
    survives unordered_map rehash (scratch buffers are heap-owned).
  - loadSidecarFromBlobWeb: async metadata load — head (16 B) -> index
    count -> tail-to-EOF -> parseSidecarHead/Tail -> applyCachedModel, then
    tags the model streaming_from_blob and frames it.
  - driveStreamingLoads: blob-sourced models route to beginWebChunkLoad
    (async vertex+index range reads -> applyStreamedChunk in the callback),
    holding is_loading until the bytes arrive. The embedded MEMFS sample
    keeps the synchronous fopen path.

shell.html stashes the File and calls _load_sidecar_from_blob_c instead of
FS.writeFile'ing the whole thing; EXPORTED_RUNTIME_METHODS=['FS'] dropped.
Desktop is untouched (the new members + driveStreamingLoads branch are all
emscripten-guarded). Web links clean; desktop rebuilds; 107/107 unit tests
pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dion Moult
2026-06-29 17:43:53 +10:00
parent a0ba3c0b98
commit 584504dcdc
6 changed files with 323 additions and 60 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ifcviewer-web/CMakeLists.txt
+13 -18
View File
@@ -97,31 +97,26 @@ target_link_options(IfcViewerWeb PRIVATE
# that starves the device promise (observed: ~10s delay in Firefox).
"-sEXIT_RUNTIME=0"
# Expose the C entry points to JS. _raf_tick_c drives the RAF loop
# (shell.html); _load_uploaded_model_c loads a user-picked sidecar
# that JS has written into MEMFS. EMSCRIPTEN_KEEPALIVE alone keeps
# the symbols in the binary but doesn't add them to Module.
"-sEXPORTED_FUNCTIONS=['_main','_raf_tick_c','_load_uploaded_model_c']"
# FS lets shell.html's file-browse handler write the picked file's
# bytes into MEMFS before calling _load_uploaded_model_c (which
# fopen()s the virtual path). Drag-drop is intentionally not used —
# it depends on an X11 drag source (file manager), which a minimal
# WM may not provide; the native file chooser is WM-independent.
"-sEXPORTED_RUNTIME_METHODS=['FS']"
# (shell.html); _load_sidecar_from_blob_c loads a user-picked File via
# byte-range Blob.slice reads; _ifcv_on_range_done is the completion
# callback the JS range reader invokes when a slice has landed in the
# heap. EMSCRIPTEN_KEEPALIVE alone keeps the symbols in the binary but
# doesn't add them to Module.
"-sEXPORTED_FUNCTIONS=['_main','_raf_tick_c','_load_sidecar_from_blob_c','_ifcv_on_range_done']"
# Streaming + chunked geometry want a heap that can grow as buffers
# arrive. 256 MB initial, 2 GB ceiling (matches the wasm32 pointer
# cap; --shared64 / MEMORY64 would lift this later if we need it).
"-sALLOW_MEMORY_GROWTH=1"
"-sINITIAL_MEMORY=268435456" # 256 MB
"-sMAXIMUM_MEMORY=2147483648" # 2 GB
# FETCH lets emscripten_fetch issue HTTP Range requests for the
# sidecar byte-range loader. Not used yet by the scaffold but
# needed by the upcoming #27 web streaming I/O backend.
# FETCH lets emscripten_fetch issue HTTP Range requests. The local
# file path (#88) reads byte ranges via Blob.slice and does NOT need
# this; it's retained for the remote-URL Range backend (follow-up).
"-sFETCH=1"
# Bundle a small sample sidecar into Emscripten's MEMFS so the
# scaffold can prove the load path end-to-end without needing
# emscripten_fetch + COOP/COEP wiring. The @ separator mounts the
# file at the virtual path the wasm uses to fopen() it. Replaced
# by an emscripten_fetch + Range backend in #88.
# Bundle a small sample sidecar into Emscripten's MEMFS so the page
# renders something on first load without a user pick. The @ separator
# mounts the file at the virtual path the wasm fopen()s. User-picked
# files instead stream via Blob.slice byte ranges (load_sidecar_from_blob_c).
"--embed-file=${CMAKE_CURRENT_SOURCE_DIR}/sample.ifcview@/sample.ifcview"
# Shell template wraps the JS output in our canvas page.
"--shell-file=${CMAKE_CURRENT_SOURCE_DIR}/shell.html"
src/ifcviewer-web/main_web.cpp
+15 -22
View File
@@ -41,10 +41,6 @@ namespace {
// WebViewportHost selector below.
constexpr const char* kCanvasSelector = "#viewer-canvas";
// MEMFS path the file-browse handler writes the picked sidecar to, and
// that load_uploaded_model_c reads back. Must match shell.html.
constexpr const char* kUploadPath = "/uploads/model.ifcview";
struct AppState {
WebViewportHost host{ kCanvasSelector };
ViewportCore core{ &host };
@@ -156,26 +152,21 @@ extern "C" EMSCRIPTEN_KEEPALIVE void raf_tick_c(void* user) {
}
}
// Called from shell.html's file-browse handler after it has written the
// picked file's bytes into MEMFS at kUploadPath. Replaces whatever is
// currently loaded (the embedded sample on first use, or a prior upload)
// with the new sidecar and frames it. Geometry becomes resident over the
// next frames via render()'s inline driveStreamingLoads. Exported to JS
// via EXPORTED_FUNCTIONS in CMakeLists.txt.
extern "C" EMSCRIPTEN_KEEPALIVE void load_uploaded_model_c() {
// Called from shell.html's file-browse handler after it has stashed the
// picked File on Module.__ifcvFile. Replaces whatever is currently loaded
// (the embedded sample on first use, or a prior pick) with the new sidecar.
// Byte-range (#88): the whole file is NOT copied into the wasm heap — the
// metadata is read via Blob.slice and chunk bytes stream per-chunk, so a
// 500 MB sidecar stays in the browser File object. Asynchronous: this
// returns immediately and the model frames itself from the JS completion
// callback. Exported to JS via EXPORTED_FUNCTIONS in CMakeLists.txt.
extern "C" EMSCRIPTEN_KEEPALIVE void load_sidecar_from_blob_c() {
if (!g_app || !g_app->ready) return;
// resetScene drops the previous model's GPU resources so a fresh load
// replaces rather than accumulates (loadSidecarFromPath appends).
// replaces rather than accumulates (loadSidecar* appends).
g_app->core.resetScene();
const unsigned int mid = g_app->core.loadSidecarFromPath(kUploadPath);
if (mid == 0) {
Log::warn() << "ifcviewer-web: uploaded model load failed";
return;
}
g_app->core.viewAll();
g_app->host.requestFrame();
Log::info() << "ifcviewer-web: loaded uploaded model (id " << mid << ")";
g_app->core.loadSidecarFromBlobWeb();
}
int main(int /*argc*/, char** /*argv*/) {
@@ -199,8 +190,10 @@ int main(int /*argc*/, char** /*argv*/) {
g_app->core.buildPickPipeline();
// Load the embedded sample sidecar (mounted into MEMFS via
// --embed-file in CMakeLists.txt). Replaced by an
// emscripten_fetch + Range backend in #88.
// --embed-file in CMakeLists.txt). The sample stays on the
// synchronous MEMFS read; user-picked files go through the
// Blob.slice byte-range path (load_sidecar_from_blob_c) so large
// sidecars never enter the wasm heap.
if (!g_app->core.loadSidecarFromPath("/sample.ifcview")) {
Log::warn() << "ifcviewer-web: sample sidecar load failed";
}
src/ifcviewer-web/shell.html
+16 -20
View File
@@ -88,35 +88,31 @@
statusEl.classList.add('error');
}
// File-browse loading. The picked file's bytes are written into MEMFS
// (the same virtual FS the wasm fopen()s) and then load_uploaded_model_c
// is invoked to read + render it. No drag-drop: that needs an X11 drag
// source (a file manager), which a minimal WM may not provide; the
// native file chooser this button opens is WM-independent.
// File-browse loading (#88, byte-range). The picked File object is stashed
// on Module.__ifcvFile and load_sidecar_from_blob_c reads it lazily via
// Blob.slice — the file is NOT copied into the wasm heap, so a 500 MB
// sidecar stays in the browser File object and only chunk-sized slices
// ever cross into wasm. No drag-drop: that needs an X11 drag source (a
// file manager), which a minimal WM may not provide; the native file
// chooser this button opens is WM-independent.
var openBtn = document.getElementById('open-btn');
var fileInput = document.getElementById('file-input');
openBtn.addEventListener('click', function() { fileInput.click(); });
fileInput.addEventListener('change', function(ev) {
var f = ev.target.files && ev.target.files[0];
if (!f) return;
if (!Module.FS || !Module._load_uploaded_model_c) {
if (!Module._load_sidecar_from_blob_c) {
statusEl.textContent += 'viewer not ready yet — wait for WebGPU init\n';
return;
}
var reader = new FileReader();
reader.onload = function() {
try {
var bytes = new Uint8Array(reader.result);
try { Module.FS.mkdir('/uploads'); } catch (e) { /* already exists */ }
Module.FS.writeFile('/uploads/model.ifcview', bytes);
Module._load_uploaded_model_c();
} catch (e) {
statusEl.textContent += 'load failed: ' + e + '\n';
statusEl.classList.add('error');
}
fileInput.value = ''; // let the same file be re-picked
};
reader.readAsArrayBuffer(f);
try {
Module.__ifcvFile = f; // kept alive for lazy Blob.slice reads
Module._load_sidecar_from_blob_c();
} catch (e) {
statusEl.textContent += 'load failed: ' + e + '\n';
statusEl.classList.add('error');
}
fileInput.value = ''; // let the same file be re-picked
});
</script>
<!-- IfcViewerWeb.js is emitted alongside this shell by the emcc build;
src/ifcviewer/ModelGpuData.h
+5
View File
@@ -277,6 +277,11 @@ struct ModelGpuData {
std::string streaming_file_path;
uint64_t streaming_vertex_section_offset = 0;
uint64_t streaming_index_section_offset = 0;
// Web only: chunk byte ranges come from the JS-registered File via
// Blob.slice (async), not from a synchronous fopen on streaming_file_path.
// Set by loadSidecarFromBlobWeb so driveStreamingLoads routes this model
// through the async blob path instead of the MEMFS sync read.
bool streaming_from_blob = false;
// For each mesh in meshes[], the chunk it lives in plus the chunk-local
// offsets into that chunk's vertex_storage and index_buffer. Populated
src/ifcviewer/ViewportCore.cpp
+257
View File
@@ -2275,6 +2275,18 @@ void ViewportCore::driveStreamingLoads() {
// queue requests with nothing to drain them. Chunks would
// never go resident.
#if defined(__EMSCRIPTEN__)
// Blob-sourced models read chunk bytes asynchronously via
// Blob.slice (the whole file is never in the heap). The chunk goes
// resident in the JS completion callback; hold is_loading until then
// so it isn't re-issued every frame. The embedded MEMFS sample falls
// through to the synchronous fopen path below.
if (cand.m->streaming_from_blob) {
c.is_loading = true;
c.last_visible_frame_idx = streaming_frame_idx_;
beginWebChunkLoad(cand.mid, cand.ci);
++enqueued;
continue;
}
const bool use_sync = true;
#else
const bool use_sync = !pending_screenshot_path_.empty();
@@ -3047,6 +3059,251 @@ std::uint32_t ViewportCore::loadSidecarFromPath(const std::string& path) {
return mid;
}
#if defined(__EMSCRIPTEN__)
// ===========================================================================
// Web byte-range streaming (#88): Blob.slice source + async chunk loads
// ===========================================================================
//
// The desktop streaming path fopen()s the sidecar and fread()s chunk byte
// ranges synchronously from a worker thread. On web there is no worker (no
// pthreads yet) and Blob.slice() is inherently async, so chunk bytes are
// pulled through the JS event loop: webReadRangesAsync issues one Blob.slice
// per coalesced read plan, scatters the bytes into the destination, then
// invokes a continuation once the whole range set has landed. The picked
// File stays in JS (Module.__ifcvFile) — only chunk-sized slices ever enter
// the wasm heap, so a 500 MB sidecar never does.
namespace {
// Size of the JS-registered File in bytes, or 0 if none. Bounds the metadata
// tail read (index-section end .. EOF).
EM_JS(double, ifcvFileSize, (void), {
return (Module["__ifcvFile"] && Module["__ifcvFile"].size)
? Module["__ifcvFile"].size : 0;
});
// Read [offset, offset+size) of the registered File into dst (which must hold
// `size` bytes), then call back _ifcv_on_range_done(reqId, ok). Async — the
// Blob is sliced and its ArrayBuffer copied into the wasm heap on resolve.
EM_JS(void, ifcvReadRangeInto, (int reqId, double offset, double size, void* dst), {
var f = Module["__ifcvFile"];
if (!f) { Module["_ifcv_on_range_done"](reqId, 0); return; }
f.slice(offset, offset + size).arrayBuffer().then(function(buf) {
HEAPU8.set(new Uint8Array(buf), dst);
Module["_ifcv_on_range_done"](reqId, 1);
}).catch(function(e) {
Module["_ifcv_on_range_done"](reqId, 0);
});
});
// One in-flight multi-range read: a sequence of coalesced plans, each read
// into `scratch` then scattered into `out`. `done(ok, out)` fires once every
// plan has landed, or on the first failure.
struct WebRangeRead {
std::vector<SidecarReadPlan> plans;
std::size_t plan_idx = 0;
std::vector<std::uint8_t> scratch;
std::vector<std::uint8_t> out;
std::function<void(bool, std::vector<std::uint8_t>&&)> done;
};
std::unordered_map<int, WebRangeRead> g_web_reads;
int g_web_read_next = 1;
// Issue the current plan's Blob.slice, or finish (success) if all plans done.
void webIssueCurrentPlan(int id) {
auto it = g_web_reads.find(id);
if (it == g_web_reads.end()) return;
WebRangeRead& r = it->second;
if (r.plan_idx >= r.plans.size()) {
auto done = std::move(r.done);
std::vector<std::uint8_t> out = std::move(r.out);
g_web_reads.erase(it);
if (done) done(true, std::move(out));
return;
}
const SidecarReadPlan& p = r.plans[r.plan_idx];
r.scratch.assign(std::size_t(p.read_size), 0);
ifcvReadRangeInto(id, double(p.file_offset), double(p.read_size),
r.scratch.data());
}
// Read `ranges` (section-relative (offset,size)) into a destination laid out
// in input order, then call done(true, bytes). On any failure: done(false,{}).
// `section_offset` makes the offsets absolute (pass 0 if already absolute).
void webReadRangesAsync(
std::uint64_t section_offset,
const std::vector<std::pair<std::uint64_t, std::uint64_t>>& ranges,
std::function<void(bool, std::vector<std::uint8_t>&&)> done) {
std::uint64_t total = 0;
for (const auto& rg : ranges) total += rg.second;
WebRangeRead r;
r.out.assign(std::size_t(total), 0);
// Coalesce within 1 MB: each Blob.slice is an async round trip, so a
// generous gap trades a few wasted bytes for far fewer JS hops.
r.plans = planSidecarReadRanges(section_offset, ranges, std::uint64_t(1) << 20);
r.done = std::move(done);
if (r.plans.empty()) { // nothing to read — complete synchronously
if (r.done) r.done(true, std::move(r.out));
return;
}
const int id = g_web_read_next++;
g_web_reads.emplace(id, std::move(r));
webIssueCurrentPlan(id);
}
} // namespace
// JS completion callback for one Blob.slice plan. Scatters the landed bytes
// and advances to the next plan, or fails the whole read. Exported as
// _ifcv_on_range_done (see ifcviewer-web/CMakeLists.txt).
extern "C" EMSCRIPTEN_KEEPALIVE void ifcv_on_range_done(int reqId, int ok) {
auto it = g_web_reads.find(reqId);
if (it == g_web_reads.end()) return;
WebRangeRead& r = it->second;
if (!ok) {
auto done = std::move(r.done);
g_web_reads.erase(it);
if (done) done(false, {});
return;
}
const SidecarReadPlan& p = r.plans[r.plan_idx];
for (const auto& s : p.slices) {
std::memcpy(r.out.data() + s.dst_offset,
r.scratch.data() + s.src_offset, std::size_t(s.bytes));
}
++r.plan_idx;
webIssueCurrentPlan(reqId);
}
void ViewportCore::beginWebChunkLoad(std::uint32_t model_id, std::size_t chunk_idx) {
auto it = models_gpu_.find(model_id);
if (it == models_gpu_.end()) return;
ModelGpuData& m = it->second;
if (chunk_idx >= m.chunks.size()) return;
const StreamingThread::Request req = makeChunkRequest(m, chunk_idx, model_id);
const std::uint64_t vsec = req.vertex_section_offset;
const std::uint64_t isec = req.index_section_offset;
const std::vector<std::pair<std::uint64_t, std::uint64_t>> v_ranges = req.v_ranges;
// i_ranges are (first_u32, count_u32); convert to byte ranges.
std::vector<std::pair<std::uint64_t, std::uint64_t>> i_byte_ranges;
i_byte_ranges.reserve(req.i_ranges.size());
for (const auto& [first_u32, count] : req.i_ranges)
i_byte_ranges.emplace_back(first_u32 * 4u, count * 4u);
// Read vertex ranges, then index ranges, then apply. Re-look-up the model
// in each callback: a resetScene() could have landed mid-flight, in which
// case the model id is gone and we simply drop the result.
webReadRangesAsync(vsec, v_ranges,
[this, model_id, chunk_idx, isec, i_byte_ranges]
(bool ok, std::vector<std::uint8_t>&& vbytes) {
auto mit = models_gpu_.find(model_id);
if (mit == models_gpu_.end()) return;
if (chunk_idx >= mit->second.chunks.size()) return;
if (!ok) { mit->second.chunks[chunk_idx].is_loading = false; return; }
auto vb = std::make_shared<std::vector<std::uint8_t>>(std::move(vbytes));
webReadRangesAsync(isec, i_byte_ranges,
[this, model_id, chunk_idx, vb]
(bool ok2, std::vector<std::uint8_t>&& ibytes) {
auto mit2 = models_gpu_.find(model_id);
if (mit2 == models_gpu_.end()) return;
ModelGpuData& mm = mit2->second;
if (chunk_idx >= mm.chunks.size()) return;
if (!ok2) { mm.chunks[chunk_idx].is_loading = false; return; }
std::vector<std::uint32_t> idx(ibytes.size() / sizeof(std::uint32_t));
if (!idx.empty())
std::memcpy(idx.data(), ibytes.data(),
idx.size() * sizeof(std::uint32_t));
if (!applyStreamedChunk(mm, chunk_idx, *vb, idx))
mm.chunks[chunk_idx].is_loading = false; // pool full; retry later
else
host_->requestFrame();
});
});
}
void ViewportCore::loadSidecarFromBlobWeb() {
if (!device_ || !queue_) {
Log::warn() << "loadSidecarFromBlobWeb: wgpu not initialised";
return;
}
const double fsize = ifcvFileSize();
if (fsize <= 0.0) {
Log::warn() << "loadSidecarFromBlobWeb: no File registered";
return;
}
// Head (16 B) → num_vertex_bytes; then the 4-byte index count after the
// vertex section; then the metadata tail (index-section end .. EOF). Each
// hop is a tiny Blob.slice; the bulk vertex/index sections are never read
// here — they stream per chunk through beginWebChunkLoad.
webReadRangesAsync(0, {{0, SIDECAR_HEAD_BYTES}},
[this, fsize](bool ok, std::vector<std::uint8_t>&& head) {
std::uint32_t nvb = 0;
if (!ok || !parseSidecarHead(head.data(), head.size(), nvb)) {
Log::warn() << "loadSidecarFromBlobWeb: bad sidecar head";
return;
}
const std::uint64_t vsec = SIDECAR_HEAD_BYTES;
const std::uint64_t idx_count_off = std::uint64_t(SIDECAR_HEAD_BYTES) + nvb;
webReadRangesAsync(0, {{idx_count_off, 4}},
[this, fsize, nvb, vsec, idx_count_off]
(bool ok2, std::vector<std::uint8_t>&& cnt) {
if (!ok2 || cnt.size() < 4) {
Log::warn() << "loadSidecarFromBlobWeb: short index count";
return;
}
std::uint32_t num_indices = 0;
std::memcpy(&num_indices, cnt.data(), 4);
const std::uint64_t isec = idx_count_off + 4;
const std::uint64_t tail_off = isec + std::uint64_t(num_indices) * 4u;
if (double(tail_off) > fsize) {
Log::warn() << "loadSidecarFromBlobWeb: tail offset past EOF";
return;
}
const std::uint64_t tail_len = std::uint64_t(fsize) - tail_off;
webReadRangesAsync(0, {{tail_off, tail_len}},
[this, vsec, nvb, isec, num_indices]
(bool ok3, std::vector<std::uint8_t>&& tail) {
if (!ok3) {
Log::warn() << "loadSidecarFromBlobWeb: tail read failed";
return;
}
StreamingSidecar sc;
sc.file_path = "blob:model.ifcview";
sc.vertex_section_offset = vsec;
sc.vertex_total_bytes = nvb;
sc.index_section_offset = isec;
sc.index_total_count = num_indices;
if (!parseSidecarTail(tail.data(), tail.size(), sc.meta)) {
Log::warn() << "loadSidecarFromBlobWeb: bad metadata tail";
return;
}
const std::size_t n_meshes = sc.meta.meshes.size();
const std::size_t n_instances = sc.meta.instances.size();
const std::uint32_t mid = next_model_id_++;
applyCachedModel(mid, std::move(sc));
auto mit = models_gpu_.find(mid);
if (mit != models_gpu_.end())
mit->second.streaming_from_blob = true;
viewAll();
host_->requestFrame();
Log::info() << "ifcviewer-web: loaded blob sidecar (id "
<< mid << ", " << n_meshes << " meshes, "
<< n_instances << " instances)";
});
});
});
}
#endif // __EMSCRIPTEN__
void ViewportCore::finalizeModel(std::uint32_t model_id) {
auto it = pending_direct_loads_.find(model_id);
if (it == pending_direct_loads_.end()) {
src/ifcviewer/ViewportCore.h
+17
View File
@@ -356,6 +356,23 @@ public:
// mismatch) and the freshly-assigned model_id on success.
std::uint32_t loadSidecarFromPath(const std::string& path);
#if defined(__EMSCRIPTEN__)
// Web byte-range load (#88). Loads a sidecar from the JS-registered
// File (Module.__ifcvFile) WITHOUT copying the whole file into the
// wasm heap: the head + tail metadata are read via Blob.slice, the
// streaming model is built, and it is tagged blob-sourced so each
// chunk's vertex/index byte ranges are pulled lazily through the async
// path. Asynchronous — returns immediately and frames the model from
// the JS completion callback. resetScene() first to replace.
void loadSidecarFromBlobWeb();
// Kick off the async blob read of one chunk's vertex + index byte
// ranges. applyStreamedChunk runs in the JS completion callback;
// c.is_loading is held until then. No-op if the model/chunk vanished
// mid-flight (e.g. a resetScene landed between issue and completion).
void beginWebChunkLoad(std::uint32_t model_id, std::size_t chunk_idx);
#endif
// Direct-load (bonsai-side) entry points. Bonsai's SceneLoader feeds
// the viewer one mesh + one instance at a time, then calls
// finalizeModel once everything's staged. The staging map lives on