Glassvision/IfcOpenShell
1
0
Fork 0
mirror of https://github.com/IfcOpenShell/IfcOpenShell.git synced 2026-08-11 02:02:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

ifcviewer: v14 chunk-contiguous sidecar + progressive network streaming

Browse Source 
Makes large-model streaming over a network actually good — fixing read
amplification, then first-paint latency — building on the byte-range work.

v14 layout + TOC (SidecarLayout, pure + unit-tested)
  The loader chunks meshes by spatial Morton order, but the sidecar stored
  geometry in mesh-id order, so a chunk's meshes were scattered through the
  file: streaming one chunk meant either hundreds of tiny range requests or
  reading (and discarding) everything between them — a 113 MB model fetched
  ~340 MB, a 531 MB model 2.25 GB (4.2x). Fix: at bake, reorder meshes into
  the loader's chunk order and rebuild vertex/index(LOD0+LOD1)/instance
  sections so each chunk is one CONTIGUOUS byte range, and bake a chunk TOC
  ({first_mesh, mesh_count}). The loader builds chunks straight from the TOC
  rather than re-deriving the plan — the float Morton quantisation isn't
  bit-identical across toolchains (x86 baker vs wasm loader), so a re-derived
  plan scatters the chunks. Format bumped to v14 (regenerate sidecars). The
  reorder buckets instances by per-instance mesh_id (the baker never sets
  MeshInfo.first_instance — trusting it scrambled every transform → geometry
  at the origin). Multiset-verified on a 28,900-instance model: every
  instance's placement + geometry preserved. Result: 531 MB fetches 531 MB
  (1.0x) in 72 requests (was 2036).

Progressive streaming (concurrency cap + small chunks)
  Even at 1x, geometry appeared only after ~the whole model arrived: the
  browser multiplexes every in-flight Range request over one HTTP/2 conn, so
  unbounded concurrency (9 in flight) split the bandwidth and nothing finished
  until the end (measured: first paint after 113 of 118 MB / 35 s @ 24 Mbps).
  Cap concurrent chunk loads (kMaxWebInflightChunks=2): the priority-sorted
  top chunks finish and paint first, then the next → first paint 9 s. Chunk
  size dropped 16->4 MB (cheap now that each chunk is one read; matches Cesium
  3D Tiles / xeokit / SVF2) for smoother progression. First-paint is now
  metadata-bound (~10 MB tail) — the next lever.

111/111 unit (new test_sidecar_layout: geometry preserved, contiguous layout,
Morton-identity) + 6/6 web smoke pass; desktop bake (SceneLoader) reorders
before writeSidecar; embedded web sample regenerated to v14.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Dion Moult
2026-06-30 21:22:47 +10:00
parent 46a695266a
commit e1be2f208c
14 changed files with 559 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ifcviewer/SidecarLayout.cpp
+135
View File
@@ -0,0 +1,135 @@
/********************************************************************************
* *
* This file is part of IfcOpenShell. *
* *
* IfcOpenShell is free software: you can redistribute it and/or modify *
* it under the terms of the Lesser GNU General Public License as published by *
* the Free Software Foundation, either version 3.0 of the License, or *
* (at your option) any later version. *
* *
* IfcOpenShell is distributed in the hope that it will be useful, *
* but WITHOUT ANY WARRANTY; without even the implied warranty of *
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the *
* Lesser GNU General Public License for more details. *
* *
* You should have received a copy of the Lesser GNU General Public License *
* along with this program. If not, see <http://www.gnu.org/licenses/>. *
* *
********************************************************************************/
#include "SidecarLayout.h"
#include "ChunkPlanner.h"
#include "InstancedGeometry.h"
#include <cstdint>
#include <vector>
void reorderSidecarByMorton(SidecarData& sd) {
const std::size_t n = sd.meshes.size();
if (n < 2) return;
// Per-mesh centroid + instance count, exactly as the loader computes them
// before chunk planning (average of instance world-AABB centres).
std::vector<float> cx(n, 0.0f), cy(n, 0.0f), cz(n, 0.0f);
std::vector<std::uint32_t> cnt(n, 0);
for (const auto& inst : sd.instances) {
if (inst.mesh_id >= n) continue;
cx[inst.mesh_id] += 0.5f * (inst.world_aabb_min[0] + inst.world_aabb_max[0]);
cy[inst.mesh_id] += 0.5f * (inst.world_aabb_min[1] + inst.world_aabb_max[1]);
cz[inst.mesh_id] += 0.5f * (inst.world_aabb_min[2] + inst.world_aabb_max[2]);
++cnt[inst.mesh_id];
}
for (std::size_t i = 0; i < n; ++i) {
if (cnt[i] > 0) {
const float inv = 1.0f / float(cnt[i]);
cx[i] *= inv; cy[i] *= inv; cz[i] *= inv;
}
}
// order[new_id] = old mesh id, in the loader's Morton order.
const std::vector<std::uint32_t> order =
ChunkPlanner::sortMeshIdsByMorton(n, cx, cy, cz, cnt);
// Greedy-pack the sorted order into chunks (the same plan the loader used
// to derive). Each chunk is a CONSECUTIVE run of `order`, so once we lay
// meshes out in `order` the chunk is a contiguous mesh range — recorded in
// the TOC as {first_mesh, mesh_count}.
std::vector<std::uint32_t> mesh_vertex_count(n, 0);
for (std::size_t i = 0; i < n; ++i) mesh_vertex_count[i] = sd.meshes[i].vertex_count;
const std::vector<std::vector<std::uint32_t>> packed = ChunkPlanner::greedyPackChunks(
order, mesh_vertex_count, INSTANCED_VERTEX_STRIDE_BYTES,
WGPU_CHUNK_VERTEX_BYTES_LIMIT);
sd.chunks.clear();
sd.chunks.reserve(packed.size());
{
std::uint32_t first = 0;
for (const auto& chunk : packed) {
sd.chunks.push_back({first, std::uint32_t(chunk.size())});
first += std::uint32_t(chunk.size());
}
}
// Bucket instances by their (authoritative) mesh_id. We must NOT rely on
// MeshInfo.first_instance: the baker leaves it 0 for every mesh and stores
// instances ungrouped, so first_instance describes nothing. Grouping here
// by mesh_id both reorders instances correctly AND fixes first_instance.
std::vector<std::vector<std::uint32_t>> insts_by_mesh(n);
for (std::uint32_t ii = 0; ii < sd.instances.size(); ++ii) {
const std::uint32_t mid = sd.instances[ii].mesh_id;
if (mid < n) insts_by_mesh[mid].push_back(ii);
}
std::vector<std::uint8_t> new_vertices; new_vertices.reserve(sd.vertices.size());
std::vector<std::uint32_t> new_indices; new_indices.reserve(sd.indices.size());
std::vector<MeshInfo> new_meshes(n);
std::vector<InstanceCpu> new_instances; new_instances.reserve(sd.instances.size());
// Pass A: vertices + LOD0 indices + instances, mesh-by-mesh in the new
// order, recording the new offsets on each MeshInfo.
for (std::uint32_t ni = 0; ni < n; ++ni) {
const std::uint32_t old = order[ni];
const MeshInfo& om = sd.meshes[old];
MeshInfo nm = om; // carries AABB; offsets/instance fields overwritten below
nm.vbo_byte_offset = std::uint32_t(new_vertices.size());
const std::size_t vbytes = std::size_t(om.vertex_count) * INSTANCED_VERTEX_STRIDE_BYTES;
new_vertices.insert(new_vertices.end(),
sd.vertices.begin() + om.vbo_byte_offset,
sd.vertices.begin() + om.vbo_byte_offset + vbytes);
nm.ebo_byte_offset = std::uint32_t(new_indices.size() * sizeof(std::uint32_t));
const std::size_t i0 = om.ebo_byte_offset / sizeof(std::uint32_t);
new_indices.insert(new_indices.end(),
sd.indices.begin() + i0,
sd.indices.begin() + i0 + om.index_count);
nm.first_instance = std::uint32_t(new_instances.size());
nm.instance_count = std::uint32_t(insts_by_mesh[old].size());
for (std::uint32_t ii : insts_by_mesh[old]) {
InstanceCpu ic = sd.instances[ii];
ic.mesh_id = ni;
new_instances.push_back(ic);
}
new_meshes[ni] = nm;
}
// Pass B: LOD1 indices appended after all LOD0 (same global layout as the
// baker), in the new order, so a chunk's LOD1 slice is contiguous too.
for (std::uint32_t ni = 0; ni < n; ++ni) {
const MeshInfo& om = sd.meshes[order[ni]];
MeshInfo& nm = new_meshes[ni];
if (om.lod1_index_count == 0) { nm.lod1_ebo_byte_offset = 0; continue; }
nm.lod1_ebo_byte_offset = std::uint32_t(new_indices.size() * sizeof(std::uint32_t));
const std::size_t l0 = om.lod1_ebo_byte_offset / sizeof(std::uint32_t);
new_indices.insert(new_indices.end(),
sd.indices.begin() + l0,
sd.indices.begin() + l0 + om.lod1_index_count);
}
sd.vertices = std::move(new_vertices);
sd.indices = std::move(new_indices);
sd.meshes = std::move(new_meshes);
sd.instances = std::move(new_instances);
}