ifcparse: sort and index per worker, merge sorted runs

After the parallel parse (and the parallel lazy index) the main thread
did, per instance, the by-type list push, the GlobalId extraction and
the name-table insert, then concatenated every worker's inverse records
and sorted the lot: on a 12-core box that serial merge was most of the
open. Now each worker sorts its own records, builds its own by-type
lists and reads its own GlobalIds; the main thread appends the by-type
lists, inserts names and GlobalIds, and merges the sorted runs pairwise
(inverse_index::merge_sorted, O(n log k)) instead of sorting again.
Instance order, GlobalId precedence and record order are unchanged.
The workers sort their runs in place (inverse_index::sort_in_place, no
radix buffer, no shrink): merge_sorted() copies the runs anyway, and a
buffer allocated on a worker's arena stays there after the open, which
showed as 3–10% more resident memory after a 12-thread open until the
runs sorted in place; peak memory at 12 threads is one transient copy
of the records higher (the merge), single-threaded it is unchanged.

Experiment on top of the series; measured in isolation against the
previous commit, five models, best of three: single-threaded within ±2%
(by construction); 12 threads, strict / lazy: TXG −16% / −25%,
210_King −23% / −26%, OKgate22 −25% / −20%, a 107 MB model −25% / −23%,
a 523 MB model of few large instances −9% / −7%. Together with the
radix sort: strict 12 threads −20% to −29%, lazy 12 threads −31% to
−35% on the ordinary models.

This commit was written by an AI coding tool and has not been verified by
a human.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL

src/ifcparse/storage.h

@@ -456,6 +456,42 @@ namespace ifcopenshell {
             // Finalizes bulk loading. Subsequent add() calls go to the delta.
             // Takes over another index's records, e.g. one built by a parser
             // worker. Both must still be in bulk-load mode (no delta).
+            // Takes over several indexes whose records are each already
+            // sorted (e.g. by their parser workers) by merging them pairwise,
+            // O(n log k), so no sort() follows. Both sides must be in bulk-load
+            // mode (no delta).
+            void merge_sorted(std::vector<inverse_index*> runs) {
+                std::vector<std::vector<inverse_record>> parts;
+                if (!base_.empty()) {
+                    sort();
+                    parts.push_back(std::move(base_));
+                    base_.clear();
+                }
+                for (auto* run : runs) {
+                    run->sort();
+                    parts.push_back(std::move(run->base_));
+                    run->clear();
+                }
+                while (parts.size() > 1) {
+                    std::vector<std::vector<inverse_record>> next;
+                    for (size_t i = 0; i + 1 < parts.size(); i += 2) {
+                        std::vector<inverse_record> merged;
+                        merged.reserve(parts[i].size() + parts[i + 1].size());
+                        std::merge(parts[i].begin(), parts[i].end(), parts[i + 1].begin(), parts[i + 1].end(), std::back_inserter(merged), record_less);
+                        next.push_back(std::move(merged));
+                    }
+                    if (parts.size() % 2 == 1) {
+                        next.push_back(std::move(parts.back()));
+                    }
+                    parts.swap(next);
+                }
+                if (!parts.empty()) {
+                    base_ = std::move(parts.front());
+                }
+                sorted_ = true;
+                invalidate_materialized();
+            }
+
             void append(inverse_index&& other) {
                 if (base_.empty()) {
                     base_ = std::move(other.base_);
@@ -521,6 +557,17 @@ namespace ifcopenshell {
                 }
             }
 
+            // For a worker's run that merge_sorted() copies anyway: no radix buffer
+            // (it would be allocated on the worker's arena and stay there) and no
+            // shrink.
+            void sort_in_place() const {
+                if (!sorted_) {
+                    std::sort(base_.begin(), base_.end(), record_less);
+                    sorted_ = true;
+                    invalidate_materialized();
+                }
+            }
+
             // Visits every live record referencing referenced_id: the base
             // records in record_less order, then the delta in insertion order.
             template <typename Fn>