Glassvision/IfcOpenShell
1
0
Fork 0
mirror of https://github.com/IfcOpenShell/IfcOpenShell.git synced 2026-09-22 00:10:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

ifcparse: give the tokenizer a compile-time policy for what it decodes

Browse Source 
spf_lexer::next() becomes next<Policy>(). full_tokens, the default, is
what the parser has always had. index_tokens is what the lazy index
needs: a string is ended but not decoded, and a number, enumeration or
binary comes back as Token_LITERAL with only its position; names,
keywords and operators are read as before. Each policy compiles to its
own loop from the one implementation, so there is no second tokenizer.

character_decoder gains skip(): the same state machine as the
conversion with the collection compiled out, so an escape such as \S\'
(an apostrophe as the page character) ends the string at the same byte
under both policies. A byte-level scan would have ended it early.

Also fixes a comment that follows a token without whitespace, ",/* x */",
which skip_comment() never saw because the slash had been consumed.

TXG (58 MB), single thread: tokenizing the whole file 194 MB/s with
full_tokens, 249 MB/s with index_tokens; through 64 KB pages 196 and
205 MB/s. The parse itself is unchanged.

This commit was written by an AI coding tool and has not been verified by
a human.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013wcN7XquTfUi4vsKQ4KchL
This commit is contained in:
Dion Moult
2026-09-14 07:08:48 +10:00
parent a026fc698c
commit 9829ebf001
6 changed files with 150 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ifcparse/parse.h
+18
View File
@@ -51,6 +51,20 @@ IFC_PARSE_API std::string encode_spf_string(const std::string& value);
IFC_PARSE_API std::string decode_spf_string(const std::string& value);
/// What a pass over the tokens has to produce. The parser needs every
/// value; the lazy index only needs to know where the tokens are and which
/// of them are instance names, so it ends strings without decoding them and
/// passes over numbers, enumerations and binaries. The choice is a template
/// parameter of spf_lexer::next(), so each pass compiles to its own loop.
struct full_tokens {
static constexpr bool decode_strings = true;
static constexpr bool decode_values = true;
};
struct index_tokens {
static constexpr bool decode_strings = false;
static constexpr bool decode_values = false;
};
/// A stream of tokens to be read from a file_reader.
template <typename Reader>
class IFC_PARSE_API spf_lexer {
@@ -82,6 +96,10 @@ class IFC_PARSE_API spf_lexer {
Reader* stream;
// file* file;
spf_lexer(Reader* stream, ifcopenshell::logger& logger = ifcopenshell::logger::root());
// The next token. With index_tokens a string, number, enumeration or
// binary comes back as Token_LITERAL (Token_STRING for a string) with
// only its position; names, keywords and operators are always read.
template <typename Policy = full_tokens>
token next();
~spf_lexer();
// void TokenString(size_t offset, std::string& result);