util.doc: fix the same truncation in pset and property descriptions

The property-set and property extraction sites had the identical
first-paragraph truncation: descriptions ending at a colon lost their
bulleted lists and follow-up paragraphs. Route all four sites (IFC2X3 +
IFC4, pset-level and property-level including recursive children) through
extract_full_description, and make the existing HISTORY: changelog split
case-insensitive, since 21 pset docs use "History:" which only becomes
reachable once the walk passes the first paragraph.

Data refreshed from the same local buildingSMART doc sources with the
merge-only-description discipline: IFC4 376 descriptions completed
(34 pset + 342 property/child), IFC2X3 147 (20 + 127), zero artifact
regressions, all other fields byte-identical.

Example: Pset_DamperOccurrence.SizingMethod previously ended at
"...nominally or with exact measurements:" and now includes
"NOMINAL: Nominal sizing method. EXACT: Exact sizing method."

Generated with the assistance of an AI coding tool.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Petru Conduraru
2026-07-12 15:17:55 +03:00
parent 5977587f03
commit 3d15f581d1
3 changed files with 539 additions and 529 deletions
@@ -480,9 +480,14 @@ class DocExtractor:
with open(md_path, "r", encoding="utf-8-sig") as fi:
# convert markdown to html for easier parsing
html = markdown(fi.read())
property_set_description = BeautifulSoup(html, features="lxml").find("p").text
property_set_description = self.extract_full_description(html)
property_set_description = property_set_description.replace("\n", " ")
property_set_description = property_set_description.split("HISTORY:", 1)[0]
# case-insensitive: some pset docs use "History:" instead of "HISTORY:",
# which only becomes reachable now that extract_full_description() walks
# past the first paragraph.
property_set_description = re.split(
r"HISTORY:", property_set_description, maxsplit=1, flags=re.IGNORECASE
)[0]
property_set_description = property_set_description.strip()
property_set_dict["description"] = property_set_description
else:
@@ -554,7 +559,7 @@ class DocExtractor:
with open(md_path, "r", encoding="utf-8-sig") as fi:
# convert markdown to html for easier parsing
html = markdown(fi.read())
description = BeautifulSoup(html, features="lxml").find("p").text
description = self.extract_full_description(html)
description = description.replace("\n", " ")
description = description.replace("\u00a0", " ")
property_dict["description"] = description
@@ -807,9 +812,14 @@ class DocExtractor:
with open(md_path, "r", encoding="utf-8-sig") as fi:
# convert markdown to html for easier parsing
html = markdown(fi.read())
property_set_description = BeautifulSoup(html, features="lxml").find("p").text
property_set_description = self.extract_full_description(html)
property_set_description = property_set_description.replace("\n", " ")
property_set_description = property_set_description.split("HISTORY:", 1)[0]
# case-insensitive: some pset docs use "History:" instead of "HISTORY:",
# which only becomes reachable now that extract_full_description() walks
# past the first paragraph.
property_set_description = re.split(
r"HISTORY:", property_set_description, maxsplit=1, flags=re.IGNORECASE
)[0]
property_set_description = property_set_description.strip()
property_set_dict["description"] = property_set_description
else:
@@ -892,7 +902,7 @@ class DocExtractor:
with open(md_path, "r", encoding="utf-8-sig") as fi:
# convert markdown to html for easier parsing
html = markdown(fi.read())
description = BeautifulSoup(html, features="lxml").find("p").text
description = self.extract_full_description(html)
description = description.replace("\n", " ")
description = description.replace("\u00a0", " ")
property_dict["description"] = description