perf(store): parse-lane index deferral — dubbo fresh init −19%, kernel-scale envelope best-ever 14.2min (§4d round 1) (#1368)
Store-architecture arc round 1 (the cbm speed bar: dubbo warm wall 10.7-11.2s vs their ~7.5). §4d measured dubbo's parse-loop as 94% store-writer busy with B-tree maintenance as the floor (statement batching and sorted inserts already killed at ~zero). This applies the resolution phase's proven edge-index window to the whole parse lane: beginBulkParseLoad/endBulkParseLoad on DatabaseConnection — FRESH-INIT ONLY (incremental runs delete per-file rows through the file_path indexes) — drop all 15 nodes/unresolved_refs/files secondary indexes plus the 4 non-unique edge indexes for the parse phase's mass insert (the UNIQUE edge identity index stays: OR-IGNORE dedup conflicts on it, and its source prefix keeps mid-window reads indexed), then rebuild each in one table scan before resolution, with a yield between builds (the endBulkEdgeLoad watchdog rationale). A crash inside the window heals on the next open — schema.sql re-applies CREATE INDEX IF NOT EXISTS. Measured: - dubbo (cbm bar repo): parse-loop 4,306 → 1,787ms (−58%), rebuild 665ms, warm fresh-init wall 10.5-11.3 → 8.46-9.39s (−19%); the bar gap vs cbm shrinks from ~3s to ~1.1s. - Linux kernel 8c: envelope ≈ 14.2min, best ever (prior 14.8). Parse itself flat (linux parse is extraction-bound, not writer-bound) and the rebuild costs 21.6s — but every downstream phase dropped (resolution 517-589 → 423.4s, edge-recreate 36.5s, synthesis 157.1s, maintenance 16.3s): bulk-rebuilt B-trees are densely packed where incrementally-grown ones are fragmented, so every index-mediated read for the rest of the run pays fewer pages. Gates: dubbo/gson/express/excalidraw full dumps byte-identical (dubbo's canonical 441,270 lines reproduced); linux counts exact 2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478 lines); full suite green ×2 (153 files / 2588 tests). Incremental sync paths untouched by construction (freshDb gate). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
9647771659
commit
f6d8e8fdab
@@ -513,6 +513,36 @@ during the parse phase, multi-file write transactions, buffer→bind without obj
|
||||
materialization. Note the #1320-arc post-mortem already measured statement batching
|
||||
and sorted inserts as ~zero on this path — B-tree maintenance is the floor.
|
||||
|
||||
**Store-arc round 1 SHIPPED (2026-07-19): parse-lane index deferral.** The
|
||||
first named lever landed as `beginBulkParseLoad`/`endBulkParseLoad`
|
||||
(fresh-init only — incremental runs delete per-file rows through the
|
||||
file_path indexes): the parse window drops all 15 nodes/unresolved_refs/files
|
||||
secondary indexes plus the 4 non-unique edge indexes (identity stays for
|
||||
OR-IGNORE dedup), and rebuilds each in one scan before resolution — the edge
|
||||
window's measured trade applied to the whole parse lane. Results:
|
||||
|
||||
- **dubbo (the cbm bar repo): parse-loop 4,306 → 1,787ms (−58%), rebuild
|
||||
665ms, warm wall 10.5-11.3 → 8.46-9.39s** — the bar gap shrank from ~3s to
|
||||
~1.1s vs cbm's ~7.5s.
|
||||
- **Linux kernel 8c: envelope ≈ 14.2min — best ever** (prior best 14.8). The
|
||||
parse-loop itself stayed ~189s (linux parse is EXTRACTION-bound — the
|
||||
wasm-deferred C tail — unlike writer-bound dubbo) and the rebuild costs
|
||||
21.6s, but every downstream phase dropped: resolution 517-589 → 423.4s,
|
||||
edge-recreate → 36.5s, synthesis → 157.1s, maintenance → 16.3s. Mechanism:
|
||||
bulk-rebuilt B-trees are densely packed where incrementally-grown ones are
|
||||
fragmented, so every index-mediated read for the rest of the run pays fewer
|
||||
pages. The rebuild is the gift that keeps giving downstream.
|
||||
- Gates: dubbo/gson/express/excalidraw dumps byte-identical (441,270-line
|
||||
dubbo dump reproduced), linux counts exact + dump sha `6dd1185b…`
|
||||
reproduced, suite green ×2.
|
||||
|
||||
Remaining store levers, re-ranked: dubbo's residual vs cbm is now resolution
|
||||
(~5.3s of the 8.5s wall) + boot (~1s) — the parse lane is no longer the
|
||||
gap. Multi-file write transactions are likely ~zero on the fastInit path
|
||||
(memory journal, synchronous OFF — same class as the killed statement
|
||||
batching); buffer→bind remains a CPU-side option if the writer re-emerges as
|
||||
the wall.
|
||||
|
||||
## 4. Per-language tracker
|
||||
|
||||
Tiers: **T1** = mostly `.scm` + mapping config. **T2** = needs bespoke pre/post passes kept
|
||||
|
||||
Reference in New Issue
Block a user