perf(store): parse-lane index deferral — dubbo fresh init −19%, kernel-scale envelope best-ever 14.2min (§4d round 1) (#1368)
Store-architecture arc round 1 (the cbm speed bar: dubbo warm wall 10.7-11.2s vs their ~7.5). §4d measured dubbo's parse-loop as 94% store-writer busy with B-tree maintenance as the floor (statement batching and sorted inserts already killed at ~zero). This applies the resolution phase's proven edge-index window to the whole parse lane: beginBulkParseLoad/endBulkParseLoad on DatabaseConnection — FRESH-INIT ONLY (incremental runs delete per-file rows through the file_path indexes) — drop all 15 nodes/unresolved_refs/files secondary indexes plus the 4 non-unique edge indexes for the parse phase's mass insert (the UNIQUE edge identity index stays: OR-IGNORE dedup conflicts on it, and its source prefix keeps mid-window reads indexed), then rebuild each in one table scan before resolution, with a yield between builds (the endBulkEdgeLoad watchdog rationale). A crash inside the window heals on the next open — schema.sql re-applies CREATE INDEX IF NOT EXISTS. Measured: - dubbo (cbm bar repo): parse-loop 4,306 → 1,787ms (−58%), rebuild 665ms, warm fresh-init wall 10.5-11.3 → 8.46-9.39s (−19%); the bar gap vs cbm shrinks from ~3s to ~1.1s. - Linux kernel 8c: envelope ≈ 14.2min, best ever (prior 14.8). Parse itself flat (linux parse is extraction-bound, not writer-bound) and the rebuild costs 21.6s — but every downstream phase dropped (resolution 517-589 → 423.4s, edge-recreate 36.5s, synthesis 157.1s, maintenance 16.3s): bulk-rebuilt B-trees are densely packed where incrementally-grown ones are fragmented, so every index-mediated read for the rest of the run pays fewer pages. Gates: dubbo/gson/express/excalidraw full dumps byte-identical (dubbo's canonical 441,270 lines reproduced); linux counts exact 2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478 lines); full suite green ×2 (153 files / 2588 tests). Incremental sync paths untouched by construction (freshDb gate). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
9647771659
commit
f6d8e8fdab
@@ -493,6 +493,11 @@ export class CodeGraph {
|
||||
// triggers, rebuild nodes_fts once from the nodes table afterwards.
|
||||
// Crash inside the window is healed on the next DatabaseConnection.open.
|
||||
this.db.beginBulkNodeLoad();
|
||||
// Fresh-init only: also drop the parse-lane secondary indexes for the
|
||||
// mass insert (the store-writer's B-tree-maintenance floor, plan §4d)
|
||||
// and rebuild each in one scan afterwards. Incremental runs keep them
|
||||
// — they delete per-file rows mid-phase through the file_path indexes.
|
||||
if (freshDb) this.db.beginBulkParseLoad();
|
||||
let result: IndexResult;
|
||||
try {
|
||||
result = await this.orchestrator.indexAll(
|
||||
@@ -506,6 +511,11 @@ export class CodeGraph {
|
||||
freshDb ? { dbPath: this.db.getPath(), fastInit } : null
|
||||
);
|
||||
} finally {
|
||||
if (freshDb) {
|
||||
const tIdx = Date.now();
|
||||
await this.db.endBulkParseLoad();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] parse-index-rebuild: ${Date.now() - tIdx}ms`);
|
||||
}
|
||||
const tFts = Date.now();
|
||||
this.db.endBulkNodeLoad();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] fts-rebuild: ${Date.now() - tFts}ms`);
|
||||
|
||||
Reference in New Issue
Block a user