perf(store): parse-lane index deferral — dubbo fresh init −19%, kernel-scale envelope best-ever 14.2min (§4d round 1) (#1368)

Store-architecture arc round 1 (the cbm speed bar: dubbo warm wall
10.7-11.2s vs their ~7.5). §4d measured dubbo's parse-loop as 94%
store-writer busy with B-tree maintenance as the floor (statement
batching and sorted inserts already killed at ~zero). This applies the
resolution phase's proven edge-index window to the whole parse lane:

beginBulkParseLoad/endBulkParseLoad on DatabaseConnection — FRESH-INIT
ONLY (incremental runs delete per-file rows through the file_path
indexes) — drop all 15 nodes/unresolved_refs/files secondary indexes
plus the 4 non-unique edge indexes for the parse phase's mass insert
(the UNIQUE edge identity index stays: OR-IGNORE dedup conflicts on it,
and its source prefix keeps mid-window reads indexed), then rebuild
each in one table scan before resolution, with a yield between builds
(the endBulkEdgeLoad watchdog rationale). A crash inside the window
heals on the next open — schema.sql re-applies CREATE INDEX IF NOT
EXISTS.

Measured:
- dubbo (cbm bar repo): parse-loop 4,306 → 1,787ms (−58%), rebuild
  665ms, warm fresh-init wall 10.5-11.3 → 8.46-9.39s (−19%); the bar
  gap vs cbm shrinks from ~3s to ~1.1s.
- Linux kernel 8c: envelope ≈ 14.2min, best ever (prior 14.8). Parse
  itself flat (linux parse is extraction-bound, not writer-bound) and
  the rebuild costs 21.6s — but every downstream phase dropped
  (resolution 517-589 → 423.4s, edge-recreate 36.5s, synthesis 157.1s,
  maintenance 16.3s): bulk-rebuilt B-trees are densely packed where
  incrementally-grown ones are fragmented, so every index-mediated read
  for the rest of the run pays fewer pages.

Gates: dubbo/gson/express/excalidraw full dumps byte-identical
(dubbo's canonical 441,270 lines reproduced); linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests). Incremental
sync paths untouched by construction (freshDb gate).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

src/db/index.ts

@@ -151,6 +151,76 @@ export class DatabaseConnection {
     this.recreateFtsTriggers();
   }
 
+  /**
+   * NON-UNIQUE secondary indexes maintained per-row during the parse phase's
+   * bulk inserts — the store-architecture arc's first lever (plan §4d: dubbo's
+   * parse-loop wall is 94% store-writer busy, and the #1320 post-mortem showed
+   * statement batching and sorted inserts are ~zero on this path because
+   * B-TREE MAINTENANCE is the floor). A fresh init writes every row of
+   * nodes/unresolved_refs/files exactly once and reads none of them until
+   * resolution, so the parse window can drop all of these and rebuild each in
+   * one table scan afterwards — the same measured trade as the resolution
+   * phase's edge-index window (2.8s → 1.1s inserting, ~0.3s recreating).
+   * Primary keys and UNIQUE constraints stay (upserts and OR-IGNORE dedup
+   * conflict on them).
+   */
+  private static readonly BULK_PARSE_INDEX_NAMES = [
+    'idx_nodes_kind',
+    'idx_nodes_name',
+    'idx_nodes_qualified_name',
+    'idx_nodes_file_path',
+    'idx_nodes_language',
+    'idx_nodes_file_line',
+    'idx_nodes_lower_name',
+    'idx_unresolved_from_node',
+    'idx_unresolved_name',
+    'idx_unresolved_file_path',
+    'idx_unresolved_from_name',
+    'idx_unresolved_status',
+    'idx_unresolved_failed_tail',
+    'idx_files_language',
+    'idx_files_modified_at',
+  ] as const;
+
+  /**
+   * Enter bulk-parse-load mode (FRESH-INIT ONLY — the caller gates on a fresh
+   * DB, because an incremental index deletes per-file rows mid-phase and needs
+   * the file_path indexes): drop every parse-lane secondary index, including
+   * the four non-unique edge indexes (parse inserts contains-edges too; the
+   * UNIQUE identity index stays for INSERT OR IGNORE dedup, and its `source`
+   * prefix keeps source-keyed reads indexed, as in the edge window). MUST be
+   * paired with endBulkParseLoad(); a crash inside the window is healed on the
+   * next DatabaseConnection open (schema.sql re-applies CREATE INDEX IF NOT
+   * EXISTS).
+   */
+  beginBulkParseLoad(): void {
+    for (const idx of DatabaseConnection.BULK_PARSE_INDEX_NAMES) {
+      this.db.exec(`DROP INDEX IF EXISTS ${idx}`);
+    }
+    this.beginBulkEdgeLoad();
+  }
+
+  /**
+   * Leave bulk-parse-load mode: recreate everything the window dropped, one
+   * table scan per index, with a yield between statements (same
+   * liveness-watchdog rationale as endBulkEdgeLoad — at kernel scale each
+   * build is a long synchronous scan). The edge indexes are rebuilt here too,
+   * so paths that never enter the resolution phase's own bulk-edge window
+   * (small runs) are left with a complete schema; the batched resolver's
+   * beginBulkEdgeLoad simply re-drops them (DROP IF EXISTS — idempotent).
+   */
+  async endBulkParseLoad(): Promise<void> {
+    const schemaPath = path.join(__dirname, 'schema.sql');
+    const schema = fs.readFileSync(schemaPath, 'utf-8');
+    for (const idx of DatabaseConnection.BULK_PARSE_INDEX_NAMES) {
+      const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${idx}\\b[^;]*;`));
+      if (!m) throw new Error(`schema.sql: parse index ${idx} not found for bulk-load recreation`);
+      this.db.exec(m[0]);
+      await new Promise((resolve) => setImmediate(resolve));
+    }
+    await this.endBulkEdgeLoad();
+  }
+
   /**
    * Names of the NON-UNIQUE edge indexes dropped for a bulk edge load.
    * idx_edges_identity deliberately stays: INSERT OR IGNORE's dedup conflicts

src/index.ts

@@ -493,6 +493,11 @@ export class CodeGraph {
         // triggers, rebuild nodes_fts once from the nodes table afterwards.
         // Crash inside the window is healed on the next DatabaseConnection.open.
         this.db.beginBulkNodeLoad();
+        // Fresh-init only: also drop the parse-lane secondary indexes for the
+        // mass insert (the store-writer's B-tree-maintenance floor, plan §4d)
+        // and rebuild each in one scan afterwards. Incremental runs keep them
+        // — they delete per-file rows mid-phase through the file_path indexes.
+        if (freshDb) this.db.beginBulkParseLoad();
         let result: IndexResult;
         try {
           result = await this.orchestrator.indexAll(
@@ -506,6 +511,11 @@ export class CodeGraph {
             freshDb ? { dbPath: this.db.getPath(), fastInit } : null
           );
         } finally {
+          if (freshDb) {
+            const tIdx = Date.now();
+            await this.db.endBulkParseLoad();
+            if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] parse-index-rebuild: ${Date.now() - tIdx}ms`);
+          }
           const tFts = Date.now();
           this.db.endBulkNodeLoad();
           if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] fts-rebuild: ${Date.now() - tFts}ms`);