perf(store): resolution ref-index window — kernel-scale resolution 423→276s, 8c envelope ≈11min (§4d round 2) (#1369)

Store-architecture arc round 2. The batched resolution loop reads
unresolved_refs ONLY through the status index + the PK keyset pager;
the other five ref indexes (from_node, name, file_path, from_name,
failed_tail) serve sync-time paths — yet every per-batch DELETE of
resolved refs maintained all of them, the biggest single main-thread
stage on the dubbo profile (deletes 1.2s of a 5.4s resolution phase)
and 50-81s at kernel scale.

beginBulkRefLoad/endBulkRefLoad on DatabaseConnection, threaded as
refIndexLoad hooks next to the existing bulkEdgeLoad pair with the
same minRefsForPool gate (small syncs never pay): drop the five for
the loop, rebuild each in one scan at the end — where the table holds
only the surviving FAILED refs (resolved rows are deleted by then),
so the recreate is near-free. Crash inside the window heals on the
next open (schema.sql re-applies CREATE INDEX IF NOT EXISTS).

Measured:
- dubbo: deletes 1.2 → 0.2s, marks 0.6 → 0.3s, recreate 219ms; wall
  ~8.5s flat — the freed main-lane time shifts into settle (the worker
  lane now binds the double-buffer at medium scale).
- Linux kernel 8c: resolution 423.4 → 275.9s (deletes 50-81 → 3.2s,
  backpressure 16.8 → 7.4s — fewer index writes mean less WAL and
  cheaper folds), ref recreate 10.3s. Envelope ≈ 11.0min, from the
  14.8min pre-arc best; <10min-on-8c now needs ~1 more minute.

Gates: dubbo/gson dumps byte-identical; linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-19 23:24:14 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent f6d8e8fdab
commit ce0ae30e09
5 changed files with 92 additions and 1 deletions
+19
View File
@@ -1388,6 +1388,10 @@ export class ReferenceResolver {
parallel?: {
dbPath: string;
bulkEdgeLoad?: { begin: () => void; end: () => void | Promise<void> };
/** unresolved_refs index window for the batched loop — the loop only
* reads the status index + PK; dropping the sync-path ref indexes cuts
* each per-batch DELETE's B-tree work (DatabaseConnection.beginBulkRefLoad). */
refIndexLoad?: { begin: () => void; end: () => void | Promise<void> };
backpressure?: () => Promise<void> | null;
}
): Promise<ResolutionResult> {
@@ -1532,6 +1536,16 @@ export class ReferenceResolver {
bulkEdgesActive = true;
} catch { /* keep the indexes; inserts just pay the per-row maintenance */ }
}
// Same gate for the ref-index window: the loop's deletes stop maintaining
// the five sync-path unresolved_refs indexes, and the end-of-loop rebuild
// is near-free (only failed refs survive the loop).
let bulkRefsActive = false;
if (parallel?.refIndexLoad && total >= minRefsForPool()) {
try {
parallel.refIndexLoad.begin();
bulkRefsActive = true;
} catch { /* keep the indexes; deletes just pay the per-row maintenance */ }
}
try {
try {
@@ -1717,6 +1731,11 @@ export class ReferenceResolver {
// Recreate the edge indexes BEFORE synthesis (kind-keyed reads) and on
// any error path. A crash before this line is healed by the next
// DatabaseConnection open (schema.sql re-applies IF NOT EXISTS).
if (bulkRefsActive) {
const tRef = Date.now();
await parallel!.refIndexLoad!.end();
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] ref-index-recreate: ${Date.now() - tRef}ms`);
}
if (bulkEdgesActive) {
const tIdx = Date.now();
await parallel!.bulkEdgeLoad!.end();