perf(store): resolution ref-index window — kernel-scale resolution 423→276s, 8c envelope ≈11min (§4d round 2) (#1369)
Store-architecture arc round 2. The batched resolution loop reads unresolved_refs ONLY through the status index + the PK keyset pager; the other five ref indexes (from_node, name, file_path, from_name, failed_tail) serve sync-time paths — yet every per-batch DELETE of resolved refs maintained all of them, the biggest single main-thread stage on the dubbo profile (deletes 1.2s of a 5.4s resolution phase) and 50-81s at kernel scale. beginBulkRefLoad/endBulkRefLoad on DatabaseConnection, threaded as refIndexLoad hooks next to the existing bulkEdgeLoad pair with the same minRefsForPool gate (small syncs never pay): drop the five for the loop, rebuild each in one scan at the end — where the table holds only the surviving FAILED refs (resolved rows are deleted by then), so the recreate is near-free. Crash inside the window heals on the next open (schema.sql re-applies CREATE INDEX IF NOT EXISTS). Measured: - dubbo: deletes 1.2 → 0.2s, marks 0.6 → 0.3s, recreate 219ms; wall ~8.5s flat — the freed main-lane time shifts into settle (the worker lane now binds the double-buffer at medium scale). - Linux kernel 8c: resolution 423.4 → 275.9s (deletes 50-81 → 3.2s, backpressure 16.8 → 7.4s — fewer index writes mean less WAL and cheaper folds), ref recreate 10.3s. Envelope ≈ 11.0min, from the 14.8min pre-arc best; <10min-on-8c now needs ~1 more minute. Gates: dubbo/gson dumps byte-identical; linux counts exact 2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478 lines); full suite green ×2 (153 files / 2588 tests). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
f6d8e8fdab
commit
ce0ae30e09
@@ -1388,6 +1388,10 @@ export class ReferenceResolver {
|
||||
parallel?: {
|
||||
dbPath: string;
|
||||
bulkEdgeLoad?: { begin: () => void; end: () => void | Promise<void> };
|
||||
/** unresolved_refs index window for the batched loop — the loop only
|
||||
* reads the status index + PK; dropping the sync-path ref indexes cuts
|
||||
* each per-batch DELETE's B-tree work (DatabaseConnection.beginBulkRefLoad). */
|
||||
refIndexLoad?: { begin: () => void; end: () => void | Promise<void> };
|
||||
backpressure?: () => Promise<void> | null;
|
||||
}
|
||||
): Promise<ResolutionResult> {
|
||||
@@ -1532,6 +1536,16 @@ export class ReferenceResolver {
|
||||
bulkEdgesActive = true;
|
||||
} catch { /* keep the indexes; inserts just pay the per-row maintenance */ }
|
||||
}
|
||||
// Same gate for the ref-index window: the loop's deletes stop maintaining
|
||||
// the five sync-path unresolved_refs indexes, and the end-of-loop rebuild
|
||||
// is near-free (only failed refs survive the loop).
|
||||
let bulkRefsActive = false;
|
||||
if (parallel?.refIndexLoad && total >= minRefsForPool()) {
|
||||
try {
|
||||
parallel.refIndexLoad.begin();
|
||||
bulkRefsActive = true;
|
||||
} catch { /* keep the indexes; deletes just pay the per-row maintenance */ }
|
||||
}
|
||||
|
||||
try {
|
||||
try {
|
||||
@@ -1717,6 +1731,11 @@ export class ReferenceResolver {
|
||||
// Recreate the edge indexes BEFORE synthesis (kind-keyed reads) and on
|
||||
// any error path. A crash before this line is healed by the next
|
||||
// DatabaseConnection open (schema.sql re-applies IF NOT EXISTS).
|
||||
if (bulkRefsActive) {
|
||||
const tRef = Date.now();
|
||||
await parallel!.refIndexLoad!.end();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] ref-index-recreate: ${Date.now() - tRef}ms`);
|
||||
}
|
||||
if (bulkEdgesActive) {
|
||||
const tIdx = Date.now();
|
||||
await parallel!.bulkEdgeLoad!.end();
|
||||
|
||||
Reference in New Issue
Block a user