fix(scale): kernel-scale hardening — OOM-safe pass skipping + watchdog-safe index recreate (#1323)

Two hazards found by running today's full stack against the Linux kernel
(70,129 files) in the cg1212 repro container:

1. The parallel-synthesis fallback retried a worker-failed pass on the MAIN
   thread. At multi-million-node scale a worker failure is usually a memory
   ceiling, so the retry would OOM the process and take the whole index with
   it. Above 1.5M nodes a failed pass is now skipped with a clear stderr
   message (its synthesized edges are absent; the index completes). Below
   that, the main-thread retry stays — small-scale worker crashes are
   transient and the retry keeps coverage.

2. endBulkEdgeLoad rebuilt all four edge indexes in one synchronous span —
   measured 79s at kernel scale, past the #850 liveness watchdog's 60s
   stall window. A daemon-triggered re-index would have been SIGKILLed right
   after doing the work. Now async with an event-loop yield between builds,
   keeping each stall to a single index (~20s at kernel scale).

Validation: full Linux kernel index to completion in the repro container —
2,048,674 nodes / 6,405,964 edges, EXIT 0, zero passes skipped, on a 2-CPU
VM (worst case: pool disabled, sequential resolution + synthesis) in ~27min.
Phase walls: parse 6.0m, resolution 19.5m (incl. synthesis 6.3m, recreate
79s), maintenance 74s. Suite green (2444).

Also adds docs/design/native-extraction-kernel.md — the spike-validated
design for the native extraction kernel (Rust parse+walk over dubbo's Java:
202ms rayon / 1.07s single-thread vs 4.7s for the current wasm pipeline).

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-16 19:09:01 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 567b4ad4be
commit 4efc6c70e2
5 changed files with 119 additions and 4 deletions
+2 -2
View File
@@ -1339,7 +1339,7 @@ export class ReferenceResolver {
// disables entirely. bulkEdgeLoad hooks (when provided) bracket the batch
// loop with drop/recreate of the non-unique edge indexes on big runs —
// see DatabaseConnection.beginBulkEdgeLoad.
parallel?: { dbPath: string; bulkEdgeLoad?: { begin: () => void; end: () => void } }
parallel?: { dbPath: string; bulkEdgeLoad?: { begin: () => void; end: () => void | Promise<void> } }
): Promise<ResolutionResult> {
// Resolution runs on the indexer's MAIN thread, and the #850 liveness
// watchdog SIGKILLs a process whose event loop stalls past its window (60s
@@ -1583,7 +1583,7 @@ export class ReferenceResolver {
// DatabaseConnection open (schema.sql re-applies IF NOT EXISTS).
if (bulkEdgesActive) {
const tIdx = Date.now();
parallel!.bulkEdgeLoad!.end();
await parallel!.bulkEdgeLoad!.end();
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] edge-index-recreate: ${Date.now() - tIdx}ms`);
}
}