fix(scale): kernel-scale hardening — OOM-safe pass skipping + watchdog-safe index recreate (#1323)
Two hazards found by running today's full stack against the Linux kernel (70,129 files) in the cg1212 repro container: 1. The parallel-synthesis fallback retried a worker-failed pass on the MAIN thread. At multi-million-node scale a worker failure is usually a memory ceiling, so the retry would OOM the process and take the whole index with it. Above 1.5M nodes a failed pass is now skipped with a clear stderr message (its synthesized edges are absent; the index completes). Below that, the main-thread retry stays — small-scale worker crashes are transient and the retry keeps coverage. 2. endBulkEdgeLoad rebuilt all four edge indexes in one synchronous span — measured 79s at kernel scale, past the #850 liveness watchdog's 60s stall window. A daemon-triggered re-index would have been SIGKILLed right after doing the work. Now async with an event-loop yield between builds, keeping each stall to a single index (~20s at kernel scale). Validation: full Linux kernel index to completion in the repro container — 2,048,674 nodes / 6,405,964 edges, EXIT 0, zero passes skipped, on a 2-CPU VM (worst case: pool disabled, sequential resolution + synthesis) in ~27min. Phase walls: parse 6.0m, resolution 19.5m (incl. synthesis 6.3m, recreate 79s), maintenance 74s. Suite green (2444). Also adds docs/design/native-extraction-kernel.md — the spike-validated design for the native extraction kernel (Rust parse+walk over dubbo's Java: 202ms rayon / 1.07s single-thread vs 4.7s for the current wasm pipeline). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
567b4ad4be
commit
4efc6c70e2
@@ -1339,7 +1339,7 @@ export class ReferenceResolver {
|
||||
// disables entirely. bulkEdgeLoad hooks (when provided) bracket the batch
|
||||
// loop with drop/recreate of the non-unique edge indexes on big runs —
|
||||
// see DatabaseConnection.beginBulkEdgeLoad.
|
||||
parallel?: { dbPath: string; bulkEdgeLoad?: { begin: () => void; end: () => void } }
|
||||
parallel?: { dbPath: string; bulkEdgeLoad?: { begin: () => void; end: () => void | Promise<void> } }
|
||||
): Promise<ResolutionResult> {
|
||||
// Resolution runs on the indexer's MAIN thread, and the #850 liveness
|
||||
// watchdog SIGKILLs a process whose event loop stalls past its window (60s
|
||||
@@ -1583,7 +1583,7 @@ export class ReferenceResolver {
|
||||
// DatabaseConnection open (schema.sql re-applies IF NOT EXISTS).
|
||||
if (bulkEdgesActive) {
|
||||
const tIdx = Date.now();
|
||||
parallel!.bulkEdgeLoad!.end();
|
||||
await parallel!.bulkEdgeLoad!.end();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] edge-index-recreate: ${Date.now() - tIdx}ms`);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user