fix(scale): kernel-scale hardening — OOM-safe pass skipping + watchdog-safe index recreate (#1323)
Two hazards found by running today's full stack against the Linux kernel (70,129 files) in the cg1212 repro container: 1. The parallel-synthesis fallback retried a worker-failed pass on the MAIN thread. At multi-million-node scale a worker failure is usually a memory ceiling, so the retry would OOM the process and take the whole index with it. Above 1.5M nodes a failed pass is now skipped with a clear stderr message (its synthesized edges are absent; the index completes). Below that, the main-thread retry stays — small-scale worker crashes are transient and the retry keeps coverage. 2. endBulkEdgeLoad rebuilt all four edge indexes in one synchronous span — measured 79s at kernel scale, past the #850 liveness watchdog's 60s stall window. A daemon-triggered re-index would have been SIGKILLed right after doing the work. Now async with an event-loop yield between builds, keeping each stall to a single index (~20s at kernel scale). Validation: full Linux kernel index to completion in the repro container — 2,048,674 nodes / 6,405,964 edges, EXIT 0, zero passes skipped, on a 2-CPU VM (worst case: pool disabled, sequential resolution + synthesis) in ~27min. Phase walls: parse 6.0m, resolution 19.5m (incl. synthesis 6.3m, recreate 79s), maintenance 74s. Suite green (2444). Also adds docs/design/native-extraction-kernel.md — the spike-validated design for the native extraction kernel (Rust parse+walk over dubbo's Java: 202ms rayon / 1.07s single-thread vs 4.7s for the current wasm pipeline). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
567b4ad4be
commit
4efc6c70e2
+10
-1
@@ -185,14 +185,23 @@ export class DatabaseConnection {
|
||||
* Leave bulk-edge-load mode: recreate the dropped indexes in one pass each
|
||||
* over the (now fully loaded) edges table — far cheaper than maintaining
|
||||
* them per-insert. DDL is extracted from schema.sql so it cannot drift.
|
||||
*
|
||||
* Async with a yield BETWEEN the four CREATE INDEX statements: each build is
|
||||
* a synchronous scan of the whole edges table (~20s apiece at Linux-kernel
|
||||
* scale, 79s total measured), and running them back-to-back is a single
|
||||
* event-loop stall longer than the #850 liveness watchdog's 60s window — a
|
||||
* daemon-triggered re-index would be SIGKILLed right after doing the work.
|
||||
* One yield per statement keeps every stall to a single index build, which
|
||||
* stays inside the window.
|
||||
*/
|
||||
endBulkEdgeLoad(): void {
|
||||
async endBulkEdgeLoad(): Promise<void> {
|
||||
const schemaPath = path.join(__dirname, 'schema.sql');
|
||||
const schema = fs.readFileSync(schemaPath, 'utf-8');
|
||||
for (const idx of DatabaseConnection.BULK_EDGE_INDEX_NAMES) {
|
||||
const m = schema.match(new RegExp(`CREATE INDEX IF NOT EXISTS ${idx}\\b[^;]*;`));
|
||||
if (!m) throw new Error(`schema.sql: edge index ${idx} not found for bulk-load recreation`);
|
||||
this.db.exec(m[0]);
|
||||
await new Promise((resolve) => setImmediate(resolve));
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user