Large-codebase indexing died at the end of "Resolving refs" two ways: watchdog kills of healthy work (24k-file Java on Windows, #1212 — third iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale, where v1.3.0 could not complete at any watchdog setting). Root causes: ~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no yield points, several materialized whole-graph snapshots (kotlin expect/actual opened with getAllNodes() — 2M nodes in one array; the C fn-pointer pass retained every C file's contents twice plus every function node), and the post-index WAL checkpoint ran minutes of synchronous IO on the main thread, killing even a successful index at the finish line. The pipeline tail now follows the same discipline as the rest: never hold O(graph) in the heap, yield everywhere. - All synthesis passes stream node-kind scans (cursors, not arrays) and yield on time-budgeted checkpoints; language gates skip passes whose filters a project's file languages provably can't satisfy. - kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded, units stream one file at a time, and the all-functions array + write-only id map are gone; spring reads each .java once, not twice. - runMaintenance moved to a worker thread (own SQLite connection); per-file store commits chunk with yields behind a serialized flush chain (preserving #1015 file-order determinism); resolver warm-up streams the DISTINCT name set; resolution batch-tail and merged-edge inserts run in bounded sub-transactions. - Daemon: fixed a socket-handoff race that could leave a fresh MCP session permanently silent (client-hello tail unshifted into a flowing stream with zero listeners — the long-standing #662 test flake was this real bug); first tool call no longer queues behind the query pool's cold start (pool.ready gate). Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully indexes in 27m8s on a 2-core/6GB container at default heap + default watchdog; llvm-project (180k files) completes under 1GB RSS including kill-and-sync recovery; synthesized-edge and full-graph parity are byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky daemon test passed 25/25 under load. Env-gated diagnostics kept: CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop tracing. Design record: docs/design/main-thread-stall-followup.md. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
58b6bf5c60
commit
a3f90089e8
+16
-3
@@ -485,23 +485,33 @@ export class CodeGraph {
|
||||
// receiver conforms to (protocol-extension / inherited / default-
|
||||
// interface). Needs the implements/extends edges the main pass just
|
||||
// built, so it runs after resolution (#750).
|
||||
const tChained = Date.now();
|
||||
await this.resolver.resolveChainedCallsViaConformance();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[synth-timing] chainedConformance: ${Date.now() - tChained}ms`);
|
||||
// Same lifecycle for `this.<member>` callback registrations whose
|
||||
// member is inherited from a supertype (#808).
|
||||
const tDeferred = Date.now();
|
||||
await this.resolver.resolveDeferredThisMemberRefs();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[synth-timing] deferredThisMember: ${Date.now() - tDeferred}ms`);
|
||||
}
|
||||
|
||||
// Refresh planner stats + checkpoint the WAL after bulk writes.
|
||||
// Cheap and non-blocking; never load-bearing for correctness.
|
||||
// Off-thread (worker connection): on a multi-GB index this is minutes
|
||||
// of IO, and inline it starved the #850 watchdog AFTER a fully
|
||||
// successful index. Never load-bearing for correctness.
|
||||
if (result.success && result.filesIndexed > 0) {
|
||||
this.db.runMaintenance();
|
||||
const tMaint = Date.now();
|
||||
await this.db.runMaintenance();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] maintenance: ${Date.now() - tMaint}ms`);
|
||||
}
|
||||
|
||||
// The orchestrator only sees extraction-phase counts; resolution and
|
||||
// synthesizer edges (often >50% of the graph on JVM repos) come later.
|
||||
// Recompute against the DB so the CLI summary reports the true totals.
|
||||
if (result.success && result.filesIndexed > 0) {
|
||||
const tCount = Date.now();
|
||||
const after = this.queries.getNodeAndEdgeCount();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] count-recompute: ${Date.now() - tCount}ms`);
|
||||
result.nodesCreated = after.nodes - before.nodes;
|
||||
result.edgesCreated = after.edges - before.edges;
|
||||
}
|
||||
@@ -612,7 +622,9 @@ export class CodeGraph {
|
||||
if (filesChanged) {
|
||||
if (result.changedFilePaths) {
|
||||
// Scope resolution to changed files (git fast path — bounded set)
|
||||
const tRefLoad = Date.now();
|
||||
const unresolvedRefs = this.queries.getUnresolvedReferencesByFiles(result.changedFilePaths);
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] sync-ref-load: ${Date.now() - tRefLoad}ms (${unresolvedRefs.length} refs)`);
|
||||
|
||||
options.onProgress?.({
|
||||
phase: 'resolving',
|
||||
@@ -687,8 +699,9 @@ export class CodeGraph {
|
||||
}
|
||||
|
||||
// Refresh planner stats + checkpoint the WAL after bulk writes.
|
||||
// Off-thread — see indexAll's call site.
|
||||
if (filesChanged || result.filesRemoved > 0 || orphanCount > 0) {
|
||||
this.db.runMaintenance();
|
||||
await this.db.runMaintenance();
|
||||
}
|
||||
|
||||
// Heal the segment vocabulary on indexes built before the table
|
||||
|
||||
Reference in New Issue
Block a user