fix(indexing): bounded-memory yielding pipeline tail + daemon session fixes (#1212) (#1226)

Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.

The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.

- All synthesis passes stream node-kind scans (cursors, not arrays) and
  yield on time-budgeted checkpoints; language gates skip passes whose
  filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
  units stream one file at a time, and the all-functions array +
  write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
  per-file store commits chunk with yields behind a serialized flush
  chain (preserving #1015 file-order determinism); resolver warm-up
  streams the DISTINCT name set; resolution batch-tail and merged-edge
  inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
  session permanently silent (client-hello tail unshifted into a
  flowing stream with zero listeners — the long-standing #662 test
  flake was this real bug); first tool call no longer queues behind
  the query pool's cold start (pool.ready gate).

Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-08 23:18:23 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 58b6bf5c60
commit a3f90089e8
21 changed files with 1036 additions and 264 deletions
+7 -3
View File
@@ -25,6 +25,7 @@
import type { Edge, Node } from '../types';
import type { ResolutionContext } from './types';
import type { MaybeYield } from './cooperative-yield';
import { GOFRAME_ROUTE_MARKER } from './frameworks/goframe';
const FANOUT_CAP = 2000; // backstop only; real apps are 1 route → 1 method.
@@ -73,13 +74,15 @@ function selectHandler(candidates: Node[], routeFile: string): Node | null {
return sameModule.length === 1 ? sameModule[0]! : null;
}
export function goframeRouteEdges(ctx: ResolutionContext): Edge[] {
export async function goframeRouteEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
let scanned255 = 0;
// Route nodes the goframe extractor created, keyed by their package-qualified
// request type (`cash.ListReq`). `wanted` holds every key a handler signature
// could match — the qualified form plus its bare type fallback.
const routesByReqType = new Map<string, Node[]>();
const wanted = new Set<string>();
for (const route of ctx.getNodesByKind('route')) {
for (const route of (ctx.iterateNodesByKind?.('route') ?? ctx.getNodesByKind('route'))) {
if ((++scanned255 & 63) === 0) await onYield();
if (route.language !== 'go') continue;
const marker = route.qualifiedName.indexOf(GOFRAME_ROUTE_MARKER);
if (marker < 0) continue;
@@ -98,7 +101,8 @@ export function goframeRouteEdges(ctx: ResolutionContext): Edge[] {
// pointer, indexed by every matching (qualified + bare) form so a route can
// match precisely on `pkg.Type` and fall back to the bare `Type`.
const handlersByKey = new Map<string, Node[]>();
for (const method of ctx.getNodesByKind('method')) {
for (const method of (ctx.iterateNodesByKind?.('method') ?? ctx.getNodesByKind('method'))) {
if ((++scanned255 & 63) === 0) await onYield();
if (method.language !== 'go' || !method.signature) continue;
for (const t of pointerParamTypes(method.signature)) {
if (!wanted.has(t)) continue;