Large-codebase indexing died at the end of "Resolving refs" two ways: watchdog kills of healthy work (24k-file Java on Windows, #1212 — third iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale, where v1.3.0 could not complete at any watchdog setting). Root causes: ~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no yield points, several materialized whole-graph snapshots (kotlin expect/actual opened with getAllNodes() — 2M nodes in one array; the C fn-pointer pass retained every C file's contents twice plus every function node), and the post-index WAL checkpoint ran minutes of synchronous IO on the main thread, killing even a successful index at the finish line. The pipeline tail now follows the same discipline as the rest: never hold O(graph) in the heap, yield everywhere. - All synthesis passes stream node-kind scans (cursors, not arrays) and yield on time-budgeted checkpoints; language gates skip passes whose filters a project's file languages provably can't satisfy. - kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded, units stream one file at a time, and the all-functions array + write-only id map are gone; spring reads each .java once, not twice. - runMaintenance moved to a worker thread (own SQLite connection); per-file store commits chunk with yields behind a serialized flush chain (preserving #1015 file-order determinism); resolver warm-up streams the DISTINCT name set; resolution batch-tail and merged-edge inserts run in bounded sub-transactions. - Daemon: fixed a socket-handoff race that could leave a fresh MCP session permanently silent (client-hello tail unshifted into a flowing stream with zero listeners — the long-standing #662 test flake was this real bug); first tool call no longer queues behind the query pool's cold start (pool.ready gate). Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully indexes in 27m8s on a 2-core/6GB container at default heap + default watchdog; llvm-project (180k files) completes under 1GB RSS including kill-and-sync recovery; synthesized-edge and full-graph parity are byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky daemon test passed 25/25 under load. Env-gated diagnostics kept: CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop tracing. Design record: docs/design/main-thread-stall-followup.md. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
58b6bf5c60
commit
a3f90089e8
@@ -25,6 +25,7 @@
|
||||
|
||||
import type { Edge, Node } from '../types';
|
||||
import type { ResolutionContext } from './types';
|
||||
import type { MaybeYield } from './cooperative-yield';
|
||||
import { GOFRAME_ROUTE_MARKER } from './frameworks/goframe';
|
||||
|
||||
const FANOUT_CAP = 2000; // backstop only; real apps are 1 route → 1 method.
|
||||
@@ -73,13 +74,15 @@ function selectHandler(candidates: Node[], routeFile: string): Node | null {
|
||||
return sameModule.length === 1 ? sameModule[0]! : null;
|
||||
}
|
||||
|
||||
export function goframeRouteEdges(ctx: ResolutionContext): Edge[] {
|
||||
export async function goframeRouteEdges(ctx: ResolutionContext, onYield: MaybeYield): Promise<Edge[]> {
|
||||
let scanned255 = 0;
|
||||
// Route nodes the goframe extractor created, keyed by their package-qualified
|
||||
// request type (`cash.ListReq`). `wanted` holds every key a handler signature
|
||||
// could match — the qualified form plus its bare type fallback.
|
||||
const routesByReqType = new Map<string, Node[]>();
|
||||
const wanted = new Set<string>();
|
||||
for (const route of ctx.getNodesByKind('route')) {
|
||||
for (const route of (ctx.iterateNodesByKind?.('route') ?? ctx.getNodesByKind('route'))) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (route.language !== 'go') continue;
|
||||
const marker = route.qualifiedName.indexOf(GOFRAME_ROUTE_MARKER);
|
||||
if (marker < 0) continue;
|
||||
@@ -98,7 +101,8 @@ export function goframeRouteEdges(ctx: ResolutionContext): Edge[] {
|
||||
// pointer, indexed by every matching (qualified + bare) form so a route can
|
||||
// match precisely on `pkg.Type` and fall back to the bare `Type`.
|
||||
const handlersByKey = new Map<string, Node[]>();
|
||||
for (const method of ctx.getNodesByKind('method')) {
|
||||
for (const method of (ctx.iterateNodesByKind?.('method') ?? ctx.getNodesByKind('method'))) {
|
||||
if ((++scanned255 & 63) === 0) await onYield();
|
||||
if (method.language !== 'go' || !method.signature) continue;
|
||||
for (const t of pointerParamTypes(method.signature)) {
|
||||
if (!wanted.has(t)) continue;
|
||||
|
||||
Reference in New Issue
Block a user