Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.
The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.
- All synthesis passes stream node-kind scans (cursors, not arrays) and
yield on time-budgeted checkpoints; language gates skip passes whose
filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
units stream one file at a time, and the all-functions array +
write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
per-file store commits chunk with yields behind a serialized flush
chain (preserving #1015 file-order determinism); resolver warm-up
streams the DISTINCT name set; resolution batch-tail and merged-edge
inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
session permanently silent (client-hello tail unshifted into a
flowing stream with zero listeners — the long-standing #662 test
flake was this real bug); first tool call no longer queues behind
the query pool's cold start (pool.ready gate).
Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The shared daemon served every session on one event loop with synchronous
node:sqlite. codegraph_explore is CPU-bound work stitched together by microtask
awaits, so N concurrent explores keep the microtask queue continuously full and
starve the macrotask phases — timers AND socket I/O. The transport freezes: no
response can flush until the whole batch drains, so with ~10 subagents on a large
repo clients routinely time out (reported via X by @symbolic2020).
Move the heavy read-tool dispatch onto a worker-thread pool. Each worker holds
its own WAL read connection (verified: a worker reader sees the main writer's
committed catch-up/watcher writes); the single watcher/writer, the catch-up gate,
codegraph_status, and the staleness/worktree notices stay on the main thread.
Concurrent reads now run in true parallel up to core count and the main loop
stays free for the MCP transport, so responses flush incrementally instead of
all-at-once after the batch drains. Enabled for the shared daemon only; direct
(single-stdio-client) mode is unchanged.
- crash recovery: respawn + retry-once, with a circuit breaker that falls back
to in-process dispatch if workers can't run on this platform
- graceful backstop: an overloaded pool returns success-shaped "busy, retry"
guidance, never isError (so it can't teach the agent to abandon codegraph)
- pending-aware growth + capped concurrent cold-starts avoid a startup
thundering herd (N simultaneous module-loads + DB opens could stall the loop)
- config: CODEGRAPH_QUERY_POOL_SIZE (default clamp(cores-1, 1, 16); 0 disables
→ in-process), CODEGRAPH_QUERY_BUSY_TIMEOUT_MS (default 45s)
10 concurrent explores on vscode (10.5k files): 31s → ~9s, staggered flush,
0 timeouts, byte-identical output; scales with cores (≈3.3× on 8, 1.8× on 2).
Full suite passes plus 10 new query-pool tests (fake-worker injection so the
scheduling logic is covered without spawning threads).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
