fix(indexing): bounded-memory yielding pipeline tail + daemon session fixes (#1212) (#1226)

Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.

The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.

- All synthesis passes stream node-kind scans (cursors, not arrays) and
  yield on time-budgeted checkpoints; language gates skip passes whose
  filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
  units stream one file at a time, and the all-functions array +
  write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
  per-file store commits chunk with yields behind a serialized flush
  chain (preserving #1015 file-order determinism); resolver warm-up
  streams the DISTINCT name set; resolution batch-tail and merged-edge
  inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
  session permanently silent (client-hello tail unshifted into a
  flowing stream with zero listeners — the long-standing #662 test
  flake was this real bug); first tool call no longer queues behind
  the query pool's cold start (pool.ready gate).

Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-08 23:18:23 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 58b6bf5c60
commit a3f90089e8
21 changed files with 1036 additions and 264 deletions
+6
View File
@@ -9,6 +9,12 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Fixes
- Indexing very large codebases no longer dies at the end of the "Resolving refs" step. Two failure modes are fixed: on multi-million-symbol projects (e.g. the Linux kernel, ~95,000 files) the final analysis phase ran out of memory and crashed the process outright, and on large projects on slower machines (reported on a 24,000-file Java project on Windows) the same phase could stall long enough that the safety watchdog killed a healthy, still-progressing index at ~98% (#1212). The whole phase now streams its work instead of holding whole-graph snapshots in memory, keeps the process responsive throughout, and skips analysis passes for languages a project doesn't contain — which also makes the tail of indexing noticeably faster on single-language repos. The resulting graph is identical, and a genuinely wedged process is still detected and killed.
- Indexing and `codegraph sync` stay responsive through their heaviest internal steps on huge projects: the post-index database maintenance (which on a multi-gigabyte index could stall the process for minutes and get a fully successful index killed by the safety watchdog at the finish line) now runs on a background thread, storing a giant generated file no longer freezes the process mid-extraction, and the reference-resolution bookkeeping between progress updates is broken into small responsive steps. The resulting graph is byte-for-byte identical.
- Fixed a race that could leave a freshly-attached MCP session permanently silent: when a client's first messages arrived glued together during the daemon's connection handshake (roughly one attach in five on a busy machine), the daemon could drop them and stop reading that connection entirely — every tool call from that session then hung with no reply. The handshake now hands the connection over losslessly, and the fix is validated by hammering the previously-flaky attach test 25× under load.
- The first tool call after the shared daemon starts no longer waits behind the query workers' cold start (which can take many seconds on a busy machine) — it's served directly until the first worker is warm, so a fresh session answers immediately.
## [1.3.0] - 2026-07-07