Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.
The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.
- All synthesis passes stream node-kind scans (cursors, not arrays) and
yield on time-budgeted checkpoints; language gates skip passes whose
filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
units stream one file at a time, and the all-functions array +
write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
per-file store commits chunk with yields behind a serialized flush
chain (preserving #1015 file-order determinism); resolver warm-up
streams the DISTINCT name set; resolution batch-tail and merged-edge
inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
session permanently silent (client-hello tail unshifted into a
flowing stream with zero listeners — the long-standing #662 test
flake was this real bug); first tool call no longer queues behind
the query pool's cold start (pool.ready gate).
Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The shared daemon's inactivity backstop (#692) reaped the daemon after
maxIdleMs (default 30 min) of no inbound query bytes whenever a client was
still connected — without ever checking whether that client was actually
alive. lastActivityAt is fed only by inbound socket data and MCP has no
keepalive, so a genuinely-live session that just hadn't queried CodeGraph in
30 min tripped it. The daemon then exited, and the proxy's onDaemonLost
degrades that session (and every other session sharing the daemon) to an
in-process engine for the rest of its life. On one dev machine over a day the
backstop fired 20 times on live sessions (clients=1) and the liveness sweep
caught 0 real dead peers — net harm.
The backstop exists only to catch a phantom client (one counted but gone,
whose socket-close was never delivered). It now consults the peer pids the
daemon already tracks: after the inactivity window it sweeps provably-dead
peers, then reaps the daemon only if NO remaining client can be proven alive
(every one is an unknown-pid connection the sweep can't verify — the sole
phantom class it can't catch). One provably-alive client keeps the daemon up.
Extracted the decision into Daemon.backstopShouldExit(isAlive) so it's unit-
testable with an injected liveness probe, mirroring reapDeadClients. All #692
guarantees preserved; the only behavior change is that a provably-alive quiet
session is no longer reaped.
- daemon-client-liveness.test.ts: 7 new deterministic cases for
backstopShouldExit (live kept, phantom reaped, mixed protects the live one,
dead-peer swept-then-held, within-window, zero-client).
- mcp-daemon.test.ts: the integration test that asserted the backstop reaps a
live connected client (it encoded the bug) now asserts the opposite — a
live-but-quiet session survives several backstop windows with its lockfile
intact and no backstop shutdown logged.
Validated end-to-end on the built bundle: a quiet session's daemon stayed up
across 4 backstop windows (maxIdle=3s), same pid throughout, zero backstop
fires. Found while fixing #1185.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "Attached to shared daemon" line is benign INFO, but it was written to
stderr — and MCP hosts render all server stderr at error level (and append an
`undefined` data field), so on every session start a healthy attach showed up
as `[error] … undefined`. It is now gated behind CODEGRAPH_MCP_LOG_ATTACH=1:
silent by default, opt-in for debugging daemon attach. Both attach sites
(runProxy + connectWithHello) route through one helper. The daemon integration
tests opt the harness into the log so their attach assertions still observe a
successful attach.
Re-applies the approach from #640 by @mturac.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When an MCP host (opencode and others) SIGTERM's the shared daemon as a new
session starts, the existing session's proxy used to exit on the dropped socket
— silently losing CodeGraph for that session, and hanging any request in flight
at the drop. The SIGTERM originates in the host's process-tree teardown, not in
CodeGraph (nothing here signals another process), so the fix is proxy
resilience, not chasing the signal.
The local-handshake proxy now treats a daemon disconnect as recoverable rather
than terminal: it falls back to its in-process engine for the rest of the
session (the same path used when no daemon is reachable at startup, and what
CODEGRAPH_NO_DAEMON does) and re-serves any requests that were in flight to the
dead daemon, so the host never hangs. The proxy still exits when the HOST goes
away (stdin close / PPID watchdog) — only daemon loss is now non-fatal.
Also replaces the over-the-wire liveness-sweep test added in #712 — which was
flaky under heavy parallel load (a raced raw-socket connect) — with a
deterministic Daemon.reapDeadClients unit test. The client-hello round-trip is
still exercised by every daemon test (the real proxy now sends it).
Validated with a reproduction (proxy stays alive, in-flight request answered,
post-drop request recovers) and a regression test in mcp-daemon.test.ts.
Confirmed on macOS (full suite green) and a Windows 11 VM.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Layer-2 defense-in-depth follow-up to the Windows PPID watchdog fix (#711).
That fix makes an orphaned proxy exit so its socket closes and the daemon
reaps via the refcount + idle timer. This adds two daemon-side safety nets for
the residual case where a socket close is never delivered (a Windows named-pipe
hazard) and a phantom client would otherwise pin the daemon forever:
- Liveness sweep: a proxy now sends an optional client-hello carrying its pid
(+ host pid) right after verifying the daemon hello; the daemon periodically
drops any client whose peer process is dead, re-arming the idle timer.
Fail-safe and version-pinned — a connection that never sends the hello just
falls back to the socket-close lifecycle, and the daemon reads it before the
transport so a non-hello first line is handed through untouched.
- Inactivity backstop: the daemon exits after a generous no-traffic window
(CODEGRAPH_DAEMON_MAX_IDLE_MS, default 30 min) even with clients attached, so
a phantom client that sends nothing can't keep it alive.
Pure helpers (parseClientHelloLine, peerIsDead) are unit-tested; the full
handshake + sweep and the backstop are covered end-to-end in mcp-daemon.test.ts.
Validated on a real Windows 11 VM: the sweep reaps a dead-pid client over a
named pipe and the backstop fires with a client still connected.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Trace endpoint relevance (overloaded names resolve to the real implementation instead of an empty protocol/delegate stub), Swift closure-collection synthesizer, multi-phase god-file explore rendering, and serve --mcp cold-start handshake sped ~811ms→~90ms (proxy answers initialize/tools-list locally). Full suite green (1090 pass).
One shared, detached daemon per project root: every `codegraph serve --mcp` is a thin stdio<->socket proxy (Unix socket / Windows named pipe) to it, so N agents in one repo share a single file watcher, SQLite connection, and tree-sitter warm-up instead of N copies. The daemon outlives any single session and reaps via client-refcount + idle timeout; `CODEGRAPH_NO_DAEMON=1` opts out.
Hardened during review: detached-process lifecycle (preserves the #277 watchdog via the proxy; the daemon no longer orphans on host SIGKILL), atomic lockfile + pid-verified stale-clear (no double-daemon on concurrent startup), realpath root canonicalization. Validated on macOS, Linux (Docker - 3x fewer inotify watches for 3 agents), and Windows (named pipes); A/B confirms byte-identical tool output vs direct mode. Closes#411.
Co-Authored-By: Colby McHenry <me@colbymchenry.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
