Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.
The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.
- All synthesis passes stream node-kind scans (cursors, not arrays) and
yield on time-budgeted checkpoints; language gates skip passes whose
filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
units stream one file at a time, and the all-functions array +
write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
per-file store commits chunk with yields behind a serialized flush
chain (preserving #1015 file-order determinism); resolver warm-up
streams the DISTINCT name set; resolution batch-tail and merged-edge
inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
session permanently silent (client-hello tail unshifted into a
flowing stream with zero listeners — the long-standing #662 test
flake was this real bug); first tool call no longer queues behind
the query pool's cold start (pool.ready gate).
Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The shared daemon's inactivity backstop (#692) reaped the daemon after
maxIdleMs (default 30 min) of no inbound query bytes whenever a client was
still connected — without ever checking whether that client was actually
alive. lastActivityAt is fed only by inbound socket data and MCP has no
keepalive, so a genuinely-live session that just hadn't queried CodeGraph in
30 min tripped it. The daemon then exited, and the proxy's onDaemonLost
degrades that session (and every other session sharing the daemon) to an
in-process engine for the rest of its life. On one dev machine over a day the
backstop fired 20 times on live sessions (clients=1) and the liveness sweep
caught 0 real dead peers — net harm.
The backstop exists only to catch a phantom client (one counted but gone,
whose socket-close was never delivered). It now consults the peer pids the
daemon already tracks: after the inactivity window it sweeps provably-dead
peers, then reaps the daemon only if NO remaining client can be proven alive
(every one is an unknown-pid connection the sweep can't verify — the sole
phantom class it can't catch). One provably-alive client keeps the daemon up.
Extracted the decision into Daemon.backstopShouldExit(isAlive) so it's unit-
testable with an injected liveness probe, mirroring reapDeadClients. All #692
guarantees preserved; the only behavior change is that a provably-alive quiet
session is no longer reaped.
- daemon-client-liveness.test.ts: 7 new deterministic cases for
backstopShouldExit (live kept, phantom reaped, mixed protects the live one,
dead-peer swept-then-held, within-window, zero-client).
- mcp-daemon.test.ts: the integration test that asserted the backstop reaps a
live connected client (it encoded the bug) now asserts the opposite — a
live-but-quiet session survives several backstop windows with its lockfile
intact and no backstop shutdown logged.
Validated end-to-end on the built bundle: a quiet session's daemon stayed up
across 4 backstop windows (maxIdle=3s), same pid throughout, zero backstop
fires. Found while fixing #1185.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On Windows, calling process.exit() while a recursive fs.watch handle is still
tearing down aborts the daemon with a libuv UV_HANDLE_CLOSING assertion
(0xC0000409) — reproducible whenever the indexed tree contains a nested repo
(submodule / embedded clone), since that's what keeps a watch active at shutdown.
A small exit delay doesn't help; only letting the loop drain is clean (verified
on a real Windows VM: close()+exit() and close()+setTimeout(exit) both abort,
while letting the loop drain exits 0).
finalizeDaemonExit() now exits immediately on POSIX (unchanged) but on Windows
marks success (exitCode=0) and lets the loop drain to a natural exit, with an
unref'd backstop that force-exits only if a stray handle would otherwise hang
shutdown. The daemon's own timers are already unref'd and its PPID watchdog lives
in the proxy, so nothing keeps the loop alive past the closing watch handles —
natural drain is fast. Pure + platform-injected so both branches unit-test off-Windows.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A project kept on an ExFAT/FAT external volume (or some network mounts /
WSL2 DrvFs) broke the background auto-sync daemon at two points, both
because the filesystem lacks POSIX features the daemon relied on:
1. Lock acquisition hard-links a temp file onto .codegraph/daemon.pid for
race-free exclusivity (#411) — these filesystems have no hard links.
2. The Unix-domain socket listen() fails regardless of path length, so the
old length-only tmpdir fallback never triggered.
Both surface as a capability error, but each OS reports a DIFFERENT errno
for the same gap (macOS ENOTSUP, Linux EPERM, Windows EISDIR), so the fix
is policy-based rather than an enumerated code-set:
- Lock: fall back to an O_EXCL create on any non-EEXIST link error. The
temp write already proved the directory is writable, so the fallback
either succeeds (still atomic + exclusive, "first writer wins") or
surfaces its own genuine error.
- Socket: an ordered candidate list [in-project, tmpdir] walked by BOTH
the daemon (binds) and the proxy (connects) — they converge on the
fallback with zero coordination. Relocate past any non-EADDRINUSE bind
error; EADDRINUSE still rethrows, preserving the #974 contract.
Normal repos are unaffected: the in-project candidate binds first, and the
hard-link lock path is unchanged.
Validated end-to-end on real removable-drive filesystems: macOS ExFAT
(hdiutil image), Linux FAT32 (Docker loop mount), Windows exFAT (diskpart
VHD) — each acquires the lock, relocates (or binds a named pipe on
Windows), and serves a real client.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared daemon served every session on one event loop with synchronous
node:sqlite. codegraph_explore is CPU-bound work stitched together by microtask
awaits, so N concurrent explores keep the microtask queue continuously full and
starve the macrotask phases — timers AND socket I/O. The transport freezes: no
response can flush until the whole batch drains, so with ~10 subagents on a large
repo clients routinely time out (reported via X by @symbolic2020).
Move the heavy read-tool dispatch onto a worker-thread pool. Each worker holds
its own WAL read connection (verified: a worker reader sees the main writer's
committed catch-up/watcher writes); the single watcher/writer, the catch-up gate,
codegraph_status, and the staleness/worktree notices stay on the main thread.
Concurrent reads now run in true parallel up to core count and the main loop
stays free for the MCP transport, so responses flush incrementally instead of
all-at-once after the batch drains. Enabled for the shared daemon only; direct
(single-stdio-client) mode is unchanged.
- crash recovery: respawn + retry-once, with a circuit breaker that falls back
to in-process dispatch if workers can't run on this platform
- graceful backstop: an overloaded pool returns success-shaped "busy, retry"
guidance, never isError (so it can't teach the agent to abandon codegraph)
- pending-aware growth + capped concurrent cold-starts avoid a startup
thundering herd (N simultaneous module-loads + DB opens could stall the loop)
- config: CODEGRAPH_QUERY_POOL_SIZE (default clamp(cores-1, 1, 16); 0 disables
→ in-process), CODEGRAPH_QUERY_BUSY_TIMEOUT_MS (default 45s)
10 concurrent explores on vscode (10.5k files): 31s → ~9s, staggered flush,
0 timeouts, byte-identical output; scales with cores (≈3.3× on 8, 1.8× on 2).
Full suite passes plus 10 new query-pool tests (fake-worker injection so the
scheduling logic is covered without spawning threads).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The client-facing MCP proxy could exit with "Transport closed" when its
connection to the shared daemon hit a socket 'error' with no listener
attached — common on WSL2 /mnt (DrvFs), where AF_UNIX is flaky. The global
fatal handler turned that uncaughtException into process.exit(1), which the
MCP client saw as a bare transport close even though the index was healthy.
proxy.ts now keeps an 'error' listener on the daemon socket for its whole
life (and skips a socket destroyed in the connect window), so a stray error
degrades to the existing in-process fallback instead of crashing. daemon.ts
releases the lockfile it acquired when it fails to bind, so the next launch
doesn't spin on a stale lock (the duplicate serve --mcp pileup).
No default behavior change for anyone; WSL /mnt users who still hit trouble
can set CODEGRAPH_NO_DAEMON=1 to skip the shared daemon entirely. Validated
on macOS (unit + live serve probe) and Linux (Docker, --init): 64/64 across
the daemon/socket/lifecycle suites, incl. real AF_UNIX.
Closes#974
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds first-class daemon control (the #845 pain point: no clean way to stop a
runaway daemon). `codegraph stop [path]` stops the current/given project's
daemon (SIGTERM -> SIGKILL fallback, sweeps artifacts); `stop --all` stops every
daemon; `list`/`ps` shows running daemons (--json for scripts).
Discovery via a small self-healing registry: each daemon records its root under
~/.codegraph/daemons/ on start, removes it on graceful shutdown; readers prune
dead pids. Cross-platform by construction (files + process.kill). Validated live
on macOS, Linux (docker), and Windows (VM): registry unit 6/6 and real-daemon
stop/list 6/6 on each.
Layer-2 defense-in-depth follow-up to the Windows PPID watchdog fix (#711).
That fix makes an orphaned proxy exit so its socket closes and the daemon
reaps via the refcount + idle timer. This adds two daemon-side safety nets for
the residual case where a socket close is never delivered (a Windows named-pipe
hazard) and a phantom client would otherwise pin the daemon forever:
- Liveness sweep: a proxy now sends an optional client-hello carrying its pid
(+ host pid) right after verifying the daemon hello; the daemon periodically
drops any client whose peer process is dead, re-arming the idle timer.
Fail-safe and version-pinned — a connection that never sends the hello just
falls back to the socket-close lifecycle, and the daemon reads it before the
transport so a non-hello first line is handed through untouched.
- Inactivity backstop: the daemon exits after a generous no-traffic window
(CODEGRAPH_DAEMON_MAX_IDLE_MS, default 30 min) even with clients attached, so
a phantom client that sends nothing can't keep it alive.
Pure helpers (parseClientHelloLine, peerIsDead) are unit-tested; the full
handshake + sweep and the backstop are covered end-to-end in mcp-daemon.test.ts.
Validated on a real Windows 11 VM: the sweep reaps a dead-pid client over a
named pipe and the backstop fires with a client still connected.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
chokidar v4 holds one OS file descriptor per watched file on macOS (libuv's
kqueue backend registers an fd per vnode; fsevents is installed but v4 no
longer uses it). On a large project the `serve --mcp` daemon accumulated tens
of thousands of open REG descriptors and exhausted kern.maxfiles — crashing
unrelated processes system-wide with ENFILE. #276 only trimmed the count by
ignoring directories; the source tree still cost one fd per file.
Replace chokidar with a pure-JS native fs.watch hybrid, keeping codegraph's
zero-native-addon "any OS builds any bundle" invariant:
- macOS / Windows: a single recursive fs.watch (one FSEvents stream /
ReadDirectoryChangesW handle) -> O(1) descriptors regardless of repo size.
- Linux: one inotify watch per directory (O(dirs), dynamic add for new
dirs, capped via CODEGRAPH_MAX_DIR_WATCHES) instead of per-file watches.
Validated empirically: macOS 0 extra fds at 6k and 12k files; Linux 31 inotify
watches at 6k files (per-file would be 6k); Windows recursive catches nested
and new-directory edits. Full test suite green.
Tests drive the watcher through an inertForTests seam (no OS watcher) for
determinism under parallel vitest, with one real-fs end-to-end test exercising
the genuine native path.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The file watcher registered a recursive watch over the entire project (node_modules, build output, caches included) and filtered only in the callback — exhausting the Linux inotify budget on large repos (#276). It now uses chokidar and excludes the same directories the indexer ignores (built-in default-ignore set + the project .gitignore) BEFORE registering a watch, so the watch count on a 900-dir node_modules drops from ~1200 to ~14 even with no .gitignore. Stacks with the shared daemon (#411): one watcher across agents, now small.
Also hardens the #411 daemon lockfile against a concurrent-startup race the new watcher timing made reproducible — the lock is now created atomically with its content (temp-write + hard-link), so racing daemons can never both win. Validated on macOS, Linux (Docker), and Windows (chokidar + fs.linkSync on NTFS).
Co-Authored-By: Colby McHenry <me@colbymchenry.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
One shared, detached daemon per project root: every `codegraph serve --mcp` is a thin stdio<->socket proxy (Unix socket / Windows named pipe) to it, so N agents in one repo share a single file watcher, SQLite connection, and tree-sitter warm-up instead of N copies. The daemon outlives any single session and reaps via client-refcount + idle timeout; `CODEGRAPH_NO_DAEMON=1` opts out.
Hardened during review: detached-process lifecycle (preserves the #277 watchdog via the proxy; the daemon no longer orphans on host SIGKILL), atomic lockfile + pid-verified stale-clear (no double-daemon on concurrent startup), realpath root canonicalization. Validated on macOS, Linux (Docker - 3x fewer inotify watches for 3 agents), and Windows (named pipes); A/B confirms byte-identical tool output vs direct mode. Closes#411.
Co-Authored-By: Colby McHenry <me@colbymchenry.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
