The liveness watchdog judged the main thread by heartbeat silence alone,
which cannot distinguish a true wedge (the #850 infinite loop it exists to
kill) from one long synchronous SQLite statement on severely degraded
storage — so it SIGKILLed valid, in-progress indexes (observed on a
150-IOPS throttled rig, and latent on real HDDs at scale).
The CLI index/init paths now hand the watchdog the project's DB + WAL
paths. On a silent timeout the watchdog child stats them first: if they
advanced during the silence, the block is a slow store making forward
progress — defer and keep watching; if not, kill at the base timeout
exactly as before. Deferral is bounded by a hard cap (10× the timeout) of
continuous silence so a wedge coinciding with unrelated file activity, or
I/O hung beyond any legitimate statement, still dies. The daemon path is
unchanged (no progress paths — pure heartbeat).
Validated with real spawned processes (defer-on-progress, kill-on-static,
hard-cap kill) and on the throttled rig: a 150-IOPS index under a 10s
watchdog window — 6× tighter than production, with store stalls measured
at 10-20s — completes cleanly where the old watchdog killed it, while
true-wedge kill latency is unchanged.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Validated the watchdog on the Windows VM: it kills a wedged process correctly,
but Windows has no real signals — process.kill(pid,'SIGKILL') maps to
TerminateProcess, seen as signal=null + non-zero code, not 'SIGKILL'. Assert
"killed" platform-agnostically and require the own exit code in the opt-out test.
Source watchdog unchanged. Windows: fatal-handler 8/8, liveness-watchdog 7/7,
mcp-daemon 9/9; mcp-initialize EPERM is pre-existing (identical with watchdog off).
The worker-thread watchdog from #856 didn't work in the real daemon — caught by
live-testing against a real serve --mcp. V8 isolates coordinate on global
safepoints, so a main thread wedged in a tight non-allocating loop (#850's
SourcePositionTableIterator::Advance) strands the watchdog worker before it can
SIGKILL.
A separate child process shares no isolate/heap with the parent, so the wedge
can't touch it; it kills via the kernel. Parent heartbeats to the child's stdin;
silence past the timeout -> SIGKILL; parent exit closes the pipe -> child exits.
Validated live (real daemon SIGKILLed in ~timeout); regression test covers the
non-allocating-wedge-under-heap-pressure case. API/install points/CHANGELOG
unchanged; the broken worker version was never released.
Belt-and-suspenders follow-up to #855. Any non-yielding sync loop on the main
thread wedges the event loop, and nothing running on that loop (timers, signal
handlers, PPID watchdog) can recover it — only another thread can.
A tiny worker thread (in the detached daemon + direct modes) watches a
shared-memory heartbeat the main thread bumps each event-loop turn; if it stops
advancing across enough consecutive checks (~CODEGRAPH_WATCHDOG_TIMEOUT_MS,
default 60s) the worker SIGKILLs the process so a fresh daemon starts on the next
connection. Counts consecutive stale checks (not wall-clock) so it's immune to
clock jumps / sleep; tuned never to fire on real work; opt out with
CODEGRAPH_NO_WATCHDOG=1.
