fix(watchdog): don't kill a healthy index on degraded storage — require heartbeat silence AND no disk progress (#1231) (#1244)

The liveness watchdog judged the main thread by heartbeat silence alone,
which cannot distinguish a true wedge (the #850 infinite loop it exists to
kill) from one long synchronous SQLite statement on severely degraded
storage — so it SIGKILLed valid, in-progress indexes (observed on a
150-IOPS throttled rig, and latent on real HDDs at scale).

The CLI index/init paths now hand the watchdog the project's DB + WAL
paths. On a silent timeout the watchdog child stats them first: if they
advanced during the silence, the block is a slow store making forward
progress — defer and keep watching; if not, kill at the base timeout
exactly as before. Deferral is bounded by a hard cap (10× the timeout) of
continuous silence so a wedge coinciding with unrelated file activity, or
I/O hung beyond any legitimate statement, still dies. The daemon path is
unchanged (no progress paths — pure heartbeat).

Validated with real spawned processes (defer-on-progress, kill-on-static,
hard-cap kill) and on the throttled rig: a 150-IOPS index under a 10s
watchdog window — 6× tighter than production, with store stalls measured
at 10-20s — completes cleanly where the old watchdog killed it, while
true-wedge kill latency is unchanged.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

src/bin/codegraph.ts

@@ -590,7 +590,7 @@ program
         return;
       }
 
-      const { default: CodeGraph } = await loadCodeGraph();
+      const { default: CodeGraph, getDatabasePath } = await loadCodeGraph();
       const cg = await CodeGraph.init(projectPath, { index: false });
       clack.log.success(`Initialized in ${projectPath}`);
 
@@ -598,10 +598,13 @@ program
       // accepted (so existing muscle memory and scripts don't break) but is a
       // no-op — initializing always builds the initial index.
       // Supervise the index: self-terminate if orphaned or wedged (#999).
+      // The DB + WAL paths let the liveness watchdog tell a slow store on
+      // degraded storage from a true wedge (#1231).
       // A closure so we can re-run the exact same supervised, progress-rendered
       // index if the user opts gitignored child repos in below (#1156).
+      const dbPath = getDatabasePath(projectPath);
       const runIndex = async (): Promise<IndexResult> => {
-        const supervision = installCommandSupervision('init');
+        const supervision = installCommandSupervision('init', { progressPaths: [dbPath, `${dbPath}-wal`] });
         try {
           if (options.verbose) {
             return await cg.indexAll({ onProgress: createVerboseProgress(), verbose: true });
@@ -727,7 +730,7 @@ program
         process.exit(1);
       }
 
-      const { default: CodeGraph } = await loadCodeGraph();
+      const { default: CodeGraph, getDatabasePath } = await loadCodeGraph();
       // `index` is a FULL re-index — identical to a fresh `init`. RECREATE the
       // database from scratch (discard .codegraph/codegraph.db + its WAL) rather
       // than opening the old graph and DELETE-ing every row. The clear-then-index
@@ -741,7 +744,10 @@ program
 
       // Supervise the indexer: self-terminate if orphaned (parent shim killed)
       // or if the main thread wedges — neither was guarded on this path (#999).
-      const supervision = installCommandSupervision('index');
+      // The DB + WAL paths let the liveness watchdog tell a slow store on
+      // degraded storage from a true wedge (#1231).
+      const dbPath = getDatabasePath(projectPath);
+      const supervision = installCommandSupervision('index', { progressPaths: [dbPath, `${dbPath}-wal`] });
       try {
         if (options.quiet) {
           // Quiet mode: no UI, just run against the freshly-recreated graph.

src/bin/command-supervision.ts

@@ -28,7 +28,7 @@
  * work — is exactly what cooperative yielding buys: a genuinely stuck span never
  * reaches its next yield, so it still trips the timeout.
  */
-import { installMainThreadWatchdog } from '../mcp/liveness-watchdog';
+import { installMainThreadWatchdog, WatchdogOptions } from '../mcp/liveness-watchdog';
 import { supervisionLostReason, parsePpidPollMs, parseHostPpid } from '../mcp/ppid-watchdog';
 import { isProcessAlive } from '../mcp/daemon-registry';
 import { EARLY_PPID } from '../mcp/early-ppid';
@@ -44,12 +44,18 @@ export interface CommandSupervision {
  * `label` is used in the shutdown notice (e.g. `"index"`). Returns a handle
  * whose `stop()` must be called when the command completes so neither watchdog
  * outlives it.
+ *
+ * Pass `watchdog.progressPaths` (the project's SQLite DB + `-wal`) so the
+ * liveness watchdog can tell a slow-but-progressing store on degraded storage
+ * (files advancing) from a true wedge (they aren't) — one long synchronous
+ * SQLite statement on a 150-IOPS disk otherwise gets a healthy index
+ * SIGKILLed (#1231).
  */
-export function installCommandSupervision(label: string): CommandSupervision {
+export function installCommandSupervision(label: string, watchdog: WatchdogOptions = {}): CommandSupervision {
   // Liveness watchdog: a separate process that SIGKILLs us if our event loop
   // stops turning for too long (a wedged synchronous loop). Self-disables on
   // CODEGRAPH_NO_WATCHDOG.
-  const liveness = installMainThreadWatchdog();
+  const liveness = installMainThreadWatchdog(watchdog);
 
   // PPID watchdog: detect that the parent (or the host threaded past the
   // relaunch shim) died and we've been orphaned, then exit instead of leaking.