fix(watchdog): don't kill a healthy index on degraded storage — require heartbeat silence AND no disk progress (#1231) (#1244)

The liveness watchdog judged the main thread by heartbeat silence alone,
which cannot distinguish a true wedge (the #850 infinite loop it exists to
kill) from one long synchronous SQLite statement on severely degraded
storage — so it SIGKILLed valid, in-progress indexes (observed on a
150-IOPS throttled rig, and latent on real HDDs at scale).

The CLI index/init paths now hand the watchdog the project's DB + WAL
paths. On a silent timeout the watchdog child stats them first: if they
advanced during the silence, the block is a slow store making forward
progress — defer and keep watching; if not, kill at the base timeout
exactly as before. Deferral is bounded by a hard cap (10× the timeout) of
continuous silence so a wedge coinciding with unrelated file activity, or
I/O hung beyond any legitimate statement, still dies. The daemon path is
unchanged (no progress paths — pure heartbeat).

Validated with real spawned processes (defer-on-progress, kill-on-static,
hard-cap kill) and on the throttled rig: a 150-IOPS index under a 10s
watchdog window — 6× tighter than production, with store stalls measured
at 10-20s — completes cleanly where the old watchdog killed it, while
true-wedge kill latency is unchanged.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

__tests__/liveness-watchdog.test.ts

@@ -57,11 +57,12 @@ describe('liveness watchdog (spawned, real watchdog process)', () => {
   function runChild(
     env: Record<string, string>,
     body: string,
-    hardTimeoutMs: number
+    hardTimeoutMs: number,
+    progressPaths?: string[]
   ): Promise<{ code: number | null; signal: NodeJS.Signals | 'TIMEOUT' | null }> {
     const src = `
       const { installMainThreadWatchdog } = require(${JSON.stringify(MODULE)});
-      installMainThreadWatchdog();
+      installMainThreadWatchdog(${progressPaths ? JSON.stringify({ progressPaths }) : ''});
       ${body}
     `;
     const child = spawn(process.execPath, ['-e', src], {
@@ -119,6 +120,69 @@ describe('liveness watchdog (spawned, real watchdog process)', () => {
     expect(code).toBe(7); // exited on its own terms
   }, 12000);
 
+  // --- disk-progress deferral (#1231): a blocked event loop is NOT a wedge
+  // when the watched DB files keep advancing (a slow synchronous SQLite
+  // statement on degraded storage). ---
+
+  /** Grow `file` every 150ms for `forMs`; resolves when done. */
+  function growFile(file: string, forMs: number): Promise<void> {
+    return new Promise((resolve) => {
+      const iv = setInterval(() => { fs.appendFileSync(file, 'x'.repeat(64)); }, 150);
+      setTimeout(() => { clearInterval(iv); resolve(); }, forMs);
+    });
+  }
+
+  it('does NOT kill a blocked loop while the watched files advance (slow store, not a wedge)', async () => {
+    const tmp = path.join(fs.mkdtempSync(path.join(require('os').tmpdir(), 'cg-wd-')), 'db-wal');
+    fs.writeFileSync(tmp, 'seed');
+    // Base timeout 500ms; the loop blocks for 2.5s (5 timeouts, under the 10×
+    // cap) while the test process grows the watched file. Old behavior: killed
+    // at ~500ms. New: deferred, exits on its own with code 5.
+    const [r] = await Promise.all([
+      runChild(
+        { CODEGRAPH_WATCHDOG_TIMEOUT_MS: '500' },
+        'setTimeout(() => { const end = Date.now() + 2500; while (Date.now() < end) {} process.exit(5); }, 200);',
+        10_000,
+        [tmp]
+      ),
+      growFile(tmp, 3200),
+    ]);
+    expect(r.signal).toBeNull();
+    expect(r.code).toBe(5);
+  }, 15000);
+
+  it('still kills a blocked loop when the watched files do NOT advance (a true wedge)', async () => {
+    const tmp = path.join(fs.mkdtempSync(path.join(require('os').tmpdir(), 'cg-wd-')), 'db-wal');
+    fs.writeFileSync(tmp, 'seed');
+    // Same blocked loop, nobody grows the file: the base timeout kills it long
+    // before its own exit(5) at 2.5s.
+    const r = await runChild(
+      { CODEGRAPH_WATCHDOG_TIMEOUT_MS: '500' },
+      'setTimeout(() => { const end = Date.now() + 2500; while (Date.now() < end) {} process.exit(5); }, 200);',
+      10_000,
+      [tmp]
+    );
+    expectKilled(r);
+  }, 15000);
+
+  it('kills at the hard cap even with ongoing file activity (bounded deferral)', async () => {
+    const tmp = path.join(fs.mkdtempSync(path.join(require('os').tmpdir(), 'cg-wd-')), 'db-wal');
+    fs.writeFileSync(tmp, 'seed');
+    // Base timeout 300ms ⇒ cap 3s. The loop blocks for 8s with continuous file
+    // growth: deferral carries it past 300ms but the cap kills it around ~3s,
+    // well before its own exit(5).
+    const [r] = await Promise.all([
+      runChild(
+        { CODEGRAPH_WATCHDOG_TIMEOUT_MS: '300' },
+        'setTimeout(() => { const end = Date.now() + 8000; while (Date.now() < end) {} process.exit(5); }, 200);',
+        15_000,
+        [tmp]
+      ),
+      growFile(tmp, 9000),
+    ]);
+    expectKilled(r);
+  }, 20000);
+
   it('does NOT kill a wedged process when CODEGRAPH_NO_WATCHDOG=1', async () => {
     const { code, signal } = await runChild(
       { CODEGRAPH_WATCHDOG_TIMEOUT_MS: '500', CODEGRAPH_NO_WATCHDOG: '1' },