fix(watchdog): don't kill a healthy index on degraded storage — require heartbeat silence AND no disk progress (#1231) (#1244)

The liveness watchdog judged the main thread by heartbeat silence alone,
which cannot distinguish a true wedge (the #850 infinite loop it exists to
kill) from one long synchronous SQLite statement on severely degraded
storage — so it SIGKILLed valid, in-progress indexes (observed on a
150-IOPS throttled rig, and latent on real HDDs at scale).

The CLI index/init paths now hand the watchdog the project's DB + WAL
paths. On a silent timeout the watchdog child stats them first: if they
advanced during the silence, the block is a slow store making forward
progress — defer and keep watching; if not, kill at the base timeout
exactly as before. Deferral is bounded by a hard cap (10× the timeout) of
continuous silence so a wedge coinciding with unrelated file activity, or
I/O hung beyond any legitimate statement, still dies. The daemon path is
unchanged (no progress paths — pure heartbeat).

Validated with real spawned processes (defer-on-progress, kill-on-static,
hard-cap kill) and on the throttled rig: a 150-IOPS index under a 10s
watchdog window — 6× tighter than production, with store stalls measured
at 10-20s — completes cleanly where the old watchdog killed it, while
true-wedge kill latency is unchanged.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-10 08:25:23 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 116cb59625
commit edb9f2f14c
5 changed files with 172 additions and 21 deletions
+10 -4
View File
@@ -590,7 +590,7 @@ program
return;
}
const { default: CodeGraph } = await loadCodeGraph();
const { default: CodeGraph, getDatabasePath } = await loadCodeGraph();
const cg = await CodeGraph.init(projectPath, { index: false });
clack.log.success(`Initialized in ${projectPath}`);
@@ -598,10 +598,13 @@ program
// accepted (so existing muscle memory and scripts don't break) but is a
// no-op — initializing always builds the initial index.
// Supervise the index: self-terminate if orphaned or wedged (#999).
// The DB + WAL paths let the liveness watchdog tell a slow store on
// degraded storage from a true wedge (#1231).
// A closure so we can re-run the exact same supervised, progress-rendered
// index if the user opts gitignored child repos in below (#1156).
const dbPath = getDatabasePath(projectPath);
const runIndex = async (): Promise<IndexResult> => {
const supervision = installCommandSupervision('init');
const supervision = installCommandSupervision('init', { progressPaths: [dbPath, `${dbPath}-wal`] });
try {
if (options.verbose) {
return await cg.indexAll({ onProgress: createVerboseProgress(), verbose: true });
@@ -727,7 +730,7 @@ program
process.exit(1);
}
const { default: CodeGraph } = await loadCodeGraph();
const { default: CodeGraph, getDatabasePath } = await loadCodeGraph();
// `index` is a FULL re-index — identical to a fresh `init`. RECREATE the
// database from scratch (discard .codegraph/codegraph.db + its WAL) rather
// than opening the old graph and DELETE-ing every row. The clear-then-index
@@ -741,7 +744,10 @@ program
// Supervise the indexer: self-terminate if orphaned (parent shim killed)
// or if the main thread wedges — neither was guarded on this path (#999).
const supervision = installCommandSupervision('index');
// The DB + WAL paths let the liveness watchdog tell a slow store on
// degraded storage from a true wedge (#1231).
const dbPath = getDatabasePath(projectPath);
const supervision = installCommandSupervision('index', { progressPaths: [dbPath, `${dbPath}-wal`] });
try {
if (options.quiet) {
// Quiet mode: no UI, just run against the freshly-recreated graph.