fix(watchdog): don't kill a healthy index on degraded storage — require heartbeat silence AND no disk progress (#1231) (#1244)
The liveness watchdog judged the main thread by heartbeat silence alone, which cannot distinguish a true wedge (the #850 infinite loop it exists to kill) from one long synchronous SQLite statement on severely degraded storage — so it SIGKILLed valid, in-progress indexes (observed on a 150-IOPS throttled rig, and latent on real HDDs at scale). The CLI index/init paths now hand the watchdog the project's DB + WAL paths. On a silent timeout the watchdog child stats them first: if they advanced during the silence, the block is a slow store making forward progress — defer and keep watching; if not, kill at the base timeout exactly as before. Deferral is bounded by a hard cap (10× the timeout) of continuous silence so a wedge coinciding with unrelated file activity, or I/O hung beyond any legitimate statement, still dies. The daemon path is unchanged (no progress paths — pure heartbeat). Validated with real spawned processes (defer-on-progress, kill-on-static, hard-cap kill) and on the throttled rig: a 150-IOPS index under a 10s watchdog window — 6× tighter than production, with store stalls measured at 10-20s — completes cleanly where the old watchdog killed it, while true-wedge kill latency is unchanged. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
116cb59625
commit
edb9f2f14c
+10
-4
@@ -590,7 +590,7 @@ program
|
||||
return;
|
||||
}
|
||||
|
||||
const { default: CodeGraph } = await loadCodeGraph();
|
||||
const { default: CodeGraph, getDatabasePath } = await loadCodeGraph();
|
||||
const cg = await CodeGraph.init(projectPath, { index: false });
|
||||
clack.log.success(`Initialized in ${projectPath}`);
|
||||
|
||||
@@ -598,10 +598,13 @@ program
|
||||
// accepted (so existing muscle memory and scripts don't break) but is a
|
||||
// no-op — initializing always builds the initial index.
|
||||
// Supervise the index: self-terminate if orphaned or wedged (#999).
|
||||
// The DB + WAL paths let the liveness watchdog tell a slow store on
|
||||
// degraded storage from a true wedge (#1231).
|
||||
// A closure so we can re-run the exact same supervised, progress-rendered
|
||||
// index if the user opts gitignored child repos in below (#1156).
|
||||
const dbPath = getDatabasePath(projectPath);
|
||||
const runIndex = async (): Promise<IndexResult> => {
|
||||
const supervision = installCommandSupervision('init');
|
||||
const supervision = installCommandSupervision('init', { progressPaths: [dbPath, `${dbPath}-wal`] });
|
||||
try {
|
||||
if (options.verbose) {
|
||||
return await cg.indexAll({ onProgress: createVerboseProgress(), verbose: true });
|
||||
@@ -727,7 +730,7 @@ program
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const { default: CodeGraph } = await loadCodeGraph();
|
||||
const { default: CodeGraph, getDatabasePath } = await loadCodeGraph();
|
||||
// `index` is a FULL re-index — identical to a fresh `init`. RECREATE the
|
||||
// database from scratch (discard .codegraph/codegraph.db + its WAL) rather
|
||||
// than opening the old graph and DELETE-ing every row. The clear-then-index
|
||||
@@ -741,7 +744,10 @@ program
|
||||
|
||||
// Supervise the indexer: self-terminate if orphaned (parent shim killed)
|
||||
// or if the main thread wedges — neither was guarded on this path (#999).
|
||||
const supervision = installCommandSupervision('index');
|
||||
// The DB + WAL paths let the liveness watchdog tell a slow store on
|
||||
// degraded storage from a true wedge (#1231).
|
||||
const dbPath = getDatabasePath(projectPath);
|
||||
const supervision = installCommandSupervision('index', { progressPaths: [dbPath, `${dbPath}-wal`] });
|
||||
try {
|
||||
if (options.quiet) {
|
||||
// Quiet mode: no UI, just run against the freshly-recreated graph.
|
||||
|
||||
Reference in New Issue
Block a user