github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(watchdog): don't kill a healthy index on degraded storage — require heartbeat silence AND no disk progress (#1231) (#1244)

Browse Source 
The liveness watchdog judged the main thread by heartbeat silence alone,
which cannot distinguish a true wedge (the #850 infinite loop it exists to
kill) from one long synchronous SQLite statement on severely degraded
storage — so it SIGKILLed valid, in-progress indexes (observed on a
150-IOPS throttled rig, and latent on real HDDs at scale).

The CLI index/init paths now hand the watchdog the project's DB + WAL
paths. On a silent timeout the watchdog child stats them first: if they
advanced during the silence, the block is a slow store making forward
progress — defer and keep watching; if not, kill at the base timeout
exactly as before. Deferral is bounded by a hard cap (10× the timeout) of
continuous silence so a wedge coinciding with unrelated file activity, or
I/O hung beyond any legitimate statement, still dies. The daemon path is
unchanged (no progress paths — pure heartbeat).

Validated with real spawned processes (defer-on-progress, kill-on-static,
hard-cap kill) and on the throttled rig: a 150-IOPS index under a 10s
watchdog window — 6× tighter than production, with store stalls measured
at 10-20s — completes cleanly where the old watchdog killed it, while
true-wedge kill latency is unchanged.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-10 08:25:23 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 116cb59625
commit edb9f2f14c
5 changed files with 172 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+3
View File
@@ -9,6 +9,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Fixes
- The safety watchdog no longer kills a healthy index on severely degraded storage. It used to judge liveness purely by the event loop, so one long database write on a struggling disk looked identical to a hung process and could get a valid, in-progress index terminated. During `codegraph index`/`codegraph init` the watchdog now also checks whether the index files on disk are advancing before it acts: slow-but-progressing work is left alone (bounded by a hard cap), while a genuinely hung process is still killed exactly as fast as before. (#1231)
## [1.4.0] - 2026-07-10