fix(db): stop watchdog-killed sessions from leaking the SQLite WAL without bound (#1431) (#1490)

A SIGKILL'd process (the #850 liveness watchdog, OOM, a crash) leaves its WAL
on disk; the next session appends to the same file; and nothing ever truncated
it — PASSIVE checkpoints fold frames but keep the file at its high-water mark,
and the one shrinking path (a clean last-connection close) is exactly what a
killed-daemon world never takes. Observed at 25.6 GB on a 5.46 GB DB, growing
until the disk filled.

- journal_size_limit on every connection: resetting checkpoints now clip the
  WAL back to the cap instead of leaving it at its high-water mark.
- healOversizedWal() fired from every DatabaseConnection.open: off-thread
  PASSIVE fold + TRUNCATE when the leftover WAL exceeds the cap (64 MB,
  CODEGRAPH_WAL_HEAL_MB to override). Single-flight per connection with
  bounded retries — concurrent passes defeat each other (each checkpoint sees
  the other as a busy reader).
- Daemon/direct MCP watchdogs now pass progressPaths (DB + WAL), extending the
  #1231 slow-disk deferral to the long-lived server so a healthy daemon mid
  slow statement isn't SIGKILL'd — fewer kills, fewer leaked WALs.
- codegraph status shows WAL size (human + JSON) and warns when it dwarfs the
  DB; daemon.log lines and the watchdog kill notice now carry ISO timestamps
  so kills can be placed in time.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-31 21:38:38 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 0682137a42
commit 02c0e2c935
9 changed files with 345 additions and 3 deletions
+14
View File
@@ -961,6 +961,7 @@ program
nodeCount: stats.nodeCount,
edgeCount: stats.edgeCount,
dbSizeBytes: stats.dbSizeBytes,
walSizeBytes: stats.walSizeBytes,
backend,
journalMode,
nodesByKind: stats.nodesByKind,
@@ -1017,6 +1018,19 @@ program
console.log(` Nodes: ${formatNumber(stats.nodeCount)}`);
console.log(` Edges: ${formatNumber(stats.edgeCount)}`);
console.log(` DB Size: ${(stats.dbSizeBytes / 1024 / 1024).toFixed(2)} MB`);
// Surface the WAL sidecar (#1431): a WAL that dwarfs the DB at rest is
// the killed-session leak — invisible before this line, it only showed
// up as a mysteriously full disk. open() above already kicked off the
// automatic heal for the oversized case.
if (stats.walSizeBytes > 0) {
const { WAL_HEAL_THRESHOLD_BYTES } = await import('../db/index');
const oversized = stats.walSizeBytes > Math.max(WAL_HEAL_THRESHOLD_BYTES, stats.dbSizeBytes);
const walLabel = `${(stats.walSizeBytes / 1024 / 1024).toFixed(2)} MB`;
console.log(` WAL Size: ${oversized ? chalk.yellow(walLabel) : walLabel}`);
if (oversized) {
warn('The write-ahead log is larger than the database — killed sessions left it behind. It is reclaimed automatically on open; if it persists across runs, another live CodeGraph process is holding it.');
}
}
// Surface the active SQLite backend (node:sqlite — Node's built-in real
// SQLite, full WAL + FTS5, no native build).
const backendLabel = chalk.green(`node:sqlite ${getGlyphs().dash} built-in (full WAL)`);