github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(indexing): HDD-class storage — false parse timeouts, dropped files, and WAL checkpoint write-back (#1231) (#1242)

Browse Source 
Parse timeouts are now judged by the worker's own clock: the base timer
only marks a job late (after a long synchronous store stall, Node runs the
timers phase before the poll phase, so the timer fired before an
already-delivered result was processed — killing workers over parses that
took milliseconds, even on 0-byte files); a result arriving before a 3×
hard-kill backstop is accepted, timed-out files are retried, and
CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are
pre-read once on the main thread and handed to every worker, so
spawns/respawns load grammars from memory instead of re-reading a
saturated disk.

Bulk indexing defers WAL auto-checkpointing for the whole run: the default
1000-page interval re-writes hot B-tree/FTS pages into the main DB file
over and over — ~95% of all disk I/O under throttled measurement. A
WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill
passes (never blocking the writer or the #850 watchdog heartbeat), pauses
the writer for a full backfill if the disk truly can't keep up, and folds
the WAL at the parse→resolution boundary so post-parse reads never page a
bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with
CODEGRAPH_WAL_VALVE_MB.

Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped
files (was 8); guava-scale completes in 7.6 min with a full graph where
v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-10 03:42:30 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent e76a355df5
commit a11a439002
12 changed files with 928 additions and 73 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/db/index.ts
+113 -7
View File
@@ -189,6 +189,98 @@ export class DatabaseConnection {
return stats.size;
}
/**
* Size of the `-wal` sidecar file in bytes. 0 when it doesn't exist (non-WAL
* journal mode, in-memory DB, or no write since the last checkpoint+reset).
*/
getWalSizeBytes(): number {
if (!this.dbPath || this.dbPath === ':memory:') return 0;
try {
return fs.statSync(`${this.dbPath}-wal`).size;
} catch {
return 0;
}
}
/** Current `wal_autocheckpoint` interval in pages (0 = disabled). */
getWalAutocheckpoint(): number {
const v = this.db.pragma('wal_autocheckpoint', { simple: true });
const n = Number(v);
return Number.isFinite(n) ? n : 0;
}
/**
* Set the connection's `wal_autocheckpoint` interval (pages; 0 disables).
* Bulk indexing defers checkpoints entirely (#1231): the default 1000-page
* auto-checkpoint re-writes hot B-tree/FTS pages into the main DB file over
* and over — measured at ~95% of ALL disk I/O during a bulk index, and the
* difference between 45s and 19+ minutes on HDD-class storage. During
* deferral a {@link WalCheckpointValve} bounds WAL growth off-thread.
*/
setWalAutocheckpoint(pages: number): void {
this.db.pragma(`wal_autocheckpoint = ${Math.max(0, Math.floor(pages))}`);
}
/**
* `PRAGMA wal_checkpoint(PASSIVE)` on a worker thread with its own
* connection. PASSIVE never blocks the writer, and running it off-thread
* means the main thread — and the #850 watchdog heartbeat — keep turning
* even when the backfill is minutes of I/O on slow storage (a synchronous
* checkpoint that exceeds the watchdog's 60s window gets a healthy index
* SIGKILLed — observed in the #1231 repro).
*
* Returns SQLite's checkpoint result row — `log === checkpointed` with
* `busy === 0` means the ENTIRE WAL was backfilled, so the writer's next
* commit restarts the WAL from the top and the file stops growing. The
* WAL valve needs that signal because a WAL file's SIZE never shrinks:
* after the first wrap, raw file size says nothing about the un-backfilled
* backlog. Best-effort: returns null on any failure (including worker
* threads being unavailable — a potentially minutes-long checkpoint must
* never run inline on the main thread).
*/
async checkpointWalPassive(): Promise<{ busy: number; log: number; checkpointed: number } | null> {
if (!this.dbPath || this.dbPath === ':memory:') {
try {
const row = this.db.prepare('PRAGMA wal_checkpoint(PASSIVE)').get() as Record<string, number> | undefined;
return row ? { busy: Number(row.busy), log: Number(row.log), checkpointed: Number(row.checkpointed) } : null;
} catch {
return null;
}
}
try {
const { Worker } = await import('node:worker_threads');
const workerSource = `
const { workerData, parentPort } = require('node:worker_threads');
let row = null;
try {
const { DatabaseSync } = require('node:sqlite');
const db = new DatabaseSync(workerData.dbPath);
try { row = db.prepare('PRAGMA wal_checkpoint(PASSIVE)').get(); } catch {}
try { db.close(); } catch {}
} catch {}
parentPort.postMessage({ row });
`;
return await new Promise((resolve) => {
let settled = false;
const finish = (row?: Record<string, number> | null): void => {
if (settled) return;
settled = true;
resolve(row ? { busy: Number(row.busy), log: Number(row.log), checkpointed: Number(row.checkpointed) } : null);
};
try {
const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath } });
worker.once('message', (m: { row?: Record<string, number> | null }) => { void worker.terminate(); finish(m?.row ?? null); });
worker.once('error', () => { void worker.terminate(); finish(null); });
worker.once('exit', () => finish(null));
} catch {
finish(null);
}
});
} catch {
return null;
}
}
/**
* Optimize database (vacuum and analyze)
*/
@@ -233,6 +325,22 @@ export class DatabaseConnection {
try { this.db.exec('PRAGMA wal_checkpoint(PASSIVE)'); } catch { /* ignore */ }
return;
}
await this.runPragmasOffThread(
['PRAGMA analysis_limit=1000', 'PRAGMA optimize', 'PRAGMA wal_checkpoint(PASSIVE)'],
// Worker threads unavailable — bounded in-line fallback, no checkpoint.
['PRAGMA analysis_limit=1000', 'PRAGMA optimize']
);
}
/**
* Run pragmas on a worker thread against its own connection to this DB
* (shared machinery for {@link runMaintenance} and
* {@link checkpointWalPassive}). Each pragma is individually best-effort;
* the whole call is best-effort. `inlineFallback` (if any) runs on THIS
* connection only when worker threads are unavailable — keep it to pragmas
* that are safe to run synchronously on the main thread.
*/
private async runPragmasOffThread(pragmas: string[], inlineFallback: string[] = []): Promise<void> {
try {
const { Worker } = await import('node:worker_threads');
const workerSource = `
@@ -240,9 +348,7 @@ export class DatabaseConnection {
try {
const { DatabaseSync } = require('node:sqlite');
const db = new DatabaseSync(workerData.dbPath);
try { db.exec('PRAGMA analysis_limit=1000'); } catch {}
try { db.exec('PRAGMA optimize'); } catch {}
try { db.exec('PRAGMA wal_checkpoint(PASSIVE)'); } catch {}
for (const p of workerData.pragmas) { try { db.exec(p); } catch {} }
try { db.close(); } catch {}
} catch {}
parentPort.postMessage('done');
@@ -253,7 +359,7 @@ export class DatabaseConnection {
if (!settled) { settled = true; resolve(); }
};
try {
const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath } });
const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath, pragmas } });
worker.once('message', () => { void worker.terminate(); finish(); });
worker.once('error', () => { void worker.terminate(); finish(); });
worker.once('exit', finish);
@@ -262,9 +368,9 @@ export class DatabaseConnection {
}
});
} catch {
// Worker threads unavailable — bounded in-line fallback, no checkpoint.
try { this.db.exec('PRAGMA analysis_limit=1000'); } catch { /* ignore */ }
try { this.db.exec('PRAGMA optimize'); } catch { /* ignore */ }
for (const p of inlineFallback) {
try { this.db.exec(p); } catch { /* ignore */ }
}
}
}