fix(indexing): HDD-class storage — false parse timeouts, dropped files, and WAL checkpoint write-back (#1231) (#1242)
Parse timeouts are now judged by the worker's own clock: the base timer only marks a job late (after a long synchronous store stall, Node runs the timers phase before the poll phase, so the timer fired before an already-delivered result was processed — killing workers over parses that took milliseconds, even on 0-byte files); a result arriving before a 3× hard-kill backstop is accepted, timed-out files are retried, and CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are pre-read once on the main thread and handed to every worker, so spawns/respawns load grammars from memory instead of re-reading a saturated disk. Bulk indexing defers WAL auto-checkpointing for the whole run: the default 1000-page interval re-writes hot B-tree/FTS pages into the main DB file over and over — ~95% of all disk I/O under throttled measurement. A WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill passes (never blocking the writer or the #850 watchdog heartbeat), pauses the writer for a full backfill if the disk truly can't keep up, and folds the WAL at the parse→resolution boundary so post-parse reads never page a bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with CODEGRAPH_WAL_VALVE_MB. Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped files (was 8); guava-scale completes in 7.6 min with a full graph where v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Fable 5
Claude Fable 5
parent
e76a355df5
commit
a11a439002
@@ -0,0 +1,7 @@
|
||||
node_modules
|
||||
dist
|
||||
.git
|
||||
.codegraph
|
||||
.kommandr
|
||||
docs
|
||||
assets
|
||||
@@ -9,6 +9,15 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### New Features
|
||||
|
||||
- Indexing is dramatically faster on slow storage — mechanical HDDs, network folders, and virtualized disks. The database no longer folds its write journal back into the main file thousands of times during a bulk index (that folding was ~95% of all disk activity); it now streams writes sequentially and folds them back in a few large, coalesced passes that run off the main thread. In a disk-throttled benchmark matching the reported hardware, a mid-size Java project went from over 25 minutes to under a minute, and there is no change on fast disks. Opt out with `CODEGRAPH_NO_WAL_DEFER=1`; tune the fold-back threshold with `CODEGRAPH_WAL_VALVE_MB`. (#1231)
|
||||
- New `CODEGRAPH_PARSE_TIMEOUT_MS` environment variable to raise the per-file parse budget on unusually slow storage, the same way `CODEGRAPH_PARSE_WORKERS` already tunes the worker count. (#1231)
|
||||
|
||||
### Fixes
|
||||
|
||||
- Indexing on slow storage (mechanical HDDs, network folders) no longer collapses into false "parse timeout" failures. When disk writes stalled the coordinating thread, parses that had already finished — including empty files — were being misjudged as hung, their workers killed, and the files silently dropped from the index. A parse result is now judged by the worker's own clock, so a stalled coordinator accepts the finished result instead of killing the worker; only a genuinely hung parse is terminated (after a wider grace window). Files that do hit the timeout are retried at the end of indexing instead of being silently lost. Thanks @KnifeOfLife for the exceptional report. (#1231)
|
||||
- Parse workers now receive their grammar files from memory instead of each re-reading them from disk on spawn, eliminating a feedback loop on slow disks where every worker restart added more disk contention — and making worker restarts cheaper everywhere. (#1231)
|
||||
|
||||
## [1.3.1] - 2026-07-09
|
||||
|
||||
|
||||
@@ -0,0 +1,43 @@
|
||||
/**
|
||||
* readGrammarWasmBytes + bytes-based grammar loading (#1231, Phase 2.1).
|
||||
*
|
||||
* The orchestrator pre-reads each needed grammar's WASM once on the main
|
||||
* thread and hands the bytes to every parse worker, so a worker respawn loads
|
||||
* grammars from memory instead of re-reading them from a (possibly slow) disk.
|
||||
* These tests pin that the byte reader resolves the same artifacts the loader
|
||||
* would, and that web-tree-sitter genuinely accepts the bytes.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { Parser, Language as WasmLanguage } from 'web-tree-sitter';
|
||||
import { readGrammarWasmBytes } from '../src/extraction/grammars';
|
||||
|
||||
describe('readGrammarWasmBytes', () => {
|
||||
it('reads bytes for a tree-sitter-wasms grammar and a vendored grammar', async () => {
|
||||
const bytes = await readGrammarWasmBytes(['typescript', 'lua']);
|
||||
expect(bytes.typescript).toBeInstanceOf(Uint8Array); // from tree-sitter-wasms
|
||||
expect(bytes.typescript.byteLength).toBeGreaterThan(10_000);
|
||||
expect(bytes.lua).toBeInstanceOf(Uint8Array); // vendored under src/extraction/wasm/
|
||||
expect(bytes.lua.byteLength).toBeGreaterThan(10_000);
|
||||
});
|
||||
|
||||
it('expands delegating languages to the grammars they need (svelte → ts/js)', async () => {
|
||||
const bytes = await readGrammarWasmBytes(['svelte']);
|
||||
expect(Object.keys(bytes).sort()).toEqual(['javascript', 'typescript']);
|
||||
});
|
||||
|
||||
it('omits languages without a WASM grammar instead of failing', async () => {
|
||||
const bytes = await readGrammarWasmBytes(['yaml', 'unknown']);
|
||||
expect(Object.keys(bytes)).toEqual([]);
|
||||
});
|
||||
|
||||
it('produces bytes web-tree-sitter can load into a working parser', async () => {
|
||||
await Parser.init();
|
||||
const bytes = await readGrammarWasmBytes(['javascript']);
|
||||
const language = await WasmLanguage.load(bytes.javascript);
|
||||
const parser = new Parser();
|
||||
parser.setLanguage(language);
|
||||
const tree = parser.parse('function hello() { return 1; }');
|
||||
expect(tree!.rootNode.hasError).toBe(false);
|
||||
expect(tree!.rootNode.toString()).toContain('function_declaration');
|
||||
});
|
||||
});
|
||||
@@ -11,7 +11,7 @@
|
||||
* parallelism safe.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest';
|
||||
import { ParseWorkerPool, resolveParsePoolSize, type ParsePoolWorker, type ParseTask } from '../src/extraction/parse-pool';
|
||||
import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs, type ParsePoolWorker, type ParseTask } from '../src/extraction/parse-pool';
|
||||
import type { Language, ExtractionResult } from '../src/types';
|
||||
|
||||
const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));
|
||||
@@ -80,6 +80,20 @@ function makePool(
|
||||
return { pool, counts: () => ({ spawned, terminated }) };
|
||||
}
|
||||
|
||||
describe('resolveParseTimeoutMs', () => {
|
||||
it('honors a positive numeric override (CODEGRAPH_PARSE_TIMEOUT_MS)', () => {
|
||||
expect(resolveParseTimeoutMs('30000')).toBe(30000);
|
||||
expect(resolveParseTimeoutMs('1500.9')).toBe(1500);
|
||||
});
|
||||
it('falls back to the 10s default when unset/blank/non-numeric/non-positive', () => {
|
||||
expect(resolveParseTimeoutMs(undefined)).toBe(10_000);
|
||||
expect(resolveParseTimeoutMs('')).toBe(10_000);
|
||||
expect(resolveParseTimeoutMs('abc')).toBe(10_000);
|
||||
expect(resolveParseTimeoutMs('0')).toBe(10_000);
|
||||
expect(resolveParseTimeoutMs('-5')).toBe(10_000);
|
||||
});
|
||||
});
|
||||
|
||||
describe('resolveParsePoolSize', () => {
|
||||
it('treats explicit 0 and 1 as a single worker (the rollback path)', () => {
|
||||
expect(resolveParsePoolSize('0', 8)).toBe(1);
|
||||
@@ -150,14 +164,61 @@ describe('ParseWorkerPool', () => {
|
||||
await pool.destroy();
|
||||
});
|
||||
|
||||
it('times out a hung parse and stays usable', async () => {
|
||||
it('times out a hung parse (at the hard-kill backstop) and stays usable', async () => {
|
||||
const { pool } = makePool(1, (m) => (m.filePath === 'hang.ts' ? { hang: true } : { result: result(9) }), { parseTimeoutMs: 30 });
|
||||
await expect(pool.requestParse(task('hang.ts'))).rejects.toThrow(/timed out/i);
|
||||
const t0 = Date.now();
|
||||
// The base timer (30ms) only marks the job late; the kill happens at the
|
||||
// 3× backstop (90ms), and the message carries the full window.
|
||||
await expect(pool.requestParse(task('hang.ts'))).rejects.toThrow(/timed out after 90ms/i);
|
||||
expect(Date.now() - t0).toBeGreaterThanOrEqual(80);
|
||||
const ok = await pool.requestParse(task('ok.ts'));
|
||||
expect(ok.durationMs).toBe(9);
|
||||
await pool.destroy();
|
||||
});
|
||||
|
||||
it('accepts a result that arrives after the base timeout instead of killing the worker (#1231 false-timeout fix)', async () => {
|
||||
// Simulates the HDD stall: the parse "finished" but its result is only
|
||||
// delivered after the base timer fired. Old behaviour killed the worker and
|
||||
// rejected; now the late result is accepted and the worker keeps serving.
|
||||
const { pool, counts } = makePool(
|
||||
1,
|
||||
(m) => (m.filePath === 'late.ts' ? { wait: sleep(80).then(() => result(11)) } : { result: result(9) }),
|
||||
{ parseTimeoutMs: 50 }
|
||||
);
|
||||
const res = await pool.requestParse(task('late.ts')); // base timer 50ms < delivery 80ms < backstop 150ms
|
||||
expect(res.durationMs).toBe(11);
|
||||
expect(counts().terminated).toBe(0); // no kill…
|
||||
expect(counts().spawned).toBe(1); // …and no respawn churn
|
||||
const ok = await pool.requestParse(task('next.ts'));
|
||||
expect(ok.durationMs).toBe(9); // same worker still serving
|
||||
await pool.destroy();
|
||||
});
|
||||
|
||||
it('forwards pre-read grammar WASM bytes to every spawned worker (#1231 respawn I/O fix)', async () => {
|
||||
const grammarBuffers = { typescript: new Uint8Array([1, 2, 3]) };
|
||||
const loadMsgs: Array<{ grammarBuffers?: Record<string, Uint8Array> }> = [];
|
||||
let worker!: FakeWorker;
|
||||
const pool = new ParseWorkerPool({
|
||||
languages: ['typescript'] as Language[],
|
||||
size: 1,
|
||||
grammarBuffers,
|
||||
createWorker: () => {
|
||||
worker = new FakeWorker(() => ({ result: result() }));
|
||||
const orig = worker.postMessage.bind(worker);
|
||||
worker.postMessage = (msg: unknown) => {
|
||||
const m = msg as { type: string; grammarBuffers?: Record<string, Uint8Array> };
|
||||
if (m.type === 'load-grammars') loadMsgs.push(m);
|
||||
orig(msg);
|
||||
};
|
||||
return worker;
|
||||
},
|
||||
});
|
||||
await pool.requestParse(task('a.ts'));
|
||||
expect(loadMsgs).toHaveLength(1);
|
||||
expect(loadMsgs[0].grammarBuffers).toBe(grammarBuffers);
|
||||
await pool.destroy();
|
||||
});
|
||||
|
||||
it('serves a queue larger than the pool size', async () => {
|
||||
const { pool } = makePool(2, (m) => ({ result: result(Number(m.filePath.replace(/\D/g, ''))) }));
|
||||
const ps = Array.from({ length: 10 }, (_, i) => pool.requestParse(task(`${i}.ts`)));
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
/**
|
||||
* WAL checkpoint deferral during bulk indexing (#1231).
|
||||
*
|
||||
* The default 1000-page wal_autocheckpoint re-writes hot pages into the main
|
||||
* DB over and over during a bulk index (~95% of all disk I/O on slow
|
||||
* storage). indexAll defers auto-checkpointing for the whole run, a
|
||||
* WalCheckpointValve bounds WAL growth via off-thread PASSIVE checkpoints,
|
||||
* and the interval is restored afterwards. These tests pin the DB helpers,
|
||||
* the valve's trigger/dedupe/backpressure logic, and the end-to-end indexAll
|
||||
* behavior (identical graph with and without deferral; interval restored).
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as os from 'os';
|
||||
import * as path from 'path';
|
||||
import { DatabaseConnection } from '../src/db';
|
||||
import { WalCheckpointValve, resolveWalValveMb } from '../src/db/wal-valve';
|
||||
import CodeGraph from '../src/index';
|
||||
|
||||
let tmpDir: string;
|
||||
|
||||
beforeEach(() => {
|
||||
tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-wal-deferral-'));
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
fs.rmSync(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
function openDb(): DatabaseConnection {
|
||||
return DatabaseConnection.initialize(path.join(tmpDir, 'test.db'));
|
||||
}
|
||||
|
||||
/** Grow the WAL: with autocheckpoint off, every commit appends and nothing folds back. */
|
||||
function writeRows(db: DatabaseConnection, rows: number): void {
|
||||
const raw = db.getDb();
|
||||
raw.exec('CREATE TABLE IF NOT EXISTS t (id INTEGER PRIMARY KEY, blob TEXT)');
|
||||
const stmt = raw.prepare('INSERT INTO t (blob) VALUES (?)');
|
||||
for (let i = 0; i < rows; i++) stmt.run('x'.repeat(4096));
|
||||
}
|
||||
|
||||
describe('resolveWalValveMb', () => {
|
||||
it('honors a positive numeric override and falls back otherwise', () => {
|
||||
expect(resolveWalValveMb('64')).toBe(64);
|
||||
expect(resolveWalValveMb('64.9')).toBe(64);
|
||||
expect(resolveWalValveMb(undefined)).toBe(256);
|
||||
expect(resolveWalValveMb('')).toBe(256);
|
||||
expect(resolveWalValveMb('abc')).toBe(256);
|
||||
expect(resolveWalValveMb('0')).toBe(256);
|
||||
expect(resolveWalValveMb('-5')).toBe(256);
|
||||
});
|
||||
});
|
||||
|
||||
describe('DatabaseConnection WAL helpers', () => {
|
||||
it('reads and writes the wal_autocheckpoint interval', () => {
|
||||
const db = openDb();
|
||||
expect(db.getWalAutocheckpoint()).toBe(1000); // SQLite default
|
||||
db.setWalAutocheckpoint(0);
|
||||
expect(db.getWalAutocheckpoint()).toBe(0);
|
||||
db.setWalAutocheckpoint(1000);
|
||||
expect(db.getWalAutocheckpoint()).toBe(1000);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('reports WAL size that grows with deferred commits', () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
const before = db.getWalSizeBytes();
|
||||
writeRows(db, 200);
|
||||
expect(db.getWalSizeBytes()).toBeGreaterThan(before);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('checkpointWalPassive backfills the WAL from a worker connection and reports the result', async () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
writeRows(db, 500);
|
||||
const dbFile = path.join(tmpDir, 'test.db');
|
||||
const mainSizeBefore = fs.statSync(dbFile).size;
|
||||
const res = await db.checkpointWalPassive();
|
||||
// Backfill moves the committed pages into the main DB file…
|
||||
expect(fs.statSync(dbFile).size).toBeGreaterThan(mainSizeBefore);
|
||||
// …and reports a full backfill (idle DB: every WAL frame checkpointed).
|
||||
expect(res).not.toBeNull();
|
||||
expect(res!.busy).toBe(0);
|
||||
expect(res!.log).toBeGreaterThan(0);
|
||||
expect(res!.checkpointed).toBe(res!.log);
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe('WalCheckpointValve', () => {
|
||||
it('check() fires an off-thread checkpoint once growth passes the soft threshold', async () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
writeRows(db, 500); // WAL well past a ~10-byte threshold
|
||||
const valve = new WalCheckpointValve(db, 0.00001); // ~10 bytes soft
|
||||
const dbFile = path.join(tmpDir, 'test.db');
|
||||
const mainSizeBefore = fs.statSync(dbFile).size;
|
||||
valve.check();
|
||||
await valve.drain();
|
||||
expect(fs.statSync(dbFile).size).toBeGreaterThan(mainSizeBefore);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('advances its baseline on a full backfill — a wrapped WAL does not retrigger it', async () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
writeRows(db, 500);
|
||||
const valve = new WalCheckpointValve(db, 0.00001);
|
||||
valve.check();
|
||||
await valve.drain(); // full backfill on an idle DB → baseline = current file size
|
||||
// The WAL file keeps its high-water size, but growth is now 0: neither
|
||||
// the timer path nor backpressure may fire again (the pre-fix bug fired
|
||||
// on raw size forever and serialized every store behind a checkpoint).
|
||||
expect(valve.backpressure()).toBeNull();
|
||||
valve.check();
|
||||
await valve.drain(); // no-op drain: nothing in flight
|
||||
// New commits recycle wrapped frames — file size is flat, still no trigger.
|
||||
writeRows(db, 5);
|
||||
expect(valve.backpressure()).toBeNull();
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('does not fire below the soft threshold', async () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
writeRows(db, 5);
|
||||
const valve = new WalCheckpointValve(db, 1024); // 1GB soft — never reached
|
||||
const dbFile = path.join(tmpDir, 'test.db');
|
||||
const mainSizeBefore = fs.statSync(dbFile).size;
|
||||
valve.check();
|
||||
await valve.drain();
|
||||
expect(fs.statSync(dbFile).size).toBe(mainSizeBefore);
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('backpressure() is null under the hard cap and a promise above it', async () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
writeRows(db, 500);
|
||||
const relaxed = new WalCheckpointValve(db, 1024);
|
||||
expect(relaxed.backpressure()).toBeNull();
|
||||
const strict = new WalCheckpointValve(db, 0.0000001); // hard cap ~0.4 bytes
|
||||
const bp = strict.backpressure();
|
||||
expect(bp).toBeInstanceOf(Promise);
|
||||
await bp;
|
||||
await strict.drain();
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('foldNow() backfills everything at a phase boundary and resets growth', async () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
writeRows(db, 500);
|
||||
const valve = new WalCheckpointValve(db, 1024); // thresholds never reached on their own
|
||||
const dbFile = path.join(tmpDir, 'test.db');
|
||||
const mainSizeBefore = fs.statSync(dbFile).size;
|
||||
await valve.foldNow();
|
||||
expect(fs.statSync(dbFile).size).toBeGreaterThan(mainSizeBefore); // pages backfilled
|
||||
expect(valve.backpressure()).toBeNull(); // baseline advanced — growth is zero
|
||||
await valve.foldNow(); // second fold is a no-op (growth 0), must not spin
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('dedupes concurrent fires into one in-flight checkpoint', () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
writeRows(db, 500);
|
||||
const valve = new WalCheckpointValve(db, 0.00001);
|
||||
valve.check();
|
||||
const first = valve.backpressure();
|
||||
const second = valve.backpressure();
|
||||
expect(second).toBe(first); // same in-flight promise, not a second worker
|
||||
db.close();
|
||||
return first ?? undefined;
|
||||
});
|
||||
});
|
||||
|
||||
describe('indexAll WAL deferral end-to-end', () => {
|
||||
function writeFixtureProject(): void {
|
||||
fs.mkdirSync(path.join(tmpDir, 'src'), { recursive: true });
|
||||
for (let i = 0; i < 8; i++) {
|
||||
fs.writeFileSync(
|
||||
path.join(tmpDir, 'src', `mod${i}.ts`),
|
||||
`export function fn${i}(x: number): number { return helper${i}(x) + ${i}; }\n` +
|
||||
`function helper${i}(x: number): number { return x * ${i}; }\n`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
it('produces the same graph with and without deferral, and restores the interval', async () => {
|
||||
writeFixtureProject();
|
||||
|
||||
const cg1 = CodeGraph.initSync(tmpDir);
|
||||
const r1 = await cg1.indexAll();
|
||||
expect(r1.success).toBe(true);
|
||||
// Deferral is scoped to the run: the connection is back on the default.
|
||||
const conn1 = (cg1 as unknown as { db: DatabaseConnection }).db;
|
||||
expect(conn1.getWalAutocheckpoint()).toBe(1000);
|
||||
const counts1 = { nodes: r1.nodesCreated, edges: r1.edgesCreated };
|
||||
await cg1.close();
|
||||
|
||||
fs.rmSync(path.join(tmpDir, '.codegraph'), { recursive: true, force: true });
|
||||
|
||||
process.env.CODEGRAPH_NO_WAL_DEFER = '1';
|
||||
try {
|
||||
const cg2 = CodeGraph.initSync(tmpDir);
|
||||
const r2 = await cg2.indexAll();
|
||||
expect(r2.success).toBe(true);
|
||||
expect({ nodes: r2.nodesCreated, edges: r2.edgesCreated }).toEqual(counts1);
|
||||
await cg2.close();
|
||||
} finally {
|
||||
delete process.env.CODEGRAPH_NO_WAL_DEFER;
|
||||
}
|
||||
});
|
||||
});
|
||||
+113
-7
@@ -189,6 +189,98 @@ export class DatabaseConnection {
|
||||
return stats.size;
|
||||
}
|
||||
|
||||
/**
|
||||
* Size of the `-wal` sidecar file in bytes. 0 when it doesn't exist (non-WAL
|
||||
* journal mode, in-memory DB, or no write since the last checkpoint+reset).
|
||||
*/
|
||||
getWalSizeBytes(): number {
|
||||
if (!this.dbPath || this.dbPath === ':memory:') return 0;
|
||||
try {
|
||||
return fs.statSync(`${this.dbPath}-wal`).size;
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
|
||||
/** Current `wal_autocheckpoint` interval in pages (0 = disabled). */
|
||||
getWalAutocheckpoint(): number {
|
||||
const v = this.db.pragma('wal_autocheckpoint', { simple: true });
|
||||
const n = Number(v);
|
||||
return Number.isFinite(n) ? n : 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Set the connection's `wal_autocheckpoint` interval (pages; 0 disables).
|
||||
* Bulk indexing defers checkpoints entirely (#1231): the default 1000-page
|
||||
* auto-checkpoint re-writes hot B-tree/FTS pages into the main DB file over
|
||||
* and over — measured at ~95% of ALL disk I/O during a bulk index, and the
|
||||
* difference between 45s and 19+ minutes on HDD-class storage. During
|
||||
* deferral a {@link WalCheckpointValve} bounds WAL growth off-thread.
|
||||
*/
|
||||
setWalAutocheckpoint(pages: number): void {
|
||||
this.db.pragma(`wal_autocheckpoint = ${Math.max(0, Math.floor(pages))}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* `PRAGMA wal_checkpoint(PASSIVE)` on a worker thread with its own
|
||||
* connection. PASSIVE never blocks the writer, and running it off-thread
|
||||
* means the main thread — and the #850 watchdog heartbeat — keep turning
|
||||
* even when the backfill is minutes of I/O on slow storage (a synchronous
|
||||
* checkpoint that exceeds the watchdog's 60s window gets a healthy index
|
||||
* SIGKILLed — observed in the #1231 repro).
|
||||
*
|
||||
* Returns SQLite's checkpoint result row — `log === checkpointed` with
|
||||
* `busy === 0` means the ENTIRE WAL was backfilled, so the writer's next
|
||||
* commit restarts the WAL from the top and the file stops growing. The
|
||||
* WAL valve needs that signal because a WAL file's SIZE never shrinks:
|
||||
* after the first wrap, raw file size says nothing about the un-backfilled
|
||||
* backlog. Best-effort: returns null on any failure (including worker
|
||||
* threads being unavailable — a potentially minutes-long checkpoint must
|
||||
* never run inline on the main thread).
|
||||
*/
|
||||
async checkpointWalPassive(): Promise<{ busy: number; log: number; checkpointed: number } | null> {
|
||||
if (!this.dbPath || this.dbPath === ':memory:') {
|
||||
try {
|
||||
const row = this.db.prepare('PRAGMA wal_checkpoint(PASSIVE)').get() as Record<string, number> | undefined;
|
||||
return row ? { busy: Number(row.busy), log: Number(row.log), checkpointed: Number(row.checkpointed) } : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
try {
|
||||
const { Worker } = await import('node:worker_threads');
|
||||
const workerSource = `
|
||||
const { workerData, parentPort } = require('node:worker_threads');
|
||||
let row = null;
|
||||
try {
|
||||
const { DatabaseSync } = require('node:sqlite');
|
||||
const db = new DatabaseSync(workerData.dbPath);
|
||||
try { row = db.prepare('PRAGMA wal_checkpoint(PASSIVE)').get(); } catch {}
|
||||
try { db.close(); } catch {}
|
||||
} catch {}
|
||||
parentPort.postMessage({ row });
|
||||
`;
|
||||
return await new Promise((resolve) => {
|
||||
let settled = false;
|
||||
const finish = (row?: Record<string, number> | null): void => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
resolve(row ? { busy: Number(row.busy), log: Number(row.log), checkpointed: Number(row.checkpointed) } : null);
|
||||
};
|
||||
try {
|
||||
const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath } });
|
||||
worker.once('message', (m: { row?: Record<string, number> | null }) => { void worker.terminate(); finish(m?.row ?? null); });
|
||||
worker.once('error', () => { void worker.terminate(); finish(null); });
|
||||
worker.once('exit', () => finish(null));
|
||||
} catch {
|
||||
finish(null);
|
||||
}
|
||||
});
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Optimize database (vacuum and analyze)
|
||||
*/
|
||||
@@ -233,6 +325,22 @@ export class DatabaseConnection {
|
||||
try { this.db.exec('PRAGMA wal_checkpoint(PASSIVE)'); } catch { /* ignore */ }
|
||||
return;
|
||||
}
|
||||
await this.runPragmasOffThread(
|
||||
['PRAGMA analysis_limit=1000', 'PRAGMA optimize', 'PRAGMA wal_checkpoint(PASSIVE)'],
|
||||
// Worker threads unavailable — bounded in-line fallback, no checkpoint.
|
||||
['PRAGMA analysis_limit=1000', 'PRAGMA optimize']
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Run pragmas on a worker thread against its own connection to this DB
|
||||
* (shared machinery for {@link runMaintenance} and
|
||||
* {@link checkpointWalPassive}). Each pragma is individually best-effort;
|
||||
* the whole call is best-effort. `inlineFallback` (if any) runs on THIS
|
||||
* connection only when worker threads are unavailable — keep it to pragmas
|
||||
* that are safe to run synchronously on the main thread.
|
||||
*/
|
||||
private async runPragmasOffThread(pragmas: string[], inlineFallback: string[] = []): Promise<void> {
|
||||
try {
|
||||
const { Worker } = await import('node:worker_threads');
|
||||
const workerSource = `
|
||||
@@ -240,9 +348,7 @@ export class DatabaseConnection {
|
||||
try {
|
||||
const { DatabaseSync } = require('node:sqlite');
|
||||
const db = new DatabaseSync(workerData.dbPath);
|
||||
try { db.exec('PRAGMA analysis_limit=1000'); } catch {}
|
||||
try { db.exec('PRAGMA optimize'); } catch {}
|
||||
try { db.exec('PRAGMA wal_checkpoint(PASSIVE)'); } catch {}
|
||||
for (const p of workerData.pragmas) { try { db.exec(p); } catch {} }
|
||||
try { db.close(); } catch {}
|
||||
} catch {}
|
||||
parentPort.postMessage('done');
|
||||
@@ -253,7 +359,7 @@ export class DatabaseConnection {
|
||||
if (!settled) { settled = true; resolve(); }
|
||||
};
|
||||
try {
|
||||
const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath } });
|
||||
const worker = new Worker(workerSource, { eval: true, workerData: { dbPath: this.dbPath, pragmas } });
|
||||
worker.once('message', () => { void worker.terminate(); finish(); });
|
||||
worker.once('error', () => { void worker.terminate(); finish(); });
|
||||
worker.once('exit', finish);
|
||||
@@ -262,9 +368,9 @@ export class DatabaseConnection {
|
||||
}
|
||||
});
|
||||
} catch {
|
||||
// Worker threads unavailable — bounded in-line fallback, no checkpoint.
|
||||
try { this.db.exec('PRAGMA analysis_limit=1000'); } catch { /* ignore */ }
|
||||
try { this.db.exec('PRAGMA optimize'); } catch { /* ignore */ }
|
||||
for (const p of inlineFallback) {
|
||||
try { this.db.exec(p); } catch { /* ignore */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,206 @@
|
||||
/**
|
||||
* WAL checkpoint valve — bounds WAL growth while auto-checkpointing is
|
||||
* deferred during a bulk index (#1231).
|
||||
*
|
||||
* Why deferral: SQLite's default `wal_autocheckpoint` (1000 pages) re-writes
|
||||
* hot B-tree/FTS pages into the main DB file over and over during a bulk
|
||||
* index — measured at ~95% of ALL disk I/O, and the difference between 45s
|
||||
* and 19+ minutes on HDD-class storage (150 random IOPS). Deferring
|
||||
* checkpoints turns the store into pure sequential WAL appends; each backfill
|
||||
* pass writes distinct pages once, in page order (≈ sequential).
|
||||
*
|
||||
* Why a valve: unbounded deferral is its own failure mode, both measured in
|
||||
* the #1231 repro. The WAL duplicates hot pages per COMMIT, so it grows far
|
||||
* faster than the DB (5.9GB WAL for a ~340MB DB on a 3.3k-file index) —
|
||||
* filling the disk, and poisoning every subsequent read that must page
|
||||
* through it (the first resolution-phase read blocked the main thread >60s
|
||||
* and the #850 liveness watchdog killed the healthy index). The valve
|
||||
* watches WAL growth on a timer and, past a soft threshold, backfills with
|
||||
* `PRAGMA wal_checkpoint(PASSIVE)` on a worker-thread connection — PASSIVE
|
||||
* never blocks the writer, and off-thread means the main thread (and the
|
||||
* watchdog heartbeat) keep turning regardless of how long a backfill takes.
|
||||
*
|
||||
* The load-bearing subtlety: a WAL file's SIZE never shrinks. After a full
|
||||
* backfill, the writer's next commit RESTARTS the WAL from the top and the
|
||||
* frames recycle inside the same file — so raw size says nothing about the
|
||||
* un-backfilled backlog, and a size-triggered valve degenerates into firing
|
||||
* (and pausing the writer) forever once the file passes its threshold
|
||||
* (measured: guava crawled at ~9min per 160 files). Instead the valve
|
||||
* tracks `sizeAtLastFullBackfill` — refreshed whenever a checkpoint reports
|
||||
* `log === checkpointed` (everything backfilled) — and triggers on GROWTH
|
||||
* beyond that baseline, which only happens when genuinely un-backfilled
|
||||
* frames push past the file's high-water mark.
|
||||
*
|
||||
* Backpressure: if the writer outruns the checkpointer past a hard cap of
|
||||
* growth (2× soft), {@link backpressure} pauses the writer (at a safe,
|
||||
* between-transactions boundary) until a FULL backfill lands. One in-flight
|
||||
* pass is not enough: on a disk saturated by the writer, every concurrent
|
||||
* PASSIVE pass is already stale by the time it finishes (the writer appended
|
||||
* past its snapshot), so neither SQLite's WAL wrap nor the baseline ever
|
||||
* trigger and the WAL grows without bound (measured: 5.9GB on guava at 150
|
||||
* IOPS, then a >60s read stall and a watchdog kill). With the writer parked,
|
||||
* the next pass covers everything, the WAL wraps on the following commit,
|
||||
* and the pause is the disk's honest catch-up cost — the correct terminal
|
||||
* mode when hardware genuinely can't keep up with the append rate.
|
||||
*/
|
||||
|
||||
import type { DatabaseConnection } from './index';
|
||||
|
||||
/** Soft WAL-growth threshold (MB) that triggers an off-thread passive checkpoint. */
|
||||
const DEFAULT_WAL_VALVE_MB = 256;
|
||||
/** Hard cap = this × soft threshold; past it the writer pauses for a full backfill. */
|
||||
const HARD_CAP_MULTIPLIER = 2;
|
||||
/** Passes attempted per writer pause before giving up (a pinned reader could stall forever). */
|
||||
const MAX_PAUSED_BACKFILL_PASSES = 20;
|
||||
/** How often the timer looks at the WAL file size. */
|
||||
const CHECK_INTERVAL_MS = 2000;
|
||||
|
||||
/**
|
||||
* Resolve the valve's soft threshold from the `CODEGRAPH_WAL_VALVE_MB`
|
||||
* override; non-numeric / non-positive values fall back to the default.
|
||||
*/
|
||||
export function resolveWalValveMb(envVal: string | undefined): number {
|
||||
if (envVal !== undefined && envVal !== '') {
|
||||
const n = Number(envVal);
|
||||
if (Number.isFinite(n) && n > 0) return Math.floor(n);
|
||||
}
|
||||
return DEFAULT_WAL_VALVE_MB;
|
||||
}
|
||||
|
||||
export class WalCheckpointValve {
|
||||
private timer: ReturnType<typeof setInterval> | null = null;
|
||||
private inflight: Promise<void> | null = null;
|
||||
/** Writer pause in progress (hard cap breached): passes loop until a full backfill. */
|
||||
private pause: Promise<void> | null = null;
|
||||
/**
|
||||
* WAL file size observed when a checkpoint last reported the ENTIRE WAL
|
||||
* backfilled. Growth is measured against this baseline — see the header
|
||||
* comment for why absolute size cannot be used.
|
||||
*/
|
||||
private sizeAtLastFullBackfill = 0;
|
||||
private readonly softBytes: number;
|
||||
private readonly hardBytes: number;
|
||||
|
||||
constructor(
|
||||
private readonly db: DatabaseConnection,
|
||||
softMb: number = resolveWalValveMb(process.env.CODEGRAPH_WAL_VALVE_MB),
|
||||
private readonly intervalMs: number = CHECK_INTERVAL_MS,
|
||||
private readonly log: (msg: string) => void = () => {}
|
||||
) {
|
||||
this.softBytes = softMb * 1024 * 1024;
|
||||
this.hardBytes = this.softBytes * HARD_CAP_MULTIPLIER;
|
||||
}
|
||||
|
||||
private mb(n: number): string {
|
||||
return `${Math.round(n / 1024 / 1024)}MB`;
|
||||
}
|
||||
|
||||
/** Un-backfilled growth estimate: bytes the WAL has grown past the last full backfill. */
|
||||
private growthBytes(): number {
|
||||
return this.db.getWalSizeBytes() - this.sizeAtLastFullBackfill;
|
||||
}
|
||||
|
||||
/** Begin watching the WAL. Idempotent; the timer never holds the loop open. */
|
||||
start(): void {
|
||||
if (this.timer) return;
|
||||
this.timer = setInterval(() => this.check(), this.intervalMs);
|
||||
this.timer.unref?.();
|
||||
}
|
||||
|
||||
/** Stop watching. Any in-flight checkpoint keeps running — await drain(). */
|
||||
stop(): void {
|
||||
if (this.timer) {
|
||||
clearInterval(this.timer);
|
||||
this.timer = null;
|
||||
}
|
||||
}
|
||||
|
||||
/** One poll: fire an off-thread passive checkpoint when growth passes the soft threshold. */
|
||||
check(): void {
|
||||
if (!this.pause && !this.inflight && this.growthBytes() > this.softBytes) this.fire();
|
||||
}
|
||||
|
||||
/**
|
||||
* Writer-side backstop, called at a between-transactions boundary. Returns
|
||||
* null (no wait) while growth is under the hard cap; past it, returns a
|
||||
* promise that resolves only once a FULL backfill has landed — see the
|
||||
* header comment for why a single pass is not enough on a saturated disk.
|
||||
*/
|
||||
backpressure(): Promise<void> | null {
|
||||
if (this.pause) return this.pause;
|
||||
if (this.growthBytes() <= this.hardBytes) return null;
|
||||
this.log(`backpressure: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)} — pausing writer for full backfill`);
|
||||
const t0 = Date.now();
|
||||
this.pause = this.backfillFully().finally(() => {
|
||||
this.pause = null;
|
||||
this.log(`backpressure released after ${Date.now() - t0}ms: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)}`);
|
||||
});
|
||||
return this.pause;
|
||||
}
|
||||
|
||||
/** Await any in-flight checkpoint and writer pause. */
|
||||
async drain(): Promise<void> {
|
||||
while (this.pause || this.inflight) {
|
||||
if (this.pause) await this.pause;
|
||||
if (this.inflight) await this.inflight;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Phase-boundary fold: backfill the ENTIRE WAL now (off-thread, awaited).
|
||||
* Called between bulk phases — e.g. after parsing, before resolution's
|
||||
* first reads — so the next phase never pages a bulk-write-sized WAL on
|
||||
* the main thread (the post-parse read against a multi-GB WAL is what
|
||||
* blew the #850 watchdog's 60s window in the #1231 repro). The await
|
||||
* keeps the event loop (and the watchdog heartbeat) turning.
|
||||
*/
|
||||
async foldNow(): Promise<void> {
|
||||
await this.drain();
|
||||
if (this.growthBytes() <= 0) return;
|
||||
this.log(`foldNow: wal=${this.mb(this.db.getWalSizeBytes())} baseline=${this.mb(this.sizeAtLastFullBackfill)}`);
|
||||
this.pause = this.backfillFully().finally(() => { this.pause = null; });
|
||||
await this.pause;
|
||||
}
|
||||
|
||||
/**
|
||||
* With the writer parked on the returned promise, loop passive passes until
|
||||
* one reports the entire WAL backfilled (typically the second: the first
|
||||
* drains the pass that was already running against a stale snapshot). Gives
|
||||
* up after a bounded number of passes — e.g. a reader pinning the WAL —
|
||||
* because unbounded WAL growth degrades; a wedged writer never recovers.
|
||||
*/
|
||||
private async backfillFully(): Promise<void> {
|
||||
for (let i = 0; i < MAX_PAUSED_BACKFILL_PASSES; i++) {
|
||||
if (this.inflight) await this.inflight; // fold in the stale in-flight pass first
|
||||
const res = await this.db.checkpointWalPassive();
|
||||
if (!res) return; // checkpoint machinery unavailable — don't spin
|
||||
this.log(`backfill pass ${i + 1}: busy=${res.busy} log=${res.log} checkpointed=${res.checkpointed} wal=${this.mb(this.db.getWalSizeBytes())}`);
|
||||
if (res.busy === 0 && res.log === res.checkpointed) {
|
||||
this.sizeAtLastFullBackfill = this.db.getWalSizeBytes();
|
||||
return;
|
||||
}
|
||||
}
|
||||
this.log(`backfill gave up after ${MAX_PAUSED_BACKFILL_PASSES} passes — WAL stays unbounded this cycle`);
|
||||
}
|
||||
|
||||
private fire(): void {
|
||||
const p = this.db
|
||||
.checkpointWalPassive()
|
||||
.then((res) => {
|
||||
// Full backfill (busy 0, every log frame checkpointed) ⇒ the writer's
|
||||
// next commit wraps the WAL; the file's current size becomes the new
|
||||
// growth baseline. A partial pass (writer appended during it, or a
|
||||
// read transaction pinned frames) leaves the baseline alone, so the
|
||||
// next tick fires again and copies the remainder. In non-WAL mode
|
||||
// SQLite reports log = checkpointed = -1, which is harmless here.
|
||||
if (res && res.busy === 0 && res.log === res.checkpointed) {
|
||||
this.sizeAtLastFullBackfill = this.db.getWalSizeBytes();
|
||||
}
|
||||
})
|
||||
.catch(() => { /* best-effort */ })
|
||||
.finally(() => {
|
||||
if (this.inflight === p) this.inflight = null;
|
||||
});
|
||||
this.inflight = p;
|
||||
}
|
||||
}
|
||||
+89
-44
@@ -7,6 +7,7 @@
|
||||
*/
|
||||
|
||||
import * as path from 'path';
|
||||
import * as fsp from 'fs/promises';
|
||||
import { Parser, Language as WasmLanguage } from 'web-tree-sitter';
|
||||
import { Language } from '../types';
|
||||
|
||||
@@ -248,31 +249,101 @@ export async function initGrammars(): Promise<void> {
|
||||
parserInitialized = true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Grammars that ship their own vendored WASMs under `dist/extraction/wasm/`
|
||||
* (not in tree-sitter-wasms, or the tree-sitter-wasms build is too old).
|
||||
* Lua: tree-sitter-wasms ships an ABI-13 build that corrupts the shared WASM
|
||||
* heap under web-tree-sitter 0.25 (drops nested calls/imports on every file
|
||||
* after the first); we vendor the upstream ABI-15 wasm instead. C#: the
|
||||
* tree-sitter-wasms build (ABI 13) has no primary-constructor support and
|
||||
* parses `class Foo(...)` as an ERROR that swallows the whole class (#237); we
|
||||
* vendor the upstream ABI-15 tree-sitter-c-sharp 0.23.5 wasm, which parses
|
||||
* primary constructors natively. Terraform: tree-sitter-wasms does not ship
|
||||
* HCL/Terraform at all, so we vendor the prebuilt tree-sitter-terraform.wasm
|
||||
* from @tree-sitter-grammars/tree-sitter-hcl 1.2.0 (Apache-2.0) —
|
||||
* byte-identical to the npm package's artifact. ArkTS: tree-sitter-wasms
|
||||
* doesn't ship it either; we vendor the prebuilt tree-sitter-arkts.wasm from
|
||||
* the tree-sitter-arkts 0.2.0 npm package (harmony-contrib/tree-sitter-arkts,
|
||||
* MIT) — byte-identical to the npm tarball's artifact. It extends the
|
||||
* tree-sitter-javascript grammar the same way tree-sitter-typescript does,
|
||||
* adding `struct_declaration` and the `arkui_component_expression` build()
|
||||
* DSL. Nix: tree-sitter-wasms doesn't ship it; we vendor a wasm built from
|
||||
* nix-community/tree-sitter-nix @ 3d0173d (MIT) with tree-sitter-cli 0.25.10
|
||||
* (`generate` + `build --wasm`, ABI 15 — upstream's checked-in parser.c is
|
||||
* still ABI 13; all 54 upstream corpus tests pass on the regenerated parser).
|
||||
*/
|
||||
const VENDORED_WASM_LANGS: ReadonlySet<GrammarLanguage> = new Set([
|
||||
'pascal', 'scala', 'lua', 'luau', 'csharp', 'r', 'cfml', 'cfscript', 'cfquery',
|
||||
'cobol', 'vbnet', 'erlang', 'terraform', 'arkts', 'nix',
|
||||
]);
|
||||
|
||||
/** Absolute path of a language's grammar WASM (vendored or tree-sitter-wasms). */
|
||||
function resolveWasmPath(lang: GrammarLanguage): string {
|
||||
const wasmFile = WASM_GRAMMAR_FILES[lang];
|
||||
return VENDORED_WASM_LANGS.has(lang)
|
||||
? path.join(__dirname, 'wasm', wasmFile)
|
||||
: require.resolve(`tree-sitter-wasms/out/${wasmFile}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Expand an index set's languages to the grammars actually needed to parse it.
|
||||
* SFC languages (svelte/vue/astro) have no grammar of their own — their
|
||||
* extractors delegate <script>/frontmatter content to the TS/JS extractor, so
|
||||
* those grammars must be loaded even when no plain .ts/.js file is in the index
|
||||
* set (e.g. a pure-.astro content site). CFML (.cfc/.cfm) likewise delegates
|
||||
* bare-script content, <cfscript> tag bodies, and <cfquery> SQL bodies to the
|
||||
* cfscript/cfquery grammars (see injections.scm in tree-sitter-cfml).
|
||||
*/
|
||||
function expandGrammarLanguages(languages: Language[]): Language[] {
|
||||
if (languages.some((l) => l === 'svelte' || l === 'vue' || l === 'astro')) {
|
||||
languages = [...languages, 'typescript', 'javascript'];
|
||||
}
|
||||
if (languages.some((l) => l === 'cfml')) {
|
||||
languages = [...languages, 'cfscript', 'cfquery'];
|
||||
}
|
||||
return languages;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-read the grammar WASM bytes for an index set, keyed by language. The
|
||||
* orchestrator reads each grammar ONCE and hands the bytes to every parse
|
||||
* worker via its `load-grammars` message, so worker spawns/respawns load
|
||||
* grammars from memory instead of re-reading them from disk — on slow storage
|
||||
* (HDD, issue #1231) each respawn's grammar re-read otherwise amplifies the
|
||||
* I/O contention that caused the respawn. Best-effort: a language whose WASM
|
||||
* can't be read here is simply omitted, and the worker falls back to its own
|
||||
* disk load (which surfaces the real error/warning path).
|
||||
*/
|
||||
export async function readGrammarWasmBytes(languages: Language[]): Promise<Record<string, Uint8Array>> {
|
||||
const out: Record<string, Uint8Array> = {};
|
||||
const toRead = [...new Set(expandGrammarLanguages(languages))].filter(
|
||||
(lang): lang is GrammarLanguage => lang in WASM_GRAMMAR_FILES
|
||||
);
|
||||
for (const lang of toRead) {
|
||||
try {
|
||||
out[lang] = await fsp.readFile(resolveWasmPath(lang));
|
||||
} catch {
|
||||
// fall through — the worker's own load reports the failure
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Load grammar WASM files for specific languages only.
|
||||
* Skips languages that are already loaded or have no WASM grammar.
|
||||
* Must be called after initGrammars().
|
||||
*
|
||||
* `wasmBytes` (optional) holds pre-read grammar bytes keyed by language (from
|
||||
* {@link readGrammarWasmBytes}, forwarded through the parse pool); when a
|
||||
* language's bytes are present they're loaded from memory instead of disk.
|
||||
*/
|
||||
export async function loadGrammarsForLanguages(languages: Language[]): Promise<void> {
|
||||
export async function loadGrammarsForLanguages(languages: Language[], wasmBytes?: Record<string, Uint8Array>): Promise<void> {
|
||||
if (!parserInitialized) {
|
||||
await initGrammars();
|
||||
}
|
||||
|
||||
// SFC languages (svelte/vue/astro) have no grammar of their own — their
|
||||
// extractors delegate <script>/frontmatter content to the TS/JS extractor,
|
||||
// so those grammars must be loaded even when no plain .ts/.js file is in
|
||||
// the index set (e.g. a pure-.astro content site).
|
||||
if (languages.some((l) => l === 'svelte' || l === 'vue' || l === 'astro')) {
|
||||
languages = [...languages, 'typescript', 'javascript'];
|
||||
}
|
||||
|
||||
// CFML (.cfc/.cfm) delegates bare-script content, <cfscript> tag bodies, and
|
||||
// <cfquery> SQL bodies to the cfscript/cfquery grammars (see injections.scm in
|
||||
// tree-sitter-cfml) — load both even when no standalone .cfs file is in the
|
||||
// index set.
|
||||
if (languages.some((l) => l === 'cfml')) {
|
||||
languages = [...languages, 'cfscript', 'cfquery'];
|
||||
}
|
||||
languages = expandGrammarLanguages(languages);
|
||||
|
||||
// Deduplicate and filter to languages that have WASM grammars and aren't already loaded
|
||||
const toLoad = [...new Set(languages)].filter(
|
||||
@@ -285,35 +356,9 @@ export async function loadGrammarsForLanguages(languages: Language[]): Promise<v
|
||||
// Load grammars sequentially to avoid web-tree-sitter WASM race condition on Node 20+
|
||||
// See: https://github.com/tree-sitter/tree-sitter/issues/2338
|
||||
for (const lang of toLoad) {
|
||||
const wasmFile = WASM_GRAMMAR_FILES[lang];
|
||||
try {
|
||||
// Some grammars ship their own WASMs (not in tree-sitter-wasms, or the
|
||||
// tree-sitter-wasms build is too old). Lua: tree-sitter-wasms ships an
|
||||
// ABI-13 build that corrupts the shared WASM heap under web-tree-sitter
|
||||
// 0.25 (drops nested calls/imports on every file after the first); we
|
||||
// vendor the upstream ABI-15 wasm instead. C#: the tree-sitter-wasms
|
||||
// build (ABI 13) has no primary-constructor support and parses
|
||||
// `class Foo(...)` as an ERROR that swallows the whole class (#237); we
|
||||
// vendor the upstream ABI-15 tree-sitter-c-sharp 0.23.5 wasm, which parses
|
||||
// primary constructors natively. Terraform: tree-sitter-wasms does not
|
||||
// ship HCL/Terraform at all, so we vendor the prebuilt
|
||||
// tree-sitter-terraform.wasm from @tree-sitter-grammars/tree-sitter-hcl
|
||||
// 1.2.0 (Apache-2.0) — byte-identical to the npm package's artifact.
|
||||
// ArkTS: tree-sitter-wasms doesn't ship it either; we vendor the prebuilt
|
||||
// tree-sitter-arkts.wasm from the tree-sitter-arkts 0.2.0 npm package
|
||||
// (harmony-contrib/tree-sitter-arkts, MIT) — byte-identical to the npm
|
||||
// tarball's artifact. It extends the tree-sitter-javascript grammar the
|
||||
// same way tree-sitter-typescript does, adding `struct_declaration` and
|
||||
// the `arkui_component_expression` build() DSL.
|
||||
// Nix: tree-sitter-wasms doesn't ship it; we vendor a wasm built from
|
||||
// nix-community/tree-sitter-nix @ 3d0173d (MIT) with tree-sitter-cli
|
||||
// 0.25.10 (`generate` + `build --wasm`, ABI 15 — upstream's checked-in
|
||||
// parser.c is still ABI 13; all 54 upstream corpus tests pass on the
|
||||
// regenerated parser).
|
||||
const wasmPath = (lang === 'pascal' || lang === 'scala' || lang === 'lua' || lang === 'luau' || lang === 'csharp' || lang === 'r' || lang === 'cfml' || lang === 'cfscript' || lang === 'cfquery' || lang === 'cobol' || lang === 'vbnet' || lang === 'erlang' || lang === 'terraform' || lang === 'arkts' || lang === 'nix')
|
||||
? path.join(__dirname, 'wasm', wasmFile)
|
||||
: require.resolve(`tree-sitter-wasms/out/${wasmFile}`);
|
||||
const language = await WasmLanguage.load(wasmPath);
|
||||
const bytes = wasmBytes?.[lang];
|
||||
const language = await WasmLanguage.load(bytes ?? resolveWasmPath(lang));
|
||||
languageCache.set(lang, language);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : String(error);
|
||||
|
||||
+31
-8
@@ -19,8 +19,8 @@ import {
|
||||
} from '../types';
|
||||
import { QueryBuilder } from '../db/queries';
|
||||
import { extractFromSource } from './tree-sitter';
|
||||
import { ParseWorkerPool, resolveParsePoolSize } from './parse-pool';
|
||||
import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages } from './grammars';
|
||||
import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs } from './parse-pool';
|
||||
import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes } from './grammars';
|
||||
import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns } from '../project-config';
|
||||
import { isCodeGraphDataDir } from '../directory';
|
||||
import { logDebug, logWarn } from '../errors';
|
||||
@@ -53,9 +53,10 @@ const SYNC_RECONCILE_YIELD_INTERVAL = 1000;
|
||||
/**
|
||||
* Maximum time (ms) to wait for a single file to parse in the worker thread.
|
||||
* If tree-sitter hangs or WASM runs out of memory, this prevents the entire
|
||||
* indexing run from freezing. The worker is restarted after a timeout.
|
||||
* indexing run from freezing. The worker is restarted after a (hard) timeout.
|
||||
* Env-overridable via CODEGRAPH_PARSE_TIMEOUT_MS for slow storage (#1231).
|
||||
*/
|
||||
const PARSE_TIMEOUT_MS = 10_000;
|
||||
const PARSE_TIMEOUT_MS = resolveParseTimeoutMs(process.env.CODEGRAPH_PARSE_TIMEOUT_MS);
|
||||
|
||||
/**
|
||||
* Number of files to parse before recycling the worker thread.
|
||||
@@ -1453,7 +1454,12 @@ export class ExtractionOrchestrator {
|
||||
async indexAll(
|
||||
onProgress?: (progress: IndexProgress) => void,
|
||||
signal?: AbortSignal,
|
||||
verbose?: boolean
|
||||
verbose?: boolean,
|
||||
// Writer-side backstop for deferred WAL checkpointing (#1231): returns
|
||||
// null in the normal case, or a promise to await (at this safe,
|
||||
// between-transactions boundary) when the WAL has outrun the off-thread
|
||||
// checkpointer past its hard cap. See db/wal-valve.ts.
|
||||
walBackpressure?: () => Promise<void> | null
|
||||
): Promise<IndexResult> {
|
||||
await initGrammars();
|
||||
const startTime = Date.now();
|
||||
@@ -1549,6 +1555,11 @@ export class ExtractionOrchestrator {
|
||||
// CODEGRAPH_PARSE_WORKERS: explicit worker count; 1 = the old single-worker
|
||||
// behaviour (the conservative rollback). Unset → clamp(cores-1, 1, 8).
|
||||
const poolSize = resolveParsePoolSize(process.env.CODEGRAPH_PARSE_WORKERS, os.cpus().length);
|
||||
// Read each needed grammar's WASM ONCE here and hand the bytes to every
|
||||
// worker, so spawns/respawns load grammars from memory instead of
|
||||
// re-reading them from disk (#1231: on an HDD, respawn re-reads amplify
|
||||
// the very I/O contention that caused the respawn).
|
||||
const grammarBuffers = await readGrammarWasmBytes(neededLanguages);
|
||||
pool = new ParseWorkerPool({
|
||||
languages: neededLanguages,
|
||||
size: poolSize,
|
||||
@@ -1556,6 +1567,7 @@ export class ExtractionOrchestrator {
|
||||
recycleInterval: WORKER_RECYCLE_INTERVAL,
|
||||
parseTimeoutMs: PARSE_TIMEOUT_MS,
|
||||
log,
|
||||
grammarBuffers,
|
||||
});
|
||||
log(`Parse worker pool: ${poolSize} worker(s)`);
|
||||
} else {
|
||||
@@ -1603,6 +1615,12 @@ export class ExtractionOrchestrator {
|
||||
const storeResult = async (filePath: string, content: string, stats: fs.Stats, result: ExtractionResult): Promise<void> => {
|
||||
processed++;
|
||||
|
||||
// WAL hard-cap backstop: between files (never mid-transaction), pause
|
||||
// the store until the off-thread checkpoint catches up. Resolves to
|
||||
// null in the normal case — a single size check, no cost.
|
||||
const bp = walBackpressure?.();
|
||||
if (bp) await bp;
|
||||
|
||||
// Store in database on main thread (SQLite is not thread-safe)
|
||||
if (result.nodes.length > 0 || result.errors.length === 0) {
|
||||
const language = detectLanguage(filePath, content, overrides);
|
||||
@@ -1815,14 +1833,19 @@ export class ExtractionOrchestrator {
|
||||
|
||||
// Retry pass: files that failed due to WASM memory corruption may succeed
|
||||
// on a fresh worker with a clean heap. Recycle before each attempt so
|
||||
// every file gets the absolute cleanest WASM state possible.
|
||||
// every file gets the absolute cleanest WASM state possible. Timeouts are
|
||||
// retried too (#1231): most are main-thread-stall artifacts, not slow
|
||||
// parses, and this pass parses one file at a time with the store strictly
|
||||
// after each parse resolves, so the stall window can't recur here.
|
||||
const retryableErrors = errors.filter(
|
||||
(e) => e.code === 'parse_error' && e.filePath &&
|
||||
(e.message.includes('Worker exited') || e.message.includes('memory access out of bounds'))
|
||||
(e.message.includes('Worker exited') ||
|
||||
e.message.includes('memory access out of bounds') ||
|
||||
e.message.includes('timed out'))
|
||||
);
|
||||
|
||||
if (retryableErrors.length > 0 && pool) {
|
||||
log(`Retrying ${retryableErrors.length} files that failed due to WASM memory errors...`);
|
||||
log(`Retrying ${retryableErrors.length} files that failed due to WASM memory errors or timeouts...`);
|
||||
|
||||
// Fresh WASM heaps for the retry phase. A retry that still crashes its
|
||||
// worker makes the pool respawn it, so later retries keep landing on clean
|
||||
|
||||
@@ -61,6 +61,18 @@ const MAX_PARSE_POOL_SIZE = 16;
|
||||
const DEFAULT_RECYCLE_INTERVAL = 250;
|
||||
/** Base per-parse timeout; scaled up for large files by the caller's formula. */
|
||||
const DEFAULT_PARSE_TIMEOUT_MS = 10_000;
|
||||
/**
|
||||
* A worker is only killed once a parse has gone this many × its budget with no
|
||||
* result. The base timer firing is NOT proof the parse is still running: after
|
||||
* a long synchronous main-thread stretch (the SQLite store on slow disks,
|
||||
* issue #1231) Node runs the timers phase before the poll phase, so the
|
||||
* expired timer fires BEFORE an already-delivered `parse-result` is processed.
|
||||
* Killing at the base timeout therefore produced false timeouts on parses that
|
||||
* finished instantly (even 0-byte files). Instead the base timer only marks
|
||||
* the job late; a result that arrives before this backstop is accepted, and
|
||||
* only a worker that stays silent the whole window is treated as hung.
|
||||
*/
|
||||
const HARD_KILL_MULTIPLIER = 3;
|
||||
/**
|
||||
* Max workers cold-starting at once. A worker's cold start is heavy (module load
|
||||
* + grammar WASM compile); starting the whole pool simultaneously thrashes CPU.
|
||||
@@ -84,6 +96,19 @@ const CRASH_BUDGET = 100;
|
||||
* - unset / blank / non-numeric → `clamp(cores - 1, 1, 8)` (leave a core for
|
||||
* the main thread + UI; never zero — parsing always needs a worker).
|
||||
*/
|
||||
/**
|
||||
* Resolve the base per-parse timeout from the `CODEGRAPH_PARSE_TIMEOUT_MS`
|
||||
* override. Slow storage (HDD, network folders) can need a larger budget; a
|
||||
* non-numeric / non-positive value falls back to the default (10s).
|
||||
*/
|
||||
export function resolveParseTimeoutMs(envVal: string | undefined): number {
|
||||
if (envVal !== undefined && envVal !== '') {
|
||||
const n = Number(envVal);
|
||||
if (Number.isFinite(n) && n > 0) return Math.floor(n);
|
||||
}
|
||||
return DEFAULT_PARSE_TIMEOUT_MS;
|
||||
}
|
||||
|
||||
export function resolveParsePoolSize(envVal: string | undefined, cpuCount: number): number {
|
||||
if (envVal !== undefined && envVal !== '') {
|
||||
const n = Number(envVal);
|
||||
@@ -102,6 +127,11 @@ interface ParseJob {
|
||||
reject: (e: Error) => void;
|
||||
settled: boolean;
|
||||
timer?: ReturnType<typeof setTimeout>;
|
||||
/** Full budget for this parse (base timeout + size scaling), for late-result logging. */
|
||||
budgetMs?: number;
|
||||
/** The base timer fired with no result yet — accept a late result, kill at the backstop. */
|
||||
timerExpired?: boolean;
|
||||
hardKillTimer?: ReturnType<typeof setTimeout>;
|
||||
}
|
||||
|
||||
/** Shape of a message a worker posts back (grammar-load ack or a parse result). */
|
||||
@@ -109,6 +139,8 @@ interface ParseWorkerMessage {
|
||||
type?: string;
|
||||
id?: number;
|
||||
result?: ExtractionResult;
|
||||
/** Worker-side parse duration — the worker's own clock, immune to main-thread stalls. */
|
||||
parseMs?: number;
|
||||
}
|
||||
|
||||
export interface ParseWorkerPoolOptions {
|
||||
@@ -126,6 +158,15 @@ export interface ParseWorkerPoolOptions {
|
||||
createWorker?: () => ParsePoolWorker;
|
||||
/** Optional verbose logger (the orchestrator's `[worker] …` logger). */
|
||||
log?: (msg: string) => void;
|
||||
/**
|
||||
* Pre-read grammar WASM bytes keyed by language, forwarded to every worker's
|
||||
* `load-grammars` message so a spawn/respawn loads grammars from memory
|
||||
* instead of re-reading them from disk — on slow storage each respawn's
|
||||
* grammar re-read otherwise amplifies the very I/O contention that caused
|
||||
* the respawn (issue #1231). Best-effort: a missing language falls back to
|
||||
* the worker's own disk read.
|
||||
*/
|
||||
grammarBuffers?: Record<string, Uint8Array>;
|
||||
}
|
||||
|
||||
export class ParseWorkerPool {
|
||||
@@ -147,9 +188,11 @@ export class ParseWorkerPool {
|
||||
private readonly parseTimeoutMs: number;
|
||||
private readonly createWorker: () => ParsePoolWorker;
|
||||
private readonly log: (msg: string) => void;
|
||||
private readonly grammarBuffers?: Record<string, Uint8Array>;
|
||||
|
||||
constructor(opts: ParseWorkerPoolOptions) {
|
||||
this.languages = opts.languages;
|
||||
this.grammarBuffers = opts.grammarBuffers;
|
||||
this.maxSize = Math.max(1, Math.min(opts.size, MAX_PARSE_POOL_SIZE));
|
||||
this.recycleInterval = opts.recycleInterval ?? DEFAULT_RECYCLE_INTERVAL;
|
||||
this.parseTimeoutMs = opts.parseTimeoutMs ?? DEFAULT_PARSE_TIMEOUT_MS;
|
||||
@@ -179,7 +222,7 @@ export class ParseWorkerPool {
|
||||
/**
|
||||
* Parse one file on the pool. Resolves with the extraction result, or REJECTS
|
||||
* if the parse times out or its worker crashes — the caller records the error
|
||||
* and (for worker-exit/OOM rejections) re-attempts in its retry pass.
|
||||
* and (for worker-exit/OOM/timeout rejections) re-attempts in its retry pass.
|
||||
*/
|
||||
requestParse(task: ParseTask): Promise<ExtractionResult> {
|
||||
if (this.destroyed) return Promise.reject(new Error('Parse pool destroyed'));
|
||||
@@ -205,7 +248,9 @@ export class ParseWorkerPool {
|
||||
w.on('error', (e) => this.onWorkerGone(w, `Worker error: ${e?.message ?? 'unknown'}`));
|
||||
w.on('exit', (code) => { if (code !== 0) this.onWorkerGone(w, `Worker exited with code ${code}`); });
|
||||
// Load grammars; the worker replies 'grammars-loaded' and only then is idle.
|
||||
w.postMessage({ type: 'load-grammars', languages: this.languages });
|
||||
// Pre-read WASM bytes (when the orchestrator provided them) make this a
|
||||
// memory load instead of a per-spawn disk read.
|
||||
w.postMessage({ type: 'load-grammars', languages: this.languages, grammarBuffers: this.grammarBuffers });
|
||||
}
|
||||
|
||||
private onMessage(w: ParsePoolWorker, m: ParseWorkerMessage): void {
|
||||
@@ -220,6 +265,22 @@ export class ParseWorkerPool {
|
||||
const job = this.inflight.get(w);
|
||||
if (!job || (m.id !== undefined && m.id !== job.id)) return; // stale (post-recycle)
|
||||
this.inflight.delete(w);
|
||||
if (job.timerExpired) {
|
||||
// The base timer fired before this result was processed. That almost
|
||||
// always means the MAIN THREAD was stalled (sync SQLite store on slow
|
||||
// disks) while the parse itself finished long ago — the worker's own
|
||||
// clock (parseMs) tells the two apart. Either way the result is here
|
||||
// and valid: accept it instead of the old behaviour (kill worker +
|
||||
// reject), which turned every main-thread stall into false timeouts
|
||||
// and dropped files (issue #1231).
|
||||
const parseMs = typeof m.parseMs === 'number' ? Math.round(m.parseMs) : undefined;
|
||||
const detail = parseMs === undefined
|
||||
? ''
|
||||
: parseMs < (job.budgetMs ?? this.parseTimeoutMs)
|
||||
? ` (parse took ${parseMs}ms in-worker — the main thread was stalled, not the parse)`
|
||||
: ` (parse genuinely took ${parseMs}ms)`;
|
||||
this.log(`Late parse-result accepted: ${job.task.filePath}${detail}`);
|
||||
}
|
||||
// Recycle the worker once it's done enough parses to have grown its WASM
|
||||
// heap; otherwise return it to the idle set for the next job.
|
||||
if ((this.parseCounts.get(w) ?? 0) >= this.recycleInterval) {
|
||||
@@ -269,6 +330,7 @@ export class ParseWorkerPool {
|
||||
// Scale the timeout for large files: base + 10s per 100KB (matches the
|
||||
// original single-worker formula so pathological-file behaviour is unchanged).
|
||||
const timeoutMs = this.parseTimeoutMs + Math.floor(job.task.content.length / 100_000) * 10_000;
|
||||
job.budgetMs = timeoutMs;
|
||||
job.timer = setTimeout(() => this.onTimeout(w, job, timeoutMs), timeoutMs);
|
||||
job.timer.unref?.();
|
||||
w.postMessage({
|
||||
@@ -281,16 +343,35 @@ export class ParseWorkerPool {
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* The base timer fired with no result processed yet. Do NOT kill or settle:
|
||||
* the timer firing doesn't prove the parse is still running — after a long
|
||||
* synchronous main-thread stretch Node services the timers phase before the
|
||||
* poll phase, so an already-delivered `parse-result` is still queued behind
|
||||
* this callback. Mark the job late (onMessage accepts a result that shows up)
|
||||
* and arm the hard-kill backstop for workers that are genuinely hung.
|
||||
*/
|
||||
private onTimeout(w: ParsePoolWorker, job: ParseJob, ms: number): void {
|
||||
if (job.settled || !this.workers.has(w)) return;
|
||||
this.log(`TIMEOUT: ${job.task.filePath} exceeded ${ms}ms — killing worker`);
|
||||
// Kill the (possibly WASM-wedged) worker and reject this parse. A timeout
|
||||
// isn't a crash — don't charge the budget — but the worker is gone, so spawn
|
||||
// a replacement to keep capacity.
|
||||
const graceMs = ms * (HARD_KILL_MULTIPLIER - 1);
|
||||
this.log(`TIMEOUT: ${job.task.filePath} exceeded ${ms}ms with no result — waiting up to ${graceMs}ms more for a late result before killing the worker`);
|
||||
job.timerExpired = true;
|
||||
job.hardKillTimer = setTimeout(() => this.onHardTimeout(w, job, ms * HARD_KILL_MULTIPLIER), graceMs);
|
||||
job.hardKillTimer.unref?.();
|
||||
}
|
||||
|
||||
/** No result after the full hard-kill window — the worker really is hung. */
|
||||
private onHardTimeout(w: ParsePoolWorker, job: ParseJob, totalMs: number): void {
|
||||
if (job.settled || !this.workers.has(w)) return;
|
||||
this.log(`TIMEOUT: ${job.task.filePath} got no result after ${totalMs}ms — killing worker`);
|
||||
// Kill the (WASM-wedged) worker and reject this parse. A timeout isn't a
|
||||
// crash — don't charge the budget — but the worker is gone, so spawn a
|
||||
// replacement to keep capacity. The rejection message contains "timed out"
|
||||
// so the orchestrator's retry pass re-attempts the file.
|
||||
this.removeWorker(w);
|
||||
this.inflight.delete(w);
|
||||
try { void w.terminate(); } catch { /* already gone */ }
|
||||
this.settle(job, undefined, new Error(`Parse timed out after ${ms}ms`));
|
||||
this.settle(job, undefined, new Error(`Parse timed out after ${totalMs}ms`));
|
||||
if (this.healthy) this.spawnOne();
|
||||
this.drain();
|
||||
}
|
||||
@@ -329,6 +410,7 @@ export class ParseWorkerPool {
|
||||
if (job.settled) return;
|
||||
job.settled = true;
|
||||
if (job.timer) clearTimeout(job.timer);
|
||||
if (job.hardKillTimer) clearTimeout(job.hardKillTimer);
|
||||
if (err) job.reject(err);
|
||||
else job.resolve(result!);
|
||||
}
|
||||
|
||||
@@ -55,12 +55,18 @@ import type { Language, ExtractionResult } from '../types';
|
||||
const PARSER_RESET_INTERVAL = 5000;
|
||||
const parseCounts = new Map<Language, number>();
|
||||
|
||||
parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: string; content?: string; languages?: Language[]; frameworkNames?: string[]; language?: Language }) => {
|
||||
parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: string; content?: string; languages?: Language[]; frameworkNames?: string[]; language?: Language; grammarBuffers?: Record<string, Uint8Array> }) => {
|
||||
if (msg.type === 'load-grammars') {
|
||||
await loadGrammarsForLanguages(msg.languages!);
|
||||
// Grammar WASM bytes pre-read by the main thread (when provided) make this
|
||||
// a memory load instead of a per-spawn disk read — see issue #1231.
|
||||
await loadGrammarsForLanguages(msg.languages!, msg.grammarBuffers);
|
||||
parentPort!.postMessage({ type: 'grammars-loaded' });
|
||||
} else if (msg.type === 'parse') {
|
||||
const { id, filePath, content, frameworkNames } = msg;
|
||||
// Worker-side parse clock: reported back with the result so the pool can
|
||||
// tell a genuinely slow parse from a result whose delivery was delayed by
|
||||
// a stalled main thread (issue #1231 false timeouts).
|
||||
const t0 = performance.now();
|
||||
try {
|
||||
// The main thread resolves the language (it holds the project's
|
||||
// codegraph.json extension overrides) and sends it; fall back to detection
|
||||
@@ -75,7 +81,7 @@ parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: st
|
||||
resetParser(language);
|
||||
}
|
||||
|
||||
parentPort!.postMessage({ type: 'parse-result', id, result });
|
||||
parentPort!.postMessage({ type: 'parse-result', id, result, parseMs: performance.now() - t0 });
|
||||
} catch (err) {
|
||||
const message = err instanceof Error ? err.message : String(err);
|
||||
|
||||
@@ -89,6 +95,7 @@ parentPort!.on('message', async (msg: { type: string; id?: number; filePath?: st
|
||||
parentPort!.postMessage({
|
||||
type: 'parse-result',
|
||||
id,
|
||||
parseMs: performance.now() - t0,
|
||||
result: {
|
||||
nodes: [],
|
||||
edges: [],
|
||||
|
||||
+50
-1
@@ -24,6 +24,7 @@ import {
|
||||
FindRelevantContextOptions,
|
||||
} from './types';
|
||||
import { DatabaseConnection, getDatabasePath, removeDatabaseFiles } from './db';
|
||||
import { WalCheckpointValve } from './db/wal-valve';
|
||||
import { QueryBuilder } from './db/queries';
|
||||
import {
|
||||
isInitialized,
|
||||
@@ -435,6 +436,29 @@ export class CodeGraph {
|
||||
} catch {
|
||||
return { success: false, filesIndexed: 0, filesSkipped: 0, filesErrored: 0, nodesCreated: 0, edgesCreated: 0, errors: [{ message: 'Could not acquire file lock - another process may be indexing', severity: 'error' as const }], durationMs: 0 };
|
||||
}
|
||||
// Defer WAL auto-checkpointing for the whole bulk run (#1231): the
|
||||
// default 1000-page interval re-writes hot pages into the main DB file
|
||||
// over and over — ~95% of all disk I/O during a bulk index, and a
|
||||
// 19+min → 45s difference on HDD-class storage. The valve bounds WAL
|
||||
// growth by backfilling PASSIVEly on a worker thread (never blocking
|
||||
// the writer or the #850 watchdog heartbeat); runMaintenance below does
|
||||
// the final fold-up before the interval is restored in the finally.
|
||||
// Kill switch: CODEGRAPH_NO_WAL_DEFER=1. Non-WAL journal modes (some
|
||||
// network filesystems) have no WAL to defer — skip.
|
||||
const deferWal = process.env.CODEGRAPH_NO_WAL_DEFER !== '1' && this.db.getJournalMode() === 'wal';
|
||||
let walValve: WalCheckpointValve | null = null;
|
||||
let priorAutocheckpoint = 1000;
|
||||
if (deferWal) {
|
||||
priorAutocheckpoint = this.db.getWalAutocheckpoint();
|
||||
this.db.setWalAutocheckpoint(0);
|
||||
walValve = new WalCheckpointValve(
|
||||
this.db,
|
||||
undefined,
|
||||
undefined,
|
||||
options.verbose ? (m) => console.log(`[wal-valve] ${m}`) : undefined
|
||||
);
|
||||
walValve.start();
|
||||
}
|
||||
try {
|
||||
const before = this.queries.getNodeAndEdgeCount();
|
||||
// Mark the index as in-flight BEFORE any writes: a run killed
|
||||
@@ -446,7 +470,19 @@ export class CodeGraph {
|
||||
// path as every file (re-)indexes below — so a full index is also the
|
||||
// orphan-cleanup pass for names deleted since the last one.
|
||||
try { this.queries.clearNameSegmentVocab(); } catch { /* vocab is advisory — never fail an index over it */ }
|
||||
const result = await this.orchestrator.indexAll(options.onProgress, options.signal, options.verbose);
|
||||
const result = await this.orchestrator.indexAll(
|
||||
options.onProgress,
|
||||
options.signal,
|
||||
options.verbose,
|
||||
walValve ? () => walValve!.backpressure() : undefined
|
||||
);
|
||||
|
||||
// Fold the parse phase's WAL BEFORE the first post-parse reads
|
||||
// (resolver re-init and resolution both read on the main thread):
|
||||
// paging a bulk-write-sized WAL there is what blew the #850
|
||||
// watchdog's 60s window in the #1231 repro. Off-thread + awaited,
|
||||
// so the event loop keeps turning.
|
||||
if (walValve) await walValve.foldNow();
|
||||
|
||||
// Re-detect frameworks now that the index is populated. The resolver
|
||||
// is constructed with createResolver() before any files exist, so
|
||||
@@ -501,6 +537,10 @@ export class CodeGraph {
|
||||
// successful index. Never load-bearing for correctness.
|
||||
if (result.success && result.filesIndexed > 0) {
|
||||
const tMaint = Date.now();
|
||||
// Quiesce the valve first so its in-flight checkpoint and the
|
||||
// maintenance checkpoint don't contend for the checkpointer lock
|
||||
// (the loser would silently no-op and leave the WAL unfolded).
|
||||
if (walValve) { walValve.stop(); await walValve.drain(); }
|
||||
await this.db.runMaintenance();
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) console.error(`[phase-timing] maintenance: ${Date.now() - tMaint}ms`);
|
||||
}
|
||||
@@ -561,6 +601,15 @@ export class CodeGraph {
|
||||
|
||||
return result;
|
||||
} finally {
|
||||
// Restore the auto-checkpoint interval AFTER the fold-up above so the
|
||||
// next ordinary write doesn't inherit a giant inline checkpoint. On
|
||||
// the error path the WAL may still be large; correctness is unchanged
|
||||
// (SQLite replays the WAL on the next open) and the follow-up write
|
||||
// that folds it is the known cost of a failed run.
|
||||
if (walValve) { walValve.stop(); await walValve.drain(); }
|
||||
if (deferWal) {
|
||||
try { this.db.setWalAutocheckpoint(priorAutocheckpoint); } catch { /* connection may be closing */ }
|
||||
}
|
||||
this.fileLock.release();
|
||||
}
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user