fix(indexing): HDD-class storage — false parse timeouts, dropped files, and WAL checkpoint write-back (#1231) (#1242)

Parse timeouts are now judged by the worker's own clock: the base timer
only marks a job late (after a long synchronous store stall, Node runs the
timers phase before the poll phase, so the timer fired before an
already-delivered result was processed — killing workers over parses that
took milliseconds, even on 0-byte files); a result arriving before a 3×
hard-kill backstop is accepted, timed-out files are retried, and
CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are
pre-read once on the main thread and handed to every worker, so
spawns/respawns load grammars from memory instead of re-reading a
saturated disk.

Bulk indexing defers WAL auto-checkpointing for the whole run: the default
1000-page interval re-writes hot B-tree/FTS pages into the main DB file
over and over — ~95% of all disk I/O under throttled measurement. A
WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill
passes (never blocking the writer or the #850 watchdog heartbeat), pauses
the writer for a full backfill if the disk truly can't keep up, and folds
the WAL at the parse→resolution boundary so post-parse reads never page a
bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with
CODEGRAPH_WAL_VALVE_MB.

Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped
files (was 8); guava-scale completes in 7.6 min with a full graph where
v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

__tests__/grammar-wasm-bytes.test.ts

@@ -0,0 +1,43 @@
+/**
+ * readGrammarWasmBytes + bytes-based grammar loading (#1231, Phase 2.1).
+ *
+ * The orchestrator pre-reads each needed grammar's WASM once on the main
+ * thread and hands the bytes to every parse worker, so a worker respawn loads
+ * grammars from memory instead of re-reading them from a (possibly slow) disk.
+ * These tests pin that the byte reader resolves the same artifacts the loader
+ * would, and that web-tree-sitter genuinely accepts the bytes.
+ */
+import { describe, it, expect } from 'vitest';
+import { Parser, Language as WasmLanguage } from 'web-tree-sitter';
+import { readGrammarWasmBytes } from '../src/extraction/grammars';
+
+describe('readGrammarWasmBytes', () => {
+  it('reads bytes for a tree-sitter-wasms grammar and a vendored grammar', async () => {
+    const bytes = await readGrammarWasmBytes(['typescript', 'lua']);
+    expect(bytes.typescript).toBeInstanceOf(Uint8Array); // from tree-sitter-wasms
+    expect(bytes.typescript.byteLength).toBeGreaterThan(10_000);
+    expect(bytes.lua).toBeInstanceOf(Uint8Array); // vendored under src/extraction/wasm/
+    expect(bytes.lua.byteLength).toBeGreaterThan(10_000);
+  });
+
+  it('expands delegating languages to the grammars they need (svelte → ts/js)', async () => {
+    const bytes = await readGrammarWasmBytes(['svelte']);
+    expect(Object.keys(bytes).sort()).toEqual(['javascript', 'typescript']);
+  });
+
+  it('omits languages without a WASM grammar instead of failing', async () => {
+    const bytes = await readGrammarWasmBytes(['yaml', 'unknown']);
+    expect(Object.keys(bytes)).toEqual([]);
+  });
+
+  it('produces bytes web-tree-sitter can load into a working parser', async () => {
+    await Parser.init();
+    const bytes = await readGrammarWasmBytes(['javascript']);
+    const language = await WasmLanguage.load(bytes.javascript);
+    const parser = new Parser();
+    parser.setLanguage(language);
+    const tree = parser.parse('function hello() { return 1; }');
+    expect(tree!.rootNode.hasError).toBe(false);
+    expect(tree!.rootNode.toString()).toContain('function_declaration');
+  });
+});