a11a439002
Parse timeouts are now judged by the worker's own clock: the base timer only marks a job late (after a long synchronous store stall, Node runs the timers phase before the poll phase, so the timer fired before an already-delivered result was processed — killing workers over parses that took milliseconds, even on 0-byte files); a result arriving before a 3× hard-kill backstop is accepted, timed-out files are retried, and CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are pre-read once on the main thread and handed to every worker, so spawns/respawns load grammars from memory instead of re-reading a saturated disk. Bulk indexing defers WAL auto-checkpointing for the whole run: the default 1000-page interval re-writes hot B-tree/FTS pages into the main DB file over and over — ~95% of all disk I/O under throttled measurement. A WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill passes (never blocking the writer or the #850 watchdog heartbeat), pauses the writer for a full backfill if the disk truly can't keep up, and folds the WAL at the parse→resolution boundary so post-parse reads never page a bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with CODEGRAPH_WAL_VALVE_MB. Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped files (was 8); guava-scale completes in 7.6 min with a full graph where v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
44 lines
2.0 KiB
TypeScript
44 lines
2.0 KiB
TypeScript
/**
|
|
* readGrammarWasmBytes + bytes-based grammar loading (#1231, Phase 2.1).
|
|
*
|
|
* The orchestrator pre-reads each needed grammar's WASM once on the main
|
|
* thread and hands the bytes to every parse worker, so a worker respawn loads
|
|
* grammars from memory instead of re-reading them from a (possibly slow) disk.
|
|
* These tests pin that the byte reader resolves the same artifacts the loader
|
|
* would, and that web-tree-sitter genuinely accepts the bytes.
|
|
*/
|
|
import { describe, it, expect } from 'vitest';
|
|
import { Parser, Language as WasmLanguage } from 'web-tree-sitter';
|
|
import { readGrammarWasmBytes } from '../src/extraction/grammars';
|
|
|
|
describe('readGrammarWasmBytes', () => {
|
|
it('reads bytes for a tree-sitter-wasms grammar and a vendored grammar', async () => {
|
|
const bytes = await readGrammarWasmBytes(['typescript', 'lua']);
|
|
expect(bytes.typescript).toBeInstanceOf(Uint8Array); // from tree-sitter-wasms
|
|
expect(bytes.typescript.byteLength).toBeGreaterThan(10_000);
|
|
expect(bytes.lua).toBeInstanceOf(Uint8Array); // vendored under src/extraction/wasm/
|
|
expect(bytes.lua.byteLength).toBeGreaterThan(10_000);
|
|
});
|
|
|
|
it('expands delegating languages to the grammars they need (svelte → ts/js)', async () => {
|
|
const bytes = await readGrammarWasmBytes(['svelte']);
|
|
expect(Object.keys(bytes).sort()).toEqual(['javascript', 'typescript']);
|
|
});
|
|
|
|
it('omits languages without a WASM grammar instead of failing', async () => {
|
|
const bytes = await readGrammarWasmBytes(['yaml', 'unknown']);
|
|
expect(Object.keys(bytes)).toEqual([]);
|
|
});
|
|
|
|
it('produces bytes web-tree-sitter can load into a working parser', async () => {
|
|
await Parser.init();
|
|
const bytes = await readGrammarWasmBytes(['javascript']);
|
|
const language = await WasmLanguage.load(bytes.javascript);
|
|
const parser = new Parser();
|
|
parser.setLanguage(language);
|
|
const tree = parser.parse('function hello() { return 1; }');
|
|
expect(tree!.rootNode.hasError).toBe(false);
|
|
expect(tree!.rootNode.toString()).toContain('function_declaration');
|
|
});
|
|
});
|