Files
codegraph/__tests__
9716fb27ae feat(extraction): parallelize indexing across a parse worker pool (#1015) (#1025)
indexAll parsed every file through a single worker thread, so a full `codegraph index` used one core no matter the machine. Add ParseWorkerPool (src/extraction/parse-pool.ts), modeled on the shipped QueryPool: indexAll now parses across clamp(cores-1,1,8) workers. CODEGRAPH_PARSE_WORKERS overrides the count; 1 reproduces the previous single-worker path exactly (the rollback).

Parses run concurrently but results commit to SQLite in file order. This matters: the post-index resolution phase selects among ambiguous same-named candidates by node DB-insertion order, so a stable commit order keeps the graph deterministic — byte-identical to the serial path — instead of drifting with parse-completion timing. A bounded reorder buffer (backpressure on dispatched-but-uncommitted count) keeps memory flat even if a file is slow at the commit cursor.

Crash/timeout of a worker rejects only that file's parse (feeding the existing retry pass) and respawns; per-worker recycle every 250 parses reclaims WASM heap. In-process fallback unchanged when the compiled worker is absent (tests).

Validated on real OSS (django +9%, redis +17%; modest and parse-fraction-dependent), graph byte-identical across worker counts, peak memory flat-to-lower since workers recycle independently — so the #320 OOM concern doesn't materialize. Adds 11 pool unit tests.

Closes #1015. Refs #320.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-27 15:32:50 -05:00
..