feat(extraction): parallelize indexing across a parse worker pool (#1015) (#1025)

indexAll parsed every file through a single worker thread, so a full `codegraph index` used one core no matter the machine. Add ParseWorkerPool (src/extraction/parse-pool.ts), modeled on the shipped QueryPool: indexAll now parses across clamp(cores-1,1,8) workers. CODEGRAPH_PARSE_WORKERS overrides the count; 1 reproduces the previous single-worker path exactly (the rollback).

Parses run concurrently but results commit to SQLite in file order. This matters: the post-index resolution phase selects among ambiguous same-named candidates by node DB-insertion order, so a stable commit order keeps the graph deterministic — byte-identical to the serial path — instead of drifting with parse-completion timing. A bounded reorder buffer (backpressure on dispatched-but-uncommitted count) keeps memory flat even if a file is slow at the commit cursor.

Crash/timeout of a worker rejects only that file's parse (feeding the existing retry pass) and respawns; per-worker recycle every 250 parses reclaims WASM heap. In-process fallback unchanged when the compiled worker is absent (tests).

Validated on real OSS (django +9%, redis +17%; modest and parse-fraction-dependent), graph byte-identical across worker counts, peak memory flat-to-lower since workers recycle independently — so the #320 OOM concern doesn't materialize. Adds 11 pool unit tests.

Closes #1015. Refs #320.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -14,6 +14,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 - You can now exclude committed directories from the index with an `exclude` list in `codegraph.json` — even when they're git-tracked. `.gitignore` can't drop a directory git already tracks, so a vendored theme or SDK that's checked into your repo (a committed Metronic theme under `static/`, a bundled vendor library) had no supported way to be kept out — it just bloated the graph and slowed indexing. Add a root `codegraph.json` with, e.g., `{ "exclude": ["static/", "**/vendor/**"] }` and those paths are skipped on indexing, sync, and file-watching, on both git and non-git projects. Patterns are gitignore-style and matched against repo-root-relative paths. This complements the existing `includeIgnored` (its opposite — opt *in* to gitignored embedded repos). (#999)
 - CodeGraph now follows C/C++ commands that are dispatched through macro-built function-pointer tables, so the handler functions they reach are no longer dead-ends in the graph. Many C projects register a handler into a struct's function-pointer field through a macro and a generated table — redis is the classic case: every command (`getCommand`, `decrbyCommand`, …) is wired into the command struct's `proc` field by a `MAKE_CMD(…)` table that lives in a generated, `#include`-d file, then invoked as `c->cmd->proc(c)`. CodeGraph now reads those macro-built tables — including ones whose struct type is itself a macro alias, whose table sits in an `#include`-d file that is never indexed on its own, or that are wrapped in conditional compilation (`#ifdef`) and defined inline with the struct. It recognizes function-pointer fields declared through a function typedef, and follows the receiver — a chained access (`c->cmd->proc`) or an array subscript through a file-scope table (`(cmdnames[i].cmd_func)(…)`) — across field types. It also follows dispatch through a bare array of function pointers with no struct wrapper at all — the opcode/handler-table pattern common in interpreters and emulators, where a table like `opcodes[op](…)` invokes one of many registered handler functions by index — linking the dispatcher to every handler in the array. The upshot: asking for the callers or blast radius of a command handler now finds the dispatcher that reaches it. For redis, `call` shows up as a caller of every command; for SQLite, the builtin SQL functions registered through `FUNCTION(...)` link to where they're invoked; for Vim, every `:ex` and normal-mode command links from the dispatcher. (#991, extending #932)
 - CodeGraph no longer times out when many agents query it at once. The shared background server that serves all your editor and agent sessions used to run every query on a single thread, so a burst of concurrent requests — for example a swarm of subagents exploring a large monorepo together — queued up behind one another and, while the heavy ones ran, froze the connection so finished answers couldn't even be sent back until the whole batch drained. Past a handful of simultaneous callers that routinely surfaced as MCP request timeouts. The shared server now answers queries across a pool of worker threads, so concurrent requests run in parallel and the connection stays responsive the whole time; when it's genuinely saturated a call returns a brief "busy, retry shortly" note (not an error) instead of hanging past your client's timeout. The pool sizes itself to your machine — roughly one worker per core, leaving one for coordination — and a single editor session is unaffected (no pool, no overhead). Set `CODEGRAPH_QUERY_POOL_SIZE` to choose a specific number of workers, or `0` to revert to single-threaded in-process queries.
+- Indexing now parses files across multiple CPU cores instead of one, so building a project's graph — `codegraph index`, the first index of a project, and the background re-index after changes — is faster on multi-core machines, most noticeably on large or parse-heavy codebases. The graph it produces is identical to before and re-indexing stays deterministic: parsing runs in parallel, but results are still committed in a fixed order, so the same project always yields the same graph. CodeGraph sizes the pool to your machine automatically (leaving a core free for everything else); set `CODEGRAPH_PARSE_WORKERS` to choose a specific number of parse workers, or `CODEGRAPH_PARSE_WORKERS=1` to restore the previous single-core behavior. Peak memory is unchanged — workers reclaim parser memory independently, so it doesn't grow with the number of cores. (#1015, #320)
 - When CodeGraph's MCP server runs with no default project of its own — started outside any repository (for example behind an MCP gateway), or at a monorepo root whose indexes live in sub-projects — it now marks `projectPath` as a required argument on every tool call. Before, `projectPath` was always optional, so an agent talking to such a server would often omit it, get back guidance to pass it, and not reliably retry — you had to nudge it by hand every time. Now the requirement is part of the tool definition the agent sees, so it supplies the path to the project it's working on the first time. When the server does have a default project — the normal case, launched inside your repo — `projectPath` stays optional and a call without it falls back to that project exactly as before. Thanks @wauxhall for the report. (#993)
 
 ### Fixes