feat(resolution): memory-aware, cgroup-honest worker-pool sizing + CODEGRAPH_RESOLVE_WORKERS (#1333)

Pool sizing used os.cpus().length, which enumerates the HOST's CPUs: inside
a 2-CPU cpuset it sized 6 resolver workers (the §7a.1 false-'sequential'
premise) and 8 parse workers, and at true 8-core concurrency six ~1GB
workers OOM-killed a 7GB container (oom_kill=5) mid-synthesis — sizing had
no memory term and no override knob.

resolvePoolSize (pure, matrix-tested): explicit CODEGRAPH_RESOLVE_WORKERS
override (0 disables, cap 16); CPU term max(2, min(availableParallelism-1,
6)) — cpuset-honest, floored at 2 so true 2-core boxes keep pooled
synthesis's ~2×; memory term floor(budget*0.7 / clamp(0.2*dbSize, 256MB,
1.5GB)) with budget = min(freemem, cgroup v2/v1 headroom). Parse pool's
core input switches to availableParallelism. Dev machines are unchanged
(still 6 workers); the 8c/7GB kernel-scale container now sizes 4.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-17 07:54:31 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 6e52295ceb
commit b8833fec57
5 changed files with 202 additions and 5 deletions
+6 -2
View File
@@ -1615,8 +1615,12 @@ export class ExtractionOrchestrator {
let pool: ParseWorkerPool | null = null;
if (useWorker) {
// CODEGRAPH_PARSE_WORKERS: explicit worker count; 1 = the old single-worker
// behaviour (the conservative rollback). Unset → clamp(cores-1, 1, 8).
const poolSize = resolveParsePoolSize(process.env.CODEGRAPH_PARSE_WORKERS, os.cpus().length);
// behaviour (the conservative rollback). Unset → clamp(cores-1, 1, 8),
// with cores from availableParallelism — cpuset/affinity-honest, where
// os.cpus() enumerates the host's CPUs and spawned 8 wasm workers (and
// their grammar heaps) inside a 2-CPU container for zero extra
// throughput (§7a.1).
const poolSize = resolveParsePoolSize(process.env.CODEGRAPH_PARSE_WORKERS, os.availableParallelism());
// Read each needed grammar's WASM ONCE here and hand the bytes to every
// worker, so spawns/respawns load grammars from memory instead of
// re-reading them from disk (#1231: on an HDD, respawn re-reads amplify