feat(resolution): memory-aware, cgroup-honest worker-pool sizing + CODEGRAPH_RESOLVE_WORKERS (#1333)
Pool sizing used os.cpus().length, which enumerates the HOST's CPUs: inside a 2-CPU cpuset it sized 6 resolver workers (the §7a.1 false-'sequential' premise) and 8 parse workers, and at true 8-core concurrency six ~1GB workers OOM-killed a 7GB container (oom_kill=5) mid-synthesis — sizing had no memory term and no override knob. resolvePoolSize (pure, matrix-tested): explicit CODEGRAPH_RESOLVE_WORKERS override (0 disables, cap 16); CPU term max(2, min(availableParallelism-1, 6)) — cpuset-honest, floored at 2 so true 2-core boxes keep pooled synthesis's ~2×; memory term floor(budget*0.7 / clamp(0.2*dbSize, 256MB, 1.5GB)) with budget = min(freemem, cgroup v2/v1 headroom). Parse pool's core input switches to availableParallelism. Dev machines are unchanged (still 6 workers); the 8c/7GB kernel-scale container now sizes 4. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
6e52295ceb
commit
b8833fec57
@@ -1615,8 +1615,12 @@ export class ExtractionOrchestrator {
|
||||
let pool: ParseWorkerPool | null = null;
|
||||
if (useWorker) {
|
||||
// CODEGRAPH_PARSE_WORKERS: explicit worker count; 1 = the old single-worker
|
||||
// behaviour (the conservative rollback). Unset → clamp(cores-1, 1, 8).
|
||||
const poolSize = resolveParsePoolSize(process.env.CODEGRAPH_PARSE_WORKERS, os.cpus().length);
|
||||
// behaviour (the conservative rollback). Unset → clamp(cores-1, 1, 8),
|
||||
// with cores from availableParallelism — cpuset/affinity-honest, where
|
||||
// os.cpus() enumerates the host's CPUs and spawned 8 wasm workers (and
|
||||
// their grammar heaps) inside a 2-CPU container for zero extra
|
||||
// throughput (§7a.1).
|
||||
const poolSize = resolveParsePoolSize(process.env.CODEGRAPH_PARSE_WORKERS, os.availableParallelism());
|
||||
// Read each needed grammar's WASM ONCE here and hand the bytes to every
|
||||
// worker, so spawns/respawns load grammars from memory instead of
|
||||
// re-reading them from disk (#1231: on an HDD, respawn re-reads amplify
|
||||
|
||||
Reference in New Issue
Block a user