ca88d3bd15
Four §7a.1 instrumented-run findings, each measured: 1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still grows the FILE without bound — the writer only restarts at frame 0 when a commit finds zero reader marks, which the instrumented run showed never happens (file marched 361→721MB through two COMPLETE backfills; 22GB by phase end). backpressure() now also trips at 4× the soft cap on raw file size and TRUNCATEs at the parked barrier; the timer path truncates opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at the same 16MB valve; dumps byte-identical under aggressive folding. 2. cgroup memory credit: memory.current counts reclaimable page cache — a post-parse container read 57MB of headroom on a 6GB box and silently disabled the pool. inactive_file is credited back (the docker-stats working-set convention); the same run now reads a sane 4.4GB budget. 3. Pool at 2 cores reversed: sequential resolution measured FASTER than pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing. cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path. 4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower (493s vs 369s) — main + store-worker don't fill the second core. Floor restores the baseline (373.5s measured). Plus the observability §7a.1 burned three 25-minute cycles for: valve armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture, pool sizing decisions (incl. the disabled path), backpressure-hook presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG. Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record runs with this build follow in the migration plan §7a.1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
81 lines
3.3 KiB
TypeScript
81 lines
3.3 KiB
TypeScript
/**
|
||
* Resolver-pool sizing (§7a.1 P1.2): cgroup-honest CPU term + memory-aware
|
||
* cap + the CODEGRAPH_RESOLVE_WORKERS override. resolvePoolSize is pure —
|
||
* these pin the whole decision matrix, including the two failure modes the
|
||
* measurement round exposed: os.cpus() cpuset-blindness (6 workers inside a
|
||
* 2-CPU container) and memory-blind sizing (six ~1GB workers OOM-killing a
|
||
* 7GB container at true 8-core concurrency).
|
||
*/
|
||
import { describe, it, expect } from 'vitest';
|
||
import { ResolverPool } from '../src/resolution/resolver-pool';
|
||
import { cgroupMemoryAvailable, memoryBudgetBytes } from '../src/resolution/memory-budget';
|
||
|
||
const GB = 1024 * 1024 * 1024;
|
||
const MB = 1024 * 1024;
|
||
|
||
function size(opts: Partial<Parameters<typeof ResolverPool.resolvePoolSize>[0]>): number | null {
|
||
return ResolverPool.resolvePoolSize({
|
||
availableParallelism: 8,
|
||
memoryBudget: 16 * GB,
|
||
dbSizeBytes: 200 * MB,
|
||
...opts,
|
||
});
|
||
}
|
||
|
||
describe('ResolverPool.resolvePoolSize', () => {
|
||
it('big dev box: CPU-capped at the long-standing 6', () => {
|
||
expect(size({})).toBe(6);
|
||
expect(size({ availableParallelism: 11 })).toBe(6);
|
||
});
|
||
|
||
it('true 2-core box gets NO pool — sequential measured faster there (§7a.1: 853s vs 1150s)', () => {
|
||
expect(size({ availableParallelism: 2, memoryBudget: 6 * GB })).toBeNull();
|
||
expect(size({ availableParallelism: 3, memoryBudget: 6 * GB })).toBe(2);
|
||
});
|
||
|
||
it('kernel-scale DB in a 7GB container: memory term shrinks the pool below the OOM line', () => {
|
||
// 4.6GB DB → ~940MB/worker estimate; 5.5GB headroom × 0.7 ≈ 3.85GB → 4 workers.
|
||
const s = size({ availableParallelism: 8, memoryBudget: 5.5 * GB, dbSizeBytes: 4.6 * GB });
|
||
expect(s).toBe(4);
|
||
expect(s!).toBeLessThan(6);
|
||
});
|
||
|
||
it('per-worker estimate is floored (small DBs) and capped (huge DBs)', () => {
|
||
// Small DB: floor 256MB/worker — memory cap = 16GB*0.7/256MB = 43 → CPU wins.
|
||
expect(size({ dbSizeBytes: 10 * MB })).toBe(6);
|
||
// Monster DB: cap 1.5GB/worker — 16GB*0.7/1.5GB = 7 → CPU still wins at 6.
|
||
expect(size({ dbSizeBytes: 40 * GB })).toBe(6);
|
||
// Same monster DB, tight memory: 4GB*0.7/1.5GB = 1 → below 2 → no pool.
|
||
expect(size({ dbSizeBytes: 40 * GB, memoryBudget: 4 * GB })).toBeNull();
|
||
});
|
||
|
||
it('starved memory disables the pool entirely', () => {
|
||
expect(size({ memoryBudget: 512 * MB, dbSizeBytes: 4 * GB })).toBeNull();
|
||
});
|
||
|
||
it('CODEGRAPH_RESOLVE_WORKERS overrides everything: 0 disables, values clamp at 16', () => {
|
||
expect(size({ explicit: '0' })).toBeNull();
|
||
expect(size({ explicit: '3', memoryBudget: 512 * MB })).toBe(3); // override skips the memory term
|
||
expect(size({ explicit: '64' })).toBe(16);
|
||
expect(size({ explicit: 'nonsense' })).toBe(6); // unparseable → computed path
|
||
});
|
||
});
|
||
|
||
describe('memory budget helpers', () => {
|
||
it('memoryBudgetBytes is positive and finite on every platform', () => {
|
||
const b = memoryBudgetBytes();
|
||
expect(b).toBeGreaterThan(0);
|
||
expect(Number.isFinite(b)).toBe(true);
|
||
});
|
||
|
||
it('cgroupMemoryAvailable is null when uncontained (non-Linux) and never throws', () => {
|
||
const v = cgroupMemoryAvailable();
|
||
if (process.platform !== 'linux') {
|
||
expect(v).toBeNull();
|
||
} else {
|
||
// Containerized CI: either uncontained (null) or a sane byte count.
|
||
expect(v === null || (v >= 0 && Number.isFinite(v))).toBe(true);
|
||
}
|
||
});
|
||
});
|