fix(db,resolution): WAL file cap + cgroup cache credit + pool/parse sizing corrections from the instrumented kernel-scale runs (#1335)
Four §7a.1 instrumented-run findings, each measured: 1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still grows the FILE without bound — the writer only restarts at frame 0 when a commit finds zero reader marks, which the instrumented run showed never happens (file marched 361→721MB through two COMPLETE backfills; 22GB by phase end). backpressure() now also trips at 4× the soft cap on raw file size and TRUNCATEs at the parked barrier; the timer path truncates opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at the same 16MB valve; dumps byte-identical under aggressive folding. 2. cgroup memory credit: memory.current counts reclaimable page cache — a post-parse container read 57MB of headroom on a 6GB box and silently disabled the pool. inactive_file is credited back (the docker-stats working-set convention); the same run now reads a sane 4.4GB budget. 3. Pool at 2 cores reversed: sequential resolution measured FASTER than pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing. cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path. 4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower (493s vs 369s) — main + store-worker don't fill the second core. Floor restores the baseline (373.5s measured). Plus the observability §7a.1 burned three 25-minute cycles for: valve armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture, pool sizing decisions (incl. the disabled path), backpressure-hook presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG. Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record runs with this build follow in the migration plan §7a.1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
8c1e821495
commit
ca88d3bd15
@@ -95,7 +95,12 @@ export class ResolverPool {
|
||||
return Math.min(n, 16);
|
||||
}
|
||||
}
|
||||
const cpuCap = Math.max(2, Math.min(opts.availableParallelism - 1, 6));
|
||||
// No floor: at ap=2 the pool LOSES to sequential outright — measured on
|
||||
// the kernel-scale 2-cpuset envelope: resolution 853s sequential vs
|
||||
// 1,150s pooled-6-on-2 (§7a.1), and synthesis is Amdahl-bound by its
|
||||
// dominant pass (cFnPtrEdges 306s of 358s) so pooling it bought nothing.
|
||||
// ap−1 < 2 ⇒ sequential is the fast path, not a fallback.
|
||||
const cpuCap = Math.min(opts.availableParallelism - 1, 6);
|
||||
const perWorker = Math.min(Math.max(opts.dbSizeBytes * 0.2, 256 * 1024 * 1024), 1.5 * 1024 * 1024 * 1024);
|
||||
const memCap = Math.floor((opts.memoryBudget * 0.7) / perWorker);
|
||||
const size = Math.min(cpuCap, memCap);
|
||||
@@ -117,18 +122,22 @@ export class ResolverPool {
|
||||
try {
|
||||
dbSizeBytes = fs.statSync(dbPath).size;
|
||||
} catch { /* fresh/missing file — the 256MB per-worker floor applies */ }
|
||||
const ap = os.availableParallelism();
|
||||
const budget = memoryBudgetBytes();
|
||||
const size = ResolverPool.resolvePoolSize({
|
||||
explicit: process.env.CODEGRAPH_RESOLVE_WORKERS,
|
||||
availableParallelism: os.availableParallelism(),
|
||||
memoryBudget: memoryBudgetBytes(),
|
||||
availableParallelism: ap,
|
||||
memoryBudget: budget,
|
||||
dbSizeBytes,
|
||||
});
|
||||
if (size === null) return null;
|
||||
// Both outcomes log under SYNTH_TIMINGS — a silent null is how §7a.1's
|
||||
// diagnostic run hid the memory-term misfire for a whole 25-minute cycle.
|
||||
if (process.env.CODEGRAPH_SYNTH_TIMINGS) {
|
||||
console.error(
|
||||
`[pool-timing] pool size=${size} (ap=${os.availableParallelism()} budget=${Math.round(memoryBudgetBytes() / 1024 / 1024)}MB db=${Math.round(dbSizeBytes / 1024 / 1024)}MB)`
|
||||
`[pool-timing] pool ${size === null ? 'disabled' : `size=${size}`} (ap=${ap} budget=${Math.round(budget / 1024 / 1024)}MB db=${Math.round(dbSizeBytes / 1024 / 1024)}MB)`
|
||||
);
|
||||
}
|
||||
if (size === null) return null;
|
||||
try {
|
||||
return new ResolverPool(workerScript, dbPath, projectRoot, size);
|
||||
} catch {
|
||||
|
||||
Reference in New Issue
Block a user