fix(db,resolution): WAL file cap + cgroup cache credit + pool/parse sizing corrections from the instrumented kernel-scale runs (#1335)
Four §7a.1 instrumented-run findings, each measured: 1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still grows the FILE without bound — the writer only restarts at frame 0 when a commit finds zero reader marks, which the instrumented run showed never happens (file marched 361→721MB through two COMPLETE backfills; 22GB by phase end). backpressure() now also trips at 4× the soft cap on raw file size and TRUNCATEs at the parked barrier; the timer path truncates opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at the same 16MB valve; dumps byte-identical under aggressive folding. 2. cgroup memory credit: memory.current counts reclaimable page cache — a post-parse container read 57MB of headroom on a 6GB box and silently disabled the pool. inactive_file is credited back (the docker-stats working-set convention); the same run now reads a sane 4.4GB budget. 3. Pool at 2 cores reversed: sequential resolution measured FASTER than pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing. cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path. 4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower (493s vs 369s) — main + store-worker don't fill the second core. Floor restores the baseline (373.5s measured). Plus the observability §7a.1 burned three 25-minute cycles for: valve armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture, pool sizing decisions (incl. the disabled path), backpressure-hook presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG. Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record runs with this build follow in the migration plan §7a.1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
8c1e821495
commit
ca88d3bd15
@@ -28,8 +28,9 @@ describe('ResolverPool.resolvePoolSize', () => {
|
||||
expect(size({ availableParallelism: 11 })).toBe(6);
|
||||
});
|
||||
|
||||
it('true 2-core box keeps a 2-worker pool (the pooled-synthesis 2×)', () => {
|
||||
expect(size({ availableParallelism: 2, memoryBudget: 6 * GB })).toBe(2);
|
||||
it('true 2-core box gets NO pool — sequential measured faster there (§7a.1: 853s vs 1150s)', () => {
|
||||
expect(size({ availableParallelism: 2, memoryBudget: 6 * GB })).toBeNull();
|
||||
expect(size({ availableParallelism: 3, memoryBudget: 6 * GB })).toBe(2);
|
||||
});
|
||||
|
||||
it('kernel-scale DB in a 7GB container: memory term shrinks the pool below the OOM line', () => {
|
||||
|
||||
@@ -103,21 +103,26 @@ describe('WalCheckpointValve', () => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('advances its baseline on a full backfill — a wrapped WAL does not retrigger it', async () => {
|
||||
it('advances its baseline on a full backfill — no infinite retrigger (at most one truncate park)', async () => {
|
||||
// Pre-§7a.1 contract was "a wrapped WAL never retriggers"; the file-size
|
||||
// trigger deliberately weakens that to "retriggers AT MOST once more, to
|
||||
// truncate the file, then goes quiet" — the pre-fix bug this test pinned
|
||||
// (firing on raw size forever, serializing every store) stays dead: a
|
||||
// successful truncate zeroes the file, so the trigger cannot loop. At
|
||||
// this test's pathological 10-BYTE soft cap, byte-level residue can trip
|
||||
// the 4×-soft file cap once; product-scale caps are 256MB/1GB.
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
writeRows(db, 500);
|
||||
const valve = new WalCheckpointValve(db, 0.00001);
|
||||
valve.check();
|
||||
await valve.drain(); // full backfill on an idle DB → baseline = current file size
|
||||
// The WAL file keeps its high-water size, but growth is now 0: neither
|
||||
// the timer path nor backpressure may fire again (the pre-fix bug fired
|
||||
// on raw size forever and serialized every store behind a checkpoint).
|
||||
expect(valve.backpressure()).toBeNull();
|
||||
await valve.drain(); // full backfill (and possibly a timer truncate)
|
||||
const first = valve.backpressure();
|
||||
if (first) await first; // one truncate park allowed — file must be 0 after
|
||||
expect(db.getWalSizeBytes()).toBe(0);
|
||||
expect(valve.backpressure()).toBeNull(); // and now: quiet
|
||||
valve.check();
|
||||
await valve.drain(); // no-op drain: nothing in flight
|
||||
// New commits recycle wrapped frames — file size is flat, still no trigger.
|
||||
writeRows(db, 5);
|
||||
await valve.drain();
|
||||
expect(valve.backpressure()).toBeNull();
|
||||
db.close();
|
||||
});
|
||||
@@ -374,3 +379,40 @@ describe('checkpointWalTruncate (§7a.1 file containment)', () => {
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
|
||||
describe('valve file-size trigger (§7a.1: backfilled WAL still grows the file)', () => {
|
||||
it('backpressure trips on file size alone once past the file cap, even with zero backlog', async () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
// Grow the file well past a 0.5MB soft cap (file cap = 4× = 2MB), then
|
||||
// fold the backlog completely so growth-vs-baseline is ~zero.
|
||||
writeRows(db, 800);
|
||||
const valve = new WalCheckpointValve(db, 0.5);
|
||||
await valve.foldNow(); // baseline := file size; backlog now 0; file unchanged
|
||||
expect(db.getWalSizeBytes()).toBe(0); // foldNow's success path truncates at the barrier
|
||||
db.close();
|
||||
});
|
||||
|
||||
it('a fully-backfilled but oversized file is chopped at the barrier', async () => {
|
||||
const db = openDb();
|
||||
db.setWalAutocheckpoint(0);
|
||||
writeRows(db, 800);
|
||||
const before = db.getWalSizeBytes();
|
||||
expect(before).toBeGreaterThan(2 * 1024 * 1024);
|
||||
const valve = new WalCheckpointValve(db, 0.5);
|
||||
const bp = valve.backpressure(); // growth past hard cap → parks
|
||||
expect(bp).not.toBeNull();
|
||||
await bp;
|
||||
expect(db.getWalSizeBytes()).toBe(0); // truncated at the parked barrier
|
||||
// And the file-size trigger alone re-arms it after regrowth:
|
||||
writeRows(db, 800);
|
||||
await valve.foldNow();
|
||||
writeRows(db, 100); // small backlog, file grows again but under hard cap
|
||||
const sizeTrigger = valve.backpressure();
|
||||
// 100 rows ≈ <1MB backlog (under 1MB hard cap) but file is past the 2MB cap
|
||||
expect(sizeTrigger).not.toBeNull();
|
||||
await sizeTrigger;
|
||||
expect(db.getWalSizeBytes()).toBe(0);
|
||||
db.close();
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user