sync() armed the WAL valve but never called backpressure(), so daemon
catch-up could grow the WAL without bound while query-pool readers pinned
frames. Wire the writer pause into sync store + batched resolution, and
abort with WalValveAbortError after parked backfills fail past the
documented hard/file caps instead of disabling parking for 60s.
Co-authored-by: Colby McHenry <colbymchenry@users.noreply.github.com>
The §7a.2 per-ref profile overturned the assumption the whole arc was
built on: resolveOne owns only ~93s of the kernel-scale ~433s batch loop.
Loop-stage attribution (CODEGRAPH_RESOLVE_PROFILE, shipped here) named the
rest: backpressure folds 111.2s, count guard 93.9s, batch reads 54.6s,
deletes/inserts/marks ~84s, settle 85.7s.
- Non-progress guard O(remaining)→O(1): the per-batch COUNT(*) walked every
remaining pending row (O(N²/batch) per run, 93.9s). The cleanup queries
now return summed SQLite , and zero-removals-from-claimed-work
is the guard signal — the DIRECT evidence the count diff inferred (a
mismatched-name resolver makes keyed cleanup no-op ⇒ changes=0). A real
COUNT runs only on that suspicious path and arbitrates exactly as before.
- Batch reads OFFSET→keyset (54.6s→O(batch)): OFFSET re-walked the
accumulated failed-row prefix every read; seeking past the last-seen
rowid is prefix-independent and enumeration-order identical.
- WAL valve caps scale with DB size (env still wins): every fold re-writes
hot pages (#1231 in bounded form — 111.2s at the flat 256MB cap);
soft=clamp(dbSize/4, 256MB, 2GB) trades ~4× fewer folds for a transient
WAL ≈ project size.
- CODEGRAPH_RESOLVE_PROFILE: per-outcome resolveOne histogram + loop-stage
attribution, main + workers, off by default.
Gates: dubbo dump byte-identical; suite 2,491 passed / 4 skipped (kernel
required). Kernel-scale payoff run lands in the plan doc next.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Four §7a.1 instrumented-run findings, each measured:
1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still
grows the FILE without bound — the writer only restarts at frame 0 when a
commit finds zero reader marks, which the instrumented run showed never
happens (file marched 361→721MB through two COMPLETE backfills; 22GB by
phase end). backpressure() now also trips at 4× the soft cap on raw file
size and TRUNCATEs at the parked barrier; the timer path truncates
opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at
the same 16MB valve; dumps byte-identical under aggressive folding.
2. cgroup memory credit: memory.current counts reclaimable page cache — a
post-parse container read 57MB of headroom on a 6GB box and silently
disabled the pool. inactive_file is credited back (the docker-stats
working-set convention); the same run now reads a sane 4.4GB budget.
3. Pool at 2 cores reversed: sequential resolution measured FASTER than
pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is
Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing.
cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path.
4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower
(493s vs 369s) — main + store-worker don't fill the second core. Floor
restores the baseline (373.5s measured).
Plus the observability §7a.1 burned three 25-minute cycles for: valve
armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture,
pool sizing decisions (incl. the disabled path), backpressure-hook
presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG.
Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record
runs with this build follow in the migration plan §7a.1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three §7a.1 run-1 lessons (kernel-scale 2c/6GB: EXIT=137, WAL 22.2GB with
the backpressure hook DEPLOYED):
1. TRUNCATE at parked barriers: a completed passive backfill bounds the
un-checkpointed backlog but the FILE only stops growing when a commit
finds zero readers holding WAL marks — rare while pool workers cycle
(dubbo debug baseline: file climbed monotonically through six completed
pass-1 backfills). At a parked barrier the no-reader window is
guaranteed, so chop the file there with wal_checkpoint(TRUNCATE)
(off-thread, 2s busy_timeout — a racing reader degrades it to a no-op).
2. Futility latch: when backfill gives up (pinned reader), parking again at
every over-cap boundary burns a 20-pass checkpoint attempt — each a
worker thread + fresh connection against a multi-GB DB — per batch. Two
consecutive give-ups now disable parking for 60s; a pinned phase degrades
to pre-valve behavior instead of OOM-amplifying.
3. CODEGRAPH_WAL_VALVE_DEBUG=1 surfaces valve decisions without the
caller's verbose plumbing, and give-up lines print under
CODEGRAPH_SYNTH_TIMINGS — run 1 failed silently because give-ups were
verbose-gated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
At kernel scale the pooled resolution/synthesis superphase grew a 22GB WAL
on a 4.6GB DB (cg1212, §7a.1): autocheckpointing is deferred for the run,
and the valve's timer-driven passive checkpoints stay perpetually partial
against the pool's continuous reads — no mechanism ever completed a
backfill, so the WAL accreted the whole phase's write volume, blowing disk
and feeding page-cache pressure into the 8-core/7GB container OOM.
The valve's writer-side backpressure() hard-cap backstop existed but was
wired only into the PARSE orchestrator. Thread it into the resolution batch
loop at the double-buffer's one pool-idle boundary (batch settled, next not
yet fanned out), after the edge-index recreate, and through the synthesis
insert loops. Parked there, the backfill completes; readers re-enter at
SQLite's backfilled mark and the next persist commit wraps the WAL.
Dubbo validation, same build: valve@16MB peak WAL 251MB (floor = the
single-transaction edge-index recreate) vs defaults 914MB; dumps
byte-identical (441,270 rows); wall unchanged (11s). Suite 2,479 green.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The #1242 fix (WAL deferral + checkpoint valve, the 26x win on
HDD-class storage) was wired only into indexAll. CodeGraph.sync never
touched wal_autocheckpoint, so every incremental run kept the default
1000-page cadence and re-triggered the #1231 per-page checkpoint
thrash — a 7-file sync took 2m 2s at 0-2% CPU on the reporter's
hardware, because the cost scales with the EXISTING database's hot
pages, not the change size.
sync now mirrors indexAll exactly: defer autocheckpoint + start the
valve for the run, fold the store phase's WAL before the post-store
reads, restore the interval in the finally. Same kill switch
(CODEGRAPH_NO_WAL_DEFER=1). Idle valve cost is one timer, so
watcher-frequency syncs stay cheap.
Fixes#1248
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parse timeouts are now judged by the worker's own clock: the base timer
only marks a job late (after a long synchronous store stall, Node runs the
timers phase before the poll phase, so the timer fired before an
already-delivered result was processed — killing workers over parses that
took milliseconds, even on 0-byte files); a result arriving before a 3×
hard-kill backstop is accepted, timed-out files are retried, and
CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are
pre-read once on the main thread and handed to every worker, so
spawns/respawns load grammars from memory instead of re-reading a
saturated disk.
Bulk indexing defers WAL auto-checkpointing for the whole run: the default
1000-page interval re-writes hot B-tree/FTS pages into the main DB file
over and over — ~95% of all disk I/O under throttled measurement. A
WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill
passes (never blocking the writer or the #850 watchdog heartbeat), pauses
the writer for a full backfill if the disk truly can't keep up, and folds
the WAL at the parse→resolution boundary so post-parse reads never page a
bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with
CODEGRAPH_WAL_VALVE_MB.
Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped
files (was 8); guava-scale completes in 7.6 min with a full graph where
v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
