perf(resolution): worker connection recycling — WAL-depth writes-under-readers fix, superphase −11.4% at 8c (#1362)
The §7a.6 anomaly probed to its mechanism with five discriminating runs (§7a.7 table): main-thread B-tree writes triple under attached readers because READERS PIN WAL checkpoint progress — the deep WAL taxes every writer page operation (deletes 42.6s pool-off vs 118.8s pool-4 on identical hardware; an aggressive 64MB valve recovers the writes but overpays +129s in full-park folds; the v2 cache resurrection was falsified — long-tail name traffic is uncacheable at any capacity). Fix: workers close and reopen their read-only connections every 8 batches at the double-buffer's worker-idle boundary (ResolverPool.recycleWorkers + QueryBuilder.rebind + a cadence call). Reopens are sub-millisecond, resolver caches survive (only prepared statements re-prepare), and the existing checkpoints advance instead of parking. Failed recycle downgrades to sequential, same as a failed fan-out. Measured (8c pool-4, linux v7.2-rc2, cadence 25 → 8 iterated): resolution superphase 715.0 → 633.6s (−11.4%), envelope best 14.8min, recreate 59.7 → 45.3s. Byte-neutral everywhere: git dumps byte-identical old-vs-new, linux dump sha 6dd1185b reproduced (10,446,478 lines), counts 2,049,153/6,413,518, suite 2517 green. 2c unchanged by construction (no pool → no recycling). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
5955d04c97
commit
971a5a0483
@@ -90,10 +90,15 @@ them are the ORIGINAL plan and carry expectations that measurement later correct
|
||||
double-buffer WORKS (8c settle 3.6s — "core-invariant" superseded);
|
||||
2c record now 16.5min, 8c range 15.0–16.4 (n=2). Two cache
|
||||
experiments killed by measurement same-day (nameCache scaling, lazy
|
||||
candidates — §7a.6 has the numbers; code reverted). Queue now:
|
||||
**writes-under-readers probe** (8c deletes+inserts +102s under the
|
||||
pool — the biggest attributed delta) > **cFnPtr native site
|
||||
extraction** (synthesis ~230s) > backpressure byte volume > recreate.
|
||||
candidates — §7a.6 has the numbers; code reverted).
|
||||
Writes-under-readers PROBED + FIXED 2026-07-18 (§7a.7): mechanism =
|
||||
WAL read-through depth under reader pins (proven by pool-off/dose/
|
||||
valve discrimination); fix = worker connection recycling at the
|
||||
pool-idle boundary (cadence 8); superphase **715 → 633.6s (−11.4%)**,
|
||||
8c envelope best **14.8min**, byte-neutral at every gate. Queue now:
|
||||
**cFnPtr native site extraction** (synthesis ~230s) >
|
||||
continuous-shallow WAL (the remaining ~45s to the valve floor) >
|
||||
backpressure byte volume > recreate.
|
||||
- [x] **R7a. C/C++ port** — DONE 2026-07-17, same-day walker+gates after the
|
||||
survey (#1344) and grammar vendoring (#1345). One dual-language walker
|
||||
(`codegraph-kernel/src/ccpp/`), preParse HOISTED to the route point
|
||||
@@ -850,6 +855,55 @@ morning's run; report ranges, never single runs on this box.** 8c parse
|
||||
(~80s paid in-container that a 24GB target-class box gets back free) —
|
||||
container numbers UNDERSTATE the true 8-core-class target.
|
||||
|
||||
#### 7a.7 Writes-under-readers probe + fix (2026-07-18) — WAL depth named, worker connection recycling shipped
|
||||
|
||||
Five discriminating runs (all 8c pool-4 unless noted, same tree/build family;
|
||||
each ~16min), then the fix in two cadence iterations:
|
||||
|
||||
| run | deletes | insertEdges | read | backpressure | settle | superphase |
|
||||
|---|---|---|---|---|---|---|
|
||||
| pool-4 baseline | 118.8 | 55.3 | 33.9 | 121.9 | 3.6 | 715.0 |
|
||||
| pool-OFF | 42.6 | 38.3 | 32.6 | 120.1 | 108.5 (main) | 685.9 |
|
||||
| workers=2 (dose) | 58.9 | 54.0 | 23.9 | 174.7 | 10.3 | 670.0 |
|
||||
| v2 caches @8c | 108.0 | 50.3 | 31.4 | 168.0 | 3.9 | 687.3 |
|
||||
| valve 64MB | **56.5** | **27.8** | **16.8** | 297.2 | 3.8 | 739.2 |
|
||||
| **recycle c25** | 99.8 | 42.9 | 32.7 | 148.8 | 3.7 | 663.3 |
|
||||
| **recycle c8 (SHIPPED)** | 104.3 | 47.8 | 32.5 | 127.6 | 4.0 | **633.6** |
|
||||
|
||||
- **Mechanism proven: WAL read-through depth under reader pins.** Pool-off
|
||||
restores writes on identical hardware (deletes 118.8 → 42.6); the dose
|
||||
scales with reader count (knee between 2 and 4); the aggressive valve —
|
||||
which forces a shallow WAL — recovers deletes/inserts/read to their floors
|
||||
(56.5/27.8/16.8) but overpays +129s in full-park folds. Reconciliation:
|
||||
checkpoints run in every topology, but READERS PIN their progress — the
|
||||
WAL runs deep exactly when workers are attached, and deep-WAL page
|
||||
operations tax the writer everywhere (including the unattributed
|
||||
between-stage spans, recreate, and synthesis reads).
|
||||
- **v2-at-8c falsified the cache resurrection** (deletes 108.0 ≈ baseline):
|
||||
name-lookup traffic is long-tail-dominated — uncacheable at any capacity —
|
||||
so reader traffic can't be reduced by caching. The caching family is
|
||||
triple-dead (2c settle, 8c writes).
|
||||
- **The fix: worker connection recycling at the pool-idle boundary**
|
||||
(`ResolverPool.recycleWorkers` + `QueryBuilder.rebind` + a cadence call at
|
||||
the double-buffer's worker-idle point). Workers close/reopen their
|
||||
read-only connections every 8 batches (~40k refs) — reopens are
|
||||
sub-millisecond, resolver caches survive (only prepared statements
|
||||
re-prepare), and the existing checkpoints advance instead of parking.
|
||||
Cadence 25 → 8 iterated by measurement; 8 wins via diffuse gains
|
||||
(backpressure −21, recreate 59.7 → 45.3). Attributed deletes stay ~100
|
||||
(the WAL still re-deepens between recycles — the valve's 56.5 floor
|
||||
needs continuous shallowness), but the SUPERPHASE captures the true win:
|
||||
**715.0 → 633.6s (−11.4%)**; envelope best-of **14.8min at 8c**
|
||||
(890s; band across the day's runs 14.8–16.4). Byte-neutral: git dumps
|
||||
byte-identical old-vs-new, linux dump sha `6dd1185b…` reproduced, counts
|
||||
2,049,153/6,413,518 every run, suite 2517 green. 2c unaffected by
|
||||
construction (no pool → no recycling).
|
||||
- Levers after this round: cFnPtr native site extraction (~230s synthesis,
|
||||
§7a.6 ranking stands) > continuous-shallow WAL (close the remaining
|
||||
~45s gap between recycling's ~100s deletes and the valve's 56.5 floor
|
||||
without full-park folds — e.g. passive-checkpoint nudges at the recycle
|
||||
boundary) > backpressure byte volume > recreate.
|
||||
|
||||
### 7b. Arc 3 — graph richness (forensics-backed; adopt cbm's real extras, skip inflation)
|
||||
Priority order, each gated by the standard A/B + node-explosion probes:
|
||||
1. **Test→subject edges** (first-class `tests` edges at index time; we compute covering
|
||||
|
||||
Reference in New Issue
Block a user