docs(kernel): P1 record runs — 2c 20.4min (-23%), 8c 18.3min no-OOM, WAL 14x contained; resolution measured core-invariant (#1338)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
2adc7f60c0
commit
19cf1ec75b
@@ -53,13 +53,17 @@ them are the ORIGINAL plan and carry expectations that measurement later correct
|
||||
multiline `^` anchors after `\r` — §0a traps) — fixed + CRLF fixtures pinned
|
||||
cross-platform in #1329. Every prebuild target platform is now validated.
|
||||
- [ ] **P1. Kernel-scale resolution speed** (§7a) — measurement round RUN 2026-07-17
|
||||
and it RESHAPED the arc (full record §7a.1): the "sequential at 2 CPUs"
|
||||
premise was FALSE (pool sizing is cpuset-blind; R6 already ran 6 workers),
|
||||
and both 8-core re-runs failed STRUCTURALLY before yielding a clean number —
|
||||
container OOM at 7GB (memory-blind pool sizing), WAL blowup to 22GB on a
|
||||
4.6GB DB (pool reader snapshots starve checkpoint truncation at kernel
|
||||
scale). Revised order: (1) WAL containment, (2) memory-aware pool sizing,
|
||||
(3) then the speed re-measure. Target unchanged: <10min on 8 cores.
|
||||
and it RESHAPED the arc (full record §7a.1); items (1) WAL containment and
|
||||
(2) memory-aware/cgroup-honest sizing **SHIPPED same day (#1332–#1335,
|
||||
§7a.2)** after an implementation arc whose three failed/diagnostic
|
||||
kernel-scale runs each corrected the design (WAL file ≠ WAL backlog;
|
||||
cgroup cache credit; pool net-negative at 2 cores; parse floor). The
|
||||
2c/6GB envelope already improved 26.4 → 21.6min with counts byte-exact.
|
||||
Record runs DONE (§7a.2): 2c/6GB **20.4min (R6 −23%), WAL 1.57GB (−14×)**;
|
||||
8c/7GB **18.3min, NO OOM, WAL 1.09GB** — all byte-exact. The <10min
|
||||
target is NOT met and the gap is named: resolution is CORE-INVARIANT
|
||||
(835.9s on 8 cores ≈ 812.5s sequential on 2) — the next arc profiles the
|
||||
per-ref main-thread path, then cFnPtrEdges (86% of synthesis), then R7a.
|
||||
- [ ] **R7a. C/C++ port** — biggest single-language effort; unlocks cg1212's parse
|
||||
expectation (6.2m → ~1.5–2m, 23% of that wall) + CARLA/UE/llvm-class repos;
|
||||
Metal + CUDA ride along (their blanking pre-passes stay TS-side — `preParse`
|
||||
@@ -597,6 +601,62 @@ the premise and surfaced two structural defects that now gate any speed work.
|
||||
sizing → (3) re-run the 8-core measurement (container at ≥12GB or the Mac with
|
||||
disk headroom) → then profile what remains.** The <10min-on-8-cores target stands.
|
||||
|
||||
#### 7a.2 P1 items (1)+(2) SHIPPED 2026-07-17 — the implementation arc (#1332–#1335)
|
||||
|
||||
Four PRs, each carrying its measurement; the arc took three failed/diagnostic
|
||||
kernel-scale runs to get right, and every failure taught a design fact:
|
||||
|
||||
| Run (2c/6GB unless noted) | Build | Outcome |
|
||||
|---|---|---|
|
||||
| R6 baseline | pre-P1 | 26.4min, EXIT 0; WAL unbounded (mid-run peak unmeasured); pooled 6-on-2 (cpuset-blind) |
|
||||
| run 1 | #1332 hook | **EXIT 137 (OOM)** — WAL 22.2GB, 0 of 5.4M frames ever backfilled; futile 20-pass parks amplified memory churn |
|
||||
| diagnostic | +latch/debug | EXIT 0, ~24min; pool KILLED by mis-measured 57MB cgroup budget → exposed **sequential resolution 853s vs 1,150s pooled** and cFnPtrEdges = 306s of synthesis's 358s |
|
||||
| instrumented | +sizing fixes | EXIT 0, **21.6min (R6 −18%)**; parse floor restores 373.5s; passives complete but the FILE marched 361→721MB → named the wrap-never-happens gap; peak 17.2GB |
|
||||
| record (first attempt) | #1335 | **EXIT 1: "database is locked"** — the timer-path truncate won the lock race after the recreate's multi-GB burst and stalled the writer past its 5s busy_timeout → truncate is barrier-only now (#1336). Bonus data: recreate 7.9s (vs 68–95s) once the WAL stays folded |
|
||||
| **record** | **#1336** | **EXIT 0, 20.4min (R6 −23%); WAL peak 1.57GB (−14×); counts byte-exact 2,048,664/6,405,964.** parse 354.9s · resolution 812.5s · synthesis 329.0s · recreate 57.5s · maintenance 43.5s |
|
||||
| **8-core retry (8c/7GB)** | **#1336** | **EXIT 0, NO OOM — 18.3min; WAL peak 1.09GB; pool sized 4 by the memory term (ap=8, budget 5.1GB, db 4.1GB); counts byte-exact.** parse 208.7s · resolution 835.9s · synthesis 338.7s |
|
||||
|
||||
**The 8-core verdict (the question P1 set out to ask): 18.3min vs the <10min
|
||||
target — infrastructure fixed, speed target NOT met, and the gap is now
|
||||
precisely characterized. Resolution is CORE-INVARIANT at kernel scale: 835.9s
|
||||
pooled-4-on-8 ≈ 812.5s sequential-on-2 — worker parallelism buys nothing, so
|
||||
the bottleneck is the per-ref main-thread path (admission + persist + per-ref
|
||||
resolver work), not topology. Of the 18.3min, ~14min is core-invariant
|
||||
resolution+synthesis. Next levers, in order: (a) profile the per-ref path
|
||||
inside resolution (the 812–836s floor), (b) `cFnPtrEdges` (306s, 86% of
|
||||
synthesis — parallelize/window WITHIN the pass), (c) the R7a C/C++ port
|
||||
(parse 209s → kernel-native). 4× cores currently buys only 2min end-to-end
|
||||
(20.4 → 18.3) because parse is the only core-scaling phase left.**
|
||||
|
||||
**Design facts these runs established (each now enforced in code + tests):**
|
||||
|
||||
1. **WAL backlog and WAL file are different resources.** Passive backfills bound
|
||||
the backlog; the FILE only stops growing when a commit finds zero reader
|
||||
marks — observed never in practice. Containment = backfill + **TRUNCATE at a
|
||||
parked barrier** (the one guaranteed no-reader window) + a raw file-size
|
||||
trigger at 4× the soft cap (#1334/#1335). Dubbo: 251MB → 69MB peak, dumps
|
||||
byte-identical under aggressive fold cycling.
|
||||
2. **cgroup v2 `memory.current` counts reclaimable page cache** — post-parse it
|
||||
read 57MB free on a 6GB box and silently disabled the pool. `inactive_file`
|
||||
is credited back (#1335); the same box reads 4.4GB.
|
||||
3. **The pool loses to sequential at 2 real cores** (853s vs 1,150s resolution;
|
||||
cold worker caches + serialization + time-slicing exceed the parallelism),
|
||||
and pooled synthesis is Amdahl-bound by `cFnPtrEdges` (306s of 358s) at
|
||||
kernel scale. Sizing: `min(availableParallelism − 1, 6)` + memory term +
|
||||
`CODEGRAPH_RESOLVE_WORKERS` knob (#1333/#1335); ap=2 → sequential by choice.
|
||||
4. **Parse needs ≥2 workers even on 2 cores** (1 worker = +34%; main + store
|
||||
worker don't fill the second core). Floored (#1335): 373.5s ≈ the 369s
|
||||
oversubscribed baseline, at a fraction of the memory.
|
||||
5. **Silent failure modes burned three 25-minute cycles**: give-ups were
|
||||
verbose-gated, sizing's null path logged nothing, the timer path logged
|
||||
nothing. All valve/sizing decisions now print under `CODEGRAPH_SYNTH_TIMINGS`
|
||||
/ `CODEGRAPH_WAL_VALVE_DEBUG` — the armed line answers "is it even alive"
|
||||
in one glance.
|
||||
|
||||
**New synthesis lever surfaced:** `cFnPtrEdges` is 86% of kernel-scale synthesis
|
||||
wall — parallelizing WITHIN that one pass (or windowing its scan) is worth more
|
||||
than pooling all 36 passes. Filed under the next P1 profiling round.
|
||||
|
||||
### 7b. Arc 3 — graph richness (forensics-backed; adopt cbm's real extras, skip inflation)
|
||||
Priority order, each gated by the standard A/B + node-explosion probes:
|
||||
1. **Test→subject edges** (first-class `tests` edges at index time; we compute covering
|
||||
|
||||
Reference in New Issue
Block a user