docs(kernel): P1 record runs — 2c 20.4min (-23%), 8c 18.3min no-OOM, WAL 14x contained; resolution measured core-invariant (#1338)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
2adc7f60c0
commit
19cf1ec75b
@@ -53,13 +53,17 @@ them are the ORIGINAL plan and carry expectations that measurement later correct
|
|||||||
multiline `^` anchors after `\r` — §0a traps) — fixed + CRLF fixtures pinned
|
multiline `^` anchors after `\r` — §0a traps) — fixed + CRLF fixtures pinned
|
||||||
cross-platform in #1329. Every prebuild target platform is now validated.
|
cross-platform in #1329. Every prebuild target platform is now validated.
|
||||||
- [ ] **P1. Kernel-scale resolution speed** (§7a) — measurement round RUN 2026-07-17
|
- [ ] **P1. Kernel-scale resolution speed** (§7a) — measurement round RUN 2026-07-17
|
||||||
and it RESHAPED the arc (full record §7a.1): the "sequential at 2 CPUs"
|
and it RESHAPED the arc (full record §7a.1); items (1) WAL containment and
|
||||||
premise was FALSE (pool sizing is cpuset-blind; R6 already ran 6 workers),
|
(2) memory-aware/cgroup-honest sizing **SHIPPED same day (#1332–#1335,
|
||||||
and both 8-core re-runs failed STRUCTURALLY before yielding a clean number —
|
§7a.2)** after an implementation arc whose three failed/diagnostic
|
||||||
container OOM at 7GB (memory-blind pool sizing), WAL blowup to 22GB on a
|
kernel-scale runs each corrected the design (WAL file ≠ WAL backlog;
|
||||||
4.6GB DB (pool reader snapshots starve checkpoint truncation at kernel
|
cgroup cache credit; pool net-negative at 2 cores; parse floor). The
|
||||||
scale). Revised order: (1) WAL containment, (2) memory-aware pool sizing,
|
2c/6GB envelope already improved 26.4 → 21.6min with counts byte-exact.
|
||||||
(3) then the speed re-measure. Target unchanged: <10min on 8 cores.
|
Record runs DONE (§7a.2): 2c/6GB **20.4min (R6 −23%), WAL 1.57GB (−14×)**;
|
||||||
|
8c/7GB **18.3min, NO OOM, WAL 1.09GB** — all byte-exact. The <10min
|
||||||
|
target is NOT met and the gap is named: resolution is CORE-INVARIANT
|
||||||
|
(835.9s on 8 cores ≈ 812.5s sequential on 2) — the next arc profiles the
|
||||||
|
per-ref main-thread path, then cFnPtrEdges (86% of synthesis), then R7a.
|
||||||
- [ ] **R7a. C/C++ port** — biggest single-language effort; unlocks cg1212's parse
|
- [ ] **R7a. C/C++ port** — biggest single-language effort; unlocks cg1212's parse
|
||||||
expectation (6.2m → ~1.5–2m, 23% of that wall) + CARLA/UE/llvm-class repos;
|
expectation (6.2m → ~1.5–2m, 23% of that wall) + CARLA/UE/llvm-class repos;
|
||||||
Metal + CUDA ride along (their blanking pre-passes stay TS-side — `preParse`
|
Metal + CUDA ride along (their blanking pre-passes stay TS-side — `preParse`
|
||||||
@@ -597,6 +601,62 @@ the premise and surfaced two structural defects that now gate any speed work.
|
|||||||
sizing → (3) re-run the 8-core measurement (container at ≥12GB or the Mac with
|
sizing → (3) re-run the 8-core measurement (container at ≥12GB or the Mac with
|
||||||
disk headroom) → then profile what remains.** The <10min-on-8-cores target stands.
|
disk headroom) → then profile what remains.** The <10min-on-8-cores target stands.
|
||||||
|
|
||||||
|
#### 7a.2 P1 items (1)+(2) SHIPPED 2026-07-17 — the implementation arc (#1332–#1335)
|
||||||
|
|
||||||
|
Four PRs, each carrying its measurement; the arc took three failed/diagnostic
|
||||||
|
kernel-scale runs to get right, and every failure taught a design fact:
|
||||||
|
|
||||||
|
| Run (2c/6GB unless noted) | Build | Outcome |
|
||||||
|
|---|---|---|
|
||||||
|
| R6 baseline | pre-P1 | 26.4min, EXIT 0; WAL unbounded (mid-run peak unmeasured); pooled 6-on-2 (cpuset-blind) |
|
||||||
|
| run 1 | #1332 hook | **EXIT 137 (OOM)** — WAL 22.2GB, 0 of 5.4M frames ever backfilled; futile 20-pass parks amplified memory churn |
|
||||||
|
| diagnostic | +latch/debug | EXIT 0, ~24min; pool KILLED by mis-measured 57MB cgroup budget → exposed **sequential resolution 853s vs 1,150s pooled** and cFnPtrEdges = 306s of synthesis's 358s |
|
||||||
|
| instrumented | +sizing fixes | EXIT 0, **21.6min (R6 −18%)**; parse floor restores 373.5s; passives complete but the FILE marched 361→721MB → named the wrap-never-happens gap; peak 17.2GB |
|
||||||
|
| record (first attempt) | #1335 | **EXIT 1: "database is locked"** — the timer-path truncate won the lock race after the recreate's multi-GB burst and stalled the writer past its 5s busy_timeout → truncate is barrier-only now (#1336). Bonus data: recreate 7.9s (vs 68–95s) once the WAL stays folded |
|
||||||
|
| **record** | **#1336** | **EXIT 0, 20.4min (R6 −23%); WAL peak 1.57GB (−14×); counts byte-exact 2,048,664/6,405,964.** parse 354.9s · resolution 812.5s · synthesis 329.0s · recreate 57.5s · maintenance 43.5s |
|
||||||
|
| **8-core retry (8c/7GB)** | **#1336** | **EXIT 0, NO OOM — 18.3min; WAL peak 1.09GB; pool sized 4 by the memory term (ap=8, budget 5.1GB, db 4.1GB); counts byte-exact.** parse 208.7s · resolution 835.9s · synthesis 338.7s |
|
||||||
|
|
||||||
|
**The 8-core verdict (the question P1 set out to ask): 18.3min vs the <10min
|
||||||
|
target — infrastructure fixed, speed target NOT met, and the gap is now
|
||||||
|
precisely characterized. Resolution is CORE-INVARIANT at kernel scale: 835.9s
|
||||||
|
pooled-4-on-8 ≈ 812.5s sequential-on-2 — worker parallelism buys nothing, so
|
||||||
|
the bottleneck is the per-ref main-thread path (admission + persist + per-ref
|
||||||
|
resolver work), not topology. Of the 18.3min, ~14min is core-invariant
|
||||||
|
resolution+synthesis. Next levers, in order: (a) profile the per-ref path
|
||||||
|
inside resolution (the 812–836s floor), (b) `cFnPtrEdges` (306s, 86% of
|
||||||
|
synthesis — parallelize/window WITHIN the pass), (c) the R7a C/C++ port
|
||||||
|
(parse 209s → kernel-native). 4× cores currently buys only 2min end-to-end
|
||||||
|
(20.4 → 18.3) because parse is the only core-scaling phase left.**
|
||||||
|
|
||||||
|
**Design facts these runs established (each now enforced in code + tests):**
|
||||||
|
|
||||||
|
1. **WAL backlog and WAL file are different resources.** Passive backfills bound
|
||||||
|
the backlog; the FILE only stops growing when a commit finds zero reader
|
||||||
|
marks — observed never in practice. Containment = backfill + **TRUNCATE at a
|
||||||
|
parked barrier** (the one guaranteed no-reader window) + a raw file-size
|
||||||
|
trigger at 4× the soft cap (#1334/#1335). Dubbo: 251MB → 69MB peak, dumps
|
||||||
|
byte-identical under aggressive fold cycling.
|
||||||
|
2. **cgroup v2 `memory.current` counts reclaimable page cache** — post-parse it
|
||||||
|
read 57MB free on a 6GB box and silently disabled the pool. `inactive_file`
|
||||||
|
is credited back (#1335); the same box reads 4.4GB.
|
||||||
|
3. **The pool loses to sequential at 2 real cores** (853s vs 1,150s resolution;
|
||||||
|
cold worker caches + serialization + time-slicing exceed the parallelism),
|
||||||
|
and pooled synthesis is Amdahl-bound by `cFnPtrEdges` (306s of 358s) at
|
||||||
|
kernel scale. Sizing: `min(availableParallelism − 1, 6)` + memory term +
|
||||||
|
`CODEGRAPH_RESOLVE_WORKERS` knob (#1333/#1335); ap=2 → sequential by choice.
|
||||||
|
4. **Parse needs ≥2 workers even on 2 cores** (1 worker = +34%; main + store
|
||||||
|
worker don't fill the second core). Floored (#1335): 373.5s ≈ the 369s
|
||||||
|
oversubscribed baseline, at a fraction of the memory.
|
||||||
|
5. **Silent failure modes burned three 25-minute cycles**: give-ups were
|
||||||
|
verbose-gated, sizing's null path logged nothing, the timer path logged
|
||||||
|
nothing. All valve/sizing decisions now print under `CODEGRAPH_SYNTH_TIMINGS`
|
||||||
|
/ `CODEGRAPH_WAL_VALVE_DEBUG` — the armed line answers "is it even alive"
|
||||||
|
in one glance.
|
||||||
|
|
||||||
|
**New synthesis lever surfaced:** `cFnPtrEdges` is 86% of kernel-scale synthesis
|
||||||
|
wall — parallelizing WITHIN that one pass (or windowing its scan) is worth more
|
||||||
|
than pooling all 36 passes. Filed under the next P1 profiling round.
|
||||||
|
|
||||||
### 7b. Arc 3 — graph richness (forensics-backed; adopt cbm's real extras, skip inflation)
|
### 7b. Arc 3 — graph richness (forensics-backed; adopt cbm's real extras, skip inflation)
|
||||||
Priority order, each gated by the standard A/B + node-explosion probes:
|
Priority order, each gated by the standard A/B + node-explosion probes:
|
||||||
1. **Test→subject edges** (first-class `tests` edges at index time; we compute covering
|
1. **Test→subject edges** (first-class `tests` edges at index time; we compute covering
|
||||||
|
|||||||
Reference in New Issue
Block a user