feat(kernel): R7a C/C++ walker — dual-lang ccpp module, preParse hoist, 7 new blanks, c/cpp default-routed (#1346)

Parity: 0 diffs on redis/git/fmt/protobuf/ALS sweeps; full-init dumps
byte-identical on all five + linux at kernel scale (10.4M dump lines,
same sha256 both arms). Linux 2c/6GB envelope: kernel-arm 19.1min vs
wasm-arm 22.9min (parse 356s vs 435s) on a much richer graph (the new
blanks recover error-swallowed code: git 2x nodes, linux kernel/+mm/ 3x).
Deferral guard corrected by measurement (C/C++ error incidence 9-42%;
--max-deferral flag); defer-reuse memo kills the 3x re-blank/re-parse
cost deferred files paid.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-07-17 16:56:41 -05:00
committed by GitHub
co-authored by Claude Fable 5
parent 44561b6aad
commit 2d72891b59
20 changed files with 3211 additions and 80 deletions
+65 -12
View File
@@ -1,17 +1,70 @@
# C/C++ kernel port (R7a) — the bug-for-bug checklist
**Status:** survey COMPLETE; grammars VENDORED + suite-green (2026-07-17):
tree-sitter-c v0.24.2 (`b780e47`, parser.c `f2883ff9…`) + tree-sitter-cpp
v0.23.4 (`f41e1a0`, parser.c `2a35a43b…`, scanner.c `cf60387d…`), built with
ts-cli 0.25.10 from checked-in parser.c, in `src/extraction/wasm/` +
VENDORED_WASM_LANGS. The walker PR adds the SAME-version crates + kernel
grammar registry (kernel-grammar-parity then pins the alignment). Walker +
gates not started.
This is §0a-recipe step 1's output for c/cpp: every TS-side branch the walker
must mirror, with file:line anchors into the reference implementation. Read it
WITH `docs/design/rust-kernel-migration-plan.md` (§0a recipe, §5 gates).
Companion walkers to crib structure from: `codegraph-kernel/src/tsjs/` (multi-
dialect module), `java.rs`, `go.rs` (receiver QNs), `python.rs`.
**Status: COMPLETE — walker SHIPPED + gates PASSED, c/cpp DEFAULT-ROUTED
(2026-07-17).** Walker: `codegraph-kernel/src/ccpp/mod.rs` (one dual-language
module, every branch below mirrored; its header comment lists the quirks).
Grammars: tree-sitter-c v0.24.2 (`b780e47`, parser.c `f2883ff9…`) +
tree-sitter-cpp v0.23.4 (`f41e1a0`, parser.c `2a35a43b…`, scanner.c
`cf60387d…`), crates pinned `=exact` in Cargo.toml, wasm vendored from the
same tags, kernel-grammar-parity green. preParse HOISTED to the route point
(`preParsedSource` in src/extraction/kernel/index.ts — both tryKernelExtract
and the raw bulk path), so no blanking ported to Rust.
**Gate results (2026-07-17):**
- Parity sweeps — **0 diffs on every compared file**: redis 592, git 790,
fmt 42, protobuf 925, ALS-Community (UE spot-check) 40 files byte-parity.
- Full-init dump-diffs — **byte-identical** kernel-arm vs wasm-arm: redis
(131,921 dump lines), git (160,844), fmt (35,433), protobuf (780,620),
ALS (3,694).
- Torture fixtures torture.c/.cpp/.hpp + CRLF variants + Metal/CUDA
hoist-parity + defer tests in `__tests__/kernel-ccpp-parity.test.ts`; new
preParse blanks unit-tested in extraction.test.ts; full suite green with
`CODEGRAPH_KERNEL_EXPECT=1`.
- **Deferral-rate guard — CORRECTED BY MEASUREMENT** (the §4f pattern): the
<10% bar was calibrated on ts/java/py/go (00.42% parse-error incidence).
Macro-heavy C/C++ genuinely parses with errors at double-digit file rates
(final sweeps: als 9%, git 16.1%, redis 25.3%, protobuf 25.8%, fmt 42% —
fmt's template metaprogramming + `.operator[]`-in-decltype shapes are
grammar-inherent), and every erroring file defers BY POLICY. Measured with
the defer disabled (`CODEGRAPH_KERNEL_CCPP_ERROR_EXTRACT=1`, sweep-only
hatch): recovery-divergence is real (21/207 redis, 8/382 git, 9/31 fmt
erroring files extract differently across UTF-8/UTF-16), so the defer
stays. The sweep harness now takes `--max-deferral` (default 0.1; use 0.5
for c/cpp — a broken walker still trips it by deferring ~everything).
- **Seven NEW/extended preParse blanks** cut real incidence (from 32%/52%
starting points; linux `kernel/`+`mm/` subtrees 79% → 58%), each
offset-preserving, TS-side, shared by both arms, and a graph-quality win
for the wasm path itself (git 7.1k → 13.3k nodes; the linux subtrees
2.4k → 7.2k): `#ifdef __cplusplus` guard bodies, lone macro lines
(`FMT_BEGIN_NAMESPACE`, `Q_OBJECT`), C statement iterator macros
(`list_for_each_entry(…) { }` and brace-less bodies), C trailing param
attrs (`int argc UNUSED`), the curated Linux/sparse annotation list
(`__init`/`__user`/… — structural matching is impossible there: `__u32
count` is shape-identical; parameterized `__printf(1,2)` guarded out) +
`container_of`'s type-keyword argument, leading attr macros extended to
cpp, and directive-line restore (stops the older blanks corrupting
`#define` lines).
- **Defer-reuse (the linux-economics fix):** a deferred file used to pay the
pipeline three times — the worker's raw kernel try, extractFromSource's
kernel RE-try, then wasm with a third preParse. A one-slot defer memo in
the route point short-circuits the repeat kernel attempt and hands the
already-blanked source to the wasm fallback (`sourceIsPreParsed`). On
linux this + the annotation blanks took the kernel-arm parse-loop from
560s (WORSE than the 426s wasm arm) to **356s vs 435s wasm-arm (18%)**,
and the 2c/6GB envelope to **19.1 min kernel-arm vs 22.9 min wasm-arm
(17%)** with a RICHER graph (2,048,295 nodes / 6,406,933 edges; two
independent kernel-arm runs byte-same counts).
- **Linux-scale dump gate:** kernel-arm and wasm-arm full graphs are
**byte-identical**`dump-graph.mjs` over both 5.2GB DBs:
10,444,551 dump lines each, sha256 `cd4182e6…` on both. (Dumping a 2M-node
DB needs a big-heap host run — `node --max-old-space-size=16000 …
> file` then hash the FILE; the in-container 6GB heap OOMs and a straight
pipe at GB scale dies with ENOBUFS, both of which silently hash truncated
output as the empty stream.)
The sections below are §0a-recipe step 1's output — every TS-side branch the
walker mirrors, with file:line anchors (as of `705e501`). Read WITH
`docs/design/rust-kernel-migration-plan.md` (§0a recipe, §5 gates).
## Architecture decisions (already made by the plan)