The kong/tokio matcher-chain residue attributed (nm:mc-* sub-stage rows,
shipped here too): matchMethodCall's cost is ~entirely
inferLocalReceiverType — 61µs per miss on kong, 99% miss rate (39k `self:`
calls hunting a local declaration Lua never writes), re-scanning the same
scope lines for every ref.
Two pure memos, both semantics-preserving by construction:
- Compiled-pattern memo: localReceiverTypePatterns/phpPropertyTypePatterns
built 2-4 fresh RegExp objects per call; patterns are a pure function of
(language, receiver) and non-global, so instances are shared via a
FIFO-capped map (no per-get mutation — the §7a.6 LRU-churn lesson).
- Incremental scan memo: refs for the same (file, scope, receiver) arrive
in ~ascending line order and the backward declaration scan is a pure
function of immutable file lines — a per-context watermark scans each
line once per key (query(c) = highest match in [start..c]; monotonic
calls extend the watermark over (hi..c]; non-monotonic calls fall back
to the plain bounded scan). componentScoped (CFML/PHP whole-file sweep)
is keyed out. States drop with the context's file caches via
clearNameMatcherMemos, wired into ReferenceResolver.clearCaches.
kong mc-infer misses 61→20µs (2.4s→0.8s combined); fresh index 3.43 →
3.03-3.20s (−8%; 4.07 → 3.14 cumulative with #1391). tokio unchanged
(tight scopes). Gates: dubbo (49k Java instance-method HITS ride this
scan), kong, tokio, Fusion dumps all byte-identical; suite 2,689 ×2 with
CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The full-README competitor matrix ranked lua/kong as the largest
legitimate fresh-index gap (2.71×). Stage attribution (RESOLVE_PROFILE=2)
pinned it: resolveLuaRequire ran getAllFiles().filter(endsWith) FOUR
times per require ref — ~7.5k string suffix scans each, measured at
~0.9ms/ref, hit or miss (2.7s combined over kong's 3k requires).
Replace the per-ref full-list scans with a per-context basename →
file-paths index (the cobolCopybookIndexes pattern). Buckets preserve
getAllFiles() iteration order, so each suffix's candidate filter yields
exactly the array the full scan produced — identical matches, identical
stable sort, identical winner, dump-proven.
kong: 4.07-4.27 → 3.40-3.63s (−16%). Gates: kong old-vs-new dump
byte-identical (157,650 rows), kong pooled-vs-sequential identical,
Fusion (luau instance-path requires) old-vs-new identical; suite
2,689 ×2 with CODEGRAPH_KERNEL_EXPECT=1.
Also registers cobolCopybookIndexes in clearImportResolverMemos — it was
never dropped on cache clears, so post-sync copybook lookups could serve
a stale file list; cache-drop is the always-safe direction.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The 9-language competitor matrix exposed tokio as the worst fresh-index
gap: 77% of its wall was resolution running SEQUENTIALLY — 56k Rust refs
sit under the fixed 150k pool gate while costing 36µs each (9× Go's
4µs/ref on prometheus). A ref-count gate can't see per-ref cost.
After each sequential batch the loop now projects the remaining
sequential settle from the measured rate and boots the pool mid-run when
it clears 400ms. The switch rides machinery that already existed: pool
boot is async and fan-out engages only when ready, admission order is
mode-independent, and the #1320 edges-before-fanout invariant holds at
every batch boundary regardless of when the pool arrives. Up-front
engagement at >=150k refs is unchanged; 2-core/low-memory hosts still
decline inside tryCreate's sizing; CODEGRAPH_NO_PARALLEL_RESOLVE still
disables; downgrade permanence is preserved (one engage attempt per run).
Measured (n=3, interleaved, caffeinated): tokio 3.06-3.12 → 2.40-2.57s
(resolution 2,443→~1,330ms); express (tiny control) unchanged with zero
engagements; dubbo unchanged (ref-count path). Gates: tokio + excalidraw
adaptive-vs-sequential dumps byte-identical (87,302 / 89,903 rows),
dubbo dump identical to the session baseline, suite 2,689 ×2 with
CODEGRAPH_KERNEL_EXPECT=1. Known follow-up: sampling the rate mid-first-
batch would close the remaining ~0.3s to the forced-engage ceiling.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Store-arc round 2 (#1388 follow-up). The synthesis pool barrier on dubbo
carried ~1.4s of passes that provably could not emit an edge for the
project: reactRenderEdges fanned out over every class before checking for
a render method (now: one indexed name lookup bounds candidates — not a
language gate, Java Litho-style render+setState still matches);
expo/rn cross-platform pairing streamed every method row without the
languages their edges require (now registry-gated: expo needs swift AND
kotlin file-languages, rn needs a JS-family caller for isBridge);
mybatis built its full java-method index before discovering there were no
mapper-XML methods (now collects the XML side first). ifaceEdges — real
work — stops re-fetching a hub interface's methods once per implementer
and skips supertype-less classes before any per-class lookup.
dubbo warm wall 8.49-8.79 → 8.14-8.24s (n=3/arm, caffeinated); barrier
784→435ms; the full removed pass work lands on low-core envelopes where
synthesis runs sequentially. Dumps byte-identical: dubbo old-vs-new,
pooled-vs-sequential, kernel-vs-wasm (441,270 rows) + excalidraw JSX-live
control (89,903 rows, 46 react-render edges reproduced). Suite 2,689 ×2
with CODEGRAPH_KERNEL_EXPECT=1.
Also ships the diagnostics that located the round (zero cost when off):
CODEGRAPH_RESOLVE_PROFILE=2 attributes per-ref time to resolveOne's
strategies (stage:*) and the name-matcher's sub-matchers (nm:*);
CODEGRAPH_SYNTH_TIMINGS now prints the store worker's decode-vs-SQL
split. Killed by measurement, recorded in the PR: import-failure negative
cache (both-outcome names exist — static imports resolve via
instance-method on jvm-miss), jvm-miss early return (1,939 later-strategy
edges), jsxEdges language gate (Java generics text produces jsx edges),
and §4d buffer→bind on Spring repos (extract() hook forces the decoded
path — kernel=0 bundles measured).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Post-R7b store-arc round 1, found by the dubbo warm-wall decomposition
(the cbm bar): resolution's loop-stage profile showed settle=3.0s — the
main thread idling on TWO resolver workers on an 11-core Mac. Pool sizing
logged `size=2 (budget=1068MB)`: memoryBudgetBytes() falls back to
os.freemem() when uncontained, and macOS keeps RAM deliberately full of
reclaimable cache, so freemem reads ~1GB on a mostly-idle 64GB machine.
The memory term then capped the pool at 2 where the CPU term allowed 6 —
the macOS sibling of §7a.1's os.cpus() cpuset-blindness (that round fixed
the CPU term; this fixes the memory term).
Fix: darwinMemoryAvailable() reads /usr/bin/vm_stat once per sizing call
and reports free + inactive + speculative + purgeable pages — what
Activity Monitor calls available, the same reclaimable-inclusive
convention the Linux branch already uses by crediting inactive_file back.
Parse failure → null → freemem fallback; Linux/cgroup and Windows paths
untouched.
Measured (dubbo 4,402 files, warm, caffeinated, n=3 each): pool now
self-sizes to 6 (budget 5.7-6.3GB) — wall 8.62-8.83s vs 9.67-10.87s
baseline, resolution phase 6.9→5.3s, loop settle 3.0→1.9s. Matches the
CODEGRAPH_RESOLVE_WORKERS=6 probe exactly (probe-before-build). Dumps
byte-identical pool-6 vs sequential (441,270 lines). Second consumer
unblocked: the cFnPtr LRU cache cap no longer spuriously degrades to 128
on Macs (its full-cache tier is worth ~60s at kernel scale).
Suite: resolver-pool-sizing gains a darwin-gated reclaimable-pages test +
an off-darwin null pin; full suite 2,689 green ×2 with
CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Flips the migration plan's R7b milestone to done: eleven languages across
four same-day batches (rust #1371; csharp/ruby/php #1378-#1380; swift/
kotlin #1381-#1382; r/lua+luau/scala/dart #1383-#1386), batch 4 going
4-for-4 first-run parity (12-of-13 arc-wide). Also records the batch-4
upfront grammar-probe method and the dart wasm byte-copy vendor.
Validation note: the full suite (2,688 tests) also ran green on linux-arm64
in a fresh rust:1-bookworm + node 22 container with the kernel built from
scratch and CODEGRAPH_KERNEL_EXPECT=1 — the Linux leg for all 11
post-R7a walkers.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R7b batch 4 #4 — the FINAL R7b language (docs/design/dart-kernel-port-checklist.md
is the authoritative quirk list). The fourth vendored-grammar-C language,
with a twist: production dart resolved its wasm from tree-sitter-wasms,
whose dart dependency is an UNPINNED github:UserNobody14/tree-sitter-dart —
a routine dependency update would have silently changed dart's grammar.
This PR byte-copies the shipping 0.1.13 artifact into src/extraction/wasm/
(VENDORED_WASM_LANGS += dart) and compiles the same-commit (d4d8f3e337d8)
parser.c/scanner.c in the kernel — table identity proven by the
kernel-grammar-parity row. crates.io tree-sitter-dart is the nielsenko
fork (different lineage) — rejected.
The center of gravity is THE SIBLING-BODY DOUBLE-WALK, reproduced
bug-for-bug: dart attaches every function/method body as a NEXT SIBLING of
its signature, and the TS walkers consume each body TWICE — once via
resolveBody (attributed to the function/method) and once via the enclosing
generic walk (attributed to the file/class). Duplicate local-function
nodes with the SAME id under different parents, duplicated
calls/instantiates refs, and file/class-attributed fn-ref twins all emit
in the exact observed interleave (a dedicated fixture pins the
duplicate-id rows; the bloc kind-census spot-check pins the counts).
Also preserved (probe-pinned): the extractBareCall selector matrix (the
first callTypes=[] language — cascades completely invisible, `?.` encodes
like `.`, the `ConfigT.load()` calls+references double emission with no
callee-of-call skip, capitalized-chain `Foo.create().run` re-encode,
const-object callee names); the constructor hooks (unnamed ctor skipped,
named ctors/factories renamed to the CTOR name with the class as
returnType, `@override (T) m()` record-misparse rescued by class-name
validation); operator methods minting `method "<anonymous>"`;
static_final_declaration constants via the visitNode hook while instance
fields mint NOTHING; the prefixed-return-type prefix bug (`other.OtherClass
f()` → returnType `other`); enum `with` mixins silent vs `implements`
working; anonymous extensions named after the ON type; deferred imports
invisible; named-argument callbacks NOT fn-ref-captured (the Flutter
`onPressed:` idiom — future accuracy PR, TS-side first); `async*`/`sync*`
NOT async; value-refs with the LIVE dart sibling-body pull and the
`$X`-vs-`${X}` interpolation asymmetry; dartdoc kept in all three comment
forms with the annotation-broken chain.
Gates: parity sweeps first-run 0-diff on shelf/bloc/flutter — 5,815 clean
files byte-parity, deferrals 10/21/1341 ≈ the survey's 10/21/~1340
(both-arm grammar reality: empty object patterns — the sealed-class
idiom — and unnamed `library;` dominate; --max-deferral 0.3); full-init
dumps byte-identical ×3 (shelf 7,959 / bloc 40,026 / flutter 1,855,319
dump lines); bloc per-kind node census identical across arms (the
double-walk duplicate rows survive the store identically);
kernel-dart-parity suite (7 fixtures + in-memory CRLF variants +
double-walk duplicate-id pin + generated-file skip pin + two defer pins);
full suite 2,688 green ×2 with CODEGRAPH_KERNEL_EXPECT=1.
DEFAULT_ROUTED += dart (20 langs — R7b COMPLETE).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R7b batch 4 #3 (docs/design/scala-kernel-port-checklist.md is the
authoritative quirk list). The third vendored-grammar-C language and the
biggest grammar in the tree (35MB parser.c): the vendored wasm is
tree-sitter/tree-sitter-scala master@0aca5d0a6f — a post-v0.26.0 generation
sync that is not a release (the 0.26.0 crate is 30 states BEHIND, so a
crate pin would be a silent downgrade). NO wasm change: production has
parsed with this exact revision since #91 — the kernel-grammar-parity row
(ABI 15, 26,650 states, 32 fields, id-by-id tables) is the whole alignment
proof.
Preserved bug-for-bug (all probe-pinned): the leak-through asymmetries —
extension methods mint NO nodes (first def's body calls leak to the
enclosing scope, later defs invisible, and the braced form resolves its
body field to the `{` TOKEN via first-match-wins field lookup → whole
extension invisible); anonymous `new T { … }` template_body members leak to
the enclosing scope (findAnonymousClassBody misses template_body); the
bodied-vs-bodiless class asymmetry (bodiless headers walk class_parameters
→ default-value calls emit FROM the class; bodied ones never see them) —
plus first-segment import names (`import com.example.C` → `com`), the
val/var hook keyed on the enclosing-definition NODE TYPE (object vals →
constants/value-ref targets, class/trait/enum/given vals → fields) with
consumed initializers, every def routed through extractMethod with the
top-level function fallback, nested defs in bodies minting NOTHING (the
inverse of kotlin) while body-local classes extract fully, curried
signatures keeping only the FIRST parameter list (type params win the
`parameters` field), enum cases positioned at the CASE node with invisible
params/extends tails, extends with-chains via scalaBaseTypeName,
`@deprecated(args)` decorates, the #750 capitalized-chain re-encode
(`WidgetS.create().render`), literal-receiver silence, static-member reads
AND writes, infix invisibility, `derives` silence, scaladoc retention with
the CRLF `\r` pin, full value-reference machinery (shadow prune, last-wins
same-name targets, `$X`/`${X}` interpolation reads), and SCALA_SPEC
fn-refs (bare ids + postfix eta unwrap + varinit, var-init non-capture).
Gates: parity sweeps first-run 0-diff on os-lib/cats/scala3-compiler-src/
scala3-library-src — 1,935 clean files byte-parity, deferrals 0/15/57/116
matching the survey's predictions exactly (scala-3's PHANTOM hasError
files — flag-true, zero ERROR nodes, capture-checking `^` — defer on the
FLAG); full-init dumps byte-identical ×3 (os-lib, cats, scala3 whole-repo
950,889 dump lines); kernel-scala-parity suite (9 fixtures + 9 in-memory
CRLF variants incl. Scala-3 indentation through the external scanner +
phantom/real-error defer pins + first-segment/namespace/value-ref pins);
full suite 2,669 green ×3 with CODEGRAPH_KERNEL_EXPECT=1
(kernel-scaffold's stays-wasm example moved scala → pascal).
DEFAULT_ROUTED += scala (19 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R7b batch 4 #2 (docs/design/lua-luau-kernel-port-checklist.md is the
authoritative quirk list). ONE walker for both dialects (ccpp precedent) —
the differences are exactly four: luau's type_definition aliases, the
`export `-slice isExported hook, the return-type signature suffix, and the
grammar handle.
Grammar prep is kernel-side only, no wasm change: lua is the SECOND
vendored-grammar-C language (the vendored wasm is the v0.4.1 tag, a revision
not on crates.io — tag artifacts compiled via build.rs, shas pinned); luau
is a plain crate pin =1.2.0 whose tarball is sha-identical to the tag (the
swift tag≠crate divergence does not recur). Grammar-parity rows replace the
bump gate entirely.
Preserved bug-for-bug (all probe-pinned): the require/visitNode-hook
ASYMMETRIES (top-level requires — including inside top-level if/for/while —
mint import nodes while the identical body-level statement emits
`calls "require"`; top-level `local x = foo()` initializers are invisible
while global `x = foo()` calls emit), the BFS string-win inside require args
(`require(script:WaitForChild("Kid"))` → import Kid) and Roblox instance
paths, receiver-QN methods (`M.sub.deep::chained`, `_G::installed`,
stack-QN nested globals like `render::leakedGlobal`), the raw-text callee
world (colon forms with `self` never stripped, bracket callees,
newline-glued chains byte-verbatim, the `(handler)` paren-conversion),
LUA_SPEC function-as-value capture with the `M.cb = cb` param-storage skip
and first-occurrence dedupe, LuaDoc `---` keeping a leading `- ` plus
`--!strict` joining docstring chains (block-comment docstrings keep interior
CRLF bytes), variable nodes at the IDENTIFIER with positional value pairing,
duplicate same-(kind,name,line) ids, and the lua↔luau isExported wire
divergence (lua functions: flag absent; luau functions: present-false;
methods: absent in both; variables: present-false in both; `export type`:
true).
Gates: parity sweeps first-run 0-diff on kong/lazy.nvim/lua-resty-core
(lua) + lune/Fusion (luau) — 1,734 clean files byte-parity, deferrals
1/0/0/3/8 matching the survey's both-arm predictions exactly (kong's 1 = a
deliberately invalid fixture; luau's = grammar-inherent generic type packs
and default type params); full-init dumps byte-identical kernel-vs-wasm ×4
(kong 157,650 dump lines); kernel-lua-parity suite (both torture fixtures +
in-memory CRLF variants + glue-chain, duplicate-id, and cross-dialect defer
pins + kernel-arm wire-flag pins); full suite 2,647 green ×2 with
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += lua, luau (18 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R7b batch 4 #1 (docs/design/r-kernel-port-checklist.md is the authoritative
quirk list; survey + probe record therein). The lightest-shared-surface,
heaviest-hook port: languages/r.ts works entirely through the visitNode hook
(every type list empty except callTypes:['call']), so the walker is a file
node + a faithful hook transcription + the generic extractCall + pre-order
recursion — four shared machineries (value-refs, static-member reads, type
annotations, fn-ref capture) are dead by language gates and stay dead.
Grammar prep is the first true no-op of the arc: the crates.io tree-sitter-r
1.2.0 tarball ships parser.c AND scanner.c sha-identical to the r-lib v1.2.0
tag the vendored wasm was built from — crate pin only, no wasm change, no
bump gate; kernel-grammar-parity gains the r row (ABI 14, same-revision).
Preserved bug-for-bug (all probe-pinned): calls "return" on every return(x)
(named node in v1.2.0), the import quintet's silent dynamic-arg consumption
vs class/generic fall-through asymmetry, library(help = pkg) importing the
named arg, class-idiom variable suppression by callee name, chained/right-
assign/precedence-ghost gaps, env$fn body-leak-to-file, raw-text callees
verbatim (pkg::fn, obj$meth, "strfn" quotes kept, (handler) conversion),
duplicate same-(kind,name,line) ids, roxygen dropped entirely, UTF-16
columns/slices.
Gates: parity sweeps first-run 0-diff on AnomalyDetection/dplyr/ggplot2/
shiny (838 files; deferrals exactly 0/0/0/1 — the 1 is the moustache-
template pseudo-R file, both-arm) — kernel-parity.mjs gained lowercased-
extension matching so .R files sweep (matches detectLanguage routing);
full-init dumps byte-identical kernel-vs-wasm on dplyr/ggplot2/shiny;
kernel-r-parity suite (torture fixture + in-memory CRLF + BOM variants +
defer pin + kernel-arm quirk pins); full suite 2,638 green ×2 with
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += r (16 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sixth R7b port — the T1½ batch finale. Checklist-first recipe
(docs/design/kotlin-kernel-port-checklist.md, 1,121 lines, dist-extractor
ground truth); parity passed FIRST RUN on all three repos.
THE NOVEL MECHANISM — vendored-grammar-C (the §4 tracker's prescription,
first use): the crates.io tree-sitter-kotlin 0.3.8 pins `tree-sitter >= 0.21,
< 0.23` (the kernel links 0.25) and tree-sitter-kotlin-ng is a DIFFERENT
grammar (8 fields vs 0, renamed kinds — extractor-breaking), so no crate dep
is possible. The fwcd 0.3.8 tag's sha-matched parser.c + scanner.c are
vendored into codegraph-kernel/grammars/kotlin and compiled by build.rs (cc),
exposed via tree-sitter-language::LanguageFn. The wasm re-vendor is
behavior-NEUTRAL (0 CST/error disagreements across 1,984 gate-repo files;
old-vs-new full-init dumps byte-identical ×3) — a reproducibility re-vendor,
ABI stays 14.
Walker firsts: extension-function receivers (getReceiverType →
`WidgetK::extend` QN OVERRIDE with no package prefix, the qualified-receiver
`com::qext` first-segment bug, and the owner-contains fallback that excludes
`interface` kinds and is source-order dependent) and extractModifiers
(expect/actual platform modifiers → the node DECORATORS wire field on every
created node — the KMP synthesizer's feed, incl. `actual typealias`).
Preserved bug-for-bug: the FIELD_COUNT-0 dead cluster (no signatures, ZERO
type-annotation refs), hook-consumed property initializers emitting nothing
(incl. `by lazy {}`), the bodiless-vs-bodied class header asymmetry, enum-
entry bodies being invisible, KDoc never a docstring AND chain-breaking,
comment-gluing into import/package extents, `@Anno(args)` emitting nothing
while `@Marker` decorates, zero instantiates refs, the paren-then-lambda
`trailing()` garbage callee, text-includes visibility/suspend false
positives, and the packaged-file value-ref target drop. The fun-interface
misparse-recovery hook is DEFER-SHIELDED (every such file has_error) and
deliberately not ported. The swift-sweep lesson pre-applied: the shared
`assignment` shadow-prune case is implemented alongside the
property_declaration case.
Gates: sweeps 0-diff okio 299/322, okhttp 531/580, kotlinx.coroutines
1031/1082 (deferrals exactly the predicted 23/49/51 — both-arm grammar
reality incl. PHANTOM hasError files with complete CSTs; the kernel trusts
the flag); full-init dumps byte-identical ×3 (46.5k/108.9k/92.3k lines); KMP
expect/actual synthesis IDENTICAL across arms (412 edges on
kotlinx.coroutines — the tracker's KMP validation); kernel-kotlin-parity
suite (torture reflowed off the phantom shapes + .kts script + CRLF variants
+ fun-interface and phantom defer pins) + kotlin grammar-parity row (the
C-build ↔ wasm table identity proof); full suite 2,633 green ×2 under
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += kotlin (15 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fifth R7b batch-3 port, checklist-first recipe
(docs/design/swift-kernel-port-checklist.md, 1,056 lines — the largest of the
arc, with a built-extractor-validated emission pin and a childForFieldName
truth table).
Grammar bump first, validated standalone: tree-sitter-wasms ^0.4.0 (ABI 13) →
crate 0.7.3 — with a provenance twist: the wasm is built from the CRATE
TARBALL's src/ (alex-pinkus keeps generated files off main and the
0.7.3-with-generated-files tag ships an older ABI-14 generation that can never
sha-match; grammar.json rules are JSON-equal; the tarball is byte-for-byte
what the kernel's cargo build compiles — table identity by construction).
Older crates evaluated and rejected: clean-parse shapes are byte-identical on
0.7.3 (53-line CST battery diff, all inert), so an older pin buys nothing and
loses the macro-era wins. Delta = error-set membership (63 old-error files
parse clean: swift-testing #expect, #Preview/#GET macros, package access,
typed throws — vapor 23.1%→9.3%; 21 NEW-only regressions in 3 probed
construct classes) + two gate-found categories: docstring boundaries near #if
directives (7 clean files, docstring-field-only — verified mechanically) and
array-literal-callee call refs (2 refs, 1 file). Every hunk classified via
the error-union rule + parked-ref↔edge ripple pairing.
Walker (the arc's biggest) centers on the #1020 DEDICATED property branch:
computed properties → property nodes with the getter walked under the
property (SwiftUI body), static let/var → constant/variable, stored → field,
decorator/type-annotation/@Siblings-attr-arg refs all attached to the
ENCLOSING TYPE, stored initializer calls attributed to the class. Preserved
bug-for-bug: the never-resolving 'parameter' field (zero param type refs,
zero signatures), present-false isAsync, open→internal visibility,
everything-is-extends inheritance (first type_identifier per specifier), no
instantiates refs ever, subscript reads as `calls arr`, `defer` as `calls
defer`, multi-case enum entries minting only the first case, /** */ block
docs ignored AND chain-breaking, init/deinit/subscript minting no nodes with
visitNode-routed bodies (calls → class, static reads → nothing), multi-
segment extension resolveName, sugar extension names, the #selector shapes,
and the value_argument label-forward skip. ONE fix found by the sweep (then
pinned in the fixture + checklist): the shared `assignment` shadow-prune case
is swift-live — declared-then-assigned `let X: T` prunes X as a value-ref
target.
Gates: sweeps 0-diff Alamofire 89/98, vapor 224/247, swift-nio 407/554
(--max-deferral 0.3 — swift error incidence is 9–27% on BOTH arms,
structural; every deferral count matches the survey's table exactly);
full-init dumps byte-identical ×3 (31.9k/20.7k/126.3k lines); the Alamofire
census reproduces property=348 (the #1020 number) on the kernel arm;
kernel-swift-parity suite (206-line torture + CRLF + the #if-between-enum-
cases defer fixture) + swift grammar-parity row; full suite 2,626 green ×2
under CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += swift (14 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
First R7b language port. Grammar: tree-sitter-rust pinned =0.24.2 + wasm
vendored from tag 77a3747 (parser.c/scanner.c sha-matched against the
crates.io tarball), replacing the 2023 ABI-14 tree-sitter-wasms build —
the bump alone is precision-positive on the wasm path (receiver-qualified
instance-method resolutions replace ambiguous bare-name matches; node
sections byte-identical on ripgrep/tokio).
Walker mirrors the TS reference bug-for-bug per
docs/design/rust-lang-kernel-port-checklist.md (survey artifact): dead-code
isAsync, impl-pushes-no-scope, the impl-Trait-for-Generic<T> trait-receiver
quirk, phantom const identifiers, use-binding triple emission,
wildcard-use-emits-nothing, scoped-supertrait drop, chained-call re-encode
gated on scoped_identifier, Rocket route macros body-only.
Gates: parity sweeps 0 diffs — ripgrep 101/101, tokio 790/790,
rust-analyzer 1217/1488 (271 deferrals are token-macro-table sources that
error on BOTH arms — grammar-inherent); full-init dump-diffs byte-identical
on all three (3,857 / 13,440 / 39,030 nodes); kernel-rustlang-parity suite
(torture + CRLF + defer) in npm test; full suite green x2 with
CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Store-architecture arc round 2. The batched resolution loop reads
unresolved_refs ONLY through the status index + the PK keyset pager;
the other five ref indexes (from_node, name, file_path, from_name,
failed_tail) serve sync-time paths — yet every per-batch DELETE of
resolved refs maintained all of them, the biggest single main-thread
stage on the dubbo profile (deletes 1.2s of a 5.4s resolution phase)
and 50-81s at kernel scale.
beginBulkRefLoad/endBulkRefLoad on DatabaseConnection, threaded as
refIndexLoad hooks next to the existing bulkEdgeLoad pair with the
same minRefsForPool gate (small syncs never pay): drop the five for
the loop, rebuild each in one scan at the end — where the table holds
only the surviving FAILED refs (resolved rows are deleted by then),
so the recreate is near-free. Crash inside the window heals on the
next open (schema.sql re-applies CREATE INDEX IF NOT EXISTS).
Measured:
- dubbo: deletes 1.2 → 0.2s, marks 0.6 → 0.3s, recreate 219ms; wall
~8.5s flat — the freed main-lane time shifts into settle (the worker
lane now binds the double-buffer at medium scale).
- Linux kernel 8c: resolution 423.4 → 275.9s (deletes 50-81 → 3.2s,
backpressure 16.8 → 7.4s — fewer index writes mean less WAL and
cheaper folds), ref recreate 10.3s. Envelope ≈ 11.0min, from the
14.8min pre-arc best; <10min-on-8c now needs ~1 more minute.
Gates: dubbo/gson dumps byte-identical; linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Store-architecture arc round 1 (the cbm speed bar: dubbo warm wall
10.7-11.2s vs their ~7.5). §4d measured dubbo's parse-loop as 94%
store-writer busy with B-tree maintenance as the floor (statement
batching and sorted inserts already killed at ~zero). This applies the
resolution phase's proven edge-index window to the whole parse lane:
beginBulkParseLoad/endBulkParseLoad on DatabaseConnection — FRESH-INIT
ONLY (incremental runs delete per-file rows through the file_path
indexes) — drop all 15 nodes/unresolved_refs/files secondary indexes
plus the 4 non-unique edge indexes for the parse phase's mass insert
(the UNIQUE edge identity index stays: OR-IGNORE dedup conflicts on it,
and its source prefix keeps mid-window reads indexed), then rebuild
each in one table scan before resolution, with a yield between builds
(the endBulkEdgeLoad watchdog rationale). A crash inside the window
heals on the next open — schema.sql re-applies CREATE INDEX IF NOT
EXISTS.
Measured:
- dubbo (cbm bar repo): parse-loop 4,306 → 1,787ms (−58%), rebuild
665ms, warm fresh-init wall 10.5-11.3 → 8.46-9.39s (−19%); the bar
gap vs cbm shrinks from ~3s to ~1.1s.
- Linux kernel 8c: envelope ≈ 14.2min, best ever (prior 14.8). Parse
itself flat (linux parse is extraction-bound, not writer-bound) and
the rebuild costs 21.6s — but every downstream phase dropped
(resolution 517-589 → 423.4s, edge-recreate 36.5s, synthesis 157.1s,
maintenance 16.3s): bulk-rebuilt B-trees are densely packed where
incrementally-grown ones are fragmented, so every index-mediated read
for the rest of the run pays fewer pages.
Gates: dubbo/gson/express/excalidraw full dumps byte-identical
(dubbo's canonical 441,270 lines reproduced); linux counts exact
2,049,153/6,413,518 and dump sha 6dd1185b… reproduced (10,446,478
lines); full suite green ×2 (153 files / 2588 tests). Incremental
sync paths untouched by construction (freshDb gate).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Two nudge shapes probed at 8c kernel scale (passive checkpoint at the pool
recycle boundary, fire-and-forget and awaited). The awaited shape reached
every §7a.7 shallow floor (read 16.6s, inserts 31.4s, deletes 50.0s) and
paid exactly what it saved (207.4s of attributed folds); fire-and-forget
regressed deletes 81→171s via fold/delete I/O contention while the
size-blind growth baseline kept the hard-cap parks firing on top (143
nudges, still 22 parks). All three arms within ~2.5% of each other — the
phase's fold I/O is a fixed budget the baseline already overlaps
off-thread, and the §7a.7 "~45s gap to the floor" is illusory. Code
reverted; #1362's valve + recycling remain the optimum of this family.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Task #5 step 2. The fuse-then-link refactor (#1364) left the extraction
sweep as a clean per-file boundary: raw text in → collected facts out.
This ports that sweep to the native kernel: `cfnptr_scan_files`
(codegraph-kernel/src/cfnptr.rs) strips and scans a batch of 16 files
per NAPI call, and the TS side only reads files, ships batches, interns
the returned facts, and resolves include paths. The JS sweep remains as
the fallback (no binary, feature detection against older binaries,
CODEGRAPH_KERNEL=0, or CODEGRAPH_KERNEL_CFNPTR=0).
Parity discipline: the JS regexes are the spec, so the scanners are
hand-rolled byte machines reproducing that engine — ASCII \w/\b next to
UNICODE \s (NBSP/U+2000-200A/FEFF decoded from UTF-8), alternation
order, lastIndex resume, and the observable backtracking dimensions
(INIT/ARRAY modifier and struct/star/bracket optionals, DISPATCH's
greedy segment loop); greedy shortcuts only where backtracking provably
can't rescue a match. The native stripper blanks per UTF-16 code unit,
so its output is string-identical to the TS stripper — pinned by a new
kernel arm on the strip differential oracle (fixtures + 500 seeded
random cases).
Gates, all green: new differential suite (adversarial fixture project —
CRLF, NBSP, continuations, decoy strings, unterminated comments,
backtracking shapes — indexed native-vs-JS: identical edge streams,
plus a record-level scanner check); repo differential on
git/redis/vim/SameBoy (identical, 705/852/433/180 edges); probe-hash on
the live linux kernel DB reproduced f6e1713d… (279,335 rows); linux
init counts exact 2,049,153/6,413,518; dump sha 6dd1185b… reproduced
(10,446,478 lines); full suite green ×2 (153 files / 2588 tests).
Measured (8c cg1212, quiet host): cFnPtr sub A=47.9s B=1.1 C=40.9
D=24.1 E=36.8 = 150.9s vs step 1's 179s and the pre-arc 230s (−34%
cumulative); the sweep itself halved (94.5→47.9s, JS strips
132.4k→68.9k). callback-synthesis phase 199.9→171.1s. E's attributed
wall grew from overlap shift under parallel synthesis; the phase total
is the honest number. Full record: plan §7a.10.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Task #5 step 1 (plan §7a.8/§7a.9). The C/C++ function-pointer dispatch
synthesizer swept every file's text four times (typedefs, registrations,
propagation, dispatch); at kernel scale the all-or-nothing source cache
declines, so that was 4.4 read+strips per file — 78s of the ~230s pass.
Now ONE extraction sweep reads+strips each file once and collects typedef
names, struct-node field declarations (structurally parsed, fn-pointer
classification deferred until the typedef sets are complete), resolved
includes, an alias-shaped-macro name set, and per-file survival filters
(init type tokens, array element types, inline-struct summaries,
field-assign pairs, dispatch fields/array names — interned, a few MB on
linux). The linking stages then replay the ORIGINAL pass bodies verbatim:
struct layouts register in kind-scan order (same-name precedence is
order-sensitive), and registration/propagation/dispatch run only for
files their filter proves can have side effects, lazily re-stripping just
those. Filters only over-approximate (full-file no-skip scans ⊇ the real
passes' jump-cursor scans), and a filtered-out file is one where every
match fails the pass's own gates before any side effect — parity by
construction. Macro tables stay lazy: a sizing probe found 6.1M #define
lines on linux (amdgpu register headers), ruling out retention.
Measured (8c cg1212, quiet host, fresh kernel init): cFnPtr pass 230s →
179s (−22%); strips 283.5k → 132.4k (4.44 → 2.08/file, 78 → 46.6s);
dispatch stage 95 → 18.5s; callback-synthesis phase 250 → 199.9s.
Standalone probe on the live DB: 139 → 122s.
Byte-parity gates, all green: probe-hash identical on the live kernel DB
(279,335 edge rows both builds); git/redis/vim/SameBoy full dumps
byte-identical old-vs-new (macro tables, commands.def, #ifdef
include-units, inline structs, bare arrays exercised); kernel-parity
0-diffs on git/redis/fmt/protobuf, deferral unchanged; linux counts
exact 2,049,153/6,413,518 and dump sha 6dd1185b… reproduced
(10,446,478 lines); full suite green ×2 (152 files / 2563 tests).
Step 2 (native per-file extractor) now has its boundary: the extraction
sweep, raw text in → records out.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three measurements before any port. The stripCStyle split('') rewrite
(byte-identical segment-builder) measured 1.0× on 15.1M chars of linux C
— V8's ~73MB/s scan rate IS the cost, and 78s ≈ 4 strips/file × that
rate: the lever is the redundancy, not the scanner. Rewrite reverted;
the differential oracle test ships so any future rewrite stays pinned
byte-identical. E's regexes alone run at ~46MB/s (~30s of its 95s; the
rest is per-match logic and slicing).
Re-ordered attack recorded in §7a.8: step 1 = TS fuse-then-link refactor
(strip once per file, collect raw matches + declared-type tables,
text-free global linking; ≈ −70-90s, parity via collector insertion
order + the §7a.4 probe-hash gate); step 2 = native per-file extractor
behind the same boundary (raw disk text — no preParse interaction).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The §7a.6 anomaly probed to its mechanism with five discriminating runs
(§7a.7 table): main-thread B-tree writes triple under attached readers
because READERS PIN WAL checkpoint progress — the deep WAL taxes every
writer page operation (deletes 42.6s pool-off vs 118.8s pool-4 on
identical hardware; an aggressive 64MB valve recovers the writes but
overpays +129s in full-park folds; the v2 cache resurrection was
falsified — long-tail name traffic is uncacheable at any capacity).
Fix: workers close and reopen their read-only connections every 8
batches at the double-buffer's worker-idle boundary
(ResolverPool.recycleWorkers + QueryBuilder.rebind + a cadence call).
Reopens are sub-millisecond, resolver caches survive (only prepared
statements re-prepare), and the existing checkpoints advance instead of
parking. Failed recycle downgrades to sequential, same as a failed
fan-out.
Measured (8c pool-4, linux v7.2-rc2, cadence 25 → 8 iterated):
resolution superphase 715.0 → 633.6s (−11.4%), envelope best 14.8min,
recreate 59.7 → 45.3s. Byte-neutral everywhere: git dumps byte-identical
old-vs-new, linux dump sha 6dd1185b reproduced (10,446,478 lines),
counts 2,049,153/6,413,518, suite 2517 green. 2c unchanged by
construction (no pool → no recycling).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fresh CODEGRAPH_RESOLVE_PROFILE tables at 2c and 8c on the round-2 build:
the 8c settle stage is 3.6s — the double-buffer absorbs the entire
resolveOne population, superseding §7a.2's core-invariant framing. The
8c cost is writes-under-readers: main-thread deletes+insertEdges run ~3×
slower (+102s) with 4 readonly workers attached — mechanism unproven,
named as the next probe. Two same-day experiments killed by measurement
and reverted: budget-scaled name caches (v1 LRU regressed via
delete+set-per-get churn + GC; v2 mutation-free second-chance cache
landed exactly on baseline — the 11µs exact-match is per-ref floor, not
refetch overhead) and lazy candidates JSON (read stage flat). New 2c
record 16.5min clean-host; 8c range 15.0–16.4min across n=2 — ranges,
never single runs. Round-2 deferral cuts did move 8c parse (202.6→178.5s).
Levers re-ranked: writes-under-readers probe > cFnPtr native site
extraction (~230s of synthesis) > backpressure byte volume > recreate.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Census-driven cut of the top-ranked post-R7a lever. All passes TS-side,
C-only (preParseCSource), shared by both arms:
- parameterized-annotation whole-blank (__free/__printf/__counted_by/
__bpf_md_ptr…; extends through a stranded field `;`)
- type-keyword-arg scanner (kzalloc_obj(struct T), list_entry, multi-line
continuations behind nested-paren args; bounded hand scanner, head
exclusions + call-vs-declaration guard; blanks trailing stars)
- static/extern CAPS-macro declaration lines at any scope; the initialized
form is REWRITTEN to its expansion (name/tail keep exact offsets)
- va_arg qualified-type blank; GNU named-variadic #define dots-only blank
(post-restore); sandwiched notrace-family; C23 auto; multi-line
iterator-macro spans (hlist_for_each_entry_rcu + lockdep arg)
- word list += cacheline family (2- and 4-underscore spellings) + 10 more
census-confirmed annotations
Gates: five-repo parity sweeps 0 diffs (git deferral 16.1→12.2%, redis
25.3→24.1%, fmt/protobuf unchanged); linux full-tree both arms
2,049,153 nodes / 6,413,518 edges (+858/+6,585 vs R7a) with byte-identical
dumps (10,446,478 lines, sha256 6dd1185b); kernel-arm parse-loop 356→306s
at 2c; suite 2517 green under CODEGRAPH_KERNEL_EXPECT=1. Honesty note
recorded in the docs: error recovery was already salvaging most SYMBOLS on
deferred files — the graph win is relationships + phantom cleanup, and the
unreleased CHANGELOG entry was rewritten off the sweep-subset framing.
Also records §7a.5: post-R7a 8-core cg1212 re-run 16.4min (was 18.3min);
8c parse sits on the single-writer floor, so the <10min-on-8c gap re-ranks
to the per-ref resolution path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parity: 0 diffs on redis/git/fmt/protobuf/ALS sweeps; full-init dumps
byte-identical on all five + linux at kernel scale (10.4M dump lines,
same sha256 both arms). Linux 2c/6GB envelope: kernel-arm 19.1min vs
wasm-arm 22.9min (parse 356s vs 435s) on a much richer graph (the new
blanks recover error-swallowed code: git 2x nodes, linux kernel/+mm/ 3x).
Deferral guard corrected by measurement (C/C++ error incidence 9-42%;
--max-deferral flag); defer-reuse memo kills the 3x re-blank/re-parse
cost deferred files paid.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
tree-sitter-wasms shipped 2023-era C/C++ grammars; the kernel walker must
compile the crates.io versions, so production wasm upgrades FIRST and in
isolation (the R2 pattern). Built with tree-sitter-cli 0.25.10 from each
tag's CHECKED-IN parser.c: tree-sitter-c v0.24.2 (b780e47, parser.c
f2883ff9), tree-sitter-cpp v0.23.4 (f41e1a0, parser.c 2a35a43b, scanner.c
cf60387d) — sha-matched against the crates.io tarballs. .metal/.cu map to
language 'cpp' so the dialects ride the same coherent grammar.
Full suite green with the upgraded grammars (2,490; one unrelated daemon
idle-timeout flake passed solo 9/9) — incl. the UE-macro, misparse-guard,
Metal, and CUDA coverage. Checklist updated with the vendored revs and the
metal/cuda routing correction.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Step 1 of the §0a recipe for the C/C++ port: every tree-sitter.ts branch,
extractor-config field, name-salvage helper, and gate the walker must
mirror, with file:line anchors. Locks the architecture: preParse (all seven
blanking passes) hoists to the kernel route point so none of it ports to
Rust; Metal/CUDA stay on wasm this round; one dual-language walker module.
Key asymmetries surveyed: value-refs are C-only (cpp absent from
VALUE_REF_LANGS), static-member refs are cpp-only, fn-ref spec is
cFamilySpec with addressOfOnly for cpp.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
cFnPtrEdges was 86% of kernel-scale synthesis (306s, §7a.3). Standalone
probe iterations against the live kernel DB attributed and fixed:
- sliceLines split the ENTIRE file per node (~1.6M full-file splits across
the B/D/E sweeps) — split once per file, slice from the line array
(D 46.2->20.5s, E 94.5->69.1s).
- The 128-entry strip cache re-stripped every file 4.4x across the four
file sweeps (71.8s). Sizing is now memory-budget-aware and ALL-OR-NOTHING
with 5% slack over the file count: a partial LRU on cyclic sweeps thrashes
to ~0% hit (measured twice: cap ~61k and cap == files both thrashed
against 63.8k + includes). Full-cache strips: exactly 1.0/file, 15.5s.
- Pass D pre-gates field names on fieldToStructs before the two regex type
resolutions — a->f = b->g matches every struct-field assignment in the
tree, ~99% of them data fields.
- recvTypeIn/varTypeIn compiled a new RegExp per call - cached per name.
- Line numbers counted incrementally per match (ascending indexes) instead
of splitting the body prefix per match; cursor rewinds between the two
dispatch scans.
- CODEGRAPH_SYNTH_TIMINGS now prints per-sweep walls + read/strip/nodes
accounting for this pass.
278.8s -> 134.8s standalone at the 2c envelope. Output identity proven at
full scale: probe edge set (merge-dedup applied, canonical sort) SHA256
21c2a971... == the original code's 274,762 edges extracted from the live
kernel DB. Pass suite 15/15.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The §7a.2 per-ref profile overturned the assumption the whole arc was
built on: resolveOne owns only ~93s of the kernel-scale ~433s batch loop.
Loop-stage attribution (CODEGRAPH_RESOLVE_PROFILE, shipped here) named the
rest: backpressure folds 111.2s, count guard 93.9s, batch reads 54.6s,
deletes/inserts/marks ~84s, settle 85.7s.
- Non-progress guard O(remaining)→O(1): the per-batch COUNT(*) walked every
remaining pending row (O(N²/batch) per run, 93.9s). The cleanup queries
now return summed SQLite , and zero-removals-from-claimed-work
is the guard signal — the DIRECT evidence the count diff inferred (a
mismatched-name resolver makes keyed cleanup no-op ⇒ changes=0). A real
COUNT runs only on that suspicious path and arbitrates exactly as before.
- Batch reads OFFSET→keyset (54.6s→O(batch)): OFFSET re-walked the
accumulated failed-row prefix every read; seeking past the last-seen
rowid is prefix-independent and enumeration-order identical.
- WAL valve caps scale with DB size (env still wins): every fold re-writes
hot pages (#1231 in bounded form — 111.2s at the flat 256MB cap);
soft=clamp(dbSize/4, 256MB, 2GB) trades ~4× fewer folds for a transient
WAL ≈ project size.
- CODEGRAPH_RESOLVE_PROFILE: per-outcome resolveOne histogram + loop-stage
attribution, main + workers, off by default.
Gates: dubbo dump byte-identical; suite 2,491 passed / 4 skipped (kernel
required). Kernel-scale payoff run lands in the plan doc next.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A truncate checkpoint started against an ACTIVE writer wins the lock and
then blocks that writer for its entire backfill; after the edge-index
recreate's multi-GB single-transaction burst that exceeds the writer's 5s
busy_timeout and fails the index with 'database is locked' (§7a.2 record
run, EXIT=1 at kernel scale — the small mid-resolution truncates folded in
ms and masked the hazard). Barrier truncates (backpressure/foldNow) are
collision-free by construction: the writer is awaiting the valve.
Dubbo gate: exit 0, peak 81MB (barrier folds carry containment), dump
byte-identical. Valve + sizing suites 27/27.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Four §7a.1 instrumented-run findings, each measured:
1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still
grows the FILE without bound — the writer only restarts at frame 0 when a
commit finds zero reader marks, which the instrumented run showed never
happens (file marched 361→721MB through two COMPLETE backfills; 22GB by
phase end). backpressure() now also trips at 4× the soft cap on raw file
size and TRUNCATEs at the parked barrier; the timer path truncates
opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at
the same 16MB valve; dumps byte-identical under aggressive folding.
2. cgroup memory credit: memory.current counts reclaimable page cache — a
post-parse container read 57MB of headroom on a 6GB box and silently
disabled the pool. inactive_file is credited back (the docker-stats
working-set convention); the same run now reads a sane 4.4GB budget.
3. Pool at 2 cores reversed: sequential resolution measured FASTER than
pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is
Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing.
cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path.
4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower
(493s vs 369s) — main + store-worker don't fill the second core. Floor
restores the baseline (373.5s measured).
Plus the observability §7a.1 burned three 25-minute cycles for: valve
armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture,
pool sizing decisions (incl. the disabled path), backpressure-hook
presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG.
Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record
runs with this build follow in the migration plan §7a.1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three §7a.1 run-1 lessons (kernel-scale 2c/6GB: EXIT=137, WAL 22.2GB with
the backpressure hook DEPLOYED):
1. TRUNCATE at parked barriers: a completed passive backfill bounds the
un-checkpointed backlog but the FILE only stops growing when a commit
finds zero readers holding WAL marks — rare while pool workers cycle
(dubbo debug baseline: file climbed monotonically through six completed
pass-1 backfills). At a parked barrier the no-reader window is
guaranteed, so chop the file there with wal_checkpoint(TRUNCATE)
(off-thread, 2s busy_timeout — a racing reader degrades it to a no-op).
2. Futility latch: when backfill gives up (pinned reader), parking again at
every over-cap boundary burns a 20-pass checkpoint attempt — each a
worker thread + fresh connection against a multi-GB DB — per batch. Two
consecutive give-ups now disable parking for 60s; a pinned phase degrades
to pre-valve behavior instead of OOM-amplifying.
3. CODEGRAPH_WAL_VALVE_DEBUG=1 surfaces valve decisions without the
caller's verbose plumbing, and give-up lines print under
CODEGRAPH_SYNTH_TIMINGS — run 1 failed silently because give-ups were
verbose-gated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Pool sizing used os.cpus().length, which enumerates the HOST's CPUs: inside
a 2-CPU cpuset it sized 6 resolver workers (the §7a.1 false-'sequential'
premise) and 8 parse workers, and at true 8-core concurrency six ~1GB
workers OOM-killed a 7GB container (oom_kill=5) mid-synthesis — sizing had
no memory term and no override knob.
resolvePoolSize (pure, matrix-tested): explicit CODEGRAPH_RESOLVE_WORKERS
override (0 disables, cap 16); CPU term max(2, min(availableParallelism-1,
6)) — cpuset-honest, floored at 2 so true 2-core boxes keep pooled
synthesis's ~2×; memory term floor(budget*0.7 / clamp(0.2*dbSize, 256MB,
1.5GB)) with budget = min(freemem, cgroup v2/v1 headroom). Parse pool's
core input switches to availableParallelism. Dev machines are unchanged
(still 6 workers); the 8c/7GB kernel-scale container now sizes 4.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
At kernel scale the pooled resolution/synthesis superphase grew a 22GB WAL
on a 4.6GB DB (cg1212, §7a.1): autocheckpointing is deferred for the run,
and the valve's timer-driven passive checkpoints stay perpetually partial
against the pool's continuous reads — no mechanism ever completed a
backfill, so the WAL accreted the whole phase's write volume, blowing disk
and feeding page-cache pressure into the 8-core/7GB container OOM.
The valve's writer-side backpressure() hard-cap backstop existed but was
wired only into the PARSE orchestrator. Thread it into the resolution batch
loop at the double-buffer's one pool-idle boundary (batch settled, next not
yet fanned out), after the edge-index recreate, and through the synthesis
insert loops. Parked there, the backfill completes; readers re-enter at
SQLite's backfilled mark and the next persist commit wraps the WAL.
Dubbo validation, same build: valve@16MB peak WAL 251MB (floor = the
single-transaction edge-index recreate) vs defaults 914MB; dumps
byte-identical (441,270 rows); wall unchanged (11s). Suite 2,479 green.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
On CRLF checkouts (every Windows autocrlf clone) the JS reference's
block-continuation strip /^\s*\*\s?/gm finds a line start after the \r and
its greedy \s* consumes the \n, leaving a bare \r in the docstring; the
kernel's (?m)^ pass matched after \n only and kept \r\n. Caught by the O2
Windows VM leg (6 kernel-tsjs-parity failures), reproduced on macOS by
CRLF-converting the fixtures.
js_multiline_strip now replicates the JS anchor set (\n, \r, U+2028, U+2029)
for all five line-marker passes; CRLF variants of every torture fixture are
pinned in kernel-tsjs-parity, derived in-memory so nothing can normalize
them away.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R1-R6 are done; §0 is now the open-work list in recommended order
(merge → Windows VM leg → P1 resolution → C/C++ port → long tail), and
§0a carries everything a fresh session needs: where the work lives, the
build/gate commands, the proven add-a-language recipe, and the paid-for
traps (encoding-dependent error recovery → defer policy, node-ID-string
dedupe, UTF-16 positions/slices, the exact-seam contract, crate+wasm
grammar lockstep). §1/§6 keep the original expectations with SUPERSEDED
annotations pointing at the measurements that corrected them; §7a now
carries the R6 numbers that make it the top open perf item.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cg1212 (Linux kernel, 2 CPU/6GB): completes in 26.4min vs the ~27min
baseline with all new machinery active — no regression; graph scale
identical (2,048,664 nodes / 6,405,964 edges). The §6 parse expectation
(6m → ~2m) was mis-premised: the tree is ~99% C, an unported T2
language, so it transfers to the C/C++ port (R7). Resolution remains
the kernel-scale wall (19.2m, 73% — P1). The tree's own Python tooling
files: 99/99 byte-parity. kernel-parity.mjs now skips dangling
symlinks (Linux dtc fixtures).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Python (codegraph-kernel/src/python.rs) and Go (src/go.rs) join the
native kernel, mirroring the wasm extractors bug-for-bug. Python:
decorated_definition docstrings/decorators (decorates refs only for
bare-identifier decorators — the call-kind quirk), function-in-class →
method, module-level assignments always extract as variable, from-import
per-name binding refs, self.x fn-ref candidates as bare names. Go:
receiver methods with Recv::name qualified names + contains edges to the
first earlier struct of that name, type_spec struct/interface
classification with embedding→extends and interface method nodes,
composite-literal instantiates keeping the package qualifier, top-level
var/const initializer walks attributed to the declared symbol (#693),
2-hop field chains (#1276), New().Method() re-encode (#645/#608), and
the GO_SPEC fn-ref layers.
Grammars: tree-sitter-python 0.23.6 + tree-sitter-go 0.23.4 crates, with
wasm vendored from the same tags (parser.c sha-matched) — both were
2023-era in tree-sitter-wasms.
Gates: extraction sweeps 100% (flask 83/83, django 3,035/3,038 +3
error-file deferrals, gin 99/99, prometheus 978/979 +1); full-init
dump-diffs byte-identical on flask (10,833 rows), gin (17,540), django
(360,794), and prometheus (213,758); torture fixtures enforced in npm
test. DEFAULT_ROUTED now covers typescript/tsx/javascript/jsx/java/
python/go. Full suite: 2,471 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kernel-routed files ship their flat tables from the parse worker to the
store worker as buffers (tryKernelExtractRaw → kernelBuffers on the
result → KernelStoreBundle); the store worker decodes and finalizes
(finalizeStoreBundle shared with the object path so filter semantics
can never drift). Files with applicable framework extract() hooks keep
the decoded path; the no-writer fallback materializes via
materializeKernelResult. Byte-identical dumps re-verified on dubbo,
excalidraw, express, gson; full suite green (2,467).
Measurement (plan §4d): dubbo's parse-loop wall is 94% store-writer
busy time — the many-core fresh-index wall is single-writer SQLite
ingest, not extraction or main-thread work. d2s improves the writer
lane ~11% (structured-clone deserialization avoided on the writer) and
frees the main thread; the remaining many-core gap is a
store-architecture arc (deferred index builds, multi-file
transactions, buffer→bind), out of the kernel project's scope.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Java joins the native kernel (codegraph-kernel/src/java.rs), mirroring
the wasm extractor's Java paths bug-for-bug: package namespaces,
imports, javadoc, annotations→decorates, type_list inheritance,
static-final constants, enum constants, anonymous classes (including
the TS side's 0-based-line quirk on the extends ref), method_invocation
calls with the this.field unwrap and the Foo.getInstance().bar() chain
encoding (#645/#608), static-member value reads, method-reference
fn-refs (#756), value-reference edges, and the full Lombok member
synthesizer (#912: Getter/Setter/Data/Value/Builder/ToString/
EqualsAndHashCode/Slf4j-family with taken-member dedup). The shared
docstring/textutil modules moved to crate level. Grammar:
tree-sitter-java 0.23.5, with the wasm grammar vendored from the same
tag (parser.c sha-matched) replacing tree-sitter-wasms' 2023-era build.
Gate (plan §4c): extraction sweeps 100% — gson 262/262, retrofit
341/341, dubbo 4,048/4,048 — plus a Java torture fixture in npm test;
full-init dump-diffs byte-identical on gson (49,766 rows), retrofit
(62,735), and dubbo (441,266 rows); all R2/R3 repos re-verified; Linux
container runs all 23 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1.
The gate caught a real cross-language bug: fn-ref dedupe and value-ref
self-target checks must compare node ID STRINGS, not node-table rows —
ids collide for same-(kind, name, line) nodes, which minified one-line
bundles hit routinely (retrofit's website JS exposed it; latent in the
TS/JS walker since R2, never released). Fixed in both walkers.
Benchmark honesty: dubbo fresh-init on an 11-core Mac is ~flat
(parse-loop wall 5,020→4,394ms; total ~11.3s both arms) because that
wall is main-thread-bound (reads + store), not worker-CPU-bound — the
§6 expectation assumed otherwise. Where worker CPU binds the kernel
delivers: dubbo on a 2-CPU/6GB container drops 27.8-28.6s → 22.3-22.8s
(~1.25×). The identified lever for the many-core headline is decoding
kernel buffers directly into store rows (skipping per-node JS object
materialization); the buffer contract already carries everything.
DEFAULT_ROUTED now includes java. Full suite: 2,467 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate evidence (docs/design/rust-kernel-migration-plan.md §4b):
- Graph parity, byte-identical (stronger than the §5 ≤0.5% bar): full
codegraph-init dump-diffs kernel-vs-wasm on express (13,712 rows),
excalidraw (89,898), and vscode (2,378,238 rows) — identical bytes.
Python control repo (flask) identical + timing unchanged. The parity
harness is now ORDER-sensitive (emission order drives rowids, which
drive resolution order) and dumps come from the new
scripts/dump-graph.mjs (natural keys, no rowids/timestamps).
- The one real find, caught by the vscode tier: tree-sitter error
RECOVERY is encoding-dependent — byte-identical grammar sources and
the same core (0.25.10) recover erroring files differently under
UTF-8 (native) vs UTF-16 (web-tree-sitter) parsing; proven by
reproducing the wasm tree with a native UTF-16 parse. Policy: the
kernel defers any file whose tree has_error() to the wasm extractor
(silent 'defer:' signal, per file) — parity by construction on
erroring files (incidence 0-0.42% across the gate repos), and the
harness fails if deferrals exceed 10% so a broken kernel can't hide
behind the fallback.
- Retrieval invariants: canonical excalidraw flow (mutateElement →
renderStaticScene) connects end-to-end on the kernel-indexed graph;
synthesized-edge families present. Agent A/B is vacuous under
byte-identical DBs (same justification as #1320-#1322).
- Perf: vscode init 105.4s → 82.1s (1.28×) on an 11-core Mac;
excalidraw on a 2-CPU/6GB Linux container (the CI-runner envelope)
6.2-7.1s → 4.3-4.8s (~1.5×). Linux arm64 in-container build: all 22
kernel tests green under CODEGRAPH_KERNEL_EXPECT=1. Windows VM leg
deferred (VM stopped; prlctl start needs Parallels Pro) — benign: a
missing .node falls back to wasm, and the release matrix builds and
gates the win32 prebuilds.
- Full suite: 2,465 tests pass WITH default-on routing, so the entire
extraction corpus now exercises the kernel for TS/JS wherever a
.node is staged.
DEFAULT_ROUTED = {typescript, tsx, javascript, jsx}. Override:
CODEGRAPH_KERNEL_LANGS (replaces the set) / CODEGRAPH_KERNEL=0 (kill).
Changelog entry added under [Unreleased].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
