Python's class-as-value idioms (return SomeClass, x = SomeClass, registry
dicts, classes passed as arguments) produced no references edges, so
callers/impact on a Django/DRF serializer missed the views that consume it.
Three gates dropped them:
- return_statement was never dispatched by PYTHON_SPEC (kernel mirrored)
- the extraction gate (definedHere) collected function/method names only
- resolution accepted function/method targets only (matchFunctionRef +
the function_ref import fast path)
Capture return_statement for Python (single expression; tuple returns not
descended), admit same-file CLASS names to the gate, and accept class
targets for Python bare identifiers — scoped to Python so the TS/JS KIND
FILTER contract is untouched. The docopt false-positive mechanism behind
the function-only rule (lowercase locals vs same-named methods) doesn't
transfer: methods stay excluded for bare ids, and the same-file/import
gate + unique-or-drop rules still apply.
Probed on django-rest-framework (~250 files): 559 new references→class
edges, 10/10 sampled genuine (serializer_class = AuthTokenSerializer, the
ModelSerializer field-mapping registry, aliases, ctor args, isinstance).
EXTRACTION_VERSION 24 → 25.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
codegraph_node / codegraph_explore read CURRENT bytes but slice them at
INDEXED line ranges; after an un-synced edit that slice can be a DIFFERENT
symbol's code served under the requested name — isError: false, introduced
by the 'verbatim … do not Read' guarantee. The watcher-based pending (#403)
and degraded (#876) banners cannot cover a project reached via projectPath:
cross-project instances have no watcher, by construction.
Freshness is now verified at the point of emission from data the index
already stores: one stat per rendered file (size + floored mtime, the sync
fast path's own test), sha256 content-hash compare only on stat mismatch
(so a touch/identical rewrite never false-positives), memoized briefly per
handler. On drift:
- codegraph_node: small files ship WHOLE and CURRENT (Read-parity, still
no Read needed); large ones omit the body with an explicit notice
steering to the tool's file-read mode or Read. Location/signature stay,
flagged as possibly shifted.
- codegraph_explore: the whole-file render (already correct by
construction) is kept and flagged; adaptive/skeleton/cluster slicing is
disabled for drifted files — a too-big drifted file is omitted with a
notice instead. The verbatim/do-not-Read header gains a per-file
exception, and a trailing note flags shifted line references (flow,
blast radius, symbol lists).
The guarantee itself is preserved: everything actually rendered is still
byte-accurate — drifted files ship whole or not at all, never as a
possibly-wrong slice. A re-sync of the target project restores normal
output (covered by test).
Adds __setLoadCodeGraphForTests (same seam pattern as __setFsWatchForTests)
so in-process tests can exercise a genuine cross-project open, which
vitest's transform cannot service through the lazy require.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A SIGKILL'd process (the #850 liveness watchdog, OOM, a crash) leaves its WAL
on disk; the next session appends to the same file; and nothing ever truncated
it — PASSIVE checkpoints fold frames but keep the file at its high-water mark,
and the one shrinking path (a clean last-connection close) is exactly what a
killed-daemon world never takes. Observed at 25.6 GB on a 5.46 GB DB, growing
until the disk filled.
- journal_size_limit on every connection: resetting checkpoints now clip the
WAL back to the cap instead of leaving it at its high-water mark.
- healOversizedWal() fired from every DatabaseConnection.open: off-thread
PASSIVE fold + TRUNCATE when the leftover WAL exceeds the cap (64 MB,
CODEGRAPH_WAL_HEAL_MB to override). Single-flight per connection with
bounded retries — concurrent passes defeat each other (each checkpoint sees
the other as a busy reader).
- Daemon/direct MCP watchdogs now pass progressPaths (DB + WAL), extending the
#1231 slow-disk deferral to the long-lived server so a healthy daemon mid
slow statement isn't SIGKILL'd — fewer kills, fewer leaked WALs.
- codegraph status shows WAL size (human + JSON) and warns when it dwarfs the
DB; daemon.log lines and the watchdog kill notice now carry ISO timestamps
so kills can be placed in time.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The standalone bundle's bin dir exposes only codegraph.cmd, and Claude
Code executes UserPromptSubmit hooks through Git Bash, which applies no
PATHEXT — so the bare `codegraph prompt-hook` the installer wrote was
"command not found" (exit 127) on every prompt. Write the platform-correct
spelling, recognize both spellings on uninstall/opt-out, and self-heal an
installer-written entry from the other platform in place on install/upgrade
re-runs (npx/hand-edited variants stay untouched).
Reproduced and validated on the Windows VM: bare form exits 127 under Git
Bash on a standalone-only PATH, codegraph.cmd exits 0; full installer suite
(165 tests, including the new migration coverage) green on Windows + macOS.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A user-level mcp.json entry using ${workspaceFolder} makes VS Code
refuse to start the server in ANY window without a folder open (loose
files, welcome tab), toasting "Variable workspaceFolder can not be
resolved" — recurring error-noise, hit live during validation.
The pin was never needed for VS Code: unlike Cursor, VS Code documents
stdio-server cwd as the workspace folder, and the codegraph server
resolves its project via roots/list with a cwd fallback. Global entries
are now variable-free (`serve --mcp`); local installs keep the absolute
--path. This supersedes the "open a folder" install note from the
previous commit, which is removed again.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
VS Code refuses to start a user-level MCP server whose entry uses
${workspaceFolder} in a window with no folder open, surfacing only a
cryptic "Variable workspaceFolder can not be resolved" toast (hit live
during validation). Global installs now note this up front.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The VS Code Copilot Chat extension writes MCP socket-handoff lock files
into ~/.copilot/ide/ on launch, so `existsSync(~/.copilot)` reported the
Copilot CLI as installed on any machine that merely has the VS Code
extension (caught live on the maintainer's Mac). Detection now counts
the dir as a CLI footprint only when it holds something besides `ide`.
Also: uninstalling a from-scratch install now deletes mcp-config.json
instead of leaving a `{}` husk that would keep detect() reporting the
CLI as installed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds three new installer targets so `codegraph install` can wire the
MCP server into GitHub Copilot surfaces:
- copilot-vscode: .vscode/mcp.json (local) or the VS Code User-dir
mcp.json (global), JSONC-surgical edits, `--path` pinned via
${workspaceFolder} for global installs
- copilot-cli: ~/.copilot/mcp-config.json
- copilot-jetbrains: github-copilot config dir (XDG / %LOCALAPPDATA%)
Detection, install, uninstall, and --print-config are covered for all
three in installer-targets.test.ts, including platform-specific path
resolution.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The Codex installer's `findNextTableHeader` skipped `[[array-of-tables]]` headers instead of treating them as a block boundary, so any `[[...]]` block after `[mcp_servers.codegraph]` in ~/.codex/config.toml was silently deleted on install/upgrade/uninstall. Now treats both `[...]` and `[[...]]` as boundaries, with a small line lexer so header-shaped text inside multiline strings/arrays isn't mistaken for a boundary. Adds round-trip regression coverage (install → reinstall → uninstall) + CHANGELOG entry.
Fixes#1351. Supersedes #624.
Thanks @KtzeAbyss.
Two changes to the watcher path (the always-on daemon every agent
session uses), which previously paid a flat 2s debounce plus a full-tree
scan-diff on every save even though the OS events name the exact files:
1. Adaptive debounce: a pending set of ≤2 files fires after a 300ms
quiet window; ≥3 keeps the full configured window so agent
multi-file bursts coalesce exactly as before. Re-arming preserves
trailing-edge semantics; a user-set CODEGRAPH_WATCH_DEBOUNCE_MS
remains the authoritative upper bound (quick window never exceeds
it, floor 100ms).
2. Scoped sync: watcher-triggered syncs pass their pending paths, and
the reconciler stats exactly those — per-path logic identical to the
full walk (stat pre-filter, hash confirm, the #1240
removal/resurrection flow) — skipping the O(repo) scan and
tracked-load. Strict fallbacks keep the full scan-diff as ground
truth: directory removals (#1285 — the events can't name the
children), empty pending sets (retry paths), and >500-file storms
(branch checkouts, which also self-heal anything event coalescing
dropped). filesChecked counts examined PATHS so a deletion-only
scoped sync can't mimic the #449 lock-unavailable signature.
Measured (warm in-process, the daemon path): dubbo one-file sync work
512→335ms, Swift compiler (27k files) 884→385ms — save-to-fresh-graph
≈0.6-0.7s end-to-end including the quick debounce, from ~2.5-6s
perceived before. Gates: scoped-vs-full dumps byte-identical on dubbo
AND the Swift compiler; watcher suite 30/30 (3 new: scoped pass-through,
dir-removal fallback, quick-fire timing); sync suite 34/34 (4 new
scoped-parity cases incl. delete-resurrection and the lock signature);
full suite 2,696 ×2 with CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Post-R7b store-arc round 1, found by the dubbo warm-wall decomposition
(the cbm bar): resolution's loop-stage profile showed settle=3.0s — the
main thread idling on TWO resolver workers on an 11-core Mac. Pool sizing
logged `size=2 (budget=1068MB)`: memoryBudgetBytes() falls back to
os.freemem() when uncontained, and macOS keeps RAM deliberately full of
reclaimable cache, so freemem reads ~1GB on a mostly-idle 64GB machine.
The memory term then capped the pool at 2 where the CPU term allowed 6 —
the macOS sibling of §7a.1's os.cpus() cpuset-blindness (that round fixed
the CPU term; this fixes the memory term).
Fix: darwinMemoryAvailable() reads /usr/bin/vm_stat once per sizing call
and reports free + inactive + speculative + purgeable pages — what
Activity Monitor calls available, the same reclaimable-inclusive
convention the Linux branch already uses by crediting inactive_file back.
Parse failure → null → freemem fallback; Linux/cgroup and Windows paths
untouched.
Measured (dubbo 4,402 files, warm, caffeinated, n=3 each): pool now
self-sizes to 6 (budget 5.7-6.3GB) — wall 8.62-8.83s vs 9.67-10.87s
baseline, resolution phase 6.9→5.3s, loop settle 3.0→1.9s. Matches the
CODEGRAPH_RESOLVE_WORKERS=6 probe exactly (probe-before-build). Dumps
byte-identical pool-6 vs sequential (441,270 lines). Second consumer
unblocked: the cFnPtr LRU cache cap no longer spuriously degrades to 128
on Macs (its full-cache tier is worth ~60s at kernel scale).
Suite: resolver-pool-sizing gains a darwin-gated reclaimable-pages test +
an off-darwin null pin; full suite 2,689 green ×2 with
CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R7b batch 4 #4 — the FINAL R7b language (docs/design/dart-kernel-port-checklist.md
is the authoritative quirk list). The fourth vendored-grammar-C language,
with a twist: production dart resolved its wasm from tree-sitter-wasms,
whose dart dependency is an UNPINNED github:UserNobody14/tree-sitter-dart —
a routine dependency update would have silently changed dart's grammar.
This PR byte-copies the shipping 0.1.13 artifact into src/extraction/wasm/
(VENDORED_WASM_LANGS += dart) and compiles the same-commit (d4d8f3e337d8)
parser.c/scanner.c in the kernel — table identity proven by the
kernel-grammar-parity row. crates.io tree-sitter-dart is the nielsenko
fork (different lineage) — rejected.
The center of gravity is THE SIBLING-BODY DOUBLE-WALK, reproduced
bug-for-bug: dart attaches every function/method body as a NEXT SIBLING of
its signature, and the TS walkers consume each body TWICE — once via
resolveBody (attributed to the function/method) and once via the enclosing
generic walk (attributed to the file/class). Duplicate local-function
nodes with the SAME id under different parents, duplicated
calls/instantiates refs, and file/class-attributed fn-ref twins all emit
in the exact observed interleave (a dedicated fixture pins the
duplicate-id rows; the bloc kind-census spot-check pins the counts).
Also preserved (probe-pinned): the extractBareCall selector matrix (the
first callTypes=[] language — cascades completely invisible, `?.` encodes
like `.`, the `ConfigT.load()` calls+references double emission with no
callee-of-call skip, capitalized-chain `Foo.create().run` re-encode,
const-object callee names); the constructor hooks (unnamed ctor skipped,
named ctors/factories renamed to the CTOR name with the class as
returnType, `@override (T) m()` record-misparse rescued by class-name
validation); operator methods minting `method "<anonymous>"`;
static_final_declaration constants via the visitNode hook while instance
fields mint NOTHING; the prefixed-return-type prefix bug (`other.OtherClass
f()` → returnType `other`); enum `with` mixins silent vs `implements`
working; anonymous extensions named after the ON type; deferred imports
invisible; named-argument callbacks NOT fn-ref-captured (the Flutter
`onPressed:` idiom — future accuracy PR, TS-side first); `async*`/`sync*`
NOT async; value-refs with the LIVE dart sibling-body pull and the
`$X`-vs-`${X}` interpolation asymmetry; dartdoc kept in all three comment
forms with the annotation-broken chain.
Gates: parity sweeps first-run 0-diff on shelf/bloc/flutter — 5,815 clean
files byte-parity, deferrals 10/21/1341 ≈ the survey's 10/21/~1340
(both-arm grammar reality: empty object patterns — the sealed-class
idiom — and unnamed `library;` dominate; --max-deferral 0.3); full-init
dumps byte-identical ×3 (shelf 7,959 / bloc 40,026 / flutter 1,855,319
dump lines); bloc per-kind node census identical across arms (the
double-walk duplicate rows survive the store identically);
kernel-dart-parity suite (7 fixtures + in-memory CRLF variants +
double-walk duplicate-id pin + generated-file skip pin + two defer pins);
full suite 2,688 green ×2 with CODEGRAPH_KERNEL_EXPECT=1.
DEFAULT_ROUTED += dart (20 langs — R7b COMPLETE).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R7b batch 4 #3 (docs/design/scala-kernel-port-checklist.md is the
authoritative quirk list). The third vendored-grammar-C language and the
biggest grammar in the tree (35MB parser.c): the vendored wasm is
tree-sitter/tree-sitter-scala master@0aca5d0a6f — a post-v0.26.0 generation
sync that is not a release (the 0.26.0 crate is 30 states BEHIND, so a
crate pin would be a silent downgrade). NO wasm change: production has
parsed with this exact revision since #91 — the kernel-grammar-parity row
(ABI 15, 26,650 states, 32 fields, id-by-id tables) is the whole alignment
proof.
Preserved bug-for-bug (all probe-pinned): the leak-through asymmetries —
extension methods mint NO nodes (first def's body calls leak to the
enclosing scope, later defs invisible, and the braced form resolves its
body field to the `{` TOKEN via first-match-wins field lookup → whole
extension invisible); anonymous `new T { … }` template_body members leak to
the enclosing scope (findAnonymousClassBody misses template_body); the
bodied-vs-bodiless class asymmetry (bodiless headers walk class_parameters
→ default-value calls emit FROM the class; bodied ones never see them) —
plus first-segment import names (`import com.example.C` → `com`), the
val/var hook keyed on the enclosing-definition NODE TYPE (object vals →
constants/value-ref targets, class/trait/enum/given vals → fields) with
consumed initializers, every def routed through extractMethod with the
top-level function fallback, nested defs in bodies minting NOTHING (the
inverse of kotlin) while body-local classes extract fully, curried
signatures keeping only the FIRST parameter list (type params win the
`parameters` field), enum cases positioned at the CASE node with invisible
params/extends tails, extends with-chains via scalaBaseTypeName,
`@deprecated(args)` decorates, the #750 capitalized-chain re-encode
(`WidgetS.create().render`), literal-receiver silence, static-member reads
AND writes, infix invisibility, `derives` silence, scaladoc retention with
the CRLF `\r` pin, full value-reference machinery (shadow prune, last-wins
same-name targets, `$X`/`${X}` interpolation reads), and SCALA_SPEC
fn-refs (bare ids + postfix eta unwrap + varinit, var-init non-capture).
Gates: parity sweeps first-run 0-diff on os-lib/cats/scala3-compiler-src/
scala3-library-src — 1,935 clean files byte-parity, deferrals 0/15/57/116
matching the survey's predictions exactly (scala-3's PHANTOM hasError
files — flag-true, zero ERROR nodes, capture-checking `^` — defer on the
FLAG); full-init dumps byte-identical ×3 (os-lib, cats, scala3 whole-repo
950,889 dump lines); kernel-scala-parity suite (9 fixtures + 9 in-memory
CRLF variants incl. Scala-3 indentation through the external scanner +
phantom/real-error defer pins + first-segment/namespace/value-ref pins);
full suite 2,669 green ×3 with CODEGRAPH_KERNEL_EXPECT=1
(kernel-scaffold's stays-wasm example moved scala → pascal).
DEFAULT_ROUTED += scala (19 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R7b batch 4 #2 (docs/design/lua-luau-kernel-port-checklist.md is the
authoritative quirk list). ONE walker for both dialects (ccpp precedent) —
the differences are exactly four: luau's type_definition aliases, the
`export `-slice isExported hook, the return-type signature suffix, and the
grammar handle.
Grammar prep is kernel-side only, no wasm change: lua is the SECOND
vendored-grammar-C language (the vendored wasm is the v0.4.1 tag, a revision
not on crates.io — tag artifacts compiled via build.rs, shas pinned); luau
is a plain crate pin =1.2.0 whose tarball is sha-identical to the tag (the
swift tag≠crate divergence does not recur). Grammar-parity rows replace the
bump gate entirely.
Preserved bug-for-bug (all probe-pinned): the require/visitNode-hook
ASYMMETRIES (top-level requires — including inside top-level if/for/while —
mint import nodes while the identical body-level statement emits
`calls "require"`; top-level `local x = foo()` initializers are invisible
while global `x = foo()` calls emit), the BFS string-win inside require args
(`require(script:WaitForChild("Kid"))` → import Kid) and Roblox instance
paths, receiver-QN methods (`M.sub.deep::chained`, `_G::installed`,
stack-QN nested globals like `render::leakedGlobal`), the raw-text callee
world (colon forms with `self` never stripped, bracket callees,
newline-glued chains byte-verbatim, the `(handler)` paren-conversion),
LUA_SPEC function-as-value capture with the `M.cb = cb` param-storage skip
and first-occurrence dedupe, LuaDoc `---` keeping a leading `- ` plus
`--!strict` joining docstring chains (block-comment docstrings keep interior
CRLF bytes), variable nodes at the IDENTIFIER with positional value pairing,
duplicate same-(kind,name,line) ids, and the lua↔luau isExported wire
divergence (lua functions: flag absent; luau functions: present-false;
methods: absent in both; variables: present-false in both; `export type`:
true).
Gates: parity sweeps first-run 0-diff on kong/lazy.nvim/lua-resty-core
(lua) + lune/Fusion (luau) — 1,734 clean files byte-parity, deferrals
1/0/0/3/8 matching the survey's both-arm predictions exactly (kong's 1 = a
deliberately invalid fixture; luau's = grammar-inherent generic type packs
and default type params); full-init dumps byte-identical kernel-vs-wasm ×4
(kong 157,650 dump lines); kernel-lua-parity suite (both torture fixtures +
in-memory CRLF variants + glue-chain, duplicate-id, and cross-dialect defer
pins + kernel-arm wire-flag pins); full suite 2,647 green ×2 with
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += lua, luau (18 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
R7b batch 4 #1 (docs/design/r-kernel-port-checklist.md is the authoritative
quirk list; survey + probe record therein). The lightest-shared-surface,
heaviest-hook port: languages/r.ts works entirely through the visitNode hook
(every type list empty except callTypes:['call']), so the walker is a file
node + a faithful hook transcription + the generic extractCall + pre-order
recursion — four shared machineries (value-refs, static-member reads, type
annotations, fn-ref capture) are dead by language gates and stay dead.
Grammar prep is the first true no-op of the arc: the crates.io tree-sitter-r
1.2.0 tarball ships parser.c AND scanner.c sha-identical to the r-lib v1.2.0
tag the vendored wasm was built from — crate pin only, no wasm change, no
bump gate; kernel-grammar-parity gains the r row (ABI 14, same-revision).
Preserved bug-for-bug (all probe-pinned): calls "return" on every return(x)
(named node in v1.2.0), the import quintet's silent dynamic-arg consumption
vs class/generic fall-through asymmetry, library(help = pkg) importing the
named arg, class-idiom variable suppression by callee name, chained/right-
assign/precedence-ghost gaps, env$fn body-leak-to-file, raw-text callees
verbatim (pkg::fn, obj$meth, "strfn" quotes kept, (handler) conversion),
duplicate same-(kind,name,line) ids, roxygen dropped entirely, UTF-16
columns/slices.
Gates: parity sweeps first-run 0-diff on AnomalyDetection/dplyr/ggplot2/
shiny (838 files; deferrals exactly 0/0/0/1 — the 1 is the moustache-
template pseudo-R file, both-arm) — kernel-parity.mjs gained lowercased-
extension matching so .R files sweep (matches detectLanguage routing);
full-init dumps byte-identical kernel-vs-wasm on dplyr/ggplot2/shiny;
kernel-r-parity suite (torture fixture + in-memory CRLF + BOM variants +
defer pin + kernel-arm quirk pins); full suite 2,638 green ×2 with
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += r (16 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sixth R7b port — the T1½ batch finale. Checklist-first recipe
(docs/design/kotlin-kernel-port-checklist.md, 1,121 lines, dist-extractor
ground truth); parity passed FIRST RUN on all three repos.
THE NOVEL MECHANISM — vendored-grammar-C (the §4 tracker's prescription,
first use): the crates.io tree-sitter-kotlin 0.3.8 pins `tree-sitter >= 0.21,
< 0.23` (the kernel links 0.25) and tree-sitter-kotlin-ng is a DIFFERENT
grammar (8 fields vs 0, renamed kinds — extractor-breaking), so no crate dep
is possible. The fwcd 0.3.8 tag's sha-matched parser.c + scanner.c are
vendored into codegraph-kernel/grammars/kotlin and compiled by build.rs (cc),
exposed via tree-sitter-language::LanguageFn. The wasm re-vendor is
behavior-NEUTRAL (0 CST/error disagreements across 1,984 gate-repo files;
old-vs-new full-init dumps byte-identical ×3) — a reproducibility re-vendor,
ABI stays 14.
Walker firsts: extension-function receivers (getReceiverType →
`WidgetK::extend` QN OVERRIDE with no package prefix, the qualified-receiver
`com::qext` first-segment bug, and the owner-contains fallback that excludes
`interface` kinds and is source-order dependent) and extractModifiers
(expect/actual platform modifiers → the node DECORATORS wire field on every
created node — the KMP synthesizer's feed, incl. `actual typealias`).
Preserved bug-for-bug: the FIELD_COUNT-0 dead cluster (no signatures, ZERO
type-annotation refs), hook-consumed property initializers emitting nothing
(incl. `by lazy {}`), the bodiless-vs-bodied class header asymmetry, enum-
entry bodies being invisible, KDoc never a docstring AND chain-breaking,
comment-gluing into import/package extents, `@Anno(args)` emitting nothing
while `@Marker` decorates, zero instantiates refs, the paren-then-lambda
`trailing()` garbage callee, text-includes visibility/suspend false
positives, and the packaged-file value-ref target drop. The fun-interface
misparse-recovery hook is DEFER-SHIELDED (every such file has_error) and
deliberately not ported. The swift-sweep lesson pre-applied: the shared
`assignment` shadow-prune case is implemented alongside the
property_declaration case.
Gates: sweeps 0-diff okio 299/322, okhttp 531/580, kotlinx.coroutines
1031/1082 (deferrals exactly the predicted 23/49/51 — both-arm grammar
reality incl. PHANTOM hasError files with complete CSTs; the kernel trusts
the flag); full-init dumps byte-identical ×3 (46.5k/108.9k/92.3k lines); KMP
expect/actual synthesis IDENTICAL across arms (412 edges on
kotlinx.coroutines — the tracker's KMP validation); kernel-kotlin-parity
suite (torture reflowed off the phantom shapes + .kts script + CRLF variants
+ fun-interface and phantom defer pins) + kotlin grammar-parity row (the
C-build ↔ wasm table identity proof); full suite 2,633 green ×2 under
CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += kotlin (15 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Fifth R7b batch-3 port, checklist-first recipe
(docs/design/swift-kernel-port-checklist.md, 1,056 lines — the largest of the
arc, with a built-extractor-validated emission pin and a childForFieldName
truth table).
Grammar bump first, validated standalone: tree-sitter-wasms ^0.4.0 (ABI 13) →
crate 0.7.3 — with a provenance twist: the wasm is built from the CRATE
TARBALL's src/ (alex-pinkus keeps generated files off main and the
0.7.3-with-generated-files tag ships an older ABI-14 generation that can never
sha-match; grammar.json rules are JSON-equal; the tarball is byte-for-byte
what the kernel's cargo build compiles — table identity by construction).
Older crates evaluated and rejected: clean-parse shapes are byte-identical on
0.7.3 (53-line CST battery diff, all inert), so an older pin buys nothing and
loses the macro-era wins. Delta = error-set membership (63 old-error files
parse clean: swift-testing #expect, #Preview/#GET macros, package access,
typed throws — vapor 23.1%→9.3%; 21 NEW-only regressions in 3 probed
construct classes) + two gate-found categories: docstring boundaries near #if
directives (7 clean files, docstring-field-only — verified mechanically) and
array-literal-callee call refs (2 refs, 1 file). Every hunk classified via
the error-union rule + parked-ref↔edge ripple pairing.
Walker (the arc's biggest) centers on the #1020 DEDICATED property branch:
computed properties → property nodes with the getter walked under the
property (SwiftUI body), static let/var → constant/variable, stored → field,
decorator/type-annotation/@Siblings-attr-arg refs all attached to the
ENCLOSING TYPE, stored initializer calls attributed to the class. Preserved
bug-for-bug: the never-resolving 'parameter' field (zero param type refs,
zero signatures), present-false isAsync, open→internal visibility,
everything-is-extends inheritance (first type_identifier per specifier), no
instantiates refs ever, subscript reads as `calls arr`, `defer` as `calls
defer`, multi-case enum entries minting only the first case, /** */ block
docs ignored AND chain-breaking, init/deinit/subscript minting no nodes with
visitNode-routed bodies (calls → class, static reads → nothing), multi-
segment extension resolveName, sugar extension names, the #selector shapes,
and the value_argument label-forward skip. ONE fix found by the sweep (then
pinned in the fixture + checklist): the shared `assignment` shadow-prune case
is swift-live — declared-then-assigned `let X: T` prunes X as a value-ref
target.
Gates: sweeps 0-diff Alamofire 89/98, vapor 224/247, swift-nio 407/554
(--max-deferral 0.3 — swift error incidence is 9–27% on BOTH arms,
structural; every deferral count matches the survey's table exactly);
full-init dumps byte-identical ×3 (31.9k/20.7k/126.3k lines); the Alamofire
census reproduces property=348 (the #1020 number) on the kernel arm;
kernel-swift-parity suite (206-line torture + CRLF + the #if-between-enum-
cases defer fixture) + swift grammar-parity row; full suite 2,626 green ×2
under CODEGRAPH_KERNEL_EXPECT=1. DEFAULT_ROUTED += swift (14 langs).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
First R7b language port. Grammar: tree-sitter-rust pinned =0.24.2 + wasm
vendored from tag 77a3747 (parser.c/scanner.c sha-matched against the
crates.io tarball), replacing the 2023 ABI-14 tree-sitter-wasms build —
the bump alone is precision-positive on the wasm path (receiver-qualified
instance-method resolutions replace ambiguous bare-name matches; node
sections byte-identical on ripgrep/tokio).
Walker mirrors the TS reference bug-for-bug per
docs/design/rust-lang-kernel-port-checklist.md (survey artifact): dead-code
isAsync, impl-pushes-no-scope, the impl-Trait-for-Generic<T> trait-receiver
quirk, phantom const identifiers, use-binding triple emission,
wildcard-use-emits-nothing, scoped-supertrait drop, chained-call re-encode
gated on scoped_identifier, Rocket route macros body-only.
Gates: parity sweeps 0 diffs — ripgrep 101/101, tokio 790/790,
rust-analyzer 1217/1488 (271 deferrals are token-macro-table sources that
error on BOTH arms — grammar-inherent); full-init dump-diffs byte-identical
on all three (3,857 / 13,440 / 39,030 nodes); kernel-rustlang-parity suite
(torture + CRLF + defer) in npm test; full suite green x2 with
CODEGRAPH_KERNEL_EXPECT=1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Task #5 step 2. The fuse-then-link refactor (#1364) left the extraction
sweep as a clean per-file boundary: raw text in → collected facts out.
This ports that sweep to the native kernel: `cfnptr_scan_files`
(codegraph-kernel/src/cfnptr.rs) strips and scans a batch of 16 files
per NAPI call, and the TS side only reads files, ships batches, interns
the returned facts, and resolves include paths. The JS sweep remains as
the fallback (no binary, feature detection against older binaries,
CODEGRAPH_KERNEL=0, or CODEGRAPH_KERNEL_CFNPTR=0).
Parity discipline: the JS regexes are the spec, so the scanners are
hand-rolled byte machines reproducing that engine — ASCII \w/\b next to
UNICODE \s (NBSP/U+2000-200A/FEFF decoded from UTF-8), alternation
order, lastIndex resume, and the observable backtracking dimensions
(INIT/ARRAY modifier and struct/star/bracket optionals, DISPATCH's
greedy segment loop); greedy shortcuts only where backtracking provably
can't rescue a match. The native stripper blanks per UTF-16 code unit,
so its output is string-identical to the TS stripper — pinned by a new
kernel arm on the strip differential oracle (fixtures + 500 seeded
random cases).
Gates, all green: new differential suite (adversarial fixture project —
CRLF, NBSP, continuations, decoy strings, unterminated comments,
backtracking shapes — indexed native-vs-JS: identical edge streams,
plus a record-level scanner check); repo differential on
git/redis/vim/SameBoy (identical, 705/852/433/180 edges); probe-hash on
the live linux kernel DB reproduced f6e1713d… (279,335 rows); linux
init counts exact 2,049,153/6,413,518; dump sha 6dd1185b… reproduced
(10,446,478 lines); full suite green ×2 (153 files / 2588 tests).
Measured (8c cg1212, quiet host): cFnPtr sub A=47.9s B=1.1 C=40.9
D=24.1 E=36.8 = 150.9s vs step 1's 179s and the pre-arc 230s (−34%
cumulative); the sweep itself halved (94.5→47.9s, JS strips
132.4k→68.9k). callback-synthesis phase 199.9→171.1s. E's attributed
wall grew from overlap shift under parallel synthesis; the phase total
is the honest number. Full record: plan §7a.10.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three measurements before any port. The stripCStyle split('') rewrite
(byte-identical segment-builder) measured 1.0× on 15.1M chars of linux C
— V8's ~73MB/s scan rate IS the cost, and 78s ≈ 4 strips/file × that
rate: the lever is the redundancy, not the scanner. Rewrite reverted;
the differential oracle test ships so any future rewrite stays pinned
byte-identical. E's regexes alone run at ~46MB/s (~30s of its 95s; the
rest is per-match logic and slicing).
Re-ordered attack recorded in §7a.8: step 1 = TS fuse-then-link refactor
(strip once per file, collect raw matches + declared-type tables,
text-free global linking; ≈ −70-90s, parity via collector insertion
order + the §7a.4 probe-hash gate); step 2 = native per-file extractor
behind the same boundary (raw disk text — no preParse interaction).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Census-driven cut of the top-ranked post-R7a lever. All passes TS-side,
C-only (preParseCSource), shared by both arms:
- parameterized-annotation whole-blank (__free/__printf/__counted_by/
__bpf_md_ptr…; extends through a stranded field `;`)
- type-keyword-arg scanner (kzalloc_obj(struct T), list_entry, multi-line
continuations behind nested-paren args; bounded hand scanner, head
exclusions + call-vs-declaration guard; blanks trailing stars)
- static/extern CAPS-macro declaration lines at any scope; the initialized
form is REWRITTEN to its expansion (name/tail keep exact offsets)
- va_arg qualified-type blank; GNU named-variadic #define dots-only blank
(post-restore); sandwiched notrace-family; C23 auto; multi-line
iterator-macro spans (hlist_for_each_entry_rcu + lockdep arg)
- word list += cacheline family (2- and 4-underscore spellings) + 10 more
census-confirmed annotations
Gates: five-repo parity sweeps 0 diffs (git deferral 16.1→12.2%, redis
25.3→24.1%, fmt/protobuf unchanged); linux full-tree both arms
2,049,153 nodes / 6,413,518 edges (+858/+6,585 vs R7a) with byte-identical
dumps (10,446,478 lines, sha256 6dd1185b); kernel-arm parse-loop 356→306s
at 2c; suite 2517 green under CODEGRAPH_KERNEL_EXPECT=1. Honesty note
recorded in the docs: error recovery was already salvaging most SYMBOLS on
deferred files — the graph win is relationships + phantom cleanup, and the
unreleased CHANGELOG entry was rewritten off the sweep-subset framing.
Also records §7a.5: post-R7a 8-core cg1212 re-run 16.4min (was 18.3min);
8c parse sits on the single-writer floor, so the <10min-on-8c gap re-ranks
to the per-ref resolution path.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parity: 0 diffs on redis/git/fmt/protobuf/ALS sweeps; full-init dumps
byte-identical on all five + linux at kernel scale (10.4M dump lines,
same sha256 both arms). Linux 2c/6GB envelope: kernel-arm 19.1min vs
wasm-arm 22.9min (parse 356s vs 435s) on a much richer graph (the new
blanks recover error-swallowed code: git 2x nodes, linux kernel/+mm/ 3x).
Deferral guard corrected by measurement (C/C++ error incidence 9-42%;
--max-deferral flag); defer-reuse memo kills the 3x re-blank/re-parse
cost deferred files paid.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The §7a.2 per-ref profile overturned the assumption the whole arc was
built on: resolveOne owns only ~93s of the kernel-scale ~433s batch loop.
Loop-stage attribution (CODEGRAPH_RESOLVE_PROFILE, shipped here) named the
rest: backpressure folds 111.2s, count guard 93.9s, batch reads 54.6s,
deletes/inserts/marks ~84s, settle 85.7s.
- Non-progress guard O(remaining)→O(1): the per-batch COUNT(*) walked every
remaining pending row (O(N²/batch) per run, 93.9s). The cleanup queries
now return summed SQLite , and zero-removals-from-claimed-work
is the guard signal — the DIRECT evidence the count diff inferred (a
mismatched-name resolver makes keyed cleanup no-op ⇒ changes=0). A real
COUNT runs only on that suspicious path and arbitrates exactly as before.
- Batch reads OFFSET→keyset (54.6s→O(batch)): OFFSET re-walked the
accumulated failed-row prefix every read; seeking past the last-seen
rowid is prefix-independent and enumeration-order identical.
- WAL valve caps scale with DB size (env still wins): every fold re-writes
hot pages (#1231 in bounded form — 111.2s at the flat 256MB cap);
soft=clamp(dbSize/4, 256MB, 2GB) trades ~4× fewer folds for a transient
WAL ≈ project size.
- CODEGRAPH_RESOLVE_PROFILE: per-outcome resolveOne histogram + loop-stage
attribution, main + workers, off by default.
Gates: dubbo dump byte-identical; suite 2,491 passed / 4 skipped (kernel
required). Kernel-scale payoff run lands in the plan doc next.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Four §7a.1 instrumented-run findings, each measured:
1. File-size trigger + truncate-at-barrier: a fully-backfilled WAL still
grows the FILE without bound — the writer only restarts at frame 0 when a
commit finds zero reader marks, which the instrumented run showed never
happens (file marched 361→721MB through two COMPLETE backfills; 22GB by
phase end). backpressure() now also trips at 4× the soft cap on raw file
size and TRUNCATEs at the parked barrier; the timer path truncates
opportunistically after complete backfills. Dubbo peak: 251MB → 69MB at
the same 16MB valve; dumps byte-identical under aggressive folding.
2. cgroup memory credit: memory.current counts reclaimable page cache — a
post-parse container read 57MB of headroom on a 6GB box and silently
disabled the pool. inactive_file is credited back (the docker-stats
working-set convention); the same run now reads a sane 4.4GB budget.
3. Pool at 2 cores reversed: sequential resolution measured FASTER than
pooled-6-on-2 at kernel scale (853s vs 1,150s), and synthesis is
Amdahl-bound by cFnPtrEdges (306s of 358s) so pooling it bought nothing.
cpuCap = min(ap−1, 6), no floor: ap=2 → sequential is the fast path.
4. Parse floor of 2: one parse worker at a 2-cpuset measured 34% slower
(493s vs 369s) — main + store-worker don't fill the second core. Floor
restores the baseline (373.5s measured).
Plus the observability §7a.1 burned three 25-minute cycles for: valve
armed/fire/timer-pass/heartbeat lines, checkpoint-worker error capture,
pool sizing decisions (incl. the disabled path), backpressure-hook
presence — all behind CODEGRAPH_SYNTH_TIMINGS / CODEGRAPH_WAL_VALVE_DEBUG.
Suite: 2,490 passed / 4 skipped (kernel required). Kernel-scale record
runs with this build follow in the migration plan §7a.1.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three §7a.1 run-1 lessons (kernel-scale 2c/6GB: EXIT=137, WAL 22.2GB with
the backpressure hook DEPLOYED):
1. TRUNCATE at parked barriers: a completed passive backfill bounds the
un-checkpointed backlog but the FILE only stops growing when a commit
finds zero readers holding WAL marks — rare while pool workers cycle
(dubbo debug baseline: file climbed monotonically through six completed
pass-1 backfills). At a parked barrier the no-reader window is
guaranteed, so chop the file there with wal_checkpoint(TRUNCATE)
(off-thread, 2s busy_timeout — a racing reader degrades it to a no-op).
2. Futility latch: when backfill gives up (pinned reader), parking again at
every over-cap boundary burns a 20-pass checkpoint attempt — each a
worker thread + fresh connection against a multi-GB DB — per batch. Two
consecutive give-ups now disable parking for 60s; a pinned phase degrades
to pre-valve behavior instead of OOM-amplifying.
3. CODEGRAPH_WAL_VALVE_DEBUG=1 surfaces valve decisions without the
caller's verbose plumbing, and give-up lines print under
CODEGRAPH_SYNTH_TIMINGS — run 1 failed silently because give-ups were
verbose-gated.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Pool sizing used os.cpus().length, which enumerates the HOST's CPUs: inside
a 2-CPU cpuset it sized 6 resolver workers (the §7a.1 false-'sequential'
premise) and 8 parse workers, and at true 8-core concurrency six ~1GB
workers OOM-killed a 7GB container (oom_kill=5) mid-synthesis — sizing had
no memory term and no override knob.
resolvePoolSize (pure, matrix-tested): explicit CODEGRAPH_RESOLVE_WORKERS
override (0 disables, cap 16); CPU term max(2, min(availableParallelism-1,
6)) — cpuset-honest, floored at 2 so true 2-core boxes keep pooled
synthesis's ~2×; memory term floor(budget*0.7 / clamp(0.2*dbSize, 256MB,
1.5GB)) with budget = min(freemem, cgroup v2/v1 headroom). Parse pool's
core input switches to availableParallelism. Dev machines are unchanged
(still 6 workers); the 8c/7GB kernel-scale container now sizes 4.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
At kernel scale the pooled resolution/synthesis superphase grew a 22GB WAL
on a 4.6GB DB (cg1212, §7a.1): autocheckpointing is deferred for the run,
and the valve's timer-driven passive checkpoints stay perpetually partial
against the pool's continuous reads — no mechanism ever completed a
backfill, so the WAL accreted the whole phase's write volume, blowing disk
and feeding page-cache pressure into the 8-core/7GB container OOM.
The valve's writer-side backpressure() hard-cap backstop existed but was
wired only into the PARSE orchestrator. Thread it into the resolution batch
loop at the double-buffer's one pool-idle boundary (batch settled, next not
yet fanned out), after the edge-index recreate, and through the synthesis
insert loops. Parked there, the backfill completes; readers re-enter at
SQLite's backfilled mark and the next persist commit wraps the WAL.
Dubbo validation, same build: valve@16MB peak WAL 251MB (floor = the
single-transaction edge-index recreate) vs defaults 914MB; dumps
byte-identical (441,270 rows); wall unchanged (11s). Suite 2,479 green.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
On CRLF checkouts (every Windows autocrlf clone) the JS reference's
block-continuation strip /^\s*\*\s?/gm finds a line start after the \r and
its greedy \s* consumes the \n, leaving a bare \r in the docstring; the
kernel's (?m)^ pass matched after \n only and kept \r\n. Caught by the O2
Windows VM leg (6 kernel-tsjs-parity failures), reproduced on macOS by
CRLF-converting the fixtures.
js_multiline_strip now replicates the JS anchor set (\n, \r, U+2028, U+2029)
for all five line-marker passes; CRLF variants of every torture fixture are
pinned in kernel-tsjs-parity, derived in-memory so nothing can normalize
them away.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Python (codegraph-kernel/src/python.rs) and Go (src/go.rs) join the
native kernel, mirroring the wasm extractors bug-for-bug. Python:
decorated_definition docstrings/decorators (decorates refs only for
bare-identifier decorators — the call-kind quirk), function-in-class →
method, module-level assignments always extract as variable, from-import
per-name binding refs, self.x fn-ref candidates as bare names. Go:
receiver methods with Recv::name qualified names + contains edges to the
first earlier struct of that name, type_spec struct/interface
classification with embedding→extends and interface method nodes,
composite-literal instantiates keeping the package qualifier, top-level
var/const initializer walks attributed to the declared symbol (#693),
2-hop field chains (#1276), New().Method() re-encode (#645/#608), and
the GO_SPEC fn-ref layers.
Grammars: tree-sitter-python 0.23.6 + tree-sitter-go 0.23.4 crates, with
wasm vendored from the same tags (parser.c sha-matched) — both were
2023-era in tree-sitter-wasms.
Gates: extraction sweeps 100% (flask 83/83, django 3,035/3,038 +3
error-file deferrals, gin 99/99, prometheus 978/979 +1); full-init
dump-diffs byte-identical on flask (10,833 rows), gin (17,540), django
(360,794), and prometheus (213,758); torture fixtures enforced in npm
test. DEFAULT_ROUTED now covers typescript/tsx/javascript/jsx/java/
python/go. Full suite: 2,471 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Java joins the native kernel (codegraph-kernel/src/java.rs), mirroring
the wasm extractor's Java paths bug-for-bug: package namespaces,
imports, javadoc, annotations→decorates, type_list inheritance,
static-final constants, enum constants, anonymous classes (including
the TS side's 0-based-line quirk on the extends ref), method_invocation
calls with the this.field unwrap and the Foo.getInstance().bar() chain
encoding (#645/#608), static-member value reads, method-reference
fn-refs (#756), value-reference edges, and the full Lombok member
synthesizer (#912: Getter/Setter/Data/Value/Builder/ToString/
EqualsAndHashCode/Slf4j-family with taken-member dedup). The shared
docstring/textutil modules moved to crate level. Grammar:
tree-sitter-java 0.23.5, with the wasm grammar vendored from the same
tag (parser.c sha-matched) replacing tree-sitter-wasms' 2023-era build.
Gate (plan §4c): extraction sweeps 100% — gson 262/262, retrofit
341/341, dubbo 4,048/4,048 — plus a Java torture fixture in npm test;
full-init dump-diffs byte-identical on gson (49,766 rows), retrofit
(62,735), and dubbo (441,266 rows); all R2/R3 repos re-verified; Linux
container runs all 23 kernel tests green under CODEGRAPH_KERNEL_EXPECT=1.
The gate caught a real cross-language bug: fn-ref dedupe and value-ref
self-target checks must compare node ID STRINGS, not node-table rows —
ids collide for same-(kind, name, line) nodes, which minified one-line
bundles hit routinely (retrofit's website JS exposed it; latent in the
TS/JS walker since R2, never released). Fixed in both walkers.
Benchmark honesty: dubbo fresh-init on an 11-core Mac is ~flat
(parse-loop wall 5,020→4,394ms; total ~11.3s both arms) because that
wall is main-thread-bound (reads + store), not worker-CPU-bound — the
§6 expectation assumed otherwise. Where worker CPU binds the kernel
delivers: dubbo on a 2-CPU/6GB container drops 27.8-28.6s → 22.3-22.8s
(~1.25×). The identified lever for the many-core headline is decoding
kernel buffers directly into store rows (skipping per-node JS object
materialization); the buffer contract already carries everything.
DEFAULT_ROUTED now includes java. Full suite: 2,467 tests pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate evidence (docs/design/rust-kernel-migration-plan.md §4b):
- Graph parity, byte-identical (stronger than the §5 ≤0.5% bar): full
codegraph-init dump-diffs kernel-vs-wasm on express (13,712 rows),
excalidraw (89,898), and vscode (2,378,238 rows) — identical bytes.
Python control repo (flask) identical + timing unchanged. The parity
harness is now ORDER-sensitive (emission order drives rowids, which
drive resolution order) and dumps come from the new
scripts/dump-graph.mjs (natural keys, no rowids/timestamps).
- The one real find, caught by the vscode tier: tree-sitter error
RECOVERY is encoding-dependent — byte-identical grammar sources and
the same core (0.25.10) recover erroring files differently under
UTF-8 (native) vs UTF-16 (web-tree-sitter) parsing; proven by
reproducing the wasm tree with a native UTF-16 parse. Policy: the
kernel defers any file whose tree has_error() to the wasm extractor
(silent 'defer:' signal, per file) — parity by construction on
erroring files (incidence 0-0.42% across the gate repos), and the
harness fails if deferrals exceed 10% so a broken kernel can't hide
behind the fallback.
- Retrieval invariants: canonical excalidraw flow (mutateElement →
renderStaticScene) connects end-to-end on the kernel-indexed graph;
synthesized-edge families present. Agent A/B is vacuous under
byte-identical DBs (same justification as #1320-#1322).
- Perf: vscode init 105.4s → 82.1s (1.28×) on an 11-core Mac;
excalidraw on a 2-CPU/6GB Linux container (the CI-runner envelope)
6.2-7.1s → 4.3-4.8s (~1.5×). Linux arm64 in-container build: all 22
kernel tests green under CODEGRAPH_KERNEL_EXPECT=1. Windows VM leg
deferred (VM stopped; prlctl start needs Parallels Pro) — benign: a
missing .node falls back to wasm, and the release matrix builds and
gates the win32 prebuilds.
- Full suite: 2,465 tests pass WITH default-on routing, so the entire
extraction corpus now exercises the kernel for TS/JS wherever a
.node is staged.
DEFAULT_ROUTED = {typescript, tsx, javascript, jsx}. Override:
CODEGRAPH_KERNEL_LANGS (replaces the set) / CODEGRAPH_KERNEL=0 (kill).
Changelog entry added under [Unreleased].
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replaces the R1 seed .scm emitter with a bespoke Rust walker
(codegraph-kernel/src/tsjs/) that mirrors TreeSitterExtractor's TS/JS
paths function-for-function: declarations (incl. #808 field/property
classification), qualified names, docstrings (#780 wrapper climbs),
signatures, imports/re-exports + per-binding refs, calls with
receiver-qualified callees (#1230 literal-receiver skip), instantiations,
decorators, inheritance, type annotations (#381), type-alias members +
tuple contracts (#359/#634), React component recognition (#841
forwardRef/memo/styled), object-of-functions / zustand-through-middleware
/ RTK Query endpoints + generated hooks / vuex + pinia store shapes,
function-as-value capture with the flush gate (#756), and value-reference
edges with the shadow prune (#895/#897). The generic query emitter is
deleted — extraction parity needs logic .scm can't express; future
languages get walkers too (migration plan §4a).
Positions and JS string-slice semantics are emitted in UTF-16 code units
natively, so kernel output is byte-identical to web-tree-sitter's — no
column diff class exists.
Parity evidence (macOS): scripts/kernel-parity.mjs (full-object multiset
diff per file) — this repo 353/353 files, excalidraw 643/643 (10,650
nodes / 10,726 edges / 68,307 refs), plus torture fixtures checked into
__tests__/fixtures/kernel-parity/ and enforced in npm test by
kernel-tsjs-parity.test.ts. The strict compare caught one real decoder
bug the loose harness missed: refs must NOT carry denormalized
filePath/language at the extractFromSource seam (the store fills them).
Perf: extraction 2.6× single-thread on excalidraw (487ms vs 1,255ms,
identical outputs). Routing stays opt-in (CODEGRAPH_KERNEL_LANGS) until
the R3 equivalence gate (large repo, DB dump-diff, retrieval invariants,
agent A/B, Linux/Windows) passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 0 of the Rust extraction-kernel migration (docs/design/
rust-kernel-migration-plan.md, now checked in with §3a recording the
shipped state):
- codegraph-kernel/ napi-rs crate: extractFile(path, content, language)
→ five flat buffers (meta/nodes/edges/refs/arena), one JS boundary
crossing per file. Node ids computed Rust-side, byte-identical to
generateNodeId (pinned by test vector). Reserved per-node metrics slot
for the Arc 3.2 code-metrics work.
- Generic .scm-driven emitter (@def.<kind>/@name/@ref.<kind> captures,
byte-range scope stack → ::-joined qualified names, contains edges,
refs attributed to the innermost enclosing symbol). Seed TS/JS queries
are smoke-level; R2 replaces them with the full port.
- Routing seam in extractFromSource with per-file wasm fallback.
DEFAULT_ROUTED is empty — no behavior change until a language passes
its equivalence gate (R3). Dev opt-in: CODEGRAPH_KERNEL_LANGS. Kill
switch: CODEGRAPH_KERNEL=0. Loader verifies ABI + kind tables before
routing; EDGE_KINDS became a runtime array because kind order is now
wire contract.
- Grammar-source parity: vendored TS/TSX/JS wasm grammars built from the
exact crate revisions (tree-sitter-typescript v0.23.2,
tree-sitter-javascript v0.25.0, checked-in parser.c, ts-cli 0.25.10) —
the tree-sitter-wasms builds were 2023-era, which the new
kernel-grammar-parity test caught on day one. Production TS/JS parsing
gets 2.5 years of grammar fixes; full suite green (2456 tests).
- Build/release wiring: scripts/build-kernel.sh + npm run build:kernel;
release.yml kernel prebuild matrix (continue-on-error — the kernel is
optional everywhere, bundles fall back to the wasm path); bundles stage
lib/kernel/codegraph-kernel.node; release job runs the kernel suites
with CODEGRAPH_KERNEL_EXPECT=1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The ~36 independent synthesis passes (callback/event/framework wiring) ran
sequentially on the indexer's main thread — 2.0s of a 4,402-file Java repo's
index, and the stage where kernel-class repos die (#1212). They now live in
an explicit registry (SYNTH_PASSES) and, when the resolver pool is alive
(>=150k-ref repos), fan out across its read-only workers: dubbo synthesis
2,024ms -> ~900ms (-55%), total fresh init 13.5s -> 11.9s. Graphs verified
byte-for-byte identical on both the pool path (dubbo) and the sequential
path (excalidraw).
Why this is safe: no pass's edges persist until the ordered merge, so every
pass sees the same committed post-resolution DB state in either mode, and
results merge in registry order regardless of completion order — the
first-seen dedup is unchanged. The pool now survives through synthesis
(destroy moved after it) instead of being torn down moments before the one
stage that could reuse it.
Robustness: a pass that fails on a worker (crash, OOM) is retried on the
main thread — a synthesizer blow-up now costs one worker instead of the
whole index, which is half the #1212 story on very large repos.
Also: ref-row cleanup deletes now run as one transaction with a cached
statement instead of one implicit commit per 500-row chunk (mechanically
fewer WAL commits; matters most on HDD-class storage). A set-based rewrite
of failed-ref parking was tried, measured ~zero on NVMe, and dropped — the
remaining persist cost is edge-index B-tree maintenance, not statement
dispatch.
SYNTH_PROGRESS_STEPS now derives from the registry (passes + fixed marks);
the pin test counts registry entries plus literal __mark sites.
Suite green (2444). Sequential-path timing unchanged on excalidraw.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three compounding defects (#1196) made a query bag of object-literal
keys (`profileInfo isTrialEligible quotaInfo billingMethod`) return
unrelated results while the defining files never surfaced:
1. Step 5b title-cased interior humps (profileInfo -> Profileinfo) and
then compared case-SENSITIVELY, dropping every row SQLite's
case-insensitive LIKE had just recovered. The hump lookup is now
case-insensitive with an explicit uppercase-at-match requirement.
2. Step 5b/5c's kind whitelist held only type-like kinds — dead code on
method-centric codebases. Callable kinds (function/method/component)
are fetched as a SEPARATE LIKE batch so hot single-word terms can't
crowd classes out of the length-ordered 200-row batch.
3. explore's named-symbol seeding was exact-name only; a field token
seeded nothing. A camelCase token with ZERO exact defs now seeds its
camel-infix definers (callables, hump-boundary or prefix, shortest
first, capped at 3) — bare lowercase words keep the #1252 stopword
guard untouched.
The reporter's acceptance query is a pinned e2e test (definer files
present, exact-name seeding unaffected). excalidraw probe: the
canonical flow query (mutateElement renderStaticScene) is byte-
identical; NL queries shift toward more-central callables
(useUIAppState/getDefaultAppState over observer periphery).
Fixes#1196
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`.gitignore: /repos/` lists `repos/` as ONE ignored entry, while the
CLI hint (#1156) suggests `includeIgnored: ["repos/a/", "repos/b/"]` —
the child spelling. findIgnoredEmbeddedRepos tested the opt-in matcher
against the PARENT path only, which a child pattern never matches, so
the documented opt-in silently indexed nothing and init looped the
byte-identical suggestion back at the user (#1295).
Ignored dirs that don't match as a whole are now descended (the walk
was already bounded: depth 4 / 2000 entries, and only runs when
includeIgnored is configured) and each nested repo root is matched
individually — parent spelling opts in everything under the dir, child
spelling exactly the named repos. findUnindexedIgnoredRepos gets the
same per-repo check so the hint stops nagging about repos that are
already configured while still naming unopted siblings.
Fixes#1295
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
", ".join(sorted(x)) resolved by bare name to a project function named
join — one nested inside a DIFFERENT function, so scope alone rules the
edge out. Both defects from #1230, fixed independently:
1. Extraction: a member call on a LITERAL receiver (string, number,
collection, regex — across grammars) emits no call ref at all. A
literal's methods are the language's builtins, never project
symbols; the bare-name fallback let them exact-match any same-named
project function. Silent miss, never a wrong edge.
2. Resolution: matchByExactName filters out candidates nested inside a
same-file FUNCTION container unless the ref originates within that
container's line range. Class members (parent is a class-like node),
top-level symbols, and C++ namespace prefixes (no parent node) are
untouched.
requests re-index: byte-identical (813 calls edges). excalidraw: -27
edges, all literal-receiver refs by construction. The issue's repro is
pinned: join has exactly one caller (format_fields), report_missing
has zero project callees.
Fixes#1230
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
target.conn.Exec("insert") with `conn *sql.DB` emitted a BARE `Exec`
ref (the receiver chain was dropped for non-identifier receivers), and
exact-match then bound it to the only local `Exec` — an unrelated
interface's method — fabricating an internal dependency (#1276).
Extraction now keeps Go 2-hop selector chains (`base.field.Method`),
and a dedicated matcher resolves them EXCLUSIVELY via two inference
hops: base's type from the enclosing scope (#1108 machinery), field's
declared type from the struct's own declaration lines (comment-
stripped, per-line — chi's "the tree router" doc comment otherwise
donates a phantom type). resolveMethodOnType validates the target.
Package-qualified field types are followed only when the package is
in-module — `handler http.Handler` must not bind a same-named local
decoy. Failure at any hop leaves the ref unresolved: chained Go
receivers never fall through to the bare-name strategies (they were
never emitted before, so no prior recall depends on that path).
chi before/after: node count stable (1,181); 8 correct field-chain
edges gained (mx.tree.FindRoute/InsertRoute/routes, validated,
including the unexported `node` type); the removed edges are the
prior bare-name guesses on external receivers.
Fixes#1276
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
reproStore.notifyJoinGuildStatus() after `import { reproStore }`
resolved its calls edge to the exported CONSTANT (resolvedBy:'import'),
while the identical same-file call resolved to the method via
local-variable receiver inference (#1108) — so `callers <method>`
missed every cross-file use and a widely-used method could look
unused (#1292).
resolveViaImport's member-descend now handles imported VALUES alongside
the #825 static-member case: when the base resolves to a
constant/variable, the value's type is inferred from ITS OWN
declaration lines in the exporting file (the shared #1108 pattern
table: `= new T(...)` initializers and type annotations) and the member
is resolved AND VALIDATED on that type via resolveMethodOnType. A
failed inference or validation keeps the existing constant edge —
never a fabricated one. Calls only; plain member reads still reference
the value.
excalidraw control: byte-identical graph (10,653 nodes / 19,483 calls
edges before and after).
Fixes#1292
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The CLI's bare-symbol branch passes includeCode=true to the
codegraph_node handler, but the symbol-pinned-to-file branch didn't —
so exactly when a user disambiguated an overloaded name to one file
(the point of -f), they got Location + trail with no code (#1284).
Fixes#1284
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A directory deletion arrives as ONE event on the directory's own path.
That path has no source extension, so handleChange dropped it at the
isSourceFile gate before ever scheduling a sync — and the files inside
may never get events of their own (Windows's recursive watcher reports
only the top-most removed entry; FSEvents can coalesce a tree deletion
the same way). Every child record then sat stale in the index until an
unrelated edit happened to trigger a sync (#1285).
A non-source path that no longer EXISTS on disk now schedules the
debounced sync; the sync's scan-diff removes whatever vanished (already
correct — verified: manual `codegraph sync` cascades fine). Events for
live non-source files stay fully ignored, so build churn schedules
nothing.
Fixes#1285
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The #1242 fix (WAL deferral + checkpoint valve, the 26x win on
HDD-class storage) was wired only into indexAll. CodeGraph.sync never
touched wal_autocheckpoint, so every incremental run kept the default
1000-page cadence and re-triggered the #1231 per-page checkpoint
thrash — a 7-file sync took 2m 2s at 0-2% CPU on the reporter's
hardware, because the cost scales with the EXISTING database's hot
pages, not the change size.
sync now mirrors indexAll exactly: defer autocheckpoint + start the
valve for the run, fold the store phase's WAL before the post-store
reads, restore the interval in the finally. Same kill switch
(CODEGRAPH_NO_WAL_DEFER=1). Idle valve cost is one timer, so
watcher-frequency syncs stay cheap.
Fixes#1248
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
SEC_ATTR UINT32 LostName(VOID) — an unknown attribute macro before a
typedef'd return type — misparses in tree-sitter's C grammar: the macro
becomes the type, the return type the declarator, and the PARAMETER
LIST is stored as the function name ("(VOID)"). The C++ grammar
recovers this shape via recoverMangledCppName, but in C the real name
never reaches the mangled string, so only a pre-parse blank can help.
Attribute macros are project-specific, so the blank keys on structure:
line-leading ALL-CAPS token followed by TWO identifiers then `(` — the
`MACRO Ret name(` definition shape. Plain typedef'd returns, ALL-CAPS
calls, #define lines, multi-word builtin returns, and mid-line uses are
all rejected by construction. Offset-preserving like the C++ blanks.
curl re-index: 5,531 C functions before and after, zero name changes;
7 nodes in memdebug.c improve start-line accuracy by 1 (the macro line
no longer counts as part of the definition).
Fixes#1211
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
An out-of-line member definition inside a namespace block takes its
qualifiedName from the declarator's receiver, which is spelled RELATIVE
to the enclosing namespace — so `namespace simulator {
ManifestStartup::Output ManifestStartup::Apply(...) {} }` indexed as
ManifestStartup::Apply while the class node carried
simulator::ManifestStartup. Fully-qualified call sites
(simulator::ManifestStartup::Apply(...)) never resolved; callers and
file impact came up empty (#1291).
The receiver-based qualifiedName now composes the active namespace
prefix, anchored at the first prefix segment the receiver re-spells
(so `namespace sim { void sim::M::f() {} }` doesn't double-prefix).
namespacePrefix is only ever non-empty for C++ — Go/Rust/Kotlin/Lua
receivers pass through unchanged.
leveldb re-index: node count byte-stable (3,044), calls edges +6,
namespace-qualified method names 947 -> 1,252.
Fixes#1291
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
template<typename T> T Box<T>::get() stored qualified_name Box<T>::get —
the <T> qualifier never matched the class node indexed as Box, so the
method didn't link to its class, while the inline form of the same
method produced Box::get. ICU-shaped multi-line template parameter
lists leaked whole <…> blocks (newlines included) into qualified_name,
exceeding NAME_MAX for downstream consumers.
extractCppReceiverType now applies stripCppTemplateArgs (the #1043
normalization for base-class refs) to the receiver qualifier.
fmt re-index: template-arg-in-qualifier names 25 -> 4 (remaining are a
FMT_BEGIN_EXPORT misparse artifact and gmock conversion-operator names,
both distinct pre-existing shapes), node count byte-stable at 7,536.
Fixes#1286
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
cache.Put("a", 1), store.Get("config", out), bus.Handle("user.created",
h) — any verb-named method with a string first arg — were indexed as
HTTP routes (38 of 82 route nodes were false positives on the
reporter's 200 KLOC Go codebase). A registration's first argument must
now start with "/" (every router style), or be a Go 1.22
"METHOD /path" mux pattern on Handle/HandleFunc — which now also
extracts the real method instead of ANY.
Validated on go-chi/chi (212 real routes retained, all path-shaped)
and golang/groupcache (0 route nodes).
Fixes#1259
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
codegraph's glyphs were ASCII on every Windows console while
@clack/prompts drew its Unicode frame around them, so one index block
mixed `|` and `│` rails (#398). supportsUnicode() now mirrors the
is-unicode-supported detection clack bundles (Windows Terminal, VS
Code, ConEmu/Cmder, Alacritty, xterm-256color, JetBrains, CI), so both
systems always pick the same glyph family.
The shimmer worker's raw fs.writeSync(1) bytes still decode through the
console codepage (OEM codepages mojibake UTF-8 even under Windows
Terminal — the #168 regression to avoid), so:
- the raw path gets its own supportsUnicodeRawWrites() that stays ASCII
on win32 unless CODEGRAPH_UNICODE=1, and
- the persistent "phase done" lines move from the worker to the parent,
written via process.stdout (wide-char console API, codepage-immune) at
phase transitions — the main thread is alive there, it's delivering
the progress callback. Only transient, self-erasing animation frames
remain on the raw path, so ASCII never lands in scrollback.
Fixes#398
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
YucandGitHub