Replaces the R1 seed .scm emitter with a bespoke Rust walker
(codegraph-kernel/src/tsjs/) that mirrors TreeSitterExtractor's TS/JS
paths function-for-function: declarations (incl. #808 field/property
classification), qualified names, docstrings (#780 wrapper climbs),
signatures, imports/re-exports + per-binding refs, calls with
receiver-qualified callees (#1230 literal-receiver skip), instantiations,
decorators, inheritance, type annotations (#381), type-alias members +
tuple contracts (#359/#634), React component recognition (#841
forwardRef/memo/styled), object-of-functions / zustand-through-middleware
/ RTK Query endpoints + generated hooks / vuex + pinia store shapes,
function-as-value capture with the flush gate (#756), and value-reference
edges with the shadow prune (#895/#897). The generic query emitter is
deleted — extraction parity needs logic .scm can't express; future
languages get walkers too (migration plan §4a).
Positions and JS string-slice semantics are emitted in UTF-16 code units
natively, so kernel output is byte-identical to web-tree-sitter's — no
column diff class exists.
Parity evidence (macOS): scripts/kernel-parity.mjs (full-object multiset
diff per file) — this repo 353/353 files, excalidraw 643/643 (10,650
nodes / 10,726 edges / 68,307 refs), plus torture fixtures checked into
__tests__/fixtures/kernel-parity/ and enforced in npm test by
kernel-tsjs-parity.test.ts. The strict compare caught one real decoder
bug the loose harness missed: refs must NOT carry denormalized
filePath/language at the extractFromSource seam (the store fills them).
Perf: extraction 2.6× single-thread on excalidraw (487ms vs 1,255ms,
identical outputs). Routing stays opt-in (CODEGRAPH_KERNEL_LANGS) until
the R3 equivalence gate (large repo, DB dump-diff, retrieval invariants,
agent A/B, Linux/Windows) passes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 0 of the Rust extraction-kernel migration (docs/design/
rust-kernel-migration-plan.md, now checked in with §3a recording the
shipped state):
- codegraph-kernel/ napi-rs crate: extractFile(path, content, language)
→ five flat buffers (meta/nodes/edges/refs/arena), one JS boundary
crossing per file. Node ids computed Rust-side, byte-identical to
generateNodeId (pinned by test vector). Reserved per-node metrics slot
for the Arc 3.2 code-metrics work.
- Generic .scm-driven emitter (@def.<kind>/@name/@ref.<kind> captures,
byte-range scope stack → ::-joined qualified names, contains edges,
refs attributed to the innermost enclosing symbol). Seed TS/JS queries
are smoke-level; R2 replaces them with the full port.
- Routing seam in extractFromSource with per-file wasm fallback.
DEFAULT_ROUTED is empty — no behavior change until a language passes
its equivalence gate (R3). Dev opt-in: CODEGRAPH_KERNEL_LANGS. Kill
switch: CODEGRAPH_KERNEL=0. Loader verifies ABI + kind tables before
routing; EDGE_KINDS became a runtime array because kind order is now
wire contract.
- Grammar-source parity: vendored TS/TSX/JS wasm grammars built from the
exact crate revisions (tree-sitter-typescript v0.23.2,
tree-sitter-javascript v0.25.0, checked-in parser.c, ts-cli 0.25.10) —
the tree-sitter-wasms builds were 2023-era, which the new
kernel-grammar-parity test caught on day one. Production TS/JS parsing
gets 2.5 years of grammar fixes; full suite green (2456 tests).
- Build/release wiring: scripts/build-kernel.sh + npm run build:kernel;
release.yml kernel prebuild matrix (continue-on-error — the kernel is
optional everywhere, bundles fall back to the wasm path); bundles stage
lib/kernel/codegraph-kernel.node; release job runs the kernel suites
with CODEGRAPH_KERNEL_EXPECT=1.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fresh init on a 4,402-file Java repo (dubbo): 18.2s -> 13.5s (-26%), with
the resolution phase going 12.6s -> 7.9s (-38%). Graphs verified
byte-identical on both the pool path (dubbo) and the sequential path
(excalidraw).
Two changes:
1. The fastInit+pool path restored WAL for the resolver workers but left
wal_autocheckpoint at its default, so the persist loop inline-checkpointed
hot pages all phase long (#1231's pathology inside resolution — measured
at 58% of resolution wall). Checkpointing is now deferred behind the
bounded valve and folded once at maintenance, mirroring the deferWal path.
2. The resolution loop is double-buffered: batch k+1 is prefetched (OFFSET
past batch k's still-pending rows, under an explicit ORDER BY rowid) and
fanned out across the pool while batch k's ref cleanup runs on the main
thread. Batch settle-waits dropped 2572ms -> 117ms.
Correctness invariant found by the byte-identical gate and now documented in
the loop: batch k+1's resolution READS batch k's edges (resolveMethodOnType
walks supertype chains over extends/implements edges that resolution itself
inserts), so edges must persist BEFORE the next batch fans out; only the ref
cleanup overlaps.
Also extends the CODEGRAPH_SYNTH_TIMINGS instrumentation with phase labels
(grammar-init, parse-loop, fts-rebuild, resolver-reinit, resolution,
callback-synthesis) and pool timings (worker open/resolve, per-batch mode,
persist), so the next profile is one env var away.
Suite green (2444). Sync path timings unchanged. Pool floor re-validated:
forced-on at 40k refs is still net-slower, so the 150k threshold stands.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`.gitignore: /repos/` lists `repos/` as ONE ignored entry, while the
CLI hint (#1156) suggests `includeIgnored: ["repos/a/", "repos/b/"]` —
the child spelling. findIgnoredEmbeddedRepos tested the opt-in matcher
against the PARENT path only, which a child pattern never matches, so
the documented opt-in silently indexed nothing and init looped the
byte-identical suggestion back at the user (#1295).
Ignored dirs that don't match as a whole are now descended (the walk
was already bounded: depth 4 / 2000 entries, and only runs when
includeIgnored is configured) and each nested repo root is matched
individually — parent spelling opts in everything under the dir, child
spelling exactly the named repos. findUnindexedIgnoredRepos gets the
same per-repo check so the hint stops nagging about repos that are
already configured while still naming unopted siblings.
Fixes#1295
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
", ".join(sorted(x)) resolved by bare name to a project function named
join — one nested inside a DIFFERENT function, so scope alone rules the
edge out. Both defects from #1230, fixed independently:
1. Extraction: a member call on a LITERAL receiver (string, number,
collection, regex — across grammars) emits no call ref at all. A
literal's methods are the language's builtins, never project
symbols; the bare-name fallback let them exact-match any same-named
project function. Silent miss, never a wrong edge.
2. Resolution: matchByExactName filters out candidates nested inside a
same-file FUNCTION container unless the ref originates within that
container's line range. Class members (parent is a class-like node),
top-level symbols, and C++ namespace prefixes (no parent node) are
untouched.
requests re-index: byte-identical (813 calls edges). excalidraw: -27
edges, all literal-receiver refs by construction. The issue's repro is
pinned: join has exactly one caller (format_fields), report_missing
has zero project callees.
Fixes#1230
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
target.conn.Exec("insert") with `conn *sql.DB` emitted a BARE `Exec`
ref (the receiver chain was dropped for non-identifier receivers), and
exact-match then bound it to the only local `Exec` — an unrelated
interface's method — fabricating an internal dependency (#1276).
Extraction now keeps Go 2-hop selector chains (`base.field.Method`),
and a dedicated matcher resolves them EXCLUSIVELY via two inference
hops: base's type from the enclosing scope (#1108 machinery), field's
declared type from the struct's own declaration lines (comment-
stripped, per-line — chi's "the tree router" doc comment otherwise
donates a phantom type). resolveMethodOnType validates the target.
Package-qualified field types are followed only when the package is
in-module — `handler http.Handler` must not bind a same-named local
decoy. Failure at any hop leaves the ref unresolved: chained Go
receivers never fall through to the bare-name strategies (they were
never emitted before, so no prior recall depends on that path).
chi before/after: node count stable (1,181); 8 correct field-chain
edges gained (mx.tree.FindRoute/InsertRoute/routes, validated,
including the unexported `node` type); the removed edges are the
prior bare-name guesses on external receivers.
Fixes#1276
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
SEC_ATTR UINT32 LostName(VOID) — an unknown attribute macro before a
typedef'd return type — misparses in tree-sitter's C grammar: the macro
becomes the type, the return type the declarator, and the PARAMETER
LIST is stored as the function name ("(VOID)"). The C++ grammar
recovers this shape via recoverMangledCppName, but in C the real name
never reaches the mangled string, so only a pre-parse blank can help.
Attribute macros are project-specific, so the blank keys on structure:
line-leading ALL-CAPS token followed by TWO identifiers then `(` — the
`MACRO Ret name(` definition shape. Plain typedef'd returns, ALL-CAPS
calls, #define lines, multi-word builtin returns, and mid-line uses are
all rejected by construction. Offset-preserving like the C++ blanks.
curl re-index: 5,531 C functions before and after, zero name changes;
7 nodes in memdebug.c improve start-line accuracy by 1 (the macro line
no longer counts as part of the definition).
Fixes#1211
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
An out-of-line member definition inside a namespace block takes its
qualifiedName from the declarator's receiver, which is spelled RELATIVE
to the enclosing namespace — so `namespace simulator {
ManifestStartup::Output ManifestStartup::Apply(...) {} }` indexed as
ManifestStartup::Apply while the class node carried
simulator::ManifestStartup. Fully-qualified call sites
(simulator::ManifestStartup::Apply(...)) never resolved; callers and
file impact came up empty (#1291).
The receiver-based qualifiedName now composes the active namespace
prefix, anchored at the first prefix segment the receiver re-spells
(so `namespace sim { void sim::M::f() {} }` doesn't double-prefix).
namespacePrefix is only ever non-empty for C++ — Go/Rust/Kotlin/Lua
receivers pass through unchanged.
leveldb re-index: node count byte-stable (3,044), calls edges +6,
namespace-qualified method names 947 -> 1,252.
Fixes#1291
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
template<typename T> T Box<T>::get() stored qualified_name Box<T>::get —
the <T> qualifier never matched the class node indexed as Box, so the
method didn't link to its class, while the inline form of the same
method produced Box::get. ICU-shaped multi-line template parameter
lists leaked whole <…> blocks (newlines included) into qualified_name,
exceeding NAME_MAX for downstream consumers.
extractCppReceiverType now applies stripCppTemplateArgs (the #1043
normalization for base-class refs) to the receiver qualifier.
fmt re-index: template-arg-in-qualifier names 25 -> 4 (remaining are a
FMT_BEGIN_EXPORT misparse artifact and gmock conversion-operator names,
both distinct pre-existing shapes), node count byte-stable at 7,536.
Fixes#1286
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* perf(index): ~34% faster fresh indexing, byte-identical graphs
Profiling a fresh init on a medium TS repo (excalidraw, 657 files) showed
the main thread as the critical path: per-row SQLite statement calls,
repeated import-resolution walks, and per-row FTS trigger firings, with
the parse workers ~75% idle behind it. This lands the semantics-preserving
tranche of fixes:
- Multi-row batched INSERTs (nodes/edges/unresolved refs/name segments)
behind cached per-batch-size prepared statements; row order preserved,
so rowid-based resolution determinism (#1015) is unchanged.
- storeFileBundle: one transaction per file instead of four; nested
transaction() calls now flatten (BEGIN-in-BEGIN previously threw, so no
caller depended on nested rollback).
- Dedicated store-writer thread for the fresh-DB bulk path (bundles
applied in file order on a single writer connection; main thread does
no DB work during the parse loop). Kill switch: CODEGRAPH_NO_STORE_WORKER=1.
- Bulk FTS mode: drop the nodes_fts sync triggers during the bulk load,
rebuild once at the end; crash inside the window self-heals on the
next open.
- Per-context memos for resolveImportPath/findExportedSymbol + a per-file
exported-symbol index, invalidated exactly where clearCaches() already
resets the resolver's own caches.
- Fast-init on completely fresh DBs (journal in memory, no fsync until
the index completes; interrupted init re-runs from scratch). Kill
switch: CODEGRAPH_NO_FAST_INIT=1.
- MaybeYield returns undefined on the not-due path so per-ref yield
checks stop paying a promise + microtask hop each.
- Parse pool prewarm for bulk indexing; compile-cache enabled at CLI and
worker entry points.
Excalidraw fresh init: 5.11s -> 3.36s median (n=5, warm cache, M-series).
Graph dumps byte-identical across init, re-index, and sync paths; full
suite green (2403 passed).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* perf(resolution): parallel reference resolution with canonical admission
Fan resolution batches across a pool of read-only worker threads, each
hosting a full ReferenceResolver over its own SQLite connection; results
are admitted on the main thread in chunk order, so edge insertion order,
row cleanup, failure parking, and deferred post-pass queues are exactly
the sequence the single-threaded loop produces. Per-ref inputs match the
baseline because the sequential path already resolves each batch against
the state committed BEFORE that batch.
Validated byte-identical on excalidraw (pool forced on) and apache/dubbo
(4,048 Java files): dubbo full index 39s -> 19s (2.05x) with identical
graph dumps (91,495 nodes / 223,953 edges).
The pool only engages when total pending refs clear a threshold (default
150k, CODEGRAPH_PARALLEL_RESOLVE_MIN to tune, CODEGRAPH_NO_PARALLEL_RESOLVE=1
to disable): measured on a ~58k-ref repo the workers' boot CPU contends
with resolution on the same cores and makes indexing slower, so small
repos keep the sequential path. When fast-init left the DB in
memory-journal mode, WAL is restored before resolution only when the pool
will run (readers + rollback-journal writers don't mix).
Also: sqlite adapter readOnly open support.
TreeCursor spine rewrite of the body walker was built, measured neutral
on real repos and equal in a 20k-child microbench (web-tree-sitter's
namedChild(i) is not quadratic in this binding), and rejected — per-node
JS<->WASM marshaling is the floor, which a traversal swap cannot remove.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Two first-run UX bugs surfaced by indexing a real 1,342-file C repo:
1. After 'Resolving refs' hit 100%, the ~40 dynamic-dispatch synthesis
passes ran with no progress surface, so the bar sat frozen at 100%
long enough to read as a hang (the C fn-pointer pass alone can hold
for a while on C-heavy repos). Synthesis now reports per-pass
progress through a new 'linking' IndexProgress phase, rendered as
'Linking dynamic dispatch'. The step total is pinned by a test to
the synthesizer's actual __mark() count so adding a pass without
bumping it fails loudly.
2. node:sqlite's ExperimentalWarning is emitted once per THREAD, so the
main process plus every parse worker printed it mid-index,
interleaved with the progress UI. All launch paths now pass
--disable-warning=ExperimentalWarning: both bundle launchers, the
Windows npm-shim invocation, and the CLI self-relaunch
(NODE_RUNTIME_FLAGS, deliberately excluded from the re-exec gate so
an older installed launcher never triggers a pointless re-exec, and
version-gated off nodes older than the flag).
Verified end-to-end on the same repo: zero warnings, live linking bar,
byte-identical graph (50,520 nodes / 148,232 edges). Full suite green.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* fix(cpp): resolve explicit operator calls (a.operator+(b)) to the operator method (#1247)
tree-sitter-cpp can't parse an operator_name in field position: the
call_expression carries `function: <receiver>` plus an ERROR child
wrapping the operator_name instead of a field_expression callee, so the
extractor emitted a calls ref named just the receiver (`a`) and the edge
never resolved — while the operator method itself indexed fine.
Two-part fix, scoped to the explicit call form (infix `a + b` / `a[i]`
need receiver type inference and are tracked in #1258):
- extraction: recover the operator_name from the ERROR child and emit
`<receiver>.operator+` (`->` receivers normalized, `this->` emits the
bare name), like any other member call
- resolution: matchMethodCall's dot pattern now admits an operator
method part (cpp-gated; symbol chars failed the \w match), so
receiver-type inference + resolveMethodOnType validate the target —
a same-named operator on an unrelated class can't capture the edge
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix(cpp): harden explicit operator-call recovery against real-world shapes (#1247)
Validated on nlohmann/json (dozens of explicit operator[] / operator* /
operator< call sites). Two refinements the synthetic fixtures missed:
- normalize spaced call-site operator names (`it.operator * ()`,
`other.operator < (*this)`) to the compact form definitions index as
- drop the ref for a complex receiver (`obj()->operator+`, member chains
ending in a call) instead of emitting a bare operator name: exact-name
fallback GUESSED among unrelated same-named operators (linked a
std::map operator[] call to an in-repo operator[]) — silent miss,
never a wrong edge
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* chore: ignore .kommandr/ directory
* fix(sync): resolve cross-file refs when an edit adds or removes the satisfying symbol (#1240)
Incremental sync scoped reference resolution to the changed files' own
refs, and a completed pass deleted every ref it failed to resolve — so
a symbol change in one file could never repair references in UNCHANGED
files, in either direction, until a full re-index:
- New-export case: a.ts imports/calls `greet` before b.ts defines it.
The failed refs were deleted at index time; when b.ts later gained
`greet`, nothing revisited a.ts — the calls/imports edges stayed
missing while status reported a clean index.
- Removal case: when a re-index (or file deletion) dropped a symbol,
the incoming edges cascade-deleted and the callers — whose resolved
refs had been consumed — never got a chance to rebind to an
alternative definition or reconnect when the symbol returned.
Fix, sharing one lifecycle:
- Schema v8: unresolved_refs gains status ('pending'/'failed') and
name_tail (last dotted segment, so `h.greet` is findable by `greet`).
Both resolver persist paths now park unresolvable refs as failed
instead of deleting them. All pending-work readers (batched drain,
non-progress guard, #1187 orphan sweep, status pendingRefs) filter to
pending, preserving their invariants and keeping status honest.
- Sync retry: after scoped resolution, failed refs whose name tail
matches a symbol name now present in the changed files are re-resolved
through a per-ref-yielding path (watchdog-safe, #1091 class). Names
matching >500 failed refs are skipped as external/builtin noise (#999
rationale).
- Removal side: createEdges stamps each resolution edge with its
originating reference (metadata.refName, + refKind when kind promotion
rewrote it). When the #899 restore misses a target or sync deletes a
file, the dropped edge is resurrected as exactly that ref — re-resolved
in the same sync (rebinding to an alternative definition) or parked
failed until the symbol reappears. Edges without the stamp (pre-upgrade,
synthesized) still drop silently: reconstructing from the target's plain
name would strip receiver context and risk a rebind a full re-index
would never make.
- Pure-removal syncs clear resolver caches so a long-lived daemon can't
resolve resurrected refs against the pre-removal graph.
Validated: issue repro now yields a graph byte-identical to a full
re-index; move/remove-readd/file-deletion scenarios all rebind or heal;
baseline-vs-new A/B on express and gin shows identical node/edge counts
and no timing regression (DB grows ~25% from the parked ref rows — pure
cache, reset by any full re-index). 8 regression tests added.
Fixes#1240
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Parse timeouts are now judged by the worker's own clock: the base timer
only marks a job late (after a long synchronous store stall, Node runs the
timers phase before the poll phase, so the timer fired before an
already-delivered result was processed — killing workers over parses that
took milliseconds, even on 0-byte files); a result arriving before a 3×
hard-kill backstop is accepted, timed-out files are retried, and
CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are
pre-read once on the main thread and handed to every worker, so
spawns/respawns load grammars from memory instead of re-reading a
saturated disk.
Bulk indexing defers WAL auto-checkpointing for the whole run: the default
1000-page interval re-writes hot B-tree/FTS pages into the main DB file
over and over — ~95% of all disk I/O under throttled measurement. A
WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill
passes (never blocking the writer or the #850 watchdog heartbeat), pauses
the writer for a full backfill if the disk truly can't keep up, and folds
the WAL at the parse→resolution boundary so post-parse reads never page a
bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with
CODEGRAPH_WAL_VALVE_MB.
Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped
files (was 8); guava-scale completes in 7.6 min with a full graph where
v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Large-codebase indexing died at the end of "Resolving refs" two ways:
watchdog kills of healthy work (24k-file Java on Windows, #1212 — third
iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale,
where v1.3.0 could not complete at any watchdog setting). Root causes:
~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no
yield points, several materialized whole-graph snapshots (kotlin
expect/actual opened with getAllNodes() — 2M nodes in one array; the
C fn-pointer pass retained every C file's contents twice plus every
function node), and the post-index WAL checkpoint ran minutes of
synchronous IO on the main thread, killing even a successful index at
the finish line.
The pipeline tail now follows the same discipline as the rest: never
hold O(graph) in the heap, yield everywhere.
- All synthesis passes stream node-kind scans (cursors, not arrays) and
yield on time-budgeted checkpoints; language gates skip passes whose
filters a project's file languages provably can't satisfy.
- kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded,
units stream one file at a time, and the all-functions array +
write-only id map are gone; spring reads each .java once, not twice.
- runMaintenance moved to a worker thread (own SQLite connection);
per-file store commits chunk with yields behind a serialized flush
chain (preserving #1015 file-order determinism); resolver warm-up
streams the DISTINCT name set; resolution batch-tail and merged-edge
inserts run in bounded sub-transactions.
- Daemon: fixed a socket-handoff race that could leave a fresh MCP
session permanently silent (client-hello tail unshifted into a
flowing stream with zero listeners — the long-standing #662 test
flake was this real bug); first tool call no longer queues behind
the query pool's cold start (pool.ready gate).
Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully
indexes in 27m8s on a 2-core/6GB container at default heap + default
watchdog; llvm-project (180k files) completes under 1GB RSS including
kill-and-sync recovery; synthesized-edge and full-graph parity are
byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky
daemon test passed 25/25 under load. Env-gated diagnostics kept:
CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop
tracing. Design record: docs/design/main-thread-stall-followup.md.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
A Git super-repo whose `.gitignore` excludes its child repositories indexed
~nothing at the parent: CodeGraph respects `.gitignore` by default (#970,
#1065), so the excluded children were skipped and `codegraph init` printed
"Done" with 0 nodes — even though `init` inside each child worked fine. The
empty index was silent and unexplained.
`init`/`index` now detect the gitignored child repos they skipped when an
index comes up empty of symbols, name them, and — in an interactive terminal
— offer to index them (writing an `includeIgnored` entry to codegraph.json and
re-indexing on the spot); non-interactive runs print the exact codegraph.json
snippet to add. Gated on nodesCreated === 0, so a project that deliberately
keeps gitignored reference clones out of a working index is never nagged.
- extraction: findUnindexedIgnoredRepos — the inverse of discoverEmbeddedRepoRoots
(bounded, skips default-ignored dirs, respects existing includeIgnored)
- project-config: addIncludeIgnoredPatterns — create/merge codegraph.json,
idempotent, refuses to clobber malformed JSON
- cli: wire the detect-name-offer flow into both `init` and `index`
- tests: +13 covering detection, config writing, and the no-nag gate
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Carries #1133 forward onto current main (rebased for conflicts). A lean Unreal-Engine-style `.h` whose only C++ signal is `class ENGINE_API Foo : public Bar` (no public:/virtual/namespace/template) was misdetected as C and its class + inheritance edge silently dropped; looksLikeCpp now recognizes the export-macro-annotated class/struct shape, matching what blankCppExportMacros already recovers.
Fixes#1159.
Co-Authored-By: robertyluo <robertyluo@tencent.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Heavily-reflected UE C++ classes (`UCharacterMovementComponent`, `AActor`, `UWorld`, …) were dropped from the index: in-body no-semicolon reflection macros (`UPROPERTY`/`UFUNCTION`/`GENERATED_BODY`/`UE_DEPRECATED_*`), member-level `*_API` prefixes, and mid-line `UMETA`/`UPARAM`/`UE_DEPRECATED` accumulate tree-sitter parse errors until the enclosing class_specifier collapses into an ERROR node. Three offset-preserving, C++-only pre-parse passes (`blankCppAnnotationMacroCalls`, `blankCppApiPrefixMacros`, `blankCppInlineAnnotationMacros`) blank the markup before parsing so the class survives.
Validated on the real Epic engine headers (CharacterMovementComponent.h / Actor.h / World.h): each main class + its base clause goes from DROPPED on main to recovered with the fix.
Closes#1160.
Thanks @luoyxy for the report and root-cause analysis.
Adds an `include` list to the root `codegraph.json` that forces gitignored first-party source (second-VCS / SVN / Perforce dual-tracked repos) into the index — discovered directly off disk on the full index, incremental sync, and file-watching, on both git and non-git projects. Gitignore-style patterns, root-relative; explicit `exclude` still wins and built-in skips (node_modules, dist, .git) are never re-included. Complements `exclude` and `includeIgnored`.
Closes#1163.
Thanks @luoyxy for the contribution.
Four gaps in the MyBatis mapper extractor, all reported and reproduced by
@ESPINS in #1182 and verified against main:
1. Single-quoted attribute values (namespace/id/refid/resultType/parameterType)
were dropped — the regexes hardcoded double quotes. Now accept either quote
via a backreference.
2. Tags inside <!-- ... --> produced phantom statement/include symbols. A
length-preserving, CDATA-aware pre-pass blanks comments before scanning,
keeping offsets/line numbers intact.
3. Legacy iBatis 2 <sqlMap> files had zero statement coverage (the root finder
gated on a <mapper namespace> root). It now also recognizes <sqlMap>
(namespaced and namespace-less DAO.method ids) and iBatis's extra
<statement>/<procedure> verbs — closing the gap with no new dependency
(option (c) from the issue; the batis-xml parser route is declined).
4. Two statements sharing a qualifiedName AND a start line (a vendor-split
databaseId pair on one line) collided on the node id, so INSERT OR REPLACE
silently dropped one. The id-hash now folds in the statement's byte offset;
the stored qualifiedName/startLine are unchanged so the Java<->XML bridge is
untouched.
Gaps 1 and 2 follow @ESPINS's fix-mybatis-quotes-comments branch. Tests add
extractor-level coverage for all four gaps plus a DB-level e2e that proves
iBatis statements land and both vendor-split nodes survive a real indexAll.
Co-authored-by: Jimin Lee <dlwlalsggg@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds ArkTS (.ets, HarmonyOS/OpenHarmony) as a first-class language:
full TypeScript-grade extraction via the harmony-contrib tree-sitter
grammar (MIT, vendored byte-identical from the tree-sitter-arkts 0.2.0
npm tarball), plus the ArkUI constructs that make HarmonyOS apps
traceable:
- @Component/@ComponentV2 structs with decorators from both grammar
positions; members extract as class members with qualified names.
- build() component trees: child instantiation edges via
arkui_component_expression, no synthesizer needed.
- Attribute chains emitted dot-prefixed and resolved ONLY against
@Extend/@Styles/@AnimatableExtend/@Builder helpers (unique-or-drop) —
bare-name fallthrough produced 36,840 wrong edges (17% of calls) on
the OpenHarmony samples monorepo. All four grammar chain shapes
handled, including the detached-chain forms.
- .onClick(this.handler) method-reference bindings.
- ohpm workspace modules: bare imports follow oh-package.json5 file:
deps (ambiguous names dropped), honoring each module's main entry —
which also lets .ts consumers resolve .ets modules.
- ArkUI dynamic-dispatch bridges, all provenance:'heuristic' with
wiring-site metadata: assignment-gated state->build() re-render
(V1 @State family + V2 @Local/@Provider/@Consumer),
@ohos.events.emitter emit->subscriber pairing on static event keys
(numeric ids same-file, named constants same-module, fan-out capped),
and router.pushUrl literal urls -> the target page's @Entry struct.
- $r/$rawfile resource intrinsics treated as built-ins; arkts joins the
web language family, value-reference edges, re-export chase, and the
other TS-applicable gates.
Also ships a language-agnostic index-completeness guard: indexAll
stamps index_state (indexing -> complete/partial/failed), reconciles
discovered vs accounted files (a loaded run silently dropped 37 files),
and codegraph status surfaces truncated/partial indexes in human and
--json output.
Validated on HarmoneyOpenEye (82 files), CoolMallArkTS (528, modular
ohpm + ArkUI V2), and openharmony/applications_app_samples (11,693
files, 202,890 nodes stable across re-index, attribute false-positive
audit 36,840 -> 588 residual all-plausible). Supersedes PRs #656 and
#988 with credit — both informed this implementation.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Follow-ups noted in #1173:
- cloudposse/atmos remote-state: module.M.outputs.X emits a scoped
module.M:remote-output.X candidate; the resolver bridges it to the
target COMPONENT's own output when every gate holds — the module
source is the stack-config remote-state module, the component name is
static (a literal, or component = var.X whose variable declares a
literal default in the same directory), and exactly one directory in
the repo matches the component name and declares that output. Dynamic
(each.value) or ambiguous wiring stays unlinked. On
cloudposse/terraform-aws-components: 254 remote-state bridge edges,
every one re-derived from a matching source declaration (789/789
cross-directory output edges explained: 528 local-module + 254
remote-state + 7 checker-artifact false alarms under deprecated/);
coverage 66.4% -> 69.1%.
- provider aliases: provider "aws" { alias = "east" } is addressed as
provider.aws.east so aliased and default configurations stop
colliding; provider = aws.east on a resource/data block (and the
values of a module's providers map) reference the selected
configuration, resolved same-directory first then up the module tree
— the one construct Terraform genuinely inherits from parents. The
selection is no longer misread as a resource reference (aws.east).
- moved/import/removed blocks reference the resource addresses they
name (anchored to the file node — no phantom symbols), so a
refactor's paper trail joins the graph; check-assert conditions
contribute their references while check-scoped data blocks keep
indexing as before. Scoped module candidates are suppressed there:
module.a.aws_x.b names a resource inside a module instance, not an
output. +91 edges on cloud-foundation-fabric's moved-heavy stages.
Also fixes a latent test bug from #1173: cg.getNodeById is not public
API (cg.getNode is) — it only passed because the asserted edge list was
empty.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(extraction): add Terraform and OpenTofu language support
Index .tf, .tfvars, and .tofu files via the tree-sitter-terraform dialect
of HCL (vendored from @tree-sitter-grammars/tree-sitter-hcl, Apache-2.0).
Symbols extracted:
- resource / data → class (qualified "type.name" / "data.type.name")
- module → module (qualified "module.name")
- variable → variable (qualified "var.name")
- output → variable (qualified "output.name")
- provider → namespace
- locals → constant per attribute (qualified "local.key")
References resolved cross-file:
- var.X, local.X, module.M[.out], data.T.N[.attr], <type>.<name>[.attr]
- built-ins skipped: each.*, count.*, self.*, path.*, terraform.workspace
The Terraform framework resolver disambiguates same-named candidates
across modules by preferring the one in the same directory as the
reference site, then by closest common-ancestor path, falling back to
the generic name matcher only when neither applies.
Validated on two Terraform monorepos (277 and 470 .tf files): indexing
runs in 1.3s and 2.4s respectively, query latency stays under 200ms,
and cross-module references resolve to the correct module 100% of the
time on inspected samples.
18 new extraction tests; full suite 1146/1148 green (2 pre-existing
flaky skips, 0 regressions).
* feat(terraform): bridge the module boundary and enforce directory scoping
Builds on #706. The module declaration was a dead end: module.M.out
resolved to the declaration and stopped, module inputs never reached the
child module's variables, and impact could not cross the boundary — on
real multi-module repos that breaks the core blast-radius question
("what breaks upstream if I change this module's variable/output").
- module blocks now wire across the boundary through :-scoped refs only
the Terraform resolver understands: module.M:var.<input> → the child's
variable node, module.M:output.<o> → the child's output node (emitted
alongside the module.M declaration ref), and module.M:file → the local
source directory's entry file (imports). Registry/git sources emit no
file ref and resolve nothing — an out-of-repo module stays a visible
boundary instead of a guess.
- .tfvars top-level assignments reference the variable they set, walking
up to the nearest ancestor directory (envs/prod.tfvars → root vars).
- Resolution now enforces Terraform's real scoping: same-directory only
(no cross-module fallback by common path prefix, no single-candidate
anywhere-in-tree binding), and terraform refs never fall through to
the generic name matcher — var.X can never legally bind outside its
module directory, so the fallback could only add wrong edges.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs(terraform): README language table + changelog entry + agent-eval corpus
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Javier Rodríguez Fernández <jfernandez@freepik.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
CUDA rides the C++ grammar via the Metal (#1121) dialect pattern:
blankCudaConstructs (offset-preserving) blanks execution-space specifiers
(__global__ family), __launch_bounds__(...), and <<<grid, block>>> launch
configs — which otherwise lex as shift operators and destroy the
host→kernel call edge entirely. Gated by .cu/.cuh extension OR by content
(looksLikeCudaSource), because much real CUDA lives in .h/.hpp headers:
cutlass launches most kernels from headers and flash-attention's launch
templates are .h. Safe by construction — no CUDA marker is valid C++
anywhere, and the launch blank is bounded + brace-balance-checked so a
stray <<< (committed merge-conflict markers) can never blank real code.
All real-world launch styles connect: plain, templated
(k<T, 256><<<...>>>), function-pointer (auto kernel = &fn<...>; with
branch reassignments each linked), dim3{...} brace-init configs, and
kernels defined through name-in-first-argument macros
(DEFINE_FLASH_FORWARD_KERNEL style — gtest TEST_F / PYBIND11_MODULE
shapes deliberately excluded by the two-lone-identifiers rule).
Two general C++ resolution wins the flow validation forced out:
- namespace blocks now prefix contained symbols' qualifiedNames
(prefix-only — no namespace nodes, avoiding #1093-style crowd-out), so
ns::fn(...) calls resolve; previously every namespace-qualified C++
call was a permanently dead edge. cutlass: +30,864 edges (~10%), node
count byte-identical.
- templated callees (fn<T, 256>(args)) strip template args at extraction
(mirroring #1043 for base classes), so they match their definitions.
Validated on llm.c (165 host→kernel launch edges, was 0),
flash-attention (run_flash_fwd → flash_fwd_kernel → compute_attn traces
in one codegraph_explore call), and NVIDIA CUTLASS; fmt as the plain-C++
control (unchanged). A/B n=2/arm: Read/Grep displacement decisive on all
three repos (flash-attention Reads 29,13 → 5,2).
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
escripts (.escript) index like any module — the ELP grammar has a
first-class shebang node, so no source transform is needed; main/1 and its
helpers get full function/call extraction.
OTP application resource files (<app>.app.src and compiled <app>.app) join
the graph as Erlang terms the grammar parses natively. They route by full
suffix (their last-dot extension, .src, is far too generic for the
extension map). The application tuple yields structure: {mod, {Mod, _}}
links the app to its callback module — the app's entry point — and
{applications, [...]} / {included_applications, [...]} connect umbrella
sibling apps, resolving through the OTP app-name == module-name convention;
kernel/stdlib and other out-of-repo apps stay unresolved.
App-file refs resolve only ever to MODULES: validation on emqx caught the
ssl OTP-app dependency resolving to a test helper FUNCTION named ssl (the
same defect class as the earlier -behaviour gate), so the matchReference
module-only gate now covers every ref an .app/.app.src file emits.
Validated on emqx: 2 app.src + 6 escripts indexed, entry-module and
umbrella-dependency edges all namespace-targeted post-gate, escript
functions extracted; a stray legacy/module.src stays unknown.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Calls hidden inside -define bodies were invisible: the extractor consumed
pp_define without walking the replacement, and macro use sites produced no
edges, so a call path routed through a macro (ejabberd's SQL upsert macros,
logging wrappers) was completely dark.
The macro's constant node now participates in the graph. The -define body's
calls are attributed to the MACRO — true exactly once, instead of a per-use
duplicate that would explode on logging macros — and each use site links
in: ?MACRO(...) with arguments emits a `calls` ref (inlined code joins the
call chain), a bare ?CONSTANT read emits `references` (answering "where is
this macro used" without polluting call paths). Compiler-predefined macros
(?MODULE, ?LINE, ?FUNCTION_NAME, ...) are excluded, macro-use arguments
keep walking so a call nested in ?assertEqual(ok, do_thing()) still
attributes to the enclosing function, and macro-to-macro chains connect.
Validated: node counts unchanged on cowboy/ejabberd/emqx; edges +26/+7.3K/
+42K with honest hub shapes (?T i18n, ?SLOG logging, ?QOS_1 protocol
constants); 40/40 sampled edges precise; +1.3s index cost on emqx's 2,273
files. The payoff chain on ejabberd: set_password_scram_t → ?SQL_UPSERT_T →
ejabberd_sql:sql_query_t — database writes through SQL macros now trace
end-to-end.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
gen_server:call/cast/send_request now connects to the TARGET module's
handle_call/handle_cast for every statically-named target, not just self:
a bare atom reaches the module of that name (OTP's {local, ?MODULE}
convention names a server after its module), and a macro defined as a bare
atom (-define(STORE, kv_store)) resolves the same way, alongside the
existing ?MODULE / -define(SERVER, ?MODULE) self paths. A registered name
that matches no module emits a qualified ref that never resolves — silent,
never guessed. Pid, var, and tuple targets ({global, Name}, {Name, Node})
stay unlinked.
Validated on emqx: 53 new edges, 53/53 precise (each source line is a real
registered-name gen_server request; each target module self-registers under
that name, macro-indirected registrations included). Nearly all are
test-suite → handler links — production code goes through API wrappers the
self path already covers — which is exactly the tests-exercising-this-
handler linkage blast-radius and test-gap reporting consume. ejabberd
yields zero (it always wraps): no false positives invented.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Vendored WhatsApp/tree-sitter-erlang 0.19 (the ELP grammar, ABI 14) with an
Erlang-shaped extractor: multi-clause/multi-arity functions merged into one
symbol, -spec signatures, records with fields, -type/-opaque aliases, -define
macros, -include/-include_lib file edges, and -export-driven visibility.
Modules wrap in a namespace so remote mod:fn(...) calls resolve through the
existing qualified-name matcher as mod::fn with zero resolver changes.
-behaviour declarations link to the behaviour module — gated to namespace
targets only (bare-name fallthrough linked -behaviour(supervisor) to an
unrelated macro constant on emqx). OTP indirection with static targets is
followed: spawn/apply/proc_lib/timer/rpc MFA-argument callees, and
gen_server:call/cast(?MODULE | ?SERVER) to the module's own
handle_call/handle_cast. Var-module dispatch and message sends stay
deliberately unlinked. codegraph_explore also normalizes Erlang-native query
spelling (mod:fn/3, init/2) so named symbols resolve as typed.
Benchmarked on cowboy (189 files), ejabberd (414), emqx (2,447): extraction
PASS on all three; with-codegraph arms reached 2/2/0 file Reads vs 10/5+/19
without, fastest on the largest repo.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Vendored patched govindbanura/tree-sitter-vbnet grammar (MIT, ~20-fix patch
+ new external scanner for XML literals and multi-line LINQ continuation;
provenance + rebuild instructions in docs/grammars/tree-sitter-vbnet.md),
vbnet extractor with VB-specific call/index disambiguation, Inherits/
Implements heritage, As New instantiation, events, Declare P/Invoke, and
MustOverride abstract members.
Parse health on five real repos: PolicyPlus 100%, CompactGUI 100%,
staxrip 95.2%, SCrawler 87.2%, PCL 87.5% (upstream grammar: 3-18%).
Retrieval A/B (sonnet): 26-43% faster with 0-5 file reads vs 7-20 without.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Programs, sections/paragraphs (reconstructed extents over the grammar's
flat header stream), PERFORM/THRU/GO TO/CALL call edges, COPY copybook
imports incl. standalone .cpy fragments, DATA DIVISION records/fields/
88-levels with write-site impact references, and CICS flows: EXEC
LINK/XCTL program targets (literal + same-file VALUE deref), EXEC SQL
INCLUDE, and pseudo-conversational RETURN/START TRANSID hops resolved
to the owning program via a CICS framework resolver. Fixed and free
source format (free format via a scanner wide-mode sentinel).
Grammar: vendored wasm built from a patched yutaro-sakamoto/
tree-sitter-cobol (EXEC blocks as an external-scanner token, copybook
fragment entry point, single-quote continuation, COPY REPLACING
pseudo-text, NOT=, CALL GIVING, ENTRY, FREE, bitwise ops, abbreviated
relations, COBOL-2002 usages, and more). Patch + provenance + upstream
PR draft in docs/grammars/. Parse health: AWS CardDemo 43/44 native
(upstream: 9/31), 44/44 through preParse; copybooks 28/29; CobolCraft
free-format 17/17 (upstream: 0); NIST COBOL85 unchanged at 373/382.
Copybook members resolve to files like C includes (basename index,
name-matcher short-circuit so compiler-supplied members stay honestly
unresolved): CardDemo imports 5 -> 285. Impact proof: ACCT-CURR-BAL
(CVACT01Y copybook) surfaces its 4 writer programs cross-file.
Also: run-all.sh now neutralizes the ambient prompt-hook in both A/B
arms (CODEGRAPH_NO_PROMPT_HOOK=1); COBOL corpus entries for agent-eval.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
CFML joins the #1108 receiver-inference family: new/createObject/typed-arg/property(inject) declarations type the receiver, variables./this. fields scan whole-file, method QNs re-scoped to Class::member in all three extraction paths. 1,649 typed edges on fw1/ColdBox/CFWheels, 1,649/1,649 audit-consistent, inherited methods resolve via #1152 extends edges.
Co-authored-by: ghedwards <125586+ghedwards@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
.metal was absent from EXTENSION_MAP, so Metal Shading Language files were
silently skipped. MSL ≈ C++14, and the C++ grammar extracts its functions,
structs, type aliases, and call edges at parity with plain C++ — except MSL's
post-declarator [[attribute]] annotations, which misparse struct fields into
spurious extends refs from the struct to the field's own type (a wrong
inheritance edge whenever the repo typedefs float3/float4x4 itself, common in
shared ShaderTypes.h). blankMetalAttributes blanks them pre-parse,
offset-preserving, following the blankCppExportMacros pattern (#1061), gated
to .metal files only — in regular C++ the attribute position is legal syntax
the grammar parses natively. The preParse hook gains an optional filePath
param to support the gate.
Validated on llama.cpp's ggml-metal.metal (10.7k lines: 130 kernels vs 113
`kernel void` ground-truth lines, rope_yarn resolves its 4 kernel callers)
and SDL's shaders (PQtoLinear ← GetOutputColor), 0 bogus extends edges.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The Ruby extractor dropped the method name from a `receiver.method` call:
`lg.log()` was recorded as a call to `lg` (the bare receiver), which
matches no symbol, so the reference resolved to nothing and no method
edge was ever produced. A Ruby method invoked through a receiver had no
recorded callers and was invisible to impact/blast-radius and explore
flow traces. This is the Ruby-specific blocker noted in #1108 — that
local-variable type-inference fix couldn't help Ruby because the call
reference itself was missing.
extractCall recognized receiver-bearing calls by the `object`/`name`/
`function` fields other grammars use; tree-sitter-ruby's `call` node uses
`receiver` + `method`, so it fell through to the generic fallback that
takes the first named child (the receiver) as the callee. Handle Ruby
`call`/`method_call` explicitly: build `receiver.method`, keep bare
`foo(...)` as the method name, emit `Foo.new` as an `instantiates` ref,
and give a capitalized (constant) receiver a `references` edge so a class
used only via its class methods still records a dependent.
With this plus #1108, `lg = Logger.new; lg.log` resolves `lg.log` to
`Logger#log`, and the two-file same-name case is same-file-correct
(#1079). Adds Ruby to the local-variable inference test matrix plus a
focused test asserting `Foo.new` stays an instantiation.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(extraction): broaden the curated C++ inline-macro library list
Since #1102 the post-parse salvage already recovers the NAME for any macro, so
adding a library now buys full return-type recovery for it. Extend the curated
list across the major C++ ecosystem: Mozilla/SpiderMonkey, Protobuf, {fmt},
Hedley + nlohmann/json, GLM, Bullet (SIMD_FORCE_INLINE), Skia, OpenCV, EASTL,
Cocos2d-x, Chromium/WebKit (NEVER_INLINE), GLib, SQLite, and the unambiguous
Windows calling conventions (WINAPI / APIENTRY / STDMETHODCALLTYPE / WINAPIV —
which sit between the return type and the name, so blanking them recovers the
return type, e.g. `HRESULT WINAPI Foo()` -> Foo : HRESULT).
Every entry is an exact, curated token matched only in specifier position, so a
real all-caps return type is never touched. Anything still missed keeps its name
via the universal salvage. CARLA control unchanged (440->6 mangles, 0
regressions — none of these libs appear there, confirming no collateral). Eleven
representative full-recovery tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note broadened C++ inline-macro library coverage (#1103)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(extraction): universal recovery of macro-mangled C/C++ function names
The curated inline-macro blank list (#1100/#1101) can't enumerate every
library's macro. Add a universal post-parse net so a function is findable by
name regardless of which macro decorates it, plus a batch of common libraries
to the curated list for full name+return-type recovery.
- recoverMangledCppName: after extraction, recover the real identifier from a
name still mangled by an un-blanked macro (`MACRO Ret name(…)` misparses to
"Ret name"). It's a new `recoverMangledName` extractor hook wired only onto
C/C++, applied to every name they produce. Safe by construction: it only
touches an already-mangled name (an internal space that isn't a legit
`operator …`/destructor), so a clean name is returned unchanged; guarded
against the `Ret (name)` parenthesized-name idiom and bare primitives. Scoped
to C/C++ so Kotlin/Scala backtick identifiers (which legitimately contain
spaces) are never touched.
- Curated list extended past UE/pugixml/Godot/Boost to Qt (Q_INVOKABLE, …),
Folly, Abseil, LLVM, V8, Eigen, and rapidjson.
Validated on CARLA (large UE project, 1131 C++/h files) vs the pre-fix baseline:
function-name mangles 440 -> 6, 431 fixed, and — critically — 0 regressions
(the salvage also recovers names that the pre-parse's own non-local error-recovery
shifts would otherwise re-mangle, erasing the 7 shifts seen in #1101). The 6
residual are all the moodycamel `Ret (name)` idiom, correctly left alone. On a
made-up macro with no list entry (`WEBKIT_EXPORT WTFString compute()`), the name
`compute` is still recovered. Full suite green; eleven regression/safety tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note universal C++ macro-mangled name recovery (#1102)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(extraction): recognize common third-party C++ inline macros, not just UE
Extend blankCppInlineMacros beyond Unreal Engine's FORCEINLINE family to the
inline/linkage macros that vendored third-party libraries define and that
mangle function names the same way:
- pugixml: PUGI__FN / PUGI__FN_NO_INLINE (before the return type) and
PUGIXML_FUNCTION (linkage macro, between return type and name — the blank
mechanism handles both positions).
- Godot: _FORCE_INLINE_ / _ALWAYS_INLINE_.
- Boost: BOOST_FORCEINLINE / BOOST_NOINLINE.
- Generic cross-ecosystem hints: ALWAYS_INLINE / FORCE_INLINE / NOINLINE.
The list now drives a single generated alternation (longest-token-first), so
adding a codebase's macro is a one-line change. Still curated exact tokens in
specifier position only — a real all-caps return type like `HRESULT DoIt()` is
never touched (verified by controls).
Validated on CARLA (large UE project, 1131 C++/h files): function-name mangles
440 -> 16 (428 fixed). The 16 residual and 7 clean->mangled shifts are all in
third-party vendored files — chiefly pugixml.cpp, a 12k-line macro amalgamation
where error recovery is non-local, so blanking one of several *stacked* macros
(PUGI__FN + PUGI__UNSIGNED_OVERFLOW …) shifts an already-imperfect extraction.
Normal C++/UE code (ActionRoguelike, ALS) sees zero regressions — blanking a
macro there only helps. Chasing pugixml's internal attribute macros is left out
of scope. Seven regression tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note third-party C++ inline macro recognition (#1101)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(extraction): recover C++ function names prefixed by an inline-specifier macro
An unknown inline-specifier macro before a function's return type
(`FORCEINLINE FString GetName(…)`) threw tree-sitter into error recovery: the
macro was read as the return type and — for a non-primitive return — the return
type was glued onto the name, so the function was indexed as
`"FString GetName"` instead of `GetName`, unfindable by name and with no caller
links. This is pervasive in Unreal Engine, where inline helpers are written
`FORCEINLINE <ret> <name>(…)` (e.g. ALS's `FORCEINLINE FString GetEnumerationToString`).
Add `blankCppInlineMacros`, a preParse that blanks the known UE inline macros
(`FORCEINLINE`, `FORCENOINLINE`, `FORCEINLINE_DEBUGGABLE`) with equal-length
spaces so byte offsets stay exact and the declaration parses as an ordinary
function — recovering both the real name AND the return type. This is the same
recover-don't-drop approach as blankCppExportMacros (#946/#1061), and the two
are composed into the cppExtractor preParse.
Matched tightly (exact known tokens, only in specifier position — followed by
the identifier that starts the return type/name), so ordinary identifiers, real
all-caps return types (`HRESULT DoIt()`), string literals, expression uses, and
longer words (`FORCEINLINE_COUNT`) are untouched — verified by controls. C++-only;
Kotlin/Scala re-index byte-for-byte identical. Five regression tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note C++ inline-specifier-macro function name fix (#1100)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(extraction): correct C++ reference-return and conversion-operator method names
Two pre-existing C++ name-extraction bugs surfaced while validating the #1093
forward-declaration fix against real Unreal Engine repos (ActionRoguelike, ALS):
1. Inline methods/functions returning a reference were named after the whole
declarator. `const int& getRef() const {…}` parses with a reference_declarator
wrapping the function_declarator; extractName unwrapped pointer_declarator but
not reference_declarator, so the method was named "& getRef() const" instead
of "getRef" — polluting search and breaking caller linkage. Ubiquitous in UE
headers (`const FGameplayTagContainer& GetActiveTags() const`). Now the
reference wrapper is unwrapped alongside the pointer wrapper.
2. User-defined conversion operators were named with their full declarator —
`operator EALSMovementState() const` — instead of `operator EALSMovementState`,
so they didn't match the symbolic-overload style (`operator+`) and carried
`() const` noise. The operator_cast declarator is now named `operator <type>`.
Both are additive and C++-scoped (reference_declarator / operator_cast are C++
grammar nodes). Pointer, value, and out-of-line reference returns, and symbolic
operator overloads, are unchanged. Six regression tests added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(changelog): note C++ reference-return and conversion-operator name fixes (#1096)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A `class Foo;` forward declaration parses as a bodiless class_specifier.
extractStruct (#831) and extractEnum already skip their bodiless forms,
but extractClass did not — so every forward decl across dozens of headers
minted a phantom bodiless `class` node that competed with, and could be
picked as the blast-radius representative over, the single real definition.
Add an opt-in `skipBodilessClass` extractor flag (set only on cppExtractor)
and skip a bodiless class node when it's set, mirroring the struct/enum
skip. The flag keeps this C/C++-scoped: languages where a bodiless class is
a complete definition (Kotlin `class Empty`, Scala `case object`/`trait`)
leave it unset and are unaffected. The body is now resolved once at the top
of extractClass and reused for the member walk.
Regression tests cover the collapse to a single definition, elaborated-type
references creating no phantom, and Kotlin/Scala staying indexed.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A C++ class annotated with an export/visibility macro between `class`/`struct`
and the type name — `class MYMODULE_API UMyComponent : public UActorComponent`
(the standard Unreal-Engine `*_API` pattern), or the equivalent `*_EXPORT`/
`*_ABI` macros in Qt, Boost, LLVM, etc. — makes tree-sitter read `class MACRO`
as an elaborated type and the whole declaration as a function. #946 dropped the
resulting phantom function, but that also discarded the recoverable class name,
members, and base-class edge, so the class never entered the graph and
"find subclasses" / type-hierarchy / impact-through-inheritance returned
nothing for effectively every gameplay class in a UE project.
Add `blankCppExportMacros` as `cppExtractor.preParse`: it blanks the macro with
equal-length spaces before parsing (offset-preserving, like C#'s
`blankCsharpPreprocessorDirectives`/#237), so the declaration parses as a normal
class_specifier and existing extraction emits the node, members, and `extends`
edge. Generalized past UE `*_API` to any all-caps export macro, with two
false-positive guards: the trailing `[:{]` definition-guard (leaves elaborated
var decls like `struct FOO var;` alone) and requiring the macro to be followed
by the real name (leaves an all-caps class NAME such as `class FOO : public Base`
alone). C++-only, so C's heavier `struct TAG var;` never reaches it. The #946
drop stays as the fallback for any residual misparse the blanking doesn't catch.
Validated on google/leveldb (LEVELDB_EXPORT, 134 files): class/struct nodes
266→293, extends edges 292→359, phantom functions 588→513; every export-macro
real definition flips function→class and `EnvWrapper extends Env` goes
absent→present.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The gitlink discovery added in #1031/#1033 indexed a tracked 160000 gitlink even when the parent .gitignore excludes its directory, pulling a gitignored reference/benchmark corpus of git add'ed clones into the index (one report: ~138k files, 4.8 GiB, wedged "Resolving refs" watchdog).
Gate both gitlink-discovery sites on the same rule the untracked-embedded path already uses: skip a gitignored gitlink unless codegraph.json includeIgnored opts it in; index non-ignored gitlinks as before. Validated real-world on macOS, Linux, and Windows.
Closes#1065
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
An Android `res/` tree (layouts, value bags, drawables, menus, navigation
graphs) holds only non-code resources that yield zero symbols, yet on an
Android app it dominates the file count (one report: 26k XML = 97% of
files, 0 symbols) — bloating the DB, slowing indexing, and padding
explore/search results and file counts with entries that have nothing to
find.
Default-ignore the Android resource type directories (`res/layout/`,
`res/values/`, `res/drawable/`, … and their `-<qualifier>` variants) at
discovery, via DEFAULT_IGNORE_PATTERNS so it applies uniformly to the git
index, the non-git walk, and change detection. The `res/<type>/`
structure is self-identifying, so non-Android projects are untouched, and
the only XML that carries symbols — MyBatis mappers under
`src/main/resources/` — never lives under `res/`, so nothing useful is
dropped. `res/raw/` is deliberately kept (arbitrary bundled assets), and
a `.gitignore` negation re-includes anything.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
`instantiates` edges came only from heap `new Calculator(0)` (a
new_expression) and copy-init `Calculator c = Calculator(0)` (a
call_expression). Stack direct-init `Calculator calc(0)` and brace-init
`Widget w{1, 2}` parse as a `declaration` whose constructor arguments hang
directly off the declarator as an argument_list / initializer_list — there
is no call/new node — so the function-body walker saw no constructor
invocation and emitted no edge. A function that built objects with the
ordinary stack syntax looked like it didn't construct them, and the
dependency was missing from impact / callers.
In the body walker, a C++ `declaration` that is a stack/brace construction
now reuses extractInstantiation (a declaration's `type` field IS the
constructed class name, and extractInstantiation already strips template
args / namespace and emits the `instantiates` ref). Gated by
isCppStackConstruction, which requires BOTH a class-like type
(type_identifier / template_type / qualified_identifier — so `int x(0)`
and `auto z = …` are excluded) AND a declarator carrying args
(argument_list / initializer_list — so default `Calculator c;` and the
most-vexing-parse `Calculator c();` are excluded). The edge targets the
class node, not the same-named constructor method.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A C++ class deriving from a template — `class Derived : public Base<int>`,
a CRTP base `class App : public CRTPBase<App>`, a struct inheriting a
template, or a templated base mixed into a multi-base clause — recorded its
base as the full instantiation text (`Base<int>`). That never name-matched
the template, which is indexed as the bare node `Base`, so the `extends`
edge never resolved and the derived class looked like it inherited from
nothing — callers/impact analysis stopped at the boundary.
Strip the template arguments from the base-type reference name in the
`base_class_clause` handler via a new `stripCppTemplateArgs` helper: it
removes every balanced `<…>` group (any nesting/position), so `Base<int>`
→ `Base` and `ns::Tpl<int>` → `ns::Tpl`. The remaining qualified head is
exactly what the non-templated base case already produces, so resolution
treats templated and non-templated bases identically; a name with no
template args passes through unchanged.
Covers same-file and same-namespace bases (the dominant real-world
patterns). A base in a different namespace referenced with its qualifier
(`other_ns::Tpl<int>`) still doesn't resolve, but that's a pre-existing,
orthogonal namespace-resolution gap — the non-templated `other_ns::Plain`
fails identically — not a template issue.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A nested git repo tracked as a gitlink (mode 160000) — a clone `git add`ed
into the super-repo without a `.gitmodules` entry, or a submodule that
isn't active/initialized in this checkout — fell through both file-collection
passes: it's tracked, so the untracked `-o` listing skips it, but it's not
an active submodule, so `--recurse-submodules` won't expand it. Indexing the
top level therefore pulled in only the outer repo's own files and stopped at
the nested repo's boundary (one report: ~10 files at the root).
Switch the tracked scan to `ls-files -s` to expose file modes, collect the
unexpanded 160000 entries, and recurse into each that has a real working tree
on disk as its own embedded repo. Mirror the same discovery in
discoverEmbeddedRepoRoots so the watcher's scope stays equal to the indexer's.
Active submodules (#147) and untracked nested clones (#193) are unchanged;
gitlinks under default-ignored dirs (vendor/, node_modules/) stay excluded
(#407); an uninitialized submodule with no checkout on disk is left alone.
Adds four-shape coverage in extraction.test.ts.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
indexAll parsed every file through a single worker thread, so a full `codegraph index` used one core no matter the machine. Add ParseWorkerPool (src/extraction/parse-pool.ts), modeled on the shipped QueryPool: indexAll now parses across clamp(cores-1,1,8) workers. CODEGRAPH_PARSE_WORKERS overrides the count; 1 reproduces the previous single-worker path exactly (the rollback).
Parses run concurrently but results commit to SQLite in file order. This matters: the post-index resolution phase selects among ambiguous same-named candidates by node DB-insertion order, so a stable commit order keeps the graph deterministic — byte-identical to the serial path — instead of drifting with parse-completion timing. A bounded reorder buffer (backpressure on dispatched-but-uncommitted count) keeps memory flat even if a file is slow at the commit cursor.
Crash/timeout of a worker rejects only that file's parse (feeding the existing retry pass) and respawns; per-worker recycle every 250 parses reclaims WASM heap. In-process fallback unchanged when the compiled worker is absent (tests).
Validated on real OSS (django +9%, redis +17%; modest and parse-fraction-dependent), graph byte-identical across worker counts, peak memory flat-to-lower since workers recycle independently — so the #320 OOM concern doesn't materialize. Adds 11 pool unit tests.
Closes#1015. Refs #320.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Swift in-class properties are extracted by a dedicated branch in
TreeSitterExtractor.visitNode, not the generic nameField/variableTypes path
swift.ts declares. That branch had a `!isComputed` gate that dropped computed
properties entirely, so `codegraph query`/`codegraph_explore` returned "No
results found" for them — including a SwiftUI view's `var body: some View`,
the most important symbol in any SwiftUI app, and the heavily-read
`var isCloudProxy: Bool` from the report.
Stored properties were already fixed in #708 (v1.0.0); the reporter tested
v0.9.9 and confirmed "still present on main" by inspecting swift.ts only,
missing the dedicated branch — so only the computed-property half was real.
- Computed properties now index as `property` nodes; the getter is walked via
visitFunctionBody so its calls attribute to the property (a SwiftUI `body`'s
subview tree becomes the property's callees — the render flow is traceable
through it), not flattened onto the enclosing type.
- Protocol property requirements (`var x: T { get }`) — a third never-indexed
category — index as `property` too.
- Routing the getter through visitFunctionBody also stops getter-local
`let`/`var` declarations from being wrongly node-ified as struct fields
(the generic child-walk used to do this): Alamofire property 0→348, field
618→588, idempotent.
Stored/static behavior is unchanged.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Colby McHenryandClaude Fable 5
xyyxrandGitHub
