`git ls-files -co --exclude-standard` only sees the submodule pointer in
the main repo's index, so projects using submodules indexed 0 files. Now
the tracked list runs with `-c --recurse-submodules` so submodule
contents are included; untracked files are gathered with a separate
`-o --exclude-standard` call (the two flags can't be combined — git only
supports --recurse-submodules with --cached/--stage).
Fixes#147.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: add framework extract wiring plan
* feat(resolution): replace extractNodes with extract() returning nodes and references
* feat(resolution): add getApplicableFrameworks helper for per-language dispatch
* feat(django): emit route nodes and route->view references in extract()
* feat(flask,fastapi): emit route nodes and route->handler references
* feat(express): emit route nodes and route->handler references
* feat(laravel): emit route nodes and route->handler references
* feat(rails): emit route nodes and route->handler references
* feat(spring): emit route nodes and route->handler references
* feat(go): emit route nodes and route->handler references
* feat(rust): emit route nodes and route->handler references
* feat(aspnet): emit route nodes and route->handler references
* feat(swift,vapor): emit route nodes and route->handler references
* chore(react,svelte): migrate resolvers to extract() interface
* feat(extraction): run framework extractors after tree-sitter parse
* docs: document framework route extraction
* feat(strip-comments): add per-language comment stripper for framework extractors
Replaces comment characters and string-literal contents with spaces (not
removal) so source offsets stay valid for downstream regex match index ->
line number conversion. Handles Python triple-quoted docstrings, Ruby
=begin/=end, Rust nested block comments, and the standard //, #, /* */
forms across the supported languages.
This is consumed by framework extract() methods in a follow-up commit so
that commented-out / docstring routing examples don't surface as phantom
route nodes in the graph.
* feat(frameworks): strip comments before regex extraction (prevents phantom routes)
Pipes the per-language stripCommentsForRegex helper into every framework
extract() that scans raw source: django/flask/fastapi (python.ts),
express, laravel, rails, spring, go, rust, aspnet, vapor, plus
swiftui/uikit struct extraction in swift.ts.
Without this, examples like:
# path('/admin/', AdminPanel.as_view())
""" path('/users/', UserListView.as_view()) """
urlpatterns = [path('/real/', RealView.as_view())]
produced 3 phantom route nodes. Now only the real one is extracted.
Each framework gets a regression test in __tests__/frameworks.test.ts
asserting that line-, block-, docstring- and (where relevant)
heredoc-style commented-out routes do not surface as nodes.
---------
Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(cli): surface lock-acquisition errors and silence Emscripten Aborted() spam
Two unrelated cosmetic but actively misleading bugs that surface when
the indexer is under load.
1) printIndexResult fell through to "No files found to index" whenever
the IndexResult had filesIndexed=0 AND filesErrored=0. The
lock-acquisition path returns success:false with a generic
"Could not acquire file lock" entry in result.errors[] (severity
'error'), but filesErrored counts only file-level parse failures,
so the user saw "No files found to index" — actively wrong.
Add a top-of-function check for the !success && !hasErrors case
that surfaces the first severity:'error' message instead.
2) parse-worker.ts let Emscripten's stderr "Aborted()" lines (plus
their "Build with -sASSERTIONS for more info" follow-ups) leak to
the parent's terminal whenever a WASM tree-sitter parser crashed
on a pathological file. Even after the JS layer caught and recovered,
the user saw dozens of `Aborted()` lines spammed to stderr. Install
a stderr filter at worker startup that drops only those specific
Emscripten internal lines; everything we log ourselves passes
through unchanged.
Verified live against ollama/ollama@v0.22.0:
- second concurrent `codegraph index` now shows
"Could not acquire file lock - another process may be indexing"
instead of "No files found to index"
- WASM-crash-prone re-index produced 0 Aborted() lines (down from 68+).
* fix(cli): null-safe error surfacing + clearer stderr-filter contract docs
Two reviewer findings on PR #128:
- printIndexResult: when result.success is false but result.errors
contains no severity:'error' entry (degenerate case but possible
if the result shape ever drifts), the find() returned undefined
and the previous if-guard fell through to the misleading
'No files found to index' branch. Now always surfaces a clear
failure message via clack.log.error, defaulting to 'Indexing
failed — no further details available' when no specific error
is in the errors list.
- parse-worker stderr filter: callback handling was already correct
but the comment didn't document it; expand the comment to spell
out the Writable-stream-contract obligation, the per-call match
semantics (split-chunk caveat), and the substring-exactness
trade-off so future readers understand the deliberate trade-offs.
Two correctness bugs in the core extraction pipeline, surfaced by an
adversarial stress corpus (5k synthetic export-const declarations
plus a deliberate 8MB single-line file):
1) Every `export const X = ...` produced TWO nodes for the same
symbol — one kind:'variable' from extractExportedVariables, plus
one kind:'constant' from extractVariable (called when the walker
descended into the export_statement child). Stress test showed
100% duplication across 5,003 export-const declarations. The
dedicated extractVariable dispatch is the correct one — it picks
kind from isConst, captures the initializer signature, and walks
type annotations; the export-statement helper was redundant
because the language extractors' isExported predicate already
walks parent chains. Remove the export_statement branch from the
dispatch (children are descended into normally) and drop the
private helper.
2) The bulk indexAll path read each file's stats but never compared
stats.size against config.maxFileSize. Vendored generated files
(multi-MB headers, minified bundles, etc.) were indexed regardless
of the user's size cap. The single-file extractFile path enforced
it; only the bulk path was missing the check. Mirror the
single-file behaviour: emit a 'size_exceeded' warning, count the
file as skipped, advance progress, and continue.
On the stress workspace (5,005 synthetic files; 50,000 fns in one
3MB file; 8MB single-line file; 5,000 export-const declarations):
before: 65,014 nodes (100% var/const duplication, every >1MB file
indexed despite maxFileSize=1MB)
after: 10,008 nodes (0 duplicates, large files correctly skipped
with size_exceeded warnings)
Tests calibrated to the duplicate behavior were updated to look for
kind:'constant' on `export const`, which is the correct kind. Full
suite: 380 passed (was 374 passed, 6 failed before this fix).
* feat(extraction): instantiates + decorates graph edges
Two new structural edges that fill gaps in the call graph for
modern JS/TS / Java / C# / Python / Kotlin codebases.
1) `instantiates` edges from `new Foo(...)`:
The bulk-extraction and visitFunctionBody dispatchers only
recognised `call_expression`; `new_expression` (and the equivalent
`object_creation_expression` / `instance_creation_expression` in
other grammars) was silently ignored. Adds INSTANTIATION_KINDS,
extractInstantiation(), and dispatch from BOTH the top-level
visitNode and the per-function-body walker. Children are still
descended so nested calls inside constructor args (`new Foo(bar())`)
get their own `calls` refs.
Output: a `bootstrap` function that does `new UserService(); new
UserController(svc)` now produces two `instantiates` edges to those
class nodes — previously zero edges.
2) `decorates` edges from `@Decorator` annotations:
Tree-sitter places decorator nodes BEFORE the symbol they apply to
in the AST, so the original walk-time dispatch saw the wrong
nodeStack head (file/class instead of class/method). Replaced with
extractDecoratorsFor(declNode, decoratedId) that runs from inside
extractClass / extractFunction / extractMethod after the symbol's
node id is known.
Looks for decorator nodes in two places:
- Direct named children of the declaration (method/property style)
- Preceding siblings in the parent (TypeScript class style:
@Foo class X {} parses as parent { decorator, class_decl })
Sibling check uses startIndex comparison rather than reference
identity — tree-sitter web bindings return fresh JS wrappers from
parent/namedChild navigation, so `===` is unreliable. Took a debug
session to spot this; flagging in the comment so the next reader
doesn't re-introduce the bug.
Output: a `@Controller` class decorator + `@Get` method decorator
on a NestJS-style controller now produce two `decorates` edges
(class→Controller, method→Get) with the correct source nodes.
Verified live on a synthetic NestJS-shape fixture; all 380
existing tests pass.
* fix(extraction): address reviewer findings — decorator boundary, generic constructors, property/field decorators, marker_annotation, tests
Five fixes from independent semantic review:
- extractDecoratorsFor sibling walk now iterates BACKWARD from the
declaration and stops at the first non-decorator/annotation
separator. Previous version walked forward up to declStart and
consumed every decorator-typed sibling — so two adjacent
decorated classes (`@A class Foo {} @B class Bar {}`) had `@A`
spuriously attributed to `Bar`.
- extractInstantiation strips the type-argument suffix from the
constructor field text. `new Map<K, V>()` was producing
referenceName 'Map<K, V>' (the constructor field is a generic_type
node) and resolution always failed.
- extractProperty and extractField now call extractDecoratorsFor
after their createNode calls. NestJS-style `@Inject() private
svc: Foo` and Java field annotations were being silently dropped.
- consider() in extractDecoratorsFor recognises 'marker_annotation'
in addition to 'decorator'/'annotation'. Java's tree-sitter grammar
emits marker_annotation for arg-less annotations like @Override
and @Deprecated; without this every Java marker annotation was
silently skipped.
- 6 new extraction tests covering: instantiates ref for new Foo(),
generic-type stripping (`new Container<string>()` -> 'Container'),
qualified-new keeps trailing identifier (`new ns.Foo()` -> 'Foo'),
decorates ref for @Foo class X {}, regression for adjacent
decorated classes (each gets its OWN decorator), decorates ref
for @Foo method().
Full test suite: 386 passed (was 380, +6 new extraction tests).
* feat(resolution): kind-aware scoring + Python instantiation promotion
Two follow-ups to the new instantiates/decorates ref kinds, surfaced
during review:
1) name-matcher previously only had a kind bonus for `calls`
(preferring function/method). When a class and a function share a
name across modules, an `instantiates` ref would tie or pick the
wrong candidate. Adds:
- `instantiates` → +25 for class/struct/interface
- `decorates` → +25 for function/method, +15 for class
(Python class decorators, Java annotation interfaces)
2) Python (and Ruby) have no `new` keyword — `Foo()` is the standard
instantiation syntax, indistinguishable from a function call at
extraction time. Resolution can tell the difference once the
target is known: when a `calls` ref resolves to a class/struct,
promote it to `instantiates`. Mirrors the existing extends→
implements promotion in createEdges.
Verified: 386 → 389 passing (+3 tests covering the kind biases and
the Python promotion).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Changes search result deduplication to use max scores across channels instead of first-seen prioritization, adds template component usage extraction for Svelte files, exempts exact matches from single-term score dampening, prioritizes structural edges in graph traversal, and increases explore tool node budget while including edge source locations in file clustering.
Addresses PHP traits extracted as classes, missing class properties, skipped constants, and invisible trait usage. Adds classifyClassNode to distinguish traits from classes, fixes property extraction for PHP's property_element AST structure (added 4,366 field nodes), and adds visitNode hook for class constants and trait use declarations (increased trait edges from 636 to 1,514). Also improves Liquid schema name handling and file path reference resolution. Verified against Laravel codebase.
Addresses JavaScript `class extends` producing zero inheritance edges due to tree-sitter grammar differences. JavaScript uses `class_heritage → identifier` (bare) while TypeScript wraps with `extends_clause`. Updates extractInheritance to handle bare identifier/type_identifier children when parent is class_heritage.
Addresses two tree-sitter misparse patterns: (1) fun interfaces with @Throws annotations parse as function_declaration > ERROR instead of user_type, (2) parent interface bodies become ERROR nodes when containing nested fun interfaces, causing methods to be skipped. Updates isFunInterfaceNode to check ERROR-nested user_type children and resolveBody to prefer ERROR bodies starting with `{`.
Addresses Kotlin interfaces/enums extracted as classes, zero function calls, and missing `fun interface` declarations. Adds classifyClassNode to distinguish interfaces/enums from classes, resolveBody hook for non-field grammar, navigation_expression call handling, getReceiverType for extension functions, and visitNode hook to detect `fun interface` misparse patterns from tree-sitter-kotlin's lack of Kotlin 1.4+ syntax support. Verified against Koin and LeakCanary codebases.
Addresses Dart method calls like `obj.method()` and `runApp()` that parse as identifier+selector combinations instead of dedicated call nodes. Adds extractBareCall hook to detect selector nodes with argument_part, handling simple function calls, method chains, constructor calls (new/const), and super/this method calls. Enables proper call relationship tracking for Dart's selector-based call syntax.
Addresses TypeScript abstract classes missing by adding abstract_class_declaration to classTypes. Fixes single-expression arrow functions being silently dropped by preventing extractName from searching identifiers in arrow_function/function_expression bodies, ensuring they return for proper parent name resolution instead of incorrectly using body identifiers.
Addresses arrow function class fields like `field = () => { ... }` where the function body is nested inside field_definition nodes. Adds resolveBody method to traverse field_definition → arrow_function/function_expression → body and handles HOF wrapper patterns like `field = throttle(() => { ... })` by searching call_expression arguments. Enables proper function body extraction for class field functions in both JavaScript and TypeScript.
Addresses Ruby bare method calls like `reset` that parse as identifier nodes instead of call expressions. Adds extractBareCall hook to detect statement-level identifiers that represent method calls, filtering out keywords, literals, and constants. Enables proper call relationship tracking for Ruby's parentheses-optional method syntax.
Addresses Ruby methods inside modules missing owner in qualified_name by adding visitNode hook to extract module AST nodes. Methods inside modules now get Module::method qualified names with proper containment relationships. Includes ExtractorContext wiring with pushScope/popScope for language hooks and updates isInsideClassLikeNode to include module kind for nested method handling.
Addresses C#'s property_declaration nodes (public string Name { get; set; }) by adding propertyTypes support and extractProperty method. Improves field extraction to handle C#'s nested variable_declaration > variable_declarator structure. Adds base_list handling in extractInheritance for C#'s `: Parent, IInterface` syntax where base class and interfaces are combined in a single colon-separated list.
Addresses C++ classes missing from .h files where extension-based detection defaults to 'c' language which has no class extraction support. Adds looksLikeCpp() heuristic that scans first 8KB for C++-specific patterns (namespace, class, template, access specifiers) to promote .h files to 'cpp' language when C++ constructs are detected. Ensures cpp grammar is loaded alongside c to handle potential .h promotion during parsing.
Addresses C++ macros like NLOHMANN_JSON_NAMESPACE_BEGIN that cause tree-sitter to misparse namespace blocks as function_definitions. Adds isMisparsedFunction hook to filter macro artifacts while still visiting their bodies to extract legitimate class/struct/enum definitions hidden inside the misparsed "function" scope.
Addresses C/C++ typedef syntax where anonymous enum/struct definitions are wrapped in typedef declarations (e.g. `typedef enum { A, B } MyEnum;`). Adds resolveTypeAliasKind to identify inner enum_specifier and struct_specifier nodes within typedefs, enabling proper extraction of enum members and struct fields from the inner anonymous definitions rather than treating them as simple type aliases.
Addresses C/C++ pointer declarator unwrapping where pointer_declarator nodes need to be resolved to find the actual function/variable name. Adds forward declaration filtering by checking for body field presence before processing struct and enum definitions, preventing extraction of incomplete type declarations.
Addresses PHP's base_clause syntax for class inheritance (extends) and implements clause for interface implementation. Adds trait_declaration support and separates property_declaration into fieldTypes. Improves PHP method call extraction by handling member_call_expression and scoped_call_expression with proper receiver name processing, including $ prefix stripping and self/this/parent/static receiver filtering.
Addresses Rust's impl block syntax where trait implementations (`impl Trait for Type`) and trait supertraits (`trait Sub: Super`) create inheritance relationships. Adds getReceiverType to extract method receiver types from impl blocks, enabling proper method-to-struct relationships and qualified name resolution. Verified against Deno codebase and moved from "Needs Verification" to completed language support.
Addresses TypeScript's AST structure where class_heritage nodes wrap extends_clause and implements_clause rather than directly indicating inheritance relationships. Moves class_heritage from direct inheritance extraction to recursive container processing to properly traverse the wrapped inheritance syntax.
Addresses Swift's inheritance_specifier syntax where type relationships are specified after colons (e.g. `class UploadRequest: DataRequest, Sendable`). Extracts user_type > type_identifier children from inheritance_specifier nodes as 'extends' references to properly model Swift's inheritance, protocol conformance, and struct conformance patterns in the code graph.
Addresses method call resolution ambiguity where bare method names couldn't be distinguished from function calls. Modifies tree-sitter extraction to include receiver names (e.g., "console.log" instead of just "log") while skipping common instance references like self/this. Updates built-in filtering to be language-specific and adds Python built-in method detection based on receiver types and method names.
Addresses Python's class definition syntax where parent classes are specified in argument_list nodes (e.g. `class Child(Parent, Mixin):`). Extracts identifier and attribute children from argument_list as 'extends' references to properly model Python's inheritance patterns in the code graph.
Addresses Go's tree-sitter parsing where method calls use selector_expression nodes with 'field' children instead of member_expression nodes with 'property' children. Extends function call extraction to handle Go's obj.method() syntax alongside existing JavaScript/TypeScript support.
Addresses qualified name pollution where file paths contaminated full-text search results. Removes file path prefix from buildQualifiedName output and method qualified names, keeping semantic hierarchy only. Also adds unresolved reference creation for Go imports to enable proper import edge resolution in the dependency graph.
Addresses Go's embedding mechanism where structs can embed other types without field names (e.g. `type DB struct { *Head; Queryable }`) and interfaces can embed other interfaces via constraint_elem nodes. Extracts these embedded types as 'extends' relationships to properly model Go's composition-based inheritance patterns in the code graph.
Addresses Go's tree-sitter parsing where structs and interfaces are wrapped in type_spec nodes rather than appearing as direct node types. Moves struct_type and interface_type detection from direct node type matching to a new resolveTypeAliasKind resolver that examines the inner type field, ensuring proper extraction with field visiting and inheritance detection.
Adds nameMatchBonus scoring to prioritize results where node names exactly or partially match query terms. Implements dedicated field extraction for Java/C# to properly categorize class fields vs variables. Optimizes BM25 search with column weights favoring name matches and increased result fetching before post-processing. Refines stop words list to preserve common programming terms like "get", "find", "list".
Extends AST parsing to identify and extract individual enum members/cases for better code analysis. Adds enumMemberTypes configuration to each language extractor with language-specific node types (e.g., 'enum_variant' for Rust, 'enum_case' for PHP, 'enum_entry' for Swift/Kotlin). Implements flexible member name resolution supporting both field-based and identifier-based extraction patterns.
Improves search accuracy by boosting results when multiple query symbols appear in the same file, addressing cases where common names like "run" return too many results. Adds Go method receiver type extraction to qualified names for better searchability (e.g., "scrapeLoop.run"). Optimizes database queries with two-pass approach to handle distinctive vs common symbol names efficiently.
Eliminates the intermediate 'finalizing' phase that was added as a progress bar transition state but served no functional purpose. Simplifies the progress flow by going directly from 'storing' to 'resolving' phases, removing associated UI labels and progress callbacks.
Adds strategic yield points and direct stdout writes to prevent progress animation from freezing when the main thread is blocked by synchronous operations. Introduces 'finalizing' phase to smooth transition between parsing and resolving steps, ensuring progress reaches 100% completion.
Recycles workers before each retry attempt instead of once per batch to maximize WASM memory headroom. Adds final fallback that strips comment-only lines from files that still crash on clean workers, reducing memory pressure from compiler test files with extensive CHECK directives while preserving line numbers for accurate node positions.
Implements file-size-based timeouts (base 10s + 10s per 100KB), more frequent worker recycling (250 files), and automatic retry logic for WASM memory corruption failures. Workers now crash immediately on memory errors to prevent cascading failures, with failed files automatically retried on fresh workers with clean heaps.
Adds --verbose flag to init and index commands that shows timestamped progress output instead of animated progress bars. Implements worker timeout protection (10s per file) and periodic worker recycling (every 500 files) to prevent WASM memory crashes from hanging the entire indexing process. Includes detailed logging of worker lifecycle events and memory usage for debugging large repository indexing issues.
Offloads tree-sitter parsing to a dedicated worker thread, keeping the main thread unblocked so shimmer progress animations render smoothly during indexing. Refactors shimmer progress renderer into separate worker for consistent 50ms animation updates. Falls back to in-process parsing when worker compilation unavailable (e.g., tests).
Replaces fixed-width padding with terminal escape sequences for proper progress line clearing across different terminal widths. Adds periodic parser reset every 5000 parses per language to prevent WASM heap fragmentation that causes "memory access out of bounds" crashes in large repositories. Includes filename truncation to fit available terminal width.
Adds expression index on lower(name) for memory-efficient case-insensitive searches, replacing in-memory caches that caused OOM on large codebases. Includes batched reference resolution, enhanced error reporting with detailed breakdown by error type, and improved CLI progress display for scanning phases.
Splits the monolithic tree-sitter.ts (3,358 lines) into modular files:
- 14 language config files under src/extraction/languages/
- 3 standalone extractors (Liquid, Svelte, DFM)
- Shared helpers and types modules to avoid circular imports
Also fixes a bug where Java's extractImport hook incorrectly set
handledRefs: true, preventing unresolved reference creation and
degrading codegraph_explore results for Java codebases.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
When a project directory is gitignored by a parent git repository, `git ls-files` returns no results even though files exist. Added detection for this scenario using `git rev-parse` and `git check-ignore` to fall back to filesystem walking when the project directory is ignored by an ancestor repo.
Adds type annotation parsing to create references edges for parameter types, return types, and variable type annotations in TypeScript and other typed languages. Expands symbol extraction from queries to capture lowercase identifiers and filters out more common English words. Removes obsolete search utility tests.
Java extraction:
- Handle Java method_invocation AST (receiver.method pattern)
- Support Java extends_interfaces and super_interfaces with type_list
- Create unresolved references for Java imports for cross-file resolution
- Extract interface inheritance via extractInheritance
MCP tools:
- Aggregate callers/callees/impact across ALL matching symbols (e.g. multiple
overloads or same-named methods in different classes)
- New findAllSymbols() helper for multi-symbol lookup
Graph traversal:
- Impact analysis now traverses into container children (class → methods)
so that callers of methods appear in the impact radius of their class
Other:
- Add deleteSpecificResolvedReferences() for precise cleanup after resolution
- Add 'instance-method' to resolvedBy union type
- Version bump to 0.6.8
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminates anonymous error reporting functionality that was collecting stack traces and error context via Sentry. Removes all telemetry-related code, configuration options, and documentation references.
andreinknvandGitHub