`codegraph init -i` from a git super-repo containing independent nested
git repositories (not submodules) reported "No files found to index":
git ls-files reports an embedded repo only as an opaque `subdir/` entry
and never lists its files. Detect embedded repos via that trailing-slash
signal and recurse `git ls-files` into each, indexing tracked + untracked
source and honoring each repo's own .gitignore.
Reported by @timxx.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`git ls-files -co --exclude-standard` only sees the submodule pointer in
the main repo's index, so projects using submodules indexed 0 files. Now
the tracked list runs with `-c --recurse-submodules` so submodule
contents are included; untracked files are gathered with a separate
`-o --exclude-standard` call (the two flags can't be combined — git only
supports --recurse-submodules with --cached/--stage).
Fixes#147.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: add framework extract wiring plan
* feat(resolution): replace extractNodes with extract() returning nodes and references
* feat(resolution): add getApplicableFrameworks helper for per-language dispatch
* feat(django): emit route nodes and route->view references in extract()
* feat(flask,fastapi): emit route nodes and route->handler references
* feat(express): emit route nodes and route->handler references
* feat(laravel): emit route nodes and route->handler references
* feat(rails): emit route nodes and route->handler references
* feat(spring): emit route nodes and route->handler references
* feat(go): emit route nodes and route->handler references
* feat(rust): emit route nodes and route->handler references
* feat(aspnet): emit route nodes and route->handler references
* feat(swift,vapor): emit route nodes and route->handler references
* chore(react,svelte): migrate resolvers to extract() interface
* feat(extraction): run framework extractors after tree-sitter parse
* docs: document framework route extraction
* feat(strip-comments): add per-language comment stripper for framework extractors
Replaces comment characters and string-literal contents with spaces (not
removal) so source offsets stay valid for downstream regex match index ->
line number conversion. Handles Python triple-quoted docstrings, Ruby
=begin/=end, Rust nested block comments, and the standard //, #, /* */
forms across the supported languages.
This is consumed by framework extract() methods in a follow-up commit so
that commented-out / docstring routing examples don't surface as phantom
route nodes in the graph.
* feat(frameworks): strip comments before regex extraction (prevents phantom routes)
Pipes the per-language stripCommentsForRegex helper into every framework
extract() that scans raw source: django/flask/fastapi (python.ts),
express, laravel, rails, spring, go, rust, aspnet, vapor, plus
swiftui/uikit struct extraction in swift.ts.
Without this, examples like:
# path('/admin/', AdminPanel.as_view())
""" path('/users/', UserListView.as_view()) """
urlpatterns = [path('/real/', RealView.as_view())]
produced 3 phantom route nodes. Now only the real one is extracted.
Each framework gets a regression test in __tests__/frameworks.test.ts
asserting that line-, block-, docstring- and (where relevant)
heredoc-style commented-out routes do not surface as nodes.
---------
Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two correctness bugs in the core extraction pipeline, surfaced by an
adversarial stress corpus (5k synthetic export-const declarations
plus a deliberate 8MB single-line file):
1) Every `export const X = ...` produced TWO nodes for the same
symbol — one kind:'variable' from extractExportedVariables, plus
one kind:'constant' from extractVariable (called when the walker
descended into the export_statement child). Stress test showed
100% duplication across 5,003 export-const declarations. The
dedicated extractVariable dispatch is the correct one — it picks
kind from isConst, captures the initializer signature, and walks
type annotations; the export-statement helper was redundant
because the language extractors' isExported predicate already
walks parent chains. Remove the export_statement branch from the
dispatch (children are descended into normally) and drop the
private helper.
2) The bulk indexAll path read each file's stats but never compared
stats.size against config.maxFileSize. Vendored generated files
(multi-MB headers, minified bundles, etc.) were indexed regardless
of the user's size cap. The single-file extractFile path enforced
it; only the bulk path was missing the check. Mirror the
single-file behaviour: emit a 'size_exceeded' warning, count the
file as skipped, advance progress, and continue.
On the stress workspace (5,005 synthetic files; 50,000 fns in one
3MB file; 8MB single-line file; 5,000 export-const declarations):
before: 65,014 nodes (100% var/const duplication, every >1MB file
indexed despite maxFileSize=1MB)
after: 10,008 nodes (0 duplicates, large files correctly skipped
with size_exceeded warnings)
Tests calibrated to the duplicate behavior were updated to look for
kind:'constant' on `export const`, which is the correct kind. Full
suite: 380 passed (was 374 passed, 6 failed before this fix).
Addresses C++ classes missing from .h files where extension-based detection defaults to 'c' language which has no class extraction support. Adds looksLikeCpp() heuristic that scans first 8KB for C++-specific patterns (namespace, class, template, access specifiers) to promote .h files to 'cpp' language when C++ constructs are detected. Ensures cpp grammar is loaded alongside c to handle potential .h promotion during parsing.
Eliminates the intermediate 'finalizing' phase that was added as a progress bar transition state but served no functional purpose. Simplifies the progress flow by going directly from 'storing' to 'resolving' phases, removing associated UI labels and progress callbacks.
Adds strategic yield points and direct stdout writes to prevent progress animation from freezing when the main thread is blocked by synchronous operations. Introduces 'finalizing' phase to smooth transition between parsing and resolving steps, ensuring progress reaches 100% completion.
Recycles workers before each retry attempt instead of once per batch to maximize WASM memory headroom. Adds final fallback that strips comment-only lines from files that still crash on clean workers, reducing memory pressure from compiler test files with extensive CHECK directives while preserving line numbers for accurate node positions.
Implements file-size-based timeouts (base 10s + 10s per 100KB), more frequent worker recycling (250 files), and automatic retry logic for WASM memory corruption failures. Workers now crash immediately on memory errors to prevent cascading failures, with failed files automatically retried on fresh workers with clean heaps.
Adds --verbose flag to init and index commands that shows timestamped progress output instead of animated progress bars. Implements worker timeout protection (10s per file) and periodic worker recycling (every 500 files) to prevent WASM memory crashes from hanging the entire indexing process. Includes detailed logging of worker lifecycle events and memory usage for debugging large repository indexing issues.
Offloads tree-sitter parsing to a dedicated worker thread, keeping the main thread unblocked so shimmer progress animations render smoothly during indexing. Refactors shimmer progress renderer into separate worker for consistent 50ms animation updates. Falls back to in-process parsing when worker compilation unavailable (e.g., tests).
Replaces fixed-width padding with terminal escape sequences for proper progress line clearing across different terminal widths. Adds periodic parser reset every 5000 parses per language to prevent WASM heap fragmentation that causes "memory access out of bounds" crashes in large repositories. Includes filename truncation to fit available terminal width.
Adds expression index on lower(name) for memory-efficient case-insensitive searches, replacing in-memory caches that caused OOM on large codebases. Includes batched reference resolution, enhanced error reporting with detailed breakdown by error type, and improved CLI progress display for scanning phases.
When a project directory is gitignored by a parent git repository, `git ls-files` returns no results even though files exist. Added detection for this scenario using `git rev-parse` and `git check-ignore` to fall back to filesystem walking when the project directory is ignored by an ancestor repo.
Eliminates anonymous error reporting functionality that was collecting stack traces and error context via Sentry. Removes all telemetry-related code, configuration options, and documentation references.
Fixes#42 — tree-sitter can produce nodes with empty names (e.g. from
complex C/C++ declarators in header files). These nodes were silently
skipped at DB insert time, but their containment edges were still
inserted, causing a FOREIGN KEY constraint violation that crashed
indexing.
Two-layer fix:
- createNode() now returns null for empty names, preventing the node
and its edges from ever being created (Option A)
- storeExtractionResult() filters edges and unresolved refs to only
reference nodes that passed validation, as a safety net (Option B)
Fixes#54 — `codegraph init -i` crashes with "Fatal process out of memory: Zone"
on large codebases because all 16 tree-sitter WASM grammar modules were compiled
upfront by V8, exhausting the WASM Zone allocator.
Changes:
- initGrammars() now only initializes the tree-sitter WASM runtime (Parser.init()),
no longer eagerly loads all grammar files
- New loadGrammarsForLanguages() loads only grammars for languages actually present
in the project (e.g. a Dart project loads ~2-3 grammars instead of 16)
- Orchestrator detects needed languages after file scan, before parsing begins
- Embedding pipeline now uses quantized model (~67MB vs ~270MB) to further reduce
WASM memory pressure when embeddings are enabled
Replace native tree-sitter with web-tree-sitter + tree-sitter-wasms for
universal cross-platform support. Add node-sqlite3-wasm as a fallback
when better-sqlite3 native bindings aren't available. Move better-sqlite3
and sqlite-vss to optionalDependencies so installs never fail.
Fix installer to use npx fallback when global npm install fails, so MCP
config, hooks, and quick-start instructions all work without the bare
codegraph command in PATH.
Fix tests: update schema version expectation, fix db test paths and
method names, extract MAX_OUTPUT_LENGTH as module constant, normalize
Windows path separators in import resolver.
- extraction/index.ts: use picomatch with static import (replacing
dynamic require) and keep normalizePath for other call sites
- utils.ts: keep normalizePath from main, take PR's PID-based FileLock
- Remove extractFunctionVariable() and its dispatch (already handled by extractVariable)
- Remove dead getGrammar() export (zero callers)
- Deduplicate indexFile by delegating to indexFileWithContent
- Remove redundant arrow function variable extraction tests (covered by existing suite)
- Create file-kind nodes for each parsed source file
- Add isInsideClassLikeNode() for method vs function detection
- Extract arrow functions and function expressions from variable declarators
- Batch file I/O with FILE_IO_BATCH_SIZE=10 using Promise.all
- Add symlink cycle detection with visitedDirs Set in scanDirectory
- Add lazy grammar loading with exported getGrammar() function
- Add indexFileWithContent() for pre-read content processing
- Add tests for file nodes and arrow function extraction
- Add validateProjectPath() to reject sensitive system directories
- Add isPathWithinRoot/isPathWithinRootReal for symlink-aware path checks
- Replace hand-rolled glob-to-regex with picomatch to prevent ReDoS
- Add isSafeRegex() to reject custom patterns with nested quantifiers
- Replace FileLock with PID-tracking version that detects stale locks
- Add symlink detection in removeDirectory/listDirectoryContents
- Add subdirectory name validation in ensureSubdirectory
- Add atomicWriteFileSync and corrupted file backup in config-writer
- Add MCP input validation (validateString) for all tool handlers
- Fix CLAUDE.md section replacement to handle ### subsections correctly
Normalize paths to forward slashes in matchesGlob() and scanDirectory()
so glob exclude patterns work on Windows. Add getGitIgnoredDirectories()
using git ls-files to skip .gitignore'd directories during indexing.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SQLite performance pragmas: synchronous=NORMAL, 64MB cache,
memory temp store, 256MB mmap (safe with WAL mode)
- Batch insert for unresolved refs: single transaction instead of
N individual inserts per file
- Symbol caching (warmCaches): pre-load all nodes into memory maps
before resolution, eliminating repeated SQLite queries per ref
- Async file I/O: fs.stat/readFile in indexFile() are now non-blocking
- Denormalize filePath/language onto UnresolvedReference: avoids N
node lookups during resolution, with schema migration v2
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements security improvements inspired by PR #16 (credit: MO2k4):
- Add validatePathWithinRoot() to prevent path traversal attacks in
extraction and context building
- Clamp MCP tool inputs (limit, depth, maxDepth) to sane ranges
- Use atomic writes (temp file + rename) for config saves
- Add symlink cycle detection in directory scanning to prevent infinite loops
- Replace all JSON.parse calls in db/queries.ts with safeJsonParse fallbacks
to handle corrupted database metadata gracefully
- Add cross-process FileLock for DB write operations (indexAll, indexFiles,
sync) to prevent concurrent writes from CLI, MCP server, and git hooks
- Remove unused path import from context/index.ts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds support for Dart and Liquid languages with tree-sitter parsing.
Improves accuracy of code symbol extraction for existing languages.
Indexes project files to enhance code navigation features.
Migrates build system to facilitate code contributions.
Removes git hook functionality.
Integrates Sentry for error tracking and reporting.
Enhances project initialization and configuration loading.
- Add evaluation test suite with TypeScript and Python fixtures
- Fix MCP server to defer CodeGraph init until rootUri received
- Fix call edge extraction by calling resolveReferences() after indexAll/sync
- Fix glob matching for root-level files (e.g., **/*.py now matches auth.py)
- Fix duplicate node extraction for methods inside classes
- Update context tests to use buildContext for semantic search + graph traversal
- Export unused formatter functions to fix build
Evaluation results:
- TypeScript: 96% precision, 79% recall, 85% F1
- Python: 99% precision, 80% recall, 85% F1
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
andreinknvandGitHub
Martin Oehlert