Batch getNodesByIds to collapse N+1 reads in graph traversal, invalidate the
insertNode LRU cache so INSERT OR REPLACE doesn't serve a stale row, and run
incremental PRAGMA optimize + passive WAL checkpoint after bulk writes.
Closes#108
* fix(db): eliminate concurrent-read "database is locked"; add node:sqlite backend (#238)
WAL + busy_timeout were already enabled, so the issue's suggested fix was a
no-op. The real causes, addressed here:
- busy_timeout is now set first (before journal_mode) and lowered 120s -> 5s,
so open-time pragmas wait out a lock instead of hanging for two minutes.
- getCodeGraph no longer opens a second connection to the default project when
a tool passes its own projectPath (the in-process lock amplifier).
- The wasm fallback (no WAL) gets a bounded read-retry on SQLITE_BUSY.
- New: node:sqlite backend, preferred over wasm, so installs whose native
better-sqlite3 build fails land on a real-WAL backend instead of no-WAL wasm.
- codegraph status / codegraph_status now report the effective journal mode, so
a lock report is triageable (wal vs delete).
- CLI hard-blocks Node < 20 to actually enforce the engines floor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(db)!: node:sqlite is the sole backend; drop better-sqlite3 + wasm
Now that distribution will bundle a Node 24 runtime, node:sqlite (real SQLite
with WAL + FTS5) is always available. Collapse the three-backend adapter to
node:sqlite only and remove the machinery the other two needed:
- Remove better-sqlite3 (optionalDependency) and node-sqlite3-wasm (dependency).
- Remove WasmDatabaseAdapter, the named->positional param translation, the
SQLITE_BUSY read-retry, the wasm fallback banner, the backend env override,
and the native/node-sqlite/wasm selection chain.
- createDatabase now opens node:sqlite directly, with a clear error pointing at
the bundled release / Node 22.5+ when the module is absent.
- NodeSqliteAdapter.close() is idempotent and pragma() supports { simple }, to
match the better-sqlite3 behavior callers relied on.
- status (CLI + MCP) reports the single node:sqlite backend; journal-mode
diagnostics and the getCodeGraph single-connection fix are retained.
- Tests repointed off better-sqlite3 onto node:sqlite.
Net -1044 lines. Running from source now requires Node 22.5+.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dist): self-contained bundle prototype (vendored Node + install channels)
Phase 3 of the node:sqlite migration: ship a vendored Node runtime so CodeGraph
runs with no system Node and no native build (node:sqlite is built in).
- scripts/build-bundle.sh: build a per-platform archive (official Node + dist +
prod deps + launcher). Same recipe per platform; pins Node v24.16.0.
- install.sh: curl|sh installer (no Node required) — detects os/arch, pulls the
archive from Releases, symlinks onto PATH; re-run to upgrade, --uninstall to
remove. The VPS/SSH path.
- scripts/npm-shim.js: thin launcher for the npm channel — resolves the
per-platform optionalDependency bundle and execs it, so `npm i -g` keeps
working and the real work runs on the bundled Node regardless of the user's.
- BUNDLING.md: distribution design + release-pipeline TODO (CI matrix, platform
packages, code signing, brew, retiring the Node-version gate).
Validated end-to-end: darwin-arm64 and linux-x64 bundles both run init + index +
status (Backend: node:sqlite, Journal: wal) + FTS query with NO system Node —
linux-x64 verified in a clean ubuntu:24.04 amd64 container. Release archives are
gitignored; CI will produce and upload them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dist): add Windows PowerShell installer (install.ps1)
The `irm … | iex` one-liner for Windows, mirroring install.sh: detect arch,
pull the matching bundle from Releases, extract to %LOCALAPPDATA%\codegraph,
add it to user PATH. Re-run to upgrade. (Windows bundle production in
build-bundle.sh is still TODO.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dist): release workflow + npm packaging; README/CHANGELOG for bundled distro
- .github/workflows/release.yml: manually-triggered (workflow_dispatch) release
matrix. Builds a self-contained bundle per platform on its own runner
(darwin-arm64/x64, linux-x64/arm64), publishes a GitHub Release with all
archives, and publishes the npm thin-installer (shim + per-platform packages).
Windows targets are TODO (build-bundle.sh is unix-only).
- scripts/pack-npm.sh: assemble the npm packages from built bundles — per-platform
packages tagged os/cpu + the main shim package with them as optionalDependencies
(esbuild pattern). Proven locally: npm-install the tarballs, run via the shim,
resolves the bundle and runs on the bundled Node 24 (node:sqlite / WAL).
- README: install section now leads with the no-Node one-liners (curl|sh, irm|iex)
then npm/npx; "bundled · none required" badge.
- CHANGELOG: standout headline for the self-contained release, plus Added/Changed/
Removed for the install channels, node:sqlite backend, and dropped deps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dist): Windows bundles + single-trigger release workflow
- build-bundle.sh: add win32-x64 / win32-arm64 targets — download Node's Windows
zip, bundle node.exe + a .cmd launcher, output a .zip. Verified structurally
(PE32+ node.exe, CRLF .cmd, portable node_modules). Since there are no native
addons, any target builds on any OS, so the whole matrix builds on one runner.
- pack-npm.sh: handle .zip bundles and win32 packages (os: win32, node.exe).
- release.yml: simplified to your spec — manual trigger reads the version from
package.json, builds all platform bundles, creates the GitHub Release with notes
pulled from CHANGELOG.md, and publishes the npm shim + platform packages.
- BUNDLING.md: Windows + build-anywhere notes; release pipeline documented.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`codegraph_callees stage_apply::run` (and `_node`, `_impact`, ...)
returned "not found" against a repo with 7-9 sibling Rust modules,
each exporting `pub async fn run`. Two underlying issues:
1. The FTS5 query builder stripped `:` as a special char without
splitting on `::`, so `stage_apply::run` collapsed to the literal
`stage_applyrun` which matches nothing. Treat `::` as whitespace
before the strip step so both halves become FTS tokens.
2. `matchesSymbol` only understood `Parent.child` qualifiers and
relied on `qualifiedName` carrying the module path. Rust file-
level functions don't have their module name in `qualifiedName`
(it's encoded in the file path instead), so even dot-style
lookups failed. Accept `::`, `.`, `/` as separators; multi-level
forms compose; Rust `crate::`/`super::`/`self::` prefixes get
stripped before path matching. Fall back to file-path containment
when the qualified-name suffix doesn't match — `stage_apply::run`
matches a `run` in any file whose path has a `stage_apply` segment.
Also tightens the no-match branch: qualified lookups no longer fall
through to a fuzzy text match. `stage_apply::nonexistent_fn` returns
`null` instead of silently resolving to an unrelated `rollback` in
the same file.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the visibility gap behind issues #138 (WASM-on-macOS) and #139
(MCP "database is locked"). `better-sqlite3` is in optionalDependencies,
so when the native build fails npm install still succeeds and the
runtime silently falls back to node-sqlite3-wasm — 5-10x slower and
without WAL, so writers block readers (which is what makes the MCP
server appear to "lock the DB" in #139). The only existing signal was
a one-line `console.warn` to stderr that MCP transports typically
swallow.
This patch does NOT change install behavior — better-sqlite3 stays in
optionalDependencies so cross-platform installs keep working. It just
makes the substitution observable + recoverable.
## Visibility (4 surfaces)
- CLI `codegraph status`: new `Backend:` line under Index Statistics.
`native` rendered green; `wasm` rendered yellow with an inline
`npm rebuild better-sqlite3` nudge. Also exposed in `--json` as
`backend: 'native' | 'wasm'`.
- MCP `codegraph_status`: new `**Backend:**` line. Native form reads
`native (better-sqlite3)`; wasm form prepends a warning glyph and
includes the full fix recipe.
- Stderr banner on fallback (`buildWasmFallbackBanner`): replaces the
bare one-line `console.warn` with a multi-line bordered banner
covering macOS + Linux fix steps and optionally appending the
native load error.
- README troubleshooting: new "Indexing is slow / MCP database is
locked / WASM fallback active" entry that walks users to the
`Backend:` line and the fix.
## Per-instance backend tracking
`createDatabase` previously set a module-level `activeBackend` global.
MCP can open multiple project DBs in one process via the
`getCodeGraph()` cache, so the global would race / overwrite. Refactor:
`createDatabase` now returns `{db, backend}`, `DatabaseConnection`
carries `private backend` and exposes `getBackend()`, and
`CodeGraph.getBackend()` is the public surface. The CLI and MCP both
call `cg.getBackend()`.
## What this does NOT fix
The root cause of users landing on WASM is environment-specific (Mac
without Xcode CLT, Node version mismatch, etc.) and not fixable in
code without changing the optionalDependencies design. The README
entry tells users what to run; `Backend: native` after rebuild is the
confirmation signal.
## Tests
New `__tests__/sqlite-backend.test.ts` (6 tests) pins the banner
recipe content (so future edits can't strip the recovery commands),
the `WASM_FALLBACK_FIX_RECIPE` constant, and per-instance
`DatabaseConnection.getBackend()` / `CodeGraph.getBackend()` reporting.
Suite: 503 → 509, all passing.
Credit to @andreinknv whose analysis on #138 (and patches on his fork
at 6d0e7a2 + 69f7001) framed the visibility approach.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Both narrow indexes are fully covered by the existing (source, kind)
and (target, kind) composites via SQLite's left-prefix scan, so
they're dead weight on every write. Empirical measurements (from the
spike script in PR #122 on a 50K-node / 250K-edge synthetic DB):
- DB size: 34.7 MB → 27.0 MB (-22.2%)
- Bulk insert (250K edges): 590ms → 431ms (1.37× faster)
- source/target lookup latency: no regression
Adds migration v4 to drop both on existing databases; fresh-DB schema
no longer creates them.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(search): field-qualified queries (kind:/lang:/path:/name:) + fuzzy typo fallback
Two UX improvements that turn a free-text search into something a
real user can drive precisely.
1) Field-qualified queries.
A new query parser (src/search/query-parser.ts) splits the raw query
into structured filters and a free-text remainder:
kind:function name:auth path:src/api authenticate
becomes
{ kinds: ['function'], nameFilters: ['auth'],
pathFilters: ['src/api'], text: 'authenticate' }
Filters compose with the SearchOptions arg (intersection). Unknown
prefixes pass through as plain text so `query "TODO:"` keeps working.
Quoted values (`path:"my dir"`) handle whitespace. When the user
specifies only filters with no text, the search uses a filter-only
candidate scan instead of bailing out.
Recognised today:
kind: any NodeKind value
lang: any Language value (alias: language:)
path: case-insensitive substring of file_path
name: case-insensitive substring of node.name
2) Fuzzy fallback.
When BOTH FTS and LIKE return nothing AND the text is at least 3
chars, the resolver scans the distinct-name set with a bounded
Damerau-Levenshtein-style edit distance (≤2 for ≥5 chars, ≤1 for
4-char queries, off for shorter). Bounded edit-distance early-exits
once the row min exceeds maxDist, so this stays O(distinct-names *
avg-name-length) with a very low constant.
Verified live against ollama/ollama@v0.22.0:
query "kind:function auth" → only function-kind hits
query "lang:go path:server route" → Go files under server/
query "getUssr" (typo) → finds getUser, SetUser
query "confg" (typo) → finds Config
Full test suite: 380 passed.
* fix(search): address reviewer findings — tokenizer mid-token quotes, fuzzy fan-out cap, larger filter-only over-fetch, unit tests
Five fixes from independent review:
- parseQuery tokenizer: quotes that appear MID-token (path:"my dir/
file") were not being recognised — only quotes at the start of a
token were treated as quoted spans. The fixture path:"my dir"
parsed as ['path:"my', 'dir"'] instead of ['path:"my dir"'].
Tokeniser is now a single state machine that scans into a token
until whitespace OR a quote, and recognises quotes anywhere within
the token (skips to the matching close quote).
- searchNodesFuzzy: cap the per-name follow-up SQL queries at
Math.max(limit*2, 50) AFTER edit-distance filtering. Without
this, a project with many similar names (getUser1, getUser2...)
could fan out far beyond limit queries before the inner-loop
break kicks in.
- searchAllByFilters (filter-only no-text path): bumped over-fetch
multiplier from 2× to 5× so a selective post-filter (e.g.
path:src/very/specific/file.ts) doesn't return fewer than limit
results despite the DB having matches.
- 23 new unit tests in __tests__/search-query-parser.test.ts:
parseQuery covers known-field filter, lang/language alias,
multiple kind: ORs, quoted spans (incl. mid-token), URL
passthrough, empty-value passthrough, unknown prefix passthrough,
unknown value passthrough, all-filters-no-text, empty input,
20k-char input. boundedEditDistance covers identity, single
insertion/deletion/substitution, length-difference shortcut,
empty inputs, case-sensitivity, early-exit correctness.
Full test suite: 853 passed (up from 830).
* refactor(search): derive parser kind/lang sets from types.ts as const
Convert NodeKind and Language to runtime-iterable as const arrays
(NODE_KINDS, LANGUAGES) so the query parser imports the canonical
list instead of duplicating it. Also fix the path: JSDoc to say
substring (matches the .includes() impl).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Removes @xenova/transformers dependency, vector storage tables, embedding generation, and semantic search APIs. Simplifies context building to use only FTS search. Eliminates visualizer server, postinstall model download, and related CLI commands. Reduces package size and complexity while maintaining core static analysis capabilities.
Uses max FTS score as baseline for exact name matches to ensure nameMatchBonus differentiation during rescoring, increases exact match limit from 5 to 20 candidates, and adds common conversational terms to stop words to reduce query noise.
Addresses cases where BM25 can bury short exact-match names (e.g. "Query") under hundreds of compound names (e.g. "QueryParserTokenManager") in large codebases, pushing them past the FTS fetch limit before post-hoc scoring can help. Supplements primary search results with direct case-insensitive name lookups for each query term, ensuring exact matches are always candidates for scoring.
Addresses cases where BFS with multiple entry points leaves most nodes disconnected after trimming. Discovers edges between already-selected nodes using specific relationship types (calls, extends, implements, references, overrides) to recover inter-node connectivity that would otherwise be lost during the node selection process.
Introduces LIKE-based substring matching to find symbols like "Search" within "TransportSearchAction" that FTS cannot match due to tokenization boundaries. Adds dedicated type hierarchy traversal to ensure parent/child classes and interfaces are included in context results, preventing BFS budget exhaustion on method-level nodes before reaching inheritance relationships.
Adds nameMatchBonus scoring to prioritize results where node names exactly or partially match query terms. Implements dedicated field extraction for Java/C# to properly categorize class fields vs variables. Optimizes BM25 search with column weights favoring name matches and increased result fetching before post-processing. Refines stop words list to preserve common programming terms like "get", "find", "list".
Improves search accuracy by boosting results when multiple query symbols appear in the same file, addressing cases where common names like "run" return too many results. Adds Go method receiver type extraction to qualified names for better searchability (e.g., "scrapeLoop.run"). Optimizes database queries with two-pass approach to handle distinctive vs common symbol names efficiently.
Replaces O(n) file scanning with O(log n) indexed database lookups by adding getAllNodeNames query and caching node lookups by name/qualified name. Pre-filters references against known symbol names to skip expensive resolution for non-existent symbols. Consolidates Go resolver helper functions into a unified resolveByNameAndKind function and moves built-in symbol sets to module-level constants for better performance.
Adds expression index on lower(name) for memory-efficient case-insensitive searches, replacing in-memory caches that caused OOM on large codebases. Includes batched reference resolution, enhanced error reporting with detailed breakdown by error type, and improved CLI progress display for scanning phases.
Java extraction:
- Handle Java method_invocation AST (receiver.method pattern)
- Support Java extends_interfaces and super_interfaces with type_list
- Create unresolved references for Java imports for cross-file resolution
- Extract interface inheritance via extractInheritance
MCP tools:
- Aggregate callers/callees/impact across ALL matching symbols (e.g. multiple
overloads or same-named methods in different classes)
- New findAllSymbols() helper for multi-symbol lookup
Graph traversal:
- Impact analysis now traverses into container children (class → methods)
so that callers of methods appear in the impact radius of their class
Other:
- Add deleteSpecificResolvedReferences() for precise cleanup after resolution
- Add 'instance-method' to resolvedBy union type
- Version bump to 0.6.8
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Eliminates anonymous error reporting functionality that was collecting stack traces and error context via Sentry. Removes all telemetry-related code, configuration options, and documentation references.
Replace native tree-sitter with web-tree-sitter + tree-sitter-wasms for
universal cross-platform support. Add node-sqlite3-wasm as a fallback
when better-sqlite3 native bindings aren't available. Move better-sqlite3
and sqlite-vss to optionalDependencies so installs never fail.
Fix installer to use npx fallback when global npm install fails, so MCP
config, hooks, and quick-start instructions all work without the bare
codegraph command in PATH.
Fix tests: update schema version expectation, fix db test paths and
method names, extract MAX_OUTPUT_LENGTH as module constant, normalize
Windows path separators in import resolver.
- Remove getNodesByIds, getNodesByKinds (zero callers)
- Remove getFileHashMap, getFileSyncMap (zero callers)
- Remove getDynamicStmt cache (only used by removed batch methods)
- Revert getStaleFiles to simple implementation (zero callers, no need to optimize)
- Remove intent field from SearchOptions (never read in search logic)
- Add provenance column on edges for tracking how edges were created
- Add project_metadata table for version/provenance tracking
- Make unresolved_refs file_path/language NOT NULL with defaults
- Add composite indexes for unresolved_refs and edges.provenance
- Update v2 migration to handle all new schema additions
- Record schema version on initialize to prevent re-migration
- Add dynamic prepared statement cache (getDynamicStmt) for varying SQL
- Add batch methods: getNodesByIds, getNodesByKinds, getFileHashMap, getFileSyncMap
- Add project metadata methods: getMetadata, setMetadata, getAllMetadata
- Optimize getStats to use single aggregate query
- Optimize getStaleFiles to use temporary table JOIN
- Add provenance parameter to getOutgoingEdges
- Add intent field to SearchOptions type
Node insertion used plain INSERT which crashes on duplicate IDs.
In large C/C++ projects, tree-sitter can produce duplicate nodes for
the same symbol (e.g. typedef struct where both struct_specifier and
type_definition resolve to the same name/kind/line, or multiple
anonymous constructs on the same line).
- Change nodes INSERT to INSERT OR REPLACE (idempotent, same data)
- Change edges INSERT to INSERT OR IGNORE (skip duplicate edges)
The node ID is sha256(filePath:kind:name:line) which already uses full
relative paths, so cross-directory collisions are not the issue.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- SQLite performance pragmas: synchronous=NORMAL, 64MB cache,
memory temp store, 256MB mmap (safe with WAL mode)
- Batch insert for unresolved refs: single transaction instead of
N individual inserts per file
- Symbol caching (warmCaches): pre-load all nodes into memory maps
before resolution, eliminating repeated SQLite queries per ref
- Async file I/O: fs.stat/readFile in indexFile() are now non-blocking
- Denormalize filePath/language onto UnresolvedReference: avoids N
node lookups during resolution, with schema migration v2
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements security improvements inspired by PR #16 (credit: MO2k4):
- Add validatePathWithinRoot() to prevent path traversal attacks in
extraction and context building
- Clamp MCP tool inputs (limit, depth, maxDepth) to sane ranges
- Use atomic writes (temp file + rename) for config saves
- Add symlink cycle detection in directory scanning to prevent infinite loops
- Replace all JSON.parse calls in db/queries.ts with safeJsonParse fallbacks
to handle corrupted database metadata gracefully
- Add cross-process FileLock for DB write operations (indexAll, indexFiles,
sync) to prevent concurrent writes from CLI, MCP server, and git hooks
- Remove unused path import from context/index.ts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds support for Dart and Liquid languages with tree-sitter parsing.
Improves accuracy of code symbol extraction for existing languages.
Indexes project files to enhance code navigation features.
Migrates build system to facilitate code contributions.
Removes git hook functionality.
Integrates Sentry for error tracking and reporting.
Enhances project initialization and configuration loading.
- Add evaluation test suite with TypeScript and Python fixtures
- Fix MCP server to defer CodeGraph init until rootUri received
- Fix call edge extraction by calling resolveReferences() after indexAll/sync
- Fix glob matching for root-level files (e.g., **/*.py now matches auth.py)
- Fix duplicate node extraction for methods inside classes
- Update context tests to use buildContext for semantic search + graph traversal
- Export unused formatter functions to fix build
Evaluation results:
- TypeScript: 96% precision, 79% recall, 85% F1
- Python: 99% precision, 80% recall, 85% F1
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
andreinknvandGitHub
Martin Oehlert