Two coupled changes addressing the issue's underlying ask — "how does the
agent know when the index lags" — without resorting to a static wait.
Per-file staleness banner
-------------------------
FileWatcher now tracks per-path `pendingFiles` (path, firstSeenMs,
lastSeenMs, indexing) — events since the last successful sync, cleared
only after a sync whose `syncStartedMs >= lastSeenMs` commits. Chokidar
initial-scan events are gated behind a `ready` flag (with `waitUntilReady()`
exposed so tests can deterministically wait through it) so a fresh startup
doesn't falsely flag every existing file as pending.
ToolHandler now wraps every code-returning response (search, context,
callers, callees, impact, trace, explore, node, files) with
`withStalenessNotice`: intersects "files referenced in the response" with
`getPendingFiles()` and emits a hybrid signal —
* banner at the top for files referenced AND pending (with edit age +
indexing/pending-sync state, telling the agent to Read those specific
files directly; the rest of the response stays fresh and codegraph
stays authoritative for it),
* compact footer for pending files elsewhere in the project not
referenced above (capped at 5).
Cost is one boolean check + N substring matches when pending; zero
allocation when idle. `codegraph_status` surfaces the same data as a
first-class `### Pending sync:` section so the agent can ask "is the index
caught up?" in one call.
Cross-project quirk: when an agent passes `projectPath` matching the
default session's project, the staleness wrapper switches from the cached
cross-project CodeGraph (no watcher) to the default one (with watcher) so
the signal still fires. Same fix applied to `handleStatus`.
CODEGRAPH_WATCH_DEBOUNCE_MS
---------------------------
MCP `serve --mcp` now reads `CODEGRAPH_WATCH_DEBOUNCE_MS` and forwards it
to `cg.watch({ debounceMs })`. Clamped to [100ms, 60s]; out-of-range or
non-numeric values fall back to the FileWatcher default (2000ms). Active
value is logged to stderr on watcher startup so it's discoverable. The
docs in `server-instructions.ts`, `installer/instructions-template.ts`,
and `.cursor/rules/codegraph.mdc` no longer claim "~500ms"; they now
describe the banner mechanism instead — since per-file staleness replaces
the "wait N ms" guidance entirely, the docs become accurate at any
debounce value.
Validation
----------
* 847 unit/integration tests pass (added 15 new ones — pending-file
tracking, banner/footer routing, status section, env-var parsing).
* Direct MCP probe through a real `codegraph serve --mcp` process: edit a
file, query within the debounce window, banner fires naming the
edited file with edit-age.
* Real Claude TUI session via `scripts/agent-eval/itrun.sh` with
`CODEGRAPH_WATCH_DEBOUNCE_MS=10000`: agent edits `math.ts`, calls
`codegraph_explore`, reads the banner, **and discloses it unprompted in
its final reply**: "note: symbol index is mid-sync for the new `divide`,
but the source it returned is verbatim from disk."
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Batch getNodesByIds to collapse N+1 reads in graph traversal, invalidate the
insertNode LRU cache so INSERT OR REPLACE doesn't serve a stale row, and run
incremental PRAGMA optimize + passive WAL checkpoint after bulk writes.
Closes#108
Remove .codegraph/config.json and the entire config surface. CodeGraph now
indexes every file whose extension maps to a supported language and respects
.gitignore everywhere — git repos via git itself, non-git projects via the
`ignore` library (root + nested .gitignore files, the same way git does).
- Remove CodeGraphConfig/DEFAULT_CONFIG, src/config.ts, and the public config
API (the `config` option on init, getConfig/updateConfig/getConfigPath).
- Derive the source-file allowlist from EXTENSION_MAP (isSourceFile); maxFileSize
is now a constant. Drop the .codegraphignore marker.
- Behavior change: committed, non-gitignored dirs (vendor/, a committed dist/)
are now indexed — .gitignore is the single source of truth.
Earlier inert fields (languages, frameworks, extractDocstrings, trackCallSites,
customPatterns) and their dead helpers are removed as part of this.
Resolves#283.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(db): eliminate concurrent-read "database is locked"; add node:sqlite backend (#238)
WAL + busy_timeout were already enabled, so the issue's suggested fix was a
no-op. The real causes, addressed here:
- busy_timeout is now set first (before journal_mode) and lowered 120s -> 5s,
so open-time pragmas wait out a lock instead of hanging for two minutes.
- getCodeGraph no longer opens a second connection to the default project when
a tool passes its own projectPath (the in-process lock amplifier).
- The wasm fallback (no WAL) gets a bounded read-retry on SQLITE_BUSY.
- New: node:sqlite backend, preferred over wasm, so installs whose native
better-sqlite3 build fails land on a real-WAL backend instead of no-WAL wasm.
- codegraph status / codegraph_status now report the effective journal mode, so
a lock report is triageable (wal vs delete).
- CLI hard-blocks Node < 20 to actually enforce the engines floor.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor(db)!: node:sqlite is the sole backend; drop better-sqlite3 + wasm
Now that distribution will bundle a Node 24 runtime, node:sqlite (real SQLite
with WAL + FTS5) is always available. Collapse the three-backend adapter to
node:sqlite only and remove the machinery the other two needed:
- Remove better-sqlite3 (optionalDependency) and node-sqlite3-wasm (dependency).
- Remove WasmDatabaseAdapter, the named->positional param translation, the
SQLITE_BUSY read-retry, the wasm fallback banner, the backend env override,
and the native/node-sqlite/wasm selection chain.
- createDatabase now opens node:sqlite directly, with a clear error pointing at
the bundled release / Node 22.5+ when the module is absent.
- NodeSqliteAdapter.close() is idempotent and pragma() supports { simple }, to
match the better-sqlite3 behavior callers relied on.
- status (CLI + MCP) reports the single node:sqlite backend; journal-mode
diagnostics and the getCodeGraph single-connection fix are retained.
- Tests repointed off better-sqlite3 onto node:sqlite.
Net -1044 lines. Running from source now requires Node 22.5+.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dist): self-contained bundle prototype (vendored Node + install channels)
Phase 3 of the node:sqlite migration: ship a vendored Node runtime so CodeGraph
runs with no system Node and no native build (node:sqlite is built in).
- scripts/build-bundle.sh: build a per-platform archive (official Node + dist +
prod deps + launcher). Same recipe per platform; pins Node v24.16.0.
- install.sh: curl|sh installer (no Node required) — detects os/arch, pulls the
archive from Releases, symlinks onto PATH; re-run to upgrade, --uninstall to
remove. The VPS/SSH path.
- scripts/npm-shim.js: thin launcher for the npm channel — resolves the
per-platform optionalDependency bundle and execs it, so `npm i -g` keeps
working and the real work runs on the bundled Node regardless of the user's.
- BUNDLING.md: distribution design + release-pipeline TODO (CI matrix, platform
packages, code signing, brew, retiring the Node-version gate).
Validated end-to-end: darwin-arm64 and linux-x64 bundles both run init + index +
status (Backend: node:sqlite, Journal: wal) + FTS query with NO system Node —
linux-x64 verified in a clean ubuntu:24.04 amd64 container. Release archives are
gitignored; CI will produce and upload them.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dist): add Windows PowerShell installer (install.ps1)
The `irm … | iex` one-liner for Windows, mirroring install.sh: detect arch,
pull the matching bundle from Releases, extract to %LOCALAPPDATA%\codegraph,
add it to user PATH. Re-run to upgrade. (Windows bundle production in
build-bundle.sh is still TODO.)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dist): release workflow + npm packaging; README/CHANGELOG for bundled distro
- .github/workflows/release.yml: manually-triggered (workflow_dispatch) release
matrix. Builds a self-contained bundle per platform on its own runner
(darwin-arm64/x64, linux-x64/arm64), publishes a GitHub Release with all
archives, and publishes the npm thin-installer (shim + per-platform packages).
Windows targets are TODO (build-bundle.sh is unix-only).
- scripts/pack-npm.sh: assemble the npm packages from built bundles — per-platform
packages tagged os/cpu + the main shim package with them as optionalDependencies
(esbuild pattern). Proven locally: npm-install the tarballs, run via the shim,
resolves the bundle and runs on the bundled Node 24 (node:sqlite / WAL).
- README: install section now leads with the no-Node one-liners (curl|sh, irm|iex)
then npm/npx; "bundled · none required" badge.
- CHANGELOG: standout headline for the self-contained release, plus Added/Changed/
Removed for the install channels, node:sqlite backend, and dropped deps.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(dist): Windows bundles + single-trigger release workflow
- build-bundle.sh: add win32-x64 / win32-arm64 targets — download Node's Windows
zip, bundle node.exe + a .cmd launcher, output a .zip. Verified structurally
(PE32+ node.exe, CRLF .cmd, portable node_modules). Since there are no native
addons, any target builds on any OS, so the whole matrix builds on one runner.
- pack-npm.sh: handle .zip bundles and win32 packages (os: win32, node.exe).
- release.yml: simplified to your spec — manual trigger reads the version from
package.json, builds all platform bundles, creates the GitHub Release with notes
pulled from CHANGELOG.md, and publishes the npm shim + platform packages.
- BUNDLING.md: Windows + build-anywhere notes; release pipeline documented.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes the visibility gap behind issues #138 (WASM-on-macOS) and #139
(MCP "database is locked"). `better-sqlite3` is in optionalDependencies,
so when the native build fails npm install still succeeds and the
runtime silently falls back to node-sqlite3-wasm — 5-10x slower and
without WAL, so writers block readers (which is what makes the MCP
server appear to "lock the DB" in #139). The only existing signal was
a one-line `console.warn` to stderr that MCP transports typically
swallow.
This patch does NOT change install behavior — better-sqlite3 stays in
optionalDependencies so cross-platform installs keep working. It just
makes the substitution observable + recoverable.
## Visibility (4 surfaces)
- CLI `codegraph status`: new `Backend:` line under Index Statistics.
`native` rendered green; `wasm` rendered yellow with an inline
`npm rebuild better-sqlite3` nudge. Also exposed in `--json` as
`backend: 'native' | 'wasm'`.
- MCP `codegraph_status`: new `**Backend:**` line. Native form reads
`native (better-sqlite3)`; wasm form prepends a warning glyph and
includes the full fix recipe.
- Stderr banner on fallback (`buildWasmFallbackBanner`): replaces the
bare one-line `console.warn` with a multi-line bordered banner
covering macOS + Linux fix steps and optionally appending the
native load error.
- README troubleshooting: new "Indexing is slow / MCP database is
locked / WASM fallback active" entry that walks users to the
`Backend:` line and the fix.
## Per-instance backend tracking
`createDatabase` previously set a module-level `activeBackend` global.
MCP can open multiple project DBs in one process via the
`getCodeGraph()` cache, so the global would race / overwrite. Refactor:
`createDatabase` now returns `{db, backend}`, `DatabaseConnection`
carries `private backend` and exposes `getBackend()`, and
`CodeGraph.getBackend()` is the public surface. The CLI and MCP both
call `cg.getBackend()`.
## What this does NOT fix
The root cause of users landing on WASM is environment-specific (Mac
without Xcode CLT, Node version mismatch, etc.) and not fixable in
code without changing the optionalDependencies design. The README
entry tells users what to run; `Backend: native` after rebuild is the
confirmation signal.
## Tests
New `__tests__/sqlite-backend.test.ts` (6 tests) pins the banner
recipe content (so future edits can't strip the recovery commands),
the `WASM_FALLBACK_FIX_RECIPE` constant, and per-instance
`DatabaseConnection.getBackend()` / `CodeGraph.getBackend()` reporting.
Suite: 503 → 509, all passing.
Credit to @andreinknv whose analysis on #138 (and patches on his fork
at 6d0e7a2 + 69f7001) framed the visibility approach.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses the need for automatic graph synchronization on file changes. Implements FileWatcher using native OS file events (FSEvents/inotify/ReadDirectoryChangesW) with 2-second debouncing to prevent thrashing on rapid saves. Filters changes against include/exclude patterns and ignores .codegraph directory modifications. Integrates with CodeGraph API (watch/unwatch/isWatching methods) and MCP server for automatic activation. Updates documentation to reflect shift from semantic to full-text search and removal of manual hook installation requirements.
Removes @xenova/transformers dependency, vector storage tables, embedding generation, and semantic search APIs. Simplifies context building to use only FTS search. Eliminates visualizer server, postinstall model download, and related CLI commands. Reduces package size and complexity while maintaining core static analysis capabilities.
Eliminates the intermediate 'finalizing' phase that was added as a progress bar transition state but served no functional purpose. Simplifies the progress flow by going directly from 'storing' to 'resolving' phases, removing associated UI labels and progress callbacks.
Adds strategic yield points and direct stdout writes to prevent progress animation from freezing when the main thread is blocked by synchronous operations. Introduces 'finalizing' phase to smooth transition between parsing and resolving steps, ensuring progress reaches 100% completion.
Adds --verbose flag to init and index commands that shows timestamped progress output instead of animated progress bars. Implements worker timeout protection (10s per file) and periodic worker recycling (every 500 files) to prevent WASM memory crashes from hanging the entire indexing process. Includes detailed logging of worker lifecycle events and memory usage for debugging large repository indexing issues.
Adds expression index on lower(name) for memory-efficient case-insensitive searches, replacing in-memory caches that caused OOM on large codebases. Includes batched reference resolution, enhanced error reporting with detailed breakdown by error type, and improved CLI progress display for scanning phases.
Testing showed semantic search produces significantly better results for
natural language queries that Claude writes. FTS alone often ranks
properties above their parent classes and misses conceptual matches.
Embeddings are now always on — the vector manager is created eagerly,
with model download and embedding generation still happening lazily.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes#54 — `codegraph init -i` crashes with "Fatal process out of memory: Zone"
on large codebases because all 16 tree-sitter WASM grammar modules were compiled
upfront by V8, exhausting the WASM Zone allocator.
Changes:
- initGrammars() now only initializes the tree-sitter WASM runtime (Parser.init()),
no longer eagerly loads all grammar files
- New loadGrammarsForLanguages() loads only grammars for languages actually present
in the project (e.g. a Dart project loads ~2-3 grammars instead of 16)
- Orchestrator detects needed languages after file scan, before parsing begins
- Embedding pipeline now uses quantized model (~67MB vs ~270MB) to further reduce
WASM memory pressure when embeddings are enabled
Replace native tree-sitter with web-tree-sitter + tree-sitter-wasms for
universal cross-platform support. Add node-sqlite3-wasm as a fallback
when better-sqlite3 native bindings aren't available. Move better-sqlite3
and sqlite-vss to optionalDependencies so installs never fail.
Fix installer to use npx fallback when global npm install fails, so MCP
config, hooks, and quick-start instructions all work without the bare
codegraph command in PATH.
Fix tests: update schema version expectation, fix db test paths and
method names, extract MAX_OUTPUT_LENGTH as module constant, normalize
Windows path separators in import resolver.
- Add validateProjectPath() to reject sensitive system directories
- Add isPathWithinRoot/isPathWithinRootReal for symlink-aware path checks
- Replace hand-rolled glob-to-regex with picomatch to prevent ReDoS
- Add isSafeRegex() to reject custom patterns with nested quantifiers
- Replace FileLock with PID-tracking version that detects stale locks
- Add symlink detection in removeDirectory/listDirectoryContents
- Add subdirectory name validation in ensureSubdirectory
- Add atomicWriteFileSync and corrupted file backup in config-writer
- Add MCP input validation (validateString) for all tool handlers
- Fix CLAUDE.md section replacement to handle ### subsections correctly
Implements security improvements inspired by PR #16 (credit: MO2k4):
- Add validatePathWithinRoot() to prevent path traversal attacks in
extraction and context building
- Clamp MCP tool inputs (limit, depth, maxDepth) to sane ranges
- Use atomic writes (temp file + rename) for config saves
- Add symlink cycle detection in directory scanning to prevent infinite loops
- Replace all JSON.parse calls in db/queries.ts with safeJsonParse fallbacks
to handle corrupted database metadata gracefully
- Add cross-process FileLock for DB write operations (indexAll, indexFiles,
sync) to prevent concurrent writes from CLI, MCP server, and git hooks
- Remove unused path import from context/index.ts
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds support for Dart and Liquid languages with tree-sitter parsing.
Improves accuracy of code symbol extraction for existing languages.
Indexes project files to enhance code navigation features.
Migrates build system to facilitate code contributions.
Removes git hook functionality.
Integrates Sentry for error tracking and reporting.
Enhances project initialization and configuration loading.
The nomic-ai model is now downloaded during npm install via a postinstall
script and stored globally in ~/.codegraph/models (shared across projects)
instead of per-project in .codegraph/models.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add evaluation test suite with TypeScript and Python fixtures
- Fix MCP server to defer CodeGraph init until rootUri received
- Fix call edge extraction by calling resolveReferences() after indexAll/sync
- Fix glob matching for root-level files (e.g., **/*.py now matches auth.py)
- Fix duplicate node extraction for methods inside classes
- Update context tests to use buildContext for semantic search + graph traversal
- Export unused formatter functions to fix build
Evaluation results:
- TypeScript: 96% precision, 79% recall, 85% F1
- Python: 99% precision, 80% recall, 85% F1
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Colby MchenryandGitHub
andreinknvandGitHub
Martin Oehlert