a11a439002dd0d7bda8683fb8da2b03deec956e6
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
a11a439002 |
fix(indexing): HDD-class storage — false parse timeouts, dropped files, and WAL checkpoint write-back (#1231) (#1242)
Parse timeouts are now judged by the worker's own clock: the base timer only marks a job late (after a long synchronous store stall, Node runs the timers phase before the poll phase, so the timer fired before an already-delivered result was processed — killing workers over parses that took milliseconds, even on 0-byte files); a result arriving before a 3× hard-kill backstop is accepted, timed-out files are retried, and CODEGRAPH_PARSE_TIMEOUT_MS overrides the budget. Grammar WASM bytes are pre-read once on the main thread and handed to every worker, so spawns/respawns load grammars from memory instead of re-reading a saturated disk. Bulk indexing defers WAL auto-checkpointing for the whole run: the default 1000-page interval re-writes hot B-tree/FTS pages into the main DB file over and over — ~95% of all disk I/O under throttled measurement. A WalCheckpointValve bounds WAL growth with off-thread PASSIVE backfill passes (never blocking the writer or the #850 watchdog heartbeat), pauses the writer for a full backfill if the disk truly can't keep up, and folds the WAL at the parse→resolution boundary so post-parse reads never page a bulk-write-sized WAL. Opt out with CODEGRAPH_NO_WAL_DEFER=1; tune with CODEGRAPH_WAL_VALVE_MB. Measured at 150 IOPS (HDD class): commons-lang 1526s → 59s with 0 dropped files (was 8); guava-scale completes in 7.6 min with a full graph where v1.3.1 needed 25 min for a repo 5× smaller. Unthrottled: no change. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
a3f90089e8 |
fix(indexing): bounded-memory yielding pipeline tail + daemon session fixes (#1212) (#1226)
Large-codebase indexing died at the end of "Resolving refs" two ways: watchdog kills of healthy work (24k-file Java on Windows, #1212 — third iteration of the #1091/#1122 class) and hard OOMs (Linux kernel scale, where v1.3.0 could not complete at any watchdog setting). Root causes: ~31 of 37 dynamic-edge synthesis passes ran start-to-finish with no yield points, several materialized whole-graph snapshots (kotlin expect/actual opened with getAllNodes() — 2M nodes in one array; the C fn-pointer pass retained every C file's contents twice plus every function node), and the post-index WAL checkpoint ran minutes of synchronous IO on the main thread, killing even a successful index at the finish line. The pipeline tail now follows the same discipline as the rest: never hold O(graph) in the heap, yield everywhere. - All synthesis passes stream node-kind scans (cursors, not arrays) and yield on time-budgeted checkpoints; language gates skip passes whose filters a project's file languages provably can't satisfy. - kotlin expect/actual filters SQL-side; c-fnptr caches are LRU-bounded, units stream one file at a time, and the all-functions array + write-only id map are gone; spring reads each .java once, not twice. - runMaintenance moved to a worker thread (own SQLite connection); per-file store commits chunk with yields behind a serialized flush chain (preserving #1015 file-order determinism); resolver warm-up streams the DISTINCT name set; resolution batch-tail and merged-edge inserts run in bounded sub-transactions. - Daemon: fixed a socket-handoff race that could leave a fresh MCP session permanently silent (client-hello tail unshifted into a flowing stream with zero listeners — the long-standing #662 test flake was this real bug); first tool call no longer queues behind the query pool's cold start (pool.ready gate). Validation: Linux kernel (70,129 files, 2.05M nodes, 6.4M edges) fully indexes in 27m8s on a 2-core/6GB container at default heap + default watchdog; llvm-project (180k files) completes under 1GB RSS including kill-and-sync recovery; synthesized-edge and full-graph parity are byte-identical vs baseline on elasticsearch/redis/vim; the ex-flaky daemon test passed 25/25 under load. Env-gated diagnostics kept: CODEGRAPH_SYNTH_TIMINGS pass/phase timings, CODEGRAPH_MCP_DEBUG hop tracing. Design record: docs/design/main-thread-stall-followup.md. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
9684b3b5a5 |
fix(index): rebuild a poisoned/oversized index by recreating the DB, not row-DELETE (#1067) (#1073)
Follow-up to #1065/#1066. Those stopped a *new* index from scanning an ignored gitlink corpus, but a project that had already built the multi-GB graph before upgrading still couldn't recover: `codegraph index` printed only "Indexing project" and was then SIGKILLed (137) by the #850 watchdog ~60s later, before scanning even started. Root cause is not the scanner. `index` cleared the old graph with a synchronous `DELETE FROM nodes/edges/files`. `nodes` carries an FTS5 `AFTER DELETE` trigger, so deleting ~1.6M rows fires ~1.6M FTS delete-markers — O(rows), and it grows the WAL further before it can finish. A deterministic probe puts the DELETE-clear at 20.4s on 1.5M synthetic nodes (WAL 1.16->2.14GB); at the report's denser ~2.6KB/node WAL that crosses the 60s main-thread watchdog. `open()` was never the wedge. A full re-index is documented as "same result as a fresh init", so make it one: discard the database files and re-initialize, instead of opening the old DB and DELETE-ing every row. - db: add removeDatabaseFiles(dbPath) — unlinks codegraph.db + its -wal/-shm sidecars (O(1) regardless of size; sidecars best-effort). - index: add CodeGraph.recreate(projectRoot) — discards the files and returns a fresh, empty instance. Never opens or migrates the poisoned DB. POSIX unlinks an open file fine (a live daemon heals via reopenIfReplaced, #925); a Windows file lock becomes an actionable "stop the daemon / remove .codegraph" error. - cli: `codegraph index` now calls recreate() instead of open()+clear(); both clear() calls dropped. The public clear() API is unchanged. This also reclaims the disk the bloated db/-wal were holding. Validated: deterministic probe (DELETE O(rows) vs recreate O(1)); an end-to-end run through the built binary recovering a real 800K-node / 419MB poisoned DB in 0.3s with no wedge and the correct small graph; new unit + CLI regression tests; existing #874 index tests still green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
0a91d0f512 |
perf(resolution): fix O(K²) import-node blowup in "Resolving refs" (#915) (#965)
* perf(resolution): resolve imports to definitions, not sibling import nodes (#915) "Resolving refs" crawled (tens of minutes) on large projects — most painfully ones mixing a big front-end and back-end. An external package or module imported across hundreds/thousands of files (react, a shared UI package, Python logging/typing) is re-declared as an `import` node in every importing file, so its unresolved import ref fell through to the exact-name matcher, which scored all K same-named import nodes via findBestMatch — K refs x K candidates = O(K^2) per package, producing only meaningless import->import edges. Fix: exclude `import`-kind nodes as name-match targets (they're statements, not definitions; real import->definition resolution is the import resolver's job). Plus two safe constant-factor wins in findBestMatch: hoist the per-candidate ref.filePath split, and skip cross-language candidates when a same-language one exists (provably the same winner — same-language scores >=50, cross-language maxes at 35). Measured: superset (Py+TS) candidates scored 7.5M -> 833K (9x), non-import edges preserved (+1618 now resolve to real defs), ~22K useless import->import edges removed; kubernetes (Go) computePathProximity 37.2s -> 5.0s; synthetic 8k-file mixed repo (K=4000) resolution 16.0s -> 1.7s. Full suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: correct stale better-sqlite3/wasm references to node:sqlite The SQLite backend has been Node's built-in node:sqlite (real SQLite, WAL + FTS5, from the bundled runtime) for a while — there is no native build step and no node-sqlite3-wasm fallback. README and the docs site were already updated; this catches the stragglers: - CLAUDE.md: the src/db/ backend description and the sqlite-backend test note. - src/db/index.ts, src/mcp/tools.ts: two code comments that still blamed "the wasm backend" for non-WAL behavior (reworded to "when WAL isn't in effect"). Leaves tree-sitter grammar wasm (web-tree-sitter / --liftoff-only) untouched — that's a different, still-current use of wasm. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * chore(telemetry): drop the dead sqlite_backend field (schema v2) node:sqlite is now the only backend, so the `index` event's `sqlite_backend` field was a constant ("native") carrying no signal — and the `install` event never actually sent it. Remove the field and the backendKind() helper, bump the telemetry SCHEMA_VERSION 1 -> 2, and update TELEMETRY.md + docs/design/telemetry.md. The ingest worker is deliberately left tolerant: `index` doesn't require the field and schema_version validates as nonNegInt(99), so v2 events ingest fine and old clients still sending v1 + sqlite_backend keep validating too. Added a legacy comment there explaining it's safe to drop once old-client share is negligible. telemetry.test.ts: the assertion pinning schema_version and a stale-claim fixture line updated 1 -> 2. All telemetry tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
e43ac82cdf |
fix(mcp): reopen the database when it's replaced on disk instead of serving a deleted inode (#925) (#940)
A long-lived `serve --mcp` process opens `.codegraph/codegraph.db` and holds the fd for its whole life. If `.codegraph/` is removed and recreated AT THE SAME PATH while it runs — `git worktree remove <p>` + re-add, or `rm -rf .codegraph` + `codegraph init` — the held fd points at the now-unlinked inode and can never see the new index. The server serves the pre-removal snapshot (renamed/removed symbols still "live", new ones missing); `codegraph sync` can't refresh it and the CLI (a fresh process) diverges. Only a restart fixed it — and because the daemon registry is keyed by path, a same-path recreate routes new clients straight back to the same stale daemon, so the fix has to self-heal inside the running process. - DatabaseConnection records the DB file's (dev, ino) at open and exposes isReplacedOnDisk() — a different inode now at the same path. POSIX-gated: Windows can't unlink an open file and its st_ino is unreliable, so it never fires there. - CodeGraph.reopenIfReplaced() opens the live file first, then swaps the connection + query layers IN PLACE (via the new wireLayers() helper), so every holder of the instance (the daemon's default project, cached projectPath connections) heals without a restart. Closing the dead handle also frees the leaked db/-wal/-shm fds pinning the unlinked inode. - ToolHandler.getCodeGraph calls it (freshen) before serving — one stat() per call, a no-op unless the inode actually changed, never throws into a tool. Tests cover isReplacedOnDisk (unchanged / replaced / absent / Windows-gated) and an end-to-end reopen that heals a held instance after a same-path recreate (asserts the pre-heal staleness too). Validated on macOS with a dist probe of the raw instance and the MCP serving path; full suite green. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
a56d9e6941 |
feat(directory): CODEGRAPH_DIR env var to override the index dir name (#636) (#741)
Two environments that share one working tree — most concretely Windows and WSL — can't safely share a single `.codegraph/`: the daemon lockfile records a platform-specific pid + socket (named pipe vs Unix socket), and SQLite locking across the WSL2/Windows filesystem boundary is unreliable, so two daemons over one index risks corruption. Add a `CODEGRAPH_DIR` env var (default `.codegraph`) that overrides the per-project data directory name, so each environment keeps its own index in the same tree (e.g. `CODEGRAPH_DIR=.codegraph-win` on Windows). The name is resolved live and validated (rejects separators / `..` / absolute, falling back to the default with a one-time stderr warning). Indexing and file-watching now skip ANY `.codegraph-*` sibling so neither side trips over the other's data. Routes the previously-hardcoded `.codegraph` literals (db path, lockfile, error log, watcher ignore, file-scan skip, installer) through the resolver. No extraction-version bump — index content is unchanged. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
 andreinknvandGitHub
|
b13f2f1ba1 |
perf(db): batch node lookups, fix insertNode cache, run maintenance after writes (#108)
Batch getNodesByIds to collapse N+1 reads in graph traversal, invalidate the insertNode LRU cache so INSERT OR REPLACE doesn't serve a stale row, and run incremental PRAGMA optimize + passive WAL checkpoint after bulk writes. Closes #108 |
||
|
|
ac52fd76c0 |
Self-contained distribution: bundle Node + node:sqlite, drop better-sqlite3/wasm (closes #238) (#282)
* fix(db): eliminate concurrent-read "database is locked"; add node:sqlite backend (#238) WAL + busy_timeout were already enabled, so the issue's suggested fix was a no-op. The real causes, addressed here: - busy_timeout is now set first (before journal_mode) and lowered 120s -> 5s, so open-time pragmas wait out a lock instead of hanging for two minutes. - getCodeGraph no longer opens a second connection to the default project when a tool passes its own projectPath (the in-process lock amplifier). - The wasm fallback (no WAL) gets a bounded read-retry on SQLITE_BUSY. - New: node:sqlite backend, preferred over wasm, so installs whose native better-sqlite3 build fails land on a real-WAL backend instead of no-WAL wasm. - codegraph status / codegraph_status now report the effective journal mode, so a lock report is triageable (wal vs delete). - CLI hard-blocks Node < 20 to actually enforce the engines floor. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * refactor(db)!: node:sqlite is the sole backend; drop better-sqlite3 + wasm Now that distribution will bundle a Node 24 runtime, node:sqlite (real SQLite with WAL + FTS5) is always available. Collapse the three-backend adapter to node:sqlite only and remove the machinery the other two needed: - Remove better-sqlite3 (optionalDependency) and node-sqlite3-wasm (dependency). - Remove WasmDatabaseAdapter, the named->positional param translation, the SQLITE_BUSY read-retry, the wasm fallback banner, the backend env override, and the native/node-sqlite/wasm selection chain. - createDatabase now opens node:sqlite directly, with a clear error pointing at the bundled release / Node 22.5+ when the module is absent. - NodeSqliteAdapter.close() is idempotent and pragma() supports { simple }, to match the better-sqlite3 behavior callers relied on. - status (CLI + MCP) reports the single node:sqlite backend; journal-mode diagnostics and the getCodeGraph single-connection fix are retained. - Tests repointed off better-sqlite3 onto node:sqlite. Net -1044 lines. Running from source now requires Node 22.5+. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dist): self-contained bundle prototype (vendored Node + install channels) Phase 3 of the node:sqlite migration: ship a vendored Node runtime so CodeGraph runs with no system Node and no native build (node:sqlite is built in). - scripts/build-bundle.sh: build a per-platform archive (official Node + dist + prod deps + launcher). Same recipe per platform; pins Node v24.16.0. - install.sh: curl|sh installer (no Node required) — detects os/arch, pulls the archive from Releases, symlinks onto PATH; re-run to upgrade, --uninstall to remove. The VPS/SSH path. - scripts/npm-shim.js: thin launcher for the npm channel — resolves the per-platform optionalDependency bundle and execs it, so `npm i -g` keeps working and the real work runs on the bundled Node regardless of the user's. - BUNDLING.md: distribution design + release-pipeline TODO (CI matrix, platform packages, code signing, brew, retiring the Node-version gate). Validated end-to-end: darwin-arm64 and linux-x64 bundles both run init + index + status (Backend: node:sqlite, Journal: wal) + FTS query with NO system Node — linux-x64 verified in a clean ubuntu:24.04 amd64 container. Release archives are gitignored; CI will produce and upload them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dist): add Windows PowerShell installer (install.ps1) The `irm … | iex` one-liner for Windows, mirroring install.sh: detect arch, pull the matching bundle from Releases, extract to %LOCALAPPDATA%\codegraph, add it to user PATH. Re-run to upgrade. (Windows bundle production in build-bundle.sh is still TODO.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dist): release workflow + npm packaging; README/CHANGELOG for bundled distro - .github/workflows/release.yml: manually-triggered (workflow_dispatch) release matrix. Builds a self-contained bundle per platform on its own runner (darwin-arm64/x64, linux-x64/arm64), publishes a GitHub Release with all archives, and publishes the npm thin-installer (shim + per-platform packages). Windows targets are TODO (build-bundle.sh is unix-only). - scripts/pack-npm.sh: assemble the npm packages from built bundles — per-platform packages tagged os/cpu + the main shim package with them as optionalDependencies (esbuild pattern). Proven locally: npm-install the tarballs, run via the shim, resolves the bundle and runs on the bundled Node 24 (node:sqlite / WAL). - README: install section now leads with the no-Node one-liners (curl|sh, irm|iex) then npm/npx; "bundled · none required" badge. - CHANGELOG: standout headline for the self-contained release, plus Added/Changed/ Removed for the install channels, node:sqlite backend, and dropped deps. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(dist): Windows bundles + single-trigger release workflow - build-bundle.sh: add win32-x64 / win32-arm64 targets — download Node's Windows zip, bundle node.exe + a .cmd launcher, output a .zip. Verified structurally (PE32+ node.exe, CRLF .cmd, portable node_modules). Since there are no native addons, any target builds on any OS, so the whole matrix builds on one runner. - pack-npm.sh: handle .zip bundles and win32 packages (os: win32, node.exe). - release.yml: simplified to your spec — manual trigger reads the version from package.json, builds all platform bundles, creates the GitHub Release with notes pulled from CHANGELOG.md, and publishes the npm shim + platform packages. - BUNDLING.md: Windows + build-anywhere notes; release pipeline documented. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
55daeffe13 |
fix(db): surface SQLite backend in status + actionable WASM-fallback banner (#148)
Closes the visibility gap behind issues #138 (WASM-on-macOS) and #139 (MCP "database is locked"). `better-sqlite3` is in optionalDependencies, so when the native build fails npm install still succeeds and the runtime silently falls back to node-sqlite3-wasm — 5-10x slower and without WAL, so writers block readers (which is what makes the MCP server appear to "lock the DB" in #139). The only existing signal was a one-line `console.warn` to stderr that MCP transports typically swallow. This patch does NOT change install behavior — better-sqlite3 stays in optionalDependencies so cross-platform installs keep working. It just makes the substitution observable + recoverable. ## Visibility (4 surfaces) - CLI `codegraph status`: new `Backend:` line under Index Statistics. `native` rendered green; `wasm` rendered yellow with an inline `npm rebuild better-sqlite3` nudge. Also exposed in `--json` as `backend: 'native' | 'wasm'`. - MCP `codegraph_status`: new `**Backend:**` line. Native form reads `native (better-sqlite3)`; wasm form prepends a warning glyph and includes the full fix recipe. - Stderr banner on fallback (`buildWasmFallbackBanner`): replaces the bare one-line `console.warn` with a multi-line bordered banner covering macOS + Linux fix steps and optionally appending the native load error. - README troubleshooting: new "Indexing is slow / MCP database is locked / WASM fallback active" entry that walks users to the `Backend:` line and the fix. ## Per-instance backend tracking `createDatabase` previously set a module-level `activeBackend` global. MCP can open multiple project DBs in one process via the `getCodeGraph()` cache, so the global would race / overwrite. Refactor: `createDatabase` now returns `{db, backend}`, `DatabaseConnection` carries `private backend` and exposes `getBackend()`, and `CodeGraph.getBackend()` is the public surface. The CLI and MCP both call `cg.getBackend()`. ## What this does NOT fix The root cause of users landing on WASM is environment-specific (Mac without Xcode CLT, Node version mismatch, etc.) and not fixable in code without changing the optionalDependencies design. The README entry tells users what to run; `Backend: native` after rebuild is the confirmation signal. ## Tests New `__tests__/sqlite-backend.test.ts` (6 tests) pins the banner recipe content (so future edits can't strip the recovery commands), the `WASM_FALLBACK_FIX_RECIPE` constant, and per-instance `DatabaseConnection.getBackend()` / `CodeGraph.getBackend()` reporting. Suite: 503 → 509, all passing. Credit to @andreinknv whose analysis on #138 (and patches on his fork at 6d0e7a2 + 69f7001) framed the visibility approach. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Colby McHenry
|
8346440592 |
Add WASM fallbacks for tree-sitter and SQLite, fix installer
Replace native tree-sitter with web-tree-sitter + tree-sitter-wasms for universal cross-platform support. Add node-sqlite3-wasm as a fallback when better-sqlite3 native bindings aren't available. Move better-sqlite3 and sqlite-vss to optionalDependencies so installs never fail. Fix installer to use npx fallback when global npm install fails, so MCP config, hooks, and quick-start instructions all work without the bare codegraph command in PATH. Fix tests: update schema version expectation, fix db test paths and method names, extract MAX_OUTPUT_LENGTH as module constant, normalize Windows path separators in import resolver. |
||
 Martin Oehlert
|
53beb1bd2f |
feat: schema v2 migration, database performance, batch query methods
- Add provenance column on edges for tracking how edges were created - Add project_metadata table for version/provenance tracking - Make unresolved_refs file_path/language NOT NULL with defaults - Add composite indexes for unresolved_refs and edges.provenance - Update v2 migration to handle all new schema additions - Record schema version on initialize to prevent re-migration - Add dynamic prepared statement cache (getDynamicStmt) for varying SQL - Add batch methods: getNodesByIds, getNodesByKinds, getFileHashMap, getFileSyncMap - Add project metadata methods: getMetadata, setMetadata, getAllMetadata - Optimize getStats to use single aggregate query - Optimize getStaleFiles to use temporary table JOIN - Add provenance parameter to getOutgoingEdges - Add intent field to SearchOptions type |
||
|
Colby McHenryandClaude Opus 4.6
|
d80900f653 |
Port performance improvements from PR #15
- SQLite performance pragmas: synchronous=NORMAL, 64MB cache, memory temp store, 256MB mmap (safe with WAL mode) - Batch insert for unresolved refs: single transaction instead of N individual inserts per file - Symbol caching (warmCaches): pre-load all nodes into memory maps before resolution, eliminating repeated SQLite queries per ref - Async file I/O: fs.stat/readFile in indexFile() are now non-blocking - Denormalize filePath/language onto UnresolvedReference: avoids N node lookups during resolution, with schema migration v2 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Colby McHenry
|
d0ee6f7fc4 |
Enhances code extraction and project indexing
Adds support for Dart and Liquid languages with tree-sitter parsing. Improves accuracy of code symbol extraction for existing languages. Indexes project files to enhance code navigation features. Migrates build system to facilitate code contributions. Removes git hook functionality. Integrates Sentry for error tracking and reporting. Enhances project initialization and configuration loading. |
||
|
Colby McHenry
|
cc6e7a5c89 | Init |