435a7fd37ab503bd87f95161a2b7af986ad5f3e3
13
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Colby MchenryandGitHub
|
cf1b0e341a |
fix(sync): refresh the watcher's scope when codegraph.json or a .gitignore changes (#1590) (#1594)
Fixes #1590. ## What was wrong The live file watcher built its scope matcher — built-in defaults + `.gitignore` + the `codegraph.json` `exclude`/`include` rules — once in `start()` and kept it for the watcher's lifetime. The MCP server is long-lived, so a `codegraph.json` created or edited after it started was invisible to the watcher, while `codegraph sync` (a fresh process with a fresh matcher) honoured it immediately. From the user's side: the CLI removed a newly excluded file, and the daemon re-indexed it a few seconds later, which reads as "`exclude` doesn't work". As the report points out, `extensions` on the very same config file *was* read live (its loader is mtime-cached), so two fields of one file behaved differently. There was a second half to it. The watcher's scoped fast path hands the exact edited paths to sync, and that path stat'ed and re-parsed them without consulting the scope matcher at all — so the stale view of scope leaked straight into the index. ## What this does **Watcher — rebuild on a scope change, then reconcile in full.** An event for the root `codegraph.json` or `.gitignore` rebuilds the matcher, marks the next sync as a full reconcile, and schedules it. A scope change has no per-file events: newly excluded files must be *removed* from the index and newly included ones *added*, and only the scan-diff (which builds its own fresh matcher) knows which those are. Two ordering details are deliberate: - the two root files are checked *before* the matcher is consulted, so a user pattern that happens to cover them (`*.json`, `.*`) can't hide their own edits; - a nested `.gitignore` (an embedded child repo's own rules, or a subdirectory rule the git-backed scan honours) is checked *after* the matcher, so the thousands of package-local `.gitignore`s an `npm install` writes under an ignored `node_modules/` can never trigger a rebuild storm. Rebuilding runs embedded-repo discovery (one `git ls-files`), which is fine per config edit and never happens per event. Replacing the field serves both watch strategies: the recursive handler and the per-directory `shouldIgnoreDir` walk read it on every call. **Scoped sync — re-check the paths it was handed.** The orchestrator now runs scoped paths through the same scope matcher and source-extension gate the full walk applies. An out-of-scope path is treated as absent: removed if tracked, never parsed on trust. The matcher is memoized on the mtimes of the two root files it derives from (two `stat`s per sync while nothing changed), so the scoped path keeps skipping O(repo) work — paying embedded-repo discovery per sync would defeat its whole point. ## Tests - `watcher.test.ts` — a `codegraph.json` edit schedules a full sync, after which an edit inside the newly excluded tree is dropped by the live matcher (not pending, no sync) while an in-scope edit still syncs scoped; a root `.gitignore` edit behaves the same; a nested `.gitignore` forces a full sync; a `.gitignore` under `node_modules/` schedules nothing; dropping the exclude again readmits the tree. - `sync.test.ts` — end-to-end through `CodeGraph`: a scoped sync of a path that `codegraph.json` now excludes removes it (`filesRemoved: 1`, nothing parsed — the symbol added to the file never appears), stays out on a repeat, and is re-added through the same scoped path once the exclude is dropped. - All five new tests fail on `main`; the `node_modules` guard passes both ways as expected. - Full suite: 189 files, 3184 passed / 9 skipped. - CLI half of the issue's repro (init with `exclude`, edit the config + the file, `codegraph sync`): the newly excluded file is removed and its new symbol never enters the index. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LxZj6W6Y1SHXwvpT3uwJpK |
||
|
c74e8b05e0 |
perf(sync): adaptive quick-fire debounce + scoped watcher sync — save-to-graph well under a second at any scale (#1397)
Two changes to the watcher path (the always-on daemon every agent session uses), which previously paid a flat 2s debounce plus a full-tree scan-diff on every save even though the OS events name the exact files: 1. Adaptive debounce: a pending set of ≤2 files fires after a 300ms quiet window; ≥3 keeps the full configured window so agent multi-file bursts coalesce exactly as before. Re-arming preserves trailing-edge semantics; a user-set CODEGRAPH_WATCH_DEBOUNCE_MS remains the authoritative upper bound (quick window never exceeds it, floor 100ms). 2. Scoped sync: watcher-triggered syncs pass their pending paths, and the reconciler stats exactly those — per-path logic identical to the full walk (stat pre-filter, hash confirm, the #1240 removal/resurrection flow) — skipping the O(repo) scan and tracked-load. Strict fallbacks keep the full scan-diff as ground truth: directory removals (#1285 — the events can't name the children), empty pending sets (retry paths), and >500-file storms (branch checkouts, which also self-heal anything event coalescing dropped). filesChecked counts examined PATHS so a deletion-only scoped sync can't mimic the #449 lock-unavailable signature. Measured (warm in-process, the daemon path): dubbo one-file sync work 512→335ms, Swift compiler (27k files) 884→385ms — save-to-fresh-graph ≈0.6-0.7s end-to-end including the quick debounce, from ~2.5-6s perceived before. Gates: scoped-vs-full dumps byte-identical on dubbo AND the Swift compiler; watcher suite 30/30 (3 new: scoped pass-through, dir-removal fallback, quick-fire timing); sync suite 34/34 (4 new scoped-parity cases incl. delete-resurrection and the lock signature); full suite 2,696 ×2 with CODEGRAPH_KERNEL_EXPECT=1. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
8dcf92f285 |
fix(watch): schedule a sync when a directory is deleted (#1313)
A directory deletion arrives as ONE event on the directory's own path. That path has no source extension, so handleChange dropped it at the isSourceFile gate before ever scheduling a sync — and the files inside may never get events of their own (Windows's recursive watcher reports only the top-most removed entry; FSEvents can coalesce a tree deletion the same way). Every child record then sat stale in the index until an unrelated edit happened to trigger a sync (#1285). A non-source path that no longer EXISTS on disk now schedules the debounced sync; the sync's scan-diff removes whatever vanished (already correct — verified: manual `codegraph sync` cascades fine). Events for live non-source files stay fully ignored, so build churn schedules nothing. Fixes #1285 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
7c7514f43f |
fix(sync): degrade auto-sync on a persistent non-lock sync failure (#1127) (#1128)
FileWatcher.flush() bounded only two failure modes — lock contention (backoff + degrade past MAX_LOCK_RETRIES) and watch-resource exhaustion (degrade at setup). Its generic catch branch — any *other* sync error — reset the only circuit breaker (lockRetryCount = 0) and fell through to scheduleSync() at the normal debounce cadence, forever, with no backoff and no degrade(). The trigger is realistic, not synthetic: CodeGraph.sync() runs the whole extract -> resolve -> maintenance pipeline inside try/finally(release) with no catch, so a deterministic failure (a tree-sitter extractor that crashes on one file, SQLITE_FULL, an OOM in batched resolution) propagates straight into that unbounded branch — wedging a long-running daemon/MCP session into ~1,800 failing syncs + log lines/hour while the auto-update guarantee is silently dead. Mirror the lock circuit breaker for the generic branch: a separate consecutive-failure counter (syncFailureRetryCount) reset only by a clean sync, exponential backoff via the shared finally, and degrade() past MAX_SYNC_FAILURE_RETRIES with an actionable reason naming the underlying error. degrade() -> onDegraded/isDegraded() is what surfaces the dead guarantee (the staleness banner already consumes it) — a lighter flat-retry would keep it hidden, which is the core of the #876/#1127 complaint. Reset-on-success means a transient hiccup never degrades. The lock path is behaviorally unchanged: in any pure-lock scenario syncFailureRetryCount stays 0, so Math.max(lockRetryCount, syncFailureRetryCount) and the degrade threshold behave exactly as before. Renamed MAX_LOCK_RETRY_DELAY_MS -> MAX_RETRY_BACKOFF_MS (shared cap). Adds two regression tests mirroring the lock-contention ones: a persistent non-lock failure degrades past the budget; a transient one recovers without degrading. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
ab107b325a |
fix(watcher): warn (don't degrade) on Linux inotify watch exhaustion (ENOSPC) (#893)
On the Linux per-directory watch path, hitting fs.inotify.max_user_watches surfaces as ENOSPC — which the degrade logic added for #876 (EMFILE/ENFILE only) did not catch, so it fell through to the silent "skip this directory" branch: a large repo got a partial watch set with no hint why edits in unwatched directories stopped auto-syncing. ENOSPC is non-fatal — raise the limit and partial watching keeps working — so it now warns ONCE, naming the exact knob (fs.inotify.max_user_watches, with the sysctl to set it), instead of degrading. It also stops attempting further doomed watches for the session (every inotify_add_watch would fail too). Installed watches keep firing; `codegraph sync` / git sync hooks cover the remainder. Validated on macOS (forced per-directory path) and real Linux (Docker) — the new test asserts a single warning naming fs.inotify.max_user_watches, no degrade, and a live partial watch. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
cea4d086f9 |
fix(watcher): degrade cleanly on watch exhaustion and prolonged lock contention (#891)
The live file watcher could stay "alive" after it had stopped being trustworthy. EMFILE/ENFILE watch-resource exhaustion only logged (and was silently tolerated on the Linux per-directory path), and prolonged LockUnavailableError retried forever at the normal debounce cadence — both left auto-sync dead while the index silently drifted stale. Especially bad for long-running MCP/daemon sessions. Add a one-way degrade(): on watch-resource exhaustion (any watch strategy) or on lock contention past a bounded exponential-backoff budget, log once, fire a new onDegraded callback, and stop. start() now returns false consistently when the per-directory path degrades at startup — it previously returned true on Linux, so the MCP server reported the watcher "active" when it had degraded. Wire onDegraded into the MCP server so callers are actually told, and expose isDegraded()/getDegradedReason(). Builds on the approach in #877 by @thismilktea. Validated on macOS (recursive), Linux (per-directory, Docker) and Windows (recursive) — 30/30 watcher + watch-policy tests on each. Closes #876 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
c9559d9991 |
fix(watcher): bound fd/watch cost with a native fs.watch hybrid (#644, #496, #555, #628, #579) (#650)
chokidar v4 holds one OS file descriptor per watched file on macOS (libuv's kqueue backend registers an fd per vnode; fsevents is installed but v4 no longer uses it). On a large project the `serve --mcp` daemon accumulated tens of thousands of open REG descriptors and exhausted kern.maxfiles — crashing unrelated processes system-wide with ENFILE. #276 only trimmed the count by ignoring directories; the source tree still cost one fd per file. Replace chokidar with a pure-JS native fs.watch hybrid, keeping codegraph's zero-native-addon "any OS builds any bundle" invariant: - macOS / Windows: a single recursive fs.watch (one FSEvents stream / ReadDirectoryChangesW handle) -> O(1) descriptors regardless of repo size. - Linux: one inotify watch per directory (O(dirs), dynamic add for new dirs, capped via CODEGRAPH_MAX_DIR_WATCHES) instead of per-file watches. Validated empirically: macOS 0 extra fds at 6k and 12k files; Linux 31 inotify watches at 6k files (per-file would be 6k); Windows recursive catches nested and new-directory edits. Full test suite green. Tests drive the watcher through an inertForTests seam (no OS watcher) for determinism under parallel vitest, with one real-fs end-to-end test exercising the genuine native path. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
72c08c2bef |
fix(watcher): retain pending files on zero-result sync (#450)
* fix(watcher): retain pending files on zero-result sync
* refactor(watcher): detect lock-unavailable at the wrapper
Replace the heuristic `(filesChanged === 0 && durationMs === 0)` check
inside `FileWatcher.flush()` with a typed `LockUnavailableError` thrown
by `CodeGraph.watch()`'s sync wrapper. The wrapper has access to the
full `SyncResult`, including `filesChecked` — which is **only** zero
when `sync()` failed to acquire the cross-process file lock (a real
empty sync always has `filesChecked > 0` because `scanDirectory` ran).
That eliminates the heuristic's edge case where a fast no-op sync
returns `durationMs === 0` by `Date.now()` rounding and gets mistaken
for a lock failure on tiny projects.
The watcher's `catch` block now distinguishes `LockUnavailableError`
from real errors: it logs at `logDebug` (not `logWarn`) and does NOT
call `onSyncError` — so a long-running external indexer holding the
lock doesn't spam stderr every debounce cycle via the MCP daemon's
`Auto-sync error` handler. The existing post-catch path already
preserves `pendingFiles` and reschedules, so no new control flow is
needed.
A/B validated end-to-end against the built dist on macOS with a
three-scenario repro (lock held, lock released mid-flight, real sync
error):
- main: lock-held silently clears pendingFiles (BUG);
lock-released never recovers (no real sync runs).
- PR-as-is: lock-held preserves pendingFiles; lock-released
drains. Same observable behavior as wrapper-level.
- wrapper-level: same outcomes; lock-failure goes through the catch
path silently (logDebug only, no onSyncError noise);
real errors still surface via onSyncError.
Updates the regression test to throw `LockUnavailableError` (the real
contract surfaced to `FileWatcher` by `CodeGraph.watch()`), and
asserts `onSyncError` stays quiet during the lock-held cycle.
Closes #449.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Colby MchenryandGitHub
|
22bc542d34 |
test: eliminate chokidar/FSEvents race in watcher + staleness-banner tests (#434)
Mocks chokidar at the module level for `__tests__/watcher.test.ts` and
`__tests__/mcp-staleness-banner.test.ts` so the pending-file-tracking and
staleness-banner tests no longer depend on OS-level file-watcher delivery
latency. Reduces full-suite failure rate from 3/10 to 0/10.
- `__tests__/__helpers__/chokidar-mock.ts` (new) — controllable
EventEmitter; `chokidarMockModule` for `vi.mock('chokidar', ...)` plus
`triggerFileEvent(root, event, relPath)` helper. `watch()` returns an
EventEmitter that fires `ready` on the next microtask.
- `__tests__/watcher.test.ts` — refactors every event-driving test to
use `triggerFileEvent` instead of `fs.writeFileSync` for the trigger.
Pending-file tests assert state synchronously. Filtering tests still
verify FileWatcher's own filter chain.
- `__tests__/mcp-staleness-banner.test.ts` — same vi.mock + same
`triggerFileEvent` pattern; tests keep `fs.writeFileSync` for on-disk
content (`cg.sync()` needs the bytes) and add the synthesized event
on top.
The watcher's debounce timer (real `setTimeout`) is left untouched — that's
the unit under test.
Total test count unchanged (928 passing + 2 pre-existing skips). Wall-clock
runtime improved (no more 8000ms waitFor polls against real chokidar).
One disclosed tradeoff: the previous node_modules filtering test
incidentally exercised chokidar's `ignored` callback at the OS level;
with chokidar mocked, that property of chokidar itself isn't covered
here. Commented inline.
|
||
|
|
b48170e69f |
feat(mcp): per-file staleness banner + tunable watcher debounce (#403) (#428)
Two coupled changes addressing the issue's underlying ask — "how does the
agent know when the index lags" — without resorting to a static wait.
Per-file staleness banner
-------------------------
FileWatcher now tracks per-path `pendingFiles` (path, firstSeenMs,
lastSeenMs, indexing) — events since the last successful sync, cleared
only after a sync whose `syncStartedMs >= lastSeenMs` commits. Chokidar
initial-scan events are gated behind a `ready` flag (with `waitUntilReady()`
exposed so tests can deterministically wait through it) so a fresh startup
doesn't falsely flag every existing file as pending.
ToolHandler now wraps every code-returning response (search, context,
callers, callees, impact, trace, explore, node, files) with
`withStalenessNotice`: intersects "files referenced in the response" with
`getPendingFiles()` and emits a hybrid signal —
* banner at the top for files referenced AND pending (with edit age +
indexing/pending-sync state, telling the agent to Read those specific
files directly; the rest of the response stays fresh and codegraph
stays authoritative for it),
* compact footer for pending files elsewhere in the project not
referenced above (capped at 5).
Cost is one boolean check + N substring matches when pending; zero
allocation when idle. `codegraph_status` surfaces the same data as a
first-class `### Pending sync:` section so the agent can ask "is the index
caught up?" in one call.
Cross-project quirk: when an agent passes `projectPath` matching the
default session's project, the staleness wrapper switches from the cached
cross-project CodeGraph (no watcher) to the default one (with watcher) so
the signal still fires. Same fix applied to `handleStatus`.
CODEGRAPH_WATCH_DEBOUNCE_MS
---------------------------
MCP `serve --mcp` now reads `CODEGRAPH_WATCH_DEBOUNCE_MS` and forwards it
to `cg.watch({ debounceMs })`. Clamped to [100ms, 60s]; out-of-range or
non-numeric values fall back to the FileWatcher default (2000ms). Active
value is logged to stderr on watcher startup so it's discoverable. The
docs in `server-instructions.ts`, `installer/instructions-template.ts`,
and `.cursor/rules/codegraph.mdc` no longer claim "~500ms"; they now
describe the banner mechanism instead — since per-file staleness replaces
the "wait N ms" guidance entirely, the docs become accurate at any
debounce value.
Validation
----------
* 847 unit/integration tests pass (added 15 new ones — pending-file
tracking, banner/footer routing, status section, env-var parsing).
* Direct MCP probe through a real `codegraph serve --mcp` process: edit a
file, query within the debounce window, banner fires naming the
edited file with edit-age.
* Real Claude TUI session via `scripts/agent-eval/itrun.sh` with
`CODEGRAPH_WATCH_DEBOUNCE_MS=10000`: agent edits `math.ts`, calls
`codegraph_explore`, reads the banner, **and discloses it unprompted in
its final reply**: "note: symbol index is mid-sync for the new `divide`,
but the source it returned is verbatim from disk."
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
b09b23cf54 |
fix(watcher): exclude ignored dirs before watching to prevent inotify exhaustion (#276)
The file watcher registered a recursive watch over the entire project (node_modules, build output, caches included) and filtered only in the callback — exhausting the Linux inotify budget on large repos (#276). It now uses chokidar and excludes the same directories the indexer ignores (built-in default-ignore set + the project .gitignore) BEFORE registering a watch, so the watch count on a 900-dir node_modules drops from ~1200 to ~14 even with no .gitignore. Stacks with the shared daemon (#411): one watcher across agents, now small. Also hardens the #411 daemon lockfile against a concurrent-startup race the new watcher timing made reproducible — the lock is now created atomically with its content (temp-write + hard-link), so racing daemons can never both win. Validated on macOS, Linux (Docker), and Windows (chokidar + fs.linkSync on NTFS). Co-Authored-By: Colby McHenry <me@colbymchenry.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
f6772dac7c |
feat: zero-config indexing driven by .gitignore (#283) (#285)
Remove .codegraph/config.json and the entire config surface. CodeGraph now indexes every file whose extension maps to a supported language and respects .gitignore everywhere — git repos via git itself, non-git projects via the `ignore` library (root + nested .gitignore files, the same way git does). - Remove CodeGraphConfig/DEFAULT_CONFIG, src/config.ts, and the public config API (the `config` option on init, getConfig/updateConfig/getConfigPath). - Derive the source-file allowlist from EXTENSION_MAP (isSourceFile); maxFileSize is now a constant. Drop the .codegraphignore marker. - Behavior change: committed, non-gitignored dirs (vendor/, a committed dist/) are now indexed — .gitignore is the single source of truth. Earlier inert fields (languages, frameworks, extractDocstrings, trackCallSites, customPatterns) and their dead helpers are removed as part of this. Resolves #283. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Colby McHenry
|
3da5c96a0b |
feat: Add file watcher with debounced auto-sync and comprehensive test coverage
Addresses the need for automatic graph synchronization on file changes. Implements FileWatcher using native OS file events (FSEvents/inotify/ReadDirectoryChangesW) with 2-second debouncing to prevent thrashing on rapid saves. Filters changes against include/exclude patterns and ignores .codegraph directory modifications. Integrates with CodeGraph API (watch/unwatch/isWatching methods) and MCP server for automatic activation. Updates documentation to reflect shift from semantic to full-text search and removal of manual hook installation requirements. |