chokidar v4 holds one OS file descriptor per watched file on macOS (libuv's
kqueue backend registers an fd per vnode; fsevents is installed but v4 no
longer uses it). On a large project the `serve --mcp` daemon accumulated tens
of thousands of open REG descriptors and exhausted kern.maxfiles — crashing
unrelated processes system-wide with ENFILE. #276 only trimmed the count by
ignoring directories; the source tree still cost one fd per file.
Replace chokidar with a pure-JS native fs.watch hybrid, keeping codegraph's
zero-native-addon "any OS builds any bundle" invariant:
- macOS / Windows: a single recursive fs.watch (one FSEvents stream /
ReadDirectoryChangesW handle) -> O(1) descriptors regardless of repo size.
- Linux: one inotify watch per directory (O(dirs), dynamic add for new
dirs, capped via CODEGRAPH_MAX_DIR_WATCHES) instead of per-file watches.
Validated empirically: macOS 0 extra fds at 6k and 12k files; Linux 31 inotify
watches at 6k files (per-file would be 6k); Windows recursive catches nested
and new-directory edits. Full test suite green.
Tests drive the watcher through an inertForTests seam (no OS watcher) for
determinism under parallel vitest, with one real-fs end-to-end test exercising
the genuine native path.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(watcher): retain pending files on zero-result sync
* refactor(watcher): detect lock-unavailable at the wrapper
Replace the heuristic `(filesChanged === 0 && durationMs === 0)` check
inside `FileWatcher.flush()` with a typed `LockUnavailableError` thrown
by `CodeGraph.watch()`'s sync wrapper. The wrapper has access to the
full `SyncResult`, including `filesChecked` — which is **only** zero
when `sync()` failed to acquire the cross-process file lock (a real
empty sync always has `filesChecked > 0` because `scanDirectory` ran).
That eliminates the heuristic's edge case where a fast no-op sync
returns `durationMs === 0` by `Date.now()` rounding and gets mistaken
for a lock failure on tiny projects.
The watcher's `catch` block now distinguishes `LockUnavailableError`
from real errors: it logs at `logDebug` (not `logWarn`) and does NOT
call `onSyncError` — so a long-running external indexer holding the
lock doesn't spam stderr every debounce cycle via the MCP daemon's
`Auto-sync error` handler. The existing post-catch path already
preserves `pendingFiles` and reschedules, so no new control flow is
needed.
A/B validated end-to-end against the built dist on macOS with a
three-scenario repro (lock held, lock released mid-flight, real sync
error):
- main: lock-held silently clears pendingFiles (BUG);
lock-released never recovers (no real sync runs).
- PR-as-is: lock-held preserves pendingFiles; lock-released
drains. Same observable behavior as wrapper-level.
- wrapper-level: same outcomes; lock-failure goes through the catch
path silently (logDebug only, no onSyncError noise);
real errors still surface via onSyncError.
Updates the regression test to throw `LockUnavailableError` (the real
contract surfaced to `FileWatcher` by `CodeGraph.watch()`), and
asserts `onSyncError` stays quiet during the lock-held cycle.
Closes#449.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Colby McHenry <me@colbymchenry.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mocks chokidar at the module level for `__tests__/watcher.test.ts` and
`__tests__/mcp-staleness-banner.test.ts` so the pending-file-tracking and
staleness-banner tests no longer depend on OS-level file-watcher delivery
latency. Reduces full-suite failure rate from 3/10 to 0/10.
- `__tests__/__helpers__/chokidar-mock.ts` (new) — controllable
EventEmitter; `chokidarMockModule` for `vi.mock('chokidar', ...)` plus
`triggerFileEvent(root, event, relPath)` helper. `watch()` returns an
EventEmitter that fires `ready` on the next microtask.
- `__tests__/watcher.test.ts` — refactors every event-driving test to
use `triggerFileEvent` instead of `fs.writeFileSync` for the trigger.
Pending-file tests assert state synchronously. Filtering tests still
verify FileWatcher's own filter chain.
- `__tests__/mcp-staleness-banner.test.ts` — same vi.mock + same
`triggerFileEvent` pattern; tests keep `fs.writeFileSync` for on-disk
content (`cg.sync()` needs the bytes) and add the synthesized event
on top.
The watcher's debounce timer (real `setTimeout`) is left untouched — that's
the unit under test.
Total test count unchanged (928 passing + 2 pre-existing skips). Wall-clock
runtime improved (no more 8000ms waitFor polls against real chokidar).
One disclosed tradeoff: the previous node_modules filtering test
incidentally exercised chokidar's `ignored` callback at the OS level;
with chokidar mocked, that property of chokidar itself isn't covered
here. Commented inline.
Two coupled changes addressing the issue's underlying ask — "how does the
agent know when the index lags" — without resorting to a static wait.
Per-file staleness banner
-------------------------
FileWatcher now tracks per-path `pendingFiles` (path, firstSeenMs,
lastSeenMs, indexing) — events since the last successful sync, cleared
only after a sync whose `syncStartedMs >= lastSeenMs` commits. Chokidar
initial-scan events are gated behind a `ready` flag (with `waitUntilReady()`
exposed so tests can deterministically wait through it) so a fresh startup
doesn't falsely flag every existing file as pending.
ToolHandler now wraps every code-returning response (search, context,
callers, callees, impact, trace, explore, node, files) with
`withStalenessNotice`: intersects "files referenced in the response" with
`getPendingFiles()` and emits a hybrid signal —
* banner at the top for files referenced AND pending (with edit age +
indexing/pending-sync state, telling the agent to Read those specific
files directly; the rest of the response stays fresh and codegraph
stays authoritative for it),
* compact footer for pending files elsewhere in the project not
referenced above (capped at 5).
Cost is one boolean check + N substring matches when pending; zero
allocation when idle. `codegraph_status` surfaces the same data as a
first-class `### Pending sync:` section so the agent can ask "is the index
caught up?" in one call.
Cross-project quirk: when an agent passes `projectPath` matching the
default session's project, the staleness wrapper switches from the cached
cross-project CodeGraph (no watcher) to the default one (with watcher) so
the signal still fires. Same fix applied to `handleStatus`.
CODEGRAPH_WATCH_DEBOUNCE_MS
---------------------------
MCP `serve --mcp` now reads `CODEGRAPH_WATCH_DEBOUNCE_MS` and forwards it
to `cg.watch({ debounceMs })`. Clamped to [100ms, 60s]; out-of-range or
non-numeric values fall back to the FileWatcher default (2000ms). Active
value is logged to stderr on watcher startup so it's discoverable. The
docs in `server-instructions.ts`, `installer/instructions-template.ts`,
and `.cursor/rules/codegraph.mdc` no longer claim "~500ms"; they now
describe the banner mechanism instead — since per-file staleness replaces
the "wait N ms" guidance entirely, the docs become accurate at any
debounce value.
Validation
----------
* 847 unit/integration tests pass (added 15 new ones — pending-file
tracking, banner/footer routing, status section, env-var parsing).
* Direct MCP probe through a real `codegraph serve --mcp` process: edit a
file, query within the debounce window, banner fires naming the
edited file with edit-age.
* Real Claude TUI session via `scripts/agent-eval/itrun.sh` with
`CODEGRAPH_WATCH_DEBOUNCE_MS=10000`: agent edits `math.ts`, calls
`codegraph_explore`, reads the banner, **and discloses it unprompted in
its final reply**: "note: symbol index is mid-sync for the new `divide`,
but the source it returned is verbatim from disk."
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The file watcher registered a recursive watch over the entire project (node_modules, build output, caches included) and filtered only in the callback — exhausting the Linux inotify budget on large repos (#276). It now uses chokidar and excludes the same directories the indexer ignores (built-in default-ignore set + the project .gitignore) BEFORE registering a watch, so the watch count on a 900-dir node_modules drops from ~1200 to ~14 even with no .gitignore. Stacks with the shared daemon (#411): one watcher across agents, now small.
Also hardens the #411 daemon lockfile against a concurrent-startup race the new watcher timing made reproducible — the lock is now created atomically with its content (temp-write + hard-link), so racing daemons can never both win. Validated on macOS, Linux (Docker), and Windows (chokidar + fs.linkSync on NTFS).
Co-Authored-By: Colby McHenry <me@colbymchenry.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Remove .codegraph/config.json and the entire config surface. CodeGraph now
indexes every file whose extension maps to a supported language and respects
.gitignore everywhere — git repos via git itself, non-git projects via the
`ignore` library (root + nested .gitignore files, the same way git does).
- Remove CodeGraphConfig/DEFAULT_CONFIG, src/config.ts, and the public config
API (the `config` option on init, getConfig/updateConfig/getConfigPath).
- Derive the source-file allowlist from EXTENSION_MAP (isSourceFile); maxFileSize
is now a constant. Drop the .codegraphignore marker.
- Behavior change: committed, non-gitignored dirs (vendor/, a committed dist/)
are now indexed — .gitignore is the single source of truth.
Earlier inert fields (languages, frameworks, extractDocstrings, trackCallSites,
customPatterns) and their dead helpers are removed as part of this.
Resolves#283.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Addresses the need for automatic graph synchronization on file changes. Implements FileWatcher using native OS file events (FSEvents/inotify/ReadDirectoryChangesW) with 2-second debouncing to prevent thrashing on rapid saves. Filters changes against include/exclude patterns and ignores .codegraph directory modifications. Integrates with CodeGraph API (watch/unwatch/isWatching methods) and MCP server for automatic activation. Updates documentation to reflect shift from semantic to full-text search and removal of manual hook installation requirements.
