fix(watcher): bound fd/watch cost with a native fs.watch hybrid (#644, #496, #555, #628, #579) (#650)

chokidar v4 holds one OS file descriptor per watched file on macOS (libuv's
kqueue backend registers an fd per vnode; fsevents is installed but v4 no
longer uses it). On a large project the `serve --mcp` daemon accumulated tens
of thousands of open REG descriptors and exhausted kern.maxfiles — crashing
unrelated processes system-wide with ENFILE. #276 only trimmed the count by
ignoring directories; the source tree still cost one fd per file.

Replace chokidar with a pure-JS native fs.watch hybrid, keeping codegraph's
zero-native-addon "any OS builds any bundle" invariant:

  - macOS / Windows: a single recursive fs.watch (one FSEvents stream /
    ReadDirectoryChangesW handle) -> O(1) descriptors regardless of repo size.
  - Linux: one inotify watch per directory (O(dirs), dynamic add for new
    dirs, capped via CODEGRAPH_MAX_DIR_WATCHES) instead of per-file watches.

Validated empirically: macOS 0 extra fds at 6k and 12k files; Linux 31 inotify
watches at 6k files (per-file would be 6k); Windows recursive catches nested
and new-directory edits. Full test suite green.

Tests drive the watcher through an inertForTests seam (no OS watcher) for
determinism under parallel vitest, with one real-fs end-to-end test exercising
the genuine native path.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-06-02 14:21:29 -05:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 434be1da58
commit c9559d9991
9 changed files with 424 additions and 365 deletions
+325 -107
View File
@@ -4,25 +4,79 @@
* Watches the project directory for file changes and triggers debounced sync
* operations to keep the code graph up-to-date.
*
* Uses chokidar, whose `ignored` callback filters directories BEFORE they are
* watched — so we never register inotify watches on excluded trees like
* node_modules/, dist/, .git/ (fixes #276: recursive fs.watch exhausted the
* kernel watch budget on large repos). The ignore decision reuses the indexer's
* `buildDefaultIgnore` (built-in default-ignore dirs + the project's .gitignore)
* so the watcher watches exactly the set the indexer indexes — in particular,
* node_modules/build/cache dirs are excluded even when the repo has no
* .gitignore (#407), which a .gitignore-only filter would miss.
* Uses Node's built-in `fs.watch` directly (no third-party watcher, no native
* addon) with a per-platform strategy chosen to keep the open-descriptor /
* kernel-watch cost BOUNDED rather than growing with the number of files:
*
* - macOS / Windows: a SINGLE recursive `fs.watch(root, {recursive:true})`.
* libuv maps this to one FSEvents stream (macOS) / one
* ReadDirectoryChangesW handle (Windows), so it costs O(1) descriptors no
* matter how large the tree. This is the fix for the macOS file-table
* exhaustion (#644 / #496 / #555 / #628): the previous watcher held one
* open fd PER WATCHED FILE on macOS (tens of thousands of REG fds), which
* exhausted `kern.maxfiles` and crashed unrelated processes system-wide.
*
* - Linux: recursive `fs.watch` is unsupported, so we watch each (non-ignored)
* DIRECTORY with one inotify watch — O(directories), NOT O(files). New
* directories are picked up dynamically and an overall watch cap bounds
* inotify usage on pathological monorepos (#579). A single inotify watch on
* a directory already reports create/modify/delete for its children, so
* per-file watches are never needed.
*
* Excluded trees (node_modules/, dist/, .git/, …) are filtered via the
* indexer's `buildDefaultIgnore` (built-in default-ignore dirs + the project's
* .gitignore) — on Linux they're never descended into (so they cost no watch),
* and on macOS/Windows the single recursive stream still covers them but their
* events are dropped before any sync is scheduled. Either way the watcher's
* scope matches the indexer's (#276 / #407).
*/
import * as fs from 'fs';
import * as path from 'path';
import type { Stats } from 'fs';
import chokidar, { FSWatcher } from 'chokidar';
import type { Ignore } from 'ignore';
import { isSourceFile, buildDefaultIgnore } from '../extraction';
import { logDebug, logWarn } from '../errors';
import { normalizePath } from '../utils';
import { watchDisabledReason } from './watch-policy';
/**
* Native recursive `fs.watch` is only reliable on macOS and Windows; on Linux
* (and AIX) it throws `ERR_FEATURE_UNAVAILABLE_ON_PLATFORM`. We branch on this
* to pick the recursive vs per-directory strategy.
*/
function supportsRecursiveWatch(): boolean {
return process.platform === 'darwin' || process.platform === 'win32';
}
/**
* Upper bound on simultaneously-watched directories on the Linux per-directory
* path. Each is one inotify watch; the kernel's `fs.inotify.max_user_watches`
* is the hard limit (commonly 8k128k). We stop adding watches past this and
* log once — partial live-watch (with `codegraph sync` as the backstop) is far
* better than exhausting the user's inotify budget and breaking watching
* system-wide (#579). Tunable via CODEGRAPH_MAX_DIR_WATCHES.
*/
const DEFAULT_MAX_DIR_WATCHES = 50_000;
function maxDirWatches(): number {
const raw = process.env.CODEGRAPH_MAX_DIR_WATCHES;
if (raw && /^\d+$/.test(raw)) {
const n = Number(raw);
if (n > 0) return n;
}
return DEFAULT_MAX_DIR_WATCHES;
}
/**
* Test seam (see {@link __emitWatchEventForTests}). Maps a watcher's project
* root to its live instance so tests can synthesize a change event
* deterministically — real fs.watch delivery latency races under parallel
* vitest (the reason the previous chokidar mock existed). Only populated under
* a test runner, so production carries no bookkeeping or retained references.
*/
const liveWatchersForTests = new Map<string, FileWatcher>();
const IS_TEST_RUNTIME = !!(process.env.VITEST || process.env.NODE_ENV === 'test');
/**
* Options for the file watcher
*/
@@ -43,6 +97,16 @@ export interface WatchOptions {
* Callback when a sync errors (for logging/diagnostics).
*/
onSyncError?: (error: Error) => void;
/**
* Test-only. When true, `start()` installs NO OS-level fs.watch — the
* watcher is "inert" and only the {@link __emitWatchEventForTests} /
* {@link FileWatcher.ingestEventForTests} seam drives its pipeline. This
* restores the deterministic, OS-free behavior the unit tests need (real
* FSEvents/inotify delivery races under parallel vitest). Production never
* sets it.
*/
inertForTests?: boolean;
}
/**
@@ -84,8 +148,9 @@ export interface PendingFile {
* debounced sync operations via a provided callback.
*
* Design goals:
* - Minimal resource usage (chokidar filters excluded directories before
* registering an inotify watch — see module docs / #276)
* - Bounded resource usage: O(1) descriptors on macOS/Windows (one recursive
* watch), O(directories) inotify watches on Linux — never O(files), which
* was the system-crashing fd leak on macOS (#644/#496/#555/#628).
* - Debounced to avoid thrashing on rapid saves
* - Filters to supported source files by extension
* - Ignores .codegraph/ and .git/ regardless of .gitignore
@@ -93,11 +158,18 @@ export interface PendingFile {
* without blocking on a sync (issue #403)
*/
export class FileWatcher {
private watcher: FSWatcher | null = null;
/** macOS/Windows: the single recursive watcher. Null on Linux. */
private recursiveWatcher: fs.FSWatcher | null = null;
/** Linux: one watcher per watched directory (keyed by absolute path). */
private dirWatchers = new Map<string, fs.FSWatcher>();
/** Set once the per-directory watch cap is hit, so we log only once. */
private dirCapWarned = false;
/** Test-only inert mode: started, but with no OS watcher installed. */
private inert = false;
private debounceTimer: ReturnType<typeof setTimeout> | null = null;
/**
* Files seen by the watcher since the last successful sync — populated on
* every chokidar event, cleared at the start of a sync, and re-populated by
* every change event, cleared at the start of a sync, and re-populated by
* events that arrive mid-sync (or restored on sync failure). Keyed by the
* same project-relative POSIX path the rest of the codebase uses, so a
* caller can intersect tool-response file paths against this map cheaply.
@@ -113,17 +185,18 @@ export class FileWatcher {
private syncing = false;
private stopped = false;
/**
* False until chokidar fires its `ready` event. Gates `pendingFiles`
* insertion so the initial crawl's `add` events (one per pre-existing
* source file) don't pollute the per-file staleness signal. The events
* still flow into `scheduleSync()` to preserve the previous "initial
* scan triggers a reconciling sync" behavior.
* True once the initial watch set is established. Unlike the previous
* chokidar implementation there is no asynchronous initial "crawl" emitting
* an `add` per existing file — `fs.watch` only reports changes from the
* moment it's installed — so this flips to true synchronously at the end of
* `start()`. The startup reconcile against on-disk state is handled
* separately by the engine's catch-up sync, not by the watcher.
*/
private chokidarReady = false;
private ready = false;
/**
* Callbacks that resolve when chokidar fires `ready`. Used by tests (and
* any production caller that cares about a clean baseline) to deterministically
* gate on the end of the initial scan instead of guessing at a sleep duration.
* Callbacks that resolve when the watch set is established. Used by tests
* (and any production caller that cares about a clean baseline) to
* deterministically gate on watcher readiness.
*/
private readyWaiters: Array<() => void> = [];
// The shared ignore matcher (built-in defaults + project .gitignore), built
@@ -136,6 +209,7 @@ export class FileWatcher {
private readonly syncFn: () => Promise<{ filesChanged: number; durationMs: number }>;
private readonly onSyncComplete?: WatchOptions['onSyncComplete'];
private readonly onSyncError?: WatchOptions['onSyncError'];
private readonly inertForTests: boolean;
constructor(
projectRoot: string,
@@ -147,6 +221,7 @@ export class FileWatcher {
this.debounceMs = options.debounceMs ?? 2000;
this.onSyncComplete = options.onSyncComplete;
this.onSyncError = options.onSyncError;
this.inertForTests = options.inertForTests ?? false;
}
/**
@@ -154,7 +229,7 @@ export class FileWatcher {
* Returns true if watching started successfully, false otherwise.
*/
start(): boolean {
if (this.watcher) return true; // Already watching
if (this.recursiveWatcher || this.dirWatchers.size > 0 || this.inert) return true; // Already watching
this.stopped = false;
// Some environments make filesystem watching unusable — most notably
@@ -168,78 +243,186 @@ export class FileWatcher {
}
// Reuse the indexer's ignore set so the watcher and indexer agree on scope.
// chokidar only registers an inotify watch on directories that pass this
// filter — that's the #276 fix.
this.ignoreMatcher = buildDefaultIgnore(this.projectRoot);
try {
this.watcher = chokidar.watch(this.projectRoot, {
// chokidar calls this for every path it encounters and only watches
// those that pass — so excluded trees (node_modules/, dist/, .git/, …)
// never get an inotify watch in the first place.
ignored: (testPath: string, stats?: Stats) => this.shouldIgnore(testPath, stats),
if (this.inertForTests) {
// Test-only: install no OS watcher; the seam drives events instead.
this.inert = true;
} else if (supportsRecursiveWatch()) {
this.startRecursive();
} else {
this.startPerDirectory();
}
// No async crawl to wait on: as soon as the watch set is installed we
// have a clean baseline (pendingFiles is only populated by post-start
// events). Clear defensively and flip ready.
this.pendingFiles.clear();
this.ready = true;
for (const cb of this.readyWaiters) cb();
this.readyWaiters.length = 0;
if (IS_TEST_RUNTIME) liveWatchersForTests.set(this.projectRoot, this);
logDebug('File watcher started', {
projectRoot: this.projectRoot,
debounceMs: this.debounceMs,
mode: this.inertForTests ? 'inert' : supportsRecursiveWatch() ? 'recursive' : 'per-directory',
watchedDirs: this.dirWatchers.size || undefined,
});
// Chokidar emits `add` for every pre-existing source file during its
// initial scan. Those events should still trigger the post-startup
// reconciling sync (preserving prior behavior), but they must NOT land
// in pendingFiles — otherwise every file in the project shows up as
// "edited but not indexed" on startup, which is the opposite of the
// signal #403 is supposed to provide. Flip the flag on chokidar's
// `ready` event; from then on, real edits populate pendingFiles.
//
// We also clear `pendingFiles` here as defense-in-depth: chokidar can
// emit late initial-scan `add` events via setImmediate AFTER the
// `ready` callback runs (observed under test-parallelism load).
// Clearing once at ready guarantees a clean baseline; real subsequent
// edits repopulate the set normally.
this.watcher.on('ready', () => {
this.chokidarReady = true;
this.pendingFiles.clear();
for (const cb of this.readyWaiters) cb();
this.readyWaiters.length = 0;
});
// chokidar emits 'all' for every event type; we only sync source files.
this.watcher.on('all', (_event: string, filePath: string) => {
if (this.stopped) return;
const normalized = normalizePath(path.relative(this.projectRoot, filePath));
// Defense in depth: `ignored` should already keep these out, but events
// can still arrive during setup or via symlink traversal.
if (this.isAlwaysIgnored(normalized)) return;
if (!isSourceFile(normalized)) return;
logDebug('File change detected', { file: normalized });
// Only track events from after chokidar's initial scan as pending
// edits — pre-existing files on disk are already represented by
// (or about to be reconciled by) the index, not a user edit.
if (this.chokidarReady) {
const now = Date.now();
const existing = this.pendingFiles.get(normalized);
this.pendingFiles.set(normalized, {
firstSeenMs: existing?.firstSeenMs ?? now,
lastSeenMs: now,
});
}
this.scheduleSync();
});
// Handle watcher errors gracefully — don't crash, the user can restart.
this.watcher.on('error', (err: unknown) => {
logWarn('File watcher error', { error: String(err) });
});
logDebug('File watcher started', { projectRoot: this.projectRoot, debounceMs: this.debounceMs });
return true;
} catch (err) {
// Watcher setup failed (e.g., permission denied, missing directory).
logWarn('Could not start file watcher', { error: String(err) });
this.stop();
return false;
}
}
/**
* macOS/Windows: one recursive watcher for the whole tree. O(1) descriptors.
* `filename` arrives relative to the project root (with subdirectories), so
* it maps straight to a project-relative path.
*/
private startRecursive(): void {
this.recursiveWatcher = fs.watch(
this.projectRoot,
{ recursive: true, persistent: true },
(_event, filename) => {
if (this.stopped || filename == null) return;
this.handleChange(normalizePath(String(filename)));
}
);
this.recursiveWatcher.on('error', (err: unknown) => {
logWarn('File watcher error', { error: String(err) });
});
}
/**
* Linux: walk the (non-ignored) tree and watch each directory. One inotify
* watch per directory reports create/modify/delete for that directory's
* direct children, so we never watch individual files.
*/
private startPerDirectory(): void {
this.watchTree(this.projectRoot, /* markExisting */ false);
}
/**
* Add an inotify watch for `dir` and recurse into its non-ignored
* subdirectories. When `markExisting` is true (a directory that appeared
* AFTER startup), the source files already inside it are recorded as pending
* — this closes the `mkdir + write` race where files created before the new
* directory's watch is installed would otherwise be missed until the next
* full sync. The initial startup walk passes false (the engine's catch-up
* sync owns the baseline).
*/
private watchTree(dir: string, markExisting: boolean): void {
if (this.dirWatchers.has(dir)) return;
if (this.dirWatchers.size >= maxDirWatches()) {
if (!this.dirCapWarned) {
this.dirCapWarned = true;
logWarn('File watcher hit directory-watch cap; remaining subtrees rely on manual/periodic sync', {
cap: maxDirWatches(),
});
}
return;
}
let w: fs.FSWatcher;
try {
w = fs.watch(dir, { persistent: true }, (_event, filename) =>
this.handleDirEvent(dir, filename)
);
} catch {
// ENOENT / EACCES / too-many-open-files — skip this directory quietly.
return;
}
w.on('error', () => this.unwatchDir(dir));
this.dirWatchers.set(dir, w);
let entries: fs.Dirent[];
try {
entries = fs.readdirSync(dir, { withFileTypes: true });
} catch {
return;
}
for (const entry of entries) {
const child = path.join(dir, entry.name);
if (entry.isDirectory()) {
if (this.shouldIgnoreDir(child)) continue;
this.watchTree(child, markExisting);
} else if (markExisting && entry.isFile()) {
this.handleChange(normalizePath(path.relative(this.projectRoot, child)));
}
}
}
/**
* Linux per-directory event handler. `filename` is relative to `dir`. A new
* sub-directory is picked up by extending the watch tree; everything else is
* routed through the shared change handler.
*/
private handleDirEvent(dir: string, filename: string | Buffer | null): void {
if (this.stopped || filename == null) return;
const full = path.join(dir, String(filename));
// A newly-created directory needs its own watch (recursive isn't available
// on Linux). statSync is cheap and these events are rare relative to file
// edits. If the path vanished (rapid create/delete) the stat throws and we
// fall through to the change handler, which no-ops on a non-source path.
try {
if (fs.statSync(full).isDirectory()) {
if (!this.shouldIgnoreDir(full)) this.watchTree(full, /* markExisting */ true);
return;
}
} catch {
// deleted/inaccessible — treat as a normal change below
}
this.handleChange(normalizePath(path.relative(this.projectRoot, full)));
}
/**
* Shared change handler for both watch strategies. `rel` is a
* project-relative POSIX path. Applies the ignore + source-file filters and,
* for a real source change, records it as pending (#403) and schedules a
* debounced sync.
*
* The recursive (macOS/Windows) watcher reports events for ignored trees too
* (one stream covers the whole repo), so the ignore check here is load-bearing
* — it drops node_modules/dist/.git churn before any sync is scheduled.
*/
private handleChange(rel: string): void {
if (!rel || rel === '.' || rel.startsWith('..')) return;
if (this.isAlwaysIgnored(rel)) return;
if (this.ignoreMatcher && this.ignoreMatcher.ignores(rel)) return;
if (!isSourceFile(rel)) return;
logDebug('File change detected', { file: rel });
if (this.ready) {
const now = Date.now();
const existing = this.pendingFiles.get(rel);
this.pendingFiles.set(rel, {
firstSeenMs: existing?.firstSeenMs ?? now,
lastSeenMs: now,
});
}
this.scheduleSync();
}
/** Close and forget the watch for a directory that errored/was removed. */
private unwatchDir(dir: string): void {
const w = this.dirWatchers.get(dir);
if (w) {
try {
w.close();
} catch {
/* already closed */
}
this.dirWatchers.delete(dir);
}
}
/** Our own dirs are always ignored, regardless of .gitignore. */
private isAlwaysIgnored(rel: string): boolean {
return (
@@ -249,20 +432,16 @@ export class FileWatcher {
}
/**
* chokidar `ignored` predicate — true for any path that should NOT be watched.
* Uses chokidar's provided `stats` to decide directory-vs-file so a dir-only
* rule like `build/` matches, without an extra `statSync` per path.
* True for any directory that should NOT be watched (used while building the
* Linux per-directory watch tree). Tests the directory form of the path so a
* dir-only ignore rule like `build/` matches.
*/
private shouldIgnore(testPath: string, stats?: Stats): boolean {
const rel = normalizePath(path.relative(this.projectRoot, testPath));
private shouldIgnoreDir(dirPath: string): boolean {
const rel = normalizePath(path.relative(this.projectRoot, dirPath));
if (!rel || rel === '.' || rel.startsWith('..')) return false; // root / outside
if (this.isAlwaysIgnored(rel)) return true;
if (!this.ignoreMatcher) return false;
if (stats) {
return this.ignoreMatcher.ignores(stats.isDirectory() ? rel + '/' : rel);
}
// Stats unknown: test both forms so a directory match isn't missed.
return this.ignoreMatcher.ignores(rel) || this.ignoreMatcher.ignores(rel + '/');
return this.ignoreMatcher.ignores(rel + '/');
}
/**
@@ -276,37 +455,61 @@ export class FileWatcher {
this.debounceTimer = null;
}
if (this.watcher) {
this.watcher.close();
this.watcher = null;
if (this.recursiveWatcher) {
try {
this.recursiveWatcher.close();
} catch {
/* already closed */
}
this.recursiveWatcher = null;
}
for (const w of this.dirWatchers.values()) {
try {
w.close();
} catch {
/* already closed */
}
}
this.dirWatchers.clear();
this.dirCapWarned = false;
this.inert = false;
this.pendingFiles.clear();
this.chokidarReady = false;
this.ready = false;
this.ignoreMatcher = null;
if (IS_TEST_RUNTIME) liveWatchersForTests.delete(this.projectRoot);
logDebug('File watcher stopped');
}
/**
* @internal Test-only: feed a synthetic project-relative change through the
* same filter → pendingFiles → debounced-sync path a real fs.watch event
* takes. Lets the watcher / staleness-banner suites stay deterministic
* instead of racing on OS watch-delivery latency. See
* {@link __emitWatchEventForTests}.
*/
ingestEventForTests(relPath: string): void {
this.handleChange(normalizePath(relPath));
}
/**
* Whether the watcher is currently active.
*/
isActive(): boolean {
return this.watcher !== null && !this.stopped;
return (this.recursiveWatcher !== null || this.dirWatchers.size > 0 || this.inert) && !this.stopped;
}
/**
* Resolves once chokidar has fired its `ready` event (or immediately if
* it has already done so). Useful for tests that need a deterministic
* boundary before asserting on `pendingFiles` — guessing a sleep duration
* is flaky under load because chokidar can take longer than expected to
* finish its initial crawl on slow filesystems / parallel test runs.
* Resolves once the watch set has been installed (or immediately if it
* already has). Useful for tests that need a deterministic boundary before
* asserting on `pendingFiles`.
*
* Production callers don't need this: `pendingFiles` is read continuously,
* the staleness banner is always correct (empty or populated), and the
* initial-scan window is a small one-time startup cost.
* the staleness banner is always correct (empty or populated), and there is
* no asynchronous initial-scan window with `fs.watch`.
*/
waitUntilReady(timeoutMs = 10000): Promise<void> {
if (this.chokidarReady) return Promise.resolve();
if (this.ready) return Promise.resolve();
return new Promise((resolve, reject) => {
const t = setTimeout(() => {
const idx = this.readyWaiters.indexOf(handler);
@@ -419,3 +622,18 @@ export class FileWatcher {
return result;
}
}
/**
* Test-only: synthesize a source-file change for the live watcher running at
* `projectRoot`, exercising the real filter → pendingFiles → debounced-sync
* logic without depending on fs.watch delivery timing (which races under
* parallel vitest). `relPath` is project-relative POSIX (e.g. "src/foo.ts").
* Returns false if no live watcher is registered for that root (e.g. outside a
* test runtime, where the registry is intentionally not populated).
*/
export function __emitWatchEventForTests(projectRoot: string, relPath: string): boolean {
const w = liveWatchersForTests.get(projectRoot);
if (!w) return false;
w.ingestEventForTests(relPath);
return true;
}