fix(watcher): bound fd/watch cost with a native fs.watch hybrid (#644, #496, #555, #628, #579) (#650)
chokidar v4 holds one OS file descriptor per watched file on macOS (libuv's kqueue backend registers an fd per vnode; fsevents is installed but v4 no longer uses it). On a large project the `serve --mcp` daemon accumulated tens of thousands of open REG descriptors and exhausted kern.maxfiles — crashing unrelated processes system-wide with ENFILE. #276 only trimmed the count by ignoring directories; the source tree still cost one fd per file. Replace chokidar with a pure-JS native fs.watch hybrid, keeping codegraph's zero-native-addon "any OS builds any bundle" invariant: - macOS / Windows: a single recursive fs.watch (one FSEvents stream / ReadDirectoryChangesW handle) -> O(1) descriptors regardless of repo size. - Linux: one inotify watch per directory (O(dirs), dynamic add for new dirs, capped via CODEGRAPH_MAX_DIR_WATCHES) instead of per-file watches. Validated empirically: macOS 0 extra fds at 6k and 12k files; Linux 31 inotify watches at 6k files (per-file would be 6k); Windows recursive catches nested and new-directory edits. Full test suite green. Tests drive the watcher through an inertForTests seam (no OS watcher) for determinism under parallel vitest, with one real-fs end-to-end test exercising the genuine native path. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
434be1da58
commit
c9559d9991
@@ -9,6 +9,9 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Fixes
|
||||
|
||||
- The background file watcher no longer exhausts your machine's file-descriptor budget. On macOS it previously kept **one open file handle per watched file**, so on a large project the running MCP server could pile up tens of thousands of handles and blow past the system-wide limit — at which point *unrelated* apps (your shell, editor, Docker, browser) started failing with "too many open files" until the codegraph process was killed. The watcher now uses a single recursive watch on macOS and Windows, and bounded per-directory watches on Linux, so its cost stays flat no matter how large the project is. (#644, #496, #555, #628, #579)
|
||||
|
||||
## [0.9.9] - 2026-06-02
|
||||
|
||||
|
||||
@@ -1,121 +0,0 @@
|
||||
/**
|
||||
* Deterministic chokidar mock for FileWatcher tests.
|
||||
*
|
||||
* The real chokidar binding goes through FSEvents (macOS) / inotify (Linux) /
|
||||
* ReadDirectoryChangesW (Windows). Under parallel vitest execution, those
|
||||
* OS-level subsystems serve multiple test files simultaneously and event
|
||||
* delivery latency grows non-deterministically — `should expose edited paths
|
||||
* via getPendingFiles before sync fires` and the `mcp-staleness-banner` tests
|
||||
* have observably raced for that reason (consistent ~30% failure rate when
|
||||
* running the full suite, 0/N when run in isolation).
|
||||
*
|
||||
* This mock replaces chokidar with a controllable in-process EventEmitter:
|
||||
*
|
||||
* - `chokidar.watch(root, opts)` returns an instance keyed by `root`.
|
||||
* - The instance fires `ready` on the next microtask, matching the
|
||||
* real chokidar shape (tests' `waitUntilReady()` resolves promptly).
|
||||
* - Tests synthesize file events via `triggerFileEvent(root, 'add', rel)`
|
||||
* instead of `fs.writeFileSync(...)` — no OS-level watcher in the loop,
|
||||
* no waitFor polling against unpredictable delivery latency.
|
||||
* - The actual debounce timer in FileWatcher is left untouched (real
|
||||
* setTimeout). That's the unit under test; deterministic timing
|
||||
* would change what the test asserts.
|
||||
*
|
||||
* Install with `vi.mock('chokidar', () => chokidarMockModule)` at the
|
||||
* top of each test file (must be hoisted, hence the static export).
|
||||
*
|
||||
* All instances live in module scope — clear them in `afterEach` if a
|
||||
* test creates watchers and needs hard isolation, but in practice the
|
||||
* `close()` plumbing handles it.
|
||||
*/
|
||||
import { EventEmitter } from 'node:events';
|
||||
|
||||
/** One mock watcher per `chokidar.watch(root, ...)` call. */
|
||||
class MockChokidarWatcher extends EventEmitter {
|
||||
private closed = false;
|
||||
private readyFired = false;
|
||||
|
||||
constructor(public readonly root: string) {
|
||||
super();
|
||||
// Mirror chokidar: `ready` fires asynchronously after the initial scan.
|
||||
// We use queueMicrotask so it's deterministic and as fast as possible —
|
||||
// tests' `await watcher.waitUntilReady()` resolves immediately.
|
||||
queueMicrotask(() => {
|
||||
if (this.closed) return;
|
||||
this.readyFired = true;
|
||||
this.emit('ready');
|
||||
});
|
||||
}
|
||||
|
||||
/** chokidar.FSWatcher#close shape. */
|
||||
close(): Promise<void> {
|
||||
this.closed = true;
|
||||
this.removeAllListeners();
|
||||
instancesByRoot.delete(this.root);
|
||||
return Promise.resolve();
|
||||
}
|
||||
|
||||
/** Test-only helper to synthesize a file event. */
|
||||
triggerEvent(event: 'add' | 'change' | 'unlink' | 'addDir' | 'unlinkDir', absPath: string): void {
|
||||
if (this.closed) return;
|
||||
// Real chokidar emits both the typed event AND the catch-all 'all'.
|
||||
// FileWatcher only listens on 'all'.
|
||||
this.emit('all', event, absPath);
|
||||
}
|
||||
|
||||
/** True once the initial-scan `ready` event has been emitted. */
|
||||
isReady(): boolean {
|
||||
return this.readyFired;
|
||||
}
|
||||
}
|
||||
|
||||
const instancesByRoot = new Map<string, MockChokidarWatcher>();
|
||||
|
||||
/**
|
||||
* The mock module — pass this to `vi.mock('chokidar', () => chokidarMockModule)`.
|
||||
* The factory must NOT close over outer-scope state because vi.mock hoists.
|
||||
*/
|
||||
export const chokidarMockModule = {
|
||||
default: {
|
||||
watch: (root: string, _opts?: unknown) => {
|
||||
const inst = new MockChokidarWatcher(root);
|
||||
instancesByRoot.set(root, inst);
|
||||
return inst;
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Test-side helper: synthesize a chokidar event on the watcher created for
|
||||
* `root`. Use after the watcher's `waitUntilReady()` has resolved, since
|
||||
* FileWatcher only adds events to its pending set when `chokidarReady` is
|
||||
* true.
|
||||
*
|
||||
* `relPath` is path.join'd with `root` before emission, matching how
|
||||
* chokidar delivers absolute paths to the `all` handler.
|
||||
*/
|
||||
export function triggerFileEvent(
|
||||
root: string,
|
||||
event: 'add' | 'change' | 'unlink' | 'addDir' | 'unlinkDir',
|
||||
relPath: string,
|
||||
): void {
|
||||
const inst = instancesByRoot.get(root);
|
||||
if (!inst) {
|
||||
throw new Error(
|
||||
`triggerFileEvent: no mock chokidar watcher registered for root '${root}' — did chokidar.watch() get called?`,
|
||||
);
|
||||
}
|
||||
// FileWatcher uses path.relative(root, eventPath) to compute the
|
||||
// normalized path it stores. We supply the absolute path here so that
|
||||
// operation produces the relPath the test wrote.
|
||||
const absPath = require('node:path').join(root, relPath);
|
||||
inst.triggerEvent(event, absPath);
|
||||
}
|
||||
|
||||
/** Reset all in-memory mock watchers — call in afterEach when needed. */
|
||||
export function resetChokidarMock(): void {
|
||||
for (const inst of instancesByRoot.values()) {
|
||||
inst.removeAllListeners();
|
||||
}
|
||||
instancesByRoot.clear();
|
||||
}
|
||||
@@ -11,27 +11,22 @@
|
||||
* decides whether to Read the specific stale file. These tests exercise
|
||||
* the full real path: real CodeGraph index + real ToolHandler.execute().
|
||||
*
|
||||
* **chokidar is mocked** (see __helpers__/chokidar-mock.ts): the real
|
||||
* FSEvents/inotify event delivery is non-deterministic under parallel
|
||||
* vitest execution and produced a consistent ~30% failure rate on these
|
||||
* tests when run inside the full suite. The mock replaces chokidar with
|
||||
* a controllable EventEmitter so the tests synthesize file events
|
||||
* deterministically via `triggerFileEvent(...)` instead of waiting on
|
||||
* the OS-level watcher to deliver. The watcher's actual debounce timer
|
||||
* (real setTimeout) is left untouched.
|
||||
* **Event delivery uses a synthetic seam** (`__emitWatchEventForTests`): the
|
||||
* real native fs.watch (FSEvents/inotify) delivery is non-deterministic under
|
||||
* parallel vitest execution and produced a consistent ~30% failure rate on
|
||||
* these tests when run inside the full suite. The seam drives the watcher's
|
||||
* pending-set pipeline directly so the tests synthesize file events
|
||||
* deterministically. The watcher's actual debounce timer (real setTimeout) is
|
||||
* left untouched.
|
||||
*/
|
||||
|
||||
import { vi } from 'vitest';
|
||||
// Hoisted: chokidar is replaced by the controllable mock for this file.
|
||||
vi.mock('chokidar', async () => (await import('./__helpers__/chokidar-mock')).chokidarMockModule);
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import CodeGraph from '../src/index';
|
||||
import { ToolHandler } from '../src/mcp/tools';
|
||||
import { triggerFileEvent } from './__helpers__/chokidar-mock';
|
||||
import { __emitWatchEventForTests } from '../src/sync/watcher';
|
||||
|
||||
function waitFor(condition: () => boolean, timeoutMs = 2000, intervalMs = 25): Promise<void> {
|
||||
return new Promise((resolve, reject) => {
|
||||
@@ -83,7 +78,7 @@ describe('MCP staleness banner', () => {
|
||||
|
||||
it('prepends a stale banner when the response references a pending file', async () => {
|
||||
// Long debounce so the edit lingers in pendingFiles while we query.
|
||||
cg.watch({ debounceMs: 4000 });
|
||||
cg.watch({ debounceMs: 4000, inertForTests: true });
|
||||
await cg.waitUntilWatcherReady();
|
||||
|
||||
// Real disk write so a later sync (if it fires) sees the new content,
|
||||
@@ -93,7 +88,7 @@ describe('MCP staleness banner', () => {
|
||||
path.join(testDir, 'src', 'alpha-only.ts'),
|
||||
'export function alphaOnly() { return 99; }\n',
|
||||
);
|
||||
triggerFileEvent(testDir, 'change', 'src/alpha-only.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/alpha-only.ts');
|
||||
|
||||
// With mocked chokidar this is synchronous — keep the wait just to
|
||||
// exercise the realistic shape (the watcher's `chokidarReady` gate
|
||||
@@ -114,7 +109,7 @@ describe('MCP staleness banner', () => {
|
||||
});
|
||||
|
||||
it('uses the footer (not the banner) when pending files are not referenced', async () => {
|
||||
cg.watch({ debounceMs: 4000 });
|
||||
cg.watch({ debounceMs: 4000, inertForTests: true });
|
||||
await cg.waitUntilWatcherReady();
|
||||
|
||||
// Edit bravo-only.ts but search for the alphaOnly symbol, whose hit is
|
||||
@@ -124,7 +119,7 @@ describe('MCP staleness banner', () => {
|
||||
path.join(testDir, 'src', 'bravo-only.ts'),
|
||||
'export function bravoOnly() { return 22; }\n',
|
||||
);
|
||||
triggerFileEvent(testDir, 'change', 'src/bravo-only.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/bravo-only.ts');
|
||||
await waitFor(() => cg.getPendingFiles().some((p) => p.path === 'src/bravo-only.ts'));
|
||||
|
||||
const res = await handler.execute('codegraph_search', { query: 'alphaOnly' });
|
||||
@@ -136,14 +131,14 @@ describe('MCP staleness banner', () => {
|
||||
});
|
||||
|
||||
it('drops the banner once the sync completes and clears the pending entry', async () => {
|
||||
cg.watch({ debounceMs: 200 });
|
||||
cg.watch({ debounceMs: 200, inertForTests: true });
|
||||
await cg.waitUntilWatcherReady();
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(testDir, 'src', 'alpha-only.ts'),
|
||||
'export function alphaOnly() { return 7; }\n',
|
||||
);
|
||||
triggerFileEvent(testDir, 'change', 'src/alpha-only.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/alpha-only.ts');
|
||||
// Wait through debounce (200ms) + sync; pendingFiles drains back to empty.
|
||||
await waitFor(() => cg.getPendingFiles().length === 0, 3000);
|
||||
|
||||
@@ -154,14 +149,14 @@ describe('MCP staleness banner', () => {
|
||||
});
|
||||
|
||||
it('lists pending files under "Pending sync" in codegraph_status', async () => {
|
||||
cg.watch({ debounceMs: 4000 });
|
||||
cg.watch({ debounceMs: 4000, inertForTests: true });
|
||||
await cg.waitUntilWatcherReady();
|
||||
|
||||
fs.writeFileSync(
|
||||
path.join(testDir, 'src', 'charlie-only.ts'),
|
||||
'export function charlieOnly() { return 33; }\n',
|
||||
);
|
||||
triggerFileEvent(testDir, 'change', 'src/charlie-only.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/charlie-only.ts');
|
||||
await waitFor(() => cg.getPendingFiles().some((p) => p.path === 'src/charlie-only.ts'));
|
||||
|
||||
const res = await handler.execute('codegraph_status', {});
|
||||
|
||||
+77
-83
@@ -3,37 +3,38 @@
|
||||
*
|
||||
* Tests for the file watcher that auto-syncs on changes.
|
||||
*
|
||||
* **Why `vi.mock('chokidar', ...)`**: real chokidar bindings go through
|
||||
* FSEvents (macOS) / inotify (Linux). Under parallel vitest execution those
|
||||
* OS-level subsystems serve many test files at once and event-delivery
|
||||
* latency becomes non-deterministic — we observed a consistent ~30%
|
||||
* failure rate on the pending-file-tracking + staleness-banner tests when
|
||||
* running the full suite, vs 0/N when run in isolation. The mock replaces
|
||||
* chokidar with a controllable EventEmitter (see
|
||||
* `__helpers__/chokidar-mock.ts`): the `ready` event fires on the next
|
||||
* microtask, and tests use `triggerFileEvent(...)` to synthesize file
|
||||
* events instead of `fs.writeFileSync(...)`. The watcher's actual
|
||||
* debounce timer (real `setTimeout`) is left untouched — that's the unit
|
||||
* under test.
|
||||
* **Why inert mode + a synthetic event seam**: the watcher now uses Node's
|
||||
* native `fs.watch` (recursive on macOS/Windows, per-directory on Linux).
|
||||
* Under parallel vitest the OS watch subsystems (FSEvents / inotify) serve
|
||||
* many test files at once and event-delivery latency becomes non-deterministic
|
||||
* — a real fs change made in `beforeEach` can even leak into a later "should
|
||||
* NOT sync" assertion. So the unit tests construct the watcher with
|
||||
* `inertForTests: true` (no OS watcher installed) and drive its filter →
|
||||
* pendingFiles → debounce pipeline directly via
|
||||
* `__emitWatchEventForTests(root, relPath)` — deterministic, the same
|
||||
* convergence point a real event reaches. The debounce timer itself is the
|
||||
* real `setTimeout` (the unit under test). One end-to-end test ("auto-sync …
|
||||
* real fs.watch") runs the genuine native watcher against a real file write.
|
||||
*/
|
||||
|
||||
import { vi } from 'vitest';
|
||||
// Hoisted: chokidar is replaced by the controllable mock for the whole file.
|
||||
vi.mock('chokidar', async () => (await import('./__helpers__/chokidar-mock')).chokidarMockModule);
|
||||
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest';
|
||||
import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import { FileWatcher, LockUnavailableError } from '../src/sync/watcher';
|
||||
import {
|
||||
FileWatcher,
|
||||
LockUnavailableError,
|
||||
__emitWatchEventForTests,
|
||||
type WatchOptions,
|
||||
} from '../src/sync/watcher';
|
||||
import CodeGraph from '../src/index';
|
||||
import { triggerFileEvent } from './__helpers__/chokidar-mock';
|
||||
|
||||
type SyncFn = () => Promise<{ filesChanged: number; durationMs: number }>;
|
||||
|
||||
/**
|
||||
* Helper to wait for a condition with timeout. Most tests no longer need
|
||||
* this because mock chokidar makes the watcher's event handler run
|
||||
* synchronously, but it's still useful for assertions that depend on the
|
||||
* debounce timer (real setTimeout) firing.
|
||||
* Helper to wait for a condition with timeout. Used for assertions that depend
|
||||
* on the debounce timer (real setTimeout) firing, or on the real watcher's
|
||||
* event delivery in the end-to-end test.
|
||||
*/
|
||||
function waitFor(
|
||||
condition: () => boolean,
|
||||
@@ -54,6 +55,11 @@ function waitFor(
|
||||
describe('FileWatcher', () => {
|
||||
let testDir: string;
|
||||
|
||||
// Inert by default — unit tests drive events via __emitWatchEventForTests
|
||||
// and never depend on real OS watch delivery.
|
||||
const newWatcher = (syncFn: SyncFn, opts: WatchOptions = {}) =>
|
||||
new FileWatcher(testDir, syncFn, { inertForTests: true, ...opts });
|
||||
|
||||
beforeEach(() => {
|
||||
testDir = fs.mkdtempSync(path.join(os.tmpdir(), 'codegraph-watcher-'));
|
||||
// Create a source file so the directory isn't empty
|
||||
@@ -71,7 +77,7 @@ describe('FileWatcher', () => {
|
||||
describe('start/stop lifecycle', () => {
|
||||
it('should start and stop without errors', () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 });
|
||||
const watcher = new FileWatcher(testDir, syncFn);
|
||||
const watcher = newWatcher(syncFn);
|
||||
|
||||
const started = watcher.start();
|
||||
expect(started).toBe(true);
|
||||
@@ -83,7 +89,7 @@ describe('FileWatcher', () => {
|
||||
|
||||
it('should be idempotent on double start', () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 });
|
||||
const watcher = new FileWatcher(testDir, syncFn);
|
||||
const watcher = newWatcher(syncFn);
|
||||
|
||||
expect(watcher.start()).toBe(true);
|
||||
expect(watcher.start()).toBe(true); // Should not throw
|
||||
@@ -94,7 +100,7 @@ describe('FileWatcher', () => {
|
||||
|
||||
it('should be idempotent on double stop', () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 });
|
||||
const watcher = new FileWatcher(testDir, syncFn);
|
||||
const watcher = newWatcher(syncFn);
|
||||
|
||||
watcher.start();
|
||||
watcher.stop();
|
||||
@@ -107,11 +113,11 @@ describe('FileWatcher', () => {
|
||||
describe('debounced sync', () => {
|
||||
it('should trigger sync after file change', async () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 1, durationMs: 10 });
|
||||
const watcher = new FileWatcher(testDir, syncFn, { debounceMs: 200 });
|
||||
const watcher = newWatcher(syncFn, { debounceMs: 200 });
|
||||
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
triggerFileEvent(testDir, 'add', 'src/new.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/new.ts');
|
||||
|
||||
// Wait for debounced sync to fire (real timer; 200ms + epsilon).
|
||||
await waitFor(() => syncFn.mock.calls.length > 0);
|
||||
@@ -122,7 +128,7 @@ describe('FileWatcher', () => {
|
||||
|
||||
it('should debounce rapid changes into a single sync', async () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 1, durationMs: 10 });
|
||||
const watcher = new FileWatcher(testDir, syncFn, { debounceMs: 400 });
|
||||
const watcher = newWatcher(syncFn, { debounceMs: 400 });
|
||||
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
@@ -131,7 +137,7 @@ describe('FileWatcher', () => {
|
||||
// Spacing them tighter than the debounce window proves the debounce
|
||||
// collapses them into one syncFn call.
|
||||
for (let i = 0; i < 5; i++) {
|
||||
triggerFileEvent(testDir, 'add', `src/file${i}.ts`);
|
||||
__emitWatchEventForTests(testDir, `src/file${i}.ts`);
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
}
|
||||
|
||||
@@ -148,14 +154,14 @@ describe('FileWatcher', () => {
|
||||
describe('filtering', () => {
|
||||
it('should ignore files not matching include patterns', async () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 });
|
||||
const watcher = new FileWatcher(testDir, syncFn, { debounceMs: 200 });
|
||||
const watcher = newWatcher(syncFn, { debounceMs: 200 });
|
||||
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
|
||||
// Synthesize a non-source-file event — FileWatcher's `isSourceFile`
|
||||
// gate must drop it before scheduling sync.
|
||||
triggerFileEvent(testDir, 'add', 'src/readme.md');
|
||||
// A non-source-file event — FileWatcher's `isSourceFile` gate must drop
|
||||
// it before scheduling sync.
|
||||
__emitWatchEventForTests(testDir, 'src/readme.md');
|
||||
|
||||
// Wait a bit longer than debounce — sync should NOT trigger.
|
||||
await new Promise((r) => setTimeout(r, 400));
|
||||
@@ -166,14 +172,14 @@ describe('FileWatcher', () => {
|
||||
|
||||
it('should ignore .codegraph directory changes', async () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 });
|
||||
const watcher = new FileWatcher(testDir, syncFn, { debounceMs: 200 });
|
||||
const watcher = newWatcher(syncFn, { debounceMs: 200 });
|
||||
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
|
||||
// Synthesize a .codegraph event — FileWatcher's `isAlwaysIgnored`
|
||||
// filter must drop it before scheduling sync.
|
||||
triggerFileEvent(testDir, 'add', '.codegraph/db.sqlite');
|
||||
// A .codegraph event — FileWatcher's `isAlwaysIgnored` filter must drop
|
||||
// it before scheduling sync.
|
||||
__emitWatchEventForTests(testDir, '.codegraph/db.sqlite');
|
||||
|
||||
await new Promise((r) => setTimeout(r, 400));
|
||||
expect(syncFn).not.toHaveBeenCalled();
|
||||
@@ -181,26 +187,17 @@ describe('FileWatcher', () => {
|
||||
watcher.stop();
|
||||
});
|
||||
|
||||
it('should not schedule sync for node_modules paths (FileWatcher-side filter)', async () => {
|
||||
// NOTE: this previously asserted chokidar's `ignored` callback excluded
|
||||
// node_modules from watching at all. With chokidar mocked, that
|
||||
// OS-level behaviour isn't exercised here — what we test is
|
||||
// FileWatcher's own filter chain (`isSourceFile` + `isAlwaysIgnored`).
|
||||
// node_modules paths AREN'T in `isAlwaysIgnored` (they're filtered by
|
||||
// chokidar's `ignored` callback in production), so this test now
|
||||
// verifies a different mechanism: a non-source extension inside
|
||||
// node_modules still drops via `isSourceFile`. The chokidar-level
|
||||
// `ignored` exclusion of `node_modules/` itself is covered by the
|
||||
// ignore-config tests under `src/sync/watcher-ignore.test.ts`-style
|
||||
// unit-level checks, which don't need a live watcher loop.
|
||||
it('should drop ignored/non-source paths but sync real source edits', async () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 0, durationMs: 0 });
|
||||
const watcher = new FileWatcher(testDir, syncFn, { debounceMs: 200 });
|
||||
const watcher = newWatcher(syncFn, { debounceMs: 200 });
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
|
||||
// A source-extension event whose path is a normal source file still
|
||||
// schedules sync (positive control).
|
||||
triggerFileEvent(testDir, 'add', 'src/live.ts');
|
||||
// node_modules is in the default-ignore set (#407) → dropped by the
|
||||
// ignore matcher even without a .gitignore.
|
||||
__emitWatchEventForTests(testDir, 'node_modules/dep/index.js');
|
||||
// A normal source file still schedules sync (positive control).
|
||||
__emitWatchEventForTests(testDir, 'src/live.ts');
|
||||
await waitFor(() => syncFn.mock.calls.length > 0);
|
||||
expect(syncFn).toHaveBeenCalled();
|
||||
|
||||
@@ -210,17 +207,16 @@ describe('FileWatcher', () => {
|
||||
|
||||
describe('pending file tracking (#403)', () => {
|
||||
it('should expose edited paths via getPendingFiles before sync fires', async () => {
|
||||
// Slow debounce — pending entries are visible until the debounce
|
||||
// fires. With mocked chokidar the event is synchronous, so we can
|
||||
// assert immediately without polling.
|
||||
// Slow debounce — pending entries are visible until the debounce fires.
|
||||
// The synthetic event is synchronous, so we can assert immediately.
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 1, durationMs: 10 });
|
||||
const watcher = new FileWatcher(testDir, syncFn, { debounceMs: 2000 });
|
||||
const watcher = newWatcher(syncFn, { debounceMs: 2000 });
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
|
||||
expect(watcher.getPendingFiles()).toEqual([]);
|
||||
|
||||
triggerFileEvent(testDir, 'add', 'src/pending.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/pending.ts');
|
||||
|
||||
const pending = watcher.getPendingFiles();
|
||||
const paths = pending.map((p) => p.path);
|
||||
@@ -236,11 +232,11 @@ describe('FileWatcher', () => {
|
||||
|
||||
it('should clear an entry only after a successful sync absorbing that edit', async () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 1, durationMs: 10 });
|
||||
const watcher = new FileWatcher(testDir, syncFn, { debounceMs: 200 });
|
||||
const watcher = newWatcher(syncFn, { debounceMs: 200 });
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
|
||||
triggerFileEvent(testDir, 'add', 'src/fresh.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/fresh.ts');
|
||||
|
||||
// Watcher saw the change → pendingFiles has the entry IMMEDIATELY.
|
||||
expect(watcher.getPendingFiles().some((p) => p.path === 'src/fresh.ts')).toBe(true);
|
||||
@@ -254,18 +250,18 @@ describe('FileWatcher', () => {
|
||||
});
|
||||
|
||||
it('should keep entries unchanged when sync fails (rescheduled work sees the same set)', async () => {
|
||||
// With chokidar mocked there's no initial-scan-triggered sync, so
|
||||
// the syncFn outcomes line up 1:1 with explicit events.
|
||||
// No initial-scan-triggered sync, so syncFn outcomes line up 1:1 with
|
||||
// explicit events.
|
||||
const syncFn = vi
|
||||
.fn()
|
||||
.mockRejectedValueOnce(new Error('boom')) // first sync rejects
|
||||
.mockResolvedValueOnce({ filesChanged: 1, durationMs: 10 }); // retry succeeds
|
||||
const onSyncError = vi.fn();
|
||||
const watcher = new FileWatcher(testDir, syncFn, { debounceMs: 100, onSyncError });
|
||||
const watcher = newWatcher(syncFn, { debounceMs: 100, onSyncError });
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
|
||||
triggerFileEvent(testDir, 'add', 'src/will-fail.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/will-fail.ts');
|
||||
|
||||
// Wait for the sync to reject.
|
||||
await waitFor(() => onSyncError.mock.calls.length > 0);
|
||||
@@ -293,7 +289,7 @@ describe('FileWatcher', () => {
|
||||
.mockResolvedValueOnce({ filesChanged: 1, durationMs: 10 });
|
||||
const onSyncComplete = vi.fn();
|
||||
const onSyncError = vi.fn();
|
||||
const watcher = new FileWatcher(testDir, syncFn, {
|
||||
const watcher = newWatcher(syncFn, {
|
||||
debounceMs: 100,
|
||||
onSyncComplete,
|
||||
onSyncError,
|
||||
@@ -301,7 +297,7 @@ describe('FileWatcher', () => {
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
|
||||
triggerFileEvent(testDir, 'add', 'src/locked.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/locked.ts');
|
||||
|
||||
await waitFor(() => syncFn.mock.calls.length >= 1);
|
||||
expect(watcher.getPendingFiles().some((p) => p.path === 'src/locked.ts')).toBe(true);
|
||||
@@ -327,14 +323,14 @@ describe('FileWatcher', () => {
|
||||
it('should call onSyncComplete after successful sync', async () => {
|
||||
const syncFn = vi.fn().mockResolvedValue({ filesChanged: 2, durationMs: 50 });
|
||||
const onSyncComplete = vi.fn();
|
||||
const watcher = new FileWatcher(testDir, syncFn, {
|
||||
const watcher = newWatcher(syncFn, {
|
||||
debounceMs: 200,
|
||||
onSyncComplete,
|
||||
});
|
||||
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
triggerFileEvent(testDir, 'add', 'src/test.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/test.ts');
|
||||
|
||||
await waitFor(() => onSyncComplete.mock.calls.length > 0);
|
||||
expect(onSyncComplete).toHaveBeenCalledWith({ filesChanged: 2, durationMs: 50 });
|
||||
@@ -345,14 +341,14 @@ describe('FileWatcher', () => {
|
||||
it('should call onSyncError when sync throws', async () => {
|
||||
const syncFn = vi.fn().mockRejectedValue(new Error('sync failed'));
|
||||
const onSyncError = vi.fn();
|
||||
const watcher = new FileWatcher(testDir, syncFn, {
|
||||
const watcher = newWatcher(syncFn, {
|
||||
debounceMs: 200,
|
||||
onSyncError,
|
||||
});
|
||||
|
||||
watcher.start();
|
||||
await watcher.waitUntilReady();
|
||||
triggerFileEvent(testDir, 'add', 'src/test.ts');
|
||||
__emitWatchEventForTests(testDir, 'src/test.ts');
|
||||
|
||||
await waitFor(() => onSyncError.mock.calls.length > 0);
|
||||
expect(onSyncError).toHaveBeenCalled();
|
||||
@@ -377,7 +373,7 @@ describe('FileWatcher', () => {
|
||||
|
||||
expect(cg.isWatching()).toBe(false);
|
||||
|
||||
const started = cg.watch({ debounceMs: 200 });
|
||||
const started = cg.watch({ debounceMs: 200, inertForTests: true });
|
||||
expect(started).toBe(true);
|
||||
expect(cg.isWatching()).toBe(true);
|
||||
|
||||
@@ -391,7 +387,7 @@ describe('FileWatcher', () => {
|
||||
});
|
||||
await cg.indexAll();
|
||||
|
||||
cg.watch({ debounceMs: 200 });
|
||||
cg.watch({ debounceMs: 200, inertForTests: true });
|
||||
expect(cg.isWatching()).toBe(true);
|
||||
|
||||
cg.close();
|
||||
@@ -400,7 +396,9 @@ describe('FileWatcher', () => {
|
||||
// but we verify no errors are thrown)
|
||||
});
|
||||
|
||||
it('should auto-sync when files change while watching', async () => {
|
||||
it('should auto-sync when files change while watching (real fs.watch end-to-end)', async () => {
|
||||
// The one test that exercises the genuine native watcher: a real file
|
||||
// write must propagate through fs.watch → debounce → sync into the graph.
|
||||
cg = CodeGraph.initSync(testDir, {
|
||||
config: { include: ['**/*.ts'], exclude: [] },
|
||||
});
|
||||
@@ -410,24 +408,20 @@ describe('FileWatcher', () => {
|
||||
const initialNodes = initialStats.nodeCount;
|
||||
|
||||
cg.watch({ debounceMs: 300 });
|
||||
// Wait through CodeGraph's internal watcher startup (the mock
|
||||
// chokidar fires `ready` on the next microtask, but cg.watch wraps
|
||||
// the watcher creation through promise plumbing).
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
// Let the watcher install before writing, so the event isn't missed.
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
|
||||
// Real fs write so cg.sync() can detect the new file on disk; then
|
||||
// synthesize the event to wake the watcher (debounce + sync).
|
||||
// Real fs write — no synthetic event. The live watcher must catch it.
|
||||
fs.writeFileSync(
|
||||
path.join(testDir, 'src', 'added.ts'),
|
||||
'export function added() { return 42; }'
|
||||
);
|
||||
triggerFileEvent(testDir, 'add', 'src/added.ts');
|
||||
|
||||
// Wait for auto-sync to pick it up.
|
||||
// Wait for auto-sync to pick it up (real OS event delivery + debounce).
|
||||
await waitFor(() => {
|
||||
const stats = cg.getStats();
|
||||
return stats.nodeCount > initialNodes;
|
||||
}, 5000);
|
||||
}, 8000);
|
||||
|
||||
// The new function should be in the graph.
|
||||
const results = cg.searchNodes('added');
|
||||
|
||||
Generated
-29
@@ -10,7 +10,6 @@
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@clack/prompts": "^1.3.0",
|
||||
"chokidar": "^4.0.3",
|
||||
"commander": "^14.0.2",
|
||||
"fast-string-width": "^3.0.2",
|
||||
"fast-wrap-ansi": "^0.2.0",
|
||||
@@ -1005,21 +1004,6 @@
|
||||
"node": ">= 16"
|
||||
}
|
||||
},
|
||||
"node_modules/chokidar": {
|
||||
"version": "4.0.3",
|
||||
"resolved": "https://registry.npmjs.org/chokidar/-/chokidar-4.0.3.tgz",
|
||||
"integrity": "sha512-Qgzu8kfBvo+cA4962jnP1KkS6Dop5NS6g7R5LFYJr4b8Ub94PPQXUksCw9PvXoeXPRRddRNC5C1JQUR2SMGtnA==",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"readdirp": "^4.0.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">= 14.16.0"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://paulmillr.com/funding/"
|
||||
}
|
||||
},
|
||||
"node_modules/commander": {
|
||||
"version": "14.0.3",
|
||||
"resolved": "https://registry.npmjs.org/commander/-/commander-14.0.3.tgz",
|
||||
@@ -1285,19 +1269,6 @@
|
||||
"node": "^10 || ^12 || >=14"
|
||||
}
|
||||
},
|
||||
"node_modules/readdirp": {
|
||||
"version": "4.1.2",
|
||||
"resolved": "https://registry.npmjs.org/readdirp/-/readdirp-4.1.2.tgz",
|
||||
"integrity": "sha512-GDhwkLfywWL2s6vEjyhri+eXmfH6j1L7JE27WhqLeYzoh/A3DBaYGEj2H/HFZCn/kMfim73FXxEJTw06WtxQwg==",
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
"node": ">= 14.18.0"
|
||||
},
|
||||
"funding": {
|
||||
"type": "individual",
|
||||
"url": "https://paulmillr.com/funding/"
|
||||
}
|
||||
},
|
||||
"node_modules/rollup": {
|
||||
"version": "4.57.1",
|
||||
"resolved": "https://registry.npmjs.org/rollup/-/rollup-4.57.1.tgz",
|
||||
|
||||
@@ -33,7 +33,6 @@
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@clack/prompts": "^1.3.0",
|
||||
"chokidar": "^4.0.3",
|
||||
"commander": "^14.0.2",
|
||||
"fast-string-width": "^3.0.2",
|
||||
"fast-wrap-ansi": "^0.2.0",
|
||||
|
||||
+2
-2
@@ -561,8 +561,8 @@ export class CodeGraph {
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves once the file watcher has finished its initial chokidar scan.
|
||||
* Useful for tests that need a deterministic boundary before asserting on
|
||||
* Resolves once the file watcher has installed its watch set. Useful for
|
||||
* tests that need a deterministic boundary before asserting on
|
||||
* `getPendingFiles()`. Resolves immediately when no watcher is active.
|
||||
*/
|
||||
waitUntilWatcherReady(timeoutMs?: number): Promise<void> {
|
||||
|
||||
+1
-1
@@ -290,7 +290,7 @@ export type AcquireResult =
|
||||
* the file existed but was empty; under concurrent daemon startup a third
|
||||
* candidate could read that empty file, decode it as `null`, and `unlink` the
|
||||
* winner's lock → two daemons (two watchers, two writers). The window was
|
||||
* normally too small to hit, but the chokidar watcher's extra startup time made
|
||||
* normally too small to hit, but the file watcher's extra startup time made
|
||||
* concurrent daemons overlap enough to reproduce it reliably.
|
||||
*
|
||||
* The fix writes the complete record to a private temp file, then hard-links it
|
||||
|
||||
+315
-97
@@ -4,25 +4,79 @@
|
||||
* Watches the project directory for file changes and triggers debounced sync
|
||||
* operations to keep the code graph up-to-date.
|
||||
*
|
||||
* Uses chokidar, whose `ignored` callback filters directories BEFORE they are
|
||||
* watched — so we never register inotify watches on excluded trees like
|
||||
* node_modules/, dist/, .git/ (fixes #276: recursive fs.watch exhausted the
|
||||
* kernel watch budget on large repos). The ignore decision reuses the indexer's
|
||||
* `buildDefaultIgnore` (built-in default-ignore dirs + the project's .gitignore)
|
||||
* so the watcher watches exactly the set the indexer indexes — in particular,
|
||||
* node_modules/build/cache dirs are excluded even when the repo has no
|
||||
* .gitignore (#407), which a .gitignore-only filter would miss.
|
||||
* Uses Node's built-in `fs.watch` directly (no third-party watcher, no native
|
||||
* addon) with a per-platform strategy chosen to keep the open-descriptor /
|
||||
* kernel-watch cost BOUNDED rather than growing with the number of files:
|
||||
*
|
||||
* - macOS / Windows: a SINGLE recursive `fs.watch(root, {recursive:true})`.
|
||||
* libuv maps this to one FSEvents stream (macOS) / one
|
||||
* ReadDirectoryChangesW handle (Windows), so it costs O(1) descriptors no
|
||||
* matter how large the tree. This is the fix for the macOS file-table
|
||||
* exhaustion (#644 / #496 / #555 / #628): the previous watcher held one
|
||||
* open fd PER WATCHED FILE on macOS (tens of thousands of REG fds), which
|
||||
* exhausted `kern.maxfiles` and crashed unrelated processes system-wide.
|
||||
*
|
||||
* - Linux: recursive `fs.watch` is unsupported, so we watch each (non-ignored)
|
||||
* DIRECTORY with one inotify watch — O(directories), NOT O(files). New
|
||||
* directories are picked up dynamically and an overall watch cap bounds
|
||||
* inotify usage on pathological monorepos (#579). A single inotify watch on
|
||||
* a directory already reports create/modify/delete for its children, so
|
||||
* per-file watches are never needed.
|
||||
*
|
||||
* Excluded trees (node_modules/, dist/, .git/, …) are filtered via the
|
||||
* indexer's `buildDefaultIgnore` (built-in default-ignore dirs + the project's
|
||||
* .gitignore) — on Linux they're never descended into (so they cost no watch),
|
||||
* and on macOS/Windows the single recursive stream still covers them but their
|
||||
* events are dropped before any sync is scheduled. Either way the watcher's
|
||||
* scope matches the indexer's (#276 / #407).
|
||||
*/
|
||||
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import type { Stats } from 'fs';
|
||||
import chokidar, { FSWatcher } from 'chokidar';
|
||||
import type { Ignore } from 'ignore';
|
||||
import { isSourceFile, buildDefaultIgnore } from '../extraction';
|
||||
import { logDebug, logWarn } from '../errors';
|
||||
import { normalizePath } from '../utils';
|
||||
import { watchDisabledReason } from './watch-policy';
|
||||
|
||||
/**
|
||||
* Native recursive `fs.watch` is only reliable on macOS and Windows; on Linux
|
||||
* (and AIX) it throws `ERR_FEATURE_UNAVAILABLE_ON_PLATFORM`. We branch on this
|
||||
* to pick the recursive vs per-directory strategy.
|
||||
*/
|
||||
function supportsRecursiveWatch(): boolean {
|
||||
return process.platform === 'darwin' || process.platform === 'win32';
|
||||
}
|
||||
|
||||
/**
|
||||
* Upper bound on simultaneously-watched directories on the Linux per-directory
|
||||
* path. Each is one inotify watch; the kernel's `fs.inotify.max_user_watches`
|
||||
* is the hard limit (commonly 8k–128k). We stop adding watches past this and
|
||||
* log once — partial live-watch (with `codegraph sync` as the backstop) is far
|
||||
* better than exhausting the user's inotify budget and breaking watching
|
||||
* system-wide (#579). Tunable via CODEGRAPH_MAX_DIR_WATCHES.
|
||||
*/
|
||||
const DEFAULT_MAX_DIR_WATCHES = 50_000;
|
||||
|
||||
function maxDirWatches(): number {
|
||||
const raw = process.env.CODEGRAPH_MAX_DIR_WATCHES;
|
||||
if (raw && /^\d+$/.test(raw)) {
|
||||
const n = Number(raw);
|
||||
if (n > 0) return n;
|
||||
}
|
||||
return DEFAULT_MAX_DIR_WATCHES;
|
||||
}
|
||||
|
||||
/**
|
||||
* Test seam (see {@link __emitWatchEventForTests}). Maps a watcher's project
|
||||
* root to its live instance so tests can synthesize a change event
|
||||
* deterministically — real fs.watch delivery latency races under parallel
|
||||
* vitest (the reason the previous chokidar mock existed). Only populated under
|
||||
* a test runner, so production carries no bookkeeping or retained references.
|
||||
*/
|
||||
const liveWatchersForTests = new Map<string, FileWatcher>();
|
||||
const IS_TEST_RUNTIME = !!(process.env.VITEST || process.env.NODE_ENV === 'test');
|
||||
|
||||
/**
|
||||
* Options for the file watcher
|
||||
*/
|
||||
@@ -43,6 +97,16 @@ export interface WatchOptions {
|
||||
* Callback when a sync errors (for logging/diagnostics).
|
||||
*/
|
||||
onSyncError?: (error: Error) => void;
|
||||
|
||||
/**
|
||||
* Test-only. When true, `start()` installs NO OS-level fs.watch — the
|
||||
* watcher is "inert" and only the {@link __emitWatchEventForTests} /
|
||||
* {@link FileWatcher.ingestEventForTests} seam drives its pipeline. This
|
||||
* restores the deterministic, OS-free behavior the unit tests need (real
|
||||
* FSEvents/inotify delivery races under parallel vitest). Production never
|
||||
* sets it.
|
||||
*/
|
||||
inertForTests?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -84,8 +148,9 @@ export interface PendingFile {
|
||||
* debounced sync operations via a provided callback.
|
||||
*
|
||||
* Design goals:
|
||||
* - Minimal resource usage (chokidar filters excluded directories before
|
||||
* registering an inotify watch — see module docs / #276)
|
||||
* - Bounded resource usage: O(1) descriptors on macOS/Windows (one recursive
|
||||
* watch), O(directories) inotify watches on Linux — never O(files), which
|
||||
* was the system-crashing fd leak on macOS (#644/#496/#555/#628).
|
||||
* - Debounced to avoid thrashing on rapid saves
|
||||
* - Filters to supported source files by extension
|
||||
* - Ignores .codegraph/ and .git/ regardless of .gitignore
|
||||
@@ -93,11 +158,18 @@ export interface PendingFile {
|
||||
* without blocking on a sync (issue #403)
|
||||
*/
|
||||
export class FileWatcher {
|
||||
private watcher: FSWatcher | null = null;
|
||||
/** macOS/Windows: the single recursive watcher. Null on Linux. */
|
||||
private recursiveWatcher: fs.FSWatcher | null = null;
|
||||
/** Linux: one watcher per watched directory (keyed by absolute path). */
|
||||
private dirWatchers = new Map<string, fs.FSWatcher>();
|
||||
/** Set once the per-directory watch cap is hit, so we log only once. */
|
||||
private dirCapWarned = false;
|
||||
/** Test-only inert mode: started, but with no OS watcher installed. */
|
||||
private inert = false;
|
||||
private debounceTimer: ReturnType<typeof setTimeout> | null = null;
|
||||
/**
|
||||
* Files seen by the watcher since the last successful sync — populated on
|
||||
* every chokidar event, cleared at the start of a sync, and re-populated by
|
||||
* every change event, cleared at the start of a sync, and re-populated by
|
||||
* events that arrive mid-sync (or restored on sync failure). Keyed by the
|
||||
* same project-relative POSIX path the rest of the codebase uses, so a
|
||||
* caller can intersect tool-response file paths against this map cheaply.
|
||||
@@ -113,17 +185,18 @@ export class FileWatcher {
|
||||
private syncing = false;
|
||||
private stopped = false;
|
||||
/**
|
||||
* False until chokidar fires its `ready` event. Gates `pendingFiles`
|
||||
* insertion so the initial crawl's `add` events (one per pre-existing
|
||||
* source file) don't pollute the per-file staleness signal. The events
|
||||
* still flow into `scheduleSync()` to preserve the previous "initial
|
||||
* scan triggers a reconciling sync" behavior.
|
||||
* True once the initial watch set is established. Unlike the previous
|
||||
* chokidar implementation there is no asynchronous initial "crawl" emitting
|
||||
* an `add` per existing file — `fs.watch` only reports changes from the
|
||||
* moment it's installed — so this flips to true synchronously at the end of
|
||||
* `start()`. The startup reconcile against on-disk state is handled
|
||||
* separately by the engine's catch-up sync, not by the watcher.
|
||||
*/
|
||||
private chokidarReady = false;
|
||||
private ready = false;
|
||||
/**
|
||||
* Callbacks that resolve when chokidar fires `ready`. Used by tests (and
|
||||
* any production caller that cares about a clean baseline) to deterministically
|
||||
* gate on the end of the initial scan instead of guessing at a sleep duration.
|
||||
* Callbacks that resolve when the watch set is established. Used by tests
|
||||
* (and any production caller that cares about a clean baseline) to
|
||||
* deterministically gate on watcher readiness.
|
||||
*/
|
||||
private readyWaiters: Array<() => void> = [];
|
||||
// The shared ignore matcher (built-in defaults + project .gitignore), built
|
||||
@@ -136,6 +209,7 @@ export class FileWatcher {
|
||||
private readonly syncFn: () => Promise<{ filesChanged: number; durationMs: number }>;
|
||||
private readonly onSyncComplete?: WatchOptions['onSyncComplete'];
|
||||
private readonly onSyncError?: WatchOptions['onSyncError'];
|
||||
private readonly inertForTests: boolean;
|
||||
|
||||
constructor(
|
||||
projectRoot: string,
|
||||
@@ -147,6 +221,7 @@ export class FileWatcher {
|
||||
this.debounceMs = options.debounceMs ?? 2000;
|
||||
this.onSyncComplete = options.onSyncComplete;
|
||||
this.onSyncError = options.onSyncError;
|
||||
this.inertForTests = options.inertForTests ?? false;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -154,7 +229,7 @@ export class FileWatcher {
|
||||
* Returns true if watching started successfully, false otherwise.
|
||||
*/
|
||||
start(): boolean {
|
||||
if (this.watcher) return true; // Already watching
|
||||
if (this.recursiveWatcher || this.dirWatchers.size > 0 || this.inert) return true; // Already watching
|
||||
this.stopped = false;
|
||||
|
||||
// Some environments make filesystem watching unusable — most notably
|
||||
@@ -168,75 +243,183 @@ export class FileWatcher {
|
||||
}
|
||||
|
||||
// Reuse the indexer's ignore set so the watcher and indexer agree on scope.
|
||||
// chokidar only registers an inotify watch on directories that pass this
|
||||
// filter — that's the #276 fix.
|
||||
this.ignoreMatcher = buildDefaultIgnore(this.projectRoot);
|
||||
|
||||
try {
|
||||
this.watcher = chokidar.watch(this.projectRoot, {
|
||||
// chokidar calls this for every path it encounters and only watches
|
||||
// those that pass — so excluded trees (node_modules/, dist/, .git/, …)
|
||||
// never get an inotify watch in the first place.
|
||||
ignored: (testPath: string, stats?: Stats) => this.shouldIgnore(testPath, stats),
|
||||
});
|
||||
if (this.inertForTests) {
|
||||
// Test-only: install no OS watcher; the seam drives events instead.
|
||||
this.inert = true;
|
||||
} else if (supportsRecursiveWatch()) {
|
||||
this.startRecursive();
|
||||
} else {
|
||||
this.startPerDirectory();
|
||||
}
|
||||
|
||||
// Chokidar emits `add` for every pre-existing source file during its
|
||||
// initial scan. Those events should still trigger the post-startup
|
||||
// reconciling sync (preserving prior behavior), but they must NOT land
|
||||
// in pendingFiles — otherwise every file in the project shows up as
|
||||
// "edited but not indexed" on startup, which is the opposite of the
|
||||
// signal #403 is supposed to provide. Flip the flag on chokidar's
|
||||
// `ready` event; from then on, real edits populate pendingFiles.
|
||||
//
|
||||
// We also clear `pendingFiles` here as defense-in-depth: chokidar can
|
||||
// emit late initial-scan `add` events via setImmediate AFTER the
|
||||
// `ready` callback runs (observed under test-parallelism load).
|
||||
// Clearing once at ready guarantees a clean baseline; real subsequent
|
||||
// edits repopulate the set normally.
|
||||
this.watcher.on('ready', () => {
|
||||
this.chokidarReady = true;
|
||||
// No async crawl to wait on: as soon as the watch set is installed we
|
||||
// have a clean baseline (pendingFiles is only populated by post-start
|
||||
// events). Clear defensively and flip ready.
|
||||
this.pendingFiles.clear();
|
||||
this.ready = true;
|
||||
for (const cb of this.readyWaiters) cb();
|
||||
this.readyWaiters.length = 0;
|
||||
if (IS_TEST_RUNTIME) liveWatchersForTests.set(this.projectRoot, this);
|
||||
|
||||
logDebug('File watcher started', {
|
||||
projectRoot: this.projectRoot,
|
||||
debounceMs: this.debounceMs,
|
||||
mode: this.inertForTests ? 'inert' : supportsRecursiveWatch() ? 'recursive' : 'per-directory',
|
||||
watchedDirs: this.dirWatchers.size || undefined,
|
||||
});
|
||||
return true;
|
||||
} catch (err) {
|
||||
// Watcher setup failed (e.g., permission denied, missing directory).
|
||||
logWarn('Could not start file watcher', { error: String(err) });
|
||||
this.stop();
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
// chokidar emits 'all' for every event type; we only sync source files.
|
||||
this.watcher.on('all', (_event: string, filePath: string) => {
|
||||
if (this.stopped) return;
|
||||
/**
|
||||
* macOS/Windows: one recursive watcher for the whole tree. O(1) descriptors.
|
||||
* `filename` arrives relative to the project root (with subdirectories), so
|
||||
* it maps straight to a project-relative path.
|
||||
*/
|
||||
private startRecursive(): void {
|
||||
this.recursiveWatcher = fs.watch(
|
||||
this.projectRoot,
|
||||
{ recursive: true, persistent: true },
|
||||
(_event, filename) => {
|
||||
if (this.stopped || filename == null) return;
|
||||
this.handleChange(normalizePath(String(filename)));
|
||||
}
|
||||
);
|
||||
this.recursiveWatcher.on('error', (err: unknown) => {
|
||||
logWarn('File watcher error', { error: String(err) });
|
||||
});
|
||||
}
|
||||
|
||||
const normalized = normalizePath(path.relative(this.projectRoot, filePath));
|
||||
/**
|
||||
* Linux: walk the (non-ignored) tree and watch each directory. One inotify
|
||||
* watch per directory reports create/modify/delete for that directory's
|
||||
* direct children, so we never watch individual files.
|
||||
*/
|
||||
private startPerDirectory(): void {
|
||||
this.watchTree(this.projectRoot, /* markExisting */ false);
|
||||
}
|
||||
|
||||
// Defense in depth: `ignored` should already keep these out, but events
|
||||
// can still arrive during setup or via symlink traversal.
|
||||
if (this.isAlwaysIgnored(normalized)) return;
|
||||
if (!isSourceFile(normalized)) return;
|
||||
/**
|
||||
* Add an inotify watch for `dir` and recurse into its non-ignored
|
||||
* subdirectories. When `markExisting` is true (a directory that appeared
|
||||
* AFTER startup), the source files already inside it are recorded as pending
|
||||
* — this closes the `mkdir + write` race where files created before the new
|
||||
* directory's watch is installed would otherwise be missed until the next
|
||||
* full sync. The initial startup walk passes false (the engine's catch-up
|
||||
* sync owns the baseline).
|
||||
*/
|
||||
private watchTree(dir: string, markExisting: boolean): void {
|
||||
if (this.dirWatchers.has(dir)) return;
|
||||
if (this.dirWatchers.size >= maxDirWatches()) {
|
||||
if (!this.dirCapWarned) {
|
||||
this.dirCapWarned = true;
|
||||
logWarn('File watcher hit directory-watch cap; remaining subtrees rely on manual/periodic sync', {
|
||||
cap: maxDirWatches(),
|
||||
});
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
logDebug('File change detected', { file: normalized });
|
||||
// Only track events from after chokidar's initial scan as pending
|
||||
// edits — pre-existing files on disk are already represented by
|
||||
// (or about to be reconciled by) the index, not a user edit.
|
||||
if (this.chokidarReady) {
|
||||
let w: fs.FSWatcher;
|
||||
try {
|
||||
w = fs.watch(dir, { persistent: true }, (_event, filename) =>
|
||||
this.handleDirEvent(dir, filename)
|
||||
);
|
||||
} catch {
|
||||
// ENOENT / EACCES / too-many-open-files — skip this directory quietly.
|
||||
return;
|
||||
}
|
||||
w.on('error', () => this.unwatchDir(dir));
|
||||
this.dirWatchers.set(dir, w);
|
||||
|
||||
let entries: fs.Dirent[];
|
||||
try {
|
||||
entries = fs.readdirSync(dir, { withFileTypes: true });
|
||||
} catch {
|
||||
return;
|
||||
}
|
||||
for (const entry of entries) {
|
||||
const child = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
if (this.shouldIgnoreDir(child)) continue;
|
||||
this.watchTree(child, markExisting);
|
||||
} else if (markExisting && entry.isFile()) {
|
||||
this.handleChange(normalizePath(path.relative(this.projectRoot, child)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Linux per-directory event handler. `filename` is relative to `dir`. A new
|
||||
* sub-directory is picked up by extending the watch tree; everything else is
|
||||
* routed through the shared change handler.
|
||||
*/
|
||||
private handleDirEvent(dir: string, filename: string | Buffer | null): void {
|
||||
if (this.stopped || filename == null) return;
|
||||
const full = path.join(dir, String(filename));
|
||||
|
||||
// A newly-created directory needs its own watch (recursive isn't available
|
||||
// on Linux). statSync is cheap and these events are rare relative to file
|
||||
// edits. If the path vanished (rapid create/delete) the stat throws and we
|
||||
// fall through to the change handler, which no-ops on a non-source path.
|
||||
try {
|
||||
if (fs.statSync(full).isDirectory()) {
|
||||
if (!this.shouldIgnoreDir(full)) this.watchTree(full, /* markExisting */ true);
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// deleted/inaccessible — treat as a normal change below
|
||||
}
|
||||
|
||||
this.handleChange(normalizePath(path.relative(this.projectRoot, full)));
|
||||
}
|
||||
|
||||
/**
|
||||
* Shared change handler for both watch strategies. `rel` is a
|
||||
* project-relative POSIX path. Applies the ignore + source-file filters and,
|
||||
* for a real source change, records it as pending (#403) and schedules a
|
||||
* debounced sync.
|
||||
*
|
||||
* The recursive (macOS/Windows) watcher reports events for ignored trees too
|
||||
* (one stream covers the whole repo), so the ignore check here is load-bearing
|
||||
* — it drops node_modules/dist/.git churn before any sync is scheduled.
|
||||
*/
|
||||
private handleChange(rel: string): void {
|
||||
if (!rel || rel === '.' || rel.startsWith('..')) return;
|
||||
if (this.isAlwaysIgnored(rel)) return;
|
||||
if (this.ignoreMatcher && this.ignoreMatcher.ignores(rel)) return;
|
||||
if (!isSourceFile(rel)) return;
|
||||
|
||||
logDebug('File change detected', { file: rel });
|
||||
if (this.ready) {
|
||||
const now = Date.now();
|
||||
const existing = this.pendingFiles.get(normalized);
|
||||
this.pendingFiles.set(normalized, {
|
||||
const existing = this.pendingFiles.get(rel);
|
||||
this.pendingFiles.set(rel, {
|
||||
firstSeenMs: existing?.firstSeenMs ?? now,
|
||||
lastSeenMs: now,
|
||||
});
|
||||
}
|
||||
this.scheduleSync();
|
||||
});
|
||||
}
|
||||
|
||||
// Handle watcher errors gracefully — don't crash, the user can restart.
|
||||
this.watcher.on('error', (err: unknown) => {
|
||||
logWarn('File watcher error', { error: String(err) });
|
||||
});
|
||||
|
||||
logDebug('File watcher started', { projectRoot: this.projectRoot, debounceMs: this.debounceMs });
|
||||
return true;
|
||||
} catch (err) {
|
||||
// Watcher setup failed (e.g., permission denied, missing directory).
|
||||
logWarn('Could not start file watcher', { error: String(err) });
|
||||
return false;
|
||||
/** Close and forget the watch for a directory that errored/was removed. */
|
||||
private unwatchDir(dir: string): void {
|
||||
const w = this.dirWatchers.get(dir);
|
||||
if (w) {
|
||||
try {
|
||||
w.close();
|
||||
} catch {
|
||||
/* already closed */
|
||||
}
|
||||
this.dirWatchers.delete(dir);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -249,20 +432,16 @@ export class FileWatcher {
|
||||
}
|
||||
|
||||
/**
|
||||
* chokidar `ignored` predicate — true for any path that should NOT be watched.
|
||||
* Uses chokidar's provided `stats` to decide directory-vs-file so a dir-only
|
||||
* rule like `build/` matches, without an extra `statSync` per path.
|
||||
* True for any directory that should NOT be watched (used while building the
|
||||
* Linux per-directory watch tree). Tests the directory form of the path so a
|
||||
* dir-only ignore rule like `build/` matches.
|
||||
*/
|
||||
private shouldIgnore(testPath: string, stats?: Stats): boolean {
|
||||
const rel = normalizePath(path.relative(this.projectRoot, testPath));
|
||||
private shouldIgnoreDir(dirPath: string): boolean {
|
||||
const rel = normalizePath(path.relative(this.projectRoot, dirPath));
|
||||
if (!rel || rel === '.' || rel.startsWith('..')) return false; // root / outside
|
||||
if (this.isAlwaysIgnored(rel)) return true;
|
||||
if (!this.ignoreMatcher) return false;
|
||||
if (stats) {
|
||||
return this.ignoreMatcher.ignores(stats.isDirectory() ? rel + '/' : rel);
|
||||
}
|
||||
// Stats unknown: test both forms so a directory match isn't missed.
|
||||
return this.ignoreMatcher.ignores(rel) || this.ignoreMatcher.ignores(rel + '/');
|
||||
return this.ignoreMatcher.ignores(rel + '/');
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -276,37 +455,61 @@ export class FileWatcher {
|
||||
this.debounceTimer = null;
|
||||
}
|
||||
|
||||
if (this.watcher) {
|
||||
this.watcher.close();
|
||||
this.watcher = null;
|
||||
if (this.recursiveWatcher) {
|
||||
try {
|
||||
this.recursiveWatcher.close();
|
||||
} catch {
|
||||
/* already closed */
|
||||
}
|
||||
this.recursiveWatcher = null;
|
||||
}
|
||||
for (const w of this.dirWatchers.values()) {
|
||||
try {
|
||||
w.close();
|
||||
} catch {
|
||||
/* already closed */
|
||||
}
|
||||
}
|
||||
this.dirWatchers.clear();
|
||||
this.dirCapWarned = false;
|
||||
this.inert = false;
|
||||
|
||||
this.pendingFiles.clear();
|
||||
this.chokidarReady = false;
|
||||
this.ready = false;
|
||||
this.ignoreMatcher = null;
|
||||
if (IS_TEST_RUNTIME) liveWatchersForTests.delete(this.projectRoot);
|
||||
logDebug('File watcher stopped');
|
||||
}
|
||||
|
||||
/**
|
||||
* @internal Test-only: feed a synthetic project-relative change through the
|
||||
* same filter → pendingFiles → debounced-sync path a real fs.watch event
|
||||
* takes. Lets the watcher / staleness-banner suites stay deterministic
|
||||
* instead of racing on OS watch-delivery latency. See
|
||||
* {@link __emitWatchEventForTests}.
|
||||
*/
|
||||
ingestEventForTests(relPath: string): void {
|
||||
this.handleChange(normalizePath(relPath));
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the watcher is currently active.
|
||||
*/
|
||||
isActive(): boolean {
|
||||
return this.watcher !== null && !this.stopped;
|
||||
return (this.recursiveWatcher !== null || this.dirWatchers.size > 0 || this.inert) && !this.stopped;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves once chokidar has fired its `ready` event (or immediately if
|
||||
* it has already done so). Useful for tests that need a deterministic
|
||||
* boundary before asserting on `pendingFiles` — guessing a sleep duration
|
||||
* is flaky under load because chokidar can take longer than expected to
|
||||
* finish its initial crawl on slow filesystems / parallel test runs.
|
||||
* Resolves once the watch set has been installed (or immediately if it
|
||||
* already has). Useful for tests that need a deterministic boundary before
|
||||
* asserting on `pendingFiles`.
|
||||
*
|
||||
* Production callers don't need this: `pendingFiles` is read continuously,
|
||||
* the staleness banner is always correct (empty or populated), and the
|
||||
* initial-scan window is a small one-time startup cost.
|
||||
* the staleness banner is always correct (empty or populated), and there is
|
||||
* no asynchronous initial-scan window with `fs.watch`.
|
||||
*/
|
||||
waitUntilReady(timeoutMs = 10000): Promise<void> {
|
||||
if (this.chokidarReady) return Promise.resolve();
|
||||
if (this.ready) return Promise.resolve();
|
||||
return new Promise((resolve, reject) => {
|
||||
const t = setTimeout(() => {
|
||||
const idx = this.readyWaiters.indexOf(handler);
|
||||
@@ -419,3 +622,18 @@ export class FileWatcher {
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Test-only: synthesize a source-file change for the live watcher running at
|
||||
* `projectRoot`, exercising the real filter → pendingFiles → debounced-sync
|
||||
* logic without depending on fs.watch delivery timing (which races under
|
||||
* parallel vitest). `relPath` is project-relative POSIX (e.g. "src/foo.ts").
|
||||
* Returns false if no live watcher is registered for that root (e.g. outside a
|
||||
* test runtime, where the registry is intentionally not populated).
|
||||
*/
|
||||
export function __emitWatchEventForTests(projectRoot: string, relPath: string): boolean {
|
||||
const w = liveWatchersForTests.get(projectRoot);
|
||||
if (!w) return false;
|
||||
w.ingestEventForTests(relPath);
|
||||
return true;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user