feat(mcp): per-file staleness banner + tunable watcher debounce (#403) (#428)

Two coupled changes addressing the issue's underlying ask — "how does the
agent know when the index lags" — without resorting to a static wait.

Per-file staleness banner
-------------------------
FileWatcher now tracks per-path `pendingFiles` (path, firstSeenMs,
lastSeenMs, indexing) — events since the last successful sync, cleared
only after a sync whose `syncStartedMs >= lastSeenMs` commits. Chokidar
initial-scan events are gated behind a `ready` flag (with `waitUntilReady()`
exposed so tests can deterministically wait through it) so a fresh startup
doesn't falsely flag every existing file as pending.

ToolHandler now wraps every code-returning response (search, context,
callers, callees, impact, trace, explore, node, files) with
`withStalenessNotice`: intersects "files referenced in the response" with
`getPendingFiles()` and emits a hybrid signal —

  * banner at the top for files referenced AND pending (with edit age +
    indexing/pending-sync state, telling the agent to Read those specific
    files directly; the rest of the response stays fresh and codegraph
    stays authoritative for it),
  * compact footer for pending files elsewhere in the project not
    referenced above (capped at 5).

Cost is one boolean check + N substring matches when pending; zero
allocation when idle. `codegraph_status` surfaces the same data as a
first-class `### Pending sync:` section so the agent can ask "is the index
caught up?" in one call.

Cross-project quirk: when an agent passes `projectPath` matching the
default session's project, the staleness wrapper switches from the cached
cross-project CodeGraph (no watcher) to the default one (with watcher) so
the signal still fires. Same fix applied to `handleStatus`.

CODEGRAPH_WATCH_DEBOUNCE_MS
---------------------------
MCP `serve --mcp` now reads `CODEGRAPH_WATCH_DEBOUNCE_MS` and forwards it
to `cg.watch({ debounceMs })`. Clamped to [100ms, 60s]; out-of-range or
non-numeric values fall back to the FileWatcher default (2000ms). Active
value is logged to stderr on watcher startup so it's discoverable. The
docs in `server-instructions.ts`, `installer/instructions-template.ts`,
and `.cursor/rules/codegraph.mdc` no longer claim "~500ms"; they now
describe the banner mechanism instead — since per-file staleness replaces
the "wait N ms" guidance entirely, the docs become accurate at any
debounce value.

Validation
----------
* 847 unit/integration tests pass (added 15 new ones — pending-file
  tracking, banner/footer routing, status section, env-var parsing).
* Direct MCP probe through a real `codegraph serve --mcp` process: edit a
  file, query within the debounce window, banner fires naming the
  edited file with edit-age.
* Real Claude TUI session via `scripts/agent-eval/itrun.sh` with
  `CODEGRAPH_WATCH_DEBOUNCE_MS=10000`: agent edits `math.ts`, calls
  `codegraph_explore`, reads the banner, **and discloses it unprompted in
  its final reply**: "note: symbol index is mid-sync for the new `divide`,
  but the source it returned is verbatim from disk."

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

src/sync/watcher.ts

@@ -45,6 +45,26 @@ export interface WatchOptions {
   onSyncError?: (error: Error) => void;
 }
 
+/**
+ * Per-file pending entry — tracks a source file the watcher saw an event for
+ * but hasn't yet synced into the index. Exposed via {@link FileWatcher.getPendingFiles}
+ * so MCP tool responses can mark stale results without forcing a wait.
+ */
+export interface PendingFile {
+  /** Project-relative POSIX path (e.g. "src/foo.ts"). */
+  path: string;
+  /** Wall-clock ms at the first event we saw for this path since the last sync. */
+  firstSeenMs: number;
+  /** Wall-clock ms at the most recent event we saw for this path. */
+  lastSeenMs: number;
+  /**
+   * True when a sync is currently in flight that began AFTER this file's most
+   * recent event — i.e. the next successful sync will pick it up. False when
+   * the file is still in the debounce window (no sync running yet).
+   */
+  indexing: boolean;
+}
+
 /**
  * FileWatcher monitors a project directory for changes and triggers
  * debounced sync operations via a provided callback.
@@ -55,13 +75,43 @@ export interface WatchOptions {
  * - Debounced to avoid thrashing on rapid saves
  * - Filters to supported source files by extension
  * - Ignores .codegraph/ and .git/ regardless of .gitignore
+ * - Tracks per-file pending state so MCP tools can flag stale results
+ *   without blocking on a sync (issue #403)
  */
 export class FileWatcher {
   private watcher: FSWatcher | null = null;
   private debounceTimer: ReturnType<typeof setTimeout> | null = null;
-  private hasChanges = false;
+  /**
+   * Files seen by the watcher since the last successful sync — populated on
+   * every chokidar event, cleared at the start of a sync, and re-populated by
+   * events that arrive mid-sync (or restored on sync failure). Keyed by the
+   * same project-relative POSIX path the rest of the codebase uses, so a
+   * caller can intersect tool-response file paths against this map cheaply.
+   */
+  private pendingFiles = new Map<string, { firstSeenMs: number; lastSeenMs: number }>();
+  /**
+   * Wall-clock ms at which the in-flight sync began. Combined with
+   * {@link pendingFiles}'s `lastSeenMs`, this distinguishes "still in the
+   * debounce window" (lastSeen > syncStarted, sync hasn't started yet for
+   * this edit) from "currently being indexed" (lastSeen <= syncStarted).
+   */
+  private syncStartedMs = 0;
   private syncing = false;
   private stopped = false;
+  /**
+   * False until chokidar fires its `ready` event. Gates `pendingFiles`
+   * insertion so the initial crawl's `add` events (one per pre-existing
+   * source file) don't pollute the per-file staleness signal. The events
+   * still flow into `scheduleSync()` to preserve the previous "initial
+   * scan triggers a reconciling sync" behavior.
+   */
+  private chokidarReady = false;
+  /**
+   * Callbacks that resolve when chokidar fires `ready`. Used by tests (and
+   * any production caller that cares about a clean baseline) to deterministically
+   * gate on the end of the initial scan instead of guessing at a sleep duration.
+   */
+  private readyWaiters: Array<() => void> = [];
   // The shared ignore matcher (built-in defaults + project .gitignore), built
   // once at start(). Same source of truth the indexer uses, so watcher scope
   // can never diverge from index scope.
@@ -116,6 +166,26 @@ export class FileWatcher {
         ignored: (testPath: string, stats?: Stats) => this.shouldIgnore(testPath, stats),
       });
 
+      // Chokidar emits `add` for every pre-existing source file during its
+      // initial scan. Those events should still trigger the post-startup
+      // reconciling sync (preserving prior behavior), but they must NOT land
+      // in pendingFiles — otherwise every file in the project shows up as
+      // "edited but not indexed" on startup, which is the opposite of the
+      // signal #403 is supposed to provide. Flip the flag on chokidar's
+      // `ready` event; from then on, real edits populate pendingFiles.
+      //
+      // We also clear `pendingFiles` here as defense-in-depth: chokidar can
+      // emit late initial-scan `add` events via setImmediate AFTER the
+      // `ready` callback runs (observed under test-parallelism load).
+      // Clearing once at ready guarantees a clean baseline; real subsequent
+      // edits repopulate the set normally.
+      this.watcher.on('ready', () => {
+        this.chokidarReady = true;
+        this.pendingFiles.clear();
+        for (const cb of this.readyWaiters) cb();
+        this.readyWaiters.length = 0;
+      });
+
       // chokidar emits 'all' for every event type; we only sync source files.
       this.watcher.on('all', (_event: string, filePath: string) => {
         if (this.stopped) return;
@@ -128,7 +198,17 @@ export class FileWatcher {
         if (!isSourceFile(normalized)) return;
 
         logDebug('File change detected', { file: normalized });
-        this.hasChanges = true;
+        // Only track events from after chokidar's initial scan as pending
+        // edits — pre-existing files on disk are already represented by
+        // (or about to be reconciled by) the index, not a user edit.
+        if (this.chokidarReady) {
+          const now = Date.now();
+          const existing = this.pendingFiles.get(normalized);
+          this.pendingFiles.set(normalized, {
+            firstSeenMs: existing?.firstSeenMs ?? now,
+            lastSeenMs: now,
+          });
+        }
         this.scheduleSync();
       });
 
@@ -187,7 +267,8 @@ export class FileWatcher {
       this.watcher = null;
     }
 
-    this.hasChanges = false;
+    this.pendingFiles.clear();
+    this.chokidarReady = false;
     this.ignoreMatcher = null;
     logDebug('File watcher stopped');
   }
@@ -199,6 +280,30 @@ export class FileWatcher {
     return this.watcher !== null && !this.stopped;
   }
 
+  /**
+   * Resolves once chokidar has fired its `ready` event (or immediately if
+   * it has already done so). Useful for tests that need a deterministic
+   * boundary before asserting on `pendingFiles` — guessing a sleep duration
+   * is flaky under load because chokidar can take longer than expected to
+   * finish its initial crawl on slow filesystems / parallel test runs.
+   *
+   * Production callers don't need this: `pendingFiles` is read continuously,
+   * the staleness banner is always correct (empty or populated), and the
+   * initial-scan window is a small one-time startup cost.
+   */
+  waitUntilReady(timeoutMs = 10000): Promise<void> {
+    if (this.chokidarReady) return Promise.resolve();
+    return new Promise((resolve, reject) => {
+      const t = setTimeout(() => {
+        const idx = this.readyWaiters.indexOf(handler);
+        if (idx >= 0) this.readyWaiters.splice(idx, 1);
+        reject(new Error(`FileWatcher.waitUntilReady timed out after ${timeoutMs}ms`));
+      }, timeoutMs);
+      const handler = () => { clearTimeout(t); resolve(); };
+      this.readyWaiters.push(handler);
+    });
+  }
+
   /**
    * Schedule a debounced sync.
    */
@@ -214,28 +319,80 @@ export class FileWatcher {
 
   /**
    * Flush pending changes by running sync.
+   *
+   * pendingFiles is NOT cleared at the start of sync — entries are removed
+   * only after sync commits successfully, and only for entries whose
+   * lastSeenMs <= syncStartedMs. That way, a query that arrives mid-sync
+   * still sees the affected files marked stale (the DB hasn't been updated
+   * yet), and an event that lands mid-sync persists into the follow-up.
+   *
+   * On sync failure pendingFiles is left untouched — every edit is still
+   * unindexed, and the rescheduled sync will absorb the same set next time.
    */
   private async flush(): Promise<void> {
     // If already syncing, the post-sync check will re-trigger
     if (this.syncing || this.stopped) return;
 
-    this.hasChanges = false;
+    this.syncStartedMs = Date.now();
     this.syncing = true;
 
     try {
       const result = await this.syncFn();
+      // Remove entries whose most recent event predates this sync — those
+      // edits are now in the DB. Entries with lastSeenMs > syncStartedMs
+      // arrived mid-sync; whether the in-flight sync captured them depends
+      // on when sync read that file, so we keep them as pending and let
+      // the follow-up sync handle them. We prefer false positives ("shown
+      // stale, actually fresh" → at worst one extra Read) over false
+      // negatives ("shown fresh, actually stale" → misleads the agent).
+      for (const [filePath, info] of this.pendingFiles) {
+        if (info.lastSeenMs <= this.syncStartedMs) {
+          this.pendingFiles.delete(filePath);
+        }
+      }
       this.onSyncComplete?.(result);
     } catch (err) {
       const error = err instanceof Error ? err : new Error(String(err));
       logWarn('Watch sync failed', { error: error.message });
+      // Failure: leave pendingFiles untouched. Every edit it tracks is
+      // still unindexed; the rescheduled sync sees the same set.
       this.onSyncError?.(error);
     } finally {
       this.syncing = false;
 
-      // If new changes arrived during sync, schedule another
-      if (this.hasChanges && !this.stopped) {
+      // If pending files remain (mid-sync events, or this sync failed),
+      // schedule another pass.
+      if (this.pendingFiles.size > 0 && !this.stopped) {
         this.scheduleSync();
       }
     }
   }
+
+  /**
+   * Snapshot of files seen by the watcher since the last successful sync.
+   *
+   * Used by MCP tool responses to mark stale results without blocking on a
+   * sync: a tool that returns a hit in `src/foo.ts` while `src/foo.ts` is in
+   * this list tells the agent "Read this file directly, the index lags."
+   *
+   * `indexing` is true when a sync is currently in flight whose start time is
+   * AFTER this file's most recent event — i.e. that sync will absorb the
+   * edit. False means the file is still inside the debounce window and no
+   * sync has started yet (a follow-up call a few hundred ms later may show
+   * `indexing: true` or the file may have left the list entirely).
+   *
+   * Cheap: O(pendingFiles.size), no I/O, no locks.
+   */
+  getPendingFiles(): PendingFile[] {
+    const result: PendingFile[] = [];
+    for (const [filePath, info] of this.pendingFiles) {
+      result.push({
+        path: filePath,
+        firstSeenMs: info.firstSeenMs,
+        lastSeenMs: info.lastSeenMs,
+        indexing: this.syncing && this.syncStartedMs >= info.lastSeenMs,
+      });
+    }
+    return result;
+  }
 }