Fixes #1590. ## What was wrong The live file watcher built its scope matcher — built-in defaults + `.gitignore` + the `codegraph.json` `exclude`/`include` rules — once in `start()` and kept it for the watcher's lifetime. The MCP server is long-lived, so a `codegraph.json` created or edited after it started was invisible to the watcher, while `codegraph sync` (a fresh process with a fresh matcher) honoured it immediately. From the user's side: the CLI removed a newly excluded file, and the daemon re-indexed it a few seconds later, which reads as "`exclude` doesn't work". As the report points out, `extensions` on the very same config file *was* read live (its loader is mtime-cached), so two fields of one file behaved differently. There was a second half to it. The watcher's scoped fast path hands the exact edited paths to sync, and that path stat'ed and re-parsed them without consulting the scope matcher at all — so the stale view of scope leaked straight into the index. ## What this does **Watcher — rebuild on a scope change, then reconcile in full.** An event for the root `codegraph.json` or `.gitignore` rebuilds the matcher, marks the next sync as a full reconcile, and schedules it. A scope change has no per-file events: newly excluded files must be *removed* from the index and newly included ones *added*, and only the scan-diff (which builds its own fresh matcher) knows which those are. Two ordering details are deliberate: - the two root files are checked *before* the matcher is consulted, so a user pattern that happens to cover them (`*.json`, `.*`) can't hide their own edits; - a nested `.gitignore` (an embedded child repo's own rules, or a subdirectory rule the git-backed scan honours) is checked *after* the matcher, so the thousands of package-local `.gitignore`s an `npm install` writes under an ignored `node_modules/` can never trigger a rebuild storm. Rebuilding runs embedded-repo discovery (one `git ls-files`), which is fine per config edit and never happens per event. Replacing the field serves both watch strategies: the recursive handler and the per-directory `shouldIgnoreDir` walk read it on every call. **Scoped sync — re-check the paths it was handed.** The orchestrator now runs scoped paths through the same scope matcher and source-extension gate the full walk applies. An out-of-scope path is treated as absent: removed if tracked, never parsed on trust. The matcher is memoized on the mtimes of the two root files it derives from (two `stat`s per sync while nothing changed), so the scoped path keeps skipping O(repo) work — paying embedded-repo discovery per sync would defeat its whole point. ## Tests - `watcher.test.ts` — a `codegraph.json` edit schedules a full sync, after which an edit inside the newly excluded tree is dropped by the live matcher (not pending, no sync) while an in-scope edit still syncs scoped; a root `.gitignore` edit behaves the same; a nested `.gitignore` forces a full sync; a `.gitignore` under `node_modules/` schedules nothing; dropping the exclude again readmits the tree. - `sync.test.ts` — end-to-end through `CodeGraph`: a scoped sync of a path that `codegraph.json` now excludes removes it (`filesRemoved: 1`, nothing parsed — the symbol added to the file never appears), stays out on a repeat, and is re-added through the same scoped path once the exclude is dropped. - All five new tests fail on `main`; the `node_modules` guard passes both ways as expected. - Full suite: 189 files, 3184 passed / 9 skipped. - CLI half of the issue's repro (init with `exclude`, edit the config + the file, `codegraph sync`): the newly excluded file is removed and its new symbol never enters the index. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01LxZj6W6Y1SHXwvpT3uwJpK
This commit is contained in:
Colby Mchenry
+54
-2
@@ -27,7 +27,7 @@ import { StoreWriter, StoreBundle, finalizeStoreBundle } from './store-writer';
|
||||
import { materializeKernelResult } from './kernel';
|
||||
import { detectGeneratedFile } from './generated-detection';
|
||||
import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes } from './grammars';
|
||||
import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns } from '../project-config';
|
||||
import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns, PROJECT_CONFIG_FILENAME } from '../project-config';
|
||||
import { isCodeGraphDataDir } from '../directory';
|
||||
import { logDebug, logWarn } from '../errors';
|
||||
import { validatePathWithinRoot, normalizePath } from '../utils';
|
||||
@@ -1448,12 +1448,48 @@ export class ExtractionOrchestrator {
|
||||
* hasn't run yet so single-file re-index paths can detect on the spot.
|
||||
*/
|
||||
private detectedFrameworkNames: string[] | null = null;
|
||||
/**
|
||||
* Scope matcher for SCOPED syncs, memoized on the mtimes of the two root
|
||||
* files it is derived from (`codegraph.json`, `.gitignore`). See
|
||||
* {@link scopedSyncMatcher}.
|
||||
*/
|
||||
private scopedMatcher: { key: string; matcher: ScopeIgnore } | null = null;
|
||||
|
||||
constructor(rootDir: string, queries: QueryBuilder) {
|
||||
this.rootDir = rootDir;
|
||||
this.queries = queries;
|
||||
}
|
||||
|
||||
/**
|
||||
* The scope matcher a scoped sync applies to the paths it was handed — the
|
||||
* same `buildScopeIgnore` the full scan uses, so an explicitly-passed path
|
||||
* that is OUT of scope (a user `exclude` in `codegraph.json`, a `.gitignore`
|
||||
* rule, a built-in default) is treated exactly as the full walk would treat
|
||||
* it: absent, hence removed if tracked, never parsed (#1590).
|
||||
*
|
||||
* Memoized on the root config + root `.gitignore` mtimes: building the
|
||||
* matcher runs embedded-repo discovery (`git ls-files`), which would defeat
|
||||
* the scoped path's whole point (skipping O(repo) work) if paid per sync.
|
||||
* Two `stat`s per sync while nothing changed. An embedded repo created
|
||||
* between config edits joins the scoped matcher on the next full sync, the
|
||||
* same lifecycle the watcher's own matcher already has.
|
||||
*/
|
||||
private scopedSyncMatcher(): ScopeIgnore {
|
||||
const key = [PROJECT_CONFIG_FILENAME, '.gitignore']
|
||||
.map((name) => {
|
||||
try {
|
||||
return String(fs.statSync(path.join(this.rootDir, name)).mtimeMs);
|
||||
} catch {
|
||||
return '-';
|
||||
}
|
||||
})
|
||||
.join('|');
|
||||
if (this.scopedMatcher && this.scopedMatcher.key === key) return this.scopedMatcher.matcher;
|
||||
const matcher = buildScopeIgnore(this.rootDir);
|
||||
this.scopedMatcher = { key, matcher };
|
||||
return matcher;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a filesystem-backed ResolutionContext sufficient for framework
|
||||
* detection. Graph-query methods (getNodesByName etc.) return empty because
|
||||
@@ -2700,7 +2736,23 @@ export class ExtractionOrchestrator {
|
||||
// reads `filesChecked === 0 && durationMs === 0` as the
|
||||
// lock-unavailable signature (#449).
|
||||
const unique = [...new Set(scopedPaths)];
|
||||
currentFiles = unique.filter((p) => fs.existsSync(path.join(this.rootDir, p)));
|
||||
// A scoped path is "present" only if it exists AND is in scope — the
|
||||
// same two gates the full walk applies (source extension, scope
|
||||
// matcher). Without the scope gate a caller's stale view of scope
|
||||
// leaked straight into the index: the watcher re-parsed a file the
|
||||
// user had just excluded in `codegraph.json` while `codegraph sync`
|
||||
// removed it (#1590). Out-of-scope paths fall out of `currentFiles`,
|
||||
// so a tracked one takes the removal branch below, exactly as a full
|
||||
// sync would treat it. (`include`-forced paths pass: ScopeIgnore
|
||||
// applies the include precedence itself.)
|
||||
const scope = this.scopedSyncMatcher();
|
||||
const overrides = loadExtensionOverrides(this.rootDir);
|
||||
currentFiles = unique.filter(
|
||||
(p) =>
|
||||
isSourceFile(p, overrides) &&
|
||||
!scope.ignores(p) &&
|
||||
fs.existsSync(path.join(this.rootDir, p))
|
||||
);
|
||||
trackedFiles = [];
|
||||
for (const p of unique) {
|
||||
const rec = this.queries.getFileByPath(p);
|
||||
|
||||
+53
-6
@@ -34,7 +34,7 @@
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { isSourceFile, buildScopeIgnore, type ScopeIgnore } from '../extraction';
|
||||
import { loadExtensionOverrides } from '../project-config';
|
||||
import { loadExtensionOverrides, PROJECT_CONFIG_FILENAME } from '../project-config';
|
||||
import { logDebug, logWarn } from '../errors';
|
||||
import { normalizePath } from '../utils';
|
||||
import { isCodeGraphDataDir } from '../directory';
|
||||
@@ -328,11 +328,13 @@ export class FileWatcher {
|
||||
* deterministically gate on watcher readiness.
|
||||
*/
|
||||
private readyWaiters: Array<() => void> = [];
|
||||
// The shared scope matcher (built-in defaults + project .gitignore, with
|
||||
// embedded child repos matched by their OWN rules — #514), built once at
|
||||
// start(). Same source of truth the indexer uses, so watcher scope can
|
||||
// never diverge from index scope. An embedded repo created after start()
|
||||
// joins the scope on the next watcher restart / re-index.
|
||||
// The shared scope matcher (built-in defaults + project .gitignore + the
|
||||
// `codegraph.json` exclude/include rules, with embedded child repos matched
|
||||
// by their OWN rules — #514), built at start() and REBUILT whenever one of
|
||||
// the files it is derived from changes (see `refreshScope`, #1590). Same
|
||||
// source of truth the indexer uses, so watcher scope can never diverge from
|
||||
// index scope. An embedded repo created after start() joins the scope on
|
||||
// the next scope refresh / watcher restart / re-index.
|
||||
private ignoreMatcher: ScopeIgnore | null = null;
|
||||
|
||||
private readonly projectRoot: string;
|
||||
@@ -573,7 +575,24 @@ export class FileWatcher {
|
||||
private handleChange(rel: string): void {
|
||||
if (!rel || rel === '.' || rel.startsWith('..')) return;
|
||||
if (this.isAlwaysIgnored(rel)) return;
|
||||
// The two root files the scope matcher is derived from are handled BEFORE
|
||||
// the matcher is consulted: a user `exclude` pattern that happens to cover
|
||||
// them (`*.json`, `.*`) must not be able to hide their own edits (#1590).
|
||||
if (rel === PROJECT_CONFIG_FILENAME || rel === '.gitignore') {
|
||||
this.refreshScope(rel);
|
||||
return;
|
||||
}
|
||||
if (this.ignoreMatcher && this.ignoreMatcher.ignores(rel)) return;
|
||||
// A nested `.gitignore` (an embedded child repo's own rules, #514, or a
|
||||
// subdirectory rule the git-backed full scan honors) is only a scope
|
||||
// change when it sits INSIDE the current scope — checked after the matcher
|
||||
// on purpose, so the thousands of package-local `.gitignore`s an
|
||||
// `npm install` writes under an ignored `node_modules/` never trigger a
|
||||
// rebuild storm.
|
||||
if (rel.endsWith('/.gitignore')) {
|
||||
this.refreshScope(rel);
|
||||
return;
|
||||
}
|
||||
if (!isSourceFile(rel, loadExtensionOverrides(this.projectRoot))) {
|
||||
this.maybeScheduleForRemovedDir(rel);
|
||||
return;
|
||||
@@ -591,6 +610,34 @@ export class FileWatcher {
|
||||
this.scheduleSync();
|
||||
}
|
||||
|
||||
/**
|
||||
* A scope-defining file changed (`codegraph.json`, a `.gitignore`): rebuild
|
||||
* the ignore matcher and make the next sync a FULL reconcile (#1590).
|
||||
*
|
||||
* The matcher used to be built once in `start()` and kept for the watcher's
|
||||
* lifetime — in a long-lived MCP daemon that meant a `codegraph.json`
|
||||
* created or edited after startup was invisible to the live watcher, while
|
||||
* `codegraph sync` (a fresh process) honoured it immediately: the CLI
|
||||
* removed a newly excluded file and the watcher re-added it seconds later.
|
||||
* `loadExtensionOverrides()` on the same filter line was already read live
|
||||
* (mtime-cached), so two fields of the same config file disagreed.
|
||||
*
|
||||
* Rebuilding costs one `git ls-files` pass (embedded-repo discovery), which
|
||||
* is fine per config edit — never per event. Replacing the field is enough
|
||||
* for both strategies: the recursive handler and the per-directory
|
||||
* `shouldIgnoreDir` walk read `this.ignoreMatcher` on every call. The full
|
||||
* scan is required because a scope change has no per-file events: newly
|
||||
* excluded files must be REMOVED from the index and newly included ones
|
||||
* added, and only the scan-diff (which builds its own fresh matcher) knows
|
||||
* which those are.
|
||||
*/
|
||||
private refreshScope(rel: string): void {
|
||||
logDebug('Scope config changed; rebuilding watcher scope', { file: rel });
|
||||
this.ignoreMatcher = buildScopeIgnore(this.projectRoot);
|
||||
this.needsFullScan = true;
|
||||
this.scheduleSync();
|
||||
}
|
||||
|
||||
/**
|
||||
* A deleted DIRECTORY arrives as one event on the directory's own path —
|
||||
* no source extension, so the source-file filter drops it, and the files
|
||||
|
||||
Reference in New Issue
Block a user