feat(config): add codegraph.json "deprioritize" for ranking-only path down-weighting (#982) (#1463)

* feat(config): add codegraph.json "deprioritize" for ranking-only path down-weighting matchesNonProductionDir hardcodes example/sample/fixture/benchmark/demo, so a peripheral tree only the project knows about — optional-skills/, scripts/ — gets no de-prioritization. When helpers there carry generic symbol names, an exact name match hands them a large bonus and they crowd out the product code that answers the query (#982). deprioritize is the RANKING counterpart to exclude: those paths stay indexed and findable, they just stop outranking first-party code. It is deliberately distinct from the corpus-frequency discount, which keys on a name being common and is near-inert on #982's own repro where only two symbols are named usage. The -15 path penalty alone is not enough, and measuring showed why: on that repro a usage() helper sits at 74.8 against 51.2 for the top product symbol, so -15 lands at 59.8 and still leads. The path penalty is additive and the name bonus it must counter is additive and larger. A de-prioritized path is saying its symbol NAMES are not the answer, so the exact-name bonus is damped to 0.25x there as well — damped, not zeroed, so the tree still ranks when it genuinely is what you asked for. Refs #982 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SKXAJMrVrdHS5Uco6ABtky * fix(config): read deprioritize lazily and apply it in explore too Review of the first cut found two real defects. The matcher was built once in wireLayers(), which runs only from the constructor and from reopenIfReplaced(). The MCP server keeps one CodeGraph per project root alive for its whole lifetime, so editing codegraph.json appeared to do nothing until the process restarted -- exclude and include do not behave that way. The predicate now reads loadDeprioritizePatterns() per call (mtime-cached, one stat) and memoizes the compiled matcher on the pattern array's identity. A regression test writes the config after opening the project and fails on the old code. Explore passed no matcher to scorePathRelevance at either of its two call sites, so the setting only half-applied -- and #982's reproduction rows B, C and D are all codegraph explore, which made this the surface the issue actually reports on. Both sites now pass it. Explore's hard early-continue filters and its non-production budget cap are deliberately NOT joined: those REMOVE content, and deprioritize is a ranking lever by definition. README narrowed accordingly -- it previously claimed this extends the built-in list, which overstated it. Also from review: scorePathRelevance takes a boolean rather than a predicate (the caller already evaluated it, and it was being invoked twice per result), the predicate body is exception-guarded so a bad path can never take a search down, the misplaced const moved out from between imports, two vacuous test assertions tightened, and tests added for the single-penalty invariant, the deliberate isTestQuery asymmetry, and a query that genuinely targets the de-prioritized tree. Refs #982 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SKXAJMrVrdHS5Uco6ABtky * fix(search): derive the deprioritize name-bonus damping instead of picking it (#982) The 0.25 scale was a guess. On a 62k-node django index it measurably breaks the "discount, don't erase" rule the lever is built on: exact-name queries for symbols that live only in the de-prioritized tree (child, parent, method) fall behind mere prefix matches (children, all_parents, method_decorator). The prefix arm of nameMatchBonus tops out below 40, and a de-prioritized node also takes the -15 path penalty, so 80 * SCALE - 15 > 40 is the bound that keeps a damped exact match ahead of a prefix match at any corpus shape. 0.75 clears it; crowd-out removal is nearly identical to 0.5 (39 vs 40 of 88 peripheral top-10 slots cleared on django), so the deeper discount bought almost nothing and cost the invariant. Two tests pin the bound, including one that fails at the old 0.25. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-22 11:55:14 -05:00
parent bc894802ff
commit 1d9de88ef1
7 changed files with 481 additions and 13 deletions
@@ -188,6 +188,7 @@ export { LOW_CONFIDENCE_MARKER } from './markers';
 export class ContextBuilder {
  private projectRoot: string;
  private queries: QueryBuilder;
+
  private traverser: GraphTraverser;

  constructor(
@@ -200,6 +201,21 @@ export class ContextBuilder {
    this.traverser = traverser;
  }

+  /**
+   * Whether the project's `codegraph.json` `deprioritize` patterns cover this
+   * path (#982). Explore ranks through its own path scorer as well as through
+   * `searchNodes`, so the lever has to be applied here too or the setting would
+   * only half-work — and explore is the surface #982 actually reports on.
+   *
+   * Only the -15 relevance penalty is shared. Explore's hard `continue` filters
+   * and its non-production budget cap are deliberately NOT joined: those REMOVE
+   * content, and `deprioritize` is a ranking lever by definition — `exclude` is
+   * the lever for taking things out of reach.
+   */
+  private isDeprioritized(filePath: string): boolean {
+    return this.queries.getDeprioritizedPathMatcher()?.(filePath) ?? false;
+  }
+
  /**
   * Build context for a task
   *
@@ -837,7 +853,12 @@ export class ContextBuilder {
          if (searchIdSet.has(r.node.id)) continue;
          if (isTestFile(r.node.filePath) && !isTestQuery) continue;

-          const pathScore = scorePathRelevance(r.node.filePath, query);
+          const pathScore = scorePathRelevance(
+            r.node.filePath,
+            query,
+            undefined,
+            this.isDeprioritized(r.node.filePath),
+          );
          const brevityBonus = Math.max(0, 6 - (name.length - titleCased.length) / 4);
          termCandidates.push({ node: r.node, score: 8 + brevityBonus + pathScore });
        }
@@ -924,7 +945,12 @@ export class ContextBuilder {
        const compoundResults: SearchResult[] = [];
        for (const [, entry] of compoundTermMap) {
          if (entry.terms.size >= 2) {
-            const pathScore = scorePathRelevance(entry.node.filePath, query);
+            const pathScore = scorePathRelevance(
+              entry.node.filePath,
+              query,
+              undefined,
+              this.isDeprioritized(entry.node.filePath),
+            );
            const brevityBonus = Math.max(0, 6 - entry.node.name.length / 8);
            compoundResults.push({
              node: entry.node,
@@ -55,6 +55,23 @@ function isLowValueFile(filePath: string, generated?: ReadonlySet<string>): bool

 const SQLITE_PARAM_CHUNK_SIZE = 500;

+/**
+ * How much of the exact-name bonus a `deprioritize`d path keeps (#982). Damped
+ * rather than zeroed: a query that genuinely targets that tree must still rank
+ * it, the same "discount, don't erase" rule the path penalty follows.
+ *
+ * Derived rather than picked. `nameMatchBonus`'s prefix arm tops out below
+ * `10 + 30 = 40`, and a de-prioritized node also takes the -15 path penalty, so
+ * `80 * SCALE - 15 > 40` is what stops a damped WHOLE-QUERY exact match from
+ * losing to a mere prefix match. 0.75 clears it (45). Measured on a 62k-node
+ * django index: at 0.25 that invariant breaks in practice — `child`, `parent`
+ * and `method` lose rank 1 to `children`, `all_parents` and `method_decorator`
+ * — while crowd-out removal is almost flat between 0.75 and 0.5 (39 vs 40 of 88
+ * peripheral top-10 slots cleared), so a deeper discount buys little and costs
+ * the invariant. Pinned by a test.
+ */
+export const DEPRIORITIZED_NAME_BONUS_SCALE = 0.75;
+
 /**
 * Database row types (snake_case from SQLite)
 */
@@ -204,6 +221,7 @@ export class QueryBuilder {
  // whole project, not a symbol, so it carries no discriminative signal (#720).
  // Set once by the CodeGraph instance; empty by default (no down-weighting).
  private projectNameTokens: Set<string> = new Set();
+  private isDeprioritizedPath: ((filePath: string) => boolean) | undefined;

  // Node cache for frequently accessed nodes (LRU-style, max 1000 entries)
  private nodeCache: Map<string, Node> = new Map();
@@ -328,6 +346,21 @@ export class QueryBuilder {
    return this.projectNameTokens;
  }

+  /**
+   * Set the predicate that marks a path as de-prioritized by the project's
+   * `codegraph.json` `deprioritize` patterns (#982). Ranking-only: those paths
+   * stay indexed and findable, they just stop outranking first-party code.
+   * Called once when the project opens; undefined disables the lever.
+   */
+  setDeprioritizedPathMatcher(matcher: ((filePath: string) => boolean) | undefined): void {
+    this.isDeprioritizedPath = matcher;
+  }
+
+  /** The `deprioritize` predicate (#982), so other rankers apply the same lever. */
+  getDeprioritizedPathMatcher(): ((filePath: string) => boolean) | undefined {
+    return this.isDeprioritizedPath;
+  }
+
  // ===========================================================================
  // Node Operations
  // ===========================================================================
@@ -1295,13 +1328,24 @@ export class QueryBuilder {
    // Apply multi-signal scoring
    if (results.length > 0 && (text || query)) {
      const scoringQuery = text || query;
-      results = results.map(r => ({
-        ...r,
-        score: r.score
-          + kindBonus(r.node.kind)
-          + scorePathRelevance(r.node.filePath, scoringQuery, this.projectNameTokens)
-          + nameMatchBonus(r.node.name, scoringQuery),
-      }));
+      results = results.map(r => {
+        // A path the project de-prioritized is saying its symbol NAMES are not
+        // the answer, so the exact-name bonus has to be damped too. The -15 path
+        // penalty alone cannot do it: the bonus is additive and larger (measured
+        // on #982's repro, a `usage()` helper sat at 74.8 vs 51.2 for the top
+        // product symbol — -15 lands at 59.8, still ahead). Damped, not zeroed,
+        // so the tree stays findable when it genuinely is what you asked for.
+        // Evaluated once and reused: the predicate stats the config file.
+        const deprioritized = this.isDeprioritizedPath?.(r.node.filePath) ?? false;
+        const nameBonus = nameMatchBonus(r.node.name, scoringQuery);
+        return {
+          ...r,
+          score: r.score
+            + kindBonus(r.node.kind)
+            + scorePathRelevance(r.node.filePath, scoringQuery, this.projectNameTokens, deprioritized)
+            + (deprioritized ? Math.round(nameBonus * DEPRIORITIZED_NAME_BONUS_SCALE) : nameBonus),
+        };
+      });
      results.sort((a, b) => b.score - a.score);
      // Trim to requested limit after rescoring
      if (results.length > limit) {
@@ -53,6 +53,8 @@ import { FileWatcher, WatchOptions, PendingFile, LockUnavailableError } from './
 import { EXTRACTION_VERSION } from './extraction/extraction-version';
 import { getCodeGraphDir } from './directory';
 import { deriveProjectNameTokens } from './search/query-utils';
+import ignore from 'ignore';
+import { loadDeprioritizePatterns } from './project-config';
 import { CodeGraphPackageVersion } from './mcp/version';
 import { extractSegmentSearchWords, segmentLookupVariants, splitIdentifierSegments } from './search/identifier-segments';
 import { createYielder } from './resolution/cooperative-yield';
@@ -186,6 +188,39 @@ export class CodeGraph {
    } catch {
      // Best-effort: ranking still works without it.
    }
+    // Down-weight the peripheral trees the project named in `codegraph.json`
+    // `deprioritize` — indexed and findable, but never outranking real code
+    // (#982). Ranking-only, so a bad pattern costs relevance, never recall.
+    //
+    // Read LAZILY, not once here: `wireLayers` runs from the constructor and
+    // from `reopenIfReplaced`, so a matcher built here would freeze at whatever
+    // the config said when the project opened. The MCP server caches one
+    // CodeGraph per root for its whole lifetime, so editing `codegraph.json`
+    // would appear to do nothing until the process restarted — `exclude` and
+    // `include` do not behave that way. `loadDeprioritizePatterns` is
+    // mtime-cached, so this costs one `stat`; the compiled matcher is memoized
+    // on the pattern array's identity, which the cache keeps stable.
+    let cachedPatterns: string[] | undefined;
+    let cachedMatcher: ReturnType<typeof ignore> | undefined;
+    this.queries.setDeprioritizedPathMatcher((filePath: string): boolean => {
+      try {
+        const patterns = loadDeprioritizePatterns(this.projectRoot);
+        if (patterns.length === 0) return false;
+        if (patterns !== cachedPatterns) {
+          cachedPatterns = patterns;
+          cachedMatcher = ignore().add(patterns);
+        }
+        const rel = path.isAbsolute(filePath)
+          ? path.relative(this.projectRoot, filePath)
+          : filePath;
+        if (!rel || rel.startsWith('..')) return false;
+        return cachedMatcher!.ignores(rel.split(path.sep).join('/'));
+      } catch {
+        // Ranking must never take the search down with it.
+        return false;
+      }
+    });
+
    this.orchestrator = new ExtractionOrchestrator(this.projectRoot, this.queries);
    this.resolver = createResolver(this.projectRoot, this.queries);
    this.graphManager = new GraphQueryManager(this.queries);
@@ -67,6 +67,21 @@ export interface ProjectConfig {
   * wins. Absent/empty (the default) forces nothing in.
   */
  include?: string[];
+  /**
+   * Gitignore-style patterns for paths that should still be INDEXED and
+   * findable, but must not outrank first-party code in search ranking (#982).
+   *
+   * The ranking counterpart to `exclude`: `exclude` is a recall lever (the
+   * content leaves the index entirely), this is a relevance lever (the content
+   * stays, it just stops winning). It generalizes the built-in
+   * example/sample/fixture/benchmark de-prioritization to trees only the
+   * project knows about — an `optional-skills/` or `scripts/` directory whose
+   * helpers share generic symbol names with real code. Matched against
+   * project-root-relative paths, so `"optional-skills/"`, a recursive glob, or
+   * `"tools/gen"` all work. Absent/empty (the default) de-prioritizes nothing
+   * beyond the built-ins.
+   */
+  deprioritize?: string[];
 }

 /** Parsed, validated view of a project's `codegraph.json`. */
@@ -74,6 +89,7 @@ interface ParsedConfig {
  extensions: Record<string, Language>;
  includeIgnored: string[];
  exclude: string[];
+  deprioritize: string[];
  include: string[];
 }

@@ -97,6 +113,7 @@ const EMPTY_CONFIG: ParsedConfig = Object.freeze({
  includeIgnored: Object.freeze([]) as unknown as string[],
  exclude: Object.freeze([]) as unknown as string[],
  include: Object.freeze([]) as unknown as string[],
+  deprioritize: Object.freeze([]) as unknown as string[],
 });

 /**
@@ -149,15 +166,17 @@ function parseConfig(file: string): ParsedConfig {
  const includeIgnored = extractIncludeIgnored(parsed, file);
  const exclude = extractExclude(parsed, file);
  const include = extractInclude(parsed, file);
+  const deprioritize = extractPatternList(parsed, file, 'deprioritize');
  if (
    extensions === EMPTY_EXTENSIONS &&
    includeIgnored.length === 0 &&
    exclude.length === 0 &&
-    include.length === 0
+    include.length === 0 &&
+    deprioritize.length === 0
  ) {
    return EMPTY_CONFIG;
  }
-  return { extensions, includeIgnored, exclude, include };
+  return { extensions, includeIgnored, exclude, include, deprioritize };
 }

 /**
@@ -210,6 +229,29 @@ function extractIncludeIgnored(parsed: object, file: string): string[] {
  return out;
 }

+/**
+ * Validate a gitignore-style pattern list under `key`. A non-array value or a
+ * non-string/blank entry warns-and-skips; never throws. Patterns are kept
+ * verbatim (trimmed) so they match exactly as a `.gitignore` line would.
+ */
+function extractPatternList(parsed: object, file: string, key: 'deprioritize'): string[] {
+  const raw = (parsed as ProjectConfig)[key];
+  if (raw === undefined) return [];
+  if (!Array.isArray(raw)) {
+    logWarn(`Ignoring "${key}" in ${PROJECT_CONFIG_FILENAME}: must be an array of gitignore-style patterns`, { file });
+    return [];
+  }
+  const out: string[] = [];
+  for (const entry of raw) {
+    if (typeof entry !== 'string' || !entry.trim()) {
+      logWarn(`Ignoring a "${key}" entry in ${PROJECT_CONFIG_FILENAME}: every pattern must be a non-empty string`, { file });
+      continue;
+    }
+    out.push(entry.trim());
+  }
+  return out;
+}
+
 /**
 * Validate the `exclude` patterns: an array of non-empty gitignore-style
 * strings naming paths to keep out of the index even when git-tracked (#999). A
@@ -325,6 +367,17 @@ export function loadExcludePatterns(rootDir: string): string[] {
  return loadParsedConfig(rootDir).exclude;
 }

+/**
+ * Load the validated `deprioritize` patterns for a project, mtime-cached.
+ *
+ * These name indexed paths that must not outrank first-party code (#982) — the
+ * ranking counterpart to `exclude`. An empty result — the zero-config default —
+ * de-prioritizes nothing beyond the built-in example/fixture/benchmark dirs.
+ */
+export function loadDeprioritizePatterns(rootDir: string): string[] {
+  return loadParsedConfig(rootDir).deprioritize;
+}
+
 /**
 * Load the validated `include` patterns for a project, mtime-cached.
 *
@@ -222,6 +222,7 @@ export function scorePathRelevance(
  filePath: string,
  query: string,
  projectNameTokens?: Set<string>,
+  isDeprioritized?: boolean,
 ): number {
  const pathLower = filePath.toLowerCase();
  const fileName = path.basename(filePath).toLowerCase();
@@ -264,10 +265,19 @@ export function scorePathRelevance(
    else if (subtokens.some((t) => pathLower.includes(t))) score += 3;
  }

-  // Deprioritize test files unless the query is explicitly about tests
+  // Deprioritize test files unless the query is explicitly about tests, and
+  // apply the same -15 to a path the project declared peripheral (#982).
+  //
+  // Two deliberate asymmetries, both pinned by tests:
+  //  - the built-in test/fixture penalty is waived for a test-y query, because
+  //    the tool inferred that classification; a `deprioritize` pattern is a
+  //    standing statement by the project, so it is NOT waived. The name-bonus
+  //    damping at the call site is what keeps such a tree findable.
+  //  - a path that is both is docked ONCE, not twice.
  const queryLower = query.toLowerCase();
  const isTestQuery = queryLower.includes('test') || queryLower.includes('spec');
-  if (!isTestQuery && isTestFile(filePath)) {
+  const offTarget = (!isTestQuery && isTestFile(filePath)) || isDeprioritized === true;
+  if (offTarget) {
    score -= 15;
  }