feat(extraction): content-based generated-file detection (CG-5, #1500)

`isGeneratedFile` was path-only, but Go's own convention is a CONTENT
marker (`// Code generated by <tool>. DO NOT EDIT.`), not a filename one.
A Go monorepo with generated CRUD in ordinarily-named files sitting beside
hand-written use-cases was therefore invisible to every generated-file
down-rank in the codebase — that is #1500.

Measured on kubernetes/client-go (2,453 Go files): the canonical banner
appears in 2,001 of them, the path check flags 0, the new content check
flags exactly those 2,001 — no false positives, no misses.

Design: decide at INDEX time (content is already in memory for parsing),
persist on `files.generated`, read from the DB. Explore never reads file
headers per request.

- `hasGeneratedHeader(content)` recognizes the standard banners — Go's,
  protoc's, `@generated`, `<auto-generated>`, Thrift, OpenAPI Generator,
  FlatBuffers, bindgen, ANTLR. Precision-first and fenced three ways: an
  8KB/60-line header window, a comment-line requirement (leader or open
  block comment), and markers tight enough that prose can't trip them. A
  generator's own source, holding the banner as a string constant in its
  body, is not flagged; neither is this module itself (pinned by test).
- `isGeneratedFile(path)` is unchanged — cheap, sync, still the fallback.
- Schema v9 adds `files.generated` + a PARTIAL index. DDL only, no
  backfill: the flag derives from content the migration cannot see, so
  rows stay 0 until a re-index and every reader unions the flag with the
  path check — an un-migrated index keeps pre-#1500 behavior rather than
  regressing. Re-index required; noted in the CHANGELOG.
- `generatedPredicateFor(paths)` gives ranking a bounded probe + O(1)
  lookups. Bounded, not cached: no invalidation, so a ranking call can
  never serve a verdict the last sync already replaced. Wired into explore
  ranking, findSymbolMatches, findAllSymbols, search (MCP + CLI), the
  context formatter, and the dominant-file/route-file hygiene filters.

Cost (acceptance bar was no measurable index-time regression): a single
unanchored `/generat/i` test over the header rejects ~every hand-written
file before any line splitting. 4.6 µs/file on client-go (worst case —
82% generated). End-to-end `codegraph init` on client-go, n=3 alternating
arms: 5.73s median with detection vs 5.76s path-only baseline; the arms
cross over between runs, so the difference is inside run-to-run noise.

Scope note: generated status remains a stable TIEBREAK at equal score,
exactly where it was. Making it a strong negative signal is CG-10, which
this unblocks by making the signal correct and available.

Two pre-existing tests hard-coded schema version 8; both now track
CURRENT_SCHEMA_VERSION (or the migration table) so future migrations
don't require editing them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

src/db/migrations.ts

@@ -9,7 +9,7 @@ import { SqliteDatabase } from './sqlite-adapter';
 /**
  * Current schema version
  */
-export const CURRENT_SCHEMA_VERSION = 8;
+export const CURRENT_SCHEMA_VERSION = 9;
 
 /**
  * Migration definition
@@ -150,6 +150,33 @@ const migrations: Migration[] = [
       `);
     },
   },
+  {
+    version: 9,
+    description:
+      'Add files.generated — index-time content-header generated-file detection for ranking (#1500)',
+    up: (db) => {
+      // DDL only — instant on any size database, and NO backfill: the flag is
+      // derived from file CONTENT, which this migration has no access to (the
+      // files table stores a hash, not the bytes). Migrated rows therefore stay
+      // 0 until the next full index re-extracts them, and every reader unions
+      // the flag with the path-only check, so an un-backfilled database keeps
+      // exactly the pre-#1500 behavior instead of regressing. `sync` heals it
+      // file-by-file as files change. This is why the CHANGELOG entry says a
+      // re-index is required to pick up the new detection.
+      //
+      // ALTER TABLE has no IF NOT EXISTS, so guard for idempotency — a database
+      // created from current schema.sql already has the column (matters when
+      // migrations are re-run from an older recorded version, as the v6
+      // regression test does). Keep in lockstep with schema.sql.
+      const cols = db.prepare('PRAGMA table_info(files)').all() as Array<{ name: string }>;
+      if (!cols.some((c) => c.name === 'generated')) {
+        db.exec('ALTER TABLE files ADD COLUMN generated INTEGER NOT NULL DEFAULT 0');
+      }
+      db.exec(
+        'CREATE INDEX IF NOT EXISTS idx_files_generated ON files(path) WHERE generated = 1'
+      );
+    },
+  },
 ];
 
 /**

src/db/queries.ts

@@ -24,13 +24,18 @@ import { isGeneratedFile } from '../extraction/generated-detection';
 import { splitIdentifierSegments } from '../search/identifier-segments';
 
 /**
- * Path-only heuristic for files that should not be candidates for
- * "dominant file" detection: test/spec files and tool-generated files.
- * Generated files (`*.pb.go`, `*.pulsar.go`, mock outputs, …) often
- * have huge in-file edge counts that dwarf the real source — etcd's
- * `rpc.pb.go` has 4× the in-file edges of `server.go`.
+ * Files that should not be candidates for "dominant file" detection: test/spec
+ * files and tool-generated files. Generated files (`*.pb.go`, `*.pulsar.go`,
+ * mock outputs, …) often have huge in-file edge counts that dwarf the real
+ * source — etcd's `rpc.pb.go` has 4× the in-file edges of `server.go`.
+ *
+ * Path patterns plus, when the caller passes the indexed set, files whose
+ * HEADER declares them generated — a `payroll.go` full of generated CRUD has
+ * exactly the same edge-density problem as `rpc.pb.go` and nothing in its name
+ * to catch it (#1500).
  */
-function isLowValueFile(filePath: string): boolean {
+function isLowValueFile(filePath: string, generated?: ReadonlySet<string>): boolean {
+  if (generated?.has(filePath)) return true;
   const lp = filePath.toLowerCase();
   return (
     /(?:^|\/)(tests?|__tests?__|spec)\//.test(lp) ||
@@ -97,6 +102,8 @@ interface FileRow {
   indexed_at: number;
   node_count: number;
   errors: string | null;
+  /** Absent on pre-v9 rows read through a stale prepared statement. */
+  generated?: number | null;
 }
 
 interface UnresolvedRefRow {
@@ -182,6 +189,7 @@ function rowToFileRecord(row: FileRow): FileRecord {
     indexedAt: row.indexed_at,
     nodeCount: row.node_count,
     errors: row.errors ? safeJsonParse(row.errors, undefined) : undefined,
+    generated: row.generated === 1,
   };
 }
 
@@ -922,7 +930,8 @@ export class QueryBuilder {
       `);
     }
     const rows = this.stmts.getDominantFile.all() as Array<{ file_path: string; edge_count: number }>;
-    const filtered = rows.filter(r => !isLowValueFile(r.file_path));
+    const generated = this.getGeneratedPathsAmong(rows.map(r => r.file_path));
+    const filtered = rows.filter(r => !isLowValueFile(r.file_path, generated));
     if (filtered.length === 0 || filtered[0]!.edge_count < 20) return null;
     return {
       filePath: filtered[0]!.file_path,
@@ -955,7 +964,8 @@ export class QueryBuilder {
       `);
     }
     const rows = this.stmts.getTopRouteFile.all() as Array<{ file_path: string; cnt: number }>;
-    const filtered = rows.filter(r => !isLowValueFile(r.file_path));
+    const generated = this.getGeneratedPathsAmong(rows.map(r => r.file_path));
+    const filtered = rows.filter(r => !isLowValueFile(r.file_path, generated));
     if (filtered.length === 0) return null;
     const totalRoutes = filtered.reduce((sum, r) => sum + r.cnt, 0);
     const top = filtered[0]!;
@@ -1006,7 +1016,8 @@ export class QueryBuilder {
       url: string; handler: string; handler_file: string; handler_line: number; handler_kind: string;
     }>;
     // Drop test/generated handlers — same hygiene as elsewhere.
-    const filtered = rows.filter(r => !isLowValueFile(r.handler_file));
+    const generated = this.getGeneratedPathsAmong(rows.map(r => r.handler_file));
+    const filtered = rows.filter(r => !isLowValueFile(r.handler_file, generated));
     if (filtered.length < 3) return null;
     // Identify the file holding the most handlers (the "primary handler file").
     const fileCounts = new Map<string, number>();
@@ -1865,8 +1876,8 @@ export class QueryBuilder {
   upsertFile(file: FileRecord): void {
     if (!this.stmts.upsertFile) {
       this.stmts.upsertFile = this.db.prepare(`
-        INSERT INTO files (path, content_hash, language, size, modified_at, indexed_at, node_count, errors)
-        VALUES (@path, @contentHash, @language, @size, @modifiedAt, @indexedAt, @nodeCount, @errors)
+        INSERT INTO files (path, content_hash, language, size, modified_at, indexed_at, node_count, errors, generated)
+        VALUES (@path, @contentHash, @language, @size, @modifiedAt, @indexedAt, @nodeCount, @errors, @generated)
         ON CONFLICT(path) DO UPDATE SET
           content_hash = @contentHash,
           language = @language,
@@ -1874,7 +1885,8 @@ export class QueryBuilder {
           modified_at = @modifiedAt,
           indexed_at = @indexedAt,
           node_count = @nodeCount,
-          errors = @errors
+          errors = @errors,
+          generated = @generated
       `);
     }
 
@@ -1887,9 +1899,59 @@ export class QueryBuilder {
       indexedAt: file.indexedAt,
       nodeCount: file.nodeCount,
       errors: file.errors ? JSON.stringify(file.errors) : null,
+      // The upsert always REWRITES the flag: a file that loses its banner in an
+      // edit must lose the flag on the next sync, not keep a stale 1.
+      generated: file.generated ? 1 : 0,
     });
   }
 
+  /**
+   * Which of `filePaths` the index flagged as tool-generated (schema v9+).
+   *
+   * Bounded-lookup by design: every consumer already holds a short candidate
+   * list (a ranked file group, an FTS result page, a LIMIT-20 aggregate), so
+   * this stays a partial-index probe over a handful of paths — no whole-repo
+   * set to materialize, and no cache to invalidate, which means a ranking call
+   * can never serve a verdict the last sync already replaced.
+   *
+   * Returns ONLY the content/index signal; callers union it with
+   * {@link isGeneratedFile} so pre-v9 databases (column present, all zeros
+   * until a re-index) keep the path-only behavior rather than regressing.
+   */
+  getGeneratedPathsAmong(filePaths: Iterable<string>): Set<string> {
+    const unique = [...new Set(filePaths)];
+    const found = new Set<string>();
+    if (unique.length === 0) return found;
+
+    for (let i = 0; i < unique.length; i += SQLITE_PARAM_CHUNK_SIZE) {
+      const chunk = unique.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
+      const placeholders = chunk.map(() => '?').join(',');
+      const rows = this.db
+        .prepare(`SELECT path FROM files WHERE generated = 1 AND path IN (${placeholders})`)
+        .all(...chunk) as Array<{ path: string }>;
+      for (const row of rows) found.add(row.path);
+    }
+    return found;
+  }
+
+  /**
+   * A reusable `(path) => boolean` over a bounded candidate list, unioning the
+   * indexed flag with the path convention. This is the shape every ranking
+   * comparator wants: one query up front, then O(1) per comparison.
+   */
+  generatedPredicateFor(filePaths: Iterable<string>): (filePath: string) => boolean {
+    const flagged = this.getGeneratedPathsAmong(filePaths);
+    return (filePath: string) => flagged.has(filePath) || isGeneratedFile(filePath);
+  }
+
+  /** How many indexed files carry the generated flag. Surfaced by `status`. */
+  countGeneratedFiles(): number {
+    const row = this.db
+      .prepare('SELECT COUNT(*) AS n FROM files WHERE generated = 1')
+      .get() as { n: number } | undefined;
+    return row?.n ?? 0;
+  }
+
   /**
    * Delete a file record and its nodes
    */

src/db/schema.sql

@@ -55,7 +55,15 @@ CREATE TABLE IF NOT EXISTS edges (
     FOREIGN KEY (target) REFERENCES nodes(id) ON DELETE CASCADE
 );
 
--- Files: Tracked source files
+-- Files: Tracked source files.
+-- `generated` is the index-time verdict from extraction/generated-detection.ts:
+-- the filename convention (*.pb.go, *.g.dart, …) OR a generation banner in the
+-- file's header. Go's convention is a CONTENT marker, so a generated
+-- `payroll.go` beside hand-written use-cases is invisible to the path check
+-- alone (#1500) — deciding it here means ranking never reads file headers per
+-- request. Migration v9 adds the column to existing databases; rows keep the
+-- 0 default until the next full index, so readers treat it as a hint that
+-- only ever ADDS to the path signal, never overrides it.
 CREATE TABLE IF NOT EXISTS files (
     path TEXT PRIMARY KEY,
     content_hash TEXT NOT NULL,
@@ -64,7 +72,8 @@ CREATE TABLE IF NOT EXISTS files (
     modified_at INTEGER NOT NULL,
     indexed_at INTEGER NOT NULL,
     node_count INTEGER DEFAULT 0,
-    errors TEXT -- JSON array
+    errors TEXT, -- JSON array
+    generated INTEGER NOT NULL DEFAULT 0
 );
 
 -- Unresolved References: References that need resolution after full indexing.
@@ -173,9 +182,13 @@ CREATE INDEX IF NOT EXISTS idx_edges_target_kind ON edges(target, kind);
 CREATE UNIQUE INDEX IF NOT EXISTS idx_edges_identity
   ON edges(source, target, kind, IFNULL(line, -1), IFNULL(col, -1));
 
--- File indexes
+-- File indexes.
+-- idx_files_generated is PARTIAL: the generated set is a small minority of any
+-- repo, so a lookup that intersects a bounded candidate list with it stays
+-- proportional to the generated files, not to the repo.
 CREATE INDEX IF NOT EXISTS idx_files_language ON files(language);
 CREATE INDEX IF NOT EXISTS idx_files_modified_at ON files(modified_at);
+CREATE INDEX IF NOT EXISTS idx_files_generated ON files(path) WHERE generated = 1;
 
 -- Unresolved refs indexes
 CREATE INDEX IF NOT EXISTS idx_unresolved_from_node ON unresolved_refs(from_node_id);