feat(extraction): content-based generated-file detection (CG-5, #1500)
`isGeneratedFile` was path-only, but Go's own convention is a CONTENT marker (`// Code generated by <tool>. DO NOT EDIT.`), not a filename one. A Go monorepo with generated CRUD in ordinarily-named files sitting beside hand-written use-cases was therefore invisible to every generated-file down-rank in the codebase — that is #1500. Measured on kubernetes/client-go (2,453 Go files): the canonical banner appears in 2,001 of them, the path check flags 0, the new content check flags exactly those 2,001 — no false positives, no misses. Design: decide at INDEX time (content is already in memory for parsing), persist on `files.generated`, read from the DB. Explore never reads file headers per request. - `hasGeneratedHeader(content)` recognizes the standard banners — Go's, protoc's, `@generated`, `<auto-generated>`, Thrift, OpenAPI Generator, FlatBuffers, bindgen, ANTLR. Precision-first and fenced three ways: an 8KB/60-line header window, a comment-line requirement (leader or open block comment), and markers tight enough that prose can't trip them. A generator's own source, holding the banner as a string constant in its body, is not flagged; neither is this module itself (pinned by test). - `isGeneratedFile(path)` is unchanged — cheap, sync, still the fallback. - Schema v9 adds `files.generated` + a PARTIAL index. DDL only, no backfill: the flag derives from content the migration cannot see, so rows stay 0 until a re-index and every reader unions the flag with the path check — an un-migrated index keeps pre-#1500 behavior rather than regressing. Re-index required; noted in the CHANGELOG. - `generatedPredicateFor(paths)` gives ranking a bounded probe + O(1) lookups. Bounded, not cached: no invalidation, so a ranking call can never serve a verdict the last sync already replaced. Wired into explore ranking, findSymbolMatches, findAllSymbols, search (MCP + CLI), the context formatter, and the dominant-file/route-file hygiene filters. Cost (acceptance bar was no measurable index-time regression): a single unanchored `/generat/i` test over the header rejects ~every hand-written file before any line splitting. 4.6 µs/file on client-go (worst case — 82% generated). End-to-end `codegraph init` on client-go, n=3 alternating arms: 5.73s median with detection vs 5.76s path-only baseline; the arms cross over between runs, so the difference is inside run-to-run noise. Scope note: generated status remains a stable TIEBREAK at equal score, exactly where it was. Making it a strong negative signal is CG-10, which this unblocks by making the signal correct and available. Two pre-existing tests hard-coded schema version 8; both now track CURRENT_SCHEMA_VERSION (or the migration table) so future migrations don't require editing them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
co-authored by
Claude Opus 5
Claude Opus 5
parent
b37f191f5a
commit
16e17495f4
@@ -8,18 +8,39 @@
|
||||
* see project_go_multi_module_audit memory). Generated stubs frequently
|
||||
* have no body to trace from, so the agent ends up reading source anyway.
|
||||
*
|
||||
* This helper is a pure path-based classifier consulted at disambiguation
|
||||
* time (findSymbol / findAllSymbols / codegraph_search formatting), NOT
|
||||
* a hard filter — generated nodes are still in the graph and remain
|
||||
* reachable; they just rank LAST when there's a real implementation
|
||||
* with the same name.
|
||||
* This is a relevance hint consulted at disambiguation time (findSymbol /
|
||||
* findAllSymbols / explore ranking / codegraph_search formatting), NOT a
|
||||
* hard filter — generated nodes are still in the graph and remain
|
||||
* reachable; they just rank LAST when there's a real implementation with
|
||||
* the same name.
|
||||
*
|
||||
* Scope: suffix patterns only. Most generated files follow the
|
||||
* `<basename>.<tool>.<ext>` convention (`.pb.go`, `_grpc.pb.go`,
|
||||
* `.g.dart`, `_pb2.py`), and that covers ~all of what we saw in the
|
||||
* Go audit. A future addition would be scanning for the canonical
|
||||
* `// Code generated by` header during extraction, for the rare files
|
||||
* that defy the suffix convention.
|
||||
* Two signals, deliberately separate:
|
||||
*
|
||||
* 1. {@link isGeneratedFile} — PATH only, pure and synchronous. Most
|
||||
* generated files follow the `<basename>.<tool>.<ext>` convention
|
||||
* (`.pb.go`, `_grpc.pb.go`, `.g.dart`, `_pb2.py`). Free to call
|
||||
* anywhere, including in a sort comparator.
|
||||
*
|
||||
* 2. {@link hasGeneratedHeader} — CONTENT banner in the file's head. Go's
|
||||
* own convention is a content marker, not a filename one, so a
|
||||
* generated `payroll.go` sitting beside hand-written use-cases is
|
||||
* invisible to (1) — that is issue #1500. Evaluated ONCE at index time
|
||||
* (the file's content is already in memory for parsing) and persisted
|
||||
* on the file record as `files.generated`; readers get it from the DB
|
||||
* rather than re-reading headers per request. See
|
||||
* GENERATED_CONTENT_PATTERNS below for the banners recognized.
|
||||
*
|
||||
* Consumers that have a bounded candidate list should use the DB-backed
|
||||
* union (`QueryBuilder.getGeneratedPathsAmong` /
|
||||
* `CodeGraph.getGeneratedFilePaths`) so both signals apply; the path-only
|
||||
* check remains the fallback for callers with no database in hand and for
|
||||
* indexes built before the flag existed.
|
||||
*
|
||||
* NOTE for future editors: the banner literals quoted in this file sit
|
||||
* BELOW the header window this detector scans, so the module does not
|
||||
* classify itself. `generated-detection.test.ts` pins that — if you move
|
||||
* the pattern table upward, the test fails rather than the repo silently
|
||||
* demoting its own file.
|
||||
*/
|
||||
|
||||
const GENERATED_PATTERNS: ReadonlyArray<RegExp> = [
|
||||
@@ -79,3 +100,146 @@ const GENERATED_PATTERNS: ReadonlyArray<RegExp> = [
|
||||
export function isGeneratedFile(filePath: string): boolean {
|
||||
return GENERATED_PATTERNS.some((p) => p.test(filePath));
|
||||
}
|
||||
|
||||
// =============================================================================
|
||||
// Content-header detection (#1500)
|
||||
// =============================================================================
|
||||
|
||||
/**
|
||||
* How much of a file's head to consider "the header". Generous enough for a
|
||||
* build-tag block + an Apache-2.0 license preamble (~15 lines) sitting above
|
||||
* the banner, tight enough that a `"// Code generated ... DO NOT EDIT."`
|
||||
* string constant in the *body* of a code generator's own source can't
|
||||
* masquerade as a banner.
|
||||
*/
|
||||
const HEADER_SCAN_CHARS = 8192;
|
||||
const HEADER_SCAN_LINES = 60;
|
||||
|
||||
/**
|
||||
* Cheap pre-filter run on the header of EVERY indexed file. Every marker
|
||||
* below contains the stem "generat", so one unanchored scan rejects ~all
|
||||
* hand-written source before any line splitting happens — this is what keeps
|
||||
* content detection off the index-time cost budget.
|
||||
*/
|
||||
const GENERATED_STEM = /generat/i;
|
||||
|
||||
/**
|
||||
* Line-comment leaders across the languages we index. A banner must sit on a
|
||||
* comment line (or inside an open block comment, tracked below): generators
|
||||
* always emit theirs as a comment, and requiring it rules out string literals
|
||||
* and identifiers that merely contain the words.
|
||||
*
|
||||
* `--` covers SQL/Haskell/Lua, `%` LaTeX/Erlang/Prolog, `;` Lisp/asm/ini,
|
||||
* `'` VB, `!` Fortran, `*` a continuation line inside a `/* … *\/` block.
|
||||
*/
|
||||
const COMMENT_LEADER =
|
||||
/^\s*(?:\/\/|\/\*+|\*+\/?|#+|--+|<!--|%+|;+|'|!|\(\*|\{-|"""|'''|=begin|<#|@rem\b|rem\b)/i;
|
||||
|
||||
/**
|
||||
* Openers/closers for block comments, so a banner on an unprefixed line
|
||||
* inside `/* … *\/` (or `<!-- … -->`, or a Python module docstring) still
|
||||
* counts. Deliberately naive — it only runs over a file's first few dozen
|
||||
* lines, where a `/*` inside a string literal is vanishingly rare, and the
|
||||
* worst case of a mis-tracked state is a ranking hint, not a wrong answer.
|
||||
*/
|
||||
const BLOCK_DELIMS: ReadonlyArray<{ open: string; close: string }> = [
|
||||
{ open: '/*', close: '*/' },
|
||||
{ open: '<!--', close: '-->' },
|
||||
{ open: '"""', close: '"""' },
|
||||
{ open: "'''", close: "'''" },
|
||||
{ open: '=begin', close: '=end' },
|
||||
{ open: '<#', close: '#>' },
|
||||
];
|
||||
|
||||
/**
|
||||
* The banners themselves. Each is a real convention emitted by a widely-used
|
||||
* generator; the list is precision-first, because a false positive silently
|
||||
* demotes hand-written code in every ranking path.
|
||||
*/
|
||||
const GENERATED_CONTENT_PATTERNS: ReadonlyArray<RegExp> = [
|
||||
// Go's codified convention — `^// Code generated .* DO NOT EDIT\.$`, defined
|
||||
// by `go generate` and honored by gofmt, golangci-lint and GitHub linguist.
|
||||
// Emitted verbatim by protoc-gen-go, mockgen, sqlc, ent, wire, stringer, and
|
||||
// by in-house generators like the FKIT CRUD in #1500 — where the file is
|
||||
// named `payroll.go` and nothing in the PATH gives it away.
|
||||
/\bcode generated\b.{0,200}?\bdo not edit\b/i,
|
||||
// protoc's Java/C#/Python banner ("Generated by the protocol buffer
|
||||
// compiler. DO NOT EDIT!"), ANTLR, Dagger, FlatBuffers, rust-bindgen,
|
||||
// Xcode asset catalogs, Bazel rules.
|
||||
/\b(?:automatically |auto[- ]?)?generated (?:by|from|with)\b.{0,200}?\bdo not (?:edit|modify|change)\b/i,
|
||||
// The `@generated` marker: the JS/TS ecosystem's convention (Relay, GraphQL
|
||||
// codegen, protobuf-es/Buf, Meta's `@generated SignedSource<<…>>`), also
|
||||
// what linguist and `git diff` collapse on. Guarded against `foo@generated`
|
||||
// and `@@generated` so only a standalone tag matches.
|
||||
/(?:^|[^\p{L}\p{N}_@])@generated\b/u,
|
||||
// .NET's `<auto-generated>` / `<auto-generated />` doc tag: Roslyn, the
|
||||
// WinForms designer, T4 templates, protoc-gen-csharp, EF scaffolding.
|
||||
/<auto-?generated\s*\/?>/i,
|
||||
// swagger-codegen / OpenAPI Generator ("NOTE: This class is auto generated
|
||||
// by OpenAPI Generator"), Thrift ("Autogenerated by Thrift Compiler"),
|
||||
// FlatBuffers ("automatically generated by the FlatBuffers compiler").
|
||||
// "by" is required — bare "automatically generated" appears in hand-written
|
||||
// prose ("the table below is automatically generated at runtime").
|
||||
/\b(?:automatically generated|auto[- ]?generated|autogenerated) by\b/i,
|
||||
// Self-declaring in-house banners that name no tool.
|
||||
/\bthis (?:file|class|code|module) (?:is|was) (?:auto[- ]?)?generated\b/i,
|
||||
// The reverse ordering: "DO NOT EDIT — this is a generated file".
|
||||
/\bdo not (?:edit|modify)\b.{0,120}?\b(?:auto[- ]?generated|generated file|generated code)\b/i,
|
||||
];
|
||||
|
||||
/**
|
||||
* Whether the head of `content` carries a recognized machine-generation
|
||||
* banner. Bounded to {@link HEADER_SCAN_CHARS} / {@link HEADER_SCAN_LINES},
|
||||
* and the marker must sit on a comment line — a generator's own source, which
|
||||
* holds the banner as a string constant in its body, is not flagged.
|
||||
*
|
||||
* Called once per file during extraction (content is already in memory), NOT
|
||||
* per query: the verdict is persisted on the file record.
|
||||
*/
|
||||
export function hasGeneratedHeader(content: string): boolean {
|
||||
if (!content) return false;
|
||||
|
||||
const head = content.length > HEADER_SCAN_CHARS ? content.slice(0, HEADER_SCAN_CHARS) : content;
|
||||
// Fast reject for ~every hand-written file: no line splitting, no allocation
|
||||
// (V8 keeps `head` as a sliced view of `content`).
|
||||
if (!GENERATED_STEM.test(head)) return false;
|
||||
|
||||
const lines = head.split('\n');
|
||||
const limit = Math.min(lines.length, HEADER_SCAN_LINES);
|
||||
let openBlock: (typeof BLOCK_DELIMS)[number] | null = null;
|
||||
|
||||
for (let i = 0; i < limit; i++) {
|
||||
const line = lines[i]!;
|
||||
const inBlock = openBlock !== null;
|
||||
|
||||
if (inBlock || COMMENT_LEADER.test(line)) {
|
||||
for (const pattern of GENERATED_CONTENT_PATTERNS) {
|
||||
if (pattern.test(line)) return true;
|
||||
}
|
||||
}
|
||||
|
||||
// Advance the block-comment state AFTER testing, so the opening line of a
|
||||
// `/* Code generated … */` block is itself matched by the leader rule.
|
||||
if (openBlock) {
|
||||
if (line.includes(openBlock.close)) openBlock = null;
|
||||
continue;
|
||||
}
|
||||
for (const delim of BLOCK_DELIMS) {
|
||||
const at = line.indexOf(delim.open);
|
||||
if (at < 0) continue;
|
||||
// Same-line close (`/* … */`, a one-line docstring) leaves no open block.
|
||||
if (line.indexOf(delim.close, at + delim.open.length) < 0) openBlock = delim;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* The union signal: path convention OR content banner. This is what the
|
||||
* indexer persists to `files.generated`.
|
||||
*/
|
||||
export function detectGeneratedFile(filePath: string, content: string): boolean {
|
||||
return isGeneratedFile(filePath) || hasGeneratedHeader(content);
|
||||
}
|
||||
|
||||
@@ -25,6 +25,7 @@ import { extractFromSource } from './tree-sitter';
|
||||
import { ParseWorkerPool, resolveParsePoolSize, resolveParseTimeoutMs } from './parse-pool';
|
||||
import { StoreWriter, StoreBundle, finalizeStoreBundle } from './store-writer';
|
||||
import { materializeKernelResult } from './kernel';
|
||||
import { detectGeneratedFile } from './generated-detection';
|
||||
import { detectLanguage, isSourceFile, isLanguageSupported, isFileLevelOnlyLanguage, initGrammars, loadGrammarsForLanguages, readGrammarWasmBytes } from './grammars';
|
||||
import { loadExtensionOverrides, loadIncludeIgnoredPatterns, loadExcludePatterns, loadIncludePatterns } from '../project-config';
|
||||
import { isCodeGraphDataDir } from '../directory';
|
||||
@@ -2275,6 +2276,11 @@ export class ExtractionOrchestrator {
|
||||
return; // No changes
|
||||
}
|
||||
|
||||
// Re-decided on every re-index of a changed file, so a banner added (or
|
||||
// removed) by an edit is reflected on the next sync (#1500). Computed after
|
||||
// the unchanged-file early return so untouched files pay nothing.
|
||||
const generated = detectGeneratedFile(filePath, content);
|
||||
|
||||
// Snapshot incoming cross-file edges BEFORE deleting this file's nodes.
|
||||
// `deleteFile` cascades to delete every edge whose source OR target is a
|
||||
// node in this file (edges.FK ... ON DELETE CASCADE). Edges whose SOURCE is
|
||||
@@ -2340,6 +2346,7 @@ export class ExtractionOrchestrator {
|
||||
indexedAt: Date.now(),
|
||||
nodeCount: result.nodes.length,
|
||||
errors: result.errors.length > 0 ? result.errors : undefined,
|
||||
generated,
|
||||
},
|
||||
});
|
||||
if (crossFileIncomingEdges.length > 0) {
|
||||
@@ -2400,6 +2407,7 @@ export class ExtractionOrchestrator {
|
||||
indexedAt: Date.now(),
|
||||
nodeCount: result.nodes.length,
|
||||
errors: result.errors.length > 0 ? result.errors : undefined,
|
||||
generated,
|
||||
};
|
||||
this.queries.upsertFile(fileRecord);
|
||||
}
|
||||
@@ -2427,6 +2435,10 @@ export class ExtractionOrchestrator {
|
||||
indexedAt: Date.now(),
|
||||
nodeCount,
|
||||
errors: resultErrors.length > 0 ? resultErrors : undefined,
|
||||
// Decided here, once, while the content is already in memory — never at
|
||||
// query time (#1500). The header scan short-circuits on a single
|
||||
// substring test for ~every hand-written file.
|
||||
generated: detectGeneratedFile(filePath, content),
|
||||
};
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user