codegraph/docs/design/generated-file-detection.md

Generated-file detection — path convention plus content banner

CG-5, issue #1500. Companion to explore-budget-allocation.md (CG-4's instrument) and a prerequisite for the scoring overhaul (CG-10).

The problem

isGeneratedFile was path-only. It matches the <basename>.<tool>.<ext> convention — .pb.go, _grpc.pb.go, .g.dart, _pb2.py — which is where most codegen output lives, and which was enough for the cosmos-sdk audit that motivated it.

It is not enough for Go. Go's convention is a content marker, not a filename one:

// Code generated by <tool>. DO NOT EDIT.

codified by go generate, honored by gofmt, golangci-lint and GitHub linguist, and emitted by protoc-gen-go, mockgen, sqlc, ent, wire, stringer — and by in-house generators. #1500 is exactly this: a Go monorepo with generated FKIT CRUD in ordinarily-named files (payroll.go) sitting beside hand-written workflow use-cases. Nothing in the path gives it away, so every generated-file down-rank in the codebase was a no-op on it.

How large the gap is

Measured on a shallow clone of kubernetes/client-go (2,453 Go files):

signal	files flagged
ground truth (grep the canonical banner in the first 60 lines)	2,001
path convention (isGeneratedFile)	0
content banner (hasGeneratedHeader)	2,001 — 0 false positives, 0 misses

82% of that repository is generated code with ordinary filenames, and the path check saw none of it. This is not a long-tail case.

Design

Decide at index time, store on the file record, read from the DB. Explore must never read file headers per request.

• isGeneratedFile(path) — unchanged. Path-only, pure, synchronous, free to call in a sort comparator. Kept for callers with no database in hand.
• hasGeneratedHeader(content) — the content signal (below).
• detectGeneratedFile(path, content) — the union, which is what the indexer persists.
• files.generated INTEGER NOT NULL DEFAULT 0 (schema v9) + a partial index idx_files_generated ON files(path) WHERE generated = 1, so lookups cost the generated minority, not the repo.
• QueryBuilder.generatedPredicateFor(paths) / CodeGraph.generatedFilePredicate(paths) — one bounded probe up front, O(1) per comparison after, unioned with the path check.

Why a bounded lookup and not a cached set

Every consumer already holds a short candidate list — a ranked file group, an FTS result page, a LIMIT 20 aggregate. Intersecting that list against the partial index needs no whole-repo set to materialize and, more importantly, no cache to invalidate: a ranking call can never serve a verdict the last sync already replaced. The alternative (a lazily materialized Set of all generated paths) has to be invalidated on every file write and goes stale on the read-only pool workers, in exchange for saving a sub-millisecond query.

Precision over recall

A false positive silently demotes hand-written code in every ranking path, so the marker table is precision-first and the scan is fenced three ways:

1. Header window only — first 8,192 chars / 60 lines. Generous enough for build tags plus an Apache-2.0 preamble above the banner; tight enough that a generator's own source, which holds the banner as a string constant in its body, is not flagged.
2. Comment lines only — the marker must sit on a line with a comment leader (//, #, --, <!--, %, ;, ', …) or inside an open block comment (/* */, <!-- -->, """, ''', =begin, <# #>), tracked with a small state machine over the window. Generators always emit banners as comments; requiring it rules out identifiers and string literals that merely contain the words.
3. Tight markers — automatically generated alone is prose ("the table is automatically generated at runtime"); automatically generated **by** is a banner. DO NOT EDIT alone is a style directive; paired with a generation claim it is a banner.

The module deliberately keeps its own quoted banner literals below the header window so it does not classify itself; generated-detection.test.ts pins that, so moving the pattern table upward fails a test rather than silently demoting this file.

Migration: no backfill, by necessity

v9 is DDL only. The flag derives from file content, which the migration cannot see — files stores a hash, not bytes. Migrated rows stay 0 until a re-index, and because every reader unions the flag with the path check, an un-backfilled database keeps exactly the pre-#1500 behavior instead of regressing. sync heals it file-by-file as files change. This is why the CHANGELOG entry says a re-index is required.

Cost

The acceptance bar was "no measurable index-time cost regression."

A single unanchored /generat/i test over the header rejects ~every hand-written file before any line splitting happens. String.prototype.slice on a long string yields a V8 sliced view, not a copy, so the fast path allocates nothing.

• Microbenchmark (detectGeneratedFile over a whole corpus, 5 passes): 4.6 µs/file on client-go (2,453 files, 14.2 MB, 82% generated — the worst case, where the gate passes and the full line scan runs), 7.3 µs/file on this repo's src.

• End-to-end codegraph init on client-go, n=3 alternating arms (current build vs. the same build with the content scan stubbed out):

  arm	runs (s)	median
  with content detection	5.66, 5.73, 5.89	5.73
  path-only baseline	5.52, 5.76, 5.88	5.76

  The arms cross over between runs — the difference is inside run-to-run noise.

What this task does NOT change

Generated status remains a stable tiebreak at equal score, exactly where it was (src/mcp/tools.ts file sort, findSymbolMatches, findAllSymbols, search formatting, getDominantFile/getTopRouteFile/getRoutingManifest, the context formatter). A generated file with a higher raw score still outranks a hand-written one. Turning generated status into a strong negative signal is CG-10, which this task unblocks by making the signal correct and available.

Verified end-to-end on a two-file Go package where a generated payroll.go and a hand-written workflow.go both define ProcessPayroll: with the flag set the hand-written file ranks first; clearing the flag in the same index (i.e. pre-#1500 behavior) puts the generated file first.