github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(extraction): content-based generated-file detection (CG-5, #1500)

Browse Source 
`isGeneratedFile` was path-only, but Go's own convention is a CONTENT
marker (`// Code generated by <tool>. DO NOT EDIT.`), not a filename one.
A Go monorepo with generated CRUD in ordinarily-named files sitting beside
hand-written use-cases was therefore invisible to every generated-file
down-rank in the codebase — that is #1500.

Measured on kubernetes/client-go (2,453 Go files): the canonical banner
appears in 2,001 of them, the path check flags 0, the new content check
flags exactly those 2,001 — no false positives, no misses.

Design: decide at INDEX time (content is already in memory for parsing),
persist on `files.generated`, read from the DB. Explore never reads file
headers per request.

- `hasGeneratedHeader(content)` recognizes the standard banners — Go's,
  protoc's, `@generated`, `<auto-generated>`, Thrift, OpenAPI Generator,
  FlatBuffers, bindgen, ANTLR. Precision-first and fenced three ways: an
  8KB/60-line header window, a comment-line requirement (leader or open
  block comment), and markers tight enough that prose can't trip them. A
  generator's own source, holding the banner as a string constant in its
  body, is not flagged; neither is this module itself (pinned by test).
- `isGeneratedFile(path)` is unchanged — cheap, sync, still the fallback.
- Schema v9 adds `files.generated` + a PARTIAL index. DDL only, no
  backfill: the flag derives from content the migration cannot see, so
  rows stay 0 until a re-index and every reader unions the flag with the
  path check — an un-migrated index keeps pre-#1500 behavior rather than
  regressing. Re-index required; noted in the CHANGELOG.
- `generatedPredicateFor(paths)` gives ranking a bounded probe + O(1)
  lookups. Bounded, not cached: no invalidation, so a ranking call can
  never serve a verdict the last sync already replaced. Wired into explore
  ranking, findSymbolMatches, findAllSymbols, search (MCP + CLI), the
  context formatter, and the dominant-file/route-file hygiene filters.

Cost (acceptance bar was no measurable index-time regression): a single
unanchored `/generat/i` test over the header rejects ~every hand-written
file before any line splitting. 4.6 µs/file on client-go (worst case —
82% generated). End-to-end `codegraph init` on client-go, n=3 alternating
arms: 5.73s median with detection vs 5.76s path-only baseline; the arms
cross over between runs, so the difference is inside run-to-run noise.

Scope note: generated status remains a stable TIEBREAK at equal score,
exactly where it was. Making it a strong negative signal is CG-10, which
this unblocks by making the signal correct and available.

Two pre-existing tests hard-coded schema version 8; both now track
CURRENT_SCHEMA_VERSION (or the migration table) so future migrations
don't require editing them.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-03 23:13:59 -05:00
co-authored by Claude Opus 5
parent b37f191f5a
commit 16e17495f4
19 changed files with 898 additions and 55 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/mcp/tools.ts
+23 -12
View File
@@ -39,7 +39,6 @@ import {
} from 'fs';
import { createHash } from 'crypto';
import { clamp, validatePathWithinRoot, validateProjectPath, isConfigLeafNode, CONFIG_LEAF_LANGUAGES } from '../utils';
import { isGeneratedFile } from '../extraction/generated-detection';
import { scanDynamicDispatch } from './dynamic-boundaries';
import { getUpdateNotice } from '../upgrade/update-check';
import { ExploreDiagnostics } from './explore-diagnostics';
@@ -1588,9 +1587,10 @@ export class ToolHandler {
// Down-rank generated files within the FTS-returned set so a search
// for "Send" surfaces the hand-written keeper before .pb.go stubs
// that share the name. Stable: only reorders generated vs. not.
const isGen = cg.generatedFilePredicate(results.map((r) => r.node.filePath));
const ranked = [...results].sort((a, b) => {
const aGen = isGeneratedFile(a.node.filePath) ? 1 : 0;
const bGen = isGeneratedFile(b.node.filePath) ? 1 : 0;
const aGen = isGen(a.node.filePath) ? 1 : 0;
const bGen = isGen(b.node.filePath) ? 1 : 0;
return aGen - bGen;
});
@@ -3113,6 +3113,13 @@ export class ToolHandler {
!MULTITERM_OFF &&
(fileTermHits.get(fp) ?? 0) >= 2 &&
(entryFiles.has(fp) || centralFiles.has(fp));
// One DB probe over the ranked candidates, then O(1) per comparison. Unions
// the index-time content-banner flag with the filename convention, so a Go
// monorepo's generated CRUD (`payroll.go` carrying a DO-NOT-EDIT banner and
// nothing in its name) down-ranks the same way `.pb.go` always has (#1500).
const isGeneratedCandidate = cg.generatedFilePredicate(relevantFiles.map(([fp]) => fp));
const sortedFiles = relevantFiles.sort((a, b) => {
const aPath = a[0].toLowerCase();
const bPath = b[0].toLowerCase();
@@ -3147,8 +3154,8 @@ export class ToolHandler {
// the response (the cosmos Q3 explore otherwise leads with
// `expected_keepers_mocks.go`, displacing the real `tally.go` content
// and forcing the agent to Read tally.go anyway).
const aGen = isGeneratedFile(a[0]);
const bGen = isGeneratedFile(b[0]);
const aGen = isGeneratedCandidate(a[0]);
const bGen = isGeneratedCandidate(b[0]);
if (aGen !== bGen) return aGen ? 1 : -1;
if (a[1].score !== b[1].score) return b[1].score - a[1].score;
@@ -3233,7 +3240,7 @@ export class ToolHandler {
entry: entryFiles.has(fp),
spine: group.nodes.some((n) => flow.pathNodeIds.has(n.id)),
lowValue: isLowValue(fp),
generated: isGeneratedFile(fp),
generated: isGeneratedCandidate(fp),
});
});
}
@@ -4753,7 +4760,8 @@ export class ToolHandler {
if (!isQualified) {
const exact = cg.getNodesByName(symbol);
if (exact.length > 0) {
return [...exact].sort((a, b) => (isGeneratedFile(a.filePath) ? 1 : 0) - (isGeneratedFile(b.filePath) ? 1 : 0));
const isGen = cg.generatedFilePredicate(exact.map((n) => n.filePath));
return [...exact].sort((a, b) => (isGen(a.filePath) ? 1 : 0) - (isGen(b.filePath) ? 1 : 0));
}
// No exact match — use the single top fuzzy result (e.g. a file basename).
const fuzzy = cg.searchNodes(symbol, { limit: 10 });
@@ -4781,10 +4789,12 @@ export class ToolHandler {
return isQualified ? [] : results[0] ? [results[0].node] : [];
}
// Down-rank generated files (.pb.go, .pulsar.go, _grpc.pb.go, …) so a flow
// query prefers the keeper implementation over the protobuf-generated stub.
// Down-rank generated files (.pb.go, .pulsar.go, _grpc.pb.go, and anything
// whose header declares it generated) so a flow query prefers the keeper
// implementation over the generated stub.
const isGen = cg.generatedFilePredicate(exactMatches.map((r) => r.node.filePath));
return [...exactMatches]
.sort((a, b) => (isGeneratedFile(a.node.filePath) ? 1 : 0) - (isGeneratedFile(b.node.filePath) ? 1 : 0))
.sort((a, b) => (isGen(a.node.filePath) ? 1 : 0) - (isGen(b.node.filePath) ? 1 : 0))
.map((r) => r.node);
}
@@ -4837,9 +4847,10 @@ export class ToolHandler {
// Same generated-file down-rank as findSymbol — keeps callers/callees
// /impact aggregation aligned (a query against "Send" returns the
// hand-written implementations before the protobuf scaffold).
const isGen = cg.generatedFilePredicate(exactMatches.map((r) => r.node.filePath));
const ranked = [...exactMatches].sort((a, b) => {
const aGen = isGeneratedFile(a.node.filePath) ? 1 : 0;
const bGen = isGeneratedFile(b.node.filePath) ? 1 : 0;
const aGen = isGen(a.node.filePath) ? 1 : 0;
const bGen = isGen(b.node.filePath) ? 1 : 0;
return aGen - bGen;
});