fix(explore): damp ambient declaration files on flow queries (CG-28)
A file that declares nothing but types and that nothing in the index depends on — a hand-written ambient `.d.ts` of global shims, vendored typings, module augmentation — cannot answer a flow question: no bodies, no call edges, no behaviour, nothing typed by it. But the identifiers it declares are exactly the generic ones a prose question uses (`Body`, `Message`, `ImageMetadata`, `ReadableStream`), so on term overlap it out-scored the implementation. Measured on the new fixture: rank #1 and 51% of delivered source, with the flow's own entry file pushed out of the response entirely. Measured first, per the issue: the Wrangler `worker-configuration.d.ts` that opened this is already handled by CG-25's banner detection, worth 15-46 points of envelope share across four flow queries. CG-25 credited; only the un-bannered case needed anything. `rankPenalty` now multiplies score and graph mass by 0.5 for such files, taken as the STRONGER of it and the generated penalty rather than multiplied — one property two signals see must not be charged twice. Detection is structural, not by extension, and four conditions deep. Two of them were forced by measurement: requiring every symbol to be type-level takes the corpus flag rate from 1-18% (which swept in Kotlin sealed classes, Rust mod.rs re-exports and django's locale tables) down to 0-4%; requiring that nothing depends on the file separates an ambient shim from a working types module, and without it the rule demoted displacement-ts's pipeline `types.ts` and broke the CG-31 gate. A query that NAMES a declared type is exempt, so a question about a type still reaches its declaration at full weight. Precise tokens only, so "…the file body…" cannot exempt a `Body` interface it never meant to name; this needs its own set because `namedSeedIds` is callable-only and a type never becomes one. Regression evidence in docs/benchmarks/explore-declaration-only-cg28.md: 6-repo envelope sweep byte-identical against a clean baseline build, zero ambient files reach the candidate set on VS Code across five queries, corpus flag rate 0-0.74%, both allocation fixtures PASS, full suite 2,978 green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
463f6e7844
commit
9efae0f8f2
@@ -1944,6 +1944,101 @@ export class QueryBuilder {
|
||||
return (filePath: string) => flagged.has(filePath) || isGeneratedFile(filePath);
|
||||
}
|
||||
|
||||
/**
|
||||
* Which of `filePaths` are AMBIENT DECLARATION files — they declare nothing
|
||||
* but types, and nothing in the index depends on them (CG-28). A hand-written
|
||||
* ambient `.d.ts` of global shims, a vendored typings file, module
|
||||
* augmentation: reachable only by name, structurally attached to nothing.
|
||||
*
|
||||
* Structural, not extension-based, so a hand-written `types.ts` and a `.d.ts`
|
||||
* are judged by the same rule and a `.d.ts` that does declare a class or a
|
||||
* const is (correctly) not caught. Four conditions, all required:
|
||||
*
|
||||
* 1. it declares at least one symbol — an empty or unparsed file is not a
|
||||
* declaration file, it is a file we know nothing about;
|
||||
* 2. EVERY declared symbol is a type-level kind (interface / type alias /
|
||||
* enum / namespace). The narrowness is deliberate and measured: a rule
|
||||
* of "no callables" alone flags 1–18% of a repo, including Kotlin sealed
|
||||
* classes, Rust `mod.rs` re-exports and django's locale constant tables —
|
||||
* real source that must not be demoted. This rule flags 0–4%;
|
||||
* 3. no symbol in it originates a `calls`/`instantiates` edge — the direct
|
||||
* evidence that nothing here has a body;
|
||||
* 4. NOTHING ELSE IN THE INDEX points at it. This is the condition that
|
||||
* separates an ambient shim from a working type module, and it is why
|
||||
* the flag is narrow enough to be safe: `displacement-ts`'s pipeline
|
||||
* `types.ts` passes 1–3 identically but carries 13 inbound imports and
|
||||
* 21 references, so the files that answer a query about the pipeline are
|
||||
* typed BY it — it is part of that answer's structure. An ambient
|
||||
* `declare global` shim has zero. Deliberately index-wide rather than
|
||||
* restricted to the candidate list: the file that imports it is usually
|
||||
* not itself a candidate.
|
||||
*
|
||||
* Bounded-lookup like {@link getGeneratedPathsAmong}: callers hold a ranked
|
||||
* candidate list, so this is a partial-index probe over a handful of paths.
|
||||
*/
|
||||
getAmbientDeclarationPathsAmong(filePaths: Iterable<string>): Set<string> {
|
||||
const unique = [...new Set(filePaths)];
|
||||
const found = new Set<string>();
|
||||
if (unique.length === 0) return found;
|
||||
|
||||
for (let i = 0; i < unique.length; i += SQLITE_PARAM_CHUNK_SIZE) {
|
||||
const chunk = unique.slice(i, i + SQLITE_PARAM_CHUNK_SIZE);
|
||||
const placeholders = chunk.map(() => '?').join(',');
|
||||
// `file`/`import`/`export`/`parameter` are structural bookkeeping, not
|
||||
// things the file declares, so they neither qualify nor disqualify.
|
||||
const rows = this.db
|
||||
.prepare(`
|
||||
SELECT file_path,
|
||||
SUM(CASE WHEN kind NOT IN ('file','import','export','parameter')
|
||||
THEN 1 ELSE 0 END) AS declared,
|
||||
SUM(CASE WHEN kind IN ('interface','type_alias','enum','enum_member','namespace')
|
||||
THEN 1 ELSE 0 END) AS typeDeclared
|
||||
FROM nodes
|
||||
WHERE file_path IN (${placeholders})
|
||||
GROUP BY file_path
|
||||
`)
|
||||
.all(...chunk) as Array<{ file_path: string; declared: number; typeDeclared: number }>;
|
||||
let candidates = rows
|
||||
.filter((r) => r.declared > 0 && r.declared === r.typeDeclared)
|
||||
.map((r) => r.file_path);
|
||||
if (candidates.length === 0) continue;
|
||||
|
||||
const disqualify = (sql: string): void => {
|
||||
if (candidates.length === 0) return;
|
||||
const hit = new Set(
|
||||
(this.db
|
||||
.prepare(sql.replace('$IN$', candidates.map(() => '?').join(',')))
|
||||
.all(...candidates) as Array<{ file_path: string }>).map((r) => r.file_path),
|
||||
);
|
||||
candidates = candidates.filter((p) => !hit.has(p));
|
||||
};
|
||||
// (3) originates behaviour
|
||||
disqualify(`
|
||||
SELECT DISTINCT n.file_path AS file_path
|
||||
FROM edges e JOIN nodes n ON n.id = e.source
|
||||
WHERE e.kind IN ('calls','instantiates') AND n.file_path IN ($IN$)
|
||||
`);
|
||||
// (4) something outside the file depends on it
|
||||
disqualify(`
|
||||
SELECT DISTINCT t.file_path AS file_path
|
||||
FROM edges e JOIN nodes t ON t.id = e.target JOIN nodes s ON s.id = e.source
|
||||
WHERE t.file_path IN ($IN$) AND s.file_path <> t.file_path
|
||||
`);
|
||||
for (const path of candidates) found.add(path);
|
||||
}
|
||||
return found;
|
||||
}
|
||||
|
||||
/**
|
||||
* A reusable `(path) => boolean` ambient-declaration test over a bounded
|
||||
* candidate list — the shape a ranking comparator wants: one query up front,
|
||||
* O(1) per comparison.
|
||||
*/
|
||||
ambientDeclarationPredicateFor(filePaths: Iterable<string>): (filePath: string) => boolean {
|
||||
const flagged = this.getAmbientDeclarationPathsAmong(filePaths);
|
||||
return (filePath: string) => flagged.has(filePath);
|
||||
}
|
||||
|
||||
/** How many indexed files carry the generated flag. Surfaced by `status`. */
|
||||
countGeneratedFiles(): number {
|
||||
const row = this.db
|
||||
|
||||
@@ -1550,6 +1550,19 @@ export class CodeGraph {
|
||||
return this.queries.generatedPredicateFor(filePaths);
|
||||
}
|
||||
|
||||
/**
|
||||
* A `(path) => boolean` ambient-declaration test over a BOUNDED candidate
|
||||
* list: true for a file that declares nothing but types, originates no call
|
||||
* edge, and that nothing in the index depends on — an ambient `.d.ts` of
|
||||
* global shims, vendored typings, module augmentation (CG-28). Structural
|
||||
* rather than extension-based, and deliberately narrow: see
|
||||
* `QueryBuilder.getAmbientDeclarationPathsAmong` for why each condition is
|
||||
* there, in particular why a `types.ts` the codebase imports is NOT flagged.
|
||||
*/
|
||||
ambientDeclarationFilePredicate(filePaths: Iterable<string>): (filePath: string) => boolean {
|
||||
return this.queries.ambientDeclarationPredicateFor(filePaths);
|
||||
}
|
||||
|
||||
/** How many indexed files are flagged tool-generated. Reported by `status`. */
|
||||
getGeneratedFileCount(): number {
|
||||
return this.queries.countGeneratedFiles();
|
||||
|
||||
@@ -66,6 +66,12 @@ export interface ExploreCandidateMeta {
|
||||
spine: boolean;
|
||||
lowValue: boolean;
|
||||
generated: boolean;
|
||||
/**
|
||||
* Nothing but type declarations in this file, and nothing in the index
|
||||
* depends on it (CG-28) — it cannot answer a flow question, so it ranks on
|
||||
* discounted signals unless the query named one of the types it declares.
|
||||
*/
|
||||
ambientDeclaration: boolean;
|
||||
/**
|
||||
* Multiplier `rankPenalty` applied to BOTH `score` and `graphScore` (1 = no
|
||||
* penalty). Generated and test/i18n files rank on discounted signals, so the
|
||||
@@ -579,6 +585,7 @@ export class ExploreDiagnostics {
|
||||
spine: r.spine,
|
||||
lowValue: r.lowValue,
|
||||
generated: r.generated,
|
||||
ambientDeclaration: r.ambientDeclaration,
|
||||
penalty: round6(r.penalty),
|
||||
kinds: r.kinds,
|
||||
allowance: r.allowance,
|
||||
@@ -796,5 +803,6 @@ function flagString(f: ExploreDiagnosticFile): string {
|
||||
if (f.spine) flags.push('spine');
|
||||
if (f.lowValue) flags.push('low-value');
|
||||
if (f.generated) flags.push('generated');
|
||||
if (f.ambientDeclaration) flags.push('ambient-decl');
|
||||
return flags.join(' ') || '-';
|
||||
}
|
||||
|
||||
+70
-3
@@ -409,6 +409,36 @@ const GENERATED_RANK_PENALTY = 0.3;
|
||||
* that case: down-weighted rather than removed.
|
||||
*/
|
||||
const LOW_VALUE_RANK_PENALTY = 0.5;
|
||||
/**
|
||||
* Ambient declaration files — a hand-written `.d.ts` of global shims, vendored
|
||||
* typings, module augmentation (CG-28). Declares nothing but types, and nothing
|
||||
* in the index depends on it.
|
||||
*
|
||||
* Such a file cannot answer a FLOW question no matter how much its identifiers
|
||||
* overlap the query: no bodies, no call edges, no behaviour, and nothing typed
|
||||
* by it. Its ceiling of usefulness is a type signature, and one follow-up
|
||||
* explore fetches that. But the identifiers it declares are exactly the generic
|
||||
* ones a prose question uses (`Body`, `Message`, `ImageMetadata`,
|
||||
* `ReadableStream`), so on term overlap it out-scores the implementation and
|
||||
* takes the envelope — measured at rank #1 and 51% of delivered source, with
|
||||
* the flow's own entry file getting none.
|
||||
*
|
||||
* Softer than {@link GENERATED_RANK_PENALTY} on purpose: "generated" is a claim
|
||||
* about provenance the file itself makes, while this is an inference about what
|
||||
* a file can be USEFUL for. A demoted declaration file that is still the best
|
||||
* candidate should keep its place; the penalty only has to stop it beating real
|
||||
* implementation. It does NOT stack with the generated penalty (see rankPenalty)
|
||||
* — penalising twice for the same property is how a file gets cliffed out of
|
||||
* answers where it is genuinely relevant.
|
||||
*/
|
||||
const AMBIENT_DECLARATION_RANK_PENALTY = 0.5;
|
||||
/**
|
||||
* The type-level NodeKinds. Must stay in step with the kind list in
|
||||
* `QueryBuilder.getAmbientDeclarationPathsAmong` — that query decides which
|
||||
* files are ambient declarations, this set decides which symbols in them the
|
||||
* agent can name to lift the penalty back off.
|
||||
*/
|
||||
const DECLARATION_KINDS = new Set(['interface', 'type_alias', 'enum', 'enum_member', 'namespace']);
|
||||
|
||||
/**
|
||||
* Score floor: `clamp(topScore * FRACTION, ABSOLUTE, MAX)`.
|
||||
@@ -3275,6 +3305,9 @@ export class ToolHandler {
|
||||
// and crowd out the real answer file (grpc's `dialoptions.go`). Corroborated
|
||||
// overloads (the query also named the type) all earn it. (#1064)
|
||||
const tierSeedIds = new Set<string>();
|
||||
// Files declaring a TYPE the query named by name — the counter-case guard
|
||||
// for the declaration-only penalty (CG-28). Populated in the token loop.
|
||||
const namedTypeFiles = new Set<string>();
|
||||
{
|
||||
const FILE_EXT = /\.(?:java|kt|kts|ts|tsx|js|jsx|mjs|cjs|cs|py|go|rb|php|swift|rs|cpp|cc|cxx|c|h|hpp|scala|lua|dart|vue|svelte|astro|erl|hrl)$/i;
|
||||
const CALLABLE = new Set(['method', 'function', 'component', 'constructor']);
|
||||
@@ -3341,6 +3374,21 @@ export class ToolHandler {
|
||||
// codegraph_node's findSymbolMatches.) Qualified tokens keep findAllSymbols.
|
||||
const isQual = /[.\/]|::/.test(t);
|
||||
const raw = isQual ? this.findAllSymbols(cg, t).nodes : cg.getNodesByName(t);
|
||||
// A query that NAMES a declared type is a question ABOUT that type, and
|
||||
// must still reach its declaration file at full weight — so record the
|
||||
// files those declarations live in and exempt them from the
|
||||
// declaration-only penalty below (CG-28). Only PRECISE tokens count, by
|
||||
// the same NL-stopword reasoning as the seeding above: "…the file body…"
|
||||
// must not exempt a `Body` interface it never meant to name. Kept
|
||||
// separate from `namedSeedIds`, which is callable-only by construction —
|
||||
// a type never becomes a named seed, so it cannot be the guard here.
|
||||
if (isPreciseToken(t)) {
|
||||
for (const n of raw) {
|
||||
if (DECLARATION_KINDS.has(n.kind) && n.name.toLowerCase() === t.toLowerCase()) {
|
||||
namedTypeFiles.add(n.filePath);
|
||||
}
|
||||
}
|
||||
}
|
||||
let cands = raw
|
||||
.filter((n) => CALLABLE.has(n.kind) && !isTestPath(n.filePath))
|
||||
.sort((a, b) => (bodyLines(b) > 1 ? 1 : 0) - (bodyLines(a) > 1 ? 1 : 0) || bodyLines(b) - bodyLines(a));
|
||||
@@ -3572,10 +3620,18 @@ export class ToolHandler {
|
||||
// DO-NOT-EDIT banner and nothing in its name) down-ranks the same way
|
||||
// `.pb.go` always has (#1500). Covers the whole subgraph, not just the
|
||||
// grouped files, because the graph-mass penalty below is keyed on it too.
|
||||
const isGeneratedCandidate = cg.generatedFilePredicate(new Set([
|
||||
const penaltyCandidates = new Set([
|
||||
...fileGroups.keys(),
|
||||
...[...subgraph.nodes.values()].map((n) => n.filePath),
|
||||
]));
|
||||
]);
|
||||
const isGeneratedCandidate = cg.generatedFilePredicate(penaltyCandidates);
|
||||
// Second bounded probe over the same set: files declaring nothing but types
|
||||
// that nothing in the index depends on (CG-28). A query that NAMED one of
|
||||
// those types is asking about the declaration, so its file is exempt and
|
||||
// ranks at full weight.
|
||||
const isAmbientDeclaration = cg.ambientDeclarationFilePredicate(penaltyCandidates);
|
||||
const isDampedDeclaration = (filePath: string): boolean =>
|
||||
isAmbientDeclaration(filePath) && !namedTypeFiles.has(filePath);
|
||||
|
||||
/**
|
||||
* Rank penalty for a file, applied to its relevance score AND (below) to its
|
||||
@@ -3583,9 +3639,19 @@ export class ToolHandler {
|
||||
* score alone would leave the #1500 case unfixed: the generated CRUD carries
|
||||
* MORE graph mass than the hand-written use-case, and graph mass outranks
|
||||
* score in the comparator.
|
||||
*
|
||||
* Generated and ambient-declaration are taken as the STRONGER of the two,
|
||||
* never multiplied: a generated `.d.ts` has one property — "not the
|
||||
* implementation" — that both signals happen to see, and charging it twice is
|
||||
* how a file gets cliffed out of answers where it is genuinely relevant
|
||||
* (CG-28). The low-value multiplier is orthogonal (a test file that is also
|
||||
* generated is two independent reasons) and still compounds.
|
||||
*/
|
||||
const rankPenalty = (filePath: string): number =>
|
||||
(isGeneratedCandidate(filePath) ? GENERATED_RANK_PENALTY : 1)
|
||||
Math.min(
|
||||
isGeneratedCandidate(filePath) ? GENERATED_RANK_PENALTY : 1,
|
||||
isDampedDeclaration(filePath) ? AMBIENT_DECLARATION_RANK_PENALTY : 1,
|
||||
)
|
||||
* (isLowValue(filePath) ? LOW_VALUE_RANK_PENALTY : 1);
|
||||
|
||||
for (const [filePath, group] of fileGroups) {
|
||||
@@ -3931,6 +3997,7 @@ export class ToolHandler {
|
||||
spine: group.nodes.some((n) => flow.pathNodeIds.has(n.id)),
|
||||
lowValue: isLowValue(fp),
|
||||
generated: isGeneratedCandidate(fp),
|
||||
ambientDeclaration: isAmbientDeclaration(fp),
|
||||
penalty: rankPenalty(fp),
|
||||
kinds: kindMix(group.nodes),
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user