Colby McHenryandClaude Opus 5
9efae0f8f2
A file that declares nothing but types and that nothing in the index depends on — a hand-written ambient `.d.ts` of global shims, vendored typings, module augmentation — cannot answer a flow question: no bodies, no call edges, no behaviour, nothing typed by it. But the identifiers it declares are exactly the generic ones a prose question uses (`Body`, `Message`, `ImageMetadata`, `ReadableStream`), so on term overlap it out-scored the implementation. Measured on the new fixture: rank #1 and 51% of delivered source, with the flow's own entry file pushed out of the response entirely. Measured first, per the issue: the Wrangler `worker-configuration.d.ts` that opened this is already handled by CG-25's banner detection, worth 15-46 points of envelope share across four flow queries. CG-25 credited; only the un-bannered case needed anything. `rankPenalty` now multiplies score and graph mass by 0.5 for such files, taken as the STRONGER of it and the generated penalty rather than multiplied — one property two signals see must not be charged twice. Detection is structural, not by extension, and four conditions deep. Two of them were forced by measurement: requiring every symbol to be type-level takes the corpus flag rate from 1-18% (which swept in Kotlin sealed classes, Rust mod.rs re-exports and django's locale tables) down to 0-4%; requiring that nothing depends on the file separates an ambient shim from a working types module, and without it the rule demoted displacement-ts's pipeline `types.ts` and broke the CG-31 gate. A query that NAMES a declared type is exempt, so a question about a type still reaches its declaration at full weight. Precise tokens only, so "…the file body…" cannot exempt a `Body` interface it never meant to name; this needs its own set because `namedSeedIds` is callable-only and a type never becomes one. Regression evidence in docs/benchmarks/explore-declaration-only-cg28.md: 6-repo envelope sweep byte-identical against a clean baseline build, zero ambient files reach the candidate set on VS Code across five queries, corpus flag rate 0-0.74%, both allocation fixtures PASS, full suite 2,978 green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
150 lines
8.4 KiB
Markdown
150 lines
8.4 KiB
Markdown
# Deterministic measurement — declaration-only files in the explore envelope (task CG-28)
|
||
|
||
**Date:** 2026-08-06 · **Baseline:** `feature/CG-24` @ `463f6e7` ·
|
||
**Harness:** `scripts/agent-eval/probe-decl-only.mjs` against a hermetic fixture
|
||
(`__tests__/fixtures/ambient-decls-ts/`, copied to a temp dir and indexed per run, so two runs on
|
||
one build give identical numbers), plus `probe-suite-envelope.mjs` and a corpus-wide flag-rate
|
||
survey for the regression side. No agent A/B: the claim under test is which FILES get selected and
|
||
in what order, and the agent runs are far too noisy to see that.
|
||
|
||
**Verdict, both halves:**
|
||
|
||
- **The motivating file is already handled — CG-25 credited.** The Wrangler `worker-configuration.d.ts`
|
||
that opened this issue is demoted by the generated penalty alone, worth 15–46 points of envelope
|
||
share on the four flow queries measured. No new mechanism needed for it.
|
||
- **The narrower gap is real and was fixed.** A declaration file with NO banner carried `pen 1.00`,
|
||
took **rank #1 and 51% of delivered source** on a prose flow query, and displaced the flow's own
|
||
entry file out of the response entirely. It is now damped — but only when nothing in the index
|
||
depends on it, which is the condition that makes the rule safe.
|
||
|
||
---
|
||
|
||
## The fixture
|
||
|
||
`__tests__/fixtures/ambient-decls-ts/` — an upload path (route → stream → metadata → queue) with
|
||
four declaration-shaped files competing against it for one envelope. All four declare nothing but
|
||
`interface`/`type_alias` and have no bodies; they differ only in the two properties under test.
|
||
|
||
| file | banner | depended on | lines |
|
||
|---|---|---|---|
|
||
| `types/worker-configuration.d.ts` | Wrangler | no | 271 |
|
||
| `types/platform-shims.d.ts` | none | no | 212 |
|
||
| `src/storage/types.ts` | none | **yes** (2 imports, 3 references) | 18 |
|
||
| implementation (`routes/`, `storage/`, `lib/`) | — | — | 37–71 each |
|
||
|
||
The declaration files carry the same generic identifiers the prose queries use — `Body`, `Message`,
|
||
`ImageMetadata`, `ReadableStream`, `Upload*` — which is the whole mechanism of the original report.
|
||
|
||
## Result 1 — what CG-25 is worth (the obsolescence leg)
|
||
|
||
Same fixture, same queries, one variable: `--variant strip-banner` deletes the two banner COMMENT
|
||
lines from `worker-configuration.d.ts` and changes nothing else, so the two declaration files become
|
||
indistinguishable to the ranker. Delivered share of the envelope for that file:
|
||
|
||
| query | with banner | banner stripped |
|
||
|---|---|---|
|
||
| flow-upload | not a candidate | 15.1% (2,271 chars) |
|
||
| flow-pipe | not a candidate | 38.5% (4,398 chars) |
|
||
| flow-generic | cliffed to a pointer, 0 chars | 35.1% (3,076 chars) |
|
||
| flow-queue | 9.4% via clusters (1,264 chars) | **46.1%, rank #1, whole file** (7,390 chars) |
|
||
|
||
The generated penalty alone is the difference between "rank #1 and nearly half the answer" and
|
||
"named in the not-shown list". **The file this issue was filed about needs nothing further.**
|
||
|
||
## Result 2 — the gap that survived
|
||
|
||
`platform-shims.d.ts` — hand-written, no banner — on the `feature/CG-24` tip:
|
||
|
||
| query | rank | score | pen | delivered |
|
||
|---|---|---|---|---|
|
||
| flow-upload | **#1** | 53.0 | 1.00 | 6,044 chars (**50.7%**) |
|
||
| flow-queue | **#1** | 21.0 | 1.00 | 6,044 chars (44.9%) |
|
||
|
||
On `flow-upload` the response carried three files and `src/routes/upload.ts` — the handler the
|
||
question is *about* — was not one of them. That is the CG-24 epic symptom, reproduced with no
|
||
generated banner anywhere in it.
|
||
|
||
Note also: `.pyi` is **not an indexed extension**, so Python stubs never enter the graph and cannot
|
||
take an envelope. That third of the issue's premise does not occur today.
|
||
|
||
## The mechanism, and why it is drawn this tight
|
||
|
||
`AMBIENT_DECLARATION_RANK_PENALTY` (0.5) multiplies score and graph mass in `rankPenalty`, for files
|
||
`QueryBuilder.getAmbientDeclarationPathsAmong` flags. Four conditions, all required — the first
|
||
three were the obvious rule, the fourth is the one that makes it safe:
|
||
|
||
1. declares ≥1 symbol;
|
||
2. **every** declared symbol is type-level (`interface`, `type_alias`, `enum`, `enum_member`,
|
||
`namespace`);
|
||
3. originates no `calls`/`instantiates` edge;
|
||
4. **nothing outside the file points at it.**
|
||
|
||
Conditions 2 and 4 were both forced by measurement, not taste:
|
||
|
||
**Why not just "no callables" (condition 2).** Surveyed across the corpus, a rule of "declares no
|
||
callable and calls nothing" flags **1.1%–18.0%** of files, and what it catches is real source:
|
||
okhttp's `SocketPolicy.kt` (19 declarations, a Kotlin sealed hierarchy), `BrotliInterceptor.kt`,
|
||
`tokio/src/runtime/mod.rs`, Alamofire's umbrella `Alamofire.swift`, and all 500+ of django's
|
||
`conf/locale/*/formats.py` constant tables. Requiring every symbol to be type-level drops that to
|
||
**0%–4%**.
|
||
|
||
**Why "nothing depends on it" (condition 4).** Without it the rule also flags
|
||
`__tests__/fixtures/displacement-ts/src/pipeline/types.ts` — pure interfaces, no bodies, structurally
|
||
identical to an ambient shim — and demoting it **broke the CG-31 displacement gate**, which is a
|
||
different invariant entirely. That file carries 13 inbound imports and 21 references: the pipeline
|
||
stages that answer a query about the pipeline are typed *by* it, so it is part of that answer's
|
||
structure. The ambient shims carry **zero** inbound edges — reachable by name, attached to nothing.
|
||
That is the real distinction, and the graph already holds it.
|
||
|
||
**The counter-case guard.** A query that NAMES a declared type is a question about the declaration,
|
||
so its file is exempt and ranks at full weight. Only shape-precise tokens count (the same
|
||
NL-stopword reasoning as named-seed selection) — "…the file **body**…" must not exempt a `Body`
|
||
interface it never meant to name. This needed its own set: `namedSeedIds` is callable-only by
|
||
construction, so a type can never become a named seed.
|
||
|
||
**No double-charging.** Generated and ambient-declaration are combined with `Math.min`, not
|
||
multiplied. A generated `.d.ts` has one property that two signals happen to see; charging it twice
|
||
(0.3 × 0.5 = 0.15) is how a file gets cliffed out of answers where it is genuinely relevant. The
|
||
low-value multiplier is orthogonal and still compounds.
|
||
|
||
## Result 3 — after the fix
|
||
|
||
| query | before | after |
|
||
|---|---|---|
|
||
| flow-upload | rank **#1**, 50.7% | rank **#2**, 38.6% — and `src/routes/upload.ts` now delivered (2,417 chars) |
|
||
| flow-queue | rank **#1**, 44.9% | rank **#3**, 41.3% |
|
||
| flow-pipe / flow-generic | not a candidate | unchanged |
|
||
| type-shim (`UploadStorage StoredUploadObject ImageMetadataShim`) | rank #1, `pen 1.00` | **unchanged** — exempt |
|
||
| type-prose (*what does the UploadStorage interface declare…*) | rank #1, `pen 1.00` | **unchanged** — exempt |
|
||
|
||
The byte share falls less than the rank does, and that is the correct outcome rather than a weak
|
||
fix: on this fixture every implementation file already delivers its entire contents, so the
|
||
declaration file is filling envelope nobody else needs. What it was actually taking was a **file
|
||
slot** — which is why the entry file came back. The issue explicitly forbids suppression, and a
|
||
damped file is still a candidate, still named in the response, and one follow-up explore away.
|
||
|
||
## Regression evidence
|
||
|
||
- **`probe-suite-envelope.mjs`, 6 repos, new build vs a clean `feature/CG-24` baseline build:
|
||
byte-identical.** django 20,878 · excalidraw 19,652 · okhttp 18,870 · tokio 21,607 · gin 10,776 ·
|
||
alamofire 11,662 source chars, same file counts, on both builds.
|
||
- **VS Code** — the repo the issue names for `.d.ts` surface — across five flow and type queries:
|
||
**zero** ambient-declaration files reach the ranked candidate set, so the output cannot differ.
|
||
- **Corpus-wide flag rate:** django 0.00% · okhttp 0.00% · gin 0.00% · alamofire 0.00% ·
|
||
tokio 0.12% · vscode 0.53% · excalidraw 0.74%. What it catches is `global.d.ts`, `vite-env.d.ts`,
|
||
`css.d.ts`, unreferenced vendored headers and test fixtures — exactly the intended shape.
|
||
- `probe-allocation.mjs`: `payroll-go` PASS, `self-query` PASS.
|
||
- Full suite: **178 files, 2,978 passed**, 6 skipped.
|
||
|
||
The change is inert everywhere the shape does not occur, which is most places. That is the point:
|
||
the defect is real but rare, and the mechanism costs nothing where it does not apply.
|
||
|
||
## Reproducing
|
||
|
||
```bash
|
||
npm run build
|
||
node scripts/agent-eval/probe-decl-only.mjs # as committed
|
||
node scripts/agent-eval/probe-decl-only.mjs --variant strip-banner # what CG-25 is worth
|
||
npx vitest run __tests__/explore-declaration-only.test.ts # the standing gate
|
||
```
|