feat(mcp): relevance scoring overhaul for explore — kill incidental name-collision matches (CG-10, #1500)
Explore's per-file relevance awarded +50/+10/+3/+1 by match class and admitted
anything scoring >= 3. Neither half held up: the tier said HOW a symbol reached
us, never whether the match was evidence, and an absolute floor admits noise on
any repo where the top file scores 50+. Three scripts/agent-eval/*.mjs harnesses
took 63% of this repo's own "how does explore allocate its output budget" answer
on nothing but an unused `const explore` and a `const BUDGET`.
Four levers:
- KIND WEIGHT (RELEVANCE_KIND_WEIGHT): callables and types 1.0, members ~0.5,
variable/constant/parameter 0.15-0.35. A weak-kind symbol with no usage edge
anywhere in the graph (`contains` excluded — nesting is not usage) drops to
0.08. Only weak kinds in the top two tiers pay for the DB probe; the subgraph's
own edges answer most cases free. No measurable latency change (210 vs 211
ms/call, n=12 interleaved).
- PERIPHERAL CAP: nodes >=2 hops from any match accumulate into a bucket capped
at 5. Uncapped they added a flat +1 each, so a file grew more relevant by being
bigger — parse-session.mjs reached 22 off one constant plus twelve unrelated
symbols.
- RANK PENALTY: generated files x0.3, low-value x0.5, applied to the score AND
the graph mass. Score alone would not have fixed #1500 — the generated CRUD
carries MORE graph mass than the hand-written use-case, and graph mass outranks
score in the comparator. Self-normalizing, never a hard exclusion.
- RELATIVE FLOOR: clamp(topScore * 0.2, 1, 10). Capped at one full-strength
direct match so concentration elsewhere can never exclude one (without it a
named-seed-heavy file pushed the floor to 21 and dropped a file the agent had
named by class name). Backfills to 3 candidates when it would leave fewer, and
drops the evidence requirement rather than return nothing at all.
excludeLowValueFiles was dead config — declared per tier, read nowhere; the
test/spec exclusion has been unconditional for a while. Removed. The real gap was
the detector: `isLowValue` anchored on a leading `/`, so a repo-ROOT `test/` dir
(express, cobra, most of npm and Go) never matched — express's routing question
spent 59% of its envelope on three test files. Anchored at `^` too, and the
filter now runs before the floor and judges "are there other candidates?" on the
whole gather.
Measured before/after on the same indexes (baseline bd86ad2):
- payroll-go fixture: generated 57.4% -> 23.5%; answer 25.6% -> 61.5%; cycle.go
delivered 0 -> 38.9%. Generated ranks #3/#4, was #1/#2.
- self-query fixture: eval scripts 72% -> 0%; tools.ts ranks #1.
- express "route a request": 59% to test/* -> lib/application.js + lib/response.js
- cobra x3, codegraph "indexing pipeline": byte-identical (control)
Diagnostic gains a per-file penalty multiplier and NodeKind mix, so "why did this
file score X" is legible. Selection stages reordered to match the pipeline.
CG-6's gates flip from it.fails to live regressions except the byte-split ones,
which stay open for CG-12 (allocation still follows file size within the ranked
set).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
bd86ad2061
commit
a3898cdc70
@@ -2,8 +2,14 @@
|
||||
"$comment": [
|
||||
"Regression fixtures for GitHub issue #1500 / epic CG-1 — relevance-proportional",
|
||||
"explore budget allocation. Run them with `node scripts/agent-eval/probe-allocation.mjs`",
|
||||
"against a built dist/. BOTH FIXTURES FAIL TODAY: that is the point — they document",
|
||||
"the bug and become the pass gate for the allocation change (CG-10 + CG-12).",
|
||||
"against a built dist/.",
|
||||
"",
|
||||
"STATUS: CG-10 (relevance scoring) closed the RANKING half of both fixtures — nothing",
|
||||
"incidental reaches the envelope any more. What still fails is the BYTE SPLIT among the",
|
||||
"files that correctly ranked in, because the render loop spends by file size: a small",
|
||||
"weakly-relevant file ships whole while the strongly-relevant one is clipped at",
|
||||
"maxCharsPerFile. That is CG-12's gate. Each fixture's `baseline` records the pre-CG-10",
|
||||
"numbers; `afterCG10` records where it stands now.",
|
||||
"",
|
||||
"`groups` partitions the files explore rendered into `answer` (what the query is",
|
||||
"actually about) and `incidental` (what wins the envelope today on name collisions).",
|
||||
@@ -65,6 +71,16 @@
|
||||
"internal/usecase/payroll/cycle.go": 0.0
|
||||
},
|
||||
"verdict": "The workflow file is allocated the single largest slice (30.6%) and delivers ZERO — the hard ceiling drops its whole section. Generated CRUD takes 57.4% of what the agent actually receives; runPayrollCycleAll, BuildPayslip and the real Upsert never reach the response."
|
||||
},
|
||||
"afterCG10": {
|
||||
"measuredOn": "2026-08-04",
|
||||
"delivered": {
|
||||
"internal/usecase/payroll/cycle.go": 0.389,
|
||||
"internal/gen/fkit/payroll/payroll_cycle.go": 0.235,
|
||||
"internal/domain/payroll/payslip.go": 0.226,
|
||||
"internal/gen/fkit/payroll/payslip.go": 0.0
|
||||
},
|
||||
"verdict": "PASSES answerShareAtLeast (61.5%), incidentalShareAtMost (23.5%), topFileGroup, and the cycle.go + runPayrollCycleAll + real-Upsert needles. The generated files now rank #3/#4 instead of #1/#2 — kind weighting plus a 0.3x generated penalty on BOTH the score and the graph mass, which is the key the comparator sorts on. STILL FAILING for CG-12: payslip_builder.go ranks #6 against the tier's maxFiles of 4, so `func (s *Service) BuildPayslip` never renders."
|
||||
}
|
||||
},
|
||||
{
|
||||
@@ -106,6 +122,14 @@
|
||||
"scripts/agent-eval/offload-eval-cost.mjs": 0.0
|
||||
},
|
||||
"verdict": "The script corpus takes 71.8% of the delivered envelope (79.4% of what was allocated) against tools.ts's 18.5%, despite tools.ts scoring 46 vs 10, carrying 2.3x the graph mass and 3x the distinct term hits."
|
||||
},
|
||||
"afterCG10": {
|
||||
"measuredOn": "2026-08-04",
|
||||
"delivered": {
|
||||
"src/resolution/memory-budget.ts": 0.512,
|
||||
"src/mcp/tools.ts": 0.329
|
||||
},
|
||||
"verdict": "PASSES incidentalShareAtMost: every scripts/agent-eval/*.mjs file is gone (0.0%), which is CG-10's acceptance bar — their sole match was an unused file-scope `explore`/`BUDGET` constant, worth 0.08x weight, and the relative floor (8.2) then cut them. tools.ts ranks #1. STILL FAILING for CG-12: memory-budget.ts scores 18 to tools.ts's 41 yet takes 51.2% of the envelope to tools.ts's 32.9%, purely because it is small enough to ship whole while tools.ts is clipped at maxCharsPerFile. Allocation still follows file size, not relevance."
|
||||
}
|
||||
}
|
||||
]
|
||||
|
||||
Reference in New Issue
Block a user