feat(mcp): relevance scoring overhaul for explore — kill incidental name-collision matches (CG-10, #1500)
Explore's per-file relevance awarded +50/+10/+3/+1 by match class and admitted
anything scoring >= 3. Neither half held up: the tier said HOW a symbol reached
us, never whether the match was evidence, and an absolute floor admits noise on
any repo where the top file scores 50+. Three scripts/agent-eval/*.mjs harnesses
took 63% of this repo's own "how does explore allocate its output budget" answer
on nothing but an unused `const explore` and a `const BUDGET`.
Four levers:
- KIND WEIGHT (RELEVANCE_KIND_WEIGHT): callables and types 1.0, members ~0.5,
variable/constant/parameter 0.15-0.35. A weak-kind symbol with no usage edge
anywhere in the graph (`contains` excluded — nesting is not usage) drops to
0.08. Only weak kinds in the top two tiers pay for the DB probe; the subgraph's
own edges answer most cases free. No measurable latency change (210 vs 211
ms/call, n=12 interleaved).
- PERIPHERAL CAP: nodes >=2 hops from any match accumulate into a bucket capped
at 5. Uncapped they added a flat +1 each, so a file grew more relevant by being
bigger — parse-session.mjs reached 22 off one constant plus twelve unrelated
symbols.
- RANK PENALTY: generated files x0.3, low-value x0.5, applied to the score AND
the graph mass. Score alone would not have fixed #1500 — the generated CRUD
carries MORE graph mass than the hand-written use-case, and graph mass outranks
score in the comparator. Self-normalizing, never a hard exclusion.
- RELATIVE FLOOR: clamp(topScore * 0.2, 1, 10). Capped at one full-strength
direct match so concentration elsewhere can never exclude one (without it a
named-seed-heavy file pushed the floor to 21 and dropped a file the agent had
named by class name). Backfills to 3 candidates when it would leave fewer, and
drops the evidence requirement rather than return nothing at all.
excludeLowValueFiles was dead config — declared per tier, read nowhere; the
test/spec exclusion has been unconditional for a while. Removed. The real gap was
the detector: `isLowValue` anchored on a leading `/`, so a repo-ROOT `test/` dir
(express, cobra, most of npm and Go) never matched — express's routing question
spent 59% of its envelope on three test files. Anchored at `^` too, and the
filter now runs before the floor and judges "are there other candidates?" on the
whole gather.
Measured before/after on the same indexes (baseline bd86ad2):
- payroll-go fixture: generated 57.4% -> 23.5%; answer 25.6% -> 61.5%; cycle.go
delivered 0 -> 38.9%. Generated ranks #3/#4, was #1/#2.
- self-query fixture: eval scripts 72% -> 0%; tools.ts ranks #1.
- express "route a request": 59% to test/* -> lib/application.js + lib/response.js
- cobra x3, codegraph "indexing pipeline": byte-identical (control)
Diagnostic gains a per-file penalty multiplier and NodeKind mix, so "why did this
file score X" is legible. Selection stages reordered to match the pipeline.
CG-6's gates flip from it.fails to live regressions except the byte-split ones,
which stay open for CG-12 (allocation still follows file size within the ranked
set).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
bd86ad2061
commit
a3898cdc70
@@ -196,11 +196,17 @@ describe('codegraph_explore allocation diagnostic', () => {
|
||||
// Totals: envelope vs budget, and the file-selection funnel with its floor.
|
||||
expect(out).toMatch(/envelope [\d,]+ chars delivered · [\d,]+ allocated of [\d,]+ budget/);
|
||||
expect(out).toMatch(/hard ceiling [\d,]+/);
|
||||
expect(out).toMatch(/files [\d,]+ grouped .*past score floor \(>=\d+\).*in output \(maxFiles \d+\)/);
|
||||
// The funnel runs low-value filter → score floor; the floor is fractional
|
||||
// now that scoring is kind-weighted (CG-10).
|
||||
expect(out).toMatch(
|
||||
/files [\d,]+ grouped .*past low-value filter .*past score floor \(>=[\d.]+\).*in output \(maxFiles \d+\)/,
|
||||
);
|
||||
// Per-file columns.
|
||||
expect(out).toMatch(/#\s+alloc%\s+deliv%\s+bytes\s+score\s+graph\s+hits\s+flags\s+render\s+file/);
|
||||
expect(out).toMatch(/#\s+alloc%\s+deliv%\s+bytes\s+score\s+graph\s+hits\s+pen\s+flags\s+render\s+file/);
|
||||
expect(out).toContain('src/session.ts');
|
||||
expect(out).toMatch(/\d+\.\d%/);
|
||||
// Kind mix — what each file's score was bought with.
|
||||
expect(out).toMatch(/kinds: (?:\w+:\d+ ?)+/);
|
||||
});
|
||||
|
||||
it('appends one JSON report per call to a sidecar path', async () => {
|
||||
@@ -222,7 +228,8 @@ describe('codegraph_explore allocation diagnostic', () => {
|
||||
expect(report.budget.maxOutputChars).toBeGreaterThan(0);
|
||||
expect(report.envelope.chars).toBeGreaterThan(0);
|
||||
expect(report.selection.scoreFloor).toBeGreaterThan(0);
|
||||
expect(report.selection.filesGrouped).toBeGreaterThanOrEqual(report.selection.filesPastScoreFloor);
|
||||
expect(report.selection.filesGrouped).toBeGreaterThanOrEqual(report.selection.filesPastLowValueFilter);
|
||||
expect(report.selection.filesPastLowValueFilter).toBeGreaterThanOrEqual(report.selection.filesPastScoreFloor);
|
||||
expect(report.selection.filesPastScoreFloor).toBeGreaterThanOrEqual(report.selection.filesRanked);
|
||||
expect(report.selection.filesRanked).toBeGreaterThanOrEqual(report.selection.filesInFinalOutput);
|
||||
expect(report.selection.filesInFinalOutput).toBeGreaterThan(0);
|
||||
|
||||
Reference in New Issue
Block a user