github/codegraph
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(mcp): relevance scoring overhaul for explore — kill incidental name-collision matches (CG-10, #1500)

Browse Source 
Explore's per-file relevance awarded +50/+10/+3/+1 by match class and admitted
anything scoring >= 3. Neither half held up: the tier said HOW a symbol reached
us, never whether the match was evidence, and an absolute floor admits noise on
any repo where the top file scores 50+. Three scripts/agent-eval/*.mjs harnesses
took 63% of this repo's own "how does explore allocate its output budget" answer
on nothing but an unused `const explore` and a `const BUDGET`.

Four levers:

- KIND WEIGHT (RELEVANCE_KIND_WEIGHT): callables and types 1.0, members ~0.5,
  variable/constant/parameter 0.15-0.35. A weak-kind symbol with no usage edge
  anywhere in the graph (`contains` excluded — nesting is not usage) drops to
  0.08. Only weak kinds in the top two tiers pay for the DB probe; the subgraph's
  own edges answer most cases free. No measurable latency change (210 vs 211
  ms/call, n=12 interleaved).

- PERIPHERAL CAP: nodes >=2 hops from any match accumulate into a bucket capped
  at 5. Uncapped they added a flat +1 each, so a file grew more relevant by being
  bigger — parse-session.mjs reached 22 off one constant plus twelve unrelated
  symbols.

- RANK PENALTY: generated files x0.3, low-value x0.5, applied to the score AND
  the graph mass. Score alone would not have fixed #1500 — the generated CRUD
  carries MORE graph mass than the hand-written use-case, and graph mass outranks
  score in the comparator. Self-normalizing, never a hard exclusion.

- RELATIVE FLOOR: clamp(topScore * 0.2, 1, 10). Capped at one full-strength
  direct match so concentration elsewhere can never exclude one (without it a
  named-seed-heavy file pushed the floor to 21 and dropped a file the agent had
  named by class name). Backfills to 3 candidates when it would leave fewer, and
  drops the evidence requirement rather than return nothing at all.

excludeLowValueFiles was dead config — declared per tier, read nowhere; the
test/spec exclusion has been unconditional for a while. Removed. The real gap was
the detector: `isLowValue` anchored on a leading `/`, so a repo-ROOT `test/` dir
(express, cobra, most of npm and Go) never matched — express's routing question
spent 59% of its envelope on three test files. Anchored at `^` too, and the
filter now runs before the floor and judges "are there other candidates?" on the
whole gather.

Measured before/after on the same indexes (baseline bd86ad2):
- payroll-go fixture: generated 57.4% -> 23.5%; answer 25.6% -> 61.5%; cycle.go
  delivered 0 -> 38.9%. Generated ranks #3/#4, was #1/#2.
- self-query fixture: eval scripts 72% -> 0%; tools.ts ranks #1.
- express "route a request": 59% to test/* -> lib/application.js + lib/response.js
- cobra x3, codegraph "indexing pipeline": byte-identical (control)

Diagnostic gains a per-file penalty multiplier and NodeKind mix, so "why did this
file score X" is legible. Selection stages reordered to match the pipeline.

CG-6's gates flip from it.fails to live regressions except the byte-split ones,
which stay open for CG-12 (allocation still follows file size within the ranked
set).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-04 00:02:45 -05:00
co-authored by Claude Opus 5
parent bd86ad2061
commit a3898cdc70
8 changed files with 897 additions and 101 deletions
Download Patch File Download Diff File Expand all files Collapse all files
__tests__/explore-allocation-1500.test.ts
+52 -32
View File
@@ -19,10 +19,12 @@
* runPayrollCycleAll → BuildPayslip → Upsert chain resolving end-to-end. If
* the fixture rots, these fail first and say so.
*
* 2. **Budget allocation** — the gate, written with `it.fails` because it
* DOCUMENTS A BUG THAT IS STILL OPEN. Vitest passes an `it.fails` test only
* while its body throws, so the suite is green today and goes RED the moment
* allocation is fixed (CG-10 scoring + CG-12 proportional bytes).
* 2. **Budget allocation** — the gate. CG-10 (relevance scoring) closed most of
* it: the generated CRUD now ranks and delivers BELOW the hand-written
* workflow, and those assertions are live regressions. What remains is
* `it.fails`, which DOCUMENTS THE PART STILL OPEN — vitest passes an
* `it.fails` test only while its body throws, so it goes RED the moment
* CG-12's proportional byte allocation lands.
* **When it goes red, delete the `.fails` — do not delete the test.**
*
* The same assertions run outside vitest, against the built dist and with the
@@ -202,54 +204,72 @@ describe('#1500 — generated Go CRUD beside a hand-written payroll workflow', (
const generatedShare = () => share((p) => p.startsWith(GENERATED_PREFIX));
/**
* BASELINE 2026-08-03 (very-tiny tier, 13,000-char budget): 23,020 chars
* allocated against it, cut to 16,011 by the 19,500 hard ceiling. The
* generated CRUD delivers 57.4%; the hand-written layer delivers 25.6%, all
* of it domain types. `cycle.go` is allocated the single largest slice
* (7,052 chars, 30.6%) and delivers ZERO — the ceiling drops its whole
* section — so runPayrollCycleAll, the hand-written BuildPayslip and the
* real Upsert never reach the agent at all.
* BASELINE 2026-08-03, BEFORE CG-10 (very-tiny tier, 13,000-char budget):
* 23,020 chars allocated, cut to 16,011 by the 19,500 hard ceiling. The
* generated CRUD delivered 57.4%; the hand-written layer 25.6%, all of it
* domain types. `cycle.go` was allocated the single largest slice (7,052
* chars, 30.6%) and delivered ZERO — the ceiling dropped its whole section —
* so runPayrollCycleAll, the hand-written BuildPayslip and the real Upsert
* never reached the agent.
*
* Each `it.fails` below passes ONLY while that is still true.
* ⚠ When one goes red, the bug is fixed: remove `.fails`, keep the test.
* AFTER CG-10 (relevance scoring): the generated files rank #3/#4 instead of
* #1/#2 — kind-weighted scoring plus a generated rank PENALTY on both the
* score and the graph mass, rather than the old tiebreak-at-equal-score.
* `cycle.go` now delivers 38.9% and the generated layer 23.5%. Four of the
* five gates below are green and are now live regressions.
*
* STILL OPEN, for CG-12 (proportional byte allocation): the render loop
* still allocates by FILE SIZE within the ranked set, and `maxFiles` is 4 at
* this tier — so `payslip_builder.go` ranks #6 and never renders, and
* `func (s *Service) BuildPayslip` is absent. Its `it.fails` passes ONLY
* while that is still true. ⚠ When it goes red, remove `.fails`, keep it.
*/
it.fails('CG-12 GATE: concentrates the envelope on the hand-written workflow', () => {
it('CG-10 GATE: concentrates the envelope on the hand-written workflow', () => {
expect(answerShare()).toBeGreaterThanOrEqual(0.55);
});
it.fails('CG-12 GATE: does not spend the envelope on the generated CRUD', () => {
it('CG-10 GATE: does not spend the envelope on the generated CRUD', () => {
expect(generatedShare()).toBeLessThanOrEqual(0.25);
});
it.fails('CG-12 GATE: delivers the workflow file it allocated the most bytes to', () => {
it('CG-10 GATE: ranks the generated CRUD below the hand-written workflow', () => {
// The #1500 report in one assertion: before CG-10 the generated layer both
// outscored AND out-delivered the use-case that implements the business rule.
expect(answerShare()).toBeGreaterThan(generatedShare());
});
it('CG-10 GATE: delivers the workflow file it allocated the most bytes to', () => {
expect(bytes.get('internal/usecase/payroll/cycle.go') ?? 0).toBeGreaterThan(0);
});
it.fails('CG-12 GATE: delivers the calculation the question asks about', () => {
expect(bytes.get('internal/usecase/payroll/payslip_builder.go') ?? 0).toBeGreaterThan(0);
});
it.fails('CG-12 GATE: puts the hand-written chain in the response, not its generated twin', () => {
// Bare `BuildPayslip`/`Upsert` also match the generated collisions — these
// needles are unique to the hand-written chain.
it('CG-10 GATE: puts the hand-written chain in the response, not its generated twin', () => {
// Bare `Upsert` also matches the generated collision — these needles are
// unique to the hand-written chain.
expect(response).toContain('runPayrollCycleAll');
expect(response).toContain('func (s *Service) BuildPayslip');
expect(response).toContain('s.store.Upsert(ctx, slip)');
});
it('records the shape of the failure so a regression is legible', () => {
// Not a gate — an assertion-free-ish snapshot of WHY the gates above fail,
// so a future change that shifts the numbers shows up in the diff rather
// than silently flipping an it.fails.
it.fails('CG-12 GATE: delivers the calculation the question asks about', () => {
// `payslip_builder.go` ranks #6; the tier's maxFiles is 4 and the render
// loop spends by file size, so it never gets bytes. Score-proportional
// allocation (CG-12) is what closes this.
expect(bytes.get('internal/usecase/payroll/payslip_builder.go') ?? 0).toBeGreaterThan(0);
expect(response).toContain('func (s *Service) BuildPayslip');
});
it('records the shape of the allocation so a regression is legible', () => {
// Not a gate — a snapshot of the split, so a future change that shifts the
// numbers shows up in the diff rather than silently flipping a gate.
const generated = generatedShare();
const answer = answerShare();
const workflow = bytes.get('internal/usecase/payroll/cycle.go') ?? 0;
expect({
generatedWinsEnvelope: generated > answer,
workflowFileDeliversNothing: workflow === 0,
workflowFileDelivers: (bytes.get('internal/usecase/payroll/cycle.go') ?? 0) > 0,
builderFileDelivers: (bytes.get('internal/usecase/payroll/payslip_builder.go') ?? 0) > 0,
}).toEqual({
generatedWinsEnvelope: true,
workflowFileDeliversNothing: true,
generatedWinsEnvelope: false,
workflowFileDelivers: true,
builderFileDelivers: false,
});
});
});