docs(agent-eval): evals must run non-nested + add ab-new-vs-baseline.sh (#734)

Running scripts/agent-eval against a `claude -p` spawned from within a Claude
Code session (nested, e.g. from a Bash tool call) makes the codegraph MCP
attach unreliable: the server is healthy (full handshake ~165ms) but the
nested client marks it status:"pending"/0-tools under CPU/timing contention,
so the agent silently runs with no codegraph. NO_DAEMON + `< /dev/null` don't
fix it — it's the nested client, not the server. Documented in CLAUDE.md's
validation methodology.

Adds ab-new-vs-baseline.sh: A/Bs a retrieval/steering change as new-build vs
baseline-build (both codegraph-on, isolating the change — vs run-all.sh's
with-vs-without), on a throwaway copy of an indexed repo. Run it in a real
terminal.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-06-08 09:59:07 -04:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 7175dc456c
commit 28c5268ee3
2 changed files with 84 additions and 0 deletions
+1
View File
@@ -137,6 +137,7 @@ For each **language × framework**, validate on **small, medium, and large** rea
1. **Pick the canonical flow** for the framework ("how does X reach Y": state→render, request→handler→view, query→SQL, action→reducer→store…).
2. **Deterministic probes** (`scripts/agent-eval/probe-{node,explore}.mjs` against the built `dist/`): `codegraph_explore` with the flow's symbol names connects from→to end-to-end with no break (its Flow section shows the path); **no node explosion** (`select count(*) from nodes` stable before/after re-index); synthesized-edge **precision** spot-check (`select … where provenance='heuristic'`).
3. **Agent A/B** (`scripts/agent-eval/run-all.sh <repo> "<Q>"`): with vs without codegraph, **≥2 runs/arm** (run-to-run variance is large — never conclude from n=1). Record **duration, total tool calls, Read, Grep**. Optional forced-Read-0 sufficiency proof via the block-read hook (`scripts/agent-eval/hook-settings.json`).
- **Run agent-evals in a REAL terminal — NEVER nested inside a Claude Code session** (don't spawn `claude -p` from a Bash tool call). The codegraph MCP server is healthy (full `initialize``tools/list` handshake ~165ms, daemon and in-process modes both fine), but a nested `claude -p` marks it `status:"pending"` / 0 tools under CPU/timing contention and the agent silently runs with no codegraph — it can connect early in a session, then degrade to consistent failure as nested spawns pile up. `CODEGRAPH_NO_DAEMON=1` and `< /dev/null` do NOT fix it (it's the nested client, not the server). Confirm via `parse-run.mjs` (`codegraph tools exposed: 0` = void run). To isolate a change — **new-build vs baseline-build, both codegraph-on** (vs run-all.sh's with-vs-without) — use `scripts/agent-eval/ab-new-vs-baseline.sh <indexed-repo> "<task>" [baseline-ref]`.
4. **Pass bar:** a normal flow question reaches **~0 Read/Grep within the repo's explore-call budget**, runs **faster** than without-codegraph, and shows **no regression on a control repo**. Record the numbers in `docs/design/dynamic-dispatch-coverage-playbook.md` (the coverage matrix).
Full playbook + per-mechanism design: `docs/design/dynamic-dispatch-coverage-playbook.md` and `docs/design/callback-edge-synthesis.md`.