docs(agent-eval): evals must run non-nested + add ab-new-vs-baseline.sh (#734)
Running scripts/agent-eval against a `claude -p` spawned from within a Claude Code session (nested, e.g. from a Bash tool call) makes the codegraph MCP attach unreliable: the server is healthy (full handshake ~165ms) but the nested client marks it status:"pending"/0-tools under CPU/timing contention, so the agent silently runs with no codegraph. NO_DAEMON + `< /dev/null` don't fix it — it's the nested client, not the server. Documented in CLAUDE.md's validation methodology. Adds ab-new-vs-baseline.sh: A/Bs a retrieval/steering change as new-build vs baseline-build (both codegraph-on, isolating the change — vs run-all.sh's with-vs-without), on a throwaway copy of an indexed repo. Run it in a real terminal. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
7175dc456c
commit
28c5268ee3
@@ -137,6 +137,7 @@ For each **language × framework**, validate on **small, medium, and large** rea
|
||||
1. **Pick the canonical flow** for the framework ("how does X reach Y": state→render, request→handler→view, query→SQL, action→reducer→store…).
|
||||
2. **Deterministic probes** (`scripts/agent-eval/probe-{node,explore}.mjs` against the built `dist/`): `codegraph_explore` with the flow's symbol names connects from→to end-to-end with no break (its Flow section shows the path); **no node explosion** (`select count(*) from nodes` stable before/after re-index); synthesized-edge **precision** spot-check (`select … where provenance='heuristic'`).
|
||||
3. **Agent A/B** (`scripts/agent-eval/run-all.sh <repo> "<Q>"`): with vs without codegraph, **≥2 runs/arm** (run-to-run variance is large — never conclude from n=1). Record **duration, total tool calls, Read, Grep**. Optional forced-Read-0 sufficiency proof via the block-read hook (`scripts/agent-eval/hook-settings.json`).
|
||||
- **Run agent-evals in a REAL terminal — NEVER nested inside a Claude Code session** (don't spawn `claude -p` from a Bash tool call). The codegraph MCP server is healthy (full `initialize`→`tools/list` handshake ~165ms, daemon and in-process modes both fine), but a nested `claude -p` marks it `status:"pending"` / 0 tools under CPU/timing contention and the agent silently runs with no codegraph — it can connect early in a session, then degrade to consistent failure as nested spawns pile up. `CODEGRAPH_NO_DAEMON=1` and `< /dev/null` do NOT fix it (it's the nested client, not the server). Confirm via `parse-run.mjs` (`codegraph tools exposed: 0` = void run). To isolate a change — **new-build vs baseline-build, both codegraph-on** (vs run-all.sh's with-vs-without) — use `scripts/agent-eval/ab-new-vs-baseline.sh <indexed-repo> "<task>" [baseline-ref]`.
|
||||
4. **Pass bar:** a normal flow question reaches **~0 Read/Grep within the repo's explore-call budget**, runs **faster** than without-codegraph, and shows **no regression on a control repo**. Record the numbers in `docs/design/dynamic-dispatch-coverage-playbook.md` (the coverage matrix).
|
||||
|
||||
Full playbook + per-mechanism design: `docs/design/dynamic-dispatch-coverage-playbook.md` and `docs/design/callback-edge-synthesis.md`.
|
||||
|
||||
Reference in New Issue
Block a user