diff --git a/docs/benchmarks/residual-context-occupancy.md b/docs/benchmarks/residual-context-occupancy.md new file mode 100644 index 0000000..bb88f90 --- /dev/null +++ b/docs/benchmarks/residual-context-occupancy.md @@ -0,0 +1,164 @@ +# Residual context occupancy + +**What it measures:** how many tokens of the context window a tool's responses +still occupy once the question has been answered — and therefore how much +headroom every following turn has to work in. + +This is the metric issue [#1500](https://github.com/colbymchenry/codegraph/issues/1500) +was actually about. The reporter was looking at a live Cursor session: explore's +output was still resident after the answer, so it was charged against everything +that came next. Our A/B harness ran one headless question to completion and +reported cost, tokens, time, and tool calls — none of which can see that. A +single-question run reports *throughput*; occupancy is a *stock*, and it only +starts costing anything on the turns that follow. + +The harness now measures it, over multi-turn sessions. + +--- + +## Running it + +```bash +# One repo, one three-turn session, both arms: +scripts/agent-eval/run-all.sh /tmp/codegraph-corpus/gin \ + "How does gin route requests through its middleware chain?||\ +Where is the 404 / no-route case handled in that same chain?||\ +What would I change to add a per-route middleware that runs before the global ones?" + +# The 7 README repos (default: 3 turns per session, RUNS=4 per arm): +CORPUS=/tmp/codegraph-corpus RUNS=2 scripts/agent-eval/bench-readme.sh +node scripts/agent-eval/parse-bench-readme.mjs /tmp/ab-readme +``` + +`||` separates turns. Turn 1 runs normally; each later turn `--resume`s the same +session, so the earlier turns' tool output is still in the window — which is the +entire point. Segments land in `run-