The A/B arms reported cost, tokens, time and tool counts for one headless
question. They could not report what issue #1500 actually measured: how much of
the context window a tool's responses still occupy once the question is
answered, which every later turn is then charged for.
parse-run.mjs now measures that. Tokens are measured, not estimated: for each
assistant request, input + cache_read + cache_creation is the exact token count
of its whole prompt, so consecutive requests differ by exactly what was appended
between them. That delta is priced against the characters in the gap, calibrated
on gaps that are >=80% tool result. Explore output lands near 2.3 chars/token, so
the usual bytes/4 estimate would have under-counted it by ~40%.
Content also leaves the window, so residual is tracked apart from contributed:
a compact_boundary clears the resident set, and a mid-run context drop is
micro-compaction, which sheds the oldest tool results first and is applied FIFO.
run-all.sh takes "Q1||Q2||Q3" and runs them as one resumed session, one segment
file per turn; parse-run.mjs stitches the segments back together. bench-readme.sh
now runs each README repo as a three-turn session (CG_TURNS=1 restores the
single-question form). parse-bench-readme.mjs reports the arms' retrieval
residual side by side -- codegraph's responses against the without-arm's
Read/Grep/Bash -- in absolute tokens, share of context, and share of window, and
says so explicitly when the rows it aggregated were single-turn.
Two transcript traps are handled and documented at the call site: Claude Code
emits one assistant event per content block, all carrying the same usage (summing
per event double-counts every turn with both thinking and a tool_use), and the
streamed output_tokens is a partial snapshot.
Occupancy lives in parse-run.mjs and is imported by the aggregator rather than
extracted to a module -- a new scripts/agent-eval/*.mjs scores into the
self-query fixture's own corpus and moves its numbers.
Replaces the old interactive publish.js script with two Claude skills and
a full agent-evaluation harness:
- `.claude/skills/audit/` — `/audit` skill drives `scripts/agent-eval/audit.sh`
to benchmark retrieval quality (with vs. without codegraph) on a chosen
real-world repo from the new `corpus.json` (17 repos across 14 languages).
- `.claude/skills/publish/` — `/publish` skill orchestrates the full release
workflow (preflight → changelog → confirmation gate → bump/build → npm
publish → GitHub release), replacing `publish.js`.
- `scripts/agent-eval/` — headless (`run-agent.sh`, `run-all.sh`) and
interactive tmux (`itrun.sh`) harnesses with stream-json parsers
(`parse-run.mjs`, `parse-session.mjs`) that report tool calls, token
usage, and a VERDICT line summarising codegraph_explore vs. Read/Grep counts.
- `run-interactive-test.md` — documents the two harnesses, idle-detection
approach, and what "good" agent behavior looks like after explore-first
guidance.
Colby McHenry