The A/B arms reported cost, tokens, time and tool counts for one headless
question. They could not report what issue #1500 actually measured: how much of
the context window a tool's responses still occupy once the question is
answered, which every later turn is then charged for.
parse-run.mjs now measures that. Tokens are measured, not estimated: for each
assistant request, input + cache_read + cache_creation is the exact token count
of its whole prompt, so consecutive requests differ by exactly what was appended
between them. That delta is priced against the characters in the gap, calibrated
on gaps that are >=80% tool result. Explore output lands near 2.3 chars/token, so
the usual bytes/4 estimate would have under-counted it by ~40%.
Content also leaves the window, so residual is tracked apart from contributed:
a compact_boundary clears the resident set, and a mid-run context drop is
micro-compaction, which sheds the oldest tool results first and is applied FIFO.
run-all.sh takes "Q1||Q2||Q3" and runs them as one resumed session, one segment
file per turn; parse-run.mjs stitches the segments back together. bench-readme.sh
now runs each README repo as a three-turn session (CG_TURNS=1 restores the
single-question form). parse-bench-readme.mjs reports the arms' retrieval
residual side by side -- codegraph's responses against the without-arm's
Read/Grep/Bash -- in absolute tokens, share of context, and share of window, and
says so explicitly when the rows it aggregated were single-turn.
Two transcript traps are handled and documented at the call site: Claude Code
emits one assistant event per content block, all carrying the same usage (summing
per event double-counts every turn with both thinking and a tool_use), and the
streamed output_tokens is a partial snapshot.
Occupancy lives in parse-run.mjs and is imported by the aggregator rather than
extracted to a module -- a new scripts/agent-eval/*.mjs scores into the
self-query fixture's own corpus and moves its numbers.
Programs, sections/paragraphs (reconstructed extents over the grammar's
flat header stream), PERFORM/THRU/GO TO/CALL call edges, COPY copybook
imports incl. standalone .cpy fragments, DATA DIVISION records/fields/
88-levels with write-site impact references, and CICS flows: EXEC
LINK/XCTL program targets (literal + same-file VALUE deref), EXEC SQL
INCLUDE, and pseudo-conversational RETURN/START TRANSID hops resolved
to the owning program via a CICS framework resolver. Fixed and free
source format (free format via a scanner wide-mode sentinel).
Grammar: vendored wasm built from a patched yutaro-sakamoto/
tree-sitter-cobol (EXEC blocks as an external-scanner token, copybook
fragment entry point, single-quote continuation, COPY REPLACING
pseudo-text, NOT=, CALL GIVING, ENTRY, FREE, bitwise ops, abbreviated
relations, COBOL-2002 usages, and more). Patch + provenance + upstream
PR draft in docs/grammars/. Parse health: AWS CardDemo 43/44 native
(upstream: 9/31), 44/44 through preParse; copybooks 28/29; CobolCraft
free-format 17/17 (upstream: 0); NIST COBOL85 unchanged at 373/382.
Copybook members resolve to files like C includes (basename index,
name-matcher short-circuit so compiler-supplied members stay honestly
unresolved): CardDemo imports 5 -> 285. Impact proof: ACCT-CURR-BAL
(CVACT01Y copybook) surfaces its 4 writer programs cross-file.
Also: run-all.sh now neutralizes the ambient prompt-hook in both A/B
arms (CODEGRAPH_NO_PROMPT_HOOK=1); COBOL corpus entries for agent-eval.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
All agent A/B arms now run claude --model sonnet --effort high by default
(MODEL/EFFORT env overrides exist). Sonnet is the deliberate floor model:
codegraph's users attach whatever host they already run (Cursor Composer,
Gemini, ...), and a stronger model's tool-use masks the salience problems a
weaker one exposes — what lands on Sonnet generalizes up; Opus/Fable-only
wins don't generalize down. Policy recorded in CLAUDE.md's validation
methodology; 11 hardcoded --model opus call sites across 9 eval scripts
switched to the env-overridable default.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Replaces the old interactive publish.js script with two Claude skills and
a full agent-evaluation harness:
- `.claude/skills/audit/` — `/audit` skill drives `scripts/agent-eval/audit.sh`
to benchmark retrieval quality (with vs. without codegraph) on a chosen
real-world repo from the new `corpus.json` (17 repos across 14 languages).
- `.claude/skills/publish/` — `/publish` skill orchestrates the full release
workflow (preflight → changelog → confirmation gate → bump/build → npm
publish → GitHub release), replacing `publish.js`.
- `scripts/agent-eval/` — headless (`run-agent.sh`, `run-all.sh`) and
interactive tmux (`itrun.sh`) harnesses with stream-json parsers
(`parse-run.mjs`, `parse-session.mjs`) that report tool calls, token
usage, and a VERDICT line summarising codegraph_explore vs. Read/Grep counts.
- `run-interactive-test.md` — documents the two harnesses, idle-detection
approach, and what "good" agent behavior looks like after explore-first
guidance.
Colby McHenry
