The three metrics existed but only run-all.sh printed them, one block per
run. ab-new-vs-baseline.sh — the harness that actually isolates a retrieval
change, both arms codegraph-on — grepped its parse output down to `by type`
and `Result`, so occupancy, sufficiency and allocation never reached the
maintainer running the A/B they were built for.
Both harnesses now print the three blocks under every run and end with one
compare-arms.mjs table: median [min–max] per arm across RUNS, sufficiency
pooled (it is per-CALL, so median-of-run-percentages would weight a 1-call
run like a 5-call one), allocation pooled by bytes and per run. The table is
"did it move?"; the per-run blocks stay the "why?" — only they name the query
that fell short and the file nothing cited. It reproduces the recorded CG-22
express result off logs already on disk: baseline 3/6 calls in the
`Read a file we returned` bucket at 82.0%, new 0/5 at 96.9%.
parse-bench-readme.mjs gets the same two metrics as a with-arm table, so the
CG-13 campaign aggregates all three rather than occupancy alone.
Also folds the CLI-block shim into no-cli-shim.sh and gives it to
ab-new-vs-baseline.sh. There it is not a with/without leak but an attribution
one, and it breaks all three metrics at once: output arriving through Bash is
charged to Bash in the occupancy table, and an explore issued through the CLI
is not a tool call at all, so it never reaches the sufficiency classifier or
the allocation parse. The run silently drops calls from every number.
The daemon pre-warm and the model policy are untouched.
Validated on one live gin arm (2 explores, 0 Read, all three blocks + table)
and against the cg22/cg15 and ab-readme logs. Selftest 68/68.
Killing ab-new-vs-baseline.sh mid-baseline-arm left the engine checked out at
the baseline ref with the post-baseline files deleted, so every later build in
the working tree was silently the OLD code.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ab-new-vs-baseline.sh now builds and indexes once per arm and runs the task
RUNS times (default 1), so the >=2-runs-per-arm rule costs one build instead
of N. Both arms run with CODEGRAPH_NO_PROMPT_HOOK=1 — the machine's ambient
front-load hook resolves to whatever is in dist/, a second uncontrolled
channel that confounds the tool-call counts — and point explore's CG-4
diagnostic at a per-arm sidecar.
parse-run.mjs gains --envelope/--answer: the per-file share of the explore
source envelope, parsed from the rendered markdown so it works on ANY build.
The CG-4 sidecar only exists post-CG-4, so it cannot measure the baseline arm;
this is the only view that measures both arms the same way. Folded into
parse-run.mjs rather than added as a new script on purpose: a new file named
after explore's budget scores into the self-query fixture's own corpus and
moved its answer share 59.9%% -> 47.9%%.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* feat(explore): announce dynamic-dispatch boundaries when a flow can't connect statically (#687)
When buildFlowFromNamedSymbols can't connect the agent's named symbols, scan
the disconnected symbols' bodies (query-time, deterministic, zero graph
mutation) for dynamic-dispatch forms — computed member calls, getattr,
reflection, typed message buses, runtime-keyed emits, Proxy — and announce
the exact site where the static path ends, with candidate runtime targets
when a dispatch key is statically visible. The honest alternative to
guessing edges: surface the boundary, don't fabricate the bridge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(agent-eval): ab-new-vs-baseline survives files added since the baseline ref
A single multi-file 'git checkout <ref> --' with one unknown pathspec checks
out nothing, so the baseline arm silently ran the NEW build. Check out
per-file and remove files that don't exist on the baseline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs(playbook): boundary surfacing as the mechanism floor for non-gateable dispatch (#687)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(explore): render a direct synthesized hop between two named symbols (#687)
A 2-node chain populates pathIds but renders nothing (Flow needs >=3), and
the dynamic-links section skipped its edge as 'already in the main chain' —
so a custom EventBus emit→handler connection was invisible. Skip-as-in-chain
now applies only when a chain actually renders, and the boundary scan treats
short-chain endpoints as connected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
All agent A/B arms now run claude --model sonnet --effort high by default
(MODEL/EFFORT env overrides exist). Sonnet is the deliberate floor model:
codegraph's users attach whatever host they already run (Cursor Composer,
Gemini, ...), and a stronger model's tool-use masks the salience problems a
weaker one exposes — what lands on Sonnet generalizes up; Opus/Fable-only
wins don't generalize down. Policy recorded in CLAUDE.md's validation
methodology; 11 hardcoded --model opus call sites across 9 eval scripts
switched to the env-overridable default.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Corrects the "run non-nested only" conclusion from #734. The codegraph server
is healthy (handshake ~165ms); the flakiness is that on a multi-step
implementation task the agent dives into Read/grep before codegraph finishes
its ~2-3s startup (worse under nested CPU contention), so it runs with no
codegraph. Fix: pre-warm a persistent daemon (high idle timeout) + skip the
startup re-exec (CODEGRAPH_WASM_RELAUNCHED=1) so claude connects before the
agent's first turn. claude's init snapshot can show status:"pending" even when
it then connects — judge by actual codegraph usage, not the init line.
ab-new-vs-baseline.sh now bakes in the pre-warm + skip-re-exec. Validated: a
clean A/B showed the new build's agent used codegraph 2x / 5 Reads vs the
baseline's 0 / 8 on the same fully-implemented task.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Running scripts/agent-eval against a `claude -p` spawned from within a Claude
Code session (nested, e.g. from a Bash tool call) makes the codegraph MCP
attach unreliable: the server is healthy (full handshake ~165ms) but the
nested client marks it status:"pending"/0-tools under CPU/timing contention,
so the agent silently runs with no codegraph. NO_DAEMON + `< /dev/null` don't
fix it — it's the nested client, not the server. Documented in CLAUDE.md's
validation methodology.
Adds ab-new-vs-baseline.sh: A/Bs a retrieval/steering change as new-build vs
baseline-build (both codegraph-on, isolating the change — vs run-all.sh's
with-vs-without), on a throwaway copy of an indexed repo. Run it in a real
terminal.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Colby McHenry
