Three additions to tighten the eval loop:
- offload-eval-styles.sh: new 4-arm eval (raw/refs/map/src) isolating the Worker's
output shape's effect on main-session tokens, latency, and accuracy. Delegation
blocked by default (DISALLOW=Agent) so variance from Haiku subagent spawning doesn't
contaminate the measurement.
- offload-eval-cost.mjs: cost/token analyzer that reads Claude Code's own per-model
accounting (modelUsage.costUSD) rather than re-deriving from raw token counts,
giving a correct main(Sonnet)/sub(Haiku) split with proper per-tier pricing.
- offload-eval-3arm.sh: adds DISALLOW env to block sub-agent delegation across all
arms, and REP_START to append reps to an existing run without clobbering earlier
jsonls (e.g. REP_START=4 REPS=3 → reps 4,5,6).
Also adds CODEGRAPH_OFFLOAD_STYLE forwarding to the managed gateway so the styles
eval can drive output shape end-to-end; the field is stripped before the upstream
model call and never sent to BYO endpoints.
Reproducible suite measuring the managed CodeGraph AI offload and the front-load
UserPromptSubmit hook (approach 1) vs raw codegraph and no-codegraph, across repo
sizes, on time / main-session tokens+cost / CodeGraph-AI tokens+cost / accuracy.
All agent arms run claude -p sonnet --effort high; eval-only, nothing shipped.
- offload-eval-setup.sh: clone + index 4 memory-probe-verified "not-trained-on" repos
(mtkruto/postybirb/shapeshift/trezor — small→large) so the no-codegraph baseline is honest.
- offload-eval-3arm.sh / -frontload.sh: one repo, the arms (offload/raw/nocg, frontload).
- offload-eval-matrix.sh / -frontload-matrix.sh: drive all 4 tiers.
- offload-eval-hook.mjs: the front-load hook (self-locates its engine; CG_FRONTLOAD_DEBUG to log).
- offload-eval-metrics.mjs / -judge.mjs (Sonnet) / -summarize.mjs: extract, score, aggregate.
- offload-eval-ground-truth.json: source-verified canonical flows (the judge's reference).
- offload-eval.md: usage + the 2026-06 findings (raw = the win; offload least-accurate;
front-load solves adoption but exposes explore's dynamic-dispatch gaps).
Scripts are path-portable (self-locating $HERE/$ENGINE; AGENT_EVAL_OUT scratch dir).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Colby McHenry
