chore(agent-eval): standing A/B model policy — sonnet + high effort, never Opus/Fable (#816)

All agent A/B arms now run claude --model sonnet --effort high by default
(MODEL/EFFORT env overrides exist). Sonnet is the deliberate floor model:
codegraph's users attach whatever host they already run (Cursor Composer,
Gemini, ...), and a stronger model's tool-use masks the salience problems a
weaker one exposes — what lands on Sonnet generalizes up; Opus/Fable-only
wins don't generalize down. Policy recorded in CLAUDE.md's validation
methodology; 11 hardcoded --model opus call sites across 9 eval scripts
switched to the env-overridable default.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-06-11 19:39:16 -05:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 823ffd1c3d
commit 0682681175
10 changed files with 14 additions and 11 deletions
+1 -1
View File
@@ -72,7 +72,7 @@ run_one() { # arm-label, run-index, use-hook(0|1)
# array expansion otherwise, which would skip the no-hook arm's claude run.
( cd "$tgt" && claude -p "$TASK" \
--output-format stream-json --verbose --permission-mode bypassPermissions \
--model opus --max-budget-usd 4 --strict-mcp-config --mcp-config "$c" ${extra[@]+"${extra[@]}"} \
--model "${MODEL:-sonnet}" --effort "${EFFORT:-high}" --max-budget-usd 4 --strict-mcp-config --mcp-config "$c" ${extra[@]+"${extra[@]}"} \
</dev/null > "$OUT/run-$label-$idx.jsonl" 2>"$OUT/run-$label-$idx.err" )
node "$PARSE" "$OUT/run-$label-$idx.jsonl" 2>&1 | grep -E "by type|Result" || echo " (parse failed — see $OUT/run-$label-$idx.jsonl)"
pkill -9 -f "serve --mcp --path $tgt" 2>/dev/null