test(agent-eval): persist offload accuracy/adoption eval harness + front-load hook
Reproducible suite measuring the managed CodeGraph AI offload and the front-load UserPromptSubmit hook (approach 1) vs raw codegraph and no-codegraph, across repo sizes, on time / main-session tokens+cost / CodeGraph-AI tokens+cost / accuracy. All agent arms run claude -p sonnet --effort high; eval-only, nothing shipped. - offload-eval-setup.sh: clone + index 4 memory-probe-verified "not-trained-on" repos (mtkruto/postybirb/shapeshift/trezor — small→large) so the no-codegraph baseline is honest. - offload-eval-3arm.sh / -frontload.sh: one repo, the arms (offload/raw/nocg, frontload). - offload-eval-matrix.sh / -frontload-matrix.sh: drive all 4 tiers. - offload-eval-hook.mjs: the front-load hook (self-locates its engine; CG_FRONTLOAD_DEBUG to log). - offload-eval-metrics.mjs / -judge.mjs (Sonnet) / -summarize.mjs: extract, score, aggregate. - offload-eval-ground-truth.json: source-verified canonical flows (the judge's reference). - offload-eval.md: usage + the 2026-06 findings (raw = the win; offload least-accurate; front-load solves adoption but exposes explore's dynamic-dispatch gaps). Scripts are path-portable (self-locating $HERE/$ENGINE; AGENT_EVAL_OUT scratch dir). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
6d5cb6b25c
commit
7ddd3fa7eb
Executable
+20
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# Drive the 3-arm campaign (offload/raw/nocg) across all 4 tiers, n reps each, into one
|
||||
# results.jsonl. Reads the canonical question per repo from offload-eval-ground-truth.json.
|
||||
# Env: REPS (default 3) AGENT_EVAL_OUT=<scratch dir>
|
||||
set -uo pipefail
|
||||
HERE="$(cd "$(dirname "$0")" && pwd)"
|
||||
OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}"
|
||||
GT="$HERE/offload-eval-ground-truth.json"
|
||||
REPS="${REPS:-3}"
|
||||
export RESULTS="$OUT/results.jsonl"
|
||||
: > "$RESULTS"
|
||||
for repo in mtkruto postybirb shapeshift trezor; do
|
||||
case "$repo" in mtkruto) tier=small;; postybirb) tier=medium;; shapeshift) tier=complex;; trezor) tier=large;; esac
|
||||
Q=$(node -e "console.log(JSON.parse(require('fs').readFileSync(process.argv[1],'utf8'))[process.argv[2]].question)" "$GT" "$repo")
|
||||
echo ""; echo "### $repo ($tier) $(date +%H:%M:%S)"
|
||||
bash "$HERE/offload-eval-3arm.sh" "$OUT/repos/$repo" "$tier" "$REPS" "$Q"
|
||||
done
|
||||
echo ""; echo "###### MATRIX DONE -> $RESULTS ($(wc -l < "$RESULTS") runs). Judge + summarize with:"
|
||||
echo " node $HERE/offload-eval-judge.mjs --results $RESULTS --truth $GT --out $OUT/judged.jsonl"
|
||||
echo " node $HERE/offload-eval-summarize.mjs $OUT/judged.jsonl"
|
||||
Reference in New Issue
Block a user