feat(eval): add agent-eval harness and /audit + /publish Claude skills
Replaces the old interactive publish.js script with two Claude skills and a full agent-evaluation harness: - `.claude/skills/audit/` — `/audit` skill drives `scripts/agent-eval/audit.sh` to benchmark retrieval quality (with vs. without codegraph) on a chosen real-world repo from the new `corpus.json` (17 repos across 14 languages). - `.claude/skills/publish/` — `/publish` skill orchestrates the full release workflow (preflight → changelog → confirmation gate → bump/build → npm publish → GitHub release), replacing `publish.js`. - `scripts/agent-eval/` — headless (`run-agent.sh`, `run-all.sh`) and interactive tmux (`itrun.sh`) harnesses with stream-json parsers (`parse-run.mjs`, `parse-session.mjs`) that report tool calls, token usage, and a VERDICT line summarising codegraph_explore vs. Read/Grep counts. - `run-interactive-test.md` — documents the two harnesses, idle-detection approach, and what "good" agent behavior looks like after explore-first guidance.
This commit is contained in:
Executable
+34
@@ -0,0 +1,34 @@
|
||||
#!/usr/bin/env bash
|
||||
# Headless Claude Code run against a repo with codegraph MCP, capturing the
|
||||
# full stream-json so we can see tool calls + token usage. Complements the
|
||||
# interactive itrun.sh: headless gives a clean per-tool breakdown + exact
|
||||
# tokens/cost, but defaults to the general-purpose subagent (not Explore).
|
||||
# To force the Explore path, ask for it in the prompt.
|
||||
#
|
||||
# Usage: run-agent.sh <repo-path> <label> "<prompt>"
|
||||
# Env: AGENT_EVAL_OUT (default /tmp/agent-eval), CG_BIN (codegraph dist binary)
|
||||
set -uo pipefail
|
||||
|
||||
REPO="$1"; LABEL="$2"; PROMPT="$3"
|
||||
CG_BIN="${CG_BIN:-$(command -v codegraph || echo /usr/local/bin/codegraph)}"
|
||||
OUT_DIR="${AGENT_EVAL_OUT:-/tmp/agent-eval}"; mkdir -p "$OUT_DIR"
|
||||
OUT="$OUT_DIR/run-${LABEL}.jsonl"
|
||||
|
||||
MCP_CONFIG=$(cat <<JSON
|
||||
{"mcpServers":{"codegraph":{"command":"${CG_BIN}","args":["serve","--mcp","--path","${REPO}"]}}}
|
||||
JSON
|
||||
)
|
||||
|
||||
echo "→ running [$LABEL] in $REPO"
|
||||
cd "$REPO" || exit 1
|
||||
|
||||
claude -p "$PROMPT" \
|
||||
--output-format stream-json --verbose \
|
||||
--permission-mode bypassPermissions \
|
||||
--model opus \
|
||||
--max-budget-usd 2 \
|
||||
--strict-mcp-config --mcp-config "$MCP_CONFIG" \
|
||||
> "$OUT" 2>"$OUT_DIR/run-${LABEL}.err"
|
||||
|
||||
echo "exit: $? | wrote $OUT ($(wc -l < "$OUT") lines)"
|
||||
node "$(cd "$(dirname "$0")" && pwd)/parse-run.mjs" "$OUT" 2>/dev/null || true
|
||||
Reference in New Issue
Block a user