refactor(eval): rename /audit skill to /agent-eval
Renames the `.claude/skills/audit/` directory and all internal references to `agent-eval`, aligning the skill name with the `/agent-eval` command it invokes.
This commit is contained in:
Colby McHenry
@@ -0,0 +1,74 @@
|
||||
---
|
||||
name: agent-eval
|
||||
description: Benchmark CodeGraph retrieval quality on a real codebase by comparing agent behavior with vs without CodeGraph. Use when the user runs /agent-eval or asks to test, benchmark, audit, or validate a codegraph version (the local dev build or a published npm version) against a language's repo.
|
||||
---
|
||||
|
||||
# CodeGraph Quality Audit
|
||||
|
||||
Measures how much CodeGraph helps an agent versus plain grep/read, for a chosen
|
||||
codegraph version on a chosen real-world repo. Drives the harness in
|
||||
`scripts/agent-eval/`.
|
||||
|
||||
## Prerequisites
|
||||
- `tmux` 3+, a logged-in `claude` CLI, `node`, `git` (macOS/Linux).
|
||||
- Run from the codegraph repo root.
|
||||
|
||||
## Workflow
|
||||
|
||||
Copy this checklist:
|
||||
```
|
||||
- [ ] 1. Pick version (local or npm)
|
||||
- [ ] 2. Pick language
|
||||
- [ ] 3. Pick repo by size
|
||||
- [ ] 4. Pick harness (headless / tmux / both)
|
||||
- [ ] 5. Run audit.sh in the background
|
||||
- [ ] 6. Report results
|
||||
```
|
||||
|
||||
**Step 1 — version.** Ask with `AskUserQuestion`: which codegraph version to test.
|
||||
Offer "Local dev build" and "Latest published"; the free-text "Other" lets the
|
||||
user type a specific version (e.g. `0.7.10`). Map the answer to a VERSION token:
|
||||
- "Local dev build" → `local`
|
||||
- "Latest published" → `latest`
|
||||
- a typed version → that string (e.g. `0.7.10`)
|
||||
|
||||
**Step 2 — language.** Read `.claude/skills/agent-eval/corpus.json`. Ask with
|
||||
`AskUserQuestion` which language to test, listing the languages that have entries.
|
||||
|
||||
**Step 3 — repo.** From the chosen language's entries, ask which repo. Label each
|
||||
option with its size and file count, e.g. `excalidraw — Medium (~600 files)`.
|
||||
Each entry carries the `repo` URL and a representative `question`.
|
||||
|
||||
**Step 4 — harness.** Ask with `AskUserQuestion` which harness to run, and map
|
||||
the answer to a MODE token:
|
||||
- "Headless" → `headless` — `claude -p` with stream-json: exact tokens/cost and a
|
||||
clean tool sequence (2 runs, fast, no TTY).
|
||||
- "Interactive (tmux)" → `tmux` — drives the real Claude TUI in tmux: faithful
|
||||
Explore-subagent behavior, metrics from session logs (2 runs, slower).
|
||||
- "Both" → `all` — headless + interactive (4 runs).
|
||||
|
||||
**Step 5 — run.** Launch in the background (sets the version, clones if missing,
|
||||
wipes + re-indexes, runs the chosen arms — several minutes):
|
||||
```bash
|
||||
scripts/agent-eval/audit.sh <VERSION> <repo-name> <repo-url> "<question>" <MODE>
|
||||
```
|
||||
|
||||
**Step 6 — report.** When the job finishes, read the log and report per arm:
|
||||
- Headless (`parse-run.mjs`): total tool calls, file `Read`s, Grep/Bash,
|
||||
codegraph-tool calls, duration, **total cost**.
|
||||
- Interactive (`parse-session.mjs`): the `VERDICT: codegraph_explore used Nx |
|
||||
Read N | Grep/Bash N` and `TOKENS:` lines.
|
||||
|
||||
Lead with cost + tool/Read counts — they are the reliable signals; raw token
|
||||
in/out are confounded by subagent delegation and prompt caching. State whether
|
||||
codegraph reduced effort and whether both arms reached a correct answer.
|
||||
|
||||
## Notes
|
||||
- The index is rebuilt every run (`audit.sh` wipes `.codegraph`) — different
|
||||
versions extract differently, so an index must be served by the same binary
|
||||
that built it.
|
||||
- `audit.sh` temporarily mutates the global `codegraph` install for the test,
|
||||
then restores your dev link via `local-install.sh`.
|
||||
- Corpus repos are cloned to `/tmp/codegraph-corpus` (reused if already present).
|
||||
- Add or edit repos in `corpus.json` (fields: `name`, `repo`, `size`, `files`,
|
||||
`question`).
|
||||
@@ -0,0 +1,63 @@
|
||||
{
|
||||
"_comment": "Test corpus for /agent-eval. Add entries freely. size: Small (<~150 files), Medium (~150-1500), Large (>~1500). 'question' is a representative architectural question that exercises cross-file understanding.",
|
||||
"TypeScript": [
|
||||
{ "name": "ky", "repo": "https://github.com/sindresorhus/ky", "size": "Small", "files": "~25", "question": "How does ky implement request retries and timeouts?" },
|
||||
{ "name": "excalidraw", "repo": "https://github.com/excalidraw/excalidraw", "size": "Medium", "files": "~600", "question": "How does Excalidraw render and update canvas elements?" },
|
||||
{ "name": "vscode", "repo": "https://github.com/microsoft/vscode", "size": "Large", "files": "~10000", "question": "How does the extension host communicate with the main process?" }
|
||||
],
|
||||
"JavaScript": [
|
||||
{ "name": "express", "repo": "https://github.com/expressjs/express", "size": "Small", "files": "~50", "question": "How does Express route a request through its middleware stack?" }
|
||||
],
|
||||
"Go": [
|
||||
{ "name": "cobra", "repo": "https://github.com/spf13/cobra", "size": "Small", "files": "~50", "question": "How does cobra parse commands and flags?" },
|
||||
{ "name": "gin", "repo": "https://github.com/gin-gonic/gin", "size": "Medium", "files": "~150", "question": "How does gin route requests through its middleware chain?" },
|
||||
{ "name": "terraform", "repo": "https://github.com/hashicorp/terraform", "size": "Large", "files": "~4000", "question": "How does Terraform build and walk the resource dependency graph?" }
|
||||
],
|
||||
"Python": [
|
||||
{ "name": "click", "repo": "https://github.com/pallets/click", "size": "Small", "files": "~60", "question": "How does click parse command-line arguments into commands?" },
|
||||
{ "name": "flask", "repo": "https://github.com/pallets/flask", "size": "Medium", "files": "~90", "question": "How does Flask dispatch a request to a view function?" },
|
||||
{ "name": "django", "repo": "https://github.com/django/django", "size": "Large", "files": "~2700", "question": "How does Django's ORM build and execute a query from a QuerySet?" }
|
||||
],
|
||||
"Rust": [
|
||||
{ "name": "clap", "repo": "https://github.com/clap-rs/clap", "size": "Medium", "files": "~200", "question": "How does clap parse arguments against a derived command definition?" },
|
||||
{ "name": "tokio", "repo": "https://github.com/tokio-rs/tokio", "size": "Large", "files": "~700", "question": "How does tokio schedule and run async tasks on its runtime?" },
|
||||
{ "name": "deno", "repo": "https://github.com/denoland/deno", "size": "Large", "files": "~1500", "question": "How does Deno load and execute a TypeScript module?" }
|
||||
],
|
||||
"Java": [
|
||||
{ "name": "gson", "repo": "https://github.com/google/gson", "size": "Medium", "files": "~200", "question": "How does Gson serialize an object to JSON?" },
|
||||
{ "name": "okhttp", "repo": "https://github.com/square/okhttp", "size": "Medium", "files": "~640", "question": "How does OkHttp process a request through its interceptor chain?" },
|
||||
{ "name": "guava", "repo": "https://github.com/google/guava", "size": "Large", "files": "~3000", "question": "How does Guava's CacheBuilder build and configure a cache?" }
|
||||
],
|
||||
"Kotlin": [
|
||||
{ "name": "koin", "repo": "https://github.com/InsertKoinIO/koin", "size": "Medium", "files": "~300", "question": "How does Koin resolve and inject dependencies?" },
|
||||
{ "name": "leakcanary", "repo": "https://github.com/square/leakcanary", "size": "Medium", "files": "~250", "question": "How does LeakCanary detect and analyze a memory leak?" }
|
||||
],
|
||||
"Swift": [
|
||||
{ "name": "alamofire", "repo": "https://github.com/Alamofire/Alamofire", "size": "Small", "files": "~100", "question": "How does Alamofire build, send, and validate a request?" }
|
||||
],
|
||||
"C#": [
|
||||
{ "name": "serilog", "repo": "https://github.com/serilog/serilog", "size": "Medium", "files": "~250", "question": "How does Serilog route a log event to its sinks?" },
|
||||
{ "name": "jellyfin", "repo": "https://github.com/jellyfin/jellyfin", "size": "Large", "files": "~2500", "question": "How does Jellyfin scan and identify items in a media library?" }
|
||||
],
|
||||
"Ruby": [
|
||||
{ "name": "sinatra", "repo": "https://github.com/sinatra/sinatra", "size": "Small", "files": "~60", "question": "How does Sinatra match a request to a route handler?" },
|
||||
{ "name": "discourse", "repo": "https://github.com/discourse/discourse", "size": "Large", "files": "~3000", "question": "How does Discourse create and render a new post?" }
|
||||
],
|
||||
"PHP": [
|
||||
{ "name": "slim", "repo": "https://github.com/slimphp/Slim", "size": "Small", "files": "~80", "question": "How does Slim handle a request through its middleware?" },
|
||||
{ "name": "laravel", "repo": "https://github.com/laravel/framework", "size": "Large", "files": "~3000", "question": "How does Laravel resolve and dispatch a route to a controller?" }
|
||||
],
|
||||
"C": [
|
||||
{ "name": "redis", "repo": "https://github.com/redis/redis", "size": "Large", "files": "~600", "question": "How does Redis parse and dispatch a client command?" }
|
||||
],
|
||||
"C++": [
|
||||
{ "name": "json", "repo": "https://github.com/nlohmann/json", "size": "Small", "files": "~100", "question": "How does nlohmann::json parse a JSON string into a value?" },
|
||||
{ "name": "grpc", "repo": "https://github.com/grpc/grpc", "size": "Large", "files": "~3000", "question": "How does gRPC dispatch an incoming RPC to its handler?" }
|
||||
],
|
||||
"Dart": [
|
||||
{ "name": "flutter", "repo": "https://github.com/flutter/flutter", "size": "Large", "files": "~6000", "question": "How does Flutter build and lay out a widget tree?" }
|
||||
],
|
||||
"Svelte": [
|
||||
{ "name": "shadcn-svelte", "repo": "https://github.com/huntabyte/shadcn-svelte", "size": "Medium", "files": "~600", "question": "How do shadcn-svelte components compose and apply their styling?" }
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user