feat(resolution): gin middleware-chain synthesizer + Opus 4.8 benchmark refresh (#547)

* fix(agent-eval): detect idle by content-stability, not spinner absence

Opus 4.8's extended-thinking TUI shows no spinner / interrupt hint / timer while it streams its final answer — those appear only during the thinking and tool-use phases. The old detector treated ~5s of not-busy + prompt-present as done, so it killed interactive runs mid-answer, silently truncating both arms of the tmux A/B (low tool counts; the final assistant message left as a mid-investigation preamble). Now a run is done only when the captured pane stops changing for ~8s; while streaming, the pane changes every poll so stability never accrues. BUSY_RE stays as the immediate busy-reset for the thinking/tool/live-timer phase. Content-stability is model-agnostic — it survives future spinner re-wordings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(readme): refresh VS Code benchmark on v0.9.7 + Opus 4.8

Re-ran the VS Code A/B (headless median-of-4) on the current build and model. Cost savings held at 26% ($0.66->$0.89), but token/time/tool-call savings narrowed (78->63%, 52->20%, 85->69%) because Opus 4.8's without-CodeGraph arm explores far more efficiently than 4.7's did (16 tool calls vs 55, no Explore-subagent fan-out); the WITH arm is unchanged at 5 calls / 0 reads. Recomputed the average row and noted that the VS Code row is now a different model/version epoch than the other six.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(resolution): synthesize gin middleware-chain edges (Next -> registered handlers)

Gin runs its entire handler chain through one dynamic line in (*Context).Next -- c.handlers[c.index](c), a slice-index dispatch tree-sitter can't resolve. So callees(Next) dead-ended at the len() helper and the flow ServeHTTP -> handleHTTPRequest -> Next stopped at the exact symbol a 'how does the middleware chain work' question is about, sending the agent to re-query and Read/grep (a measured gin WITH-arm rabbit-hole: 2/4 headless runs spiraled to ~5min, one mis-firing the opt-in Workflow orchestration tool). Find the chain dispatcher (a Go method invoking a handlers slice by index) and link it -> every HandlerFunc registered via .Use/.GET/.../.Handle, so callees(Next) and trace(ServeHTTP, handler) connect end-to-end. Gated on the dispatcher existing (inert on non-gin Go repos), named handlers only (inline closures skipped), capped; provenance heuristic / synthesizedBy gin-middleware-chain, registeredAt = the registration site. Validated: gin callees(Next) now surfaces Logger/Recovery/ErrorLogger + handlers (node count stable at 2,544; 5 precise edges); agent A/B (headless median-of-4, Opus 4.8) flipped gin from -58% cost / -129% time to +7% cost / +35% tokens / +8% time / 38% tool calls, all 4 WITH runs clean (0 Read/Grep/Bash). 167/167 unit tests pass incl. the new gin-middleware-chain test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(readme): publish uniform Opus 4.8 benchmark + per-repo breakdown accordion

Refresh all 7 benchmark rows to the v0.9.7 / Opus 4.8 headless median-of-4 (was a mix of the 4.8 VS Code row + six 4.7 rows). New average 18% cheaper / 51% fewer tokens / 16% faster / 57% fewer tool calls; headline + methodology note updated 4.7->4.8. The gap is smaller than the prior 4.7 numbers because Opus 4.8's native grep/read is more efficient (the without-arm no longer fans out into large Explore-subagent sweeps) -- not a codegraph regression; CodeGraph still cuts tool calls and tokens on all 7 repos, with cost marginal/negative only on django + okhttp. Adds a top-level 'Per-repo breakdown' accordion (per-metric Time/Reads/Grep-Bash/Tool calls/Tokens/Cost, WITH vs WITHOUT, per repo) directly below the condensed summary; methodology/queries/why-wins move to a second accordion.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(changelog): note Gin middleware-chain synthesizer under [Unreleased]

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-05-28 23:41:11 -05:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 7a75c82dd9
commit f58de8a391
6 changed files with 350 additions and 38 deletions
@@ -182,7 +182,7 @@ Status legend: ✅ done+validated · 🔬 hole identified · ⬜ not started.
| Python | Django ORM | QuerySet → SQL compiler | R | ✅ |
| Python | Django / DRF (views) | url → view → model | R + X | ✅ url→view (`path`/`url`/`as_view`) + **DRF `router.register`→ViewSet** (realworld S / wagtail M / saleor L); ORM QuerySet→SQL (prior work). 🔬 signals (`post_save`→receiver), DRF viewset CRUD actions (inherited), saleor GraphQL resolvers |
| Python | Flask / FastAPI | request → route → handler → dependency | R + X | ✅ **Flask: handler resolved across intervening decorators (`@login_required`) + stacked `@x.route` lines** (microblog S 6→27, redash L decorator routes 6/6); **FastAPI: empty-path router-root routes `@router.get("")` incl. multi-line** (realworld S 12→20 / Netflix dispatch L **290/290 100%**) + **bare-name builtin guard** — a handler named after a Python builtin method (`index`/`get`/`update`/`count`…) was filtered as a builtin and lost its route→handler edge. + **Flask-RESTful `add_resource(Resource,'/x')` → Resource class** (redash 6→**77**) + **tuple `methods=('GET',)`** (was mislabeled GET) + **broadened detection** (requirements/Pipfile/setup + subdir app-factory entrypoints — flask-realworld 0→**19**). 🔬 FastAPI `Depends()` dependency edges (light validation) |
| Go | Gin / chi / gorilla/mux / net-http | request → route → handler → service | X | ✅ **routes on ANY group var** (`v1.GET`, `PublicGroup.GET`) not just `r/router` (gin-vue-admin S→M 4→259 / realworld S / gitness L) — was missing all group-routed apps; named handlers resolve precisely. **gorilla/mux confirmed covered** by the any-receiver `HandleFunc`/`Handle` handling (subrouter-var `s.HandleFunc(...)` + namespaced handlers; `.Methods()` chain ignored). 🔬 inline `func(c){}` handlers (anonymous, body lost); subrouter/`PathPrefix` path-prefix not prepended (label only); gitness chi custom (26/321) |
| Go | Gin / chi / gorilla/mux / net-http | request → route → handler → service; middleware chain (`Use`→`Next`) | S + X | ✅ **routes on ANY group var** (`v1.GET`, `PublicGroup.GET`) not just `r/router` (gin-vue-admin S→M 4→259 / realworld S / gitness L) — was missing all group-routed apps; named handlers resolve precisely. **gorilla/mux confirmed covered** by the any-receiver `HandleFunc`/`Handle` handling (subrouter-var `s.HandleFunc(...)` + namespaced handlers; `.Methods()` chain ignored). + **gin middleware-chain synthesizer** (`ginMiddlewareChainEdges`): gin runs its entire chain through one dynamic line — `(*Context).Next` does `c.handlers[c.index](c)`, a slice-index dispatch tree-sitter can't resolve, so `callees(Next)` dead-ended at the `len()` helper (`safeInt8`) and the agent rabbit-holed re-querying it. Find the dispatcher (a Go method invoking a `handlers` slice by index) and link it → every HandlerFunc registered via `.Use`/`.GET`/…/`.Handle`; gated on the dispatcher existing (inert on non-gin Go repos), named handlers only (closures skipped), capped. gin L: `callees(Next)` now surfaces `Logger`/`Recovery`/`ErrorLogger`+handlers (node count stable 2,544; 5 precise edges with `registeredAt` wiring sites). **Agent A/B (headless median-of-4, Opus 4.8): gin flipped from codegraph 58% cost / 129% time (the rabbit-hole, incl. a stray `Workflow` mis-fire on 2/4 WITH runs) → +7% cost / +35% tokens / +8% time / 38% tool calls, all 4 WITH runs clean (0 Read/Grep/Bash, no Workflow, no duplicate calls).** 🔬 inline `func(c){}` handlers (anonymous, body lost); subrouter/`PathPrefix` path-prefix not prepended (label only); gitness chi custom (26/321) |
| Rust | Axum / actix / Rocket | request → route → handler | R + X | ✅ **Axum chained methods + namespaced handlers** — `.route("/x", get(h1).post(h2))` emitted only the first method+handler, and `get(mod::handler)` captured the module not the fn (realworld-axum S **12→19, 19/19**); balanced-paren scan + per-method nodes + last-`::`-segment handler. **Rocket attribute macros 550/556 (99%)** (Rocket repo L) — already strong. crates.io named axum routes resolve (6/8; rest are closures/var handlers; its API is mostly the utoipa `routes!` macro = frontier). Cargo-workspace module resolution (prior work). **actix builder API** `web::resource("/x").route(web::get().to(h))` / `.to(h)` / App `.route("/x", web::get().to(h))` (actix-examples **51→128 routes, 35→112 resolved**) — was the dominant actix style and fully missed (the handler is in `.to(h)`, not `get(h)`). 🔬 actix `web::scope("/api")` prefix (not prepended to nested resource paths) + anonymous `.to` closure handlers |
| Java | Spring | request → @RestController → @Autowired service → repo | R + X | ✅ **bare `@GetMapping`/`@PostMapping` + class `@RequestMapping` prefix join → route→method** (realworld S / mall M / halo L) — was missing all path-less method mappings; DI controller→service resolves (name + dir) + **interface→impl dispatch synthesizer** (`interfaceOverrideEdges`: a class's `implements`/`extends` → link each interface/base method → its same-name override; JVM-gated, capped, **overload-aware**; mall **310** / halo **734** synth edges, node count unchanged) so trace follows controller→service-**interface**→**impl** instead of dead-ending at the abstract method — `trace("PmsProductController.getList","PmsProductServiceImpl.list")` connects in **3 hops** (probe-validated). + **field-injected concrete-bean trace** (#389): `this.<field>.method()` strips the `this.` receiver at extraction, and the resolver looks up the receiver name in the enclosing class's field declarations to get the declared type, then resolves the method on it — closes the controller→bean hop when the field-name doesn't capitalize to the type (`@Resource(name="userBO") UserBO userbo` → `userbo.toLogin2()` reaches `UserBO.toLogin2`). + **`@Value("${k}")` / `@ConfigurationProperties(prefix="X")` → application.{yml,yaml,properties}** binding with Spring's relaxed binding (kebab↔camel↔snake), incl. `${k:default}`. mall-tiny S: 11/11 `@Value` resolved. ⚠️ **agent A/B null** (n=2: the agent went context→explore→Read and never invoked `trace`, so the synth edges weren't exercised — adoption-gated, the recurring wall; see `docs/benchmarks/call-sequence-analysis.md`). The fix is correct + improves trace/callees/impact/context connectivity regardless; agent-visible read reduction needs trace adoption. 🔬 Spring Data JPA derived queries (`findByEmail`) — metaprogramming frontier; `@PropertySource` external files; Spring Cloud Config; mapper-class simple-name collisions across packages (dropped to avoid mis-resolution) |
| Java | MyBatis (XML mappers) | DAO interface method → `<select\|insert\|update\|delete id="X">` SQL | R (XML extract) + S (Java↔XML synthesizer) | ✅ **XML mapper as first-class language** (#389) — `src/extraction/mybatis-extractor.ts` parses files containing `<mapper namespace="...">`; emits one method-shaped node per statement qualified `<namespace>::<id>` + `<sql id="X">` fragments + `<include refid>` references. Non-mapper XML (pom, log4j) → file node only. `mybatisJavaXmlEdges` synthesizer indexes Java methods by `<ClassName>::<methodName>` and joins to XML qualified names by suffix-match — ambiguous simple-name collisions dropped (precision over recall). mall-tiny S **6/6 custom-SQL mapper methods bridge** to their XML statements; full enterprise chain `trace(controller.action → mapper.method-xml)` connects across controller / service-iface / impl / mapper / XML. 🔬 cross-mapper `<include>` via unqualified refid; MyBatis Plus dynamic methods (`BaseMapper<T>` CRUD inherited from framework, not in project); annotation-driven mappers (`@Select("SELECT ...")` on Java methods — the SQL lives in the annotation, not XML) |