docs(playbook): first P7 agent A/B numbers (proshop, Sonnet, 2 runs/arm) and the plan's P7 status

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01REFyW9hmNrxhwN5wxRoAkC
This commit is contained in:
Colby McHenry
2026-08-28 14:57:43 -05:00
co-authored by Claude Fable 5
parent bc45e9071d
commit 7f461269c4
2 changed files with 17 additions and 2 deletions
@@ -278,7 +278,7 @@ Status legend: ✅ done+validated · 🔬 hole identified · ⬜ not started.
(Verify the exact supported set against `src/extraction/languages/` and
`src/resolution/frameworks/` before starting — this table is a starting point.)
| TypeScript/JS | Next.js (App Router + Pages Router) | page → `<Link>` / `router.push` / `redirect` → page; page load → data; client component → `'use server'` action → DB → `redirect`; `route.ts` handler → DB → response | R + S | ✅ 2026-08-28 (`frameworks/nextjs.ts`, `next-router-synthesizer.ts`, Steps `load` trigger + server-action crossing): fixture `__tests__/nextjs.test.ts` end to end (Screens `routed`, the push attributed back under its condition, the action's redirect); `leerob/next-saas-starter` indexes its pages and links. 🔬 agent A/B (`--model sonnet`, ≥2 runs/arm) not run yet |
| TypeScript/JS | Express + React (MERN monorepo) | client `axios` / `fetch` literal path → own route → handler → Mongoose → response rows | S + R | ✅ 2026-08-28 (`tier-synthesizer.ts` `http-client` + Express mounts / chained `router.route()` / wrapped `const h = asyncHandler(…)` handlers / nested `package.json` detection): `bradtraversy/proshop_mern` 49 routes (19 pages + 30 endpoints), 23 client→route edges, every one spot-checked correct; `login → ⇢ POST /api/users/login → User.findOne → 401 rows → jwt.sign`. Node count stable across re-index. 🔬 A/B not run |
| TypeScript/JS | Express + React (MERN monorepo) | client `axios` / `fetch` literal path → own route → handler → Mongoose → response rows | S + R | ✅ 2026-08-28 (`tier-synthesizer.ts` `http-client` + Express mounts / chained `router.route()` / wrapped `const h = asyncHandler(…)` handlers / nested `package.json` detection): `bradtraversy/proshop_mern` 49 routes (19 pages + 30 endpoints), 23 client→route edges, every one spot-checked correct; `login → ⇢ POST /api/users/login → User.findOne → 401 rows → jwt.sign`. Node count stable across re-index. **Agent A/B (2026-08-28, `run-all.sh`, Sonnet/high, 2 runs/arm, warm daemon, CLI shim on — 0 CLI leaks):** *"How does submitting the login form reach the database, and what does the API respond with when the password is wrong?"* — with: **14s / 14s, 2 tool calls each, 0 Read, 0 Grep, 1 and 2 explores** (explore sufficiency: answered after 1 call in run 1, after 2 in run 2), answer names `submitHandler → login → POST /api/users/login → authUser → User.findOne → matchPassword → 401` end to end; without: **18s / 37s, 14 and 8 tool calls, 7 and 1 Read** (+7 Bash `cat`s in run 2, a subagent in run 1). Tokens processed 153k/165k vs 237k/369k. Pass bar met on the small repo (0 Read/Grep, faster, within 12 explores against a budget of 1). 🔬 medium/large (Ghost) and ≥3 prompts still to run |
| TypeScript/JS | NestJS queues / events / sockets | `queue.add('job')` → `@Process('job')`; `emit('x')` → `@OnEvent('x')`; `socket.emit` → `@SubscribeMessage` and `server.emit` → `socket.on` | S | ✅ 2026-08-28 (`tier-synthesizer.ts` `queue-job`, `event-bus`): `nestjs/nest` `sample/26-queues` (`transcode` → `handleTranscode`) and `sample/30-event-emitter` (`order.created` → its listener) exact, 0 wrong edges after the `e2e/` and generic-event guards; `immich-app/immich` 0 edges — a generated SDK client and a wrapped queue API carry no literal, so silence (correct). 🔬 A/B not run |
| Python | FastAPI (prefixed routers) | request → `Depends` → handler → session → `HTTPException` rows | R | ✅ 2026-08-28 `python.ts` `postExtract` composes `APIRouter(prefix=)` + literal `include_router(prefix=)`: `fastapi/full-stack-fastapi-template` 23 routes named `GET /items/{id}`, `POST /login/access-token` (were `GET /`); a computed mount prefix is skipped. 🔬 A/B not run |
| C# | ASP.NET Minimal API endpoint groups | request → group handler → `ISender.Send` → `TypedResults` rows | R | ✅ 2026-08-28 `csharp.ts` handler-first `MapPost(Handler[, path])` under the class + `$"/api/{groupName}"` head (`jasontaylordev/CleanArchitecture` shape); fixture end to end in `ui-steps-api-servers.test.ts` (`PUT /api/TodoItems/{id}` → `204 · 400` rows); `jasontaylordev/CleanArchitecture` 10 routes, all the app's endpoints (were 0). 🔬 A/B not run |
@@ -2,7 +2,8 @@
**Status:** plan, written 2026-08-28 at the end of the session that built the Steps view and the
readings it rests on (Expo + React Native app, `amniservices-mobile-app`). **Updated the same day, later
sessions: P0P6 are built** (see the per-item notes marked *Built*); P7's agent A/B numbers are open. Every claim about what a
sessions: P0P6 are built** (see the per-item notes marked *Built*); P7 has its first agent A/B (proshop, small) and
the medium / large rows are open. Every claim about what a
resolver emits *today* was verified against the source on this date — re-verify before building on it,
the resolvers move. What was learned building it, beyond the plan: the index keeps only the LAST
segment of a deep member call (`create` for `prisma.user.create`) and name-matches it — often to the
@@ -354,6 +355,8 @@ its `redirect` on the FILE node (Screens now re-attributes a file-scope navigati
`useActionState(signIn, …)` leaves no function-as-value edge (a plain call argument — Screens now falls back to the
functions that MENTION the value in the files importing it, read from the source; the principled fix is an extractor
fnRef rule for `useActionState` / `useFormState` / `startTransition` arguments, with the Rust kernel twin).
With both: 12 screens, 17 links, 8 origins — `/sign-in → /dashboard via Login > signIn WHEN userWithTeam.length !== 0 &&
isPasswordValid && redirectTo !== 'checkout'`, `/dashboard/security → /sign-in via deleteAccount WHEN isPasswordValid`.
*Where:* `resolution/frameworks/react.ts` (split a `nextjs.ts` out of it — the pages/app routing is
already there), a `next-router-synthesizer.ts` modelled on `expo-router-synthesizer.ts`.
@@ -421,6 +424,18 @@ names and the frameworks detected, so the viewer does not guess).
### P7 — Validation set and the numbers
*First numbers* (2026-08-28, later session) — the deterministic half is in the playbook §6 rows (Next.js, MERN, Nest
channels, FastAPI prefixed routers, ASP.NET endpoint groups: node counts, edge precision spot-checks, pictures). One agent
A/B so far, `bradtraversy/proshop_mern` (small, Express + React), `scripts/agent-eval/run-all.sh`, Sonnet/high, 2 runs per
arm, a daemon pre-warmed before each with-run, the CLI shim on (0 leaks): *"How does submitting the login form reach the
database, and what does the API respond with when the password is wrong?"* — with codegraph 14s / 14s, 2 tool calls, 0 Read,
0 Grep, 12 explores, the full path named (`submitHandler → login → POST /api/users/login → authUser → User.findOne →
matchPassword → 401`); without 18s / 37s, 14 / 8 tool calls, 7 / 1 Read plus Bash `cat`s and a subagent. Tokens 153k/165k vs
237k/369k. The pass bar (§4 of the playbook) holds on the small repo. **Still open:** the medium / large rows (Ghost,
immich, cal.com / twenty), ≥3 prompts per framework, and a control repo. The driver lives in the session scratchpad
(`ab-proshop.sh`: pre-warm `serve --mcp` with `CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS` high and `CODEGRAPH_WASM_RELAUNCHED=1`,
then `run-all.sh` per run with its own `AGENT_EVAL_OUT`); re-create it from the memory note.
Small fixtures live in the tests. For the real bar, index these and record the results in
`docs/design/dynamic-dispatch-coverage-playbook.md` (new rows) exactly as the playbook asks
(**≥3 flow prompts × small/medium/large, node count stable, synthesized-edge precision spot-check,