Commit Graph
4 Commits
Author SHA1 Message Date
df6f4bec43 feat(explore): dynamic-dispatch boundary surfacing — announce where a flow ends instead of guessing edges (#687) (#835)
* feat(explore): announce dynamic-dispatch boundaries when a flow can't connect statically (#687)

When buildFlowFromNamedSymbols can't connect the agent's named symbols, scan
the disconnected symbols' bodies (query-time, deterministic, zero graph
mutation) for dynamic-dispatch forms — computed member calls, getattr,
reflection, typed message buses, runtime-keyed emits, Proxy — and announce
the exact site where the static path ends, with candidate runtime targets
when a dispatch key is statically visible. The honest alternative to
guessing edges: surface the boundary, don't fabricate the bridge.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(agent-eval): ab-new-vs-baseline survives files added since the baseline ref

A single multi-file 'git checkout <ref> --' with one unknown pathspec checks
out nothing, so the baseline arm silently ran the NEW build. Check out
per-file and remove files that don't exist on the baseline.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(playbook): boundary surfacing as the mechanism floor for non-gateable dispatch (#687)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(explore): render a direct synthesized hop between two named symbols (#687)

A 2-node chain populates pathIds but renders nothing (Flow needs >=3), and
the dynamic-links section skipped its edge as 'already in the main chain' —
so a custom EventBus emit→handler connection was invisible. Skip-as-in-chain
now applies only when a chain actually renders, and the boundary scan treats
short-chain endpoints as connected.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 11:26:33 -05:00
0682681175 chore(agent-eval): standing A/B model policy — sonnet + high effort, never Opus/Fable (#816)
All agent A/B arms now run claude --model sonnet --effort high by default
(MODEL/EFFORT env overrides exist). Sonnet is the deliberate floor model:
codegraph's users attach whatever host they already run (Cursor Composer,
Gemini, ...), and a stronger model's tool-use masks the salience problems a
weaker one exposes — what lands on Sonnet generalizes up; Opus/Fable-only
wins don't generalize down. Policy recorded in CLAUDE.md's validation
methodology; 11 hardcoded --model opus call sites across 9 eval scripts
switched to the env-overridable default.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-11 19:39:16 -05:00
0e2789ab71 docs(agent-eval): nested MCP attach is startup-latency, not a hard block (#735)
Corrects the "run non-nested only" conclusion from #734. The codegraph server
is healthy (handshake ~165ms); the flakiness is that on a multi-step
implementation task the agent dives into Read/grep before codegraph finishes
its ~2-3s startup (worse under nested CPU contention), so it runs with no
codegraph. Fix: pre-warm a persistent daemon (high idle timeout) + skip the
startup re-exec (CODEGRAPH_WASM_RELAUNCHED=1) so claude connects before the
agent's first turn. claude's init snapshot can show status:"pending" even when
it then connects — judge by actual codegraph usage, not the init line.

ab-new-vs-baseline.sh now bakes in the pre-warm + skip-re-exec. Validated: a
clean A/B showed the new build's agent used codegraph 2x / 5 Reads vs the
baseline's 0 / 8 on the same fully-implemented task.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 10:25:40 -04:00
28c5268ee3 docs(agent-eval): evals must run non-nested + add ab-new-vs-baseline.sh (#734)
Running scripts/agent-eval against a `claude -p` spawned from within a Claude
Code session (nested, e.g. from a Bash tool call) makes the codegraph MCP
attach unreliable: the server is healthy (full handshake ~165ms) but the
nested client marks it status:"pending"/0-tools under CPU/timing contention,
so the agent silently runs with no codegraph. NO_DAEMON + `< /dev/null` don't
fix it — it's the nested client, not the server. Documented in CLAUDE.md's
validation methodology.

Adds ab-new-vs-baseline.sh: A/Bs a retrieval/steering change as new-build vs
baseline-build (both codegraph-on, isolating the change — vs run-all.sh's
with-vs-without), on a throwaway copy of an indexed repo. Run it in a real
terminal.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-08 09:59:07 -04:00