Merge remote-tracking branch 'origin/main'
# Conflicts: # CHANGELOG.md
This commit is contained in:
@@ -0,0 +1,36 @@
|
||||
# AGENTS.md (docs/)
|
||||
|
||||
Nested Codex guidance under `docs/`. Loaded with the repo-root `AGENTS.md` when cwd is under `docs/` (Codex walks root to cwd; shared `project_doc_max_bytes` budget).
|
||||
|
||||
Root `AGENTS.md` already carries the non-negotiable retrieval principles (adapt-the-tool, explore budgets, end-to-end synthesis). This file holds the longer validation methodology and Excalidraw worked example that were moved out of root to fit the budget.
|
||||
|
||||
### Validation methodology (REQUIRED for every new language/framework)
|
||||
|
||||
For each **language × framework**, validate on **small, medium, and large** real repos with **≥3 different flow prompts** each:
|
||||
|
||||
1. **Pick the canonical flow** for the framework ("how does X reach Y": state→render, request→handler→view, query→SQL, action→reducer→store…).
|
||||
2. **Deterministic probes** (`scripts/agent-eval/probe-{node,explore}.mjs` against the built `dist/`): `codegraph_explore` with the flow's symbol names connects from→to end-to-end with no break (its Flow section shows the path); **no node explosion** (`select count(*) from nodes` stable before/after re-index); synthesized-edge **precision** spot-check (`select … where provenance='heuristic'`).
|
||||
3. **Agent A/B** (`scripts/agent-eval/run-all.sh <repo> "<Q>"`): with vs without codegraph, **≥2 runs/arm** (run-to-run variance is large — never conclude from n=1). Record **duration, total tool calls, Read, Grep**. Optional forced-Read-0 sufficiency proof via the block-read hook (`scripts/agent-eval/hook-settings.json`).
|
||||
- **Every run also reports three feedback metrics** — residual context occupancy, explore sufficiency (what the agent did NEXT after each explore), and allocation efficiency (share of returned bytes the answer cited) — under each run, plus a side-by-side arm table (`compare-arms.mjs`). Entry point: `docs/benchmarks/agent-eval-feedback-metrics.md`. Reading them: `Read a file we returned` is an allocation miss, `Read a file we did NOT return`/`Grep` is recall; allocation efficiency is **relative** (attribution is by citation) so it is only valid between builds on the same question; occupancy *shares* are Claude Code / 200k and don't transfer to another host — the arm ratio does.
|
||||
- **The `codegraph` CLI is blocked in every arm** (`no-cli-shim.sh`: sanitized PATH + a PreToolUse hook, shared by both harnesses). Without it 14 of 15 without-arm runs in one 7-repo pass reached codegraph through Bash. Check the contamination row before believing any number: `CLI calls that RETURNED output` > 0 invalidates the run (in a new-vs-baseline A/B it silently drops calls from all three metrics, since a CLI explore is not a tool call).
|
||||
- **Model policy — every A/B arm runs Claude with `--model sonnet --effort high`. Always. Never Opus/Fable.** All `scripts/agent-eval/*.sh` default to this (`MODEL`/`EFFORT` env override exists — don't raise it without an explicit reason from the maintainer). Two reasons, and the second matters more than cost: (a) Sonnet doesn't burn tokens; (b) **Sonnet is the deliberate floor model** — codegraph's real users attach it to whatever agent they already run (Cursor Composer, Gemini, etc.), so we validate on a "dumber" model on purpose: a stronger model's tool-use covers up the salience/sufficiency problems a weaker one exposes. An affordance that lands on Sonnet generalizes up to every host; one that only works on Opus/Fable doesn't generalize down to the agents most users actually have. Both arms always use the same model.
|
||||
- **MCP attach is a startup-latency issue, not a hard block.** On a multi-step task the agent dives into Read/grep before codegraph finishes its ~2-3s startup (worse when the eval is itself run nested inside a Claude session, under CPU contention), so it runs with no codegraph. Fix: **pre-warm a persistent daemon** for the target (`CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS` high; spawn `serve --mcp --path <target> </dev/null &`; wait for `.codegraph/daemon.sock`) **and skip the startup re-exec** (`CODEGRAPH_WASM_RELAUNCHED=1`) so claude connects before the agent's first turn. Don't trust claude's `init` snapshot — it can read `status:"pending"` / 0 tools even when it then connects; judge by actual codegraph usage in `parse-run.mjs`'s `by type`. To isolate a change — **new-build vs baseline-build, both codegraph-on** (vs run-all.sh's with-vs-without) — use `scripts/agent-eval/ab-new-vs-baseline.sh <indexed-repo> "<task>" [baseline-ref]` (it bakes in the pre-warm).
|
||||
4. **Pass bar:** a normal flow question reaches **~0 Read/Grep within the repo's explore-call budget**, runs **faster** than without-codegraph, and shows **no regression on a control repo**. Record the numbers in `docs/design/dynamic-dispatch-coverage-playbook.md` (the coverage matrix).
|
||||
|
||||
Full playbook + per-mechanism design: `docs/design/dynamic-dispatch-coverage-playbook.md` and `docs/design/callback-edge-synthesis.md`.
|
||||
|
||||
### Worked example — Excalidraw (TS/React, medium, 643 files)
|
||||
|
||||
The template to replicate per language/framework. Question: *"how does updating an element re-render the canvas on screen?"* (the full flow crosses three React boundaries: observer callback, `setState`→`render`, and JSX child).
|
||||
|
||||
| Stage | duration | Read | Grep | codegraph |
|
||||
|---|---|---|---|---|
|
||||
| Without codegraph | 115–139s | 9–10 | 10–11 | 0 |
|
||||
| Broken (explore-budget regression) | 131–139s | 5–10 | 3–5 | 6–14 |
|
||||
| Fixed (budget + msgs + synthesis) | 64–112s | 0–2 | 2–4 | 3–**10** |
|
||||
| + trace-first steering | **51–74s** | **0–2** | 0–4 | **3–4** |
|
||||
|
||||
n=4 unhooked runs/stage, same prompt. After steering flow questions to `codegraph_trace` first: **best run 0 Read / 0 Grep / 3 codegraph / 51s**; **2 of 4 fully clean** (0 Read, 0 Grep). Steering eliminated the over-drill variance — call count tightened from 3–10 to 3–4, trace adoption went 3/4 → 4/4, and the `search`+`callers` path-reconstruction floundering dropped to 0. Run-to-run variance is still real; report the range, never a single run. **Residual reads/greps are all the nonce data-flow** (`canvasNonce` — a local prop with no graph edges); that's the def-use/data-flow frontier, left deliberately uncovered (tracking every local would explode the graph). Validated: `trace(mutateElement, renderStaticScene)` connects in **6 hops** across all three boundaries (`mutateElement → triggerUpdate → [callback] triggerRender → [react-render] render → [jsx] StaticCanvas → renderStaticScene`), each hop showing inline source + the wiring site; node count stable at 9,289; 1 callback + 46 react-render + 280 jsx-render synthesized edges (no explosion, precision-checked).
|
||||
|
||||
|
||||
Also see: `docs/design/dynamic-dispatch-coverage-playbook.md`, `docs/design/callback-edge-synthesis.md`, `docs/benchmarks/call-sequence-analysis.md`, `docs/benchmarks/agent-eval-feedback-metrics.md`.
|
||||
@@ -278,10 +278,18 @@ Hooks PRESENT (port each exactly):
|
||||
but createNode's extractModifiers merge still runs, so `expect val` /
|
||||
`actual val` DO get decorators. Return true → the dispatcher runs
|
||||
`scanFnRefSubtree(node, 0)` (capture-only, halts at nested
|
||||
function/lambda types) and NEVER descends → **property initializers
|
||||
emit NO calls/instantiates refs anywhere** (`val SHARED = WidgetK(0)`
|
||||
→ nothing; `by lazy { compute() }` → nothing, the scan halts at the
|
||||
lambda_literal). Consequences pinned in `extract-torture.txt`.
|
||||
function/lambda types) and never descends on its own. **The hook itself
|
||||
then walks the property's RHS under the property's scope** — the named
|
||||
child after the `=` token plus a `property_delegate` — via
|
||||
`ctx.visitFunctionBody`, so `val SHARED = WidgetK(0)`, `val cb =
|
||||
Runnable { hit() }` and `by lazy { compute() }` all emit their calls
|
||||
FROM the property node (Go's #693 initializer walk, ported). The
|
||||
declaration's own children — modifiers, `val`/`var`, the name+type, an
|
||||
extension receiver's type and type parameters, `getter`/`setter` — are
|
||||
NOT walked: a same-line `val c get() = f()` still emits nothing, a
|
||||
next-line accessor still attributes to the class, and a
|
||||
hook-DECLINED destructuring RHS is still invisible.
|
||||
Consequences pinned in `extract-torture.txt`.
|
||||
2. **`lambda_literal` after a fun-interface ERROR (:139-143)** and
|
||||
3. **fun-interface misparse recovery (:145-214)** (ERROR/
|
||||
function_declaration shapes; `isFunInterfaceNode` :46; Pattern 1 walks
|
||||
@@ -391,7 +399,7 @@ Hooks ABSENT (the walker must NOT do these): `preParse`, `resolveName`,
|
||||
| `anonymous_initializer` (`init { }`) | no branch | recursed → its statements' calls → **`calls` refs FROM THE CLASS node**; its `val` locals → hook 'local' → nothing (pinned: `calls "register" from=class:WidgetK`) |
|
||||
| `secondary_constructor` | no branch | **NO constructor node**; recursed → body calls attribute to the CLASS (`calls "log" from=class:WidgetK`); the `constructor_delegation_call`'s value_arguments still feed fn-ref capture |
|
||||
| `getter`/`setter` as SIBLINGS (accessor on its own line) | no branch | recursed → accessor-body calls attribute to the CLASS (or file). See §Properties for the sibling/child split |
|
||||
| `object_literal` (`object : T { … }` initializer) | no branch anywhere | never a node; see §Body walker for the method-leak quirk |
|
||||
| `object_literal` (`object : T { … }` initializer) | no branch anywhere | never a node itself; inside a PROPERTY initializer the hook's walk reaches its `fun`s, which leak out as FUNCTIONS under the property (see §Body walker for the same method-leak quirk) |
|
||||
| `file_annotation` (`@file:JvmName("x")`) | no branch | recursed; its value_arguments feed fn-ref capture (string args → nothing). No decorates ref |
|
||||
| INSTANTIATION_KINDS (354-361) | **no kotlin member** | extractInstantiation:4610 is **UNREACHABLE** for kotlin — constructor calls `Foo()` are call_expressions → plain `calls` refs named `Foo` (capitalized). Kotlin emits **zero `instantiates` refs**, ever |
|
||||
| `impl_item`:1274 / property_signature:1282 / export_statement / swift property:1121 | never | not kotlin node kinds (the swift `property_declaration` branch at 1121-1193 is gated `language === 'swift'` — kotlin property_declarations never enter it) |
|
||||
@@ -659,7 +667,8 @@ refs — kotlin emits NO instantiates, §dispatch table); backticked
|
||||
### Static-member / value-read refs (4750-4808) — kotlin IS in STATIC_MEMBER_LANGS (345-347)
|
||||
|
||||
Called ONLY from the body walker (5218) — top-level/class-scope reads emit
|
||||
nothing (hook-consumed property initializers doubly so).
|
||||
nothing — EXCEPT a property initializer, which the hook now walks through
|
||||
visitFunctionBody under the property's own scope (§Properties).
|
||||
`navigation_expression` ∈ MEMBER_ACCESS_TYPES (326). Mechanics:
|
||||
|
||||
- callee-of-call skip (4772-4778): parent ∈ callTypes AND parent.namedChild(0)
|
||||
@@ -850,12 +859,13 @@ unwrap/ungatedModes/addressOfOnly.
|
||||
- Capture points: visitNode:990 (top-level/class-scope call args),
|
||||
visitFunctionBody:5137, scanFnRefSubtree (hook-consumed property
|
||||
subtrees — `val x = register(::f)` captures via the inner
|
||||
value_arguments; **the scan halts at `lambda_literal` (610), so refs
|
||||
inside `by lazy { }`/trailing lambdas under a hook-consumed property are
|
||||
NOT captured**). **NOT captured anywhere: property/local initializer
|
||||
callable refs (`val m = ::caller`, `val bound = w::render`) — kotlin's
|
||||
dispatch has NO property_declaration/varinit key** (unlike SWIFT_SPEC —
|
||||
do not borrow it). Pinned: torture emits exactly three function_refs —
|
||||
value_arguments; **the scan halts at `lambda_literal` (610)**, but the
|
||||
hook's own initializer walk (§Properties) covers the same subtree with the
|
||||
PROPERTY on the stack, so refs inside `by lazy { }`/trailing lambdas are
|
||||
captured there — a shallow `::ref` reachable by BOTH is emitted twice, once
|
||||
from the class and once from the property). **NOT captured anywhere:
|
||||
local initializer callable refs — kotlin's dispatch has NO
|
||||
property_declaration/varinit key** (unlike SWIFT_SPEC — do not borrow it). Pinned: torture emits exactly three function_refs —
|
||||
`topLevel` (definedHere), `OtherClass::handle`, `this.caller`.
|
||||
- Flush gate (639-728): generated-file skip; `this.`-prefixed +
|
||||
`::`-containing candidates always flush; bare names need definedHere
|
||||
@@ -1039,7 +1049,7 @@ unwrap/ungatedModes/addressOfOnly.
|
||||
`Unit` / nullable / lambda return / `: T` generic leak; `expect fun`
|
||||
(bodiless + dec) / `actual fun`; tailrec self-call in expression body;
|
||||
top-level `val`/`var`/`const val`/`by lazy {}` (constant/variable kinds,
|
||||
NO initializer refs, NO capture inside the delegate lambda) +
|
||||
initializer + delegate refs attributed TO the property) +
|
||||
**destructuring (`val (a,b)` → nothing, both scopes)** + next-line-getter
|
||||
top-level `val` (getter calls → file/namespace); class with primary ctor
|
||||
(props invisible, defaults not walked), class-body val/var/computed
|
||||
|
||||
@@ -140,7 +140,7 @@ undefined; **no isConst means `const_item`/`static_item` extract as kind
|
||||
|---|---|---|
|
||||
| `function_item` (top level) | functionTypes, tree-sitter.ts:994 → extractFunction:1517 | not inside class-like at file scope → extractFunction; **first line of extractFunction (1522): if getReceiverType returns a value → extractMethod instead** (this is how impl-block fns become methods — impl_item does NOT push a scope) |
|
||||
| `function_signature_item` | same | in a trait body (trait pushed, class-like) → extractMethod; no `body` field → no body walk |
|
||||
| `struct_item` | structTypes:1059 → extractStruct:1869 | `body` field required: **unit structs `struct Unit;` have no body → NO node minted** (1876, `record_declaration` exemption is C#-only). Tuple structs have body `ordered_field_declaration_list` → extracted. `field_declaration` children make NO nodes (rust has no fieldTypes) — visitNode recurses into them and finds nothing |
|
||||
| `struct_item` | structTypes:1059 → extractStruct:1869 | ~~`body` field required: unit structs `struct Unit;` have no body → NO node minted~~ — **superseded: Rust now sets `allowBodilessStruct`, so `struct Unit;` mints a node with no members.** Rust has no forward declarations, so the bodiless skip (meant for C/C++) never applied here; the `record_declaration` exemption is the C# form of the same carve-out. Tuple structs have body `ordered_field_declaration_list` → extracted. `field_declaration` children make NO nodes (rust has no fieldTypes) — visitNode recurses into them and finds nothing |
|
||||
| `enum_item` | enumTypes:1064 → extractEnum:1914 | body `enum_variant_list`; `enum_variant` children → extractEnumMembers:1958 — **`name` field path: one `enum_member` node from `getChildByField(node,'name')`, then return** (variant payload bodies `B(u32)` / `C { x }` are never walked). Non-variant children (e.g. `attribute_item`) → visitNode (no-op) |
|
||||
| `trait_item` | interfaceTypes:1054 → extractInterface:1834 | kind `'trait'` (interfaceKind); extractInheritance sees the `trait_bounds` child (see below); body `declaration_list` children visited with the trait pushed → fn items become methods with QN `Trait::name` via nodeStack |
|
||||
| `impl_item` | dedicated branch:1273-1276 → extractRustImplItem:5690 | emits the implements back-reference (below); **skipChildren stays false** → the `declaration_list` is then visited normally by the loop at 1295 (that's how impl members are reached; impl pushes NOTHING on the nodeStack) |
|
||||
@@ -490,7 +490,7 @@ inner `array_expression`, but `const CB: fn() = handler;` captures nothing
|
||||
## Gates (per plan §5, no exceptions)
|
||||
|
||||
- **Torture fixture `torture.rs`** (+ CRLF variant, derived in-memory), pinning
|
||||
at minimum: unit struct (NO node) / tuple struct / field struct; enum with
|
||||
at minimum: unit struct (node, no members) / tuple struct / field struct; enum with
|
||||
unit+tuple+struct variants; trait with supertraits incl. a SCOPED one
|
||||
(`fmt::Debug` — dropped) + `function_signature_item` + default method +
|
||||
associated type/const (no node; const value call attributes to trait);
|
||||
|
||||
Reference in New Issue
Block a user