diff --git a/.cursor/rules/codegraph.mdc b/.cursor/rules/codegraph.mdc index 00a3f81..17d144a 100644 --- a/.cursor/rules/codegraph.mdc +++ b/.cursor/rules/codegraph.mdc @@ -1,37 +1,22 @@ --- -description: CodeGraph MCP usage guide — when to use which tool +description: CodeGraph MCP usage guide — one tool, codegraph_explore alwaysApply: true --- ## CodeGraph -This project has a CodeGraph MCP server (`codegraph_*` tools) configured. CodeGraph is a tree-sitter-parsed knowledge graph of every symbol, edge, and file. Reads are sub-millisecond and return structural information grep cannot. +This project has a CodeGraph MCP server configured, exposing a single tool: `codegraph_explore`. CodeGraph is a tree-sitter-parsed knowledge graph of every symbol, edge, and file. Reads are sub-millisecond and return structural information grep cannot. -### When to prefer codegraph over native search +### Use codegraph_explore instead of reading files -Use codegraph for **structural** questions — what calls what, what would break, where is X defined, what is X's signature. Use native grep/read only for **literal text** queries (string contents, comments, log messages) or after you already have a specific file open. - -| Question | Tool | -|---|---| -| "Where is X defined?" / "Find symbol named X" | `codegraph_search` | -| "What calls function Y?" | `codegraph_callers` | -| "What does Y call?" | `codegraph_callees` | -| "How does X reach/become Y? / trace the flow from X to Y" | `codegraph_trace` (one call = the whole path, incl. callback/React/JSX dynamic hops) | -| "What would break if I changed Z?" | `codegraph_impact` | -| "Show me Y's signature / source / docstring" | `codegraph_node` | -| "Give me focused context for a task/area" | `codegraph_context` | -| "See several related symbols' source at once" | `codegraph_explore` | -| "What files exist under path/" | `codegraph_files` | -| "Is the index healthy?" | `codegraph_status` | +Reach for `codegraph_explore` before grep/find or Read for any **structural** question — how does X work, how does X reach Y, what calls what, where is X defined, or surveying an area. It takes a natural-language question or a bag of symbol/file names and returns the relevant symbols' **verbatim, line-numbered source** grouped by file (the same `\t` shape Read gives you, safe to Edit from), plus the call paths between them — including dynamic-dispatch hops (callbacks, React re-render, JSX children) grep can't follow — and a blast-radius summary of what depends on them. Name a file or symbol in the query to read its current source. ### Rules of thumb -- **Answer directly — don't delegate exploration.** For "how does X work" / architecture questions, answer with 2-3 codegraph calls: `codegraph_context` first, then ONE `codegraph_explore` for the source of the symbols it surfaces. For a specific **flow** ("how does X reach Y") start with `codegraph_trace` from→to — one call returns the whole path with dynamic hops bridged — then ONE `codegraph_explore` for the bodies; don't rebuild the path with `codegraph_search` + `codegraph_callers`. Codegraph IS the pre-built index, so spawning a separate file-reading sub-task/agent — or running a grep + read loop — repeats work codegraph already did and costs more for the same answer. +- **Answer directly — don't delegate exploration.** ONE `codegraph_explore` usually answers the whole question; follow up with another `codegraph_explore` naming more specific symbols if you need more. Codegraph IS the pre-built index, so spawning a separate file-reading sub-task/agent — or running a grep + read loop — repeats work codegraph already did and costs more for the same answer. - **Trust codegraph results.** They come from a full AST parse. Do NOT re-verify them with grep — that's slower, less accurate, and wastes context. -- **Don't grep first** when looking up a symbol by name. `codegraph_search` is faster and returns kind + location + signature in one call. -- **Don't chain `codegraph_search` + `codegraph_node`** when you just want context — `codegraph_context` is one call. -- **Don't loop `codegraph_node` over many symbols** — one `codegraph_explore` call returns several symbols' source grouped in a single capped call, while each separate node/Read call re-reads the whole context and costs far more. -- **Index lag — check the staleness banner, don't guess a wait.** When a codegraph response starts with "⚠️ Some files referenced below were edited since the last index sync…", the listed files are pending re-index — Read those specific files for accurate content. Files NOT in that banner are fresh and codegraph is authoritative for them. `codegraph_status` also lists pending files under "Pending sync". +- **Don't grep or Read first** to find or understand indexed code — one `codegraph_explore` returns the relevant source in a single round-trip. Reach for raw Read/Grep only to confirm a specific detail codegraph didn't cover, or for what it doesn't index (configs, docs). +- **Index lag — check the staleness banner, don't guess a wait.** When a codegraph response starts with "⚠️ Some files referenced below were edited since the last index sync…", the listed files are pending re-index — Read those specific files for accurate content. Files NOT in that banner are fresh and codegraph is authoritative for them. ### If `.codegraph/` doesn't exist diff --git a/.gitignore b/.gitignore index 69791d3..4963a4b 100644 --- a/.gitignore +++ b/.gitignore @@ -45,6 +45,10 @@ npm-debug.log* # Parallels Windows VM SSH/connection config (local machine, see CLAUDE.md) .parallels +# Confidential business / product / strategy docs — must NOT land in the +# public engine repo (see the IP boundary in CLAUDE.md) +docs/business/ + # CodeGraph data directories (in test projects) .codegraph/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 0b1a325..443d3a6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,18 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### New Features +- **Claude Code:** an optional front-load hook makes your agent reach for CodeGraph automatically. When you ask a structural question — "how does X work", "what calls Y", "trace the flow from A to B" — CodeGraph injects the relevant source and call paths into the prompt up front, so the agent answers from the graph instead of grepping around to rebuild it. You're asked during `codegraph install` (default yes; Claude Code only, since it's the agent with prompt hooks), it's removed by `codegraph uninstall`, and `codegraph upgrade` turns it on for existing Claude setups. It's strictly additive and degradable — non-structural prompts and un-indexed projects are left alone — and you can switch it off any time without uninstalling by setting `CODEGRAPH_NO_PROMPT_HOOK=1`. +- Vue store actions, mutations, and getters are now indexed as symbols you can find and read. Whether your store is **Vuex** (`mutations` / `actions` objects in a module) or **Pinia** — both the options form (`defineStore({ actions: { … } })`) and the setup form (`defineStore('id', () => { … })`, where actions are local functions) — each action, mutation, and getter is now a real node. So `codegraph search` finds `login` or `getSessionList`, and `codegraph_explore` / `codegraph_node` show its body and what it calls, instead of "not found" because the function only existed as an object-literal property. +- `codegraph_explore` now connects a Vue component to the **Pinia** store action it calls. When code does `const store = useUserStore()` and then `store.fetchUser()`, that call now links through to the `fetchUser` action in the store module — so "what happens when this view loads its data?" traces from the component into the action's body instead of stopping at the `store.fetchUser()` line. Works for both Pinia store styles (options and setup), and stays precise (a built-in like `store.$patch()` or an unrelated same-named method isn't mislinked). +- `codegraph_explore` now follows **Vuex** string dispatch. A `dispatch('user/login')` or `commit('SET_TOKEN')` call — namespaced `'module/action'` keys included — now links to the action or mutation it names, resolved to the correct store module even when several modules share an action name (and without being fooled by a same-named `api/` helper). So "what runs when this dispatches?" traces from the call into the store handler and on to the mutations it commits. Vuex's canonical `export default { namespaced, actions, mutations }` module shape is now indexed too, so those handlers are findable symbols. +- `codegraph_explore` now connects React data-fetching flows built on **RTK Query** (Redux Toolkit's `createApi`). An endpoint defined inside `createApi({ endpoints })` and the `useGetXQuery` / `useUpdateYMutation` hook it generates were both invisible to analysis — so "what does this component fetch?" or "where does `useGetThingQuery` get its data?" dead-ended, because the hook, the endpoint, and the component had nothing linking them. CodeGraph now indexes each endpoint and each generated hook as real symbols and wires the path `component → useGetXQuery → getX → queryFn`, so the flow resolves in one explore call instead of reading the API slice by hand. Both the arrow (`endpoints: build => ({ … })`) and method (`endpoints(builder) { return { … } }`) styles are recognized, along with the `useLazyGetXQuery` variant; hand-written hooks of a similar name are left untouched. +- `codegraph_explore` now follows **Celery** task dispatch in Python. A `send_email.delay(...)` or `send_email.apply_async(...)` call now links to the `@shared_task` / `@app.task` function it runs — typically defined in a different module (`tasks.py`) from where it's triggered (a view or service) — so "what actually happens when this is dispatched?" traces from the call site straight into the task body instead of stopping at the `.delay()` line. Both decorator dialects are recognized (bare `@shared_task` and the arg'd `@app.task(bind=True, …)` form), including the module-qualified `tasks.invalidate_cache.apply_async()` call style. It stays precise: a `.delay()` on something that isn't a Celery task is never mislinked, so a project that doesn't use Celery is unaffected. +- `codegraph_explore` now follows **Spring application events** in Java. A `publishEvent(new OrderShippedEvent(...))` call now links to every `@EventListener` that handles that event — usually in a different class — so "what reacts when this is published?" traces from the publisher straight into each listener method instead of dead-ending at `publishEvent(...)`. The link is by event type, and all the common listener styles are recognized: a `@EventListener` typed on its parameter, the `@EventListener(SomeEvent.class)` form, `@TransactionalEventListener`, and the older `implements ApplicationListener`. One event fans out to all its listeners, and a plain Spring app with no event bus is unaffected. +- `codegraph_explore` now follows **MediatR** request and notification dispatch in C#/.NET. A `_mediator.Send(command)` or `_mediator.Publish(notification)` call now links to the `Handle` method of the matching `IRequestHandler<>` / `INotificationHandler<>` — usually in a different file in a Clean Architecture layout — so "what handles this command?" traces from the controller straight into the handler instead of stopping at the mediator call. The sent type is recognized whether it's constructed inline (`Send(new GetFooQuery())`), built into a local first (`var cmd = new …; Send(cmd)`), or passed in as a parameter, and it's matched by type — so a `MessagingCenter.Send(...)` or a same-named DTO that isn't a request is never mislinked, and a project without MediatR is unaffected. +- `codegraph_explore` now follows **Sidekiq** background-job dispatch in Ruby. A `DestroyUserWorker.perform_async(id)` (or `.perform_in` / `.perform_at`) call now links to that worker's `perform` method — usually in `app/workers/` away from the controller or model that enqueues it — so "what runs in the background here?" traces from the enqueue straight into the job body. Both the modern `include Sidekiq::Job` and the older `Sidekiq::Worker` are recognized, namespaced workers resolve to the right class even when several share a name (e.g. `Comments::NotifyWorker` vs `Articles::NotifyWorker`), and Rails ActiveJob's `perform_later` — a different mechanism — is intentionally left alone. +- `codegraph_explore` now follows **Laravel events** in PHP. An `event(new OrderShipped($order))` call now links to every listener that handles it — each listener's `handle()` method, usually a separate `app/Listeners/` class — so "what reacts to this event?" traces from the dispatch straight into the listener bodies. Listeners are found both ways Laravel registers them: by a typed `handle(OrderShipped $event)` (auto-discovery, including a `handle(A|B $event)` union that listens for two events) and by the `protected $listen` map in your `EventServiceProvider` (which also catches a listener whose `handle()` has no type-hint). One event fans out to all its listeners, and queued jobs — dispatched via `::dispatch()` rather than `event()` — are correctly left out. + +- `codegraph_explore` now surfaces the right code in large multi-layer projects. When you ask a backend-flow question in a repo that pairs an API server with a big frontend that mirrors the same domain words — say an `app/` admin UI sitting over an `api/` server — the server-side file that genuinely matches several of your query's terms is no longer pushed out of the results by the larger, more interconnected frontend layer. A file corroborated by two or more distinct query terms is now kept in the answer even when a denser unrelated layer would otherwise crowd it out, so "how does X read items / handle the request" returns the service or handler that does the work instead of a wall of frontend views. Single-layer projects are unaffected; set `CODEGRAPH_RANK_NO_MULTITERM=1` to revert to the previous ranking. - Impact and blast-radius analysis for TypeScript, JavaScript, Go, Python, Rust, Ruby, C, Java, C#, PHP, Scala, Kotlin, Swift, Dart, and Pascal/Delphi now understands the readers of a constant. When you change a file-scope, package-level, module-level, or class-level constant — a config object, a lookup table, a shared constant — the other symbols in that file that read it now show up as affected, where before they were invisible (impact only followed calls, imports, and inheritance, so a constant's consumers looked like "nothing depends on this"). This makes `codegraph impact`, and the impact trail in `codegraph_explore`/`codegraph_node`, catch the "change this table, break its readers" class of change. It's on by default and adds no nodes to your graph; bundled/minified files and ambiguously-shadowed names are skipped to keep results precise. Set `CODEGRAPH_VALUE_REFS=0` to turn it off. - C file-scope constants and globals — `static const` scalars, pointer/array lookup tables, and shared mutable globals — are now recognized as symbols in their own right. They previously weren't extracted at all, so they never appeared in search or carried any dependents; now they show up in `codegraph search` and participate in impact analysis (see above), so changing a C lookup table surfaces the same-file functions that read it. - Java `static final` constants, C# `const` / `static readonly` constants, Scala `object` vals, and Kotlin top-level / `object` / `companion object` `val`s are now classified as constants rather than generic fields, so they participate in the constant-reader impact analysis above — change a `public static final` table, a `const string`, a Scala `object Config { val Timeout = … }`, or a Kotlin `companion object { const val … }` and the methods that read it now show up as affected. (Per-object Java `final` / C# `readonly` / Scala & Kotlin `class` instance properties are unchanged.) Kotlin constants were previously not indexed as their own symbols at all, so they now also appear in `codegraph search`. @@ -19,6 +31,10 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ### Fixes +- `codegraph install` now wires up your agents and stops there — it no longer indexes the current directory. Building a project's graph is always the explicit `codegraph init` (or `codegraph index`), so you decide what gets indexed and when, and the steps are the same whether you installed globally or just for one project. This clears up the confusion where a project-local install silently indexed but a global one didn't, and where the docs and the tool disagreed about whether you still had to run `init`. (#826) +- React components declared with `forwardRef`, `memo`, or styled-components / emotion (`const Button = forwardRef(...)`, `const Card = memo(...)`, `const Box = styled.button\`…\``) are now recognized as components, so finding where they're used works. Before, they were indexed as plain constants, so `codegraph callers` and impact analysis reported "no callers found" even when the component was rendered across dozens of files — a dangerous false "safe to change" right before refactoring a shared component. Now every `; +} +` + ); + const db = await index(); + + // The render edge exists and is the synthesized jsx-render kind. + const edgeRows = db + .prepare( + `SELECT s.name caller FROM edges e + JOIN nodes s ON s.id = e.source + JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata, '$.synthesizedBy') = 'jsx-render' + AND t.kind = 'component' AND t.name = 'Button'` + ) + .all(); + expect(edgeRows.map((r: any) => r.caller)).toContain('Page'); + + // ...and it surfaces through the public callers API (the issue's symptom: + // "No callers found" before the fix). + const buttonId = db + .prepare("SELECT id FROM nodes WHERE name='Button' AND kind='component'") + .get().id as string; + const callers = cg.getCallers(buttonId).map((c: any) => c.node.name); + expect(callers).toContain('Page'); + }); + + it('captures the inner render-fn body callees under the component', async () => { + fs.writeFileSync( + path.join(dir, 'widget.tsx'), + `import * as React from 'react'; +function useThing() { return 1; } +export const Widget = React.forwardRef((props, ref) => { + const v = useThing(); + return
{v}
; +}); +` + ); + const db = await index(); + const rows = db + .prepare( + `SELECT t.name FROM edges e + JOIN nodes s ON s.id = e.source + JOIN nodes t ON t.id = e.target + WHERE s.name = 'Widget' AND s.kind = 'component' + AND e.kind = 'calls' AND t.name = 'useThing'` + ) + .all(); + expect(rows.length).toBeGreaterThanOrEqual(1); + }); + + it('does not misclassify non-component PascalCase consts (precision)', async () => { + fs.writeFileSync( + path.join(dir, 'controls.tsx'), + `import * as React from 'react'; +const cache = memo(expensiveFn); +export const Config = loadConfig(); +export const Client = new ApiClient(); +export const Styles = styledHelper(); +export const Total = [1, 2].reduce((a, b) => a + b, 0); +export const Theme = { color: 'red' }; +` + ); + const db = await index(); + for (const name of ['Config', 'Client', 'Styles', 'Total', 'Theme']) { + expect(kindsOf(db, name), `${name} must stay a constant`).toContain('constant'); + expect(kindsOf(db, name), `${name} must not be a component`).not.toContain('component'); + } + // A lowercase-named memo() result is a memoization util, not a component. + expect(kindsOf(db, 'cache')).not.toContain('component'); + }); +}); diff --git a/__tests__/redux-thunk-synthesizer.test.ts b/__tests__/redux-thunk-synthesizer.test.ts new file mode 100644 index 0000000..bde12e2 --- /dev/null +++ b/__tests__/redux-thunk-synthesizer.test.ts @@ -0,0 +1,129 @@ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; + +/** + * End-to-end test for the redux-thunk dispatch-chain synthesizer. + * + * `createAsyncThunk(prefix, async (a, api) => {...})` passes the async body as an argument, so + * tree-sitter never makes it its own function node — the thunk `constant`'s body calls (incl. + * `dispatch(nextThunk(...))`) are orphaned and `callees(thunk)` is empty. Verify the synthesizer + * body-scans each thunk constant and links it → each dispatched thunk, so the chain + * `outer → inner → deep` connects end-to-end; and that a non-thunk constant is skipped. + */ +describe('redux-thunk synthesizer', () => { + let dir: string; + beforeEach(() => { + dir = fs.mkdtempSync(path.join(os.tmpdir(), 'redux-thunk-fixture-')); + }); + afterEach(() => { + fs.rmSync(dir, { recursive: true, force: true }); + }); + + it('links each thunk constant to the thunks it dispatches, and skips non-thunks', async () => { + fs.writeFileSync( + path.join(dir, 'package.json'), + JSON.stringify({ name: 'app', dependencies: { '@reduxjs/toolkit': '^2' } }) + ); + fs.writeFileSync( + path.join(dir, 'thunks.ts'), + `import { createAsyncThunk } from '@reduxjs/toolkit'; + +export const deepThunk = createAsyncThunk('app/deep', async (n: number) => { + return n * 2; +}); + +export const innerThunk = createAsyncThunk('app/inner', async (n: number, { dispatch }) => { + return dispatch(deepThunk(n)); +}); + +export const outerThunk = createAsyncThunk('app/outer', async (n: number, { dispatch }) => { + await dispatch(innerThunk(n)); +}); + +// Non-thunk constant that only MENTIONS dispatch in a string — must be skipped. +export const notAThunk = 'dispatch(innerThunk())'; +` + ); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + + const db = (cg as any).db.db; + const rows = db + .prepare( + `SELECT s.name source_name, s.kind source_kind, t.name target_name, + json_extract(e.metadata,'$.via') via, + json_extract(e.metadata,'$.registeredAt') registeredAt + FROM edges e + JOIN nodes s ON s.id = e.source + JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata,'$.synthesizedBy') = 'redux-thunk'` + ) + .all(); + cg.close?.(); + + // The dispatch chain connects: outer → inner → deep. + const pairs = new Set(rows.map((r: any) => `${r.source_name}>${r.target_name}`)); + expect(pairs.has('outerThunk>innerThunk')).toBe(true); + expect(pairs.has('innerThunk>deepThunk')).toBe(true); + + // Sources are thunk constants; the non-thunk string constant is never a source. + expect(rows.every((r: any) => r.source_kind === 'constant')).toBe(true); + expect(rows.some((r: any) => r.source_name === 'notAThunk')).toBe(false); + + // Edges are 'calls' with the wiring site surfaced for the agent. + const outer = rows.find((r: any) => r.source_name === 'outerThunk'); + expect(outer.via).toBe('innerThunk'); + expect(outer.registeredAt).toMatch(/thunks\.ts:\d+/); + }); + + it('on a name collision, a dispatch resolves to the THUNK, not a same-named service function', async () => { + // Regression for the octo-call case: `leaveCall` exists as BOTH a `createAsyncThunk` + // const and an unrelated service function. `dispatch(leaveCall())` targets the thunk, + // but the old first-match resolver could pick the function. The resolver now prefers a + // thunk-signature const > other const > same-file > first. + fs.writeFileSync( + path.join(dir, 'package.json'), + JSON.stringify({ name: 'app', dependencies: { '@reduxjs/toolkit': '^2' } }) + ); + // A plain service function that shares the name `leaveCall` with the thunk below. + fs.writeFileSync(path.join(dir, 'service.ts'), `export function leaveCall(id: string) { return id; }\n`); + fs.writeFileSync( + path.join(dir, 'thunks.ts'), + `import { createAsyncThunk } from '@reduxjs/toolkit'; + +export const leaveCall = createAsyncThunk('call/leave', async () => { + return 1; +}); + +export const logout = createAsyncThunk('user/logout', async (_: void, { dispatch }) => { + dispatch(leaveCall()); +}); +` + ); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + + const db = (cg as any).db.db; + const row = db + .prepare( + `SELECT t.kind target_kind, t.file_path target_file + FROM edges e + JOIN nodes s ON s.id = e.source + JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata,'$.synthesizedBy') = 'redux-thunk' + AND s.name = 'logout' AND t.name = 'leaveCall'` + ) + .get(); + cg.close?.(); + + expect(row).toBeTruthy(); + // Resolved to the createAsyncThunk constant in thunks.ts, NOT service.ts's function. + expect(row.target_kind).toBe('constant'); + expect(row.target_file).toMatch(/thunks\.ts$/); + }); +}); diff --git a/__tests__/rtk-query-synthesizer.test.ts b/__tests__/rtk-query-synthesizer.test.ts new file mode 100644 index 0000000..3afef8f --- /dev/null +++ b/__tests__/rtk-query-synthesizer.test.ts @@ -0,0 +1,197 @@ +/** + * RTK Query generated-hook → endpoint synthesizer. + * + * RTK Query's `createApi({ endpoints })` defines endpoints as object-literal + * properties (`getX: build.query(...)`) and generates one `useGetXQuery` / + * `useUpdateYMutation` hook per endpoint, exported via a `const {…} = api` + * destructuring. Neither the endpoint nor the generated hook is otherwise a node, + * so a `component → useGetXQuery → getX → queryFn` flow has nothing to connect to. + * + * This validates the two halves: extraction mints a function node for each + * endpoint (named by its key, both the `build => ({...})` arrow form and the + * `endpoints(build){ return {...} }` method-shorthand form) and for each generated + * hook binding; then the synthesizer bridges hook→endpoint by the naming + * convention (incl. the `useLazyGetXQuery` variant → the same endpoint). Precision + * is gated to genuinely-generated hooks: a hand-written `use*Query` arrow is never + * bridged, and no edge ever crosses files. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; + +describe('rtk-query synthesizer', () => { + let dir: string; + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'rtk-query-')); }); + afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); }); + + it('extracts endpoints + generated hooks and bridges hook→endpoint (arrow + method + lazy + factory forms)', async () => { + // Arrow form (shapeshift-style): `endpoints: build => ({...})`, `queryFn: () => {}`. + fs.writeFileSync( + path.join(dir, 'fiatRampApi.ts'), + `import { createApi, fetchBaseQuery } from '@reduxjs/toolkit/query/react'; +import { fetchRamps } from './ramps'; + +export const fiatRampApi = createApi({ + reducerPath: 'fiatRampApi', + baseQuery: fetchBaseQuery({ baseUrl: '/' }), + endpoints: build => ({ + getFiatRamps: build.query({ + queryFn: async () => { + const data = await fetchRamps(); + return { data }; + }, + }), + placeOrder: build.mutation({ + query: body => ({ url: 'order', method: 'POST', body }), + }), + }), +}); + +export const { useGetFiatRampsQuery, usePlaceOrderMutation, useLazyGetFiatRampsQuery } = fiatRampApi; +` + ); + // Method-shorthand form (basetool-style): `endpoints(builder){ return {...} }`, + // `query(){}` method handler, plus a factory-handler endpoint (no fn literal). + fs.writeFileSync( + path.join(dir, 'dashApi.ts'), + `import { createApi, fetchBaseQuery } from '@reduxjs/toolkit/query/react'; +import { makeCheckFn } from './factory'; + +export const dashApi = createApi({ + reducerPath: 'dash', + baseQuery: fetchBaseQuery({ baseUrl: '/' }), + endpoints(builder) { + return { + getDashboards: builder.query({ + query() { + return '/dashboards'; + }, + }), + checkConnection: builder.mutation({ + queryFn: makeCheckFn('/check'), + }), + }; + }, +}); + +export const { useGetDashboardsQuery, useCheckConnectionMutation } = dashApi; +` + ); + // Components consuming the generated hooks. + fs.writeFileSync( + path.join(dir, 'Views.tsx'), + `import { useGetFiatRampsQuery, useLazyGetFiatRampsQuery } from './fiatRampApi'; +import { useGetDashboardsQuery } from './dashApi'; + +export function FiatForm() { + const { data } = useGetFiatRampsQuery(); + return data; +} +export function DashList() { + const { data } = useGetDashboardsQuery(); + return data; +} +export function LazyForm() { + const [load] = useLazyGetFiatRampsQuery(); + return load; +} +` + ); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + + // Endpoints are extracted as function nodes named by their key. + const endpoints = db + .prepare(`SELECT name, kind FROM nodes WHERE name IN ('getFiatRamps','placeOrder','getDashboards','checkConnection')`) + .all(); + expect(endpoints.length).toBe(4); + expect(endpoints.every((n: any) => n.kind === 'function')).toBe(true); + + // Generated hooks are extracted as function nodes carrying the sentinel. + const hooks = db + .prepare(`SELECT name FROM nodes WHERE signature = '= RTK Query generated hook' ORDER BY name`) + .all() + .map((r: any) => r.name); + expect(hooks).toEqual([ + 'useCheckConnectionMutation', + 'useGetDashboardsQuery', + 'useGetFiatRampsQuery', + 'useLazyGetFiatRampsQuery', + 'usePlaceOrderMutation', + ]); + + // hook → endpoint synth edges, including the Lazy variant mapping to the same endpoint. + const synth = db + .prepare( + `SELECT s.name source, t.name target, s.file_path sf, t.file_path tf + FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata,'$.synthesizedBy') = 'rtk-query'` + ) + .all(); + const pairs = synth.map((r: any) => `${r.source}->${r.target}`).sort(); + expect(pairs).toEqual([ + 'useCheckConnectionMutation->checkConnection', + 'useGetDashboardsQuery->getDashboards', + 'useGetFiatRampsQuery->getFiatRamps', + 'useLazyGetFiatRampsQuery->getFiatRamps', + 'usePlaceOrderMutation->placeOrder', + ]); + // Every synth edge stays within one file (RTK colocates api + hooks). + expect(synth.every((r: any) => r.sf === r.tf)).toBe(true); + + // The component reaches the hook (normal import/call resolution), so the full + // `component → hook → endpoint` chain is connected. + const compToHook = db + .prepare( + `SELECT s.name source, t.name target FROM edges e + JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE s.name = 'FiatForm' AND t.name = 'useGetFiatRampsQuery' AND e.kind = 'calls'` + ) + .all(); + expect(compToHook.length).toBeGreaterThan(0); + + cg.close?.(); + }); + + it('does not bridge a hand-written use*Query hook (no createApi, no sentinel) — 0 synth edges', async () => { + // A real custom hook of the same name shape, plus a same-file `getThing` + // function it could spuriously map to. Without the generated-hook sentinel + + // createApi destructuring, the synthesizer must produce nothing. + fs.writeFileSync( + path.join(dir, 'useGetThingQuery.ts'), + `export function getThing() { return 42; } +export const useGetThingQuery = () => { + return getThing(); +}; +` + ); + fs.writeFileSync( + path.join(dir, 'Thing.tsx'), + `import { useGetThingQuery } from './useGetThingQuery'; +export function Thing() { + return useGetThingQuery(); +} +` + ); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + + const synth = db + .prepare(`SELECT count(*) c FROM edges WHERE json_extract(metadata,'$.synthesizedBy') = 'rtk-query'`) + .get(); + expect(synth.c).toBe(0); + // The hand-written hook keeps its real body (not a sentinel binding). + const sentinel = db + .prepare(`SELECT count(*) c FROM nodes WHERE signature = '= RTK Query generated hook'`) + .get(); + expect(sentinel.c).toBe(0); + + cg.close?.(); + }); +}); diff --git a/__tests__/sidekiq-dispatch-synthesizer.test.ts b/__tests__/sidekiq-dispatch-synthesizer.test.ts new file mode 100644 index 0000000..46f8b9e --- /dev/null +++ b/__tests__/sidekiq-dispatch-synthesizer.test.ts @@ -0,0 +1,128 @@ +/** + * Sidekiq job-dispatch bridge (Ruby). + * + * Sidekiq decouples a job enqueue from the worker's `perform`, linked by the WORKER CLASS + * NAME: `DestroyUserWorker.perform_async(id)` has no static edge to `DestroyUserWorker#perform` + * (usually a different file). This bridges each `Worker.perform_async`/`.perform_in`/`.perform_at` + * site to that worker's instance `perform`, gated on the class including `Sidekiq::Job`/`Worker`. + * Covers both include aliases, the scheduled forms, namespace disambiguation (two `NotifyWorker`s + * in different modules resolve to the right one by qualified name), and the precision boundary: a + * non-worker class with a `perform`, and an ActiveJob `perform_later`, both produce no edge. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; + +describe('sidekiq-dispatch synthesizer', () => { + let dir: string; + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'sidekiq-dispatch-')); }); + afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); }); + + const write = (rel: string, body: string) => { + const p = path.join(dir, rel); + fs.mkdirSync(path.dirname(p), { recursive: true }); + fs.writeFileSync(p, body); + }; + + it('bridges perform_async/_in to #perform, disambiguates namespaces, ignores non-workers and ActiveJob', async () => { + write('app/workers/destroy_user_worker.rb', `class DestroyUserWorker + include Sidekiq::Worker + def perform(user_id) + User.find(user_id).destroy! + end +end +`); + // Modern Sidekiq::Job alias + the scheduled form. + write('app/workers/send_email_worker.rb', `class SendEmailWorker + include Sidekiq::Job + def perform(addr) + end +end +`); + // Namespace collision: two NotifyWorkers, same simple name, different modules. + write('app/workers/comments/notify_worker.rb', `module Comments + class NotifyWorker + include Sidekiq::Job + def perform(id) + end + end +end +`); + write('app/workers/articles/notify_worker.rb', `module Articles + class NotifyWorker + include Sidekiq::Job + def perform(id) + end + end +end +`); + // A non-worker class that happens to have a `perform` method — never a target. + write('app/services/report.rb', `class Report + def perform(x) + end +end +`); + // An ActiveJob — dispatched via perform_later, a different shape, not matched. + write('app/jobs/cleanup_job.rb', `class CleanupJob < ApplicationJob + def perform + end +end +`); + write('app/services/user_service.rb', `class UserService + def deactivate(user) + DestroyUserWorker.perform_async(user.id) + SendEmailWorker.perform_in(5, user.email) + Comments::NotifyWorker.perform_async(1) + Articles::NotifyWorker.perform_async(2) + Report.perform_async(3) + CleanupJob.perform_later + end +end +`); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + + const edges = db + .prepare( + `SELECT s.name source, t.name target, t.file_path tf, json_extract(e.metadata,'$.via') via + FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata,'$.synthesizedBy') = 'sidekiq-dispatch'` + ) + .all(); + + // Four enqueues bridge: both aliases, perform_async + perform_in, two namespaced. + expect(edges.map((r: any) => r.via).sort()).toEqual([ + 'Articles::NotifyWorker', 'Comments::NotifyWorker', 'DestroyUserWorker', 'SendEmailWorker', + ]); + expect(edges.every((r: any) => r.target === 'perform' && r.source === 'deactivate')).toBe(true); + // Namespace disambiguation: each NotifyWorker hits its OWN module's file, not the other. + expect(edges.find((r: any) => r.via === 'Comments::NotifyWorker').tf).toMatch(/comments[\\/]notify_worker\.rb$/); + expect(edges.find((r: any) => r.via === 'Articles::NotifyWorker').tf).toMatch(/articles[\\/]notify_worker\.rb$/); + // PRECISION: a non-worker `perform`, and ActiveJob `perform_later`, contribute nothing. + expect(edges.some((r: any) => r.via === 'Report')).toBe(false); + expect(edges.some((r: any) => /Cleanup/.test(r.via))).toBe(false); + + cg.close?.(); + }); + + it('produces no edges in a Ruby project with no Sidekiq (clean control)', async () => { + write('lib/calc.rb', `class Calc + def add(a, b) + a + b + end +end +`); + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + const count = db + .prepare(`SELECT count(*) c FROM edges WHERE json_extract(metadata,'$.synthesizedBy') = 'sidekiq-dispatch'`) + .get(); + expect(count.c).toBe(0); + cg.close?.(); + }); +}); diff --git a/__tests__/spring-event-synthesizer.test.ts b/__tests__/spring-event-synthesizer.test.ts new file mode 100644 index 0000000..760cc88 --- /dev/null +++ b/__tests__/spring-event-synthesizer.test.ts @@ -0,0 +1,132 @@ +/** + * Spring application-event bridge (Java). + * + * Spring decouples an event publisher from its listener(s) through the application + * event bus, linked by the EVENT TYPE: `eventPublisher.publishEvent(new XEvent(...))` + * has no static edge to the `@EventListener void on(XEvent e)` that handles it (usually + * in a different file). This bridges each `publishEvent(new XEvent(...))` site to every + * listener of XEvent. Covers all four listener forms — param-typed `@EventListener`, + * annotation-typed `@EventListener(XEvent.class)`, `@TransactionalEventListener`, and the + * older `implements ApplicationListener` / `onApplicationEvent` — fans out to + * multiple listeners of the same event, and proves precision: a published event with no + * listener, and a same-file non-annotated method, both produce no edge. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; + +describe('spring-event synthesizer', () => { + let dir: string; + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'spring-event-')); }); + afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); }); + + const write = (rel: string, body: string) => { + const p = path.join(dir, rel); + fs.mkdirSync(path.dirname(p), { recursive: true }); + fs.writeFileSync(p, body); + }; + + it('bridges publishEvent(new X) to every listener form of X, ignoring unheard events and non-listeners', async () => { + write('shop/OrderEvents.java', `package shop; +class OrderShippedEvent { } +class OrderCancelledEvent { } +class UnheardEvent { } +`); + // Publisher — two events, one of them (UnheardEvent) has no listener. + write('shop/OrderService.java', `package shop; +import org.springframework.context.ApplicationEventPublisher; +class OrderService { + private ApplicationEventPublisher publisher; + void ship() { + publisher.publishEvent(new OrderShippedEvent()); + publisher.publishEvent(new UnheardEvent()); + } + void cancel() { + publisher.publishEvent(new OrderCancelledEvent()); + } +} +`); + // Form 1: param-typed @EventListener — plus a same-file NON-listener (no annotation). + write('shop/ShippingListener.java', `package shop; +import org.springframework.context.event.EventListener; +class ShippingListener { + @EventListener + public void onShipped(OrderShippedEvent event) { } + + public void helper(OrderShippedEvent event) { } +} +`); + // Form 2: annotation-typed @EventListener(X.class) — fan-out, a 2nd OrderShipped listener. + write('shop/AuditListener.java', `package shop; +import org.springframework.context.event.EventListener; +class AuditListener { + @EventListener(OrderShippedEvent.class) + public void audit(OrderShippedEvent event) { } +} +`); + // Form 3: @TransactionalEventListener — a 3rd OrderShipped listener. + write('shop/TxListener.java', `package shop; +import org.springframework.transaction.event.TransactionalEventListener; +class TxListener { + @TransactionalEventListener + public void afterShipped(OrderShippedEvent event) { } +} +`); + // Form 4: older implements ApplicationListener / onApplicationEvent. + write('shop/LegacyListener.java', `package shop; +import org.springframework.context.ApplicationListener; +class LegacyListener implements ApplicationListener { + @Override + public void onApplicationEvent(OrderCancelledEvent event) { } +} +`); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + + const edges = db + .prepare( + `SELECT s.name source, t.name target, json_extract(e.metadata,'$.via') via + FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata,'$.synthesizedBy') = 'spring-event'` + ) + .all(); + + const targets = (src: string) => + edges.filter((r: any) => r.source === src).map((r: any) => r.target).sort(); + // ship() → all three OrderShippedEvent listeners (param-typed, annotation-typed, transactional). + expect(targets('ship')).toEqual(['afterShipped', 'audit', 'onShipped']); + // cancel() → the ApplicationListener form. + expect(targets('cancel')).toEqual(['onApplicationEvent']); + // Every shipped edge is keyed by the event type. + expect(edges.filter((r: any) => r.source === 'ship').every((r: any) => r.via === 'OrderShippedEvent')).toBe(true); + // PRECISION: UnheardEvent has no listener → no edge; the non-annotated helper is never a target. + expect(edges.some((r: any) => r.via === 'UnheardEvent')).toBe(false); + expect(edges.some((r: any) => r.target === 'helper')).toBe(false); + + cg.close?.(); + }); + + it('produces no edges in a Spring app with no event bus (clean control)', async () => { + write('shop/PlainService.java', `package shop; +import org.springframework.stereotype.Service; +@Service +class PlainService { + private final Repo repo; + PlainService(Repo repo) { this.repo = repo; } + String find(String id) { return repo.get(id); } +} +`); + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + const count = db + .prepare(`SELECT count(*) c FROM edges WHERE json_extract(metadata,'$.synthesizedBy') = 'spring-event'`) + .get(); + expect(count.c).toBe(0); + cg.close?.(); + }); +}); diff --git a/__tests__/vue-store-extraction.test.ts b/__tests__/vue-store-extraction.test.ts new file mode 100644 index 0000000..9fc2f9e --- /dev/null +++ b/__tests__/vue-store-extraction.test.ts @@ -0,0 +1,138 @@ +/** + * Vue store action/mutation/getter extraction (the foundation for finding and + * reading store logic — `codegraph_node login` / `getSessionList`). + * + * Vuex/Pinia define a store's callable surface as object-literal members nested + * under `actions`/`mutations`/`getters`, or as body-local consts in a Pinia setup + * store — none of which were extracted, so the symbols an agent looks for didn't + * exist as nodes. This covers the three dominant forms: + * - Vuex module: non-exported `const actions = {…}` / `const mutations = {…}`. + * - Pinia options: `defineStore({ actions: {…}, getters: {…} })`. + * - Pinia setup: `defineStore('id', () => { const foo = …; return { foo } })`. + * And the precision gate: a non-exported `const actions = {…}` in a file that + * isn't a Vue store contributes nothing. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; + +describe('vue store extraction', () => { + let dir: string; + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'vue-store-')); }); + afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); }); + + it('extracts Vuex module + Pinia options + Pinia setup store members as function nodes', async () => { + // Vuex MODULE form: non-exported `const mutations`/`const actions` collections, + // wired via a default export (element-admin style). Method shorthand + arrow pairs. + fs.writeFileSync( + path.join(dir, 'userModule.js'), + `import { persistToken } from './auth-utils'; +const state = { token: '' }; +const mutations = { + SET_TOKEN: (state, token) => { state.token = token; }, +}; +const actions = { + login({ commit }, info) { + persistToken(info.token); + }, + async logout({ commit }) { + commit('SET_TOKEN', ''); + }, +}; +export default { namespaced: true, state, mutations, actions }; +` + ); + fs.writeFileSync( + path.join(dir, 'auth-utils.js'), + `export function persistToken(token) { return token; } +` + ); + // Pinia OPTIONS form: actions + getters as object properties of a defineStore config. + fs.writeFileSync( + path.join(dir, 'authStore.ts'), + `import { defineStore } from 'pinia'; +export const useAuthStore = defineStore({ + id: 'auth', + state: () => ({ name: '' }), + getters: { + upperName: state => state.name.toUpperCase(), + }, + actions: { + async fetchMenu() { return loadMenu(); }, + setName(n: string) { this.name = n; }, + }, +}); +` + ); + // Pinia SETUP form: actions are body-local consts exposed via the return block. + fs.writeFileSync( + path.join(dir, 'chatStore.ts'), + `import { defineStore } from 'pinia'; +export const useChatStore = defineStore('chat', () => { + const list = reactive([]); + const getList = async () => { return fetchList(); }; + function pushItem(x) { list.push(x); } + return { list, getList, pushItem }; +}); +` + ); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + const fn = (name: string) => + db.prepare(`SELECT count(*) c FROM nodes WHERE name = ? AND kind = 'function'`).get(name).c; + + // Vuex module: actions + mutations extracted. + expect(fn('login')).toBeGreaterThan(0); + expect(fn('logout')).toBeGreaterThan(0); + expect(fn('SET_TOKEN')).toBeGreaterThan(0); + // Pinia options: actions + getter extracted. + expect(fn('fetchMenu')).toBeGreaterThan(0); + expect(fn('setName')).toBeGreaterThan(0); + expect(fn('upperName')).toBeGreaterThan(0); + // Pinia setup: body-local actions extracted (and reachable via their bodies). + expect(fn('getList')).toBeGreaterThan(0); + expect(fn('pushItem')).toBeGreaterThan(0); + + // The extracted action spans its real body — `login`'s `persistToken(...)` + // call attributes to it (extraction, not the deferred dispatch synthesis). + const loginCalls = db + .prepare( + `SELECT t.name FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE s.name = 'login' AND e.kind = 'calls'` + ) + .all() + .map((r: any) => r.name); + expect(loginCalls).toContain('persistToken'); + + cg.close?.(); + }); + + it('does not extract a non-exported `const actions = {…}` outside a Vue store file', async () => { + // A plain module that happens to hold a non-exported `const actions` object of + // functions, but lacks any second Vue-store signal — the gate must not fire. + fs.writeFileSync( + path.join(dir, 'commands.js'), + `const actions = { + doThing() { return 1; }, + doOther() { return 2; }, +}; +export function run(key) { return actions[key](); } +` + ); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + + expect(db.prepare(`SELECT count(*) c FROM nodes WHERE name = 'doThing'`).get().c).toBe(0); + expect(db.prepare(`SELECT count(*) c FROM nodes WHERE name = 'doOther'`).get().c).toBe(0); + // The real exported function is still extracted normally. + expect(db.prepare(`SELECT count(*) c FROM nodes WHERE name = 'run' AND kind='function'`).get().c).toBeGreaterThan(0); + + cg.close?.(); + }); +}); diff --git a/__tests__/vuex-dispatch-synthesizer.test.ts b/__tests__/vuex-dispatch-synthesizer.test.ts new file mode 100644 index 0000000..b2f9a71 --- /dev/null +++ b/__tests__/vuex-dispatch-synthesizer.test.ts @@ -0,0 +1,100 @@ +/** + * Vuex string-keyed dispatch/commit bridge. + * + * Vuex dispatches actions/mutations by a runtime STRING key — `dispatch('user/login')`, + * `commit('SET_TOKEN')` — with no static edge to the handler (an object-literal + * method in a store module). This bridges the key to its function node: the last + * `/` segment is the action/mutation name, the preceding segment is the namespace + * (≈ the module file). It resolves to a node IN A STORE FILE (excluding a same-named + * `api/` helper — a real collision), disambiguated by the namespace appearing in the + * path, or the same file for a root `commit('M')` inside an action. Redux-style + * `dispatch(actionCreator())` (no string key) produces nothing. + * + * Also exercises the canonical Vuex MODULE shape `export default { namespaced, + * actions: {…}, mutations: {…} }` — whose methods only become nodes via the + * store-collection extraction this bridge depends on. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import * as os from 'node:os'; +import { CodeGraph } from '../src'; + +describe('vuex-dispatch synthesizer', () => { + let dir: string; + beforeEach(() => { dir = fs.mkdtempSync(path.join(os.tmpdir(), 'vuex-dispatch-')); }); + afterEach(() => { fs.rmSync(dir, { recursive: true, force: true }); }); + + it('bridges namespaced dispatch + local commit to the right store handler, excluding an api collision', async () => { + fs.mkdirSync(path.join(dir, 'store', 'modules'), { recursive: true }); + fs.mkdirSync(path.join(dir, 'api'), { recursive: true }); + // Canonical Vuex module: `export default { namespaced, actions, mutations }`. + fs.writeFileSync( + path.join(dir, 'store', 'modules', 'user.js'), + `import { login as apiLogin } from '../../api/user'; +export default { + namespaced: true, + state: { token: '' }, + mutations: { + SET_TOKEN(state, t) { state.token = t; }, + }, + actions: { + login({ commit }, info) { + apiLogin(info); + commit('SET_TOKEN', info.token); // root/local key → SET_TOKEN in THIS module + }, + }, +}; +` + ); + // Collision: an api helper ALSO named `login` — must never be the dispatch target. + fs.writeFileSync( + path.join(dir, 'api', 'user.js'), + `export function login(info) { return info; } +` + ); + // Consumer dispatches by namespaced string key. + fs.writeFileSync( + path.join(dir, 'app.js'), + `import store from './store'; +export function bootstrap() { + store.dispatch('user/login', { token: 'x' }); +} +` + ); + // Redux-style control: a non-string dispatch must produce no vuex edge. + fs.writeFileSync( + path.join(dir, 'reduxy.js'), + `export function reduxy(dispatch) { + dispatch(someAction()); +} +` + ); + + const cg = await CodeGraph.init(dir, { silent: true }); + await cg.indexAll(); + const db = (cg as any).db.db; + + const edges = db + .prepare( + `SELECT s.name source, t.name target, t.file_path tf, json_extract(e.metadata,'$.via') via + FROM edges e JOIN nodes s ON s.id = e.source JOIN nodes t ON t.id = e.target + WHERE json_extract(e.metadata,'$.synthesizedBy') = 'vuex-dispatch'` + ) + .all(); + + // bootstrap → login, resolving to the STORE module (not api/user.js). + const loginEdge = edges.find((r: any) => r.source === 'bootstrap' && r.target === 'login'); + expect(loginEdge).toBeTruthy(); + expect(loginEdge.tf).toMatch(/store[\\/]modules[\\/]user\.js$/); + expect(loginEdge.via).toBe('user/login'); + // The api helper of the same name was never targeted. + expect(edges.some((r: any) => /api[\\/]user\.js$/.test(r.tf))).toBe(false); + // Local commit('SET_TOKEN') inside the action → the same module's mutation. + expect(edges.some((r: any) => r.source === 'login' && r.target === 'SET_TOKEN')).toBe(true); + // Redux-style non-string dispatch contributed nothing. + expect(edges.some((r: any) => r.source === 'reduxy')).toBe(false); + + cg.close?.(); + }); +}); diff --git a/docs/design/dispatch-synthesizer-backlog.md b/docs/design/dispatch-synthesizer-backlog.md new file mode 100644 index 0000000..9e32771 --- /dev/null +++ b/docs/design/dispatch-synthesizer-backlog.md @@ -0,0 +1,173 @@ +# Dispatch-Synthesizer Backlog — the "dispatch-through-indirection" family + +**Audience:** a Claude agent continuing the coverage mission. +**Relationship to the playbook:** this is a *cross-cutting* companion to +[`dynamic-dispatch-coverage-playbook.md`](./dynamic-dispatch-coverage-playbook.md). +The playbook's §6 matrix is organized by **language × framework**. This doc is +organized by **dispatch *shape*** — because a single framework can contain several +distinct indirection shapes (Redux alone is ≥2: hand-written thunks vs RTK Query), +and several shapes recur identically across many frameworks/languages (a name→class +registry is the same problem in trezor `connect`, n8n nodes, and a VS Code command +palette). Redux-thunk (`synthesizedBy:'redux-thunk'`) was the first member shipped; +this is the queue behind it. + +Status legend (matches the playbook): ✅ done+validated · 🟡 shipped but under-validated +· 🔬 hole identified · ⬜ not started · ⛔ deliberately not built (silent beats wrong). + +--- + +## The discipline (lessons already paid for — read before building any of these) + +1. **Build against ≥2 real repos that *contain the pattern*, from the start.** + redux-thunk was tuned on **trezor-suite alone (n=1)**. The obvious second repo, + **shapeshift/web**, fires **0** redux-thunk edges — and that 0 is *correct*: + shapeshift has **zero** `createAsyncThunk`/`createThunk` (it's an **RTK Query** + codebase, 14 `createApi` files). So shapeshift could neither confirm nor refute + generalization — it doesn't contain the shape. **A synthesizer validated on one + repo is unvalidated.** Pick the validation repos *by grepping for the pattern + first*, not by reputation. + +2. **"One framework" ≠ "one shape."** The trezor→shapeshift split is the proof: + - `createAsyncThunk` + thunk→thunk `dispatch(Y())` chains → **redux-thunk** ✅ (trezor) + - `createApi` + `builder.query/mutation` endpoints → hooks/components → **RTK Query** 🔬 (shapeshift) — a *different, unbuilt* synthesizer + - plain `dispatch(action)` → matching `reducer`/slice `case` → **slice-dispatch** ⬜ + Don't let "we did Redux" hide two-thirds of Redux. + +3. **Precision is free recall's price.** redux-thunk's 0-on-shapeshift is the *good* + kind of zero (no false edges on a non-thunk repo — same bar as the playbook's + "0 on every non-pattern control"). Every synthesizer below must show **0 on a + control that lacks the shape** *and* **non-zero + precise on ≥2 that have it**. + +4. **Two-part master lever still governs.** An edge only helps if a *realistic + symbol-named explore seeds a path it lies on*. A synthesizer whose far endpoint + no normal query names buys nothing (the trezor "11 explores" tail). Prefer shapes + where both endpoints are names an agent would actually type. + +5. **Partial coverage is worse than none** (playbook §7). Close each flow + *end-to-end* and re-measure; never ship a half-bridged flow. + +--- + +## The backlog (prioritized by frequency × static-resolvability × query-seedability) + +### Tier A — high traffic, cleanly static, build next + +| Shape | Ecosystem | The static anchor that bridges it | Mechanism | Status | +|---|---|---|---|---| +| **Name→class registry / command bus** | any (TS/JS first) | object-literal registry `{key: Handler}` + computed-key dispatch `(new) reg[var](…)` | S (fan-out, `object-registry`) | ✅ **SHIPPED v1 (2026-06-20)** — `objectRegistryEdges`. Links each dispatcher fn → each registered handler's callable entry (a class's `execute`/run/handle method — preferring the method chained at the dispatch — or the function value). Precise on **xrengine** (CommandManager, 64 edges, class registry → `.execute`), **Prebid.js** (7: builder/consent/message dispatch, fn registry), **warp-drive** (1). **0 false positives** after: minified-file skip (avg line >200), **depth-aware** entry parse (top-level `key: Ident` only — method-shorthand/nested-object bodies don't leak), callable-only targets (no data `constant`), dynamic-dispatch gate. Handles constructor + field-initializer (`this.` normalized) forms. **Deferred (recall, documented):** assign-then-call (`const h=reg[k]; h()` — warp-drive's main `COMMANDS`), augmentation (`reg[k]=H` — Prebid single-entry), method-shorthand entry recall, and the **cross-file barrel-namespace** variant (trezor `getMethod`: `import * as M; M[method]→new` + computed dynamic import + camel↔Pascal — the hard tier, still 🔬). | +| **RTK Query** | TS / Redux Toolkit | `createApi({ endpoints: b => ({ getX: b.query(...) }) })` → generated `useGetXQuery` hook → component; endpoint name ↔ hook name (`getX`↔`useGetXQuery`) is convention | X (extract endpoints) + S (endpoint→hook) | ✅ **SHIPPED (2026-06-20)** — `synthesizedBy:'rtk-query'`. **X:** extraction mints a function node per endpoint (named by its key, spanning the `queryFn`/`query` handler so its calls attribute; both `endpoints: b => ({…})` arrow and `endpoints(b){ return {…} }` method forms; a factory-handler endpoint `queryFn: makeFn(url)` falls back to a bare node spanning the builder call) **and** per generated-hook binding from `export const {…} = api` (carrying the sentinel signature `= RTK Query generated hook`). **S:** `rtkQueryEdges` bridges hook→same-file endpoint by the naming convention (strip `use` + optional `Lazy` + `Query`/`Mutation`, lc head). Component→hook is normal import/call resolution; hook→endpoint surfaces in explore as `dynamic: rtk query`. Validated **100% precision** (hooks == synth edges, **0 cross-file**) on **basetool** (small, 54 edges, both forms + factory fallback), **minusx-metabase** (small, 11), **shapeshift** (large, 13); **0** on the uwave-web control (no `createApi` → a complete no-op, 0 nodes/edges added). Sentinel gate correctly ignores hand-written look-alikes (shapeshift's `useFoxyQuery` is a real custom hook, never bridged). **Deferred:** cross-module `injectEndpoints` where the hook destructuring's RHS isn't the same bare api const (synth requires same-file endpoint). | +| **Vuex / Pinia** | Vue | `store.dispatch('ns/action')` / `commit('mutation')` → action/mutation by string key (namespaced); Pinia `useStore().action()` instance call | **X (extract collections) ✅ + S (dispatch bridge) ⬜** | 🟡 **EXTRACTION FOUNDATION SHIPPED (2026-06-20)** — store actions/mutations/getters are now nodes (`codegraph_node login`/`getSessionList` works). Corpus probe found this is **NOT one clean string-keyed shape** — it's ~5: **(1)** Vuex MODULE non-exported `const actions/mutations = {…}` (element-admin), **(2)** Vuex split-file `export default {…}` + computed-key `commit(CONST)` + `mapActions` (vue2-elm), **(3)** Pinia OPTIONS `defineStore({actions:{…}})` (Geeker), **(4)** Pinia SETUP `defineStore('id',()=>{const f=…;return{f}})` body-locals (MallChat), **(5)** Pinia `useStore().action()` instance dispatch. Extraction covers **1, 3, 4** (`extractObjectLiteralFunctions` on `actions`/`mutations`/`getters` collections + a `findPiniaSetupFn`/`extractPiniaSetupBody` for setup locals; `looksLikeVueStoreFile` ≥2-signal gate + the shape gate make it a **0-node no-op on a Redux control** despite the word "actions"). Validated findable on element-admin (50 fns), Geeker (21), MallChat (68); vue2-elm form-2 + computed-key **deferred** (n=1, needs export-default dispatch + const-string resolution). **The dispatch BRIDGE synth, 2 members — BOTH ✅ SHIPPED (2026-06-20):** **(a)** Vuex string-key `dispatch('ns/action')`/`commit('M')` → action/mutation node — `synthesizedBy:'vuex-dispatch'` (`vuexDispatchEdges`): last `/` segment = action name, preceding = namespace; resolve to a function node IN A STORE FILE (the ≥2-signal `isStoreFile` gate excludes a same-named `api/` helper — `getInfo`/`login` collide), disambiguated by the immediate namespace segment in the path (handles DEEP nesting `d2admin/user/set`) or same-file for a root local `commit('M')`. Also added `export default { namespaced, actions:{…}, mutations:{…} }` extraction (the canonical Vuex module form — `extractStoreCollectionMethods` off the export_statement, store-file gated) since d2-admin needs it. **100% precision: element-admin 55 edges, vue-admin-template 12, d2-admin 63; 0 non-store targets, 0 namespace mismatches (54/54 namespaced edges route to the correct module); 0 on Redux controls (basetool/uwave — non-string `dispatch()` ignored).** `+ vuex-dispatch-synthesizer.test.ts`. **(b)** Pinia `useStore().action()` → action — ✅ **SHIPPED (2026-06-20)** `synthesizedBy:'pinia-store'` (`piniaStoreEdges`): maps each `const useXStore=defineStore(…)` factory → its file, binds `const s=useXStore()` per consumer file, links the enclosing fn (or the `.vue` component, via fallback) → the `s.method()` action node IN THE STORE'S FILE (same-store-file gate ⇒ `$patch`/built-ins/unrelated same-named methods resolve to nothing). Covers options + setup forms uniformly. **100% precision** (Geeker 41 edges, MallChat 64; 0 targets outside a store file), 0 on the Vuex-only element-admin control; surfaces as `dynamic: pinia store`; suite 1612 + `pinia-store-synthesizer.test.ts`. Corpus: `/tmp/cg-vuex-eval/{vue-element-admin,vue2-elm,Geeker-Admin,MallChatWeb}`. | +| **NgRx effects** | Angular | `createEffect(() => actions.pipe(ofType(LoginAction), …))` → effect handler; `Store.dispatch(new LoginAction())` → effect by action type/class | S (type/class-keyed) | ⬜ | + +### Tier B — backend command/event/message buses (each needs its own canonical flow + ≥2 repos) + +| Shape | Ecosystem | Anchor | Mechanism | Status | +|---|---|---|---|---| +| **MediatR / CQRS** | .NET | `_mediator.Send(x)`/`.Publish(x)` → the `Handle` of `IRequestHandler`/`INotificationHandler` by request type | S (type-keyed, 2-pass + arg resolution) | ✅ **SHIPPED (2026-06-20)** — `synthesizedBy:'mediatr-dispatch'` (`mediatrDispatchEdges`). Same 2-pass type-keyed shape as Spring, with a twist: **C# method nodes have NO `signature`** (csharp.ts defines no `getSignature`), so Pass 1 reads the request type from the handler **class base-list source** (`: IRequestHandler` — first generic arg), not a param signature, and binds the class's `Handle` method. **The dominant .NET idiom is VARIABLE-passed, not inline** — eShop had **0** genuine inline `Send(new X)` (every send is `mediator.Send(command)`), so Pass 2 RESOLVES the sent type from the argument three ways within the enclosing method: inline `new X(…)`, a local `var v = new X(…)` (backward scan, wins), or a parameter/local declared `X v`. **Two precision gates:** (1) receiver must be mediator-ish (`/mediator|sender|publisher/i` — excludes MAUI `MessagingCenter.Send`, `HttpClient.Send`), (2) resolved type must be in the handler map (so eShop's same-named `CancelOrderCommand` DTO in ClientApp, which has no handler, is never bridged). Handles the `IdentifiedCommand` wrapper (sent + handled at that layer) and void single-arg `IRequestHandler`. **100% precision: jasontaylordev/CleanArchitecture (small, 9 edges, inline + param forms) + dotnet/eShop (medium, 9 edges, 0 FP, variable-passed + IdentifiedCommand + DTO-collision avoided); 0 on the Newtonsoft.Json control.** Node-stable (pure edge synth). Surfaces `dynamic: mediatr dispatch`. `+ mediatr-dispatch-synthesizer.test.ts`. **Deferred (recall):** generic `_mediator.Publish(domainEvent)` over a collection (concrete type erased at the publish site — eShop's DDD AddDomainEvent fan-out), `record`-positional or factory-built args whose type isn't a `new X`/param, the `ICommandHandler` facade indirection (modular-monolith). | +| **Celery** | Python | `@shared_task`/`@app.task`/`@.task`/`@task` def + `.delay()`/`.apply_async()` call → task body | S (decorator-gated name) | ✅ **SHIPPED (2026-06-20)** — `synthesizedBy:'celery-dispatch'` (`celeryDispatchEdges`). Link the enclosing fn at each `.delay(`/`.apply_async(` site → the task fn. Precision rests on the DECORATOR gate: the dispatched name must resolve to a Python `function` carrying a task decorator, read from the source lines ABOVE its `def` (the def's own startLine excludes the decorator; no `decorates` edge exists — `@shared_task` is an unresolved external import). `kind==='function'` filter drops the same-named test-method collision (`consume_file`). Canvas forms (`group(t).delay()`, `t.s()`/`.si()`) have no single identifier before `.delay` → skipped, not mis-bridged. Cross-module name collision → same-file preference else bail. **100% precision: paperless-ngx (small, `@shared_task`, 31 edges, 31/31 real), pretix (medium, `@app.task`, 63 edges across 21 tasks, 0/21 FP); 0 on the httpie control (no Celery).** Node-stable (pure edge synth, no extraction change). Surfaces as `dynamic: celery dispatch @site` via the generic fallback. `+ celery-dispatch-synthesizer.test.ts`. **Deferred (recall):** canvas dispatch, class-based `Task` subclasses, `app.send_task('dotted.name')` string dispatch, aliased imports (`import send_email as s; s.delay()`). | +| **Sidekiq** | Ruby | `W.perform_async(...)`/`.perform_in`/`.perform_at` → `W#perform`, gated on `include Sidekiq::Job`/`Worker` | S (name-keyed class→perform) | ✅ **SHIPPED (2026-06-20)** — `synthesizedBy:'sidekiq-dispatch'` (`sidekiqDispatchEdges`). Name-keyed (like Celery): link the enclosing method at each `Worker.perform_async/_in/_at(…)` site → the worker's instance `perform`. The receiver class must be a Sidekiq worker — gated by reading `include Sidekiq::Job|Worker` from the class BODY source (the mixin is an external gem module → no resolvable edge, like Celery's decorator / Spring's annotation). **Namespace disambiguation (the n>1 fix):** loomio's flat workers hid a collision bug forem exposed — 4 `SendEmailNotificationWorker`s across modules; simple-name resolution mis-targeted 7/143 edges to the wrong namespace. Fixed by resolving a namespaced ref (`Comments::SendEmailNotificationWorker`) via EXACT `getNodesByQualifiedName` first, falling back to simple-name only for a unique worker (ambiguous unqualified collision bails). ActiveJob's `perform_later`/`_now` deliberately NOT matched (different shape → ActiveJob-only app yields 0). **100% precision: loomio (medium, `Sidekiq::Worker`, 47 edges) + forem (large, both aliases — 131 `Sidekiq::Job` + 11 `Sidekiq::Worker`, 142 edges, 0 worker-FP, 0 source-FP, 0 namespace-mismatch); 0 on the jekyll control.** Node-stable. Surfaces `dynamic: sidekiq dispatch`. `+ sidekiq-dispatch-synthesizer.test.ts`. **Deferred (recall):** the superclass-chain variant (diaspora: `class Foo < Base` where only `Base` has the include — worker detection must follow `< Base`), the `Jobs.enqueue(:sym)` facade (Discourse), dispatch from non-method contexts (admin DSL blocks → no enclosing method). | +| **Spring events** | Java | `publishEvent(new XEvent(…))` → `@EventListener`/`@TransactionalEventListener`/`ApplicationListener` by event type | S (type-keyed, 2-pass) | ✅ **SHIPPED (2026-06-20)** — `synthesizedBy:'spring-event'` (`springEventEdges`). Pass 1 builds `Map` — listeners are `@EventListener`/`@TransactionalEventListener` methods (event type = the first param type off the node `signature`, or the `@EventListener(X.class)` value form) + `class … implements ApplicationListener` `onApplicationEvent` methods (name + file `ApplicationListener<` gate). Pass 2 links each `publishEvent(new XEvent(…))` site's enclosing method → every listener of XEvent. **KEY Java fact:** a method node's range INCLUDES its leading annotations (`startLine` = first `@…` line, NOT the `public void` decl), so the annotation gate scans DOWNWARD from startLine, bounded to consecutive `@`-lines (no bleed into an adjacent method). Keyed by EXACT type name, no name resolution — precision is structural (param type ↔ `new X` type). Multi-line `publishEvent(\n new X(…))` handled (`\s*` spans newlines). **100% precision: halo (medium, 1254 java, 33 edges across 24 events, 0 publisher/listener FP, all 3 listener forms + fan-out) + thombergs/code-examples (4 edges incl. the `@TransactionalEventListener` form halo lacks); 0 on the gson control (no Spring).** Node-stable (pure edge synth). Surfaces `dynamic: spring event @site`. `+ spring-event-synthesizer.test.ts`. **Deferred (recall):** `publishEvent(bareVar)` (needs the var's declared type), Spring's listener-return-value re-publish, `@DomainEvents`/`AbstractAggregateRoot.registerEvent`, generic `PayloadApplicationEvent` params. | +| **Laravel events** | PHP | `event(new XEvent(...))` → each listener's `handle`, via a typed `handle(XEvent $e)` AND the `$listen` map | X+S (two registration sources) | ✅ **SHIPPED (2026-06-21)** — `synthesizedBy:'laravel-event'` (`laravelEventEdges`). Pass 1 builds `Map` from BOTH mechanisms (both real, both needed): **(A)** a typed listener `handle(EventType $e)` first param (read from the method decl source — PHP has no `signature`, like C#; splits a `handle(A|B $e)` UNION into two events); **(B)** the `protected $listen = [ XEvent::class => [Listener::class, …] ]` map in an EventServiceProvider — parsed from comment-stripped source (so firefly's fully-commented map on auto-discovery contributes nothing), keys/values as `::class` or string literals — which is the ONLY way to link a listener whose `handle()` is UNTYPED (koel's `PruneLibrary`). Pass 2 links each `event(new XEvent(...))` site → every handle of XEvent. **Job exclusion is free:** queued jobs dispatch via `::dispatch()`/`dispatch()` (not matched) and their `handle()` takes an injected service (never an event type) — so matching ONLY `event(new X)` excludes them by construction; no job-vs-event ambiguity. `use Dispatchable` is NOT keyed on (unreliable — koel 1/9, firefly 5/50 events use it). **100% precision: koel (small, populated `$listen` map, 9 edges incl. the untyped-handle case + a fan-out) + firefly-iii (large, pure auto-discovery / empty `$listen`, 141 edges, 0 source/target FP, 0 namespace-mismatch via use-import check, union split verified); 0 on the guzzle control.** Namespace-agnostic (`FireflyIII\` not hardcoded). Node-stable. Surfaces `dynamic: laravel event`. `+ laravel-event-synthesizer.test.ts`. **Deferred (recall):** `XEvent::dispatch()` static-trait dispatch (neither repo uses it for events — would reintroduce job ambiguity), `Event::listen(closure)`, string-literal `$listen` keys for framework events (parsed but never `event(new)`-dispatched), event simple-name collisions across namespaces (none in the corpus — add qualified disambiguation like Sidekiq if a repo needs it). | + +### Tier C — frontier, ⛔ do **not** build (no static anchor; would add noise) + +| Shape | Why not | +|---|---| +| **RxJS subscribe** | observable→observer is predominantly *anonymous* closures; no name to seed (playbook ⬜, deferred) | +| **MobX / Vue-reactivity / Solid signals** | Proxy reactive runtime — the edge doesn't exist statically at all; silent beats wrong (matches vue-core deferral) | +| **Redux-Saga** | generator `yield put()` / `takeEvery(ACTION, saga*)` — generator-body dispatch, materially harder; revisit only if a real repo demands it | + +### Already shipped (for context) + +| Shape | `synthesizedBy` | Validated on | +|---|---|---| +| Redux thunk | `redux-thunk` | ✅ **generalizes (2026-06-20)** — precise on uwave-web (small, 5 edges), session-desktop (medium, 2), trezor (large, 211); control shapeshift (RTK Query, no thunks) = 0. Receiver-agnostic (`api.dispatch`/`thunkApi.dispatch`/`window.…dispatch` all matched). **⚠️ 2 follow-ups below.** | +| Object-literal registry | `object-registry` | ✅ **shipped (2026-06-20)** — xrengine `CommandManager` (64), Prebid.js (7), warp-drive (1); 0 false positives after 4 precision gates. | +| RTK Query | `rtk-query` | ✅ **shipped (2026-06-20)** — 100% precision (hooks == synth edges, 0 cross-file) on basetool (54), minusx-metabase (11), shapeshift (13); 0 on uwave-web control. Extraction mints endpoint + generated-hook nodes; synth bridges hook→endpoint by convention. | +| Pinia store | `pinia-store` | ✅ **shipped (2026-06-20)** — `useStore().action()` instance dispatch → action; 100% precision Geeker (41) / MallChat (64), 0 on element-admin (Vuex) control. | +| Vuex dispatch | `vuex-dispatch` | ✅ **shipped (2026-06-20)** — string `dispatch('ns/action')`/`commit('M')` → handler; 100% precision element-admin (55) / vue-admin-template (12) / d2-admin (63), 0 on Redux controls. | +| Celery | `celery-dispatch` | ✅ **shipped (2026-06-20)** — `.delay()`/`.apply_async()` → `@shared_task`/`@app.task` body; 100% precision paperless-ngx (31) / pretix (63 across 21 tasks), 0 on httpie control. Decorator-gated via source above the `def`. | +| Spring events | `spring-event` | ✅ **shipped (2026-06-20)** — `publishEvent(new XEvent)` → `@EventListener`/`@TransactionalEventListener`/`ApplicationListener` by event type; 100% precision halo (33 across 24 events) / code-examples (4), 0 on gson control. Type-keyed 2-pass, no name resolution. | +| MediatR | `mediatr-dispatch` | ✅ **shipped (2026-06-20)** — `_mediator.Send(x)`/`.Publish(x)` → the `Handle` of `IRequestHandler`/`INotificationHandler` by request type; 100% precision jasontaylor (9) / eShop (9, variable-passed), 0 on Newtonsoft control. Type from class base-list (C# has no signature) + arg resolved inline/local/param; receiver + handler-map gates. | +| Sidekiq | `sidekiq-dispatch` | ✅ **shipped (2026-06-20)** — `W.perform_async/_in/_at(…)` → `W#perform`, gated on `include Sidekiq::Job`/`Worker`; 100% precision loomio (47) / forem (142, both aliases), 0 on jekyll control. Name-keyed; namespaced collisions disambiguated by qualified name; ActiveJob `perform_later` excluded. | +| Laravel events | `laravel-event` | ✅ **shipped (2026-06-21)** — `event(new XEvent)` → each listener's `handle`, via typed `handle(XEvent $e)` (auto-discovery, union-split) AND the `$listen` map (covers untyped handles); 100% precision koel (9, `$listen`) / firefly (141, auto-discovery), 0 on guzzle control. Jobs excluded (they use `::dispatch`). | +| (see playbook §6 / `callback-synthesizer.ts` for the other ~20 channels) | | | + +### redux-thunk follow-ups (found by the n>1 validation — this is exactly what it's for) + +1. **Precision: name-collision target resolution — ✅ FIXED (2026-06-20).** `reduxThunkEdges` + resolved the dispatched name via `getNodesByName(name).find(kind ∈ {constant,function, + method})` — first match wins, no preference for the thunk. On **octo-call**, `leaveCall` + collides (a `createAsyncThunk` const at `state/call.ts:201` *and* a service `function` + at `services/firestore-signaling.ts:253`); **both** edges mis-resolved to the *service + function*. trezor's long unique thunk names hid this. **Fix:** resolution now prefers a + thunk-signature const > other const > same-file callable > first match (single-candidate + unaffected). Verified: octo-call's 2 edges now target the thunk (`call.ts:201`); uwave's 5 + unchanged; regression test in `__tests__/redux-thunk-synthesizer.test.ts`. +2. **Surfacing: synth edges between non-callable nodes were invisible — ✅ ROOT-CAUSED + FIXED + (2026-06-20).** redux-thunk connects `constant` nodes (thunks are `const X=createAsyncThunk`), + but explore's flow machinery assumed callables, so the hop fell through both surfacing + paths: **(a)** `buildFlowFromNamedSymbols` filtered its named set to + `CALLABLE={method,function,component,constructor}` (tools.ts:1554) → constants never entered + the Flow scan / #687 Dynamic-dispatch-links loop, at any tier; **(b)** the kind-agnostic + `### Relationships` section (which *does* render constant→constant) is + `includeRelationships:false` below 500 files. Net: redux-thunk edges surfaced ONLY via + Relationships, ONLY on repos ≥500 files (uwave/octo-call showed nothing). **Fix (surgical, + tier-independent):** a `dynNamed` set of named CONSTANT/VARIABLE/FIELD nodes that participate + in a heuristic edge feeds the `## Dynamic-dispatch links` scan (main call-chain stays + callable-only); plus a generic `synthEdgeNote` fallback so any synth hop reads + `dynamic: @wiring-site`, not a bare `[calls]`. Verified: uwave `shufflePlaylist→ + loadPlaylist` and `register→login→initState` now surface; trezor unchanged; full suite + + new `__tests__/explore-synth-constant-endpoints.test.ts` pass. **No-op for callable flows** + (dynNamed stays empty) — so it generalizes: any future constant/variable/field-connecting + synth (RTK Query, Vuex) surfaces for free. + +--- + +## Per-synthesizer validation protocol (condensed from the playbook) + +For each shape, before marking ✅: +1. **Grep ≥3 real repos for the pattern**; keep the **2+ that contain it** (small/medium) + + **1 control that lacks it**. (Graph-level precision/recall validation does **not** + need not-trained-on repos — that constraint is only for *agent A/B baselines*.) +2. **Measure the hole**: `select count(*) from edges where synthesizedBy='X'` → + non-zero + node count stable (no explosion) on the pattern repos; **0 on the control**. +3. **Precision spot-check**: sample ~12 edges; source & target must both be real and the + indirection must actually exist in the source body. +4. **Seed a flow**: `scripts/agent-eval/probe-explore.mjs` with the shape's endpoint + symbol names → the Flow section shows the path through the synthesized hop. +5. **Agent A/B** (only for the headline repo, not every control): `--model sonnet + --effort high`, n≥2/arm, record Read/Grep/duration. + +--- + +## Immediate next actions + +- [ ] **Validate redux-thunk for real (workstream 1):** clone a small + medium + `createAsyncThunk`-using app (grep-confirmed), re-index, repeat the protocol. + Promote `redux-thunk` 🟡→✅ or fix the overfit. *(None of the 4 already-cloned + eval repos contain `createAsyncThunk`.)* +- [x] **Decide trezor end (workstream 3):** ✅ **RESOLVED (2026-06-21) — SHELVED as single-lineage / likely-overfit.** + The same-file object-literal half shipped as `object-registry` (xrengine/Prebid/warp-drive). The + remaining **cross-file barrel-namespace** half (`import * as M from './api'` → `M[runtimeKey]` → + `new` → `.run()`) was the open Tier-A item. A grep-confirmed discovery across **15 independent diverse + repos + GitHub-wide code search** found the STRICT shape in exactly **2 repos — trezor-suite AND + OneKey hardware-js-sdk — but OneKey is a `@trezor/connect` FORK** (same `findMethod`/`MethodConstructor`/ + `ApiMethods[method]`/`new`+`BaseMethod.run()` skeleton, 130 vs 61 methods). So it's **2 indexable repos + but ONE design lineage = effectively n=1**. The hypothesis that it "*also* closes n8n/VS-Code registries" + is **DISPROVEN**: every independent "registry by runtime key" is a DIFFERENT shape the trezor-tuned synth + won't catch — n8n = dynamic `import()`-of-computed-PATH + DI (`Container.get`), polkadot = array-of- + constructors + numeric index, ccxt = object-literal/external-pkg (already covered by `object-registry`), + typeorm/bcoin/xrpl = `switch`. The barrel synth is the HARD tier (cross-file barrel re-export enumeration + + computed index + camel↔Pascal + entry-method fan-out) — **meaningful complexity for a single-lineage win**, + which the overfit discipline (see #1 above — the redux-thunk-on-trezor-alone lesson) says **don't build**. + Corpus `/tmp/cg-barrel-eval/` (trezor-suite, onekey-hardware + 15 non-matches). Reopen ONLY if an + independent (non-trezor-lineage) repo with the strict shape turns up. The **facade** + (`connect-common/factory.ts`) remains **low-value** (single `call` fan-in, no per-method disambiguation) — skip. +- [x] **RTK Query (workstream 2 spillover):** ✅ **shipped (2026-06-20)** — + `synthesizedBy:'rtk-query'`, validated on basetool / minusx-metabase / + shapeshift (+ uwave control). See the Tier-A row for the mechanism. + **Next RTK spillover:** the cross-module `injectEndpoints` case (hooks + destructured off an enhanced api in a different file than the base) — the + synth's same-file gate skips it today; would need a same-`reducerPath` or + import-following relaxation, validated on a repo that splits endpoints. diff --git a/docs/design/dynamic-dispatch-coverage-playbook.md b/docs/design/dynamic-dispatch-coverage-playbook.md index 213942a..8a5bb72 100644 --- a/docs/design/dynamic-dispatch-coverage-playbook.md +++ b/docs/design/dynamic-dispatch-coverage-playbook.md @@ -7,6 +7,9 @@ each one the same way, so cross-symbol *flows* exist in the graph everywhere. > This is the top-level playbook. The deep design for one mechanism (the callback > synthesizer) is in [`callback-edge-synthesis.md`](./callback-edge-synthesis.md). +> The cross-cutting **dispatch-shape** queue (Redux/RTK Query/NgRx/MediatR/registries — +> organized by indirection shape, not language×framework) is in +> [`dispatch-synthesizer-backlog.md`](./dispatch-synthesizer-backlog.md). > Full investigation context + findings: auto-memory `project_codegraph_read_displacement`. > **Update (2026-06-01):** the `codegraph_trace` and `codegraph_context` MCP tools were diff --git a/scripts/agent-eval/offload-eval-3arm.sh b/scripts/agent-eval/offload-eval-3arm.sh new file mode 100755 index 0000000..b17c39c --- /dev/null +++ b/scripts/agent-eval/offload-eval-3arm.sh @@ -0,0 +1,72 @@ +#!/usr/bin/env bash +# 3-arm offload eval for ONE indexed repo + ONE question, n reps each. +# ARM offload : codegraph attached, managed offload ON (per-run AI usage log) +# ARM raw : codegraph attached, CODEGRAPH_OFFLOAD_DISABLE=1 (raw source) +# ARM nocg : no codegraph (empty MCP config) -> Read/Grep baseline +# All arms: claude -p sonnet --effort high. One JSON metrics line/run -> $RESULTS. +# +# Usage: offload-eval-3arm.sh "" +# Env: MODEL=sonnet EFFORT=high RESULTS= AGENT_EVAL_OUT= +set -uo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +ENGINE="$(cd "$HERE/../.." && pwd)" +BIN="$ENGINE/dist/bin/codegraph.js" +OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}" +TARGET="${1:?usage: offload-eval-3arm.sh \"\"}" +TIER="${2:?tier}"; REPS="${3:?reps}"; Q="${4:?question}" +RUNS="$OUT/runs" +EXTRACT="$HERE/offload-eval-metrics.mjs" +RESULTS="${RESULTS:-$OUT/results.jsonl}" +REPO=$(basename "$TARGET") +mkdir -p "$RUNS" +command -v claude >/dev/null || { echo "no claude on PATH"; exit 1; } +[ -d "$TARGET/.codegraph" ] || { echo "not indexed: $TARGET (run offload-eval-setup.sh first)"; exit 1; } +# Physical path so pkill matches the daemon's real cmdline (macOS /tmp->/private/tmp symlink +# otherwise makes the kill miss the daemon, and the next arm connects to the SURVIVING daemon +# — contaminating the raw arm with offload). +TARGET=$(cd "$TARGET" && pwd -P) + +prewarm() { # path extra-env (e.g. "FOO=bar") + pkill -9 -f "serve --mcp --path $1" 2>/dev/null; rm -f "$1/.codegraph/daemon.sock" 2>/dev/null; sleep 0.6 + env ${2:-} CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS=1800000 node "$BIN" serve --mcp --path "$1" /dev/null 2>&1 & + node -e 'const fs=require("fs");let n=0;const t=setInterval(()=>{if(fs.existsSync(process.argv[1]+"/.codegraph/daemon.sock")){clearInterval(t);process.exit(0)}if(n++>150){clearInterval(t);process.exit(1)}},100)' "$1" \ + && echo " daemon warm" || echo " WARN daemon never bound" +} + +run() { # arm rep mcp-config usage-log-or-dash + local arm="$1" rep="$2" cfg="$3" usage="$4" tag="$REPO-$1-$2" + [ "$usage" != "-" ] && : > "$usage" + # DISALLOW (optional): block sub-agent delegation across all arms so the A/B + # measures the retrieval mode, not whether Sonnet decides to spawn a codegraph-blind + # Explore subagent (which thrashes regardless and adds huge variance). + ( cd "$TARGET" && claude -p "$Q" \ + --output-format stream-json --verbose --permission-mode bypassPermissions \ + --model "${MODEL:-sonnet}" --effort "${EFFORT:-high}" --max-budget-usd 4 \ + ${DISALLOW:+--disallowedTools "$DISALLOW"} \ + --strict-mcp-config --mcp-config "$cfg" \ + "$RUNS/$tag.jsonl" 2>"$RUNS/$tag.err" ) + node "$EXTRACT" --run "$RUNS/$tag.jsonl" --usage "$usage" --arm "$arm" --rep "$rep" \ + --repo "$REPO" --tier "$TIER" --q "$Q" >> "$RESULTS" + node -e 'const o=JSON.parse(require("fs").readFileSync(process.argv[1],"utf8").trim().split("\n").pop());console.log(` [${o.arm} #${o.rep}] ${o.durationSec}s | main $${o.costUsdMain} ${o.tokBillable} tok | read=${o.read} grep=${o.grep} explore=${o.explore} offload=${o.offloadFired} | AI ${o.ai.calls}call/${o.ai.totalTokens}tok/$${o.ai.costUsd.toFixed(4)} | ok=${o.ok}`)' "$RESULTS" +} + +CFG_OFF="$RUNS/mcp-offload-$REPO.json"; CFG_RAW="$RUNS/mcp-raw-$REPO.json"; CFG_NOCG="$RUNS/mcp-nocg.json" +USAGE="$RUNS/$REPO-usage.jsonl" +printf '{"mcpServers":{"codegraph":{"command":"env","args":["CODEGRAPH_WASM_RELAUNCHED=1","CODEGRAPH_OFFLOAD_USAGE_LOG=%s","node","%s","serve","--mcp","--path","%s"]}}}' "$USAGE" "$BIN" "$TARGET" > "$CFG_OFF" +printf '{"mcpServers":{"codegraph":{"command":"env","args":["CODEGRAPH_WASM_RELAUNCHED=1","CODEGRAPH_OFFLOAD_DISABLE=1","node","%s","serve","--mcp","--path","%s"]}}}' "$BIN" "$TARGET" > "$CFG_RAW" +printf '{"mcpServers":{}}' > "$CFG_NOCG" + +# REP_START lets a later batch ADD reps without clobbering earlier jsonls +# (e.g. REP_START=4 REPS=3 -> reps 4,5,6; default starts at 1). +START="${REP_START:-1}"; END=$((START + REPS - 1)) +echo "###### repo=$REPO tier=$TIER reps=$START..$END model=${MODEL:-sonnet}/${EFFORT:-high}" +echo "###### Q=$Q" +echo "== ARM offload =="; prewarm "$TARGET" "CODEGRAPH_OFFLOAD_USAGE_LOG=$USAGE" +for r in $(seq "$START" "$END"); do run offload "$r" "$CFG_OFF" "$USAGE"; done +pkill -9 -f "serve --mcp --path $TARGET" 2>/dev/null; rm -f "$TARGET/.codegraph/daemon.sock" 2>/dev/null; sleep 1 +echo "== ARM raw =="; prewarm "$TARGET" "CODEGRAPH_OFFLOAD_DISABLE=1" +for r in $(seq "$START" "$END"); do run raw "$r" "$CFG_RAW" "-"; done +pkill -9 -f "serve --mcp --path $TARGET" 2>/dev/null; rm -f "$TARGET/.codegraph/daemon.sock" 2>/dev/null; sleep 1 +echo "== ARM nocg ==" +for r in $(seq "$START" "$END"); do run nocg "$r" "$CFG_NOCG" "-"; done +echo "###### DONE $REPO" diff --git a/scripts/agent-eval/offload-eval-cost.mjs b/scripts/agent-eval/offload-eval-cost.mjs new file mode 100644 index 0000000..90b5bda --- /dev/null +++ b/scripts/agent-eval/offload-eval-cost.mjs @@ -0,0 +1,133 @@ +#!/usr/bin/env node +// Cost/token analysis for the 3-arm offload eval, with a MAIN-vs-SUBAGENT split. +// +// The explore-subagent question. With delegation ALLOWED, the nocg arm spawns a +// Claude Code Explore subagent; the codegraph arms do all work in the main agent. +// Two facts make naive accounting wrong: +// 1. The Explore subagent runs on HAIKU 4.5; the main agent on SONNET 4.6. +// So per-token cost differs ~3x between them — you cannot price both the same. +// 2. The subagent's consumption is ~95% cache-reads. At Haiku's $0.10/MTok +// cache-read rate, a huge TOKEN volume is a small DOLLAR cost. +// +// Rather than re-derive cost from raw token counts (and guess the cache TTL — +// Claude Code uses 1-hour ephemeral cache here, 2x write, not 5-min), we read +// Claude Code's OWN authoritative accounting from the `result` event: +// result.modelUsage[model].costUSD — per-model cost CC itself billed +// result.total_cost_usd — their sum (INCLUDES the Haiku subagent; +// the handoff's "excludes subagent" was wrong) +// The model split IS the agent split here: sonnet => main, haiku => Explore subagent +// (only nocg spawns one, and only nocg shows haiku usage). Token volume is still +// summed per-model from modelUsage for the separate "tokens" story. +// +// Usage: offload-eval-cost.mjs [reps] +// e.g. offload-eval-cost.mjs /tmp/cg-offload-eval/runs trezor 3 +import { readFileSync, existsSync } from 'fs'; + +const MAIN_TIER = /sonnet/; // main agent +const SUB_TIER = /haiku/; // Claude Code Explore subagent + +const [,, runsDir, repo, repsArg] = process.argv; +if (!runsDir || !repo) { console.error('usage: offload-eval-cost.mjs [reps] (env ARMS=nocg,raw,offload)'); process.exit(1); } +const REPS = Number(repsArg || 3); +// Arms to analyze (file stems `--.jsonl`). Override for the style A/B: +// ARMS=raw,refs,map,src. nocg's Haiku subagent is the only sub-tier; the rest are main-only. +const ARMS = (process.env.ARMS || 'nocg,raw,offload').split(',').map((s) => s.trim()).filter(Boolean); + +const toks = (u) => (u.inputTokens||0)+(u.outputTokens||0)+(u.cacheReadInputTokens||0)+(u.cacheCreationInputTokens||0); + +function analyzeRun(file) { + let result = null, agentCalls = 0; + const tools = {}, subPids = new Set(); + for (const line of readFileSync(file, 'utf8').split('\n')) { + if (!line) continue; + let e; try { e = JSON.parse(line); } catch { continue; } + if (e.parent_tool_use_id && e.message?.usage) subPids.add(e.parent_tool_use_id); + if (e.type === 'assistant' && Array.isArray(e.message?.content)) + for (const b of e.message.content) + if (b.type === 'tool_use') { tools[b.name] = (tools[b.name]||0)+1; if (b.name === 'Agent') agentCalls++; } + if (e.type === 'result') result = e; + } + // Authoritative cost + tokens from Claude Code's per-model accounting. + const mu = result?.modelUsage || {}; + const main = { cost: 0, tok: 0 }, sub = { cost: 0, tok: 0 }; + for (const [model, u] of Object.entries(mu)) { + const bucket = SUB_TIER.test(model) ? sub : main; // sonnet/anything-else => main + bucket.cost += u.costUSD || 0; + bucket.tok += toks(u); + } + return { + main, sub, subagents: subPids.size, agentCalls, + ccTotal: result?.total_cost_usd ?? null, + ok: result?.subtype === 'success', + durationSec: result?.duration_ms ? +(result.duration_ms/1000).toFixed(1) : null, + models: Object.keys(mu), tools, + }; +} + +const k = (n) => (n/1000).toFixed(0).padStart(5) + 'K'; +const d = (n) => '$' + n.toFixed(3); +const cost = (b) => b.cost; +const tot = (b) => b.tok; + +const byArm = {}; +for (const arm of ARMS) { + const runs = []; + for (let r = 1; r <= REPS; r++) { + const f = `${runsDir}/${repo}-${arm}-${r}.jsonl`; + if (existsSync(f)) runs.push({ rep: r, ...analyzeRun(f) }); + } + byArm[arm] = runs; +} + +// Per-run detail. Cost is Claude Code's own modelUsage.costUSD (authoritative, +// per-model pricing + correct cache TTL). MAIN=Sonnet, SUB=Haiku Explore subagent. +// cc-check: main$+sub$ must equal result.total_cost_usd (delta should be ~0). +console.log(`\n=== ${repo}: per-run main(Sonnet)/sub(Haiku) split — Claude Code's own cost accounting ===`); +console.log('arm rep | subAg | MAIN(sonnet) tok / $ | SUB(haiku) tok / $ | TOTAL tok / $ | cc_total Δ | dur reads'); +for (const arm of ARMS) for (const r of byArm[arm]) { + const mC = cost(r.main), sC = cost(r.sub), mT = tot(r.main), sT = tot(r.sub); + const reads = r.tools['Read'] || 0, grep = (r.tools['Grep']||0)+(r.tools['Bash']||0)+(r.tools['Glob']||0); + const explore = r.tools['mcp__codegraph__codegraph_explore'] || 0; + const delta = (mC + sC) - (r.ccTotal || 0); // should be ~0 + console.log( + `${arm.padEnd(8)} #${r.rep} | ${String(r.subagents).padStart(2)} | ${k(mT)} ${d(mC).padStart(7)} | ${k(sT)} ${d(sC).padStart(7)} | ${k(mT+sT)} ${d(mC+sC).padStart(7)} | ${d(r.ccTotal||0).padStart(7)} ${(delta>=0?'+':'')+delta.toFixed(4)} | ${String(r.durationSec).padStart(5)} r=${reads} g=${grep} x=${explore}` + ); +} + +// Per-arm means +const mean = (arr, f) => arr.length ? arr.reduce((s,x)=>s+f(x),0)/arr.length : 0; +console.log(`\n=== ${repo}: per-arm MEANS (n per arm) ===`); +console.log('arm n | main $ sub $ TOTAL $ | main tok sub tok TOTAL tok | %$ in sub | %tok in sub'); +for (const arm of ARMS) { + const runs = byArm[arm]; if (!runs.length) continue; + const mC = mean(runs, r=>cost(r.main)), sC = mean(runs, r=>cost(r.sub)); + const mT = mean(runs, r=>tot(r.main)), sT = mean(runs, r=>tot(r.sub)); + const pctSubC = (mC+sC) ? (100*sC/(mC+sC)) : 0; + const pctSubT = (mT+sT) ? (100*sT/(mT+sT)) : 0; + console.log( + `${arm.padEnd(8)} ${runs.length} | ${d(mC).padStart(7)} ${d(sC).padStart(7)} ${d(mC+sC).padStart(7)} | ${k(mT)} ${k(sT)} ${k(mT+sT)} | ${pctSubC.toFixed(0).padStart(3)}% | ${pctSubT.toFixed(0).padStart(3)}%` + ); +} + +// Headline ladders — cost, tokens, duration, all vs a baseline (nocg if present, else first arm). +console.log(`\n=== Ladders (mean, incl. subagent) ===`); +const totals = ARMS.map(a => ({ a, c: mean(byArm[a], r=>cost(r.main)+cost(r.sub)), t: mean(byArm[a], r=>tot(r.main)+tot(r.sub)) })).filter(x=>byArm[x.a].length); +const base = totals.find(x=>x.a==='nocg') ?? totals[0]; +const bn = base?.a ?? '?'; +console.log(` COST (vs ${bn}):`); +for (const x of totals) { + const vs = base && base.c ? ` (${((x.c/base.c-1)*100>=0?'+':'')}${((x.c/base.c-1)*100).toFixed(0)}%)` : ''; + console.log(` ${x.a.padEnd(8)} ${d(x.c)}${vs}`); +} +console.log(` TOKENS (vs ${bn}):`); +for (const x of totals) { + const vs = base && base.t ? ` (${((x.t/base.t-1)*100>=0?'+':'')}${((x.t/base.t-1)*100).toFixed(0)}%)` : ''; + console.log(` ${x.a.padEnd(8)} ${k(x.t)}${vs}`); +} +console.log(` DURATION (wall-clock, vs ${bn}):`); +const durs = ARMS.map(a => ({ a, s: mean(byArm[a].filter(r=>r.durationSec!=null), r=>r.durationSec) })).filter(x=>byArm[x.a].length); +const dbase = durs.find(x=>x.a==='nocg') ?? durs[0]; +for (const x of durs) { + const vs = dbase && dbase.s ? ` (${((x.s/dbase.s-1)*100>=0?'+':'')}${((x.s/dbase.s-1)*100).toFixed(0)}%)` : ''; + console.log(` ${x.a.padEnd(8)} ${x.s.toFixed(0)}s${vs}`); +} diff --git a/scripts/agent-eval/offload-eval-effort.mjs b/scripts/agent-eval/offload-eval-effort.mjs new file mode 100644 index 0000000..c6d275e --- /dev/null +++ b/scripts/agent-eval/offload-eval-effort.mjs @@ -0,0 +1,108 @@ +#!/usr/bin/env node +// Effort A/B — does CODEGRAPH_OFFLOAD_EFFORT=high improve offload SYNTHESIS FIDELITY vs low? +// Probe-based (no agent): for each repo × effort × rep, run codegraph_explore with the offload +// ON on the canonical question, capture the synthesized answer + AI tokens/cost/latency, then +// Sonnet-judge that answer's fidelity vs source-verified ground truth. Isolates the synthesis +// from agent/adoption noise. Requires `codegraph login` (managed offload) + indexed repos. +// +// Env: REPS (default 3) · CG_ENGINE (engine repo) · AGENT_EVAL_OUT (repos under /repos) · CONC (judge concurrency) +import { pathToFileURL, fileURLToPath } from 'node:url'; +import { resolve, dirname, join } from 'node:path'; +import { readFileSync, writeFileSync, existsSync, rmSync } from 'node:fs'; +import { execFile } from 'node:child_process'; +import { tmpdir } from 'node:os'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const ENGINE = process.env.CG_ENGINE || resolve(HERE, '..', '..'); +const OUT = process.env.AGENT_EVAL_OUT || '/tmp/cg-offload-eval'; +const REPOS = join(OUT, 'repos'); +const GT = JSON.parse(readFileSync(resolve(HERE, 'offload-eval-ground-truth.json'), 'utf8')); +const REPS = Number(process.env.REPS || 3); +const CONC = Number(process.env.CONC || 4); +const EFFORTS = (process.env.EFFORTS_FILTER || 'low,high').split(','); +const ONLY = process.env.REPOS_FILTER ? new Set(process.env.REPOS_FILTER.split(',')) : null; +const TIER = { mtkruto: 'small', postybirb: 'medium', shapeshift: 'complex', trezor: 'large' }; + +const load = async (rel) => import(pathToFileURL(resolve(ENGINE, rel)).href); +const idx = await load('dist/index.js'); +const toolsMod = await load('dist/mcp/tools.js'); +const CodeGraph = idx.default?.default ?? idx.default ?? idx.CodeGraph; +const ToolHandler = toolsMod.ToolHandler ?? toolsMod.default?.ToolHandler; +if (typeof CodeGraph?.openSync !== 'function' || typeof ToolHandler !== 'function') { + console.error('could not load engine from', ENGINE); process.exit(2); +} + +const fidPrompt = (gt, ans) => `You are scoring the FIDELITY of a machine-synthesized code-exploration answer against verified ground truth. Do NOT use any tools. + +QUESTION: ${gt.question} + +VERIFIED GROUND TRUTH (the actual call path + files): +${gt.truth} + +SYNTHESIZED ANSWER (to score): +${ans || '(empty)'} + +Judge: (1) is the traced call path correct vs ground truth? (2) are the cited files/symbols correct (not fabricated)? (3) if it gave a "Coverage:" verdict, was it honest? A confident WRONG trace is the worst outcome — penalize it harder than an honest partial. +Output ONLY minified JSON: {"verdict":"pass|partial|fail","score":<0-100>,"fabrication":,"coverageHonest":,"note":"<=20 words"}`; + +const askJudge = (prompt) => new Promise((res) => { + execFile('claude', ['-p', prompt, '--model', 'sonnet', '--effort', 'high', '--max-budget-usd', '0.5', + '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}'], + { cwd: OUT, maxBuffer: 1 << 24, timeout: 120000 }, (err, stdout) => { + const m = (stdout || '').match(/\{[\s\S]*\}/); + if (!m) return res({ verdict: 'error', score: null, note: (err ? err.message : 'no json').slice(0, 60) }); + try { res(JSON.parse(m[0])); } catch { res({ verdict: 'error', score: null }); } + }); +}); + +// ---- 1. Probe: collect synthesized answers at each effort ------------------- +const records = []; +for (const repo of Object.keys(GT)) { + if (ONLY && !ONLY.has(repo)) continue; + const dir = join(REPOS, repo); + if (!existsSync(join(dir, '.codegraph'))) { console.error('skip (not indexed):', repo); continue; } + const cg = CodeGraph.openSync(dir); + const h = new ToolHandler(cg); + for (const effort of EFFORTS) { + for (let rep = 1; rep <= REPS; rep++) { + process.env.CODEGRAPH_OFFLOAD_EFFORT = effort; + const usageLog = join(tmpdir(), `effort-${repo}-${effort}-${rep}.jsonl`); + try { rmSync(usageLog); } catch { /* none */ } + process.env.CODEGRAPH_OFFLOAD_USAGE_LOG = usageLog; + let answer = ''; + try { answer = (await h.execute('codegraph_explore', { query: GT[repo].question }))?.content?.[0]?.text ?? ''; } + catch (e) { console.error(` ${repo}/${effort}#${rep} explore failed: ${e?.message}`); } + const fired = /Synthesized by CodeGraph/.test(answer); + const ai = { tokens: 0, cost: 0, ms: 0 }; + if (existsSync(usageLog)) for (const e of readFileSync(usageLog, 'utf8').split('\n').filter(Boolean).map(JSON.parse)) { + ai.tokens += e.totalTokens || 0; ai.cost += e.costUsd || 0; ai.ms += e.ms || 0; + } + records.push({ repo, tier: TIER[repo], effort, rep, fired, ai, answer }); + console.error(` ${repo}/${effort}#${rep}: fired=${fired} ${ai.tokens}tok $${ai.cost.toFixed(4)} ${ai.ms}ms`); + } + } + try { cg.close?.(); } catch { /* none */ } +} + +// ---- 2. Judge fidelity (concurrency) --------------------------------------- +console.error(`\njudging ${records.length} answers (concurrency ${CONC})...`); +let done = 0; +const q = [...records]; +async function worker() { while (q.length) { const r = q.shift(); r.fid = await askJudge(fidPrompt(GT[r.repo], r.answer)); console.error(` [${++done}/${records.length}] ${r.repo}/${r.effort}#${r.rep}: ${r.fid.verdict} ${r.fid.score ?? ''}`); } } +await Promise.all(Array.from({ length: CONC }, worker)); +writeFileSync(join(OUT, 'effort-results.jsonl'), records.map((r) => JSON.stringify(r)).join('\n') + '\n'); + +// ---- 3. Aggregate: low vs high per repo ------------------------------------ +const med = (a) => { a = a.filter((x) => x != null).sort((x, y) => x - y); return a.length ? (a.length % 2 ? a[(a.length - 1) / 2] : (a[a.length / 2 - 1] + a[a.length / 2]) / 2) : null; }; +console.log(`\n${'='.repeat(80)}\nEFFORT A/B — offload synthesis fidelity (probe, n=${REPS}/cell)\n${'='.repeat(80)}`); +console.log(`${'repo'.padEnd(11)} ${'tier'.padEnd(8)} ${'effort'.padEnd(6)} fired ${'fid(med)'.padStart(8)} ${'fab%'.padStart(5)} ${'AItok'.padStart(7)} ${'AIcost'.padStart(8)} ${'ms(med)'.padStart(8)}`); +for (const repo of Object.keys(GT)) { + for (const effort of EFFORTS) { + const rs = records.filter((r) => r.repo === repo && r.effort === effort); + if (!rs.length) continue; + const fids = rs.map((r) => r.fid?.score).filter((x) => x != null); + const fab = rs.filter((r) => r.fid?.fabrication === true).length; + console.log(`${repo.padEnd(11)} ${TIER[repo].padEnd(8)} ${effort.padEnd(6)} ${rs.filter((r) => r.fired).length}/${rs.length} ${String(med(fids) ?? '—').padStart(8)} ${String(Math.round(100 * fab / rs.length) + '%').padStart(5)} ${String(Math.round(med(rs.map((r) => r.ai.tokens)) / 1000) + 'k').padStart(7)} ${('$' + (med(rs.map((r) => r.ai.cost)) ?? 0).toFixed(4)).padStart(8)} ${String(med(rs.map((r) => r.ai.ms)) ?? '—').padStart(8)}`); + } +} +console.log(''); diff --git a/scripts/agent-eval/offload-eval-frontload-matrix.sh b/scripts/agent-eval/offload-eval-frontload-matrix.sh new file mode 100755 index 0000000..8571888 --- /dev/null +++ b/scripts/agent-eval/offload-eval-frontload-matrix.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Run the FRONTLOAD arm across all 4 tiers (n reps), then judge + merge with the existing +# matrix (offload/raw/nocg in $OUT/judged.jsonl, if present) + emit a combined summary. +# Env: REPS (default 3) AGENT_EVAL_OUT= +set -uo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}" +GT="$HERE/offload-eval-ground-truth.json" +REPS="${REPS:-3}" +export RESULTS="$OUT/results-fl.jsonl" +: > "$RESULTS"; rm -f "$OUT/runs/hook-debug.log" +for repo in mtkruto postybirb shapeshift trezor; do + case "$repo" in mtkruto) tier=small;; postybirb) tier=medium;; shapeshift) tier=complex;; trezor) tier=large;; esac + Q=$(node -e "console.log(JSON.parse(require('fs').readFileSync(process.argv[1],'utf8'))[process.argv[2]].question)" "$GT" "$repo") + echo ""; echo "### $repo ($tier) $(date +%H:%M:%S)" + bash "$HERE/offload-eval-frontload.sh" "$OUT/repos/$repo" "$tier" "$REPS" "$Q" +done +echo "" +echo "frontload: $(wc -l < "$RESULTS") runs | hook injections: $(grep -c INJECTED "$OUT/runs/hook-debug.log" 2>/dev/null) | errors: $(grep -c ERROR "$OUT/runs/hook-debug.log" 2>/dev/null)" +echo "=== JUDGE frontload ===" +node "$HERE/offload-eval-judge.mjs" --results "$RESULTS" --truth "$GT" --out "$OUT/judged-fl.jsonl" --concurrency 4 2>&1 | tail -4 +if [ -f "$OUT/judged.jsonl" ]; then cat "$OUT/judged.jsonl" "$OUT/judged-fl.jsonl" > "$OUT/judged-all.jsonl"; else cp "$OUT/judged-fl.jsonl" "$OUT/judged-all.jsonl"; fi +echo "=== COMBINED SUMMARY ===" +node "$HERE/offload-eval-summarize.mjs" "$OUT/judged-all.jsonl" +echo "###### FRONTLOAD MATRIX DONE" diff --git a/scripts/agent-eval/offload-eval-frontload.sh b/scripts/agent-eval/offload-eval-frontload.sh new file mode 100755 index 0000000..abf6a54 --- /dev/null +++ b/scripts/agent-eval/offload-eval-frontload.sh @@ -0,0 +1,47 @@ +#!/usr/bin/env bash +# FRONTLOAD arm (approach 1): codegraph attached (offload-disabled) + the front-load +# UserPromptSubmit hook (offload-eval-hook.mjs), n reps, appended to $RESULTS. Compare against +# the matrix's raw/nocg baselines. Usage: offload-eval-frontload.sh "" +# Env: MODEL=sonnet EFFORT=high RESULTS= AGENT_EVAL_OUT= +set -uo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +ENGINE="$(cd "$HERE/../.." && pwd)" +BIN="$ENGINE/dist/bin/codegraph.js" +OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}" +TARGET="${1:?repo}"; TIER="${2:?tier}"; REPS="${3:?reps}"; Q="${4:?question}" +RUNS="$OUT/runs" +EXTRACT="$HERE/offload-eval-metrics.mjs" +RESULTS="${RESULTS:-$OUT/results-fl.jsonl}" +REPO=$(basename "$TARGET") +mkdir -p "$RUNS" +[ -d "$TARGET/.codegraph" ] || { echo "not indexed: $TARGET"; exit 1; } +TARGET=$(cd "$TARGET" && pwd -P) + +CFG="$RUNS/mcp-fl-$REPO.json" +printf '{"mcpServers":{"codegraph":{"command":"env","args":["CODEGRAPH_WASM_RELAUNCHED=1","CODEGRAPH_OFFLOAD_DISABLE=1","node","%s","serve","--mcp","--path","%s"]}}}' "$BIN" "$TARGET" > "$CFG" +# Generate the hook settings pointing at the persisted hook; enable its debug log so we can +# count injections (claude passes this env down to the spawned hook process). +HOOKCFG="$RUNS/frontload-settings.json" +printf '{"hooks":{"UserPromptSubmit":[{"hooks":[{"type":"command","command":"node %s/offload-eval-hook.mjs"}]}]}}' "$HERE" > "$HOOKCFG" +export CG_FRONTLOAD_DEBUG="$RUNS/hook-debug.log" + +prewarm() { + pkill -9 -f "serve --mcp --path $1" 2>/dev/null; rm -f "$1/.codegraph/daemon.sock" 2>/dev/null; sleep 0.6 + env CODEGRAPH_OFFLOAD_DISABLE=1 CODEGRAPH_DAEMON_IDLE_TIMEOUT_MS=1800000 node "$BIN" serve --mcp --path "$1" /dev/null 2>&1 & + node -e 'const fs=require("fs");let n=0;const t=setInterval(()=>{if(fs.existsSync(process.argv[1]+"/.codegraph/daemon.sock")){clearInterval(t);process.exit(0)}if(n++>150){clearInterval(t);process.exit(1)}},100)' "$1" \ + && echo " daemon warm" || echo " WARN no daemon" +} + +echo "###### FRONTLOAD repo=$REPO tier=$TIER reps=$REPS" +prewarm "$TARGET" +for r in $(seq 1 "$REPS"); do + tag="$REPO-frontload-$r" + ( cd "$TARGET" && claude -p "$Q" --output-format stream-json --verbose --permission-mode bypassPermissions \ + --model "${MODEL:-sonnet}" --effort "${EFFORT:-high}" --max-budget-usd 4 \ + --strict-mcp-config --mcp-config "$CFG" --settings "$HOOKCFG" \ + "$RUNS/$tag.jsonl" 2>"$RUNS/$tag.err" ) + node "$EXTRACT" --run "$RUNS/$tag.jsonl" --usage "-" --arm frontload --rep "$r" --repo "$REPO" --tier "$TIER" --q "$Q" >> "$RESULTS" + node -e 'const o=JSON.parse(require("fs").readFileSync(process.argv[1],"utf8").trim().split("\n").pop());console.log(` [frontload #${o.rep}] ${o.durationSec}s | main $${o.costUsdMain} ${o.tokBillable}tok | read=${o.read} grep=${o.grep} agentExplore=${o.explore} | ok=${o.ok}`)' "$RESULTS" +done +pkill -9 -f "serve --mcp --path $TARGET" 2>/dev/null; rm -f "$TARGET/.codegraph/daemon.sock" 2>/dev/null +echo "###### FRONTLOAD DONE $REPO (cumulative hook injections: $(grep -c INJECTED "$CG_FRONTLOAD_DEBUG" 2>/dev/null))" diff --git a/scripts/agent-eval/offload-eval-ground-truth.json b/scripts/agent-eval/offload-eval-ground-truth.json new file mode 100644 index 0000000..3fdcf6c --- /dev/null +++ b/scripts/agent-eval/offload-eval-ground-truth.json @@ -0,0 +1,18 @@ +{ + "mtkruto": { + "question": "How does calling the high-level client.sendMessage() method get the message serialized into a TL/MTProto request and sent over the network transport to Telegram's servers? Trace the call path.", + "truth": "Verified call path (small TS Telegram-client lib):\n1. Client.sendMessage — client/6_client.ts:1852 → calls this.#messageManager.sendMessage(...) (1853)\n2. MessageManager.sendMessage — client/3_message_manager.ts:330 → builds the TL function {_:\"messages.sendMessage\", peer, message,...} and calls this.#c.invoke(...) (~377)\n3. c.invoke closure / Client.#invoke — client/6_client.ts:279/887 → resolves a ClientEncrypted and calls client.invoke(function_) (897)\n4. ClientEncrypted.invoke — client/2_client_encrypted.ts:324 → this.#send(function_) (325)\n5. ClientEncrypted.#send — client/2_client_encrypted.ts:261 → SERIALIZES via Api.serializeObject(function_) (290), then this.session.send(body) (296)\n - Serialization: Api.serializeObject (tl/2_telegram.ts:38) → new TLWriter().writeObject(...) (tl/1_tl_writer.ts) writes constructor id + fields\n6. SessionEncrypted.send — session/2_session_encrypted.ts:193 → ENQUEUES a PendingMessage and wakes the send loop (#awakeSendLoop) [DYNAMIC: async queue, not a direct call]\n7. SessionEncrypted.#sendLoopBody (AbortableLoop) — session/2_session_encrypted.ts:282 → #encryptMessage (serializeMessage tl/2_message.ts + IGE-256 AES) then this.transport.transport.send(payload) (339)\n8. Transport.send — TransportAbridged.send (transport/1_transport_abridged.ts:71) or TransportIntermediate → this.#connection.write(encrypt(framed))\n9. ConnectionTCP.write (connection/1_connection_tcp.ts:96) or ConnectionWebSocket.write — bytes exit the process.\nKEY SYMBOLS a correct answer must hit: sendMessage → MessageManager.sendMessage → invoke (ClientEncrypted.invoke/#send) → Api.serializeObject/TLWriter → SessionEncrypted.send (queue) → #sendLoopBody/#encryptMessage → Transport.send → Connection.write.\nDYNAMIC BOUNDARIES: the invoke indirection (closure→#invoke→ClientEncrypted), and the send-loop QUEUE between SessionEncrypted.send and #sendLoopBody (async, no direct edge)." + }, + "postybirb": { + "question": "How does submitting/queueing a post reach the website-specific code that actually uploads to a target website? Trace the path from the submission entry point in the NestJS server to a concrete website service's post implementation.", + "truth": "Verified call path (NestJS+Electron, server under electron-app/src/server/):\n1. Entry: PostController.queue (submission/post/post.controller.ts:33, POST queue/:id) OR SubmissionService.queueScheduledSubmissions (submission/submission.service.ts, @Interval(60000) scheduler) — both call PostService.queue\n2. PostService.queue — submission/post/post.service.ts:68 → this.post(submission)\n3. PostService.post (private) — post.service.ts:206 → this.createPoster(...) for each non-default SubmissionPart\n4. PostService.createPoster — post.service.ts:373 → website = this.websites.getWebsiteModule(part.website) [part.website is a STRING]\n5. WebsiteProvider.getWebsiteModule — websites/website-provider.service.ts:86 → websiteModulesMap[name.toLowerCase()] [DYNAMIC: string-keyed registry of DI-injected Website singletons]; then new Poster(..., website, ...)\n6. Poster constructor — submission/post/poster.ts:117 → setTimeout(this.post.bind(this), delay) [DYNAMIC: async timer, min ~5s]\n7. Poster.post (poster.ts:131) → performPost (171) → attemptPost (217)\n8. Poster.attemptPost — poster.ts:217 → this.website.postFileSubmission(token, data, accountData) [DYNAMIC: polymorphic dispatch on abstract Website] (or postNotificationSubmission)\n9. Website.postFileSubmission (abstract) — websites/website.base.ts:102\n10. Concrete e.g. FurAffinity.postFileSubmission — websites/fur-affinity/fur-affinity.service.ts:231 → multi-step HTTP upload (GET /submit, POST /submit/upload multipart, POST /submit/finalize)\nKEY SYMBOLS: PostService.queue → PostService.post → createPoster → WebsiteProvider.getWebsiteModule (registry) → Poster (setTimeout) → Poster.attemptPost → Website.postFileSubmission (abstract base) → a concrete website service (e.g. FurAffinity).\nDYNAMIC BOUNDARIES: NestJS DI builds the website registry; string-keyed map lookup; setTimeout defers Poster.post; polymorphic dispatch on the abstract Website base. A correct answer must reach a concrete website's post via the registry + base class, not stop at PostService." + }, + "shapeshift": { + "question": "How does executing a swap work in ShapeShift — from the code that fetches quotes and selects a swapper down to a specific swapper's execute/trade? Name the swapper interface, the registry, and one concrete swapper, and trace the path.", + "truth": "Verified call path (large multi-package monorepo; swap logic in packages/swapper + execution in src/lib):\nQUOTE/REGISTRY layer:\n- Swapper interface = the `Swapper` type (execute methods) + `SwapperApi` type (getTradeQuote/getUnsignedTx) — packages/swapper/src/types.ts (~846/897)\n- Registry = `swappers: Record` — packages/swapper/src/constants.ts:52 (merges e.g. thorchainSwapper + thorchainApi)\n- Aggregator: swapperApi RTK endpoint getTradeQuote/getTradeRates (src/state/apis/swapper/swapperApi.ts:78/156) → getTradeQuotes (packages/swapper/src/swapper.ts:18/26): swapper = swappers[swapperName]; swapper.getTradeQuote(...) [DYNAMIC interface dispatch]\n- Concrete (THORChain): thorchainApi.getTradeQuote (swappers/ThorchainSwapper/endpoints.ts → getTradeQuote/getTradeQuote.ts:15) → getL1RateOrQuote → getQuote (thorService HTTP)\nSELECTION: tradeQuoteSlice selectors rank quotes (selectSortedTradeQuotes / selectActiveSwapperName) [DYNAMIC: winner chosen by ranking/user]\nEXECUTION layer:\n- useTradeExecution (src/components/MultiHopTrade/.../hooks/useTradeExecution.tsx:200/476) → new TradeExecution(); execution.execEvmTransaction(...) (CowSwap: execEvmMessage)\n- TradeExecution.execEvmTransaction — src/lib/tradeExecution.ts:326 → _execWalletAgnostic(...) (372)\n- TradeExecution._execWalletAgnostic — tradeExecution.ts:136 → swapper = swappers[swapperName] (149) [DYNAMIC registry]; buildSignBroadcast → swapper.getUnsignedEvmTransaction(...) (355) then swapper.executeEvmTransaction(unsignedTx, {signAndBroadcastTransaction}) (367) [DYNAMIC SwapperApi/Swapper interface]\n- Concrete (THORChain): thorchainSwapper.executeEvmTransaction (swappers/ThorchainSwapper/ThorchainSwapper.ts → utils.ts:181) delegates to callbacks.signAndBroadcastTransaction (wallet). CowSwap alt: cowSwapper.executeEvmMessage → signCowOrder + cowService.post.\nKEY SYMBOLS: Swapper/SwapperApi types, swappers registry (constants.ts), getTradeQuotes (swapper.ts), TradeExecution._execWalletAgnostic, swapper.getUnsignedEvmTransaction/executeEvmTransaction, one concrete swapper (thorchainSwapper/zrxSwapper/cowSwapper).\nDYNAMIC BOUNDARIES: swappers[name] registry lookup (2 sites); all hops into a concrete swapper are via the Swapper/SwapperApi interface, never a direct function. A correct answer must name the interface + registry and reach a concrete swapper through interface dispatch." + }, + "trezor": { + "question": "How does sending a crypto transaction flow from the send form's review/sign action through to signing it via @trezor/connect (TrezorConnect.signTransaction)? Trace the call path.", + "truth": "Verified call path (trezor-suite monorepo; app in packages/suite, shared logic in suite-common/wallet-core, device API in packages/connect):\n1. ReviewButton.handleButtonReviewClick — packages/suite/src/views/wallet/send/TotalSent/ReviewButton.tsx:120 → signTransaction() (= useSendForm's sign)\n2. useSendForm.sign (exported as signTransaction) — packages/suite/src/hooks/wallet/useSendForm.ts:278 → dispatch(signAndPushSendFormTransactionThunk({formState, precomposedTransaction, selectedAccount})) [DYNAMIC: redux thunk]\n3. signAndPushSendFormTransactionThunk — packages/suite/src/actions/wallet/send/sendFormThunks.ts:237 → (first enhancePrecomposedTransactionThunk) then dispatch(signTransactionThunk({...})) [cross-package: thunk from suite-common/wallet-core]\n4. signTransactionThunk (coin-routing hub) — suite-common/wallet-core/src/send/sendFormThunks.ts:532/573 → networkType branch → dispatch(signBitcoinSendFormTransactionThunk) (ethereum→signEthereumSendFormTransactionThunk, etc.) [DYNAMIC: runtime coin dispatch]\n5. signBitcoinSendFormTransactionThunk — suite-common/wallet-core/src/send/sendFormBitcoinThunks.ts:394 → await TrezorConnect.signTransaction(signPayload)\n6. TrezorConnect.signTransaction (facade) — packages/connect-common/src/factory.ts → closure calls impl.call({method:'signTransaction',...}) [DYNAMIC: facade / iframe-or-module boundary]\n7. CoreInModule.call — packages/connect/src/impl/core-in-module.ts:171 → posts CORE_CALL to core (deferred promise)\n8. Core.onCall → getMethod — packages/connect/src/core/index.ts → getMethod resolves 'signTransaction' → new SignTransaction(message) [DYNAMIC: name→class]\n9. SignTransaction.run — packages/connect/src/api/signTransaction.ts:317 → signTx via device.getCommands().typedCall (protobuf to device). After signing, signAndPushSendFormTransactionThunk → pushSendFormTransactionThunk → TrezorConnect.pushTransaction.\nKEY SYMBOLS: ReviewButton → useSendForm.sign → signAndPushSendFormTransactionThunk (suite) → signTransactionThunk (wallet-core, coin hub) → signBitcoinSendFormTransactionThunk → TrezorConnect.signTransaction → connect factory/CoreInModule.call → SignTransaction.run.\nDYNAMIC BOUNDARIES: every suite→wallet-core hop is a redux thunk dispatch; signTransactionThunk branches by networkType at runtime; TrezorConnect is a dynamically-built facade crossing an iframe/module boundary; the SignTransaction class is resolved by name. A correct answer must cross suite→wallet-core→connect and reach TrezorConnect.signTransaction / SignTransaction.run, not stop at the UI." + } +} diff --git a/scripts/agent-eval/offload-eval-hook.mjs b/scripts/agent-eval/offload-eval-hook.mjs new file mode 100644 index 0000000..b2f23d9 --- /dev/null +++ b/scripts/agent-eval/offload-eval-hook.mjs @@ -0,0 +1,84 @@ +#!/usr/bin/env node +// UserPromptSubmit hook — APPROACH 1: additive context-injection. +// Front-loads codegraph's structural answer for flow/impact/"how/where" prompts so the +// agent's reflex grep/read has nothing left to find. Strictly additive (never blocks), +// gated to structural prompts (no cost otherwise), and uses RAW explore (offload disabled) +// so the injected context is accurate — never the (currently low-fidelity) synthesis. +// +// Reads {prompt, cwd} as JSON on stdin; prints the explore result to stdout (which Claude +// Code injects into the agent's context). Any failure -> silent exit 0 (degradable). +import { pathToFileURL, fileURLToPath } from 'node:url'; +import { resolve, join, dirname } from 'node:path'; +import { existsSync, readFileSync, appendFileSync } from 'node:fs'; + +// Resolve the engine repo from this script's own location (scripts/agent-eval/ -> ../..), +// overridable with CG_ENGINE. The hook ships inside the repo, so it finds its own dist. +const HERE = dirname(fileURLToPath(import.meta.url)); +const ENGINE = process.env.CG_ENGINE || resolve(HERE, '..', '..'); +const BUDGET = Number(process.env.CG_FRONTLOAD_BUDGET || 16000); + +// Debug log only when CG_FRONTLOAD_DEBUG is set to a file path (the harness points it at a +// log to count injections); off by default so the shipped hook writes nothing extra. +const DBG = process.env.CG_FRONTLOAD_DEBUG; +const dbg = (m) => { if (!DBG) return; try { appendFileSync(DBG, `[${new Date().toISOString()}] ${m}\n`); } catch { /* ignore */ } }; + +let input = {}; +try { input = JSON.parse(readFileSync(0, 'utf8')); } catch (e) { dbg('stdin parse fail: ' + e.message); } +const prompt = String(input.prompt || ''); +const cwd = String(input.cwd || process.cwd()); +dbg(`invoked: promptLen=${prompt.length} cwd=${cwd}`); + +// Gate: only structural / flow / impact / where-how questions. Cheap regex; silent no-op +// otherwise so non-structural prompts ("fix this typo") cost nothing. +const STRUCTURAL = /\b(how|where|trace|flow|path|reach(es|ed)?|call(s|ed|er|ers|ee)?|depend|impact|affect|wire[ds]?|connect|implement|architect|structure|breaks?|what calls|why does)\b/i; +if (!prompt || !STRUCTURAL.test(prompt)) { dbg('gate: non-structural, no-op'); process.exit(0); } +dbg('gate: structural PASS'); + +// Find the index: cwd, then walk up a few levels. +let root = cwd, found = null; +for (let i = 0; i < 6 && root; i++) { + if (existsSync(join(root, '.codegraph'))) { found = root; break; } + const parent = resolve(root, '..'); if (parent === root) break; root = parent; +} +if (!found) { dbg(`no .codegraph found from cwd=${cwd}`); process.exit(0); } +dbg(`found index at ${found}`); + +try { + process.env.CODEGRAPH_OFFLOAD_DISABLE = '1'; // raw, accurate — never the unfixed offload + process.env.CODEGRAPH_TELEMETRY = '0'; process.env.DO_NOT_TRACK = '1'; + const load = async (rel) => import(pathToFileURL(resolve(ENGINE, rel)).href); + const idx = await load('dist/index.js'); + const tools = await load('dist/mcp/tools.js'); + const CodeGraph = idx.default?.default ?? idx.default ?? idx.CodeGraph; + const ToolHandler = tools.ToolHandler ?? tools.default?.ToolHandler; + if (typeof CodeGraph?.openSync !== 'function' || typeof ToolHandler !== 'function') process.exit(0); + + // Retry once on a transient busy/locked index (the hook's openSync can race a + // freshly-warming daemon on the first prompt of a session). + let text = ''; + for (let attempt = 1; attempt <= 2; attempt++) { + try { + const cg = CodeGraph.openSync(found); + const h = new ToolHandler(cg); + const res = await h.execute('codegraph_explore', { query: prompt }); + text = res?.content?.[0]?.text ?? ''; + try { cg.close?.(); } catch { /* ignore */ } + dbg(`explore attempt ${attempt} returned ${text.length} chars`); + break; + } catch (e) { + dbg(`explore attempt ${attempt} failed: ${e?.message || e}`); + if (attempt === 2) throw e; + await new Promise((r) => setTimeout(r, 800)); + } + } + if (!text.trim()) { dbg('empty explore result, no-op'); process.exit(0); } + if (text.length > BUDGET) text = text.slice(0, BUDGET) + '\n…[front-load truncated to budget]'; + + process.stdout.write( + `## CodeGraph structural context (auto-retrieved for this question)\n` + + `The code graph was queried for your question; the relevant symbols, source, and call flow are below. ` + + `Treat the quoted source as already read. If you need more, call codegraph_explore with specific symbol names rather than grepping or reading files.\n\n` + + text + '\n' + ); + dbg(`INJECTED ${text.length} chars`); +} catch (e) { dbg('ERROR: ' + (e?.stack || e?.message || e)); process.exit(0); } // degradable diff --git a/scripts/agent-eval/offload-eval-judge.mjs b/scripts/agent-eval/offload-eval-judge.mjs new file mode 100644 index 0000000..32a15dd --- /dev/null +++ b/scripts/agent-eval/offload-eval-judge.mjs @@ -0,0 +1,103 @@ +#!/usr/bin/env node +// Accuracy judge. For each run in results.jsonl: +// - end-to-end: agent finalAnswer vs verified ground truth (all arms) +// - fidelity: offload synthesized answer vs ground truth (offload arm only) +// Judge = claude -p sonnet --effort high, no tools, run from a neutral cwd, +// JSON-only verdicts. Writes judged.jsonl (one line per run, verdicts merged). +// +// Usage: judge.mjs --results --truth --out [--concurrency 4] +import { readFileSync, writeFileSync, existsSync } from 'fs'; +import { execFile } from 'child_process'; + +const A = {}; +for (let i = 2; i < process.argv.length; i += 2) A[process.argv[i].replace(/^--/, '')] = process.argv[i + 1]; +const results = readFileSync(A.results, 'utf8').split('\n').filter(Boolean).map(l => JSON.parse(l)); +const truth = JSON.parse(readFileSync(A.truth, 'utf8')); +const OUT = A.out || '/tmp/cg-offload-eval/judged.jsonl'; +const CONC = Number(A.concurrency || 4); + +function askJudge(prompt) { + return new Promise((resolve) => { + execFile('claude', ['-p', prompt, '--model', 'sonnet', '--effort', 'high', + '--max-budget-usd', '0.5', '--strict-mcp-config', '--mcp-config', '{"mcpServers":{}}'], + // Run from a neutral dir with no repo files so the judge can't "cheat" by reading source. + { cwd: process.env.AGENT_EVAL_OUT || '/tmp', maxBuffer: 1 << 24, timeout: 120000 }, + (err, stdout) => { + const raw = (stdout || '').trim(); + const m = raw.match(/\{[\s\S]*\}/); + if (!m) return resolve({ verdict: 'error', score: null, note: (err ? 'exec ' + err.message : 'no json').slice(0, 80) }); + try { resolve(JSON.parse(m[0])); } catch { resolve({ verdict: 'error', score: null, note: 'parse fail' }); } + }); + }); +} + +const e2ePrompt = (gt, ans) => `You are scoring whether an AI coding agent correctly answered a code-flow question about a repository. Judge ONLY against the verified ground truth. Do NOT use any tools. + +QUESTION: ${gt.question} + +VERIFIED GROUND TRUTH (the actual call path + files): +${gt.truth} + +AGENT'S ANSWER: +${ans || '(empty)'} + +Score how correct the agent's answer is vs the ground truth. A "pass" means it identifies the core mechanism and the major hops with the right files/symbols and makes no materially wrong claim. "partial" = right area but misses major hops or has notable errors. "fail" = wrong layer, fabricated, or misses the mechanism. +Output ONLY minified JSON, no prose, no code fences: +{"verdict":"pass|partial|fail","score":<0-100>,"missedHops":["..."],"wrongClaims":["..."],"note":"<=20 words"}`; + +const fidPrompt = (gt, ans) => `You are scoring the FIDELITY of a machine-synthesized code-exploration answer against verified ground truth. The synthesized answer claims to trace a flow and cite file:line locations. Do NOT use any tools. + +QUESTION: ${gt.question} + +VERIFIED GROUND TRUTH (the actual call path + files): +${gt.truth} + +SYNTHESIZED ANSWER (to score): +${ans || '(empty)'} + +Judge: (1) is the traced call path correct vs ground truth? (2) are the cited files/symbols correct (not fabricated)? (3) if it gave a "Coverage:" verdict, was that verdict honest about what it actually covered? A confident WRONG trace is the worst outcome — penalize it harder than an honest "partial/not found". +Output ONLY minified JSON, no prose, no code fences: +{"verdict":"pass|partial|fail","score":<0-100>,"fabrication":,"coverageHonest":,"missedHops":["..."],"note":"<=20 words"}`; + +// Build the job list +const jobs = []; +for (const r of results) { + const gt = truth[r.repo]; + if (!gt) { r._nojudge = true; continue; } + jobs.push({ r, kind: 'e2e', prompt: e2ePrompt(gt, r.finalAnswer) }); + if (r.arm === 'offload' && Array.isArray(r.offloadAnswers)) + r.offloadAnswers.forEach((ans, i) => { if (ans && ans.trim()) jobs.push({ r, kind: 'fid', idx: i, prompt: fidPrompt(gt, ans) }); }); +} +console.error(`judging ${jobs.length} verdicts across ${results.length} runs (concurrency ${CONC})...`); + +let done = 0; +async function worker(queue) { + while (queue.length) { + const job = queue.shift(); + const v = await askJudge(job.prompt); + if (job.kind === 'e2e') job.r.e2e = v; else (job.r._fid ??= []).push(v); + console.error(` [${++done}/${jobs.length}] ${job.r.repo}/${job.r.arm}#${job.r.rep} ${job.kind}: ${v.verdict}${v.score != null ? ' ' + v.score : ''}`); + } +} +const q = [...jobs]; +await Promise.all(Array.from({ length: CONC }, () => worker(q))); + +// Aggregate per-answer fidelity verdicts into one fidelity object per offload run. +const medOf = (a) => { a = [...a].sort((x, y) => x - y); return a.length ? (a.length % 2 ? a[(a.length - 1) / 2] : (a[a.length / 2 - 1] + a[a.length / 2]) / 2) : null; }; +for (const r of results) { + if (r._fid?.length) { + const scores = r._fid.map(v => v.score).filter(x => x != null); + r.fidelity = { + n: r._fid.length, scores, + max: scores.length ? Math.max(...scores) : null, + min: scores.length ? Math.min(...scores) : null, + median: medOf(scores), + anyFabrication: r._fid.some(v => v.fabrication === true), + allCoverageHonest: r._fid.every(v => v.coverageHonest !== false), + verdicts: r._fid.map(v => v.verdict), + }; + } + delete r._fid; +} +writeFileSync(OUT, results.map(r => JSON.stringify(r)).join('\n') + '\n'); +console.error(`wrote ${OUT}`); diff --git a/scripts/agent-eval/offload-eval-matrix.sh b/scripts/agent-eval/offload-eval-matrix.sh new file mode 100755 index 0000000..6361305 --- /dev/null +++ b/scripts/agent-eval/offload-eval-matrix.sh @@ -0,0 +1,20 @@ +#!/usr/bin/env bash +# Drive the 3-arm campaign (offload/raw/nocg) across all 4 tiers, n reps each, into one +# results.jsonl. Reads the canonical question per repo from offload-eval-ground-truth.json. +# Env: REPS (default 3) AGENT_EVAL_OUT= +set -uo pipefail +HERE="$(cd "$(dirname "$0")" && pwd)" +OUT="${AGENT_EVAL_OUT:-/tmp/cg-offload-eval}" +GT="$HERE/offload-eval-ground-truth.json" +REPS="${REPS:-3}" +export RESULTS="$OUT/results.jsonl" +: > "$RESULTS" +for repo in mtkruto postybirb shapeshift trezor; do + case "$repo" in mtkruto) tier=small;; postybirb) tier=medium;; shapeshift) tier=complex;; trezor) tier=large;; esac + Q=$(node -e "console.log(JSON.parse(require('fs').readFileSync(process.argv[1],'utf8'))[process.argv[2]].question)" "$GT" "$repo") + echo ""; echo "### $repo ($tier) $(date +%H:%M:%S)" + bash "$HERE/offload-eval-3arm.sh" "$OUT/repos/$repo" "$tier" "$REPS" "$Q" +done +echo ""; echo "###### MATRIX DONE -> $RESULTS ($(wc -l < "$RESULTS") runs). Judge + summarize with:" +echo " node $HERE/offload-eval-judge.mjs --results $RESULTS --truth $GT --out $OUT/judged.jsonl" +echo " node $HERE/offload-eval-summarize.mjs $OUT/judged.jsonl" diff --git a/scripts/agent-eval/offload-eval-metrics.mjs b/scripts/agent-eval/offload-eval-metrics.mjs new file mode 100644 index 0000000..97cb35f --- /dev/null +++ b/scripts/agent-eval/offload-eval-metrics.mjs @@ -0,0 +1,94 @@ +#!/usr/bin/env node +// Extract one eval run's metrics from its Claude stream-json transcript + the +// offload usage sidecar log, emit ONE merged JSON line. +// +// Usage: extract-metrics.mjs --run --usage \ +// --arm --rep --repo --tier --q +import { readFileSync, existsSync } from 'fs'; + +const args = {}; +for (let i = 2; i < process.argv.length; i += 2) args[process.argv[i].replace(/^--/, '')] = process.argv[i + 1]; + +const runFile = args.run; +const lines = existsSync(runFile) ? readFileSync(runFile, 'utf8').split('\n').filter(Boolean) : []; + +const toolCounts = {}; +let result = null; +const tok = { gen: 0, fresh: 0, cached: 0 }; +const offloadAnswers = []; +let exploreResults = 0; // tool_results from explore (offload or raw) +let lastAssistantText = ''; + +for (const line of lines) { + let ev; try { ev = JSON.parse(line); } catch { continue; } + + // per-turn token usage (authoritative token measure; result.usage is last-turn only) + const u = ev.message?.usage; + if (u) { + tok.gen += u.output_tokens || 0; + tok.fresh += (u.input_tokens || 0) + (u.cache_creation_input_tokens || 0); + tok.cached += u.cache_read_input_tokens || 0; + } + + if (ev.type === 'assistant' && Array.isArray(ev.message?.content)) { + for (const b of ev.message.content) { + if (b.type === 'tool_use') toolCounts[b.name] = (toolCounts[b.name] || 0) + 1; + if (b.type === 'text' && b.text?.trim()) lastAssistantText = b.text.trim(); + } + } + // tool_results arrive in user messages + if (ev.type === 'user' && Array.isArray(ev.message?.content)) { + for (const b of ev.message.content) { + if (b.type !== 'tool_result') continue; + const text = Array.isArray(b.content) + ? b.content.map(c => (typeof c === 'string' ? c : c.text || '')).join('') + : (typeof b.content === 'string' ? b.content : ''); + // An offload answer is either the 'plain'/'report' synthesis (carries the + // "Synthesized by CodeGraph" footer) or a 'refs' answer (carries the re-expanded + // "### Referenced source — verbatim" appendix). A refs call that cited nothing + // valid falls back to RAW source, which is correctly counted as a raw explore below. + if (/Synthesized by CodeGraph|### Referenced source — verbatim/.test(text)) { offloadAnswers.push(text); exploreResults++; } + else if (/Found \d+ symbols? across|## Exploration:/.test(text)) exploreResults++; + } + } + if (ev.type === 'result') result = ev; +} + +// offload usage sidecar (CodeGraph AI tokens + cost) — one JSON line per offload call +const ai = { calls: 0, promptTokens: 0, completionTokens: 0, totalTokens: 0, credits: 0, costUsd: 0, ms: 0 }; +if (args.usage && args.usage !== '-' && existsSync(args.usage)) { + for (const line of readFileSync(args.usage, 'utf8').split('\n').filter(Boolean)) { + let e; try { e = JSON.parse(line); } catch { continue; } + ai.calls++; + ai.promptTokens += e.promptTokens || 0; + ai.completionTokens += e.completionTokens || 0; + ai.totalTokens += e.totalTokens || 0; + ai.credits += e.creditsCharged || 0; + ai.costUsd += e.costUsd || 0; + ai.ms += e.ms || 0; + } +} + +// front-load hook fired iff its injected header appears in the transcript +const frontload = lines.some(l => l.includes('auto-retrieved for this question')); +const get = (n) => toolCounts[n] || 0; +const read = get('Read'); +const grep = get('Grep') + get('Bash') + get('Glob'); +const explore = get('mcp__codegraph__codegraph_explore'); +const cgAny = Object.keys(toolCounts).filter(k => /mcp__codegraph__/.test(k)).reduce((s, k) => s + toolCounts[k], 0); + +const out = { + repo: args.repo, tier: args.tier, arm: args.arm, rep: Number(args.rep), question: args.q, + ok: result?.subtype === 'success', + durationSec: result ? +(result.duration_ms / 1000).toFixed(1) : null, + numTurns: result?.num_turns ?? null, + costUsdMain: result ? +(result.total_cost_usd || 0).toFixed(4) : null, + tokGen: tok.gen, tokFresh: tok.fresh, tokCached: tok.cached, tokBillable: tok.gen + tok.fresh, + read, grep, explore, cgAny, frontload, + offloadFired: offloadAnswers.length, + ai, + // text payloads for the accuracy judge (kept separate; large) + finalAnswer: (result?.result || lastAssistantText || '').slice(0, 8000), + offloadAnswers: offloadAnswers.map(a => a.slice(0, 6000)), +}; +process.stdout.write(JSON.stringify(out) + '\n'); diff --git a/scripts/agent-eval/offload-eval-refs1.sh b/scripts/agent-eval/offload-eval-refs1.sh new file mode 100755 index 0000000..aac6fb1 --- /dev/null +++ b/scripts/agent-eval/offload-eval-refs1.sh @@ -0,0 +1,50 @@ +#!/usr/bin/env bash +# ONE offload run on ONE indexed repo at a given offload STYLE (plain|refs), so we can +# watch a single agent transcript at a time (the user's one-run-at-a-time methodology). +# The OFFLOAD reasoning runs in the prewarmed DAEMON process, so the style env must be +# set on BOTH the daemon and the client MCP config. Writes one metrics line to RESULTS +# and leaves the raw stream-json at $RUNS/-