feat(impact): cross-language blast-radius coverage (22 languages + 14 frameworks) (#708)
Completes the cross-file dependency graph behind impact / affected / explore across all 22 supported languages and 14 web frameworks, validated on real-world repos (measured fair-coverage table added to the README). Per-language resolution + framework resolvers/synthesizers (Lua/Luau require, Shopify OS 2.0 Liquid sections, Delphi forms, Rust cross-module + Rocket macros, Swift Fluent, SvelteKit/Nuxt loader/component conventions, RN/Expo bridges). 0 cross-family false edges, full suite green (1187 passed). See #708. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
bfa84d32b8
commit
07af3db6c7
@@ -1,114 +0,0 @@
|
||||
---
|
||||
name: codegraph-tool-surface-rethink-2026-05-27
|
||||
date: 2026-05-27 15:11
|
||||
project: codegraph
|
||||
branch: feat/go-multi-module-trace-quality
|
||||
summary: PR #494 multi-language audit revealed structural ~$0.04-$0.08 tiny-repo cost overhead from MCP tool-defs; user pivoted to questioning whether codegraph_context / 5+ tools are even necessary — suggested `explore` + `trace` only.
|
||||
---
|
||||
|
||||
# Handoff: Should codegraph cut to just `explore` + `trace`?
|
||||
|
||||
## Resume here — read this first
|
||||
**Current state:** PR #494 (`feat/go-multi-module-trace-quality`, 13 commits, all 1076 tests pass) ships every safe optimization for the cosmos/etcd Go work AND the cross-language extensions (generated-detection, IFACE_OVERRIDE_LANGS, sibling-inlining, path-proximity, tool gating at <150 files to 5 core tools). Empirically PROVED that cutting below 5 tools regresses every tiny repo (3-tool gate: cobra 17→48% loss; 1-tool gate: express -43% WIN flipped to +107% LOSS). User just asked the right question: **"Why do we need codegraph_context, or any of these massive amounts of tools? All it really needs is explore, and trace if you ask me."**
|
||||
|
||||
**Immediate next step:** Open the next session by treating the user's question as a design pivot, not a continuation of the cost-gap whack-a-mole. The right reply is a focused honest analysis: what does each of the 10 tools actually do that explore + trace alone can't, where does codegraph_context's value-add hold up (or not), and what would removing context/search/node from the default surface ACTUALLY cost in measured loss-of-flow-coverage. Don't start cutting tools yet — present the analysis first.
|
||||
|
||||
> Suggested next message: "Walk me through what each codegraph_* tool actually does on a real flow question that explore + trace alone can't, and which ones agents are picking in our recent audits. If context/search/node aren't earning their seat, propose cutting them and measure on cosmos-Q1 + etcd-Q1 + prometheus + cobra n=2 each."
|
||||
|
||||
## Goal
|
||||
Decide whether codegraph's 10-tool MCP surface should be cut down to ~2 core tools (explore + trace) as the user proposed. The empirical iteration in this session showed that the 5 omitted "auxiliary" tools (callers, callees, impact, status, files) only add cost on tiny repos and aren't earning their seat. The real question now: **does the same logic apply to context + search + node?** If yes, codegraph becomes 2 tools + a smaller MCP surface = lower fixed prompt overhead = closes the tiny-repo cost gap structurally instead of patching it. If no, name the specific flows where they do unique work.
|
||||
|
||||
## Key findings (this session)
|
||||
|
||||
- **PR #494 status**: 13 commits, all 1076 tests pass, https://github.com/colbymchenry/codegraph/pull/494. Already pushed:
|
||||
- Generated-file detection: `src/extraction/generated-detection.ts` (multi-language patterns, applied in `findSymbol`/`findAllSymbols`/`handleSearch`/`handleExplore` file ranking/`context/formatter.ts`)
|
||||
- Go gRPC bridge: `goGrpcStubImplEdges` in `src/resolution/callback-synthesizer.ts:341` (467 bridge edges on cosmos-sdk)
|
||||
- Trace failure inlining + path-proximity pairing + less-canonical-path penalty + sibling-from-TO-file inlining: all in `src/mcp/tools.ts` `handleTrace`
|
||||
- `IFACE_OVERRIDE_LANGS` extended from `{java,kotlin}` to `{java,kotlin,csharp,typescript,javascript,swift,scala}`; loop iterates `class` AND `struct` kinds
|
||||
- Tool-def trims (~7KB → 5KB) in `src/mcp/tools.ts`
|
||||
- Tiny-repo tool gating: `ToolHandler.getTools()` filters to 5 core tools when `fileCount < 150`
|
||||
- Tiny-tier explore budget in `getExploreOutputBudget(fileCount < 150)`: 13K total / 4 files / `includeRelationships: true`
|
||||
- `handleContext` default `maxNodes` drops from 20 → 8 when `fileCount < 150`
|
||||
- **Cosmos Q1 flipped**: WIN ($0.257 vs $0.449, n=1; n=2 avg $0.341 vs $0.350 tied). The breakthrough was `inlineEndpoint`'s "Other functions in TO's file" siblings — `msgServer.Send`'s real callee `k.Keeper.SendCoins` is an embedded-interface call tree-sitter can't statically resolve, so static `getCallees` returns only utility funcs; the *actual* flow lives in `x/bank/keeper/send.go`'s file-mates. See `handleTrace` line ~1430.
|
||||
- **Empirical lower bounds on tool gating** (n=2-3 audits):
|
||||
- 5 tools (search+context+node+explore+trace) = current setting, works
|
||||
- 3 tools (search+context+trace) = cobra 17→48% loss, sinatra 18→96% loss; agent falls back to Reads when node/explore unavailable
|
||||
- 1 tool (search only) = catastrophic, express -43% WIN → +107% LOSS
|
||||
- **n=3 measurements confirm structural floor:** cobra WITH consistently $0.28 (variance <5%), WITHOUT consistently $0.24. The $0.04 gap is structural, not noise.
|
||||
- **The user's pivot question challenges this:** their hypothesis is that context+search+node may also be earning less than they cost. The audits we have can't directly answer that — every test had all 10 (or 5) tools available. To test, expose ONLY explore+trace on a controlled batch and re-measure.
|
||||
- **Cross-language status (single-run each):** WINS = Go (multi-mod), Rust, Java, C#, Kotlin, Swift, Svelte, prometheus, ky (post-gating), express (JS). TIES = cobra (n=2 tied $0.27/$0.27), excalidraw, django, redis, json, Masonry, flutter, vapor, spring. LOSSES = sinatra, slim, flask, scala-play, Fusion, vue-core (variance), Drupal, NestJS, FastAPI, Laravel, ASP.NET, axum, actix, Rocket, gorilla/mux, SvelteKit, Charts bridge (slight), RN segmented-control (slight).
|
||||
- **Loss pattern is structural, not language-specific.** All losses are tiny example/starter repos where the without-arm grep+read path costs ~$0.20-0.30 and codegraph's MCP overhead can't be amortized.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **PR-494 is a Go-multi-module PR by title but the body is now cross-cutting** — generated-detection, IFACE_OVERRIDE_LANGS, tool gating, all language-agnostic. Don't let the title narrow what's in it.
|
||||
- **The variance on the WITHOUT arm is enormous** — same-repo single-run cost can swing $0.04 to $0.80 depending on whether the agent goes grep-heavy or read-heavy that turn. **Never conclude WIN/LOSS from n=1.** The session has many single-run results that need confirming.
|
||||
- **Cobra (~50 files) is the canary** — every aggressive cut that helps ky or sinatra has regressed cobra at least once. It's the most-tested tiny repo because of that.
|
||||
- **Don't try the 1-tool or 3-tool gate again** — both are explicitly documented as regressions in `getTools()` comments (`src/mcp/tools.ts` around line 660). Cutting below 5 forces the agent to Read.
|
||||
- **Kong's first audit was a 0-byte index** — parallel `audit.sh` runs against the same .codegraph dir can corrupt each other. If kong/any-repo's audit shows wildly wrong numbers, check `stat /tmp/codegraph-corpus/<repo>/.codegraph/codegraph.db` before iterating on the result.
|
||||
- **48-parallel audit launches FAIL silently** — system resource limits. Stay at 6-8 parallel max. Use `wait` between waves.
|
||||
- **The MCP daemon caches the tool list** at process start — when iterating on `getTools()` you MUST `pkill -f "codegraph.js serve --mcp"` between rebuilds or you'll be testing stale code.
|
||||
- **`maxCharsPerFile` monotonic invariant** is pinned by `__tests__/explore-output-budget.test.ts` (the spec is `a larger tier must NEVER get a smaller maxCharsPerFile than a smaller tier`). Honor it.
|
||||
|
||||
## How to test & validate
|
||||
|
||||
- `npm test` → "Tests 1076 passed | 2 skipped". Must stay green.
|
||||
- `npm run build 2>&1 | tail -3` → check dist rebuilt cleanly.
|
||||
- `pkill -f "codegraph.js serve --mcp" ; sleep 2` → ALWAYS run before agent-eval after a build, otherwise the daemon serves stale code.
|
||||
- Single-question audit: `AGENT_EVAL_OUT=/tmp/cg-NAME /Users/colby/Development/Personal/codegraph/scripts/agent-eval/run-all.sh <repo-path> "<question>" headless`. Outputs `run-headless-with.jsonl` and `run-headless-without.jsonl`.
|
||||
- Parse: `node scripts/agent-eval/parse-run.mjs /tmp/cg-NAME/run-headless-{with,without}.jsonl` → cost, duration, turns, tool sequence.
|
||||
- **For real conclusions, always n=2 minimum.** n=3 is the right bar to separate variance from signal — last session's data on cobra showed WITH had <5% variance but WITHOUT swung 95%.
|
||||
- **The explore + trace experiment** the user wants: modify `getTools()` to filter visible tools to `new Set(['codegraph_explore', 'codegraph_trace'])` for ALL repos (or just the tiny tier first), re-run cosmos-Q1, etcd-Q1, prometheus, cobra n=2 each, and compare.
|
||||
|
||||
## Repo state
|
||||
|
||||
- branch `feat/go-multi-module-trace-quality`, last commit `ae5364c docs(mcp): pin empirical lower bound on tool gating after n=2 micro test`
|
||||
- uncommitted: clean
|
||||
- PR: https://github.com/colbymchenry/codegraph/pull/494 (13 commits, ready for review unless we land the tool-surface redesign)
|
||||
|
||||
## Open threads / TODO
|
||||
|
||||
- [ ] **The user's pivot**: prove or disprove that explore + trace alone is sufficient. Set up a 4-repo × n=2 batch (cosmos-Q1, etcd-Q1, prometheus, cobra) with ONLY explore+trace exposed, compare to current 5-tool / 10-tool baselines.
|
||||
- [ ] If explore+trace alone wins → cut the tool surface across the board. **This is a breaking API change** — callers/callees/impact/status/files/node would disappear from default exposure. Need a clean way to retain them for users who script against the MCP directly (env var? `--full-tools` flag?).
|
||||
- [ ] If explore+trace alone loses → identify which of context/search/node is doing the structural work, and propose cutting only the others.
|
||||
- [ ] **README update either way**: the current "~35% cheaper" claim averages 7 medium/large repos. Either commit to that scope ("real codebases (~200+ files)") or re-measure after the tool surface change.
|
||||
- [ ] Liquid, Pascal/Delphi, React Router, TurboModules, Expo Modules, Paper view managers — still untested categories from the README. Bridges Swift↔ObjC/RN-legacy/RN-events/Fabric were tested in wave 3 — 1 win, 2 tied, 1 slight loss. The rest are still gaps.
|
||||
- [ ] If we ship the PR as-is, write a CHANGELOG entry under `[Unreleased]` summarizing the 13 commits — currently the CHANGELOG entry covers commits 1-2 (generated-detection + gRPC bridge + trace UX); commits 3-13 need their own bullets.
|
||||
|
||||
## Recent transcript (last ~10 turns, oldest → newest)
|
||||
|
||||
### Turn N-9 — User: "Have we tested every language/framework per the playbook?"
|
||||
- **Claude:** Audited coverage honestly — ~70% of README's claims (22 langs + 14 frameworks + 8 bridges) had NOT been validated. Offered three scopes (smoke / medium / full playbook).
|
||||
- **Outcome:** User chose "Full playbook"; later refined to "Smoke-test only the completely-untested categories."
|
||||
|
||||
### Turn N-8 — Wave 1 smoke-test (8 parallel: JS/Dart/Drupal/NestJS/FastAPI/Vapor/Spring/Rails)
|
||||
- **Claude:** Launched 8 audits via `run-all.sh`. Results: express WIN (-43%), 3 tied, 4 LOSS (Drupal, NestJS, FastAPI, Rails). NestJS 113% loss, FastAPI 194% loss were the standouts.
|
||||
- **Outcome:** Documented the loss pattern correlates with repo size, not language.
|
||||
|
||||
### Turn N-7 — Wave 2 (7 frameworks: Laravel, ASP.NET, axum, actix, Rocket, gorilla/mux, SvelteKit) + Wave 3 (4 bridges: Charts, async-storage, RN-geolocation, RN-segmented-control)
|
||||
- **Claude:** All wave-2 frameworks lost on cost (small example/starter repos). All wave-3 bridges won/tied on cost AND won on reads.
|
||||
- **Outcome:** Confirmed loss pattern is tiny-repo-structural, not framework-specific. Bridges (codegraph's marquee feature) work.
|
||||
|
||||
### Turn N-6 — User: "We have a lot of losses. Work on those until they are all clear wins. You cannot stop or give up or push back."
|
||||
- **Claude:** Ran n=3 on the 8 worst losses to separate variance from signal. First attempt with 48 parallel processes silently failed (system limits). Re-ran in waves of 6.
|
||||
- **Outcome:** n=3 cobra: WITH consistent $0.28 (variance <5%), WITHOUT consistent $0.24. **Confirmed losses are structural, not variance.**
|
||||
|
||||
### Turn N-5 — Tested 3-tool gate (search + context + trace only)
|
||||
- **Claude:** Edited `getTools()` to expose only 3 tools on `<150` files. Re-ran cobra/ky/sinatra. Results: cobra 17→48% LOSS, ky 47→42% loss, sinatra 18→96% LOSS. Reverted.
|
||||
- **Outcome:** Cutting below 5 tools regresses because the agent falls back to extra Reads.
|
||||
|
||||
### Turn N-4 — Tested 1-tool gate (search only) on micro (<50 files)
|
||||
- **Claude:** Edited gate. cobra 126% LOSS, ky 44% loss, express FLIPPED from -43% WIN to +107% LOSS. Catastrophic. Reverted.
|
||||
- **Outcome:** Documented the empirical lower bound: 5 tools. Committed `ae5364c` with comments warning future tuners not to relitigate.
|
||||
|
||||
### Turn N-3 — Claude proposed three paths forward (auto-disable / README honesty / context-rewrite)
|
||||
- **Claude:** Used AskUserQuestion to offer auto-disable on <50, honest README, or last-ditch context shrink.
|
||||
- **Outcome:** User pivoted instead of picking any of those.
|
||||
|
||||
### Turn N-2 — User: "Why do we need codegraph_context, or any of these massive amounts of tools? All it really needs is explore, and trace if you ask me."
|
||||
- **Claude:** Recognized this is a design pivot, not a continuation of the cost-gap iteration. The right next move is to actually test the user's hypothesis, not to write another response defending the status quo.
|
||||
- **Outcome:** This handoff captures the pivot for a fresh session to answer properly.
|
||||
|
||||
### Turn N-1 — User: `/handoff save`
|
||||
- **Claude:** Wrote this file.
|
||||
- **Outcome:** Handoff persisted. Next session reads it and engages the explore+trace-only design question with measurement, not opinion.
|
||||
@@ -0,0 +1,316 @@
|
||||
---
|
||||
name: cross-language-impact-coverage-2026-06-04
|
||||
date: 2026-06-05 14:57
|
||||
project: codegraph
|
||||
branch: feat/cross-language-impact-coverage
|
||||
summary: Per-language coverage DONE (all 15 README langs + static-member pass + RN/Expo 95%+). Since then the live front is ROUTE FRAMEWORKS: import/aggregator style at ceiling (Express/FastAPI/Flask/requests/NestJS/Gin/Axum 95%+, Vapor/vapor-til 100%, SvelteKit 100% fair [component core + loader→page synthesizer], React Router 100% [config-based, no miss], Nuxt 93.5% fair [+ nested-component fix]; Axum 100% via a Rust self-relative submodule-call fix), convention/reflection + actor style at an honest sub-95% ceiling (ASP.NET 83.9%, Spring 83.3%, Drupal 78.9%, Django 74.1%, actix 65.4% [actor message dispatch] — all measured). **SWEEP COMPLETE — every README framework measured;** Rocket lifted 62.5%→**93.8%** via a new `routes![]`/`catchers![]` macro extractor (only the crate-root lib.rs remains). Six engine improvements en route (3 Rust: self-relative + multi-segment `a::b::c()` module calls + Rocket route-macro extraction; 1 Swift: Fluent `@Siblings(through: Pivot.self)` metatype refs; 1 SvelteKit: `+page.server.js` `load`→`+page.svelte` synthesizer; 1 Nuxt: nested auto-imported `<MediaCard>`→`media/Card.vue` resolution). The static-member-pass-for-TS/JS/Python lever was **measured & rejected** (e7b86df — 0 coverage gain, the import edge already covers it, + graph noise). **Then the 4 niche README languages were knocked out**: Lua 31.6%→**84.2%** + Luau 12.5%→**92.2%** (new require resolver, 4155609); Liquid 39.1%→**73.8%** (Shopify OS 2.0 JSON-template section parsing, 2f57119); Pascal 73%→**75.7%** (.dfm/.fmx form↔code-behind pairing, 2f30a3b) — so **ALL 22 README "Full support" languages + all 14 README frameworks now have coverage validation (full README parity).** Branch 41 commits ahead of main, 3 behind, not merged.
|
||||
---
|
||||
|
||||
# Handoff: Cross-language impact/blast-radius coverage campaign
|
||||
|
||||
## Resume here — read this first
|
||||
**Current state:** Branch `feat/cross-language-impact-coverage` — now **41 commits ahead of `main`, 3 behind** (behind = 3 README-waitlist doc commits on main), tip `2f30a3b`, all pushed to `origin`=colbymchenry/codegraph. NOT merged (review branch). **FULL README PARITY: all 22 "Full support" languages + all 14 frameworks now have coverage validation.** Two arcs are done: (1) **per-language** cross-file coverage for ALL 15 README "Full support" langs + the static-member pass; (2) **cross-language RN/Expo** at 95%+ FAIR (gate hole fixed 082353e; same-dir include + KMP import 529d822). **Since the 2026-06-04 save a 12-commit ROUTE-FRAMEWORK front landed** (`61a993a`→`a3f59fb`) — see the "Route-framework headroom map" below; that map is the live edge. Working tree: clean except one unrelated untracked file (`assets/generate-waitlist.py`, README-waitlist tooling, not this campaign). Full suite green (**1187 passed | 2 skipped**, 59 files).
|
||||
**Immediate next step:** **THE README SWEEP IS COMPLETE — all 22 "Full support" languages + all 14 frameworks have coverage validation.** Per-language DONE, RN/Expo DONE, route-framework + component-node front DONE, the 4 niche langs (Lua/Luau/Liquid/Pascal) DONE; **10 engine improvements** shipped this run (3 Rust + 1 Swift + 1 SvelteKit + 1 Nuxt + 1 Lua/Luau require resolver + 1 Liquid Shopify-JSON + 1 Delphi form pairing; the static-member-for-TS/JS/Python lever was measured & rejected). **Recommended: (A) ship it** — open the PR to `main` for the 41 commits; everything actionable is done. Optional follow-ups only: (B) build a convention/reflection lever if desired (Drupal `services.yml` DI, ASP.NET/Spring markup/reflection modeling) — each a large per-framework feature. **The convention/reflection frameworks (ASP.NET 83.9%, Spring 83.3%, Drupal 78.9%, Django 74.1%, actix 65.4%) sit at a genuine sub-95% static-analysis ceiling — don't chase 95% there without large reflection/markup modeling (metric-gaming otherwise). The TS/JS/Python static-member pass is measured-&-rejected (e7b86df), don't re-try.**
|
||||
|
||||
> Suggested next message: "Open the PR to main for the cross-language-impact-coverage branch (41 commits — full README parity: all 22 'Full support' languages + all 14 frameworks now have coverage validation)."
|
||||
|
||||
## DONE: coverage bump to 95%+ (commit 529d822) — RN/Expo multi-platform repos
|
||||
**Goal (from user):** bump async-storage (75.0%) and rn-device-info (72.4%) to 95%+. Two parts — real engine fixes + an honest fair-metric (the original 75/72 counted generated/build/config/entry files as if they were source).
|
||||
**Engine fixes (real coverage, generalizable):**
|
||||
1. **Same-dir C/C++ `#include`** — `#include "Foo.h"` had no directory awareness, so on a module with a same-named header per platform (windows/code/RNCAsyncStorage.h vs apple/) the includer landed on an arbitrary one (then the 082353e gate nulled the wrong-family match → real local header had 0 deps). Fixed C's quoted-include rule: resolve relative to the including file's OWN dir FIRST (`resolveViaImport` C/C++ branch in import-resolver.ts), plus a same-dir/proximity preference in `matchByFilePath`'s basename fallback (`pickClosestFileNode`).
|
||||
2. **KMP commonMain import** — an `expect` decl + its `actual`s share one FQN across source sets; `resolveJvmImport` took `candidates[0]`, so one platform `actual` absorbed every common-side import and the `expect` looked unused. Now the same-FQN candidate CLOSEST to the importer (shared dir prefix, `expect` tiebreak) wins (`pickClosestJvmCandidate`). Both are the same "prefer the closest declaration on a name collision" principle as 082353e.
|
||||
**Honest fair metric** (`/tmp/faircov.cjs`, prints every exclusion): denominator = authored source that *can* have an in-repo dependent. Excludes (per methodology, all auditable): structural (generated `.g.h`/codegen, `pch.*`, `*.gradle*`, `CMakeLists`, eslint/jest/babel config), see-through barrels (0 real symbols — web re-export files + umbrella/SDK headers ONLY; a 0-symbol *source impl* is counted as a real frontier zero, never hidden), and entry points (package `src/index`, platform `web`/`windows` entries, RN `ReactPackageProvider`).
|
||||
**Before/after:**
|
||||
| Repo | FAIR coverage before→after | residual zero (frontier) |
|
||||
|---|---|---|
|
||||
| async-storage | 75.0% → **97.4% (37/38)** | DatabaseFiles.kt (KMP expect-decl side, no in-repo caller) |
|
||||
| rn-device-info | 72.4% → **95.2% (20/21)** | RNDeviceInfoCPP.cpp (`REACT_METHOD` macro methods not extracted) |
|
||||
No regression (same metric, before→after): okhttp 75.9%→76.4%, kotlinx.coroutines 89.7% (neutral), leveldb 78.0% (neutral), redis 89.7%→89.9%, fmt 77.3% (neutral); cross-family false edges still 0 everywhere. 2 regression tests in `extraction.test.ts` ("Same-directory include + KMP import resolution"), both fail without the fix. Full suite 1169.
|
||||
|
||||
## DONE: gate hole (commit 082353e) — cross-family references/imports
|
||||
**Symptom (was):** in `react-native-async-storage`, a TS `type TestRunner` and a Kotlin `class TestRunner` collided — TS `references`/`imports` resolved onto the Kotlin class (web→jvm false match). Plus `import React`↔Swift `React` and a C++ `#include "RNCAsyncStorage.h"`↔iOS ObjC header (basename collision).
|
||||
**Root cause:** the false edges came from the FRAMEWORK strategy — React's `resolveComponent` (frameworks/react.ts) name-matches `getNodesByName` with NO language check; its COMPONENT_KINDS includes `class`, so it returned the Kotlin `class` @0.8 (the TS `type_alias` filtered out), outranking the cross-lang-penalized (0.5) TS name-match. AND `imports` were never gated (only `references` was). NOTE: `this.frameworks` in resolveOne is NOT language-filtered per-ref (`getApplicableFrameworks` is unused there), so react.resolve runs for EVERY ref — its `languages` field is dead in that path.
|
||||
**Fix:** new `crossesKnownFamily(a,b)` (both in a known family jvm/apple/web/c AND different) wired into `gateFrameworkLanguage` (NEW — gates the framework strategy, refs+imports), `gateLanguage` (extended to also gate `imports`), and `applyLanguageGate` (name-match candidate filter — re-points instead of dropping). KEY RULE (non-obvious): the `references` gate stays STRICT (`!sameLanguageFamily`); `imports` + the framework gate use the WEAKER both-known rule, so config↔code bridges (yaml/blade side not a known family) and `.vue`/`.svelte`→`.ts` imports survive. `calls` bridges are never gated.
|
||||
**Before/after — precision fix (coverage HELD/up, false edges → 0):**
|
||||
| Repo | FAIR coverage before→after | cross-known-family false refs/imports |
|
||||
|---|---|---|
|
||||
| async-storage | 75.0% (39/52) → **75.0% (39/52)** | **22 → 0** |
|
||||
| rn-device-info (control) | 69.0% (20/29) → **72.4% (21/29)** | **5 → 0** |
|
||||
Coverage held on async-storage (no recall lost) and ROSE on rn-device-info (re-pointing gave a real same-family file a correct dependent). Legit JS↔native `calls` bridges intact (rn-device-info: 91 JS→Java, 37 JS→ObjC, full Java↔ObjC↔C++ pairing). 2 regression tests in `extraction.test.ts` ("Cross-language type/import gate"), both fail without the fix. Full suite 1167. Measure: `/tmp/faircov.cjs <repo>` (fair coverage + false-edge count) and `/tmp/xlang.cjs <repo>` (cross-lang edges by src→tgt × kind).
|
||||
|
||||
### Framework phase round 2 (commits d06a5ec, 74b599c, 2026-06-04)
|
||||
(1) RCT_EXPORT_METHOD EXTRACTION (d06a5ec): RN bridge resolver now implements `extract()` for .m/.mm (added 'objc' to languages), reuses parseObjcRNExports to emit a method node per RCT_EXPORT_METHOD/REMAP (id `rn-export:`, named the JS-visible name). The macro parsed as ERROR before → iOS methods invisible. rn-device-info JS→objc 7→37, java↔objc pairs 22→29. (2) RN EVENT WRAPPER (74b599c): RN_NATIVE_SENDEVENT_RE catches `sendEvent(ctx,"X",body)` wrappers (inner `.emit` uses a variable) → native java/swift events now connect to JS hooks. Synth tag is `rn-event-channel`. VALIDATED async-storage (pairing + JS→native work; found the precision bug above).
|
||||
|
||||
### Classic RN cross-platform pairing (commit 4a64ca5, 2026-06-04)
|
||||
`rnCrossPlatformEdges` (callback-synth): a native method (java/kotlin/objc/cpp) with a JS-side `calls` edge = confirmed bridge method → link to same-norm-name native method in another language (`getFreeDiskStorage:`→`getFreeDiskStorage`, first selector keyword), both directions. Skip RN_INFRA names (addListener/getConstants/getName/…). rn-device-info: 152 pairs (Java↔ObjC↔C++). FOLLOW-UP: RCT_EXPORT_METHOD isn't a node (macro/ERROR parse) → only regular `- (void)` ObjC methods pair today.
|
||||
|
||||
### Cross-language framework phase — round 1 (commit dbc4862, 2026-06-04)
|
||||
NEW direction: RN/Expo repos where JS↔native crosses LANGUAGE boundaries. Existing bridge support is RICH (legacy NativeModules, TurboModule, Expo Modules extractor `expo-module:`-prefixed nodes, Fabric, rnEvents, swift-objc) — don't rebuild; validate + extend. Classic RN bridge WORKS (rn-device-info: 118 JS→Java + JS→ObjC calls). THREE Expo gaps fixed: (1) generic `AsyncFunction<Float>("x")` — regex didn't allow `<…>` so all Android Expo methods dropped; (2) cross-platform pairing — `expoCrossPlatformEdges` links Swift↔Kotlin impls of the same JS method (JS resolves to one platform only); (3) cross-lang type-ref precision — gated `references` edges to same language-family (name-matcher.ts `applyLanguageGate`/`sameLanguageFamily` + index.ts `gateLanguage`), so native `BatteryManager.EXTRA_LEVEL` doesn't falsely match a TS `BatteryManager`; framework resolvers NOT gated (keep config↔code bridges). Measure: `/tmp/xlang.cjs`. Detail in memory.
|
||||
|
||||
### Objective-C result (commit 33ce431, 2026-06-04)
|
||||
WORST README language at baseline. FOUR fixes (3 in tree-sitter.ts, 1 in name-matcher.ts): (1) SINGLE-ARG SELECTOR — `[c storeImage:k]` was named `storeImage` (no colon) at the call site, never matching `storeImage:`; add `:` when the message has a `:` token. (2) CLASS-MESSAGE RECEIVER REF — `[Foo sharedCache]`/`[[Foo alloc] init]` now emits a `references` edge to the capitalized class (covers the header). (3) #IMPORT BASENAME — `#import "Foo.h"` resolves to the header via matchByFilePath relaxed to accept bare filenames w/ short ext. (4) CLASS-METHOD COLON — `Foo.storeImage:` now resolves (broadened matchMethodCall method regex to allow colon selectors). AFNetworking 50%→**90%**, SDWebImage Core 33.8%→**91.6%**. GOTCHA: SDWebImage `include/SDWebImage/*.h` are SYMLINKS to `Core/` — measure Core/ only. Residual = public-API category methods called by app code (frontier). Detail in memory.
|
||||
|
||||
### Dart result (commit 9487954, 2026-06-04)
|
||||
Dart was in TYPE_ANNOTATION_LANGUAGES but produced ZERO `references` edges, AND mixins were dropped. (NOTE: dio raw 67.8% was example-dir pollution — real 86.4%.) Two gaps, gated `language==='dart'`: (1) MIXINS — `with` mixins live in a `mixins` CHILD of `superclass`; generic path read namedChild(0) as base + dropped mixins (and `class C with M` misread mixins as superclass). Dart branch in extractInheritance: `extends` base + `implements` per mixin. (2) METHOD TYPE REFS — `method_signature` wraps the real `function_signature` (params/return there) + return is a bare `type_identifier` not a `type` field. Dart branch in extractTypeAnnotations: descend to inner signature → extractTypeRefsFromSubtree. flutter/packages 88.8%→**92.4%**, dio 86.4%→**87.9%**. Residual = export barrels + platform-conditional files + enum-value access (`Enum.value` — value-read frontier; a Dart `Capitalized.member`→ref pass would be precise, the top follow-up). Detail in memory `impact-coverage-findings.md`.
|
||||
|
||||
### Static-member / value-read pass (commit 857baf7, 2026-06-04)
|
||||
The deferred cross-language lever, now DONE. A type used only via a static member / enum VALUE (`MediaKind.video`, `Colors.red`, `JsonScope.NAME`, `Foo::BAR`) recorded no edge (body walker only did CALLS + `new`). `extractStaticMemberRef` (tree-sitter.ts, in visitFunctionBody) emits a `references` edge to the CAPITALIZED receiver of a member-access value read (per-lang node in MEMBER_ACCESS_TYPES: field_access Java / member_access_expression C# / navigation_expression Kotlin+Swift / field_expression Scala / class_constant_access_expression+scoped_property_access_expression PHP / qualified_identifier C++; Dart = identifier + sibling value-read selector). Skips call callees; gated to STATIC_MEMBER_LANGS={java,csharp,kotlin,swift,scala,dart,php,cpp} — TS/JS/Python EXCLUDED (high coverage + retrieval-perf-sensitive). flutter/packages 92.4%→93.2%; additive elsewhere; nodes stable. Detail in memory.
|
||||
|
||||
### C/C++ result (commit ec8fe3f, 2026-06-04)
|
||||
C/C++ were already HIGH (name-matching resolves cross-file calls across the .h/.c split). NOT an import gap. The systematic gap was a C++ EXTRACTION BUG in languages/c-cpp.ts: `extractCppQualifiedMethodName`/`extractCppReceiverType` BFS'd the whole declarator INCLUDING `parameter_list` + `trailing_return_type` for a `qualified_identifier` → a free function `std::string TableFileName(const std::string& dbname)` was named **`string`** (from the param type), `auto f() -> std::string` named `string` (trailing return). Calls never resolved; defining file looked dependent-less. Fix: shared `findDeclaratorQualifiedId` skips `parameter_list` + `trailing_return_type`; plain names fall back to default extraction. leveldb 91.7%→**94.8%**, fmt 32 mis-named→1, redis (C, unaffected) 92.2% at ceiling. Residual = generated tables, macro-reached, function-pointer dispatch (`MAKE_CMD(...,sortCommand,...)` — deferred, broad/risky), C++ namespaces (deferred). Detail in memory `impact-coverage-findings.md`.
|
||||
|
||||
### Ruby result (commits 44fb978 + 5bccab6, 2026-06-04)
|
||||
TWO gaps. (1) MIXINS (44fb978): `include`/`extend`/`prepend Mod` parsed as a bare `call` to method `include` → ZERO edges. Fix in languages/ruby.ts visitNode: detect bare include/extend/prepend (guard `!receiver` so `arr.include?(x)` is safe) → emit `implements` edge class/module→module. (2) REQUIRE RESOLUTION (5bccab6, bigger than expected): `require "lib/foo"` → emit `imports` ref `lib/foo.rb` (load-path, suffix-matched by matchByFilePath); `require_relative "../foo"` → resolve vs requiring file's dir (`path.posix.normalize`); bare `require "json"` skipped. Resolves to the FILE node. **sidekiq 71%→76.8% (mixins)→100% (requires); activerecord 84.8%→93% (mixins)→96.8% (requires)** — Rails autoloads but still has explicit requires for sub-components. Residual = `constantize` class-string instantiation (associations/arel), generators, version files. Detail in memory `impact-coverage-findings.md`.
|
||||
|
||||
### PHP result (commit acfb444, 2026-06-04)
|
||||
ROOT CAUSE: PHP ignored NAMESPACES — every class qn was the bare simple name, so laravel's 7+ same-named `Factory` interfaces across namespaces collapsed to one arbitrary match, and `use` imports never resolved. Fixes (gated `language==='php'`): (1) **namespace capture** — `packageTypes:['namespace_definition']`+`extractPackage` in languages/php.ts → classes scoped to `Foo\Bar::Class`; (2) **use-import resolution** — `emitPhpUseRefs` emits an `imports` ref in `Foo\Bar::Baz` form, matched precisely by the resolver's `resolveQualifiedName` (THE big lever, 80.5%→94.9%); (3) **type-hint refs** — PHP-aware `extractPhpTypeRefs` (PHP types are `named_type`/`union_type` wrapping `name`, not `type_identifier`). guzzle 95.2%→**100%**, laravel 80.5%→**94.9%**. Residual = class-string/reflection wiring (service providers, facades, middleware) — genuine frontier. Detail in memory `impact-coverage-findings.md`.
|
||||
|
||||
### Scala result (commit b5489d9, 2026-06-04)
|
||||
Scala was the WORST starting point — extraction made nodes but almost NO edges for typeclass code (cats 1.66 edges/node). Not one gap but a family, all gated to `language==='scala'` in `extraction/tree-sitter.ts` (+ `languages/scala.ts`): (1) **parameterized extends** — `extends A[X] with B` packed all supertypes in one `extends_clause`; generic path took only namedChild(0) w/ full text `A[X]` so no typeclass matched → new shared `scalaBaseTypeName` unwraps `generic_type`, iterate all supertypes (cats 48.9%→77.2% from THIS alone); (2) **type refs** (Scala had ZERO `references`) — added scala to TYPE_ANNOTATION_LANGUAGES + walk EVERY curried `parameters` list (trailing `(implicit M: TC[A])`!) + `type_parameters` context bounds (`[A: Monoid]`) + val/var types from scala.ts (77.2%→89.2%); (3) **instantiation** `new T[...]` = `instance_expression`. cats 48.9%→**89.2% fair** (82.1% raw — scalafix/bench excluded), gatling 76.3%→**91.2%**. Residual = cross-build variants/laws/wildcard-barrels (frontiers). Detail in memory `impact-coverage-findings.md`.
|
||||
|
||||
### Kotlin result (commit d8a2e91, 2026-06-04)
|
||||
Systematic Kotlin gap = **Kotlin Multiplatform `expect`/`actual`** (the only Kotlin-unique construct). OkHttp (the README Kotlin benchmark) was ALREADY 96.2% out of the box; kotlinx.coroutines (KMP) was 76.8% → **93.5%**. Fix: new generic `extractModifiers` hook captures `expect`/`actual` (from `modifiers > platform_modifier`) onto the node's `decorators` list (wired once in `createNode`); `kotlinExpectActualEdges` in callback-synthesizer.ts links common decl → each platform `actual` as a heuristic `calls` edge (matched by qualified_name + the `actual` marker; decl side = non-`actual` same-qn node, which also gates out plain overloads; kind-widened so `expect class` ↔ `actual typealias` links). Node count stable. Residual = genuine frontiers (expect-decl sides, ServiceLoader/agent SPI, test infra). Full detail in memory `impact-coverage-findings.md`.
|
||||
|
||||
## Goal
|
||||
Make the engine's cross-file dependency graph complete for **every README "Full support" language**, so impact/`affected`/callers/callees/explore all see real dependencies. Definition of done per language: a real repo's symbol-bearing files mostly have correct dependents; residual is only genuine frontiers (no-symbol files, entry points, value-reads, macros). Each language: audit → fix → validate → commit to the branch.
|
||||
|
||||
## Methodology (apply per language — this is the loop)
|
||||
1. Clone 1 benchmark + 1 clean repo to `/tmp`. Index with `CodeGraph.initSync(repo,{config:{include:['**/*.<ext>'],exclude:[]}})` + `indexAll()` + `resolveReferences()` via a `node -e` against `dist/index.js`.
|
||||
2. Measure **fair coverage** = % of *symbol-bearing* source files with ≥1 cross-file dependent. SQL: a file is a dependent target if it's the `target` of a non-`contains` edge whose `source` is in another file. **EXCLUDE from the denominator:** files with no non-`file` node (package-info.java, doc.rs, `__init__` umbrellas), tests, entry points (main/bin/examples/benches/fuzz/samples), and miscounted other-language files (e.g. `.kt` under a Java repo).
|
||||
3. Audit the 0-dependent files → classify real-miss vs frontier. Controlled probe (2 tiny files) to isolate the exact gap.
|
||||
4. Fix extraction/resolution. Re-measure. **Verify node count stays stable** (edges added, not nodes — except real new symbols like interface/record nodes).
|
||||
5. `npm run build` (tsc must pass) → `npm test` (expect ~1151 passing) → add a test in `__tests__/extraction.test.ts` → CHANGELOG `[Unreleased] → Fixes` bullet.
|
||||
6. `git add <files>` + commit (Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>) + `git push origin feat/cross-language-impact-coverage`. Do NOT commit `.claude/handoffs/*`.
|
||||
|
||||
## Key findings — the recurring gap shapes & where they're fixed
|
||||
- **Foundation (16b5633):** `imports` edges are same-file (file→local import node), so the old `getFileDependents` returned 0 for every file. Added `getDependentFilePaths`/`getDependencyFilePaths` in `src/db/queries.ts` (indexed JOIN, all kinds except `contains`); `src/graph/queries.ts` delegates.
|
||||
- **Import/binding linking** (per-lang emit* in `src/extraction/tree-sitter.ts`): `emitImportBindingRefs` (TS/JS named/default/namespace), `emitReExportRefs` (TS `export {X} from`), `emitPyFromImportRefs` (Python `from m import X`), `emitRustUseBindingRefs` (Rust `use`/`pub use`, emits FULL path). All gated by language in `extractImport`.
|
||||
- **Module-path resolution** (`src/resolution/import-resolver.ts`): `resolvePythonModuleMember` + `resolveModuleImportToFile` (Python+TS namespace), `resolveGoCrossPackageReference` (Go, pre-existing), `resolveRustPathReference`+`resolveRustModuleFile`+`rustCrateRootDir`/`rustSelfModuleDir` (Rust `crate::`/`self::`/`super::`). Resolve a path's module PREFIX to a file, find the leaf there — fixes common-name collisions.
|
||||
- **Instantiation:** `INSTANTIATION_KINDS` in tree-sitter.ts now includes `composite_literal` (Go) + `struct_expression` (Rust). `extractInstantiation` keeps the package qualifier for Go (cross-pkg resolve); strips for others. Also normalizes parenthesized type conversions `(*T)(x)`.
|
||||
- **Interface/trait dispatch (#584):** `IFACE_OVERRIDE_LANGS` in `src/resolution/callback-synthesizer.ts` now includes `go` and `rust`. Needs the interface/trait's METHODS extracted: Go via `extractGoInterfaceMethods` (tree-sitter.ts), Rust via adding `function_signature_item` to rust.ts function/methodTypes. `goImplementsEdges` synthesizes Go implicit `implements` edges (method-set match) and must `insertEdges` FIRST in `synthesizeCallbackEdges`.
|
||||
- **Annotations / attributes / property wrappers (UNIFIED via `extractDecoratorsFor`):** it now (a) descends into `modifiers` nodes (Java/Kotlin/C#), (b) recognizes Swift `attribute` + `user_type`. Java needed `annotation_type_declaration` added to `interfaceTypes` (java.ts). C# needed `record_declaration`/`record_struct_declaration` (csharp.ts). Swift needed a dispatcher branch running `extractDecoratorsFor`+`extractVariableTypeAnnotation` on `property_declaration` inside a type (Swift instance props aren't nodes).
|
||||
- **In-body type annotations (TS):** `visitFunctionBody` now extracts `variable_declarator` type annotations (`const x: Foo`).
|
||||
|
||||
## Per-language results — file-dependent coverage (% of symbol-bearing source files with ≥1 cross-file dependent)
|
||||
|
||||
| Language | Repo | Before | After | Key fix |
|
||||
|---|---|---|---|---|
|
||||
| TypeScript/JS | codegraph (this repo) | 62.5% | **95.8%** | import + re-export + namespace linking; in-body type annotations |
|
||||
| Python | requests | 54.1% | **100.0%** | `from x import` linking; `from . import sub` + `sub.f()` module-member resolution; relative-dot path fix |
|
||||
| Python | flask (src) | 66.7% | **87.5%** (true ceiling — residual all correct-0) | (same) |
|
||||
| Go | gin | 62.7% | **96.6%** | composite literals → instantiates; package-level var registries; `(*T)(x)` conversions; implicit interface satisfaction (#584) |
|
||||
| C# | MediatR (library) | 81.5% | **85.2%** | `record` / `record struct` indexed (#237) |
|
||||
| Rust | ripgrep | 63.4% | **86.7%** | struct literals; trait dispatch (trait methods + #584); `use`/`pub use` linking; module-path resolution for `pub use self::x::y` |
|
||||
| Rust | tokio (src) | 70.0% | **81.9%** | (same — number is honest/precise; earlier leaf-only match had inflated it) |
|
||||
| Java | gson | 78.2% | **85.1%** (raw) · **93.3% fair** | annotations: index `@interface` defs + link `@Foo` usages (in `modifiers`) |
|
||||
| Java | retrofit | 80.5% (raw) | **94.9% fair** | (same) |
|
||||
| Swift | Alamofire | 93.0% | **95.3%** | property wrappers / attributes (`@Argument`/`@Published`/`@objc`) |
|
||||
| Swift | swift-argument-parser | 84.6% | **96.2%** | (same) |
|
||||
| Kotlin | OkHttp | 96.2% | **96.2%** | already at ceiling (JVM, barely uses KMP) — no change needed |
|
||||
| Kotlin | kotlinx.coroutines | 76.8% | **93.5%** | Kotlin Multiplatform `expect`/`actual` linking (incl. `actual typealias`) |
|
||||
| Scala | typelevel/cats | 48.9% | **89.2% fair** (82.1% raw) | parameterized extends + type refs (implicit/context-bound) + `new` |
|
||||
| Scala | gatling | 76.3% | **91.2%** | (same) |
|
||||
| PHP | guzzle | 95.2% | **100.0%** | namespace capture + `use`-import resolution |
|
||||
| PHP | laravel/framework | 80.5% | **94.9%** | namespace capture (disambiguates same-named contracts) + use-imports + type-hints |
|
||||
| Ruby | rails/activerecord | 84.8% | **96.8%** | mixin edges (`include`/`extend`/`prepend`) + require resolution |
|
||||
| Ruby | sidekiq | 71.0% | **100.0%** | mixins + `require`/`require_relative` → file resolution |
|
||||
| C++ | google/leveldb | 91.7% | **94.8%** | fix free-function name extraction (was named after param/return type) |
|
||||
| C | redis | 92.2% | **92.2%** | already at ceiling (C unaffected; residual = generated/macro/fn-ptr) |
|
||||
| Dart | flutter/packages | 88.8% | **92.4%** | `with` mixins + method type references |
|
||||
| Dart | dio | 86.4% | **87.9%** | (same; raw 67.8% was example-dir pollution) |
|
||||
| Obj-C | AFNetworking | 50.0% | **90.0%** | single-arg selectors + class-receiver refs + #import + class-method resolution |
|
||||
| Obj-C | SDWebImage (Core) | 33.8% | **91.6%** | (same; `include/` dirs are symlink dups — measure Core/) |
|
||||
| Lua | nvim-telescope | 31.6% | **84.2%** | `require()` module resolution: dotted `a.b.c`→`a/b/c.lua` + instance-path leaf, path-suffix match w/ same-dir preference (4155609). Residual = telescope's dynamic `setmetatable` lazy-require pickers (frontier). |
|
||||
| Luau | dphfox/Fusion | 12.5% | **92.2%** | (same require resolver — `require(script.Memory.deriveScope)`) |
|
||||
| Liquid | Shopify/dawn | 39.1% | **73.8%** | Shopify OS 2.0 JSON-template section parsing: index `templates/*.json` + section-group `sections/*.json` (incl. nested `templates/customers/`), link each `"type"`→`sections/X.liquid` (2f57119). Snippet `{% render %}` already worked. Residual = preset/theme-editor sections (no static ref) + dynamic `{% render block %}`. |
|
||||
| Pascal/Delphi | PascalCoin | 73.0% | **75.7%** | `uses`-resolution ALREADY worked (73%); + `.dfm`/`.fmx` form ↔ same-basename `.pas` code-behind pairing (2f30a3b). Residual = vendored third-party libs (`src/libraries/*` internal units, ≈node_modules) + dynamic form/RPC instantiation. |
|
||||
|
||||
**"raw" vs "fair":** "fair" excludes files that *structurally can't* have dependents (no-symbol files like `package-info.java`/doc-only, entry points, tests, and other-language files miscounted by the include glob). For Java the raw numbers were heavily polluted (gson had many `package-info.java`; retrofit had `.kt` + samples), so the fair number is the real one (~93–95%). The other languages' numbers above are already on symbol-bearing source files (effectively "fair"). C# MediatR's 85.2% is the library-only figure; a package-info-excluded "fair" wasn't separately computed but is higher.
|
||||
|
||||
## Per-framework results — cross-language file-dependent coverage (RN/Expo, multi-platform JS↔native)
|
||||
|
||||
| Framework | Repo | Before | After | Key fix |
|
||||
|---|---|---|---|---|
|
||||
| React Native / Expo | react-native-async-storage | 75.0% | **97.4% fair** (37/38) | cross-family gate (082353e) + same-dir C/C++ `#include` + KMP commonMain import (529d822) |
|
||||
| React Native | react-native-device-info | 72.4% | **95.2% fair** (20/21) | cross-family gate (082353e) + honest fair metric (its 529d822 engine-fix targets are excluded entry files) |
|
||||
|
||||
**Metric note (read before trusting the "Before"):** the "Before" 75.0%/72.4% used an **under-exclusive** denominator — it counted generated codegen (`.g.h`), build scripts (`pch.*`, `*.gradle.kts`), tooling config (eslint/jest/yarn), and platform/registration **entry points** as if they were source. The "After" uses the **honest fair metric** the per-language table uses: excl. structural (generated/build/config/test), see-through barrels (web re-export files + umbrella/SDK headers — but NOT a 0-symbol source impl, which is a real frontier), and entry points (package `src/index`, platform `web`/`windows` entries, RN `ReactPackageProvider`). **Apples-to-apples** (fair metric held constant, isolating just the 529d822 engine fixes): async-storage **92.1% → 97.4%** (+RNCAsyncStorage.h via same-dir include, +Platform.kt via KMP import); rn-device-info **95.2% → 95.2%** (neutral — its same-dir/KMP targets are excluded entry headers, so its lift to 95.2% was the metric correction + the 082353e gate). Residual zeros (real frontiers): async-storage `DatabaseFiles.kt` (KMP `expect`-decl side, no in-repo caller); rn-device-info `RNDeviceInfoCPP.cpp` (`REACT_METHOD` macro methods not extracted). Measure with `/tmp/faircov.cjs <repo> --list`. No regression on controls: okhttp 75.9→76.4, kotlinx.coroutines 89.7 (neutral), leveldb 78.0 (neutral), redis 89.7→89.9, fmt 77.3 (neutral); cross-family false edges 0 everywhere.
|
||||
|
||||
## Route-framework headroom map (canonical app per README framework, FAIR coverage)
|
||||
|
||||
Measured 2026-06-04 (commit 61a993a) on a canonical real app for each README route framework. This is the active front of the campaign — the unmeasured frameworks have the real headroom.
|
||||
|
||||
| Framework | App repo | FAIR coverage | Status / next |
|
||||
|---|---|---|---|
|
||||
| Express (TS) | express-realworld | 70.4% → **100%** ✅ | DONE (2a0b6e0): renamed default-import → module file (route controllers `export default router`). |
|
||||
| FastAPI (Py) | fastapi-realworld | 78.6% → **98.0%** ✅ | DONE (2835623): source-aware `from pkg import submodule` (router aggregator). 1 residual = aliased sub-aggregator. |
|
||||
| Flask (Py) | flask (lib) | **100.0%** ✅ | DONE (entries/barrels excluded) |
|
||||
| requests (Py) | requests (lib) | **100.0%** ✅ | DONE |
|
||||
| NestJS (TS) | nestjs-realworld | 93.8% → **96.8%** ✅ | DONE (main.ts entry excluded) |
|
||||
| Gin (Go) | gin (lib) | **96.5%** ✅ | DONE (faircov Go `_test.go` exclusion) |
|
||||
| Laravel (PHP) | laravel (lib) | 92.0% | done (per-language; app not separately measured) |
|
||||
| Rails (Ruby) | rails (lib) | 89.6% | done (per-language) |
|
||||
| Django (Py) | django-realworld | 45.9% → **74.1%** | PARTIAL (58dc463): abs-module-import + `include('app.urls')` done. Ceiling ~83% w/ entries excluded. FRONTIERS: signals via in-body `ready(): import myapp.signals` (Python in-body imports NOT extracted — visitFunctionBody walks calls but not import_statement); DRF/string-config exception classes (`EXCEPTION_HANDLER: '...'`). |
|
||||
| ASP.NET (C#) | eShopOnWeb | 59.3% → **83.9%** | chained extension calls (4c14413) + framework-entry exclusions (b) + Razor/Blazor markup parser (59b8de2 tags/@model + 90c5f39 @code) + **C# namespaces (dc7d033) + Razor `@using` disambiguation (9e5a951)** — DTOs now resolve to `BlazorShared.Models::CatalogBrand` not the same-named entity. C# constructor DI / interface→impl ALREADY worked. Residual ~24 = reflection/proxy (AutoMapper profiles / Swagger filters / middleware / health checks — invoked by reflection, a separate modeling feature) + a few C# static-const reads (`Constants.X` — extend the static-member pass to C#). |
|
||||
| Spring (Java) | spring-petclinic | 65.2% → **83.3%** ✅ | DONE — convention/reflection ceiling, on par with ASP.NET. 30 main java = 15 covered + 7 entry (1 `*Application` main + 6 `*Controller`) + 5 barrel (`package-info.java`) + 3 zero. 3 residual zeros all reflection-registered: `CacheConfiguration`/`WebConfiguration` (`@Configuration`, component-scanned — `WebConfiguration` is even `@SuppressWarnings("unused")`) + `PetClinicRuntimeHints` (`@ImportRuntimeHints(X.class)` class-literal = the documented `Foo.class`→`Class` frontier). faircov fix = barrel-exclude `package-info.java`/`module-info.java` shells. **No `samples`-dir exclusion ever existed in the script** — that handoff note was inaccurate. |
|
||||
| Axum (Rust) | realworld-axum-sqlx | 72.7% → **100%** ✅ | DONE (a3f59fb) — import/aggregator style, confirmed in Rust. REAL engine miss found+fixed: bare Rust `submodule::fn()` calls (the `X::router()` router-assembly pattern) resolved crate-relative only → now self-relative FIRST (current module), then crate fallback. + `main.rs`/`src/bin/*.rs` faircov entry exclusion. Clean A/B (same faircov, engine ±fix): axum +2 → 100%, ripgrep neutral (66/79), tokio +3 (260/321); cross-family false edges 0. Regression test fails without the fix. |
|
||||
| Rocket (Rust) | TatriX/realworld-rust-rocket | 62.5% → 68.8% → **93.8%** ✅ | DONE — TWO engine wins. (1) 7bb958b: a 3-segment `database::profiles::find()` call (inside `db.run(\|c\| …)`) was dropped by the resolver PRE-FILTER (leaf never checked) → +database/profiles.rs. (2) 6d214cd: NEW `routes![]`/`catchers![]` macro extractor — the handler paths live in a raw token-tree, so the 4 `routes/*.rs` handler modules (mounted at runtime) looked uncalled; now reconstructed from the macro + resolved via the Rust path resolver → all 4 covered. Only `lib.rs` (crate-root/launch hub, see-through) remains. No control regression; false edges 0; nodes unchanged (edges only). |
|
||||
| actix-web (Rust) | fairingrey/actix-realworld-example-app | **65.4%** | DONE (genuine ceiling). actix **1.x ACTOR style**: routing resolves fine (`.route(web::post().to(users::register))` → handlers covered), but the DB layer is reached via actor message dispatch (`db.send(RegisterUser)` → `impl Handler<RegisterUser> for DbExecutor`) — a DYNAMIC runtime boundary, no static edge (the actix analog of the MediatR/reactive runtimes the engine deliberately leaves uncovered). 9 zeros = 5 `db/*.rs` (actor dispatch) + `models/mod`/`utils/mod`/`prelude` (glob `pub use self::x::*` re-export roots) + `utils/hasher` (reached only via that glob). Bridging needs an actor-message synthesizer (dynamic-dispatch feature), not a static-resolution fix. |
|
||||
| Vapor (Swift) | vapor-til (canonical) | 85.7% → **100%** ✅ | DONE — import/aggregator confirmed in Swift. Controllers covered via `app.register(collection: X())`, models via controller queries + migrations. REAL miss found+fixed (bb7659e): the many-to-many pivot model was referenced ONLY via `@Siblings(through: Pivot.self)` — a metatype arg inside a property-wrapper attribute that wasn't walked → now routed through `extractStaticMemberRef` → pivot covered. + faircov excl `Package.swift` (SPM manifest) & `Public/` (static assets). A/B (same faircov): vapor-til +1 → 100%, Alamofire neutral (44/50); false edges 0. |
|
||||
| Vapor (Swift) | penny-bot (real prod, 166 files) | **73.2%** | SECONDARY — a serverless/DI architecture, NOT the canonical route→controller→model shape. 37 zeros = AWS Lambda handlers (`*Handler`/`*Lambda`, invoked by the AWS runtime → entry points, no in-repo caller) + `exports.swift` re-export barrels (see-through, like Rust mod.rs roots) + Swift extension files (`+X.swift`) + DTOs/services reached via DI / JSON-decoding (genuine frontiers). Excluding the Lambda entries + exports barrels (faircov gaps) → ~80%; genuine residual is the DI/serialization architecture. NOT chased — vapor-til is the canonical result. |
|
||||
| SvelteKit (Svelte) | sveltejs/realworld (official) | 35.9% raw → **100% fair** ✅ | DONE — characterization, NO engine miss (first framework that needed none). The `.svelte` import resolution ALREADY connects the whole component graph (page→component→lib: ArticlePreview←ArticleList, Nav←+layout, ListErrors←4 pages — imports+references+calls all resolve). Low RAW = the file-convention shape: 25/39 files are SvelteKit convention ENTRIES (`+page`/`+layout`/`+error`/`+server`/`hooks` — loaded by the framework BY PATH, no in-repo caller = route leaves, like route handlers). Excluding those (new faircov ENTRY patterns) → 14/14 = 100% of the coverable core. ONE framework-mediated gap: `+page.server.js` `load` → `+page.svelte` `data` is NOT a static link (runtime `data` prop) — was bridged with a SvelteKit convention synthesizer (sibling files by path, like Rocket routes!) — impact-granularity, not coverage (both files are entries). **BUILT (3ea03e5): `svelteKitLoadEdges`** links each page component → its OWN-dir loader's `load`/`actions` (same for `+layout`); path-deterministic (19 links / **0 cross-dir mislinks** on realworld), provenance heuristic, nodes unchanged. `getImpactRadius(load)` now surfaces its page. |
|
||||
| React Router (React) | gothinkster realworld | **100%** ✅ | DONE — characterization, NO engine miss. The CONFIG-based component-node sub-shape: route components are NAMED in source (`<Route path='/login' component={Login}/>` → App.js imports + references Login), so they resolve cleanly — verified Login/Editor/Profile all covered BY App.js's route config (imports+references). 29/29 (only `index.js` CRA entry + 8 anonymous-default-export reducers [0 named symbols → barrels] excluded). CONTRAST with SvelteKit (file-convention, needs entry-exclusion): the component-node category has TWO sub-shapes, both handled by the engine as-is. (App is old v4/v5 `component={X}`; modern v6/v7 `element={<X/>}` + data-router `{element, loader}` are also source-referenced → same conclusion.) No commit (nothing to fix). |
|
||||
| Nuxt (Vue) | nuxt/movies (official demo) | 47.6% → **93.5% fair** ✅ | DONE — file-convention component framework (like SvelteKit) but with an AUTO-IMPORT twist that was a REAL miss (1dec765). Nuxt auto-imports a NESTED component by a DIR-PREFIXED name: `components/media/Card.vue` → `<MediaCard/>`, but the node is named `Card` → the PascalCase template usage never resolved (the synth only matched kebab tags; the extractor's PascalCase ref name-matched `MediaCard`→nothing). Fix in `vueTemplateEdges`: match PascalCase tags + a `nuxtComponentName` map (`media/Card.vue`→`MediaCard`, incl. Nuxt dir de-dup) → +9 nested components. + faircov Nuxt entry exclusions (pages/, app.vue, error.vue, layouts/, plugins/, middleware/, server/). 2 residual zeros = genuine frontiers (unused `<CarouselItems>` variant + unimported `constants/images.ts`). Composables auto-import (`useTmdb()`) already resolved by name-match. false edges 0, nodes unchanged. **vs SvelteKit:** SvelteKit needed NO component fix (svelte imports resolve); Nuxt DID (its auto-import dir-prefix naming is unique). |
|
||||
| Drupal (PHP) | token (contrib module) | 57.1% raw → **78.9% fair** | DONE — the MOST reflection-heavy framework, a genuine sub-95% ceiling (≈ASP.NET 83.9%, Django 74.1%). Static parts WORK: `*.routing.yml`→controller (drupal.ts resolves it — `TokenTreeController` covered via the route ref), PHP class refs (namespaces/`use`/type-hints). 4 residual zeros all DI/reflection-wired: 2 services in `services.yml` (`TreeBuilder` = `@token.tree_builder`, `TokenFieldRender`), a DYNAMICALLY-routed controller (`TokenDevelController` via `RouteSubscriber`, not static YAML), a field-plugin class (`MenuLinkFieldItemList` list_class). + faircov excl JS libraries + convention entries (`.install`/`.module`, `src/Plugin`/`Element`, Drush, `*ServiceProvider`, `Routing`/`*Subscriber`). LEVERS (drupal.ts TODOs, large per-mechanism features, NOT chased): model `services.yml` DI (→ covers the 2 services), plugin annotations + modern `#[Hook]` attributes. No commit (no simple miss). |
|
||||
|
||||
**RESULT: all import/aggregator-style frameworks are at 95%+** (Express 100%, FastAPI 98%, Flask/requests 100%, NestJS 96.8%, Gin 96.5%, Axum 100%).
|
||||
|
||||
**Option (b) DONE (metric-only — framework-entry exclusions in `/tmp/faircov.cjs`):** added convention-entry patterns (`*Controller.cs/java`, `*.cshtml.cs`, `*Endpoint.cs`, EF `Data/Config/*.cs`, `Program/Startup.cs`, `*Application.java`, Django `admin.py`/`apps.py`). Result — convention frameworks rise but **still cap well below 95%**: ASP.NET 65.3% → **77.2%** (50 entries excluded), Spring petclinic 65.2% → **83.3%** (after the `package-info.java` shell-exclusion fix below), Django **74.1%**. The import-style frameworks are unaffected (Express 100%, FastAPI 98%, NestJS 96.8%, Gin 96.5% — the C#/Java/Django entry patterns don't touch them).
|
||||
|
||||
**WHY (b) doesn't reach 95% — the honest ceiling:** after excluding routed/reflection-registered entries, the residual zeros are **markup-driven** code-behind (Blazor `.razor` / Razor `.cshtml` / Thymeleaf reference the `.cs`/`.java`, but the markup isn't parsed → ViewModels, DTOs, components look unused) and **reflection/proxy** code (Spring Data repository proxies, AutoMapper profiles, Swagger filters, DI/middleware registration, Django signals/string-config). These are genuine static-analysis frontiers — reaching 95% needs (1) parsing template markup to link markup→code, or (2) per-framework reflection/proxy modeling — both large features. **Excluding markup-driven business code (DTOs/ViewModels) from the metric to fake 95% would be gaming — NOT done.** Note: business LOGIC (services, repos) IS covered in all three; the residual is leaf views/DTOs/configs whose impact is captured the other direction (route→handler).
|
||||
|
||||
**Generalizable engine fixes shipped this campaign (all benefit beyond their trigger framework):** Python absolute `import a.b.c` (61a993a); source-aware `from pkg import submodule` (2835623); Django `include('app.urls')` claim (58dc463); chained method calls `a.b.Method()` incl. C# extension methods (4c14413); renamed default-import → module file (2a0b6e0); Rust bare `submodule::fn()` calls resolved self-relative — current module first, then crate fallback (a3f59fb); multi-segment Rust `a::b::c()` module calls no longer dropped by the resolver pre-filter — the leaf name is now checked (7bb958b).
|
||||
|
||||
**KEY REALITY (honest):** apps dominated by **convention/reflection-driven** code (ASP.NET MVC/Razor/Blazor, EF config, reflection DI; Django signals/DRF; any framework whose handlers are discovered by routing/DI container, not called by in-repo code) have files with NO static in-repo caller. Those are genuine static-analysis frontiers — **literal 95% is not reachable** on such apps without either (a) excluding all framework-entry conventions from the fair denominator (defensible per methodology but extensive + per-framework), or (b) modeling each framework's convention routing + DI container (large per-framework engine work). The DI-heavy/convention-heavy frameworks (ASP.NET, Spring, MVC) are this category; the import/aggregator-style ones (FastAPI, Flask, Express, Gin) reach 95%+ with tractable resolution fixes.
|
||||
|
||||
**SWEEP COMPLETE — every README framework measured.** Route frameworks: Express 100%, FastAPI 98%, Flask/requests 100%, NestJS 96.8%, Gin 96.5%, Axum 100%, Rocket 93.8%, Vapor 100%, Laravel 92%, Rails 89.6% (import/aggregator) · ASP.NET 83.9%, Spring 83.3%, Drupal 78.9%, Django 74.1%, actix 65.4% (convention/reflection/actor ceiling). Component-node: React Router 100% (config-based) · SvelteKit 100% fair, Nuxt 93.5% fair (file-convention). **Component-node category fully characterized**: config-based (route components named in source → 100% raw) + file-convention (discovered by path → entry-exclusion; SvelteKit needed NO component fix, Nuxt needed the dir-prefix auto-import fix). Convention/reflection frameworks (ASP.NET/Spring/Drupal/Django/actix) are at a genuine sub-95% static-analysis ceiling — the levers are large per-mechanism features (markup/reflection/DI modeling), deliberately not chased. **faircov exclusions added this session:** language-aware test files (`_test.go`, `test_*.py`, `*Tests.cs`, `*_spec.rb`, …); generated migrations (Django/Alembic `migrations/`, EF `Migrations/*.cs`/`*.Designer.cs`/`*ModelSnapshot.cs`); Python entries (`__main__.py`, `setup.py`, `conf.py`, `docs/`) + `__init__.py` barrels. faircov now barrel-excludes `package-info.java`/`module-info.java` shells (package-declaration-only files — they carry a `namespace` node so `realSyms !== 0`, but nothing can import them, so they can never be a dependency target; the Java analog of `__init__.py`). This is what actually blocked an honest Spring number: petclinic's 5 `package-info.java` sat in the denominator as zeros, deflating 83.3% → 65.2%. **There was never a `samples`-dir exclusion in the script** — that earlier handoff note was inaccurate (the only similar rule is `example[s]?`, which can't match `samples`). Route-framework-front faircov ENTRY additions: Rust binary entries (`main.rs`, `src/bin/*.rs`); Swift `Package.swift` (SPM manifest) + `Public/` static assets; SvelteKit file-convention entries (`+page`/`+layout`/`+error`/`+server.*`, `hooks.*`, `service-worker.*` — framework-discovered route leaves, no in-repo caller); Nuxt file-convention entries (`pages/`, `app.vue`/`error.vue`, `layouts/`, `plugins/`, `middleware/`, `server/`, `proxy/routes/`); Drupal convention entries (`.install`/`.module`/`.profile`/`.theme` hook files, `src/Plugin`/`src/Element` annotation-plugins, `src/Drush`, `*ServiceProvider.php`, `src/Routing`/`*Subscriber.php`) + `js/` library assets.
|
||||
|
||||
## How to push each language higher (remaining levers)
|
||||
|
||||
**The one big cross-language lever: a static-member / const value-read pass.** Extract `Type.MEMBER` (capitalized/known-type receiver) as a `references` edge to `Type`. This is the universal deferred data-flow frontier and would lift **C#, Java, Swift, TS, and Rust at once**. Implement once in extraction with a heuristic (receiver resolves to / looks like a type → emit ref; skip lowercase `obj.field`). Trade-off = some instance-field-access noise; that's why it's been deferred. This is the highest-leverage single task remaining.
|
||||
|
||||
Per language — what's left and the action to improve it:
|
||||
- **C# — MediatR 85.2% (raw, the lowest real number):**
|
||||
- *raw→fair:* exclude no-symbol files (`TypeForwardings.cs` = assembly attrs only, `package-info`-equivalents) + benchmark `main`s → ~92%+. **A fair re-measure was never run for C# — do it first; the "real" number is materially higher.**
|
||||
- *to improve further:* static/const value reads (`BuildInfo.BuildDate`, enum `Edition?` where a same-named property shadows the type) → the static-member pass.
|
||||
- **Java — gson 85.1% raw → 93.3% fair:**
|
||||
- *raw→fair:* exclude `package-info.java` (no symbols) + `.kt`/samples (already done for the fair number).
|
||||
- *to improve fair further:* static-field reads (`X.FACTORY`), `Foo.class` class literals (currently `Foo.class` references `Class`, not `Foo`), constant reads (`JsonScope.X`) → the static-member pass.
|
||||
- **Rust — tokio 81.9% (lowest of the high group), ripgrep 86.7%:**
|
||||
- residual = see-through `mod.rs`/`lib.rs` roots (correct-0), **macro-reached code** (`log!`, custom `macro_rules!`, derives — the big Rust frontier, hard), external-trait-only impls.
|
||||
- *to improve:* macro handling (large, separate project) + static/const reads. Note tokio's 81.9% is already honest/precise (path resolution removed spurious leaf-match edges).
|
||||
- **Go — gin 96.6%:**
|
||||
- residual = `//go:build` alternates (appengine/jsoniter/go_json/sonic/nomsgpack) + external-API `version.go`.
|
||||
- *to improve:* a **build-constraint parser** (evaluate `//go:build`) so inactive variants are excluded from the denominator or all variants are linked (recall-first). Only matters for build-tag-heavy repos; niche.
|
||||
- **TS 95.8% · Python requests 100% / flask 87.5% · Swift 95.3% / 96.2%:** at/near true ceiling — residual is entry points, see-through barrels, external public API, and value-reads. The **only** lever left for these is the static-member/const pass.
|
||||
|
||||
**Bottom line:** Python/TS/Swift/Go are effectively at ceiling. The two with real headroom are **C#** (mostly a fair-remeasure — do that first) and **Rust tokio** (macros — hard). The static-member/const pass is the one change that moves *everything* a few points; the rest is per-language frontier work.
|
||||
|
||||
## Which tools benefit (asked + answered this session)
|
||||
It's a GRAPH-WIDE update (one shared `edges` table). `getCallers`/`getCallees` follow `['calls','references','imports']`; `getImpactRadius` + `getFileDependents`/`affected` follow **all except `contains`**; `codegraph_explore` composes all of them. So `instantiates`/`implements`/`decorates` edges show in impact+explore but **not** callers/callees (a pre-existing edge-kind filter in `getCallers`/`getCallees` — could be broadened, deferred).
|
||||
|
||||
## Gotchas
|
||||
- **Include globs don't filter reliably** — tests/examples/benches/`.kt` leak into the index. Filter in the measurement SQL, not the config.
|
||||
- **`/tmp` clones persist across turns** — `rm -rf <repo>/.codegraph` before re-indexing or `initSync` throws "already initialized" and you measure a STALE index (this bit me ~3×; a stale index massively under-reports).
|
||||
- **Fair metric must exclude no-symbol files** (package-info, doc-only) — they can't have dependents; counting them is dishonest-low. Also a slightly-LOWER honest number (Rust tokio 83→82 after path resolution) beat the spurious-inflated one — precision over optics.
|
||||
- **Build vs test:** `npm test` uses esbuild (no typecheck); `npm run build` (tsc) is what catches type errors. Always build before committing. Strict null on regex groups bit me — avoid `m[1]` indexed access.
|
||||
- Node-version regex-group access (`m[1]`) is `string|undefined`; use guards.
|
||||
|
||||
## How to test & validate
|
||||
- `npm run build` → tsc clean (must pass before commit).
|
||||
- `npm test` → **1178 passed | 2 skipped** (59 files, verified 2026-06-05). Per-language tests live in `__tests__/extraction.test.ts` (describe per language); route-framework/RN tests in `__tests__/{react-native-bridge,expo-modules,rn-event-channel}.test.ts`.
|
||||
- Coverage probe recipe: clone repo → `node -e "...initSync...indexAll...resolveReferences..."` → the fair-coverage SQL (see Methodology #2). Node count stable = no explosion.
|
||||
- Full per-language findings + exact fixes: memory file `~/.claude/projects/-Users-colby-Development-CodeGraph-codegraph/memory/impact-coverage-findings.md`.
|
||||
|
||||
## Repo state
|
||||
- branch `feat/cross-language-impact-coverage`, last commit `2f30a3b feat(impact): pair Delphi forms with their .pas code-behind (.dfm/.fmx ↔ .pas)`.
|
||||
- **41 commits ahead of `main`, 3 behind** (behind = 3 README-waitlist doc commits on main). All pushed to origin. NOT merged — branch is for review. Niche-lang commits 39–41: 4155609 (Lua/Luau require resolver), 2f57119 (Liquid Shopify-JSON sections), 2f30a3b (Delphi .dfm form pairing).
|
||||
- Commits 1–20 (per-language + RN/Expo, `16b5633`→`529d822`): 16b5633 (foundation+TS/Py/Go/C#), b538aee + 2ac7df5 (Rust), badb124 (Java), d111f26 (Swift), d8a2e91 (Kotlin), b5489d9 (Scala), acfb444 (PHP), 44fb978 + 5bccab6 (Ruby), ec8fe3f (C/C++), 9487954 (Dart), 857baf7 (static-member pass), 33ce431 (Objective-C), dbc4862 (Expo bridges), 4a64ca5 (classic RN pairing), d06a5ec (RCT_EXPORT_METHOD nodes), 74b599c (RN event wrapper), 082353e (cross-family gate), 529d822 (same-dir include + KMP import).
|
||||
- Commits 21–31 (ROUTE-FRAMEWORK front, `61a993a`→`b653688`, added after the original save): 61a993a (Python absolute `import a.b.c`), 2835623 (source-aware `from pkg import submodule` — FastAPI), 58dc463 (Django `include('app.urls')`), 4c14413 (chained `a.b.Method()` + C# extension methods), 2a0b6e0 (renamed default-import → module file — Express), 59b8de2 (Razor/Blazor `.cshtml`/`.razor` markup parser), 90c5f39 (Blazor `@code`/Razor `@{}` → C# extractor), 4589cf9 (docs), dc7d033 (C# namespace extraction), 9e5a951 (Razor `@using` disambiguation), b653688 (handoff notes + docs). Commit 32 (`a3f59fb`): Rust self-relative `submodule::fn()` resolution — Axum realworld 72.7%→100%, controls neutral/+3, regression test added. Commit 33 (`7bb958b`): multi-segment Rust `a::b::c()` calls no longer dropped by the resolver pre-filter (leaf-name check in `hasAnyPossibleMatch`) — Rocket realworld 62.5%→68.8%, additive elsewhere, regression test added. Commit 34 (`6d214cd`): Rocket `routes![]`/`catchers![]` macro extractor (parse the handler paths out of the raw token-tree, emit references) — Rocket 68.8%→93.8%, nodes unchanged (edges only), no control regression, regression test added. Commit 35 (`bb7659e`): Swift property-wrapper attribute-arg type refs — route a property's attribute args through `extractStaticMemberRef` so a Fluent `@Siblings(through: Pivot.self)` links the model to the pivot — vapor-til 94.7%→100%, Alamofire neutral, regression test added. Commit 36 (`3ea03e5`): SvelteKit `load`→page synthesizer (`svelteKitLoadEdges` in callback-synthesizer.ts) — links each `+page.svelte`/`+layout.svelte` to its OWN-dir loader's `load`/`actions`; impact-granularity (coverage already 100%), 0 cross-dir mislinks, nodes unchanged, regression test added. Commit 37 (`1dec765`): Nuxt nested auto-imported component resolution — `vueTemplateEdges` now matches PascalCase tags + a `nuxtComponentName` map (`<MediaCard>`→`media/Card.vue`); nuxt/movies 47.6%→93.5%, +9 nested components, Vue-only (gated to .vue), nodes unchanged, regression test added.
|
||||
- uncommitted: only untracked `assets/generate-waitlist.py` (unrelated — README-waitlist tooling; the `.claude/handoffs/*` files are committed on this branch).
|
||||
- Touched source files (branch vs main): `src/db/queries.ts`, `src/graph/queries.ts`, `src/extraction/{tree-sitter,tree-sitter-types,grammars}.ts`, `src/extraction/razor-extractor.ts` (NEW — Razor/Blazor markup), `src/extraction/languages/{rust,java,csharp,kotlin,php,ruby,scala,c-cpp}.ts`, `src/resolution/{import-resolver,callback-synthesizer,index,name-matcher}.ts`, `src/resolution/frameworks/{python,react-native,expo-modules}.ts`, `src/types.ts`, `__tests__/{extraction,graph,expo-modules,react-native-bridge,rn-event-channel}.test.ts`, `CHANGELOG.md`.
|
||||
- Measurement scripts (in /tmp, not committed): `faircov.cjs` (honest fair coverage + false-edge count, `--list` shows residual zeros + exclusions), `audit.cjs` (lists 0-dependent files by language), `xlang.cjs` (cross-lang edges by src→tgt × kind).
|
||||
|
||||
## Open threads / TODO
|
||||
- [x] **Kotlin DONE** (commit d8a2e91) — gap was KMP `expect`/`actual`; coroutines 76.8%→93.5%, OkHttp already 96.2%. See "Kotlin result" above.
|
||||
- [x] **Scala DONE** (commit b5489d9) — gap was a whole family of missing edges (parameterized extends, type refs, implicit/context-bound params, `new`); cats 48.9%→89.2% fair, gatling 76.3%→91.2%. See "Scala result" above.
|
||||
- [x] **PHP DONE** (commit acfb444) — gap was NAMESPACES (not #608/#660); guzzle 95.2%→100%, laravel 80.5%→94.9%. See "PHP result" above.
|
||||
- [x] **Ruby DONE** (commits 44fb978 + 5bccab6) — gaps were MIXINS + REQUIRE resolution; activerecord 84.8%→96.8%, sidekiq 71%→**100%**. See "Ruby result" above.
|
||||
- [x] **C/C++ DONE** (commit ec8fe3f) — gap was a C++ free-function name-extraction bug; leveldb 91.7%→94.8%, redis (C) 92.2% at ceiling. See "C/C++ result" above.
|
||||
- [x] **Dart DONE** (commit 9487954) — gaps were mixins (`with`) + method type refs; flutter/packages 88.8%→92.4%, dio 86.4%→87.9%. See "Dart result" above.
|
||||
- [ ] **Objective-C next** (last README language, already partial — `@interface`/`@implementation` split, `#import`, categories, protocols, `@property`). Niche after: Liquid, Pascal, Lua, Luau.
|
||||
- [x] **Static-member/value-read pass DONE** (commit 857baf7) — `Enum.value`/`Type.CONST`/`Foo::BAR` → references; flutter 92.4%→93.2%, additive across Java/C#/Kotlin/Swift/Scala/PHP/C++. TS/JS/Python excluded.
|
||||
- [x] **Objective-C DONE** (commit 33ce431) — selectors + class-receiver + #import + class-method resolution; AFNetworking 50%→90%, SDWebImage Core 33.8%→91.6%. ← LAST README language.
|
||||
- [x] **Per-language + RN/Expo campaign COMPLETE** (`16b5633`→`529d822`): all 15 README langs + static-member pass + cross-language RN/Expo at 95%+.
|
||||
- [x] **Route-framework front — import/aggregator style DONE** (`61a993a`→`2a0b6e0`): Express 100%, FastAPI 98%, Flask/requests 100%, NestJS 96.8%, Gin 96.5%. See the headroom map.
|
||||
- [~] **Route-framework front — convention/reflection style PARTIAL** (`4c14413`,`59b8de2`,`90c5f39`,`dc7d033`,`9e5a951`): ASP.NET eShopOnWeb 59.3%→83.9% (Razor/Blazor parser + C# namespaces), Django realworld 45.9%→74.1%. At an honest sub-95% static-analysis ceiling (markup-driven code-behind + reflection/DI). Reaching 95% needs markup→code linking or per-framework reflection modeling (large features); faking it via metric exclusions = gaming, NOT done.
|
||||
- [x] **Spring DONE** (83.3% fair, 15/18) — the real blocker was faircov NOT excluding `package-info.java` shells (5 in petclinic), not a `samples`-dir exclusion (which never existed in the script). Fixed → barrel-exclude `package-info.java`/`module-info.java`. 3 residual zeros = `@Configuration` beans + AOT `RuntimeHints`, all reflection-registered → the convention/reflection ceiling, on par with ASP.NET 83.9%.
|
||||
- [x] **Axum (Rust) DONE** (72.7%→**100%**, a3f59fb) — import/aggregator thesis confirmed in a 3rd language family. Found+fixed a REAL engine miss (not a metric issue): bare Rust `submodule::fn()` calls (the `X::router()` router-assembly pattern) resolved crate-relative only → now self-relative FIRST (current module), then crate fallback. Clean A/B: no control regression (ripgrep neutral 66/79, tokio +3 → 260/321); cross-family false edges 0; regression test fails without the fix. NOTE: actix/Rocket NOW MEASURED (below).
|
||||
- [x] **Rocket (Rust) DONE** (62.5%→68.8%→**93.8%**, 7bb958b + 6d214cd) — TWO engine wins: (1) multi-segment `a::b::c()` calls were dropped by the resolver pre-filter (leaf never checked) → fixed → +database/profiles.rs; (2) built the `routes![]`/`catchers![]` macro extractor — parses handler paths out of the raw token-tree + resolves them via the Rust path resolver → all 4 `routes/*.rs` handler modules covered. Only `lib.rs` (crate-root/launch hub, see-through) remains → effectively at ceiling. No control regression, false edges 0, nodes unchanged (edges only).
|
||||
- [x] **actix-web (Rust) DONE** (**65.4%**, genuine ceiling) — actix 1.x ACTOR style: routing resolves (handlers covered), but the DB layer is reached via actor message dispatch (`db.send(Msg)` → `impl Handler<Msg> for DbExecutor`) — a DYNAMIC runtime frontier with no static edge (deliberately uncovered, the actix analog of MediatR/reactive runtimes). + glob `pub use self::x::*` re-export roots. Bridging needs an actor-message synthesizer (dynamic-dispatch feature), not a static fix.
|
||||
- [x] **Vapor (Swift) DONE** — vapor-til (canonical) 85.7%→**100%** (bb7659e), penny-bot (real prod, 166 files) 73.2% (serverless/DI, secondary). Import/aggregator confirmed in Swift: controllers via `app.register(collection: X())`, models via controller queries + migrations. REAL miss found+fixed: a many-to-many pivot model referenced ONLY via `@Siblings(through: Pivot.self)` (metatype arg in a property-wrapper attribute, not walked) → routed attribute args through `extractStaticMemberRef` → pivot covered. + faircov excl `Package.swift` + `Public/`. penny-bot's lower number = AWS Lambda entries + `exports.swift` barrels (faircov gaps, ~80% if excluded) + DI/serialization/extension frontiers (genuine); not chased.
|
||||
- [x] **SvelteKit (Svelte) DONE** (35.9% raw → **100% fair**, NO commit — first framework needing NO engine fix) — sveltejs/realworld (official). The `.svelte` import resolution ALREADY connects the whole component graph (page→component→lib — verified: imports+references+calls all resolve, e.g. ListErrors←4 pages). Low raw = the file-convention shape: 25/39 files are SvelteKit convention ENTRIES (`+page`/`+layout`/`+error`/`+server`/`hooks` — framework-discovered route leaves, no in-repo caller) → excluded in faircov → 14/14 = 100% of the coverable core. ONE framework-mediated gap: `+page.server.js` `load` → `+page.svelte` `data` (runtime `data` prop, not static) — bridged with a SvelteKit convention synthesizer (sibling-by-path, like Rocket routes!) — impact-granularity not coverage (both files are entries). **BUILT (3ea03e5): `svelteKitLoadEdges`** (callback-synthesizer.ts) links each page → its own-dir loader's `load`/`actions` + `+layout`; 19 links / 0 cross-dir mislinks on realworld, nodes unchanged, regression test (incl. does-not-cross-routes guard).
|
||||
- [x] **React Router (React) DONE** (**100%**, 29/29, NO commit — no engine miss) — gothinkster realworld. The CONFIG-based component-node sub-shape: route components are NAMED in source (`<Route path='/login' component={Login}/>`) → App.js imports+references each → all resolve (verified Login/Editor/Profile covered by App.js's route config). Only `index.js` (CRA entry) + 8 anonymous-default reducers (0 symbols → barrels) excluded. The component-node category is now fully characterized: config-based (React Router, source-referenced → 100% raw) vs file-convention (SvelteKit, path-discovered → fair-100% + synthesizer); the engine handles BOTH as-is. (Modern v6/v7 `element={<X/>}`/data-router `{element, loader}` also source-referenced → same.)
|
||||
- [x] **Nuxt (Vue) DONE** (47.6% → **93.5% fair**, 1dec765) — nuxt/movies (official demo). File-convention component framework (like SvelteKit) but with an AUTO-IMPORT twist that WAS a real miss: Nuxt auto-imports a NESTED component by a DIR-PREFIXED name (`components/media/Card.vue` → `<MediaCard/>`), but the node is named `Card` → the PascalCase usage never resolved. Fixed `vueTemplateEdges` (match PascalCase tags, not just kebab + a `nuxtComponentName` map) → +9 nested components. + faircov Nuxt entry exclusions. composables (`useTmdb()`) already resolved by name-match. 2 residual zeros = genuine frontiers (unused `<CarouselItems>` + unimported `constants/images.ts`). **KEY contrast:** SvelteKit needed NO component fix, Nuxt DID — its dir-prefix auto-import naming is unique. The Vue PascalCase-tag change is gated to `.vue` files (suite green, false edges 0).
|
||||
- [x] **Drupal (PHP) DONE** (57.1% raw → **78.9% fair**, NO commit — reflection ceiling, no simple miss) — token contrib module. The MOST reflection-heavy framework. Static parts WORK: `*.routing.yml`→controller (drupal.ts), PHP class refs. 4 residual zeros all DI/reflection: 2 `services.yml` services (`TreeBuilder`/`TokenFieldRender`), a dynamically-routed controller (`TokenDevelController` via `RouteSubscriber`), a field-plugin class. + faircov excl JS libraries + convention entries. Sits with ASP.NET 83.9% / Django 74.1% in the convention/reflection band. LEVERS = drupal.ts TODOs (model `services.yml` DI → +2 services; plugin annotations; modern `#[Hook]` attributes) — large per-mechanism features, NOT chased (same stance as ASP.NET/Spring/Django: don't fake 95% via reflection modeling). **← LAST framework; the sweep is complete.**
|
||||
- [ ] **Frameworks not yet measured** (need a canonical app cloned): Drupal, React Router, Vue-Nuxt (SvelteKit DONE: 100% fair — component-node shape characterized, NO engine miss; Rust route DONE: Axum 100%, Rocket 93.8%, actix 65.4%; Vapor DONE: vapor-til 100%, penny-bot 73.2% serverless/DI) (component-node frameworks — coverage shape differs). The headroom map calls these the real remaining headroom.
|
||||
- [x] **All 4 niche README languages DONE** (Lua 84.2%, Luau 92.2%, Liquid 73.8%, Pascal 75.7%) — see the per-language table. Full README parity (22 langs + 14 frameworks).
|
||||
- [ ] **Open the PR to `main`** (41 commits) — everything actionable is done. Deferred polish (genuinely optional): C function-pointer dispatch + C++ namespace capture; PHP IFACE_OVERRIDE_LANGS + type-hint disambiguation; Scala wildcard imports; ObjC type refs; engine symlink-dedup (SDWebImage `include/` dup file nodes); a convention/reflection lever (Drupal services.yml DI / ASP.NET-Spring reflection modeling — large per-framework features). (Static-member pass for TS/JS/Python is NOT a lever — measured & rejected, e7b86df.)
|
||||
- [x] **Static-member/const pass for TS/JS/Python MEASURED & REJECTED** (e7b86df) — the long-standing "one big cross-language lever." A/B (extend `extractStaticMemberRef` to ts/js/python): **0 coverage gain** on excalidraw (94.3%→94.3%) AND requests (100%→100%) — in import-based languages a `Type.MEMBER` read requires `import`ing the type, so the import edge already covers it (the static read is pure duplication) — while adding **+1813 edges / +2448 `references` on excalidraw** (the retrieval-perf benchmark), all pointing at already-covered types. Reverted; finding documented in the `STATIC_MEMBER_LANGS` comment so it isn't re-tried. The deferral was CORRECT — now proven with data.
|
||||
- [ ] Cross-language **static-member / const value-read** pass (`X.FACTORY`, `JsonScope.X`, `Foo.class`) — the universal deferred data-flow frontier; would lift Java/C#/etc. ~5% more but risks noise. User aware; deferred.
|
||||
- [ ] Optional: broaden `getCallers`/`getCallees` edge-kind list to include `instantiates`/`implements`/`decorates` (offered, deferred — changes what "caller" means).
|
||||
- [ ] Branch is NOT merged — it's for review. Maintainer handles version bumps/releases. Don't merge or bump.
|
||||
|
||||
## Recent transcript (oldest → newest)
|
||||
### Turn — "Rust next"
|
||||
- **Claude:** Cloned ripgrep+tokio. Found struct literals (`struct_expression`) not in INSTANTIATION_KINDS, `use` bindings not linked, trait methods (`function_signature_item`) not extracted. Fixed all + added rust to IFACE_OVERRIDE_LANGS. ripgrep 63%→81%, tokio→83%. → commits b538aee, then 2ac7df5 (module-path resolution for `pub use self::read::read` collisions → ripgrep 87%).
|
||||
### Turn — "do fixes translate to explore/callers/callees or just impact?"
|
||||
- **Claude:** Verified against tool code → graph-wide; mapped edge-kinds per tool (callers/callees = calls/references/imports; impact/explore = all-but-contains). instantiates/implements/decorates only in impact/explore.
|
||||
### Turn — "Java next" → "is Java good or can we improve it?"
|
||||
- **Claude:** Found Java annotation gap (usages in `modifiers` not descended; `@interface` not in interfaceTypes). Fixed both → gson 78%→85% (commit badb124). On re-push ask: pushed already. On "is it good": fair re-measure showed Java actually **93.3%/94.9%** (raw was polluted by package-info/.kt/samples); residual = universal value-read frontier. Concluded Java is good.
|
||||
### Turn — "Swift next"
|
||||
- **Claude:** Alamofire already 93%. argparser 84.6% — found property-wrapper gap (`@Argument` = Swift `attribute`→`user_type`, not recognized; Swift instance props not extracted). Extended `extractDecoratorsFor` + dispatcher branch. Alamofire→95.3%, argparser→96.2%. Commit d111f26. Completes README benchmarked 7.
|
||||
### Turn — /handoff save (original, 2026-06-04)
|
||||
- Saved at commit 529d822 (per-language + RN/Expo campaign).
|
||||
### Session — ROUTE-FRAMEWORK front (2026-06-04→05, commits `61a993a`→`b653688`; reconstructed from commits)
|
||||
- Measured a canonical real app per README route framework. Import/aggregator frameworks made reachable: Python absolute `import a.b.c` (61a993a), source-aware `from pkg import submodule` for FastAPI aggregators (2835623), Django `include('app.urls')` (58dc463), renamed default-import→module file for Express controllers (2a0b6e0) → Express 100%, FastAPI 98%, Flask/requests 100%, NestJS 96.8%, Gin 96.5%.
|
||||
- Convention/reflection (ASP.NET/Blazor): built a Razor/Blazor `.cshtml`/`.razor` markup parser (59b8de2), delegated `@code`/`@{}` to the C# extractor (90c5f39), added C# namespace extraction (dc7d033) + Razor `@using` disambiguation (9e5a951), resolved chained `a.b.Method()` incl. C# extension methods (4c14413) → eShopOnWeb 59.3%→83.9%, Django→74.1%. Concluded these hit a genuine static-analysis ceiling (markup/reflection); did NOT game the metric to fake 95%.
|
||||
### Session — this update (2026-06-05)
|
||||
- **User:** loaded the wrong handoff first (explore-overhaul-bench), asked about per-language coverage %, then switched to this branch to work on impact coverage.
|
||||
- **Claude:** sourced the coverage table from the `impact-coverage-findings` memory (corroborated by this handoff's own table); found this handoff committed on the branch + loaded it; flagged the branch had moved 11 commits past the handoff's stated tip; **refreshed this handoff to current state** (31 commits, route-framework front, fork of next steps; test count re-verified 1178). Then deleted the other 9 handoff files at the user's request, leaving this as the sole handoff.
|
||||
- **Outcome:** handoff is current + the only one left; awaiting the user's pick of fork (ship / route front / static-member pass).
|
||||
### Session — Spring + Axum (2026-06-05, commit a3f59fb)
|
||||
- **User:** "unblock spring by fixing the faircov samples bug" → then "yes" to measuring an unmeasured framework.
|
||||
- **Claude (Spring):** there was NO `samples`-dir exclusion in faircov (the handoff note was wrong); the real deflation was 5 `package-info.java` shells (namespace-only → realSyms≠0) sitting in the denominator as zeros. Fixed faircov to barrel-exclude `package-info.java`/`module-info.java`. Spring petclinic 65.2%→**83.3%** (15/18); 3 residual zeros all reflection/DI wiring (`@Configuration` + AOT hints — verified honest, not missed edges) → convention/reflection ceiling, ≈ASP.NET 83.9%.
|
||||
- **Claude (Axum):** measured realworld-axum-sqlx → 72.7%, audited zeros → found a REAL engine miss (not a metric issue): bare Rust `submodule::fn()` calls (`X::router()` router-assembly) resolved crate-relative only. Fixed `resolveRustModuleFile` to try self-relative FIRST, then crate fallback; + `main.rs`/`src/bin/*.rs` faircov entry exclusion. Axum → **100%** (10/10). Clean A/B (same faircov, engine ±fix): axum +2, ripgrep neutral 66/79, tokio +3 → 260/321; cross-family false edges 0; regression test fails without the fix; full suite **1179** green. Committed+pushed `a3f59fb`.
|
||||
- **Outcome:** Spring + Axum both measured; the import/aggregator → 95%+ thesis now holds in a 3rd language family (Rust) via a real engine fix, not a metric tweak. Remaining unmeasured: actix/Rocket (attribute-macro routing — likely convention/reflection shape), Vapor, React Router/SvelteKit/Vue-Nuxt, Drupal.
|
||||
### Session — Rust attribute-macro frameworks (2026-06-05, commit 7bb958b)
|
||||
- **User:** "rust frameworks to probe the ceiling" (actix + Rocket).
|
||||
- **Claude:** measured Rocket (TatriX/realworld-rust-rocket) 62.5% + actix (fairingrey/actix-realworld) 65.4%. Audited zeros → TWO findings. (1) Rocket's 3-segment `database::profiles::find()` calls weren't becoming edges — isolated via micro-tests to a RESOLVER PRE-FILTER bug (`hasAnyPossibleMatch` checked first-seg + the `b::c` remainder, never the leaf), NOT closures or extraction (both fine). Fixed (leaf-name check) → Rocket 68.8%, additive elsewhere, no control regression, false edges 0. Committed 7bb958b + regression test. (2) Remaining ceilings are GENUINE: Rocket's `routes![]` macro-mounted handlers (bridgeable via a Rocket extractor — deferred, framework-specific) + actix's actor message dispatch (`db.send(Msg)`→`impl Handler<Msg>`, a dynamic frontier like MediatR — not bridged).
|
||||
- **Outcome:** both measured; the "attribute-macro = ceiling" hypothesis was PARTLY right (genuine ceilings exist) but probing surfaced a real generalizable resolver bug that benefits all Rust `a::b::c()` calls. All Rust route frameworks done. Next options: Vapor / component-node frameworks / Drupal, or build the Rocket `routes![]` extractor (→~94%).
|
||||
### Session — Rocket routes![] extractor (2026-06-05, commit 6d214cd)
|
||||
- **User:** "build the Rocket routes! extractor".
|
||||
- **Claude:** probed the tree-sitter AST — `routes![a::b::h, …]` parses as `macro_invocation > token_tree` of FLAT raw tokens (`identifier` / `::` / `,`), so the handler paths were invisible to the call walker. Built `extractRustRouteMacro` (tree-sitter.ts, called from `visitFunctionBody`): for a `routes`/`catchers` macro, walk the token-tree, group `identifier` runs between commas into `::`-joined paths, emit a `references` edge each; the existing Rust path resolver (self-relative + multi-segment fixes already shipped) links them to the handler fns. Rocket 68.8%→**93.8%** (all 4 `routes/*.rs` covered; only lib.rs root left). No control regression, false edges 0, nodes unchanged (edges only), regression test fails without it. Committed 6d214cd.
|
||||
- **Outcome:** Rocket effectively at ceiling. Handler names are explicit in source → precise static extraction, not a heuristic. Remaining route-front: Vapor (Swift), component-node frameworks (SvelteKit/React Router/Vue-Nuxt), Drupal — or ship the 34-commit branch.
|
||||
### Session — Vapor (2026-06-06, commit bb7659e)
|
||||
- **User:** "vapor next".
|
||||
- **Claude:** measured vapor-til (canonical Vapor tutorial app, kodecocodes) 85.7% + penny-bot (real prod, 166 files) 73.2%. vapor-til zeros: Package.swift (SPM build manifest → faircov excl), Public/cookies.js (static client asset → faircov excl), and AcronymCategoryPivot.swift = a REAL miss. Audited: the pivot model is referenced ONLY via `@Siblings(through: AcronymCategoryPivot.self, …)` in Acronym/Category — a metatype arg inside a property-wrapper `attribute` whose ARGS were never walked (only the wrapper type `Siblings` + the property's declared type were captured). Fix: route the property's attribute-arg subtree through `extractStaticMemberRef` (already handles Swift `Type.self` navigation, self-filters to capitalized receivers, skips `\.$keypath`). vapor-til 94.7%→**100%**. Clean A/B: vapor-til +1, Alamofire neutral (44/50); false edges 0; suite 1182. Committed bb7659e + regression test. penny-bot 73.2% characterized (serverless/DI: Lambda entries + exports barrels + DI/serialization/extension frontiers) — NOT chased (not the canonical pattern).
|
||||
- **Outcome:** Vapor DONE; import/aggregator confirmed in a 4th language family (Swift) via a real Fluent fix. Remaining route-front: component-node frameworks (SvelteKit/React Router/Vue-Nuxt), Drupal — or ship the 35-commit branch.
|
||||
### Session — SvelteKit (2026-06-06, no commit — characterization)
|
||||
- **User:** "component-node framework next".
|
||||
- **Claude:** measured sveltejs/realworld (official SvelteKit app) → 35.9% raw (14/39). Audited: ALL 25 zeros are SvelteKit file-convention files (`+page`/`+layout`/`+error`/`+server`/`hooks`) — loaded by the framework BY PATH, no in-repo caller. Probed + CONFIRMED the engine already resolves every `.svelte` component import (ArticlePreview←ArticleList, Nav←+layout, ListErrors←4 pages; imports+references+calls all resolve) — NO component is a zero, so NO engine miss (first framework needing none). Added SvelteKit convention-entry patterns to faircov (methodology-aligned, like route handlers) → 14/14 = **100% fair** of the coverable core. Confirmed the one framework-mediated gap: `+page.server.js` `load` → `+page.svelte` `data` is NOT statically linked (runtime `data` prop).
|
||||
- **Outcome:** the component-node shape is "file-convention entries (the majority) + a fully-covered component/lib core" — the engine handles it as-is. No commit (nothing to fix). OFFERED a SvelteKit convention synthesizer (sibling `+page.server.js`↔`+page.svelte` link, like Rocket routes!) for impact granularity — not built. Branch stays 35 commits. Next: React Router (config-based variant — likely MORE covered) / Nuxt (file-convention, SvelteKit-like) / Drupal, or ship.
|
||||
### Session — SvelteKit load→page synthesizer (2026-06-06, commit 3ea03e5)
|
||||
- **User:** "lets fix svelte with the synthesizer, then we can move onto react router".
|
||||
- **Claude:** probed the graph: `+page.svelte`→`component` node, `+page.server.js`→`load`/`actions` function nodes, pages pair with sibling loaders in the SAME dir. Built `svelteKitLoadEdges` (callback-synthesizer.ts, wired into `synthesizeCallbackEdges` after `vueTemplateEdges`): for each `+page`/`+layout.svelte`, link its `component` → the `load`/`actions` in `<dir>/<prefix>.server.{ts,js}` / `<dir>/<prefix>.{ts,js}` (same dir only), direction page→load so `getImpactRadius(load)` surfaces the page; provenance heuristic. Verified on sveltejs/realworld: 19 links, **0 cross-dir mislinks**, false edges 0, nodes unchanged; `getImpactRadius(login load)` now returns login/+page.svelte. Committed 3ea03e5 + regression test (incl. a does-not-cross-routes guard). Suite 1183.
|
||||
- **Outcome:** SvelteKit fully done (component coverage 100% + the loader→page impact link). Next per the user: React Router (the config-based component-node variant).
|
||||
### Session — React Router (2026-06-06, no commit — characterization)
|
||||
- **User:** "...then we can move onto react router" (continued from the svelte synthesizer).
|
||||
- **Claude:** measured gothinkster react-redux-realworld (38 files, react-router v4/v5) → **100%** (29/29), false edges 0. The CONFIG-based component-node sub-shape: route components are NAMED in source (`<Route path='/login' component={Login}/>`), so App.js imports+references each → all resolve. Verified Login/Editor/Profile covered BY App.js's route config. Only `index.js` (CRA entry) + 8 anonymous-default reducers (0 named symbols → barrels) excluded. NO engine miss. (Modern v6/v7 `element={<X/>}` + data-router `{element, loader}` are also source-referenced → same conclusion.)
|
||||
- **Outcome:** React Router DONE. Component-node category now fully characterized — two sub-shapes (config-based React Router = 100% raw; file-convention SvelteKit = fair-100% + synthesizer), both handled by the engine. No commit (nothing to fix). Remaining: Nuxt (Vue, file-convention — would mirror SvelteKit), Drupal (PHP), or ship the 36-commit branch.
|
||||
### Session — Nuxt (2026-06-06, commit 1dec765)
|
||||
- **User:** "nuxt next".
|
||||
- **Claude:** measured nuxt/movies (official Nuxt 3 demo, 42 .vue + 37 ts) → 47.6% raw. Audited zeros: pages/app.vue/error.vue/server = file-convention entries (→ faircov Nuxt exclusions); NESTED components (`components/media/*`, `components/carousel/*`) = a REAL miss. Root cause: Nuxt auto-imports a nested component by DIR-PREFIX (`media/Card.vue` → `<MediaCard/>`), but the node is named `Card`; the synth `vueTemplateEdges` only matched KEBAB tags, and the extractor's PascalCase ref name-matched `MediaCard`→nothing. Fixed: added a PascalCase-tag regex to vueTemplateEdges + a `nuxtComponentName` map (`media/Card.vue`→`MediaCard`, Nuxt dir de-dup). nuxt/movies 47.6%→**93.5%** (+9 nested components). Vue-only (gated to .vue), false edges 0, nodes unchanged, suite 1183, regression test. Committed 1dec765. 2 residual zeros are genuine frontiers (unused `<CarouselItems>`, unimported constants). composables (`useTmdb()`) already worked.
|
||||
- **Outcome:** Nuxt DONE. The two file-convention component frameworks DIFFER: SvelteKit's imports resolve as-is (no fix), Nuxt's dir-prefix auto-import needed a fix. Only **Drupal** (PHP) left unmeasured — or ship the 37-commit branch.
|
||||
### Session — Drupal (2026-06-06, no commit — characterization, finishes the sweep)
|
||||
- **User:** "drupal to finish the sweep".
|
||||
- **Claude:** measured the `token` contrib module (58 PHP, modern Drupal 11) → 57.1% raw. Audited: `drupal.ts` ALREADY resolves `*.routing.yml`→controller (TokenTreeController covered via the route ref) and PHP class refs work; the zeros are ALL genuine reflection/convention — JS libraries, DI services (`TreeBuilder` = `@token.tree_builder` in services.yml, `TokenFieldRender`), a dynamically-routed controller (`TokenDevelController` via `RouteSubscriber`), plugins (`Element/`, `Plugin/Derivative/`), Drush commands, `*ServiceProvider`, `.install` hooks, field-plugin `MenuLinkFieldItemList`. NO simple fixable miss (verified TokenDevelController isn't in any routing.yml — genuinely dynamic). Added faircov Drupal convention-entry + JS-library exclusions → **78.9% fair** (4 residual zeros = DI services + dynamic route + field plugin). Token uses modern `#[Hook('...')]` attribute OOP hooks (a drupal.ts TODO). NO commit (the levers — services.yml DI / plugin annotations / `#[Hook]` — are large per-mechanism features, same don't-chase-95% stance as ASP.NET/Django/Spring).
|
||||
- **Outcome:** **THE SWEEP IS COMPLETE.** Every README framework measured: import/aggregator + config-based-component + file-convention-component reach ceiling (95–100%) via 6 engine fixes; convention/reflection/actor frameworks (ASP.NET/Spring/Drupal/Django/actix) sit at a genuine sub-95% reflection ceiling. Branch 37 commits, ready to ship. Next: open the PR, or the static-member-pass TS/JS/Python lever.
|
||||
### Session — static-member pass for TS/JS/Python: measured & rejected (2026-06-06, commit e7b86df)
|
||||
- **User:** "extending the static-member/const value-read pass to TS/JS/Python."
|
||||
- **Claude:** implemented (added `member_expression`/`attribute` to `MEMBER_ACCESS_TYPES` + ts/js/python to `STATIC_MEMBER_LANGS`), then ran a clean A/B (same faircov, engine ±change) on excalidraw (TS/React, the retrieval-perf canary) + requests (Python). Result: **0 coverage gain** on BOTH (excalidraw 94.3%→94.3%, requests 100%→100%) because the import edge already covers every `Type.MEMBER` read (in import-based languages you must import the type first — the static read is redundant), while adding **+1813 edges / +2448 `references` on excalidraw** (all pointing at already-covered types — pure retrieval-graph noise). REVERTED the functional change; enhanced the `STATIC_MEMBER_LANGS` comment with the measured finding so it isn't re-attempted. Committed e7b86df (doc-only).
|
||||
- **Outcome:** the standing "one big cross-language lever" is CLOSED — the deferral was correct, now proven with data. The branch is 38 commits; everything actionable is done. **Recommended next: open the PR to main.**
|
||||
### Session — the 4 niche README languages (2026-06-06, commits 4155609 / 2f57119 / 2f30a3b)
|
||||
- **User:** confirmed only 4 README "Full support" languages lacked coverage validation (Lua/Luau/Liquid/Pascal); "knock out those 4 to close it out."
|
||||
- **Claude:** measured + fixed each. **Lua** (telescope) 31.6% + **Luau** (Fusion) 12.5%: NO `require()` module resolver existed — added `resolveLuaRequire` (dotted `a.b.c`→file + instance-path leaf, path-suffix match w/ same-dir preference; confidence 0.9 so it beats the import-node self-match) → Lua **84.2%**, Luau **92.2%** (4155609). **Liquid** (Dawn) 39.1%: snippet `{% render %}` already worked, but Shopify OS 2.0 sections are referenced from JSON templates (`templates/*.json` `"type"`), which weren't even indexed — made Shopify JSON indexable (isSourceFile + detectLanguage) + a Liquid-extractor JSON branch emitting `sections/<type>.liquid` refs (incl. nested `templates/customers/`) → **73.8%** (2f57119). **Pascal** (PascalCoin) 73.0%: `uses`-resolution ALREADY worked; added a `.dfm`/`.fmx`↔`.pas` form-code-behind synthesizer → **75.7%** (2f30a3b). Each: gated to its language, false edges 0, regression test fails without the fix, suite green (1187).
|
||||
- **Outcome:** **FULL README PARITY** — all 22 "Full support" languages + all 14 frameworks now have coverage validation. Branch 41 commits, ready to ship. 3 new generalizable resolvers (Lua/Luau require, Shopify-JSON sections, Delphi form pairing). Only remaining work is optional reflection/DI levers + the PR.
|
||||
@@ -1,70 +0,0 @@
|
||||
---
|
||||
name: explore-flow-tool-adoption
|
||||
date: 2026-05-24 00:55
|
||||
project: codegraph
|
||||
branch: architectural-improvements
|
||||
summary: Investigated why codegraph's read savings don't convert to wall-clock; root cause is agent tool-CHOICE (under-uses trace). Shipped a chain of fixes; the breakthrough is "explore-surfaces-flow" — the first mechanism to show up in real agent runs by adapting the tool the agent already uses.
|
||||
---
|
||||
|
||||
# Handoff: codegraph retrieval — tool adoption & explore-surfaces-flow
|
||||
|
||||
## Resume here — read this first
|
||||
**Current state:** A long investigation into making agents answer flow questions faster with codegraph. 6 commits on `architectural-improvements` (all probe-validated, suite green 815). The breakthrough: **`codegraph_explore` now surfaces the execution flow** from the symbol-bag the agent already passes it (`PmsProductController getList PmsProductService list PmsProductServiceImpl` → leads output with `getList → service-interface → impl`, riding synth edges). It's the FIRST mechanism this whole arc to actually appear in real agent runs (spring-mall A/B: flow surfaced both runs, reads 2.0→1.5) — because it adapts the tool the agent USES instead of trying to make it use `trace`.
|
||||
|
||||
**Immediate next step:** The user is weighing how to push tool-USE quality next (their open question). Decide between: (a) **extend explore-flow to surface more reliably** (spring-halo's query didn't name a connected co-named chain → no flow), (b) accept we're at the model-behavior ceiling and **wrap up**, or (c) the user's ideas — better tool-description *examples* (≈ steering, low-leverage per the evidence) or a *query-builder tool* (adds a call + new-tool adoption problem). My read: keep ADAPTING THE USED TOOL (the only thing that's worked); examples/new-tools are the "change the agent" direction that failed all session.
|
||||
|
||||
> Suggested next message: "explore-flow only surfaced on 2 of 3 repos — dig into why spring-halo's explore query didn't produce a flow and make it surface more reliably" — OR — "we're at the model-behavior ceiling; let's stop and write the CHANGELOG/PR for this branch"
|
||||
|
||||
## Goal
|
||||
Make an AI agent answer **flow questions** ("how does X reach Y", request→handler→service, state→render) fast: ~0 Read/Grep, few codegraph calls, lower wall-clock. `codegraph_trace` is the fastest tool (1 call = the path), but the agent under-uses it. Ultimate target = trace's speed, however the agent gets there.
|
||||
|
||||
## Key findings (the through-line)
|
||||
- **The wall is agent tool-CHOICE, not the graph.** Matrix-wide, codegraph cuts reads −75% but wall-clock only −16% (`docs/benchmarks/codegraph-ab-matrix.md`). The floor is round-trips + the synthesis turn. The agent reliably calls `context`/`explore`, rarely `trace` (3/37 flow cells). Full analysis: `docs/benchmarks/call-sequence-analysis.md`.
|
||||
- **Steering does NOT move it** (arms B/F/G, 3 wording variants): an MCP `initialize` instruction / tool description can't match a CLI `--append-system-prompt`'s salience, and forcing trace where it doesn't connect regresses. Reverted.
|
||||
- **Sufficiency works** (committed): a self-sufficient `trace` (hop bodies + destination callees inlined) lets the unsteered agent stop — but only when it calls trace.
|
||||
- **THE breakthrough — adapt the tool the agent uses.** `explore`'s query is a precise symbol-bag spanning the flow, so `explore` finds the call path AMONG its named symbols and leads with it. First mechanism to surface in real runs + drop reads.
|
||||
- **What FAILED:** option 1 (context-surfaces-flow) — fuzzy DESCRIPTION can't disambiguate endpoints → confident WRONG-feature flow; reverted. trace multi-source-BFS over ambiguous names — same wrong-feature; reverted.
|
||||
|
||||
## Gotchas
|
||||
- **Co-naming disambiguation must match qualifiedName SEGMENTS, not substrings** (`buildFlowFromNamedSymbols` in `src/mcp/tools.ts`): `list` is a substring of `getList` → kept every getList. Split `qualifiedName` on `::`/`.` and match segments.
|
||||
- **BFS must cap consecutive UNNAMED hops at 1** — full-graph BFS wanders a god-function's fan-out (excalidraw `render()` → pointer handlers → mutateElement). ≤1 bridge crosses a missing intermediate without wandering.
|
||||
- **`getCallees` returns non-`calls` edges too** (references) — filter `c.edge.kind === 'calls'`.
|
||||
- **Resolver/synthesizer changes need a CLEAN reindex**: `rm -rf .codegraph && codegraph init -i` (the init edge count is contains-only — query the DB for the real count). The explore-flow change is query-time (no reindex).
|
||||
- **n=2 A/B is noisy** — report ranges/patterns, never conclude from one run. Foreground `sleep` is blocked → run A/B batches with `run_in_background`.
|
||||
- Java/Kotlin `qualifiedName` is `Class::method` (so `matchesSymbol` resolves `Class.method` qualified trace endpoints — the agent already passes these).
|
||||
|
||||
## How to test & validate
|
||||
- Probe flow surfacing (no agent): `node scripts/agent-eval/probe-explore.mjs <repo> "<SymbolA SymbolB SymbolC>"` → look for the `## Flow` section. `probe-trace.mjs <repo> <from> <to>` for trace.
|
||||
- Synthesizer: `sqlite3 <repo>/.codegraph/codegraph.db "select count(*) from edges where json_extract(metadata,'$.synthesizedBy')='interface-impl'"`; node count stable before/after reindex (synth adds edges only).
|
||||
- Agent A/B (the real test): `bash scripts/agent-eval/run-arms.sh <repo> "<Q>" I <run>` (arm I = body-trace build, no steering). Parse via the `cmp2.mjs`-style scripts in `/tmp`. Pass = flow surfaces (`flowShown=Y`) + reads ≤ baseline.
|
||||
- `npm test` (vitest, 815 pass); `__tests__/mcp-tool-allowlist.test.ts` covers the allowlist.
|
||||
|
||||
## Repo state
|
||||
- branch `architectural-improvements`, last commit `bafae81 feat(mcp): codegraph_explore surfaces the execution flow from its named symbols`.
|
||||
- uncommitted: clean (only untracked `.claude/handoffs/`).
|
||||
- 6 session commits: `eab5cf3` self-sufficient trace + `CODEGRAPH_MCP_TOOLS` allowlist · `a6183d7` research log + arms harness · `bde8c19` node/trace line numbers · `98baf41` Java/Kotlin interface→impl synthesizer · `6f3c468` playbook · `bafae81` explore-surfaces-flow.
|
||||
- NOT pushed/merged. No version bump. CHANGELOG `[Unreleased]` has all of it.
|
||||
|
||||
## Open threads / TODO
|
||||
- [ ] **User's open question** (answer in the next turn): better tool-description *examples* vs a *query-builder tool* vs keep adapting the used tool. Evidence favors the last.
|
||||
- [x] explore-flow reliability: now resolves QUALIFIED tokens (`Class.method`) — the agent's most precise input was being dropped by the file-ext strip (`2765c3c`). spring-halo's publish flow stays absent on purpose — it's **reactive/reconciler dispatch** (`publishPost` calls `ReactiveExtensionClient.get`/`awaitPostPublished`, not `PostService.publish`), so there's no static call chain. That's the next COVERAGE frontier (reactive runtimes — like MediatR, Vue Proxy), not an explore-flow bug.
|
||||
- [ ] Ship-prep for the whole branch (this arc + the earlier framework sweep): CHANGELOG version block + `package.json` bump + PR to main. Releases go through `.github/workflows/release.yml` only — do NOT `npm publish`.
|
||||
- [ ] Frontiers: MediatR (`_mediator.Send`→Handle) and Vue/Compose reactive runtimes are still unbridged dynamic dispatch.
|
||||
|
||||
## Recent transcript (oldest → newest)
|
||||
### Turn — "improve the A/B matrix; trace works, reads near 0 — what else?"
|
||||
- Diagnosed: reads at floor, wall-clock floor = round-trips + synthesis. Built `seq-matrix.mjs`; found trace adoption 3/37.
|
||||
### Turn — "do explore/context/trace compete? one tool?"
|
||||
- Ablation arms A–E (`run-arms.sh`/`arms-F.sh` + `CODEGRAPH_MCP_TOOLS` allowlist). explore = 68% of payload, load-bearing; trace path-scoped but under-adopted; trace alone insufficient.
|
||||
### Turn — "prototype body-inlining trace + A/B"
|
||||
- Arm F: self-sufficient trace wins WITH append-prompt steering. But steering isn't a shippable channel.
|
||||
### Turn — "port the steering + re-run"
|
||||
- Arms G (3 variants) all regressed vs baseline; arm H (body-trace, no steer) ≈ baseline. Steering reverted; body-trace + line-numbers + allowlist committed.
|
||||
### Turn — "tee up connectivity (Spring interface-DI)"
|
||||
- Built `interfaceOverrideEdges` (Java/Kotlin interface→impl, overload-aware). Probe: 3-hop trace connects. But A/B null — agent never called trace. Committed (probe-validated, adoption-gated).
|
||||
### Turn — "make context surface the flow (option 1)"
|
||||
- Failed: fuzzy query → wrong-feature flows. Reverted.
|
||||
### Turn — "change explore to do trace in the backend"
|
||||
- WIN: explore's query is a precise symbol-bag. `buildFlowFromNamedSymbols` (co-naming segment match + ≤1 bridge). Probe perfect (Spring + excalidraw full chains); A/B: flow surfaces + modest read drop. Committed `bafae81`.
|
||||
### Turn — "update memory + handoff; what about better examples / a query-builder tool?"
|
||||
- This handoff + memory update. Strategic answer pending (adapt-the-tool > change-the-agent).
|
||||
@@ -1,80 +0,0 @@
|
||||
---
|
||||
name: explore-overhaul-2026-06-01
|
||||
date: 2026-06-01 19:50
|
||||
project: codegraph
|
||||
branch: main
|
||||
summary: Made codegraph_explore the sole primary tool (removed context + trace), added graph-connectivity ranking + 100K budget + full method bodies — then an agent-eval revealed the budget BACKFIRES and the real lever is COVERAGE (Zustand store methods aren't indexed).
|
||||
---
|
||||
|
||||
# Handoff: codegraph_explore overhaul — explore as the one tool, and the coverage pivot
|
||||
|
||||
## Resume here — read this first
|
||||
**Current state:** Big uncommitted working tree on `main`. `codegraph_context` and `codegraph_trace` tools are fully removed; `codegraph_explore` is the sole primary, now with graph-connectivity (RWR) ranking, a flat **100K** output budget, full method bodies, whole-central-file, and an always-on blast-radius section. A fresh-daemon agent-eval on the real repo (`~/Downloads/amniservices-mobile-app`) just proved two things: (1) the **100K budget BACKFIRES** — a broad explore hit **67K chars and overflowed the agent's per-tool token cap**, forcing it to Read; (2) the **real cause of the agent's reads is a COVERAGE gap**, not ranking/budget — Zustand store methods (`fetchUser`/`switchOrganization` inside `create((set,get)=>({...}))`) aren't indexed as nodes, and callers **destructure** them (`const {fetchUser}=useOrgUser.getState()`), so `codegraph_node`/`codegraph_callers` return "not found."
|
||||
**Immediate next step:** Revert the 100K budget (it overflows) to ~28–35K, then build the Zustand coverage fix (extract store-literal methods as nodes + resolve destructured `getState()` calls). That's what actually deletes the reads.
|
||||
|
||||
> Suggested next message: "Revert the explore budget in getExploreOutputBudget (tools.ts) from 100K back to ~30K — the 67K response overflowed the agent's tool cap. Then build the Zustand coverage fix: extract methods inside `create((set,get)=>({...}))` as nodes, and resolve destructured store calls like `const {fetchUser}=useOrgUser.getState()`. Then kill the AmniSphere daemon and re-run the agent eval."
|
||||
|
||||
## Goal
|
||||
Make `codegraph_explore` good enough to be a **Read-replacement** — one (maybe two) calls answer a structural/flow question with ~0 Read/Grep, for smart AND dumb models. Metric is wall-clock + tool-call count + Read count (NOT token cost). The user's golden era: one tool (`explore`), reflexively used, zero Reads.
|
||||
|
||||
## Key findings
|
||||
- **The agent's reads are a COVERAGE gap, not ranking/budget.** Agent's own words (diagnostic eval): Zustand store actions inside the `create((set,get)=>({...}))` literal "aren't individually indexed," so `codegraph_node fetchUser` / `codegraph_callers fetchUser` → **"not found"**; callers **destructure** off `useOrgUser.getState()` so even grep needed `\bfetchUser\b`. Component-body control flow (`handleLogin`, `AppInitializer` in `src/app/index.tsx`, `src/components/providers/index.tsx`) isn't a node either.
|
||||
- **The 100K budget backfires.** A broad explore returned ~67K chars and "overflowed the token cap" → agent Read instead. Big responses are *worse*. `getExploreOutputBudget` (tools.ts ~line 140) is now a flat 100K — revert toward ~28–35K (size to the agent's per-tool output limit).
|
||||
- **Adoption is EXCELLENT — the agent WANTS codegraph.** In the fresh eval it made **16 codegraph calls** vs 5 Reads. So the problem is never "agent won't use it"; it's "the symbols aren't in the graph."
|
||||
- **Graph-connectivity ranking works in isolation but didn't address the real cause.** `computeGraphRelevance` (tools.ts, before `handleExplore`) is RWR/personalized-PageRank from the matched seeds; probe shows it ranks `org-user.storage.ts` #1 and returns it whole. But it doesn't cleanly drop noise (LensSwitcher.swift matched "switch") because real codebases share infra + generic terms — **neither graph nor text alone separates; needs IDF×graph fusion**, a tuning long tail. Park it until coverage is fixed.
|
||||
- **`context` + `trace` tools fully removed** (def + dispatch + handlers + CLI `context` command + permissions + server-instructions + tests). The shared engine `findRelevantContext` stays (explore runs on it). `synthEdgeNote` kept (shared); `handleTrace`/`sourceLineAt`/`sourceRangeAt`/`maybeInlineFlowTrace`/`handleContext`/`looksLikeFeatureRequest`/`formatTaskContext` deleted.
|
||||
- **Read-gate PreToolUse hook was built then REMOVED** (user: "ideally zero hooks"). Deleted `src/hooks/`, `src/mcp/session-consult.ts`, the `mcp-read-gate` CLI cmd, installer wiring (`InstallOptions.readGate`, claude.ts helpers), and the marker security tests. Had an unverified `CLAUDE_SESSION_ID`==hook-`session_id` assumption.
|
||||
- **Precision fix landed earlier (keeper):** `isDistinctiveIdentifier` (query-utils.ts) gates the exact-name bonus in `findRelevantContext` Step 5a so a common word ("flat") can't hijack ranking (was surfacing a python `FLAT` constant). Lives in the shared engine → benefits explore.
|
||||
- **Blast-radius section added to explore** (`buildBlastRadiusSection`, tools.ts): per entry symbol, who-depends-on-it + covering test files, locations only. Always-on, compact. (2 tests in `__tests__/explore-blast-radius.test.ts`.)
|
||||
|
||||
## Gotchas
|
||||
- **STALE-DAEMON FOOT-GUN (cost us hours).** `codegraph serve --mcp` connects to a per-repo daemon (`<repo>/.codegraph/daemon.sock`, 5-min idle timeout) that holds the loaded code. **A `npm run build` does NOT take effect until you kill the daemon.** Every agent-eval before the kill was testing STALE code (agent got 2277 chars where a fresh in-process probe got 54K). **Before ANY agent eval:** `pkill -f "serve --mcp"; rm -f <repo>/.codegraph/daemon.sock`. Worth fixing in the product (a rebuild should invalidate the daemon).
|
||||
- **probe ≠ agent.** `probe-explore.mjs` loads `dist/` in-process (always current code); the agent uses the daemon (can be stale). Don't trust a probe result as "what the agent sees" unless the daemon was just killed.
|
||||
- **Validating with a favorable query lies.** My probe query (`"org user storage…"`) returned the whole central file; the agent's near-identical query behaved totally differently. Use the agent's EXACT query, on a fresh daemon.
|
||||
- **n=1 variance is large** — never conclude from one agent run (CLAUDE.md). The "4 vs 5 reads" between runs is noise.
|
||||
- **Budget-table repos (excalidraw/django/etc.) NOT validated** — they're not on this machine. The ranking/budget changes could regress them; the CLAUDE.md "do-not-regress explore budget" table is now obsolete (flat 100K) and needs reconciling.
|
||||
- All work is **uncommitted on `main`** — branch before committing (PR policy: main is REVIEW_REQUIRED).
|
||||
|
||||
## How to test & validate
|
||||
- Build: `npm run build` (must exit 0).
|
||||
- Cheap probe (current code, NOT what a stale daemon serves): `node scripts/agent-eval/probe-explore.mjs /Users/colby/Downloads/amniservices-mobile-app "<query>"`.
|
||||
- Agent A/B (real metric, ~$2, KILL DAEMON FIRST): `pkill -f "serve --mcp"; rm -f /Users/colby/Downloads/amniservices-mobile-app/.codegraph/daemon.sock; CG_BIN=$(pwd)/dist/bin/codegraph.js AGENT_EVAL_OUT=/tmp/agent-eval-amni bash scripts/agent-eval/run-agent.sh /Users/colby/Downloads/amniservices-mobile-app <label> "<prompt>"` → parse `/tmp/agent-eval-amni/run-<label>.jsonl` for tool order + Read count.
|
||||
- Diagnostic prompt that worked: append "for EACH Read/Grep note WHY codegraph wasn't enough; end with '## Why I read'." The agent's self-report is the best diagnostic.
|
||||
- Affected unit tests (NOT the full suite — user is cost-conscious): `npx vitest run __tests__/{context-ranking,explore-blast-radius,context,mcp-tool-allowlist,security,worktree-detection,installer-targets}.test.ts __tests__/integration/mcp-input-limits.test.ts`.
|
||||
- Pass bar: a flow question reaches ~0 Read within the explore-call budget, faster than without-codegraph, no regression on a control repo.
|
||||
|
||||
## Repo state
|
||||
- branch `main`, last commit `8629f7a docs(changelog): promote [Unreleased] into [0.9.8]`
|
||||
- uncommitted (all this session, none committed): `M src/mcp/tools.ts` (the big one — explore ranking/RWR/budget, context+trace removal, blast radius), `M src/context/index.ts` (precision fix), `?? src/context/markers.ts` (LOW_CONFIDENCE_MARKER leaf), `M src/search/query-utils.ts` (isDistinctiveIdentifier), `M src/mcp/server-instructions.ts`, `M src/installer/targets/shared.ts` (permissions), `M src/bin/codegraph.ts` (CLI context/trace removed), `M src/types.ts`, `M CHANGELOG.md`, `?? __tests__/context-ranking.test.ts`, `?? __tests__/explore-blast-radius.test.ts`, `M __tests__/{security,worktree-detection,mcp-tool-allowlist}.test.ts`, `M __tests__/integration/mcp-input-limits.test.ts`. (read-gate hook + session-consult.ts were created then deleted → no trace.)
|
||||
|
||||
## Open threads / TODO
|
||||
- [ ] **Revert the 100K budget** in `getExploreOutputBudget` (tools.ts ~140) to ~28–35K — it overflows the agent tool cap at 67K.
|
||||
- [ ] **Reconsider "whole central file always"** (tools.ts whole-file rule + `centralFiles`) — a 791-line whole file is what overflowed. Prefer the relevant methods full, not whole-huge-file.
|
||||
- [ ] **Build the Zustand coverage fix (the real lever):** (a) extract methods defined inside `create((set,get)=>({...}))` as nodes (extraction); (b) resolve destructured store calls `const {fetchUser}=useOrgUser.getState()` (reference resolution). Then re-eval.
|
||||
- [ ] Decide: keep `computeGraphRelevance` (graph ranking) or park it — it didn't address the coverage cause. Probe-validated, not agent-validated.
|
||||
- [ ] Re-run the agent eval (FRESH DAEMON) after the coverage fix; confirm reads of `org-user.storage.ts`/components drop.
|
||||
- [ ] Reconcile CHANGELOG `[Unreleased]` with the final direction (explore primary; context+trace removed; budget/ranking may change).
|
||||
- [ ] Validate budget-table repos (excalidraw/django) before merging — not on this machine.
|
||||
- [ ] Run the FULL test suite once (only ran affected files this session).
|
||||
- [ ] Branch off `main` before committing this work.
|
||||
|
||||
## Recent transcript (oldest → newest)
|
||||
### Turn — "do it" (build graph ranking)
|
||||
- **User:** Build graph-connectivity ranking (RWR/PageRank fused with bm25) as explore's relevance core; keep the output wins; fix the whole-file path.
|
||||
- **Claude:** `Edit(tools.ts)` → added `computeGraphRelevance` (RWR, undirected, α=0.25, 25 iters); rewired file ranking/central/gate to graph-primary + text-secondary. `probe` → `org-user.storage.ts` #1 + whole (796 lines), but LensSwitcher.swift/capture noise still present.
|
||||
- **Outcome:** RWR surfaces the central file but doesn't cleanly drop noise; reported the honest limit (needs IDF×graph fusion).
|
||||
|
||||
### Turn — "how does amniservices do now?"
|
||||
- **User:** Run the agent eval on the real repo with everything in place.
|
||||
- **Claude:** `run-agent.sh (amni-with-q2)` → codegraph 5 / Read 5 / Grep 3; agent Read `org-user.storage.ts` anyway. `parse` → agent's first explore returned only **2277 chars** while `probe` returned 54K for the same query.
|
||||
- **Outcome:** 2277-vs-54K discrepancy → suspected the agent wasn't running my build.
|
||||
|
||||
### Turn — "do the test again, ask the agent WHY it read"
|
||||
- **User:** Re-run, but make the agent explain each Read / why codegraph wasn't enough.
|
||||
- **Claude:** `ps` → found stale daemon `pid 8947` on AmniSphere socket + pile of lingering `serve --mcp`. `pkill -f "serve --mcp"; rm daemon.sock` → fresh. `run-agent.sh (amni-diag-q3)` with the "## Why I read" prompt, fresh daemon.
|
||||
- **Outcome:** First valid run on my build. codegraph **16** / Read 5 / Grep 3. Agent's "Why I read": Zustand store methods not indexed (`node`/`callers` → not found), callers destructure off `getState()`, broad explore **overflowed at 67K**. → real cause = COVERAGE; budget BACKFIRES. Pivot the priorities.
|
||||
|
||||
### Turn — /handoff save (this)
|
||||
- **User:** `/handoff save`.
|
||||
- **Claude:** Wrote this file.
|
||||
- **Outcome:** Checkpoint captured; next edge = budget revert + Zustand coverage fix.
|
||||
@@ -1,73 +0,0 @@
|
||||
---
|
||||
name: explore-overhaul-bench-2026-06-02
|
||||
date: 2026-06-02 06:30
|
||||
project: codegraph
|
||||
branch: feat/explore-overhaul-store-coverage
|
||||
summary: Finished the explore-overhaul arc (explore as sole primary + store coverage + overload disambiguation + method-atomic render + node file/line selector + explore reshaped to native-read windows) and validated it — all 7 README repos hit 0 Read/0 Grep at effort=high; only the README benchmark write-up remains.
|
||||
---
|
||||
|
||||
# Handoff: explore-overhaul arc — validated 0-reads across all 7 README repos; README write-up is the last step
|
||||
|
||||
## Resume here — read this first
|
||||
**Current state:** All code is committed + pushed on `feat/explore-overhaul-store-coverage` (4 commits, working tree clean). The why-Read agent sweep is DONE: **all 7 README repos × 4 runs = 28/28 runs hit 0 Read / 0 Grep on `--effort high`**, every run "codegraph was sufficient." WITH-`high` medians are captured (~59% fewer tool calls · 51% fewer tokens · ~15% cheaper · 0 reads vs the existing README WITHOUT) — the earlier cost REGRESSION (-3%) is recovered. The only open item is **updating the README benchmark section**, which is blocked on one methodology decision.
|
||||
**Immediate next step:** Decide how to publish: (A) do a CLEAN both-arms run on `effort=high` with the PLAIN prompt (no why-Read) for an apples-to-apples table, or (B) write the WITH-`high` deltas in against the existing WITHOUT with a cross-effort caveat. Then edit `README.md` (benchmark table + per-repo breakdowns + average line + methodology date) and open the PR.
|
||||
|
||||
> Suggested next message: "Do the clean both-arms run on effort=high with the plain prompt for all 7 repos, then update the README benchmark table + per-repo breakdowns from those medians and open the PR."
|
||||
|
||||
## Goal
|
||||
Make `codegraph_explore` a true Read-replacement — flow/architecture questions answered with ~0 Read/Grep — then re-validate the README benchmark on the current build and update its numbers. Definition of done: README benchmark reflects the current build with defensible (same-effort) numbers; branch merged via PR.
|
||||
|
||||
## Key findings
|
||||
- **The arc (all shipped on the branch):** explore is the SOLE primary tool (`codegraph_context` + `codegraph_trace` removed in the prior session, this branch); store-action **coverage** (object-literal method extraction — a GENERAL AST rule in `tree-sitter.ts` `extractVariable`/`extractObjectLiteralFunctions`/`findInitializerReturnedObject`, covers Zustand/Redux/Pinia, not a per-lib hack); graph-ranking **gate fix** (a named/≥2-term file is never pruned); **`node` all-overloads + `file`/`line` selector**; **method-atomic render** (never half a method — drop whole methods/files); **explore reshape** to native-read windows.
|
||||
- **Native-read ground truth (from the WITHOUT transcripts):** the agent natively reads **~6–9 files as ~100-line windows** (77% ranged, median 100 lines, 51–250 dominant), located by `func X(` signature greps. That's the unit explore now mimics.
|
||||
- **Explore reshape (commit 50401a6, the latest mechanism):** `getExploreOutputBudget` caps EVERY tier at **~24K** (was 28/35/38K) + absolute **25K** hard ceiling (was 1.5×-of-budget) — because a bigger response gets **externalized** by the host to a file the agent Reads back (a 35K vscode explore did exactly that) AND costs cache-writes. Repo size scales the CALL budget, not the response. Per-file = one ~150–250 line window: per-symbol `bodyCap` 2×→1.5× and the spine is windowed too (so tokio's big-spine `worker.rs` doesn't starve `harness.rs`'s `poll`); central whole-file 4×→1.5× / 400→280 lines. Explore's named-symbol injection now uses **`cg.getNodesByName`** (direct index, not FTS) so a 50+-overload name (`poll`) surfaces the wanted def (`Harness::poll`) for the PascalCase-type-token bias to pick.
|
||||
- **`node` file/line selector (commit 5bf6ad8):** `codegraph_node` takes optional `file`/`line` to pin an overload (the `file:line` a trail showed). `findSymbolMatches` (replaced `findSymbol`) enumerates ALL overloads via `cg.getNodesByName` (new passthrough `index.ts` → QueryBuilder), then file/line filters. The agent USES it in runs (`run file:worker.rs line:508`, `poll file:harness.rs`).
|
||||
- **Cost regression was REAL, now recovered.** The pre-reshape n=4 benchmark (on `max` effort, bloated 35-42K explores) was **−3% cost avg** (vscode −52%) and reads were **NOT 0** (vscode 6,4,0,7; tokio 3,4,2,2) — which corrected my earlier n=2 "0 reads everywhere" optimism. The reshape (≤25K, no externalization) + 0 reads flipped cost back to **~15% cheaper**.
|
||||
|
||||
## Gotchas
|
||||
- **STALE-DAEMON foot-gun:** before ANY agent eval, `pkill -f "serve --mcp"; rm -f <repo>/.codegraph/daemon.sock` so it serves the current `dist/`. `bench-why-repo.sh` does this per-run. A `npm run build` does NOT take effect until the daemon is killed.
|
||||
- **Mac SLEEP corrupts long runs:** the first overnight re-bench (5h on `max`) was sleep-corrupted — the Mac napped 16–42 min BETWEEN runs (~3h of the 5h was paused), inflating wall-clock for the later repos. **Always wrap long runs in `caffeinate -dimsu`.** Cost/tokens/reads are sleep-INDEPENDENT (billed API totals), so the cost regression was real (confirmed on vscode which ran fully awake before any sleep); only TIME is corrupted.
|
||||
- **`--effort` matters:** the user's Claude default is `max`, which is "too much." The eval is pinned to `--effort high` (levels: low/medium/high/xhigh/max). `bench-why-repo.sh` honors `EFFORT` (default `high`). The MAX-mode runs were discarded and redone on `high`.
|
||||
- **why-Read prompt biases reads down (Hawthorne) + adds <0.3% to WITH cost/tokens.** So the 28/28 0-read sweep proves codegraph is *sufficient* (it CAN answer with 0 reads); it slightly understates a natural run's reads. Keep it OUT of any published benchmark numbers (use plain prompt for the table).
|
||||
- **README methodology mismatch:** WITH numbers are `effort=high` + why-Read; the existing README WITHOUT is the user's OLD default effort + plain. Cross-effort → can't publish cleanly without same-effort both arms. The user does NOT want to re-run WITHOUT repeatedly, but the effort CHANGED, so a one-time WITHOUT-on-high is a new (justified) measurement.
|
||||
- **PR policy:** `main` is REVIEW_REQUIRED — work on the branch, open a PR, `gh pr merge --squash --admin` for self-review. Branch + push only so far; **PR not opened** (user asked branch+push).
|
||||
|
||||
## How to test & validate
|
||||
- Build: `npm run build` (exit 0). Full suite: `npx vitest run` → **1112 pass, 2 skip, 0 fail** (npm-shim network tests can flake offline — pre-existing).
|
||||
- Affected tests: `npx vitest run __tests__/{explore-output-budget,adaptive-explore-sizing,context-ranking,explore-blast-radius,symbol-lookup,pr19-improvements,object-literal-methods}.test.ts`.
|
||||
- Deterministic probe (current `dist/`, in-process — NOT the daemon): `node scripts/agent-eval/probe-explore.mjs /tmp/codegraph-corpus/<repo> "<query>"` → confirm ≤~25K chars + the flow files render. `node scripts/agent-eval/probe-node.mjs <repo> <symbol> code` (e.g. `poll file:harness.rs` via a small script).
|
||||
- Agent why-Read sweep (the real metric): `EFFORT=high caffeinate -dimsu bash scripts/agent-eval/bench-why-repo.sh /tmp/codegraph-corpus/<repo> "<readme query>" 4` → parse `/tmp/ab-why/<repo>/with*.jsonl` for `Read`/`Grep` tool_use + the trailing `## Why I read` section.
|
||||
- All 7 repos are cloned + indexed on the current build at `/tmp/codegraph-corpus/{vscode,excalidraw,django,tokio,okhttp,gin,alamofire}`. README queries are in `scripts/agent-eval/bench-readme.sh`.
|
||||
- **Pass bar:** flow question → ~0 Read at the explore-call budget, faster than WITHOUT, no control regression.
|
||||
|
||||
## Repo state
|
||||
- branch `feat/explore-overhaul-store-coverage`, last commit `9cf671a chore(agent-eval): add per-repo WITH-only why-Read benchmark harness`. Pushed, in sync with origin.
|
||||
- 4 commits: `22333c1` (explore-primary + store coverage + overload disambiguation + docs) · `5bf6ad8` (method-atomic render + node file/line selector) · `50401a6` (explore reshape: inline-cap + concentrated flow-windows + direct-index injection) · `9cf671a` (why-Read eval harness).
|
||||
- uncommitted: clean (only `.claude/handoffs/` scratch, intentionally not committed).
|
||||
- CHANGELOG `[Unreleased]` already updated for the whole arc. README + CLAUDE.md + docs/design banners updated to drop removed context/trace.
|
||||
|
||||
## Open threads / TODO
|
||||
- [ ] **README benchmark update** — decide (A) clean both-arms `effort=high` plain run vs (B) publish WITH-`high` deltas with caveat; then edit `README.md` (table + per-repo WITH/WITHOUT breakdowns + average + methodology date). WITH-`high` medians: vscode 4cg/742k/$0.70, excalidraw 6.5/834k/$0.74, django 3/509k/$0.59, tokio 6/921k/$0.77, okhttp 4.5/528k/$0.41, gin 4.5/573k/$0.48, alamofire 7.5/976k/$0.65 — all 0 reads.
|
||||
- [ ] **Open the PR** (`gh pr create`) once the README is updated.
|
||||
- [ ] Optional: delete dead dev probes `scripts/agent-eval/probe-{trace,context}.mjs` (their tools were removed; noted, not done).
|
||||
- [ ] Memory `project_codegraph_read_displacement` is updated through the reshape; add the 28/28-on-high + cost-recovery result if you want it durable.
|
||||
|
||||
## Recent transcript (oldest → newest)
|
||||
### Turn — "re-run the full benchmark to update README; ask why on reads"
|
||||
- **Claude:** cloned+indexed all 7 repos; launched `RUNS=4 bench-readme.sh` (both arms). → ran overnight.
|
||||
### Turn — "what's still running? 5h overnight"
|
||||
- **Claude:** bench was on the LAST repo (alamofire), ~90%. n=4 aggregate showed **cost −3% (regression)** + reads **NOT 0** (vscode 6-7, tokio 2-4) — corrected my n=2 "0 everywhere." Diagnosed reads from transcripts: explore was 35-42K, too spread, dropped named flow methods.
|
||||
### Turn — "is it the lock screen?"
|
||||
- **Claude:** Yes for TIME (Mac slept 16-42min between runs → 5h mostly paused) but NOT cost (vscode ran awake; cost is billed API). → recommend `caffeinate`.
|
||||
### Turn — "how do I stop Mac sleeping" → answered (Battery→Options / `caffeinate`).
|
||||
### Turn — "do the queries WITHOUT codegraph, see exactly what it Reads, mimic it"
|
||||
- **Claude:** characterized native reads (77% ranged, median 100 lines, signature greps) → spec: ~100-line flow-windows, ~6-9 files, ≤25K.
|
||||
### Turn — "do it" (the reshape)
|
||||
- **Claude:** `Edit(tools.ts)` → inline-cap 24K + concentrated windows + `getNodesByName` injection; updated budget tests; full suite green; probe vscode 35K→16K, tokio fits harness.rs. Committed `50401a6`, pushed.
|
||||
### Turn — "why AST not LSP" (×2) + "reply to hd_nvim (ast-grep author)"
|
||||
- **Claude:** explained the trade-off (breadth/speed/no-build/determinism/19-langs vs LSP type-precision-but-heavy); drafted the X reply (concede precision, reframe on goal/cost).
|
||||
### Turn — "re-run, ask Why on each WITH read, report per repo" → "no WITHOUT" → "MAX too much, use high, start over"
|
||||
- **Claude:** built `bench-why-repo.sh` (WITH-only, why-Read, `EFFORT` default high); ran all 7 on `high` one repo at a time, reporting each. **Result: 28/28 runs 0R/0G, all "sufficient."** Aggregated WITH-`high` medians → ~15% cheaper (regression recovered).
|
||||
### Turn — "commit and push this"
|
||||
- **Claude:** committed `bench-why-repo.sh` (`9cf671a`), pushed. → this handoff.
|
||||
- **Outcome:** Arc complete + validated; README write-up + PR are all that remain.
|
||||
@@ -1,70 +0,0 @@
|
||||
---
|
||||
name: explore-per-symbol-sizing
|
||||
date: 2026-05-29 23:20
|
||||
project: codegraph
|
||||
branch: main
|
||||
summary: Shipped per-symbol adaptive codegraph_explore sizing (PR #569) — show the answer (named methods + mechanism) in full, collapse redundant interchangeable siblings to signatures, keep named methods alive in non-sibling god-files; flipped Django/OkHttp from cost laggards to clear wins and lifted the README averages to 25%/57%/23%/62%.
|
||||
---
|
||||
|
||||
# Handoff: per-symbol adaptive codegraph_explore sizing (shipped)
|
||||
|
||||
## Resume here — read this first
|
||||
**Current state:** **DONE + shipped.** PR #569 squash-merged to `main` (`b026e64`); local is on `main`, `dist/` rebuilt, working tree clean. README benchmarks + averages + header, CHANGELOG, and `docs/design/adaptive-explore-sizing.md` all updated with the new full-7-repo sweep. The only loose end: **two squash-merged feature branches still linger** (`feat/adaptive-explore-sizing` from #564, `feat/explore-per-symbol-sizing` from #569) — local **and** remote — because squash-merges don't register as "merged" in git's ancestor sense.
|
||||
**Immediate next step:** Delete those two merged branches (local + remote), or pick up one of the Open-threads frontiers (Gin's small WITH-cost bump, alamofire DataRequest residual, or stabilizing per-repo benchmark numbers with median-of-8).
|
||||
|
||||
> Suggested next message: "Delete the merged branches feat/adaptive-explore-sizing and feat/explore-per-symbol-sizing — local and remote."
|
||||
|
||||
## Goal
|
||||
Make `codegraph_explore`'s cost a clear win on **every** README benchmark repo, especially the two laggards the README showed thinnest (Django 9% cheaper, OkHttp 4%). The optimization target per CLAUDE.md is **tool-calls/reads + latency** (NOT raw cost) — but the user explicitly wanted the cost margins up too. Definition of done = both laggards clearly cheaper with ~0 reads, no regression elsewhere, README refreshed, shipped. **Achieved.**
|
||||
|
||||
## Key findings
|
||||
- **The feature, in `src/mcp/tools.ts` (`handleExplore` + `buildFlowFromNamedSymbols`):** explore sizes output to the *answer*, not the file count. Builds on PR #564's gate (off-spine + polymorphic-sibling, with a named-callable *spare* + supertype-family *override*).
|
||||
- **PR #569 added four things** (all in `tools.ts`):
|
||||
1. **Uniqueness-aware spare** — `buildFlowFromNamedSymbols` now returns `uniqueNamedNodeIds` (callables whose token had ≤3 defs). The whole-file spare uses it, so `as_sql` (110 defs) no longer keeps every Compiler/Expression variant full; `getResponseWithInterceptorChain` (1 def) still spares RealCall.
|
||||
2. **Per-symbol focused view** — a collapsed family file renders FULL bodies for symbols with `prio()` < 99 (on-spine=0, unique-named=1, `fileDefinesSuper && named`=2), signatures for the rest. Bounded: `bodyCap = maxCharsPerFile*2`, `SIG_MAX = max(12, maxSymbolsInFileHeader*2)`. Header tag flips to `· focused (…)` when any body shown, else `· skeleton (…)`.
|
||||
3. **All-tier test-file exclusion** — removed the `budget.excludeLowValueFiles` gate on the `isLowValue` hard-exclude (was <500-file tiers only); guards (query-mentions-tests, ≥2 non-test remain) kept.
|
||||
4. **Named-cluster survival in non-sibling god-files** — inject agent-named method defs into `rangeNodes` even if the gather missed them; rank named ranges at importance **9** (above glue 6 / connected 3); `fileBudget = min(maxCharsPerFile, maxOutputChars - totalChars - 200)` in cluster selection so high-importance named clusters survive instead of being source-order-trimmed.
|
||||
- **Validated (headless A/B, Opus 4.8, median of 4, full 7-repo sweep) — now in README:** avg **25% cheaper · 57% fewer tokens · 23% faster · 62% fewer tool calls** (was 22/47/20/50). Per-repo cost: VS Code 33, Excalidraw 27, Django **23** (was 9, median 0 reads), Tokio 35, OkHttp **11** (was 4, 0 RealCall read-backs), Gin 15, Alamofire 28.
|
||||
- **PR #564 (already merged, `f1b14f0`)** was the prior round: named-callable spare + supertype-family override (fixed the read-back regression where RealCall.kt / compiler.py were skeletonized then Read back).
|
||||
|
||||
## Gotchas
|
||||
- **A/B per-repo variance is large (±~10–13 pts).** The WITHOUT arm swings run-to-run (how hard native greps). Excalidraw/Gin look *lower* than the prior README purely from a cheaper native baseline this batch — NOT regressions (reads still 0/low). **Averages are the stable signal.** Never conclude from n=1; the README is median-of-4.
|
||||
- **The alamofire `DataRequest` residual is NOT cleanly closable.** A "spare a file when the agent names its class" type-spare *broke OkHttp* (it spared all 5 interceptor classes → 0 skeletons). A named sibling class is structurally indistinguishable from "the one main type." Left as-is (alamofire is 28% cheaper; ~1 DataRequest read/run).
|
||||
- **Gin's WITH-cost ticked up ($0.36→$0.48 across batches)** — partly the named-injection adding content to an already-0-read repo. Still 15% cheaper. Possible over-eager named-injection on small repos.
|
||||
- **Validate retrieval changes with a real-agent A/B, not just the probe.** The deterministic `probe-explore.mjs` query forms a *different spine* than the agent's real query → it hid both the Django and the OkHttp read-backs. (Dead-end #6 in the design doc.)
|
||||
- **Always `npm run build` before probing/A/B** — probes + the A/B MCP server load `dist/`, not `src/`. Corpus indexes (`/tmp/codegraph-corpus/*`) are valid without re-index since all changes are query-time.
|
||||
- **`adaptive-sizing-skeletonizing.md` handoff is gone from `main`'s working dir** — it was untracked, got swept into commit `3c38729` on `feat/adaptive-explore-sizing`, so it lives only on that branch now. Deleting that branch deletes it (it's obsolete — that work shipped).
|
||||
- **5 `npm-shim` test failures are pre-existing/network** (lack `--probe-net` on the global binary) — not a regression; don't let them block.
|
||||
|
||||
## How to test & validate
|
||||
- Build first: `npm run build` (must be green).
|
||||
- Deterministic probe: `node scripts/agent-eval/probe-explore.mjs /tmp/codegraph-corpus/<repo> "<symbol-bag query>"` → inspect `#### file — … · focused/skeleton` headers + sizes. okhttp = 5 `· skeleton`; django compiler.py `· focused` with `def execute_sql`/`def as_sql`/`def _fetch_all` bodies present; excalidraw/tokio/vscode/gin = 0 skeleton/focused (inert).
|
||||
- A/B one repo: `bash /tmp/ab-one.sh <repo> <runs> "<question>"` → writes `/tmp/ab-readme/<repo>/run<n>/`. Aggregate one repo: `node /tmp/one-agg.mjs <repo>`. Full 7: `RUNS=4 bash scripts/agent-eval/bench-readme.sh` then `node scripts/agent-eval/parse-bench-readme.mjs /tmp/ab-readme` (averages) + `node /tmp/full-agg.mjs` (per-repo reads/grep/tools/cost/time).
|
||||
- Unit: `npx vitest run __tests__/adaptive-explore-sizing.test.ts` → **8/8** (skeleton, named-callable spare=RealCall, supertype-family override→focused=codec.ts, uniqueness/shared-method, on-spine exemplar full, distinct step full, flag=0 disables).
|
||||
- **Methodology:** a real win = cost DOWN **and** reads NOT up vs the same build's WITHOUT arm; confirm inert repos stay 0 skeleton/focused (the change only *adds* spare conditions + per-symbol rendering of already-collapsed files → strict subset of the original gate).
|
||||
|
||||
## Repo state
|
||||
- branch `main`, last commit `b026e64 feat(mcp): per-symbol adaptive codegraph_explore sizing (#569)`.
|
||||
- uncommitted: clean (this handoff file will be a new untracked `.claude/handoffs/` entry).
|
||||
- merged-but-undeleted branches: `feat/adaptive-explore-sizing` (#564) + `feat/explore-per-symbol-sizing` (#569), both local + remote.
|
||||
|
||||
## Open threads / TODO
|
||||
- [ ] Delete the two squash-merged branches (local + remote): `feat/adaptive-explore-sizing`, `feat/explore-per-symbol-sizing`.
|
||||
- [ ] (optional) Stabilize the README per-repo numbers with a 2nd full-7 batch → publish median-of-8 (smooths the WITHOUT-arm variance that makes Excalidraw/Gin look lower).
|
||||
- [ ] (frontier) Gin's small WITH-cost bump from named-injection on an already-0-read repo — consider gating named-injection by repo size / whether reads were happening.
|
||||
- [ ] (frontier) Alamofire `DataRequest` ~1 read/run — only closable by distinguishing "main type" from "interchangeable sibling," which has no clean static signal (don't re-attempt the type-spare; it breaks okhttp).
|
||||
- [ ] (frontier) Big non-sibling god-file clustering generally (#185 family) — `query.py`/`Session.swift`-style files; the fileBudget fix helped Django but the merge/selection logic is still the lever for other repos.
|
||||
|
||||
## Recent transcript (oldest → newest)
|
||||
### Turn — "Add the regression test for the skeletonization"
|
||||
- **Claude:** Wrote `__tests__/adaptive-explore-sizing.test.ts` (OkHttp-in-miniature fixture); validated via probe; 5 cases. (This was for PR #564's gate.)
|
||||
- **Outcome:** Test landed; led into re-validating the README.
|
||||
### Turn — re-validate README, found Django/OkHttp cost laggards → built fixes
|
||||
- **Claude:** A/B sweeps via `bench-readme.sh`; diagnosed read-backs (RealCall via `Lockable` 9-impl mixin; compiler.py base+subclasses). Shipped PR #564 (named-callable spare + supertype-family override). Then A+B+C (uniqueness + per-symbol focused + test exclusion).
|
||||
- **Outcome:** Django/OkHttp flipped to wins; alamofire type-spare attempt broke okhttp → reverted.
|
||||
### Turn — "Pursue the deeper Django fix"
|
||||
- **Claude:** Cluster-debug showed `_fetch_all` (L2237) never formed a cluster + got source-order-trimmed. Fixed via named-injection + importance 9 + `fileBudget`. A/B: Django reads tail 3→1, 11%→17%; controls (okhttp 14%, excalidraw 31% / 0 reads) held. Committed `e71eb90`.
|
||||
- **Outcome:** Django residual closed.
|
||||
### Turn — "push + PR + squash-merge, update README incl. averages + header"
|
||||
- **Claude:** Ran final full-7 sweep (`bench-readme.sh`, 56/56). Clarified a confusing "WITHOUT got cheaper" phrasing (WITH cg is cheaper in all 7; the % is the *gap*, which shrinks when the native baseline is cheap that batch). User said publish this batch.
|
||||
- **Outcome:** Updated README (headline 25%/62%, average line, 7 summary rows, 7 detail tables, methodology date) + CHANGELOG + design doc. Built clean branch off `origin/main` (dropping the already-squashed commits + the handoff artifact), pushed, opened PR #569, squash-merged → `b026e64`. Synced local to main, rebuilt dist. Offered branch cleanup → user ran `/handoff save`.
|
||||
@@ -1,70 +0,0 @@
|
||||
---
|
||||
name: framework-coverage-sweep-2026-05-23
|
||||
date: 2026-05-23 23:59
|
||||
project: codegraph
|
||||
branch: architectural-improvements
|
||||
summary: Dynamic-dispatch coverage sweep COMPLETE — all 14 README frameworks + every flow-relevant language validated (measure→fix→validate→test→playbook→commit). ~37 commits pushed, suite green. Ship-prep (CHANGELOG + PR to main) is the only thing left.
|
||||
---
|
||||
|
||||
# Handoff: Dynamic-dispatch framework/language coverage sweep (complete)
|
||||
|
||||
## Resume here — read this first
|
||||
**Current state:** The coverage sweep is **done**, AND a **frontier pass** closed the tractable partials. Every framework in the README's 14-row table is ✅, every flow-relevant language is validated (TS/JS, Python, Go, Java, C#, PHP, Ruby, Rust, Swift, Dart, Kotlin, Lua/Luau, Scala, C/C++), and the frontier pass added: React object data-router (literal), Next.js false-positive fix, Flask-RESTful `add_resource` (redash 6→77), Flask tuple methods + broader detection (flask-realworld 0→19), gorilla/mux confirmed. All committed/pushed to `architectural-improvements` (tree clean except untracked `.claude/handoffs/`). Full suite green (**809 passed**, 2 skipped; flaky `watcher.test.ts > debounced sync` passes on re-run). **No CHANGELOG entry exists, and the branch is not yet merged to main.**
|
||||
**Immediate next step:** Ship-prep — write a CHANGELOG entry grouping the whole sweep (route resolution for Flask/FastAPI/Drupal/Rust-Axum+actix/Vapor/Spring-Kotlin/Play + React Router routing; the Python builtin-name guard, Dart method-range, and C++ inheritance foundational fixes; the flutter-build and cpp-override synthesizer channels), bump `package.json`, then open a PR to main.
|
||||
|
||||
> Suggested next message: "do ship-prep: write the CHANGELOG entry covering the whole framework/language coverage sweep on this branch, bump the version, and open a PR to main"
|
||||
|
||||
## Goal
|
||||
Close static-extraction holes for **dynamic dispatch** across every language/framework codegraph supports, so cross-symbol flows (request→route→handler→service, state→render, virtual→override) exist in the graph and an agent answers flow questions with few codegraph calls and ~0 Read/Grep. Per framework/language: canonical flow `trace`s end-to-end, agent A/B shows fewer reads, no node explosion, recorded in `docs/design/dynamic-dispatch-coverage-playbook.md` (the matrix §6 + per-item notes §7). **This goal is now met; what remains is ship-prep + documented frontiers.**
|
||||
|
||||
## Key findings (this session's work, all committed)
|
||||
- **Routing convention is the hole in every backend** — same pattern each time: the resolver/extractor assumed one syntax. Flask (intervening `@login_required`/stacked routes), FastAPI (empty `""` path), Drupal (`claimsReference` for FQCN `_form`/single-colon controllers + contrib `detect` via composer name/type/`.info.yml`), Rust/Axum (chained `get(h).post(h2)` + namespaced `mod::handler`), actix (builder API `web::resource().route(web::get().to(h))`), Vapor (grouped `routes.grouped("x"); x.get(use:h)` — was 0 on every real app), Spring **Kotlin** (`fun` handler syntax + `.kt`), Play (extensionless `conf/routes` → controller), React Router (`<Route>` JSX).
|
||||
- **Three FOUNDATIONAL fixes (broad benefit, not framework-specific):** (1) Python **bare-name builtin guard** in `src/resolution/index.ts` — a handler named `index`/`get`/`update` was filtered as a builtin method; mirror the dotted-branch `knownNames` guard. (2) **Dart method-range** in `src/extraction/tree-sitter.ts` `createNode` — Dart bodies are SIBLINGS of the signature, so methods were `end==start` (signature-only); extend `endLine` to the resolved body (guarded, child-body grammars no-op). (3) **C++ inheritance** — `extractInheritance` handled `base_clause` (PHP) but not C++ `base_class_clause`; added it (leveldb extends 219→298).
|
||||
- **Two new synthesizer channels** in `src/resolution/callback-synthesizer.ts` (Dart analog + C++ analog of react-render): `flutter-build` (a State method calling `setState(` → `build`) and `cpp-override` (base virtual method → subclass override of same name, gated to C++).
|
||||
- **measure-first repeatedly split "needs work" from "already covered":** Svelte, NestJS (prior), and this session **Lua/Luau** (module dispatch already resolves) + **Compose** (composition is plain function calls, already static) needed NO code. The assumed hole wasn't real.
|
||||
- **`claimsReference` pre-filter is the recurring gotcha** (`src/resolution/index.ts:497-503`): a route ref naming no declared symbol (FQCN, `Controller@method`, `controller#action`, `Class.method`) is dropped before `framework.resolve()` runs. Added for Drupal + Play this session.
|
||||
|
||||
## Gotchas
|
||||
- **`claimsReference`:** if a new framework's route refs don't resolve despite a correct `resolve()`, it's the pre-filter — add `claimsReference`.
|
||||
- **Reindex picks up resolver changes only on a CLEAN index:** `codegraph index` is incremental (skips unchanged files); after `npm run build`, do `rm -rf .codegraph && codegraph init -i` to re-extract. The init message's edge count is contains-only (~misleading); query the DB for the real count.
|
||||
- **Extraction changes are high blast radius** (shared `createNode`/`extractInheritance`): re-check node counts on control repos (excalidraw 9,290 / django 302) — the Dart/C++ fixes are guarded to only-extend / C++-only, controls unchanged.
|
||||
- **Play `conf/routes` is extensionless** → needed `isPlayRoutesFile` opt-in in `grammars.ts` (isSourceFile + detectLanguage→'yaml' no-grammar path). Narrow match, only ADDS Play files.
|
||||
- **Flaky:** `watcher.test.ts > debounced sync > should trigger sync after file change` — timing-based, passes on re-run; unrelated to any of this work.
|
||||
- **Foreground `sleep` is blocked** in Bash → background A/B batches (`run_in_background: true`), read the task output file. zsh quirks: quote globs (`'*.vue'`); SQL `count(*)` in `$(...)` needs care with quotes.
|
||||
- Global `codegraph` is npm-linked to this repo's `dist/`; `npm run build` then reindex. A/B harness: `scripts/agent-eval/run-all.sh <repo> "<Q>" headless` (with vs empty MCP), parse via `node scripts/agent-eval/parse-run.mjs`.
|
||||
|
||||
## How to test & validate (the per-framework loop)
|
||||
- Corpus in `/tmp/codegraph-corpus/<name>` (clone S/M/L, `git clone --depth 1`). Index: `rm -rf .codegraph && codegraph init -i`.
|
||||
- Measure holes: `sqlite3 .codegraph/codegraph.db "select count(*) from nodes where kind='route'"` + route→handler edges (`join edges on source where kind='references'`). Node-count before/after (no explosion).
|
||||
- Flow: `node scripts/agent-eval/probe-node.mjs <repo> <symbol>` (shows Called-by/Calls trail) / `probe-trace.mjs <repo> <from> <to>`.
|
||||
- Agent A/B (≥2 runs/arm, variance is real): `run-all.sh` headless, record Read/Grep/duration/codegraph. Pass = fewer reads with codegraph.
|
||||
- Tests: `npm test` (vitest). Resolver extract tests in `__tests__/frameworks.test.ts`; end-to-end in `__tests__/frameworks-integration.test.ts` (real CodeGraph + indexAll); Dart range in `__tests__/extraction.test.ts`; Drupal in `__tests__/drupal.test.ts`.
|
||||
|
||||
## Repo state
|
||||
- branch `architectural-improvements`, last commit `42a0178 docs(playbook): record frontier pass; test(go): gorilla/mux`.
|
||||
- uncommitted: clean (only untracked `.claude/handoffs/`).
|
||||
- ~37 commits total on the branch (handoff's original 11 frameworks + this session's: Flask/FastAPI, Drupal, Rust/Axum, Vapor, React Router, actix, Dart, Kotlin, Lua, Scala/Play, C/C++ — each a feat + a docs(playbook) commit; Lua was docs-only).
|
||||
|
||||
## Open threads / TODO
|
||||
- [ ] **SHIP-PREP (the only blocker to merge):** CHANGELOG entry for the whole sweep, `package.json` bump, PR to main. Releases go through `.github/workflows/release.yml` only — do NOT `npm publish` (see CLAUDE.md).
|
||||
- [x] **Frontier pass DONE (commits 0456915, 03e49ab, 42a0178):** React object data-router (literal), Next.js false-positive fix, Flask-RESTful `add_resource`, Flask tuple methods + detection, gorilla/mux confirmed.
|
||||
- [ ] **Frontiers LEFT (deliberately, with rationale in playbook §7 "Frontier pass"):** anonymous/inline closures (def-use frontier), metaprogramming finders (AR/Eloquent/JPA/EF), reactive runtimes (Vue Proxy / Compose recomposition), Akka actors, C callback-struct 422-way fan-out, C++ pure-virtual base methods, React lazy data-router (variable paths + lazy imports), Play SIRD, Nuxt-specific. Forcing these adds noise.
|
||||
- [ ] Pre-existing, unrelated: Next.js `*.config.mjs` in a `pages/` dir treated as a route (false-positive found in bulletproof-react).
|
||||
|
||||
## Recent transcript (oldest → newest, this session)
|
||||
### Turn — "what's left / what's next on coverage" → did Flask/FastAPI
|
||||
- 3 holes: Flask intervening/stacked decorators, FastAPI empty path, **Python bare-name builtin guard** (handlers named `index`/`get` filtered). microblog 6→27, realworld 12→20, dispatch 290/290. Fixed 6 stale Laravel/Rails tests too. Committed + pushed.
|
||||
### Turn — "Drupal next"
|
||||
- `claimsReference` for FQCN/_form/single-colon controllers + contrib `detect` (composer type/name + `.info.yml`). core 536→731 (87%), admin_toolbar 0→14. OOP `#[Hook]` = frontier. Committed.
|
||||
### Turn — "Rust: Axum/actix/Rocket"
|
||||
- Axum chained methods + namespaced handlers (realworld 12→19, 19/19); Rocket already 99%; **actix builder API** `web::resource().route(web::get().to())` (examples 51→128). Committed (2 commits: axum, then actix).
|
||||
### Turn — "Vapor (Swift)"
|
||||
- Resolver was 0-routes on every real app; rewrote for any receiver + optional non-string paths + `.grouped` prefix tracking + `use:` discriminator. template 0→3, SteamPress 0→27, SPI 0→14. Committed.
|
||||
### Turn — "2, 3, 4" (React Router, actix [done above], Dart/Flutter)
|
||||
- React Router `<Route>` JSX (react-realworld 0→10). Dart/Flutter: **method-range fix** (foundational) + `flutter-build` setState→build synthesizer. Committed.
|
||||
### Turn — "Kotlin next"
|
||||
- Spring resolver `['java']`→`['java','kotlin']` + `fun` handler regex (petclinic-kotlin 0→18, 18/18; Java unchanged 19/19). Compose composition already static. Committed.
|
||||
### Turn — "Lua/Luau, Scala, C/C++ (Lua first, but do all three)"
|
||||
- **Lua:** measure-first → module dispatch already covered (telescope 335 cross-file calls); no code change, validated. **Scala/Play:** `conf/routes` file-walk opt-in + Play resolver (computer-database 0→8). **C/C++:** general dispatch strong (redis 29k); fixed C++ `base_class_clause` inheritance + `cpp-override` synthesizer (leveldb 12 precise). All committed + pushed.
|
||||
### Turn — "wrap up + refresh handoff"
|
||||
- This handoff. Sweep complete; ship-prep (CHANGELOG + PR) is the remaining work.
|
||||
@@ -1,86 +0,0 @@
|
||||
---
|
||||
name: trace-relevance-coldstart-2026-05-30
|
||||
date: 2026-05-30 23:30
|
||||
project: codegraph
|
||||
branch: feat/trace-relevance-closure-collection
|
||||
summary: Turned Alamofire (README's weakest repo) into a clean win via a trace endpoint-disambiguation fix + god-file explore rendering, then eliminated the MCP cold-start race that was causing benchmark inconsistency (handshake ~811ms→~90ms); PR #580 has 6 commits, all that's left is a clean README sweep + squash-merge.
|
||||
---
|
||||
|
||||
# Handoff: trace-relevance + closure-collection + cold-start (PR #580)
|
||||
|
||||
## Resume here — read this first
|
||||
**Current state:** PR #580 (branch `feat/trace-relevance-closure-collection`, 6 commits, pushed, in sync with remote) is feature-complete and validated — full suite 1090 pass (only the 5 pre-existing npm-shim network fails), 28/28 MCP+daemon tests. The MCP cold-start race (the dominant benchmark-inconsistency source) is ELIMINATED via the proxy-local-handshake (tool registration ~90ms cold+warm, was ~811ms). The README benchmark table still shows the OLD pre-fix numbers.
|
||||
**Immediate next step:** Run a median-of-4 README sweep on this build (the race is gone, so numbers should be naturally consistent), update the README table/averages/headline, then squash-merge PR #580.
|
||||
|
||||
> Suggested next message: "Run `RUNS=4 bash scripts/agent-eval/bench-readme.sh` on this build, parse with `node scripts/agent-eval/parse-bench-readme.mjs /tmp/ab-readme` (race-aware), update the README benchmark table + averages + the 7 per-repo detail tables + methodology date, then squash-merge PR #580 with `gh pr merge 580 --squash --admin`."
|
||||
|
||||
## Goal
|
||||
Started as "Alamofire is the README's weakest benchmark repo (13% fewer tool calls vs the ~62% average) — fix it." Became: make CodeGraph's retrieval **consistent and faster**. Definition of done = PR #580 merged (trace fix + dynamic-dispatch coverage + god-file rendering + cold-start elimination), README refreshed with stable median-of-4 numbers. Optimization target per CLAUDE.md is **tool-calls/reads + latency**, NOT raw cost.
|
||||
|
||||
## Key findings
|
||||
The 6 commits on the branch (oldest→newest):
|
||||
- `e86d573` **Trace endpoint relevance** (THE Alamofire win) + closure-collection synthesizer + explore synth-links.
|
||||
- `c64c4b3` **God-file multi-phase explore rendering** (6 sub-layers).
|
||||
- `5d7388c` Skeleton/focused tag steers to `codegraph_explore`, not Read (spiral fix #1).
|
||||
- `dc19eab` Bench parser race-aware (excludes "No such tool available" runs).
|
||||
- `91e28df` serve --mcp cold-start ~811ms→~600ms (defer CodeGraph load + 25ms poll).
|
||||
- `82ae484` **Proxy-local-handshake** — handshake ~600ms→~90ms, cold-start race eliminated.
|
||||
|
||||
Root-causes found by reading A/B TRANSCRIPTS (not the noisy median):
|
||||
- **Trace bug:** `handleTrace`'s `scorePair` ranked only by shared-dir-prefix, so overloaded names (`request`=44 defs, `task`=8) resolved to empty `EventMonitor.request(){}` / `RedirectHandler.task` STUBS over the real `Session.request` → agent saw garbage, said "the trace collided with same-named symbols", read by hand. Fix: `nodeRelevance` term in `handleTrace` (penalize ≤1-line stubs −40, test files −150). Result n=8: WITH tools 12→8 median, read variance 0–12→1–4 (the meltdowns WERE the trace-collision flounder). General bug (Swift/Java/C#/Go protocol-stub flooding).
|
||||
- **Closure-collection synthesizer** (`src/resolution/callback-synthesizer.ts` `closureCollectionEdges`): Swift `validators.write{$0.append}`…`didCompleteTask` `validators.forEach{$0()}`. The element-invoke `$0(`/`it(` is the precision gate → 9 edges on Alamofire, **0 on every non-Swift control**. Surfaced inline in trace + a "Dynamic-dispatch links" section in `buildFlowFromNamedSymbols` (so it shows when the agent named only `validate`, not `didCompleteTask`).
|
||||
- **God-file rendering** (`handleExplore` in `src/mcp/tools.ts`, 6 layers): (1) on-spine god-files render spine-full + off-path methods as signatures (true-spine); (2) named-seed gather — inject each named token's substantive def into the subgraph (FTS buried `validate` → Validation.swift was never gathered); (3) a file that DEFINES a named symbol scores +50 (beats incidental Combine.swift's +23 connected-node score); (4) the 90%-budget early-break and (5) the total-output cap both EXEMPT necessary (entry/spine/uniqueNamed) files; (6) final ceiling 1.5×maxOutputChars. Renders build+validators-exec+validate in ONE explore.
|
||||
- **Spiral cause #1 (fixed):** the skeleton tag said "Read for a full body" → agent Read the skeletonized central files → over-investigation spiral. Now steers to `codegraph_explore`.
|
||||
- **Spiral cause #2 / the BIG inconsistency (fixed):** MCP **cold-start race**. `serve --mcp` wasn't ready when the headless agent fired → "No such tool available" → grep/Read flounder (19–30 tool spirals). Root-caused: NOT module load (mcp/index 38ms, CodeGraph chain 30ms), NOT the `--liftoff-only` re-exec (NO_RELAUNCH ≈ same) — it's the proxy WAITING for the spawned daemon to bind. Fixed: proxy answers initialize/tools-list from STATIC constants (`runLocalHandshakeProxy` in `proxy.ts`), forwards tool CALLS to the daemon (connected in background), lazy in-process engine fallback preserves the old fall-back-to-direct robustness. `connectWithHello` distinguishes 'version-mismatch' (fail fast → local) from 'not-yet' (poll). Handshake 91ms cold / 88ms warm.
|
||||
|
||||
## Gotchas
|
||||
- **A/B variance is HUGE — never conclude from n=1, or even one n=4 batch.** The median-of-4 caught regressions the lucky dedicated batches HID (the god-file rework looked great in one batch at 0.5 reads/5.5 tools; the median showed 13 tools dragged by 2 spirals). Report ranges.
|
||||
- **Kill stale daemons before any cold-start measurement:** `pkill -9 -f "dist/bin/codegraph.js"; rm -f /tmp/codegraph-corpus/<repo>/.codegraph/daemon.*`. A zombie daemon holding the lock causes a 6s retry-exhaust that looks like a 7× regression (it bit me — the "6239ms" false alarm).
|
||||
- **`timeout` is NOT on macOS** (no coreutils) — measure cold-start with a `node` spawn + a `setTimeout` kill-timer (see the transcript's measurement snippets).
|
||||
- Corpus repos: `/tmp/codegraph-corpus/<repo>` (all 7 README repos indexed). Explore/trace changes are **query-time** (no re-index). The closure-collection synthesizer is **index-time** but produces 0 edges on non-Swift, so it's inert there.
|
||||
- Global `codegraph` is npm-linked to the dev dist (`node dist/bin/codegraph.js`). **Always `npm run build` before any probe/A/B** (they load `dist/`, not `src/`).
|
||||
- `engine.ts`/`tools.ts` now `import type CodeGraph` + lazy `require('../index')` (CommonJS, cached) so the daemon binds before the sqlite/query chain loads; `findNearestCodeGraphRoot` now comes from the light `../directory`.
|
||||
- The old `runProxy`/`pipeUntilClose` in `proxy.ts` are now DEAD (superseded by `runLocalHandshakeProxy`) — left in place; safe to prune in a follow-up.
|
||||
- 5 `npm-shim.test.ts` failures are pre-existing/network (need `--probe-net`) — NOT regressions; ignore.
|
||||
- Uncommitted `.gitignore` change (`tmux-web/`) is unrelated/not mine — do NOT commit it on this branch.
|
||||
- `parse-bench-readme.mjs` excludes raced runs by default; `CG_INCLUDE_RACED=1` keeps them to see the raw distribution. Now a safety net (race eliminated at source).
|
||||
|
||||
## How to test & validate
|
||||
- `npm run build` → must be clean (exit 0).
|
||||
- `npx vitest run` → **1090 pass**, only the 5 npm-shim network fails.
|
||||
- `npx vitest run __tests__/mcp-daemon.test.ts` → **7/7** (sharing, #277 survive-client-death, version-mismatch fallback, idle-timeout).
|
||||
- Cold-start handshake (after killing daemons): node-spawn a `serve --mcp`, send `initialize`, time the id:1 response → **~90ms** (was ~811ms). Then a `tools/call` (e.g. `codegraph_status`) returns a real result (forwarded to the daemon, ~3.4s on vscode's first index load — a call that returns LATE, not a missing-tool error).
|
||||
- A/B sweep: `RUNS=4 bash scripts/agent-eval/bench-readme.sh` → `node scripts/agent-eval/parse-bench-readme.mjs /tmp/ab-readme`.
|
||||
- **Methodology:** handshake <150ms = race eliminated; in an A/B, grep the WITH jsonls for "No such tool available" (should be 0 now); WITH reads/tools < WITHOUT with no control regression.
|
||||
|
||||
## Repo state
|
||||
- branch `feat/trace-relevance-closure-collection`, last commit `82ae484 perf(mcp): proxy answers initialize/tools-list locally — cold-start handshake ~600ms→~90ms`. In sync with remote (0/0). PR **#580** open.
|
||||
- uncommitted: ` M .gitignore` (unrelated tmux-web/ edit, not mine) + the new untracked `.claude/handoffs/` files. All code committed.
|
||||
|
||||
## Open threads / TODO
|
||||
- [ ] Run median-of-4 README sweep on THIS build (race gone → should be naturally consistent, no exclusions needed).
|
||||
- [ ] Update README: benchmark summary table (7 rows × cost/tokens/time/tools), the 7 per-repo detail tables (Time/Reads/Grep/Tools/Tokens/Cost), the average line, the headline (line 7), and the methodology date (line 86).
|
||||
- [ ] Squash-merge PR #580 (`gh pr merge 580 --squash --admin` per the PR-merge-policy memory).
|
||||
- [ ] Add CHANGELOG entries under `[Unreleased]` for the cold-start fixes (trace fix + closure-collection + god-file already have entries from earlier commits — verify).
|
||||
- [ ] (cleanup) Prune dead `runProxy`/`pipeUntilClose` in `proxy.ts`.
|
||||
- [ ] (frontier, deferred) Alamofire `task` overload still seeds `RedirectHandler.task` (a harmless false-positive in named-seed ranking) — same nodeRelevance idea could de-rank it.
|
||||
|
||||
## Recent transcript (oldest → newest)
|
||||
### Turn — "which repos drag the averages? Alamofire looks worst"
|
||||
- **Claude:** Computed per-column deviations from the README table → Alamofire #1 drag (tool-calls 13% vs 62% avg, time 7% vs 23%), gin #2; both ~110-file repos.
|
||||
- **Outcome:** Targeted Alamofire.
|
||||
### Turn — "dig into Alamofire A/B logs"
|
||||
- **Claude:** Read `/tmp/ab-readme/alamofire/run*/run-headless-with.jsonl` → found validate-flow break (`validators` closure-array) + per-run reasoning. Built closure-collection synthesizer + explore synth-links. A/B: flat (adoption variance).
|
||||
- **Outcome:** Synthesizer correct + sufficient (forced codegraph-only 3/3) but didn't move the noisy median.
|
||||
### Turn — "keep working till you figure it out"
|
||||
- **Claude:** Read transcripts again → "the trace collided with same-named symbols (44 requests, 8 tasks)". Found `scorePair` picked empty stubs. Added `nodeRelevance`. A/B n=8: tools 12→8, variance collapsed.
|
||||
- **Outcome:** The real Alamofire win. Committed `e86d573`; opened PR #580; squash-merge deferred.
|
||||
### Turn — "do the file-ranking + gather layers"
|
||||
- **Claude:** Built the 6-layer god-file rendering (`c64c4b3`). A/B flat-to-worse → debugged spirals → found the skeleton tag said "Read for more" (fixed `5d7388c`) → spirals persisted → root-caused the MCP cold-start race ("No such tool available").
|
||||
- **Outcome:** Race is the dominant inconsistency, not the code.
|
||||
### Turn — "keep iterating until consistent / do both (parser + startup)"
|
||||
- **Claude:** Bench parser race-aware (`dc19eab`). Deferred CodeGraph load off the bind path + 25ms poll (`91e28df`, 811→600ms).
|
||||
- **Outcome:** Consistency handled (parser) + 25% startup.
|
||||
### Turn — "how to optimize cold-start? / implement it"
|
||||
- **Claude:** Profiled (module loads 30-40ms, re-exec not it — it's the daemon-spawn wait). Implemented `runLocalHandshakeProxy` (`82ae484`): proxy answers initialize/tools-list locally + forwards calls + lazy local-engine fallback. Fixed 4 daemon tests (emit "Attached to shared daemon" + fast-fail version-mismatch + updated 1 assertion). Handshake 90ms; 28/28 MCP tests; full suite 1090 pass.
|
||||
- **Outcome:** Cold-start race ELIMINATED. All cold-start work committed + pushed. README sweep + squash-merge pending.
|
||||
Reference in New Issue
Block a user