docs(benchmarks): record the CG-24 epic resolution

Four shipped fixes, one open defect (CG-36), and five issues closed because
measurement contradicted them. The headline is that the reported symptom was
not an explore bug at all — it was a degraded index (CG-33), and the reported
query answers correctly on a clean rebuild with no explore change.

Records the two traps that cost real time and are now guarded in tooling: the
nonexistent .codegraph/graph.db path that sqlite3 silently creates, and
ab-new-vs-baseline.sh swapping src/ mid-run so a commit captures baseline
sources.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Colby McHenry
2026-08-06 14:46:11 -05:00
co-authored by Claude Opus 5
parent ed6c56f3fa
commit 76ab1fe130
@@ -0,0 +1,95 @@
# Epic resolution — explore response noise (CG-24)
Worked 2026-08-05 → 2026-08-06. Started from one bad `codegraph_explore` response
in a real session and ended with four shipped fixes, one open defect, and five
issues closed because measurement contradicted them.
**The headline: the reported symptom was not an explore bug.** It was a degraded
index. The explore defects the investigation found are real and were fixed, but
none of them caused the report.
## The report
A prose flow query returned an unusable response: the symbol the agent had named
never rendered, and a 12k-line generated Cloudflare ambient-types file took 60.7%
of the output envelope.
```
# deliv% bytes reserved score pen flags file
1 1.0% 251 10,970 87.0 1.00 named entry central <the named file>
2 60.7% 15,043 6,484 49.0 1.00 entry central worker-configuration.d.ts
4 — — — 19.9 1.00 dropped: budget <a third file>
```
## Root cause
**Index drift ([CG-33](index-drift-cg33.md)).** The live incrementally-synced index
diverged from a clean rebuild by 4.3% of distinct edges, bidirectionally,
overwhelmingly `calls`. RWR graph mass is relative and normalized, so call edges
missing elsewhere inflate an unaffected file's share — the `.d.ts` carried mass
0.24750 drifted vs 0.13119 rebuilt (~1.9×), score 49.0 vs 27.0.
Two causes, both fixed: incremental sync re-resolved only references *in* changed
files, and `getNodesByName` had no `ORDER BY`, so ties broke by rowid — i.e. by
the order files happened to be **written**. The second is why scope alone could
never converge. Stale edges dropped 671 → 2 across an 80-commit replay.
On a freshly rebuilt index the reported query answers correctly **with no explore
change at all**.
## Shipped
| | what |
|---|---|
| **CG-30** | Bounded how far an oversize cluster member may overshoot; windows on whole lines past 1.5× instead of emitting whole — or, when larger than the response ceiling, dropping the file silently. |
| **CG-31** | Gave the cluster path the `owedBelow` displacement guard the whole-file BUY arm always had, holding back only the prefix of what is owed below that the response can actually pay. |
| **CG-26** | Closed the remaining holes: whole-file arms had no displacement guard at all, section overhead was charged at a flat 200 against a real 300500, and `owedPayableBelow` held all-or-nothing. |
| **CG-25** | Recognize `Generated by <tool> by running <command>` banners. Precision held by requiring two `by` clauses, so ordinary prose does not match. |
| **CG-28** | Damp declaration-only files that nothing in the index depends on. Does not stack with the generated penalty (`Math.min`), and naming a declaration symbol exempts its file. |
| **CG-33 / CG-35** | Incremental sync converges with a rebuild, plus a regression suite that fails when the fix is disabled. |
Deterministic across the 6-repo suite: no repo truncates, none loses a file,
okhttp gains one, every repo lands at or under the 25,000 hard ceiling.
## Open
**[CG-36]** — a file's non-first clusters are never shrunk, so a trivial cluster
starves the answer-bearing one. Found as a byproduct of CG-27's measurement and
confirmed firing on the epic tip: `query.py` leaves 81% of its budget unspent,
keeping a score-14 cluster and dropping a score-290 one. Two candidate fix
points — the density tiebreak in selection, and the never-shrink rule — and
density-first must keep working (the `Session.swift` case).
## Closed because measurement contradicted them
Five, which is the story of this epic as much as the fixes are.
| | why |
|---|---|
| **CG-32** | Named file "didn't render first." Drift artifact; on a clean index it renders first and takes 89%. |
| **CG-34** | "Allocator over-reserves for low-scoring files." Filed on a runner's diagnosis without checking the numbers. The file was never over-reserved (4,314 in both arms) — it was over-*spending*, which is CG-31. |
| **CG-27** | Adding `function`/`method` to `ENVELOPE_KINDS` measured as a **large regression** — rank #1 fell from 7,539 delivered chars to 397, 7 of 11 inner closures to 0. The enclosing range was holding the file together as one cluster, inside which `shrinkCluster` already did the per-symbol ranking the issue wanted. A careful version was noise (69 vs 68 across nine queries). |
| **CG-29** | Prose-vs-symbol query gap. Inverted on measurement: prose matches symbol on django and delivers 63% more source on okhttp. The founding observation was drift. |
| **CG-37** | Duplicate of CG-36, filed without seeing it. |
## What this epic is actually a lesson in
**A confident diagnosis is worth less than a cheap measurement.** Every issue
above was filed by someone — human or agent — who had read the code and had a
plausible mechanism. Five were wrong. The ones that survived did so because a
deterministic probe disagreed with them and the probe won.
Two specific traps this cost real time on, both now guarded in tooling:
- **`.codegraph/graph.db` does not exist** — the index is `codegraph.db`, and
`sqlite3` against a mistyped path *creates* an empty database rather than
failing. An empty schema reads exactly like a stale pre-migration index. This
produced a wrong root cause. `diff-index-drift.mjs` refuses a missing path.
- **`ab-new-vs-baseline.sh` checks the engine out at the baseline ref mid-run.**
A commit made while it runs captures baseline sources and silently reverts the
fix under test. This happened during CG-30. Check the `changed:` line before
believing any A/B result.
And one measurement discipline worth keeping: **compare sets, not totals.** The
drift that started all of this shows up as +0.7% on raw edge counts, because it
is bidirectional and nets out. On distinct edge triples it is 4.3%.