fix(extraction): capture & clean docstrings across all README languages (#780) (#806)

* fix(extraction): capture docstrings for export/const/decorator-wrapped symbols (#780)

getPrecedingDocstring walked previousNamedSibling from the EMITTED
declaration node, so it only found a leading comment when the comment was
a direct sibling of that node. For a declaration nested under a wrapper —
`export class X` / `export const f = () => {}` (export_statement /
lexical_declaration), a plain const arrow (variable_declarator), or a
decorated Python def/class (decorated_definition) — the comment is a
sibling of the WRAPPER, so the inner node had no preceding comment and
the docstring was stored as NULL.

Climb out through the wrapper node(s) before scanning for the comment.
Each wrapper holds exactly one declaration, so this can't mis-attribute a
comment to a sibling (verified: an uncommented method does NOT inherit its
class's comment). Also strip leading `#` from Python/Ruby/shell line
comments, which the cleanup chain missed (Python docstrings used to keep
their `#`).

Query/extraction-layer change to a parse helper; re-index to pick up
docstrings on already-indexed files. Verified on the reporter's JS/TS and
Python repros (8/8 now captured) plus over-walk controls; +3 tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(extraction): clean comment markers across all supported languages (#780)

Validating docstring capture across every README language surfaced that
the marker cleanup only knew C-style `//` and `/* */`, plus the `#` added
earlier this branch. Doc comments in other styles were captured but left
their markers in the stored text:

  - Rust/Swift/Kotlin doc lines `///` and `//!`  -> leading `/` / `!` leaked
  - Lua/Luau `--` and `--[[ ]]`                  -> not stripped
  - Pascal `{ }` and `(* *)`                     -> not stripped

Extract the cleanup into cleanCommentMarkers() and handle every style.
Paired block delimiters are stripped only when the comment OPENS with one,
so a line comment that happens to end with `}` / `*)` / `]]` is never
truncated; per-line markers stay anchored at line start.

Validated end-to-end (extract -> index -> codegraph_node output) across
all 19 tree-sitter code languages plus Svelte/Vue `<script>` blocks: every
one now stores and returns a clean docstring. +1 cross-language test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
2026-06-11 12:38:22 -05:00
committed by GitHub
co-authored by Claude Opus 4.8
parent 0b1a2eed97
commit 0df9246752
3 changed files with 142 additions and 12 deletions
+83
View File
@@ -184,6 +184,89 @@ export class PaymentService {
expect(chargeMethod).toBeDefined();
});
it('captures docstrings for export- and const-wrapped declarations (#780)', () => {
const code = `
// plain class control
class Ledger {}
// exported class
export class Invoice {}
// export default
export default function settle() { return true; }
// exported arrow const
export const refund = (amount: number) => amount;
// non-export arrow const
const audit = (amount: number) => amount;
`;
const byName = new Map(extractFromSource('doc.ts', code).nodes.map((n) => [n.name, n]));
expect(byName.get('Ledger')?.docstring).toBe('plain class control'); // control still works
expect(byName.get('Invoice')?.docstring).toBe('exported class');
expect(byName.get('settle')?.docstring).toBe('export default');
expect(byName.get('refund')?.docstring).toBe('exported arrow const');
expect(byName.get('audit')?.docstring).toBe('non-export arrow const');
});
it('does not mis-attribute a class comment to an uncommented member (#780)', () => {
const code = `
// Comment for Box
export class Box {
noComment() {}
// own comment
withComment() {}
}
`;
const byName = new Map(extractFromSource('box.ts', code).nodes.map((n) => [n.name, n]));
expect(byName.get('Box')?.docstring).toBe('Comment for Box');
expect(byName.get('noComment')?.docstring ?? null).toBeNull(); // no over-walk
expect(byName.get('withComment')?.docstring).toBe('own comment');
});
it('captures docstrings for decorated Python declarations, stripping `#` (#780)', () => {
const code = [
'# decorated function',
'@app.route("/x")',
'def py_handler():',
' return 1',
'',
'',
'# plain function control',
'def py_plain():',
' return 1',
'',
'',
'# decorated class',
'@dataclass',
'class PyModel:',
' pass',
'',
].join('\n');
const byName = new Map(extractFromSource('mod.py', code).nodes.map((n) => [n.name, n]));
expect(byName.get('py_handler')?.docstring).toBe('decorated function');
expect(byName.get('py_plain')?.docstring).toBe('plain function control'); // `#` stripped
expect(byName.get('PyModel')?.docstring).toBe('decorated class');
});
it('cleans comment markers across language styles (#780)', () => {
const doc = (file: string, code: string, name: string) =>
new Map(extractFromSource(file, code).nodes.map((n) => [n.name, n])).get(name)?.docstring;
// Rust doc lines (`///`, `//!`) — the trailing slash used to leak through.
expect(doc('m.rs', '/// rust doc line\nfn rs_fn() {}', 'rs_fn')).toBe('rust doc line');
// Lua line + long-bracket comments.
expect(doc('m.lua', '-- lua line\nfunction lua_fn() end', 'lua_fn')).toBe('lua line');
expect(doc('b.lua', '--[[ lua block ]]\nfunction lua_b() end', 'lua_b')).toBe('lua block');
// Pascal brace and paren-star comments.
const pasUnit = (c: string) =>
`unit U;\ninterface\n${c}\nprocedure P;\nimplementation\nprocedure P;\nbegin\nend;\nend.\n`;
expect(doc('a.pas', pasUnit('{ pascal brace }'), 'P')).toBe('pascal brace');
expect(doc('c.pas', pasUnit('(* pascal paren *)'), 'P')).toBe('pascal paren');
// C block comment still clean (no regression).
expect(doc('m.c', '/* c block */\nvoid c_fn(void) {}', 'c_fn')).toBe('c block');
});
it('should extract interfaces', () => {
const code = `
export interface User {