Programs, sections/paragraphs (reconstructed extents over the grammar's flat header stream), PERFORM/THRU/GO TO/CALL call edges, COPY copybook imports incl. standalone .cpy fragments, DATA DIVISION records/fields/ 88-levels with write-site impact references, and CICS flows: EXEC LINK/XCTL program targets (literal + same-file VALUE deref), EXEC SQL INCLUDE, and pseudo-conversational RETURN/START TRANSID hops resolved to the owning program via a CICS framework resolver. Fixed and free source format (free format via a scanner wide-mode sentinel). Grammar: vendored wasm built from a patched yutaro-sakamoto/ tree-sitter-cobol (EXEC blocks as an external-scanner token, copybook fragment entry point, single-quote continuation, COPY REPLACING pseudo-text, NOT=, CALL GIVING, ENTRY, FREE, bitwise ops, abbreviated relations, COBOL-2002 usages, and more). Patch + provenance + upstream PR draft in docs/grammars/. Parse health: AWS CardDemo 43/44 native (upstream: 9/31), 44/44 through preParse; copybooks 28/29; CobolCraft free-format 17/17 (upstream: 0); NIST COBOL85 unchanged at 373/382. Copybook members resolve to files like C includes (basename index, name-matcher short-circuit so compiler-supplied members stay honestly unresolved): CardDemo imports 5 -> 285. Impact proof: ACCT-CURR-BAL (CVACT01Y copybook) surfaces its 4 writer programs cross-file. Also: run-all.sh now neutralizes the ambient prompt-hook in both A/B arms (CODEGRAPH_NO_PROMPT_HOOK=1); COBOL corpus entries for agent-eval. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Colby Mchenry
co-authored by
Claude Fable 5
Claude Fable 5
parent
7d624ecfac
commit
41620c60fa
@@ -0,0 +1,128 @@
|
||||
# tree-sitter-cobol.wasm — provenance & rebuild
|
||||
|
||||
`src/extraction/wasm/tree-sitter-cobol.wasm` is built from
|
||||
[yutaro-sakamoto/tree-sitter-cobol](https://github.com/yutaro-sakamoto/tree-sitter-cobol)
|
||||
at commit `e99dbdc3d800d5fa2796476efd60af91f6b43d93` with the patch in
|
||||
`tree-sitter-cobol.patch` applied (grammar.js + src/scanner.c; everything else
|
||||
is regenerated by `tree-sitter generate`).
|
||||
|
||||
## What the patch adds
|
||||
|
||||
The upstream grammar (COBOL85, derived from opensource-cobol, NIST-tested)
|
||||
parses batch COBOL well but chokes on the constructs that dominate real
|
||||
mainframe and GnuCOBOL code:
|
||||
|
||||
1. **`EXEC ... END-EXEC` blocks** (CICS / SQL / DLI). Upstream has no EXEC
|
||||
support at all — the tokens get absorbed into the preceding statement until
|
||||
one breaks the parse, cascading hundreds of lines into one ERROR. The patch
|
||||
adds an `EXEC_BLOCK` external-scanner token that consumes the whole block,
|
||||
surfaced as a single `exec_statement` node (valid as a procedure statement
|
||||
and as a data-division entry, for `EXEC SQL INCLUDE`/`DECLARE`).
|
||||
2. **`NOT=`** without a space (IBM COBOL accepts it).
|
||||
3. **`FD <name>.` followed by `COPY`** for the record layout — upstream
|
||||
required a literal record description after every FD (also fixed for
|
||||
LINKAGE SECTION).
|
||||
4. **`COPY ... REPLACING ==pseudo-text== BY ==pseudo-text==`** with multiple
|
||||
replacement pairs. (Upstream's `replacing_clause` never consumed the
|
||||
REPLACING keyword and allowed only one WORD/string pair.)
|
||||
5. **Single-quote string continuation lines** (hyphen in the indicator
|
||||
column). Upstream's scanner handled continuation only for double-quoted
|
||||
strings; the patch generalizes the quote character and handles doubled-
|
||||
quote escapes (`'DON''T'`).
|
||||
6. **`FUNCTION <intrinsic>(refmod)`** — upstream's generic FUNCTION fallback
|
||||
took arguments but not a reference-modification suffix
|
||||
(`FUNCTION CURRENT-DATE(1:4)`; the dedicated intrinsic tokens like
|
||||
`CURRENT-DATE-FUNC` only match opensource-cobol's *preprocessed* names).
|
||||
7. **Standalone copybook entry point** — a `copybook_fragment` start
|
||||
alternative so `.cpy` files (data records or procedure paragraphs, no
|
||||
IDENTIFICATION DIVISION) parse without a wrapper. A file is either
|
||||
programs or one fragment, keeping program suffixes unambiguous.
|
||||
8. **Wide mode for free-format source** — the extractor converts free-format
|
||||
COBOL to fixed by left-padding 7 columns and plants a `CGWIDE` sentinel in
|
||||
the first line's sequence area; the scanner then relaxes the column-72
|
||||
right margin (free-format lines routinely exceed it). One byte of scanner
|
||||
state, carried through serialize/deserialize. Fixed-format files are
|
||||
untouched.
|
||||
9. **COBOL-2002 / GnuCOBOL surface**: `BINARY-LONG-LONG`, `FLOAT-LONG`,
|
||||
`FLOAT-SHORT` usages; `PROGRAM-ID. X IS RECURSIVE.`; relational `WHEN`
|
||||
objects (`WHEN > 0`); abbreviated combined relations
|
||||
(`WHEN X < 0 OR > Y`); bitwise operators (`B-AND`, `B-OR`, `B-XOR`,
|
||||
`B-NOT`, `B-SHIFT-L/-LC`, `B-SHIFT-R/-RC`); the `FREE` statement;
|
||||
`VALUE <constant-name>`; `PIC X(CONSTANT-NAME)`; `>>` compiler
|
||||
directives as comments; `ENTRY 'literal' USING ...` (IMS batch
|
||||
alternate entry points); empty `DATE-COMPILED.` / `DATE-WRITTEN.` /
|
||||
`SECURITY.` headers.
|
||||
10. **`CALL ... GIVING`** — upstream *intended* to support it but a misnested
|
||||
`field()` call swallowed the GIVING alternative entirely.
|
||||
|
||||
## Measured parse health (at vendoring time)
|
||||
|
||||
| Corpus | Clean parses |
|
||||
|---|---|
|
||||
| AWS CardDemo, all programs incl. DB2/IMS/MQ variants (44 `.cbl`) | 43/44 — upstream: 9/31 on the base set alone. The one residual (a period-less `EXEC SQL INCLUDE` between paragraphs) is repaired by the extractor's preParse, which terminates the single-line form with a period. |
|
||||
| AWS CardDemo copybooks (29 `.cpy`) | 28/29 — upstream: 0/29 (the failure is a `COPY REPLACING` template containing `(TESTVAR1)` placeholders, not valid COBOL pre-substitution) |
|
||||
| NIST COBOL85 suite (382 programs) | 373/382 — unchanged from upstream |
|
||||
| CobolCraft (17 free-format GnuCOBOL programs, via wide mode) | 17/17 — upstream: unparseable (free format) |
|
||||
| Upstream corpus tests | 1 pre-existing failure (`comment`), no new failures |
|
||||
|
||||
|
||||
## Rebuild
|
||||
|
||||
```bash
|
||||
git clone https://github.com/yutaro-sakamoto/tree-sitter-cobol
|
||||
cd tree-sitter-cobol
|
||||
git checkout e99dbdc3d800d5fa2796476efd60af91f6b43d93
|
||||
git apply path/to/tree-sitter-cobol.patch
|
||||
# tree-sitter 0.24.x needs a tree-sitter.json; grammar name must stay COBOL
|
||||
# (the C symbols are tree_sitter_COBOL*):
|
||||
cat > tree-sitter.json <<'JSON'
|
||||
{
|
||||
"grammars": [
|
||||
{ "name": "COBOL", "camelcase": "COBOL", "scope": "source.cobol",
|
||||
"path": ".", "file-types": ["cbl", "cob", "cpy"] }
|
||||
],
|
||||
"metadata": { "version": "0.1.1", "license": "MIT",
|
||||
"description": "COBOL grammar for tree-sitter",
|
||||
"links": { "repository": "https://github.com/yutaro-sakamoto/tree-sitter-cobol" } }
|
||||
}
|
||||
JSON
|
||||
npm install tree-sitter-cli@0.24.5
|
||||
npx tree-sitter generate
|
||||
npx tree-sitter build --wasm -o tree-sitter-cobol.wasm # needs emscripten or Docker
|
||||
```
|
||||
|
||||
The patches are written to be upstreamable — each is independent and comes
|
||||
with the failing construct documented above.
|
||||
|
||||
## Upstreaming (prepared, not sent)
|
||||
|
||||
`git apply tree-sitter-cobol.patch` on upstream commit `e99dbdc3` reproduces
|
||||
the fork exactly, so the patch file IS the prepared series. Draft PR body,
|
||||
ready to post to yutaro-sakamoto/tree-sitter-cobol once approved:
|
||||
|
||||
> **Parse real-world CICS/DB2 and GnuCOBOL sources**
|
||||
>
|
||||
> This adds the constructs that block the grammar on production COBOL, found
|
||||
> while integrating it into a code-indexing tool. Measured on public corpora:
|
||||
> AWS CardDemo goes from 9/31 to 43/44 clean parses, CobolCraft (free-format
|
||||
> GnuCOBOL) from 0 to 17/17, NIST COBOL85 unchanged at 373/382, and the
|
||||
> existing corpus tests keep their single pre-existing failure (`comment`).
|
||||
>
|
||||
> - `EXEC ... END-EXEC` blocks as an external-scanner token (`exec_statement`)
|
||||
> - Fixed-format single-quote string continuation + doubled-quote escapes
|
||||
> - `COPY ... REPLACING ==pseudo-text==` with multiple pairs
|
||||
> - `FD`/`LINKAGE SECTION` record descriptions supplied via `COPY`
|
||||
> - `NOT=`, `CALL ... GIVING` (a misnested `field()` dropped it), `FREE`,
|
||||
> `ENTRY`, `PROGRAM-ID ... IS RECURSIVE`, empty `DATE-COMPILED.` headers
|
||||
> - `FUNCTION <name>(refmod)`, `VALUE <constant>`, `PIC X(CONSTANT)`
|
||||
> - Relational and abbreviated-combined `WHEN` objects, bitwise `B-*` ops,
|
||||
> `>>` directives-as-comments, COBOL-2002 usages (`BINARY-LONG-LONG`, ...)
|
||||
> - A `copybook_fragment` entry point so standalone `.cpy` files parse
|
||||
> - An opt-in wide mode (sentinel in the first line's sequence area) that a
|
||||
> free-format preprocessor can use to relax the column-72 margin
|
||||
>
|
||||
> Happy to split any of these out or adjust naming/style. Each item is
|
||||
> independent; `tree-sitter test` passes minus the one pre-existing failure.
|
||||
|
||||
The maintainer may prefer separate PRs per item — the diff hunks split
|
||||
cleanly along the list above.
|
||||
Reference in New Issue
Block a user