grammars-v4

# Claude Notes

@GLOSSARY.md

## Trash Toolkit

The Trash Toolkit is a collection of .NET tools for working with Antlr4 grammars.
Source and documentation: https://github.com/kaby76/Trash

The tools are declared in `.config/dotnet-tools.json` at the repo root and must be
restored once before use (or after a fresh clone):

```sh
cd <repo-root>       # i.e. the grammars-v4 clone root
dotnet tool restore
```

This installs all `tr*` tools (`trgen`, `trwdog`, `trperf`, `trglob`, `triconv`,
`trparse`, `trquery`, `trxml2`, etc.) as local .NET tools available via `dotnet <toolname>`.

## Testing a Grammar

To test a grammar (e.g. `csharp/v8-spec`), follow these steps from the repo root.

**1. Restore the Trash Toolkit (once per clone), and note location:**

```sh
dotnet tool restore
cloneroot=`pwd`
```

**2. Generate the C# sandbox:**

```sh
cd csharp/v8-spec
dotnet trash gen -t CSharp
```

`trgen` reads `desc.xml` in the current directory for grammar and test configuration,
discovers all `.g4` files, resolves their dependencies, and writes the complete driver
application into `Generated-CSharp/`.

> **trgen rules:**
> - Only ever use `-t <target>` — no other options (no `--template-sources-directory`, etc.).
> - Always run `dotnet trash gen` from the directory that contains `desc.xml` (the grammar dir).
> - Never run it from the repo root or any other directory.
> - For Java: `dotnet trash gen -t Java` → generates `Generated-Java/`.

**3. Build the driver:**

```sh
cd Generated-CSharp
bash build.sh
```

`build.sh` runs `antlr4` to regenerate the lexer/parser C# sources from the `.g4`
files, then `dotnet build` to compile the driver. You will need to install prerequisites
for building and compiling the generated driver.

**4. Run the test suite, and test individual parses:**

```sh
bash test.sh
```

`test.sh` parses all example files listed in `desc.xml`, writes `.tree` and `.errors`
files alongside the inputs, and diffs them against the committed baselines. It prints
`Test succeeded.` on success or `Test failed.` with a diff on failure.

```sh
bash run.sh *.txt
```

`run.sh` parses individual files.

## Maven Antlr4 Tester (old/Java-based)

Each grammar directory contains a `pom.xml` that uses the `antlr4test-maven-plugin`.
For the `pom.xml` format, see https://maven.apache.org/guides/introduction/introduction-to-the-pom.html.

**Test a single grammar:**

```sh
cd <grammar-dir>   # directory containing pom.xml
mvn clean test
```

**Test all grammars:**

```sh
cd <repo-root>
mvn clean test
```

**Test only grammars changed in a PR:**

```sh
_scripts/maven.sh <before-git-sha> <after-git-sha>
```

### Common Maven test failure: "no errors found, but errors file exists"

This happens when zero-length `.errors` files (generated by the Trash Toolkit test runner)
are left in the examples directory. The Maven plugin treats any `.errors` file as a baseline
that expects parse failures, so if parsing succeeds it fails the test.

Fix: delete only the zero-length `.errors` files from the test directory (check `pom.xml` for
`<exampleFiles>` to confirm the directory; default is `examples/`):

```sh
find examples/ -name "*.errors" -size 0 -delete
```

This is safer than `git clean -f .`, which would remove *all* untracked files (including any
new test inputs you've added but not yet staged with `git add`).

### Common Maven test failure: unexpected file extensions in examples/

If people add test inputs or documentation into `examples/` with extensions that the plugin
doesn't expect, the plugin may try to parse them and fail. This is caused by not specifying
`<testFileExtension>` in `pom.xml`. Check the `pom.xml` for that element; if it's absent,
the plugin will attempt to parse every file in the examples directory.

## Checking for Ambiguity (Trash Toolkit / C# target)

The generated C# test harness is in `.../Generated-CSharp/`.

**Build** (only needed once, or after grammar changes):

```sh
cd Generated-CSharp
bash build.sh
```

**Check a single file for ambiguity:**

Using the built binary directly:

```sh
./bin/Debug/net10.0/Test.exe --ambig <file>
```

The `--ambig` flag enables ANTLR's profiling mode. After parsing, it reports every
ATN decision where more than one alternative was viable for the actual input (a true
grammar ambiguity). For each ambiguity it prints all distinct parse trees, one per
line, prefixed with `d=<decision>.a=<alt>` (e.g. `d=195.a=1`, `d=195.a=2`).

**Restrict output to specific decisions:**

```sh
./bin/Debug/net10.0/Test.exe --ambig=195,42 <file>
```

**Limit the number of parse trees returned:**

```sh
./bin/Debug/net10.0/Test.exe --ambig --limit=4 <file>
```

**Using trparse/trtree to show ambiguities:**

```sh
dotnet trash parse --ambig <file> | dotnet trash tree
```