which once initialized comprises a number of elements. This UI element is responsible for the display, and the generation of edit events. 1. The editor relies on a Session, which manages events and configuration. 1. The code itself is stored in a Document. Any insertion or deletion of text is reflected in the Document. 1. Keyword highlighting, indentation and brace matching are delegated to a mode. There is no direct equivalent of an ACE mode in ANTLR. While keywords are the equivalent of ANTLR lexer tokens, indentation and brace matching are edit tasks, not parsing ones. A given ACE editor can only have one mode, which corresponds to the language being edited. There is no need for ANTLR integration to support keyword highlighting, indentation and brace matching. 1. Syntax checking is delegated to a worker. This is where ANTLR integration is needed. If syntax checking is enabled, ACE asks the mode to create a worker. In JavaScript, workers run in complete isolation i.e. they don't share code or variables with other workers, or with the HTML page itself. 1. The below diagram describes how the whole system works. In green are the components *you* need to provide. You'll notice that there is no need to load ANTLR in the HTML page itself. You'll also notice that ACE maintains a document in each thread. This is done through low level events sent by the ACE session to the worker which describe the delta. Once applied to the worker document, a high level event is triggered, which is easy to handle since at this point the worker document is a perfect copy of the UI document.

## Step-by-step guide The first thing to do is to create an editor in your html page. This is thoroughly described in the ACE documentation, so we'll just sum it up here: ```xml ``` This should give you a working editor. You may want to control its sizing using CSS. I personally load the editor in an iframe and set its style to position: absolute, top: 0, left: 0 etc... but I'm sure you know better than me how to achieve results. The second thing to do is to configure the ACE editor to use your mode i.e. language configuration. A good place to start is to inherit from the built-in TextMode. The following is a very simple example, which only caters for comments, literals, and a limited subset of separators and keywords : ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Now if you store the above in a file called "my-mode.js", setting the ACE Editor becomes straightforward: ```xml ``` At this point you should have a working editor, able to highlight keywords. You may wonder why you need to set the tokens when you have already done so in your ANTLR lexer grammar. First, ACE expects a classification (control, operator, type...) which does not exist in ANTLR. Second, there is no need for ANTLR to achieve this, since ACE comes with its own lexer. Ok, now that we have a working editor comes the time where we need syntax validation. This is where the worker comes in the picture. Creating the worker is the responsibility of the mode you provide. So you need to enhance it with something like the following: ```javascript var WorkerClient = require("ace/worker/worker_client").WorkerClient; this.createWorker = function(session) { this.$worker = new WorkerClient(["ace"], "ace/worker/my-worker", "MyWorker", "../js/my-worker.js"); this.$worker.attachToDocument(session.getDocument()); this.$worker.on("errors", function(e) { session.setAnnotations(e.data); }); this.$worker.on("annotate", function(e) { session.setAnnotations(e.data); }); this.$worker.on("terminate", function() { session.clearAnnotations(); }); return this.$worker; }; ``` The above code needs to be placed in the existing worker, after: ```javascript this.$id = "ace/mode/my-mode"; ``` Please note that the mode code runs on the UI side, not the worker side. The event handlers here are for events sent by the worker, not to the worker. Obviously the above won't work out of the box, because you need to provide the "my-worker.js" file. Creating a worker from scratch is not something I've tried. Simply put, your worker needs to handle all messages sent by ACE using the WorkerClient created by the mode. This is not a simple task, and is better delegated to existing ACE code, so we can focus on tasks specific to our language. What I did is I started from "mode-json.js", a rather simple worker which comes with ACE, stripped out all JSON validation related stuff out of it, and saved the remaining code in a file name "worker-base.js" which you can find [here](resources/worker-base.js). Once this done, I was able to create a simple worker, as follows: ```javascript importScripts("worker-base.js"); ace.define('ace/worker/my-worker',["require","exports","module","ace/lib/oop","ace/worker/mirror"], function(require, exports, module) { "use strict"; var oop = require("ace/lib/oop"); var Mirror = require("ace/worker/mirror").Mirror; var MyWorker = function(sender) { Mirror.call(this, sender); this.setTimeout(200); this.$dialect = null; }; oop.inherits(MyWorker, Mirror); (function() { this.onUpdate = function() { var value = this.doc.getValue(); var annotations = validate(value); this.sender.emit("annotate", annotations); }; }).call(MyWorker.prototype); exports.MyWorker = MyWorker; }); var validate = function(input) { return [ { row: 0, column: 0, text: "MyMode says Hello!", type: "error" } ]; }; ``` At this point, you should have an editor which displays an error icon next to the first line. When you hover over the error icon, it should display: MyMode says Hello!. Is that not a friendly worker? Yum. What remains to be done is have our validate function actually validate the input. Finally ANTLR comes in the picture! To start with, let's load ANTLR and your parser, listener etc.. The preferred approach for loading parser code is to bundle your parser, [as described here](javascript-target.md). You can then load it as part of the importScripts instruction at the start of your worker code. Another approach is to load it using 'require'. Easy, since you could write: ```js var antlr4 = require('antlr4/index'); ``` This may work, but it's actually unreliable. The reason is that the 'require' function that comes with ACE uses a different syntax than the 'require' function used by ANTLR, which follows the NodeJS 'require' convention. So we need to bring in a NodeJS compatible 'require' function that conforms to the NodeJS syntax. I personally use one that comes from Torben Haase's Honey project, which you can find in li/require.js. But hey, now we're going to have 2 'require' functions not compatible with each other! Indeed, this is why you need to take special care, as follows: ```js // load nodejs compatible require var ace_require = require; require = undefined; var Honey = { 'requirePath': ['..'] }; // walk up to js folder, see Honey docs importScripts("../lib/require.js"); var antlr4_require = require; require = ace_require; ``` Now it's safe to load antlr and the parsers generated for your language. Assuming that your language files (generated or hand-built) are in a folder with an index.js file that calls require for each file, your parser loading code can be as simple as follows: ```js // load antlr4 and myLanguage var antlr4, mylanguage; try { require = antlr4_require; antlr4 = require('antlr4/index'); mylanguage = require('mylanguage/index'); } finally { require = ace_require; } ``` Please note the try-finally construct. ANTLR uses 'require' synchronously so it's perfectly safe to ignore the ACE 'require' while running ANTLR code. ACE itself does not guarantee synchronous execution, so you are much safer always switching 'require' back to 'ace_require'. Now detecting deep syntax errors in your code is a task for your ANTLR listener or visitor or whatever piece of code you've delegated this to. We're not going to describe this here, since it would require some knowledge of your language. However, detecting grammar syntax errors is something ANTLR does beautifully (isn't that why you went for ANTLR in the first place?). So what we will illustrate here is how to report grammar syntax errors. I have no doubt that from there, you will be able to extend the validator to suit your specific needs. Whenever ANTLR encounters an unexpected token, it fires an error. By default, the error is routed to an error listener which simply writes to the console. What we need to do is replace this listener by our own listener, se we can route errors to the ACE editor. First, let's create such a listener: ```js // class for gathering errors and posting them to ACE editor var AnnotatingErrorListener = function(annotations) { antlr4.error.ErrorListener.call(this); this.annotations = annotations; return this; }; AnnotatingErrorListener.prototype = Object.create(antlr4.error.ErrorListener.prototype); AnnotatingErrorListener.prototype.constructor = AnnotatingErrorListener; AnnotatingErrorListener.prototype.syntaxError = function(recognizer, offendingSymbol, line, column, msg, e) { this.annotations.push({ row: line - 1, column: column, text: msg, type: "error" }); }; ``` With this, all that remains to be done is plug the listener in when we parse the code. Here is how I do it: ```js var validate = function(input) { var stream = CharStreams.fromString(input); var lexer = new mylanguage.MyLexer(stream); var tokens = new antlr4.CommonTokenStream(lexer); var parser = new mylanguage.MyParser(tokens); var annotations = []; var listener = new AnnotatingErrorListener(annotations) parser.removeErrorListeners(); parser.addErrorListener(listener); parser.parseMyRule(); return annotations; }; ``` You know what? That's it! You now have an ACE editor that does syntax validation using ANTLR! I hope you find this useful, and simple enough to get started. Now wait, hey! How do you debug this? Well, as usual, using Chrome, since no other browser is able to debug worker code. What a shame... --- ### Doc Actions (`doc/actions.md`) # Actions and Attributes In Chapter 10, Attributes and Actions, we learned how to embed actions within grammars and looked at the most common token and rule attributes. This section summarizes the important syntax and semantics from that chapter and provides a complete list of all available attributes. (You can learn more about actions in the grammar from the free excerpt on listeners and actions.) Actions are blocks of text written in the target language and enclosed in curly braces. The recognizer triggers them according to their locations within the grammar. For example, the following rule emits "found a decl" after the parser has seen a valid declaration: ``` decl: type ID ';' {System.out.println("found a decl");} ; type: 'int' | 'float' ; ``` Most often, actions access the attributes of tokens and rule references: ``` decl: type ID ';' {System.out.println("var "+$ID.text+":"+$type.text+";");} | t=ID id=ID ';' {System.out.println("var "+$id.text+":"+$t.text+";");} ; ``` ## Token Attributes All tokens have a collection of predefined, read-only attributes. The attributes include useful token properties such as the token type and text matched for a token. Actions can access these attributes via `$label.attribute` where label labels a particular instance of a token reference (`a` and `b` in the example below are used in the action code as `$a` and `$b`). Often, a particular token is only referenced once in the rule, in which case the token name itself can be used unambiguously in the action code (token `INT` can be used as `$INT` in the action). The following example illustrates token attribute expression syntax: ``` r : INT {int x = $INT.line;} ( ID {if ($INT.line == $ID.line) ...;} )? a=FLOAT b=FLOAT {if ($a.line == $b.line) ...;} ; ``` The action within the `(...)?` subrule can see the `INT` token matched before it in the outer level. Because there are two references to the `FLOAT` token, a reference to `$FLOAT` in an action is not unique; you must use labels to specify which token reference you’re interested in. Token references within different alternatives are unique because only one of them can be matched for any invocation of the rule. For example, in the following rule, actions in both alternatives can reference `$ID` directly without using a label: ``` r : ... ID {System.out.println($ID.text);} | ... ID {System.out.println($ID.text);} ; ``` To access the tokens matched for literals, you must use a label: ``` stat: r='return' expr ';' {System.out.println("line="+$r.line);} ; ``` Most of the time you access the attributes of the token, but sometimes it is useful to access the Token object itself because it aggregates all the attributes. Further, you can use it to test whether an optional subrule matched a token: ``` stat: 'if' expr 'then' stat (el='else' stat)? {if ( $el!=null ) System.out.println("found an else");} | ... ; ``` `$T` and `$L` evaluate to `Token` objects for token name `T` and token label `L`. `$ll` evaluates to `List` for list label `ll`. `$T.attr` evaluates to the type and value specified in the following table for attribute `attr`: |Attribute|Type|Description| |---------|----|-----------| |text|String|The text matched for the token; translates to a call to getText. Example: $ID.text.| |type|int|The token type (nonzero positive integer) of the token such as INT; translates to a call to getType. Example: $ID.type.| |line|int|The line number on which the token occurs, counting from 1; translates to a call to getLine. Example: $ID.line.| |pos|int|The character position within the line at which the token’s first character occurs counting from zero; translates to a call to getCharPositionInLine. Example: $ID.pos.| |index|int|The overall index of this token in the token stream, counting from zero; translates to a call to getTokenIndex. Example: $ID.index.| |channel|int|The token’s channel number. The parser tunes to only one channel, effectively ignoring off-channel tokens. The default channel is 0 (Token.DEFAULT_CHANNEL), and the default hidden channel is Token.HIDDEN_CHANNEL. Translates to a call to getChannel. Example: $ID.channel.| |int|int|The integer value of the text held by this token; it assumes that the text is a valid numeric string. Handy for building calculators and so on. Translates to Integer.valueOf(text-of-token). Example: $INT.int.| ## Parser Rule Attributes ANTLR predefines a number of read-only attributes associated with parser rule references that are available to actions. Actions can access rule attributes only for references that precede the action. The syntax is `$r.attr` for rule name `r` or a label assigned to a rule reference. For example, `$expr.text` returns the complete text matched by a preceding invocation of rule `expr`: ``` returnStat : 'return' expr {System.out.println("matched "+$expr.text);} ; ``` Using a rule label looks like this: ``` returnStat : 'return' e=expr {System.out.println("matched "+$e.text);} ; ``` You can also use `$` followed by the name of the attribute to access the value associated with the currently executing rule. For example, `$start` is the starting token of the current rule. ``` returnStat : 'return' expr {System.out.println("first token "+$start.getText());} ; ``` `$r` and `$rl` evaluate to `ParserRuleContext` objects of type `RContext` for rule name `r` and rule label `rl`. `$rll` evaluates to `List` for rule list label `rll`. `$r.attr` evaluates to the type and value specified in the following table for attribute `attr`: |Attribute|Type|Description| |---------|----|-----------| |text|String|The text matched for a rule or the text matched from the start of the rule up until the point of the `$text` expression evaluation. Note that this includes the text for all tokens including those on hidden channels, which is what you want because usually that has all the whitespace and comments. When referring to the current rule, this attribute is available in any action including any exception actions.| |start|Token|The first token to be potentially matched by the rule that is on the main token channel; in other words, this attribute is never a hidden token. For rules that end up matching no tokens, this attribute points at the first token that could have been matched by this rule. When referring to the current rule, this attribute is available to any action within the rule.| |stop|Token|The last nonhidden channel token to be matched by the rule. When referring to the current rule, this attribute is available only to the after and finally actions.| |ctx|ParserRuleContext|The rule context object associated with a rule invocation. All of the other attributes are available through this attribute. For example, `$ctx.start` accesses the start field within the current rules context object. It’s the same as `$start`.| |parser|Parser|The parser itself. This attribute can be used, for example, to invoke a method defined in the parser's `@members` section from a semantic predicate.| ## Dynamically-Scoped Attributes You can pass information to and from rules using parameters and return values, just like functions in a general-purpose programming language. Programming languages don’t allow functions to access the local variables or parameters of invoking functions, however. For example, the following reference to local variable `x` form a nested method call is illegal in Java: ```java void f() { int x = 0; g(); } void g() { h(); } void h() { int y = x; // INVALID reference to f's local variable x } ``` Variable `x` is available only within the scope of `f`, which is the text lexically delimited by curly brackets. For this reason, Java is said to use lexical scoping. Lexical scoping is the norm for most programming languages. Languages that allow methods further down in the call chain to access local variables defined earlier are said to use dynamic scoping. The term dynamic refers to the fact that a compiler cannot statically determine the set of visible variables. This is because the set of variables visible to a method changes depending on who calls that method. It turns out that, in the grammar realm, distant rules sometimes need to communicate with each other, mostly to provide context information to rules matched below in the rule invocation chain. (Naturally, this assumes that you are using actions directly in the grammar instead of the parse-tree listener event mechanism.) ANTLR allows dynamic scoping in that actions can access attributes from invoking rules using syntax `$r::x` where `r` is a rule name and `x` is an attribute within that rule. It is up to the programmer to ensure that `r` is in fact an invoking rule of the current rule. A runtime exception occurs if `r` is not in the current call chain when you access `$r::x`. To illustrate the use of dynamic scoping, consider the real problem of defining variables and ensuring that variables in expressions are defined. The following grammar defines the symbols attribute where it belongs in the block rule but adds variable names to it in rule `decl`. Rule `stat` then consults the list to see whether variables have been defined. ``` grammar DynScope; prog: block ; block /* List of symbols defined within this block */ locals [ List symbols = new ArrayList() ] : '{' decl* stat+ '}' // print out all symbols found in block // $block::symbols evaluates to a List as defined in scope {System.out.println("symbols="+$symbols);} ; /** Match a declaration and add identifier name to list of symbols */ decl: 'int' ID {$block::symbols.add($ID.text);} ';' ; /** Match an assignment then test list of symbols to verify * that it contains the variable on the left side of the assignment. * Method contains() is List.contains() because $block::symbols * is a List. */ stat: ID '=' INT ';' { if ( !$block::symbols.contains($ID.text) ) { System.err.println("undefined variable: "+$ID.text); } } | block ; ID : [a-z]+ ; INT : [0-9]+ ; WS : [ \t\r\n]+ -> skip ; ``` Here’s a simple build and test sequence: ```bash $ antlr4 DynScope.g4 $ javac DynScope*.java $ grun DynScope prog => { => int i; => i = 0; => j = 3; => } => EOF <= undefined variable: j symbols=[i] ``` There’s an important difference between a simple field declaration in a `@members` action and dynamic scoping. symbols is a local variable and so there is a copy for each invocation of rule `block`. That’s exactly what we want for nested blocks so that we can reuse the same input variable name in an inner block. For example, the following nested code block redefines `i` in the inner scope. This new definition must hide the definition in the outer scope. ``` { int i; int j; i = 0; { int i; int x; x = 5; } x = 3; } ``` Here’s the output generated for that input by DynScope: ```bash $ grun DynScope prog nested-input symbols=[i, x] undefined variable: x symbols=[i, j] ``` Referencing `$block::symbols` accesses the `symbols` field of the most recently invoked `block`’s rule context object. If you need access to a symbols instance from a rule invocation farther up the call chain, you can walk backwards starting at the current context, `$ctx`. Use `getParent` to walk up the chain. --- ### Doc Antlr Project Testing (`doc/antlr-project-testing.md`) # ANTLR project unit tests ## Introduction Because ANTLR supports multiple target languages, the unit tests are broken into two groups: the unit tests that test the tool itself (in `tool-testsuite`) and the unit tests that test the parser runtimes (in `antlr4/runtime-testsuite`). The tool tests are straightforward because they are Java code testing Java code; see the section at the bottom of this file. The runtime tests must be specified in a generic fashion to work across language targets. Furthermore, the various targets from Java must be tested. This usually means Java launching processes to compile, say, C++ and run parsers. As of 4.10, a Java descriptor file held as an [RuntimeTestDescriptor.java](../runtime-testsuite/test/org/antlr/v4/test/runtime/RuntimeTestDescriptor.java) is used to represent each runtime test. Each test is described with a text file with various sections and resides in a group directory; see [directories under descriptors' dir](../runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors). Here is a sample test descriptor: ``` [notes] This is a regression test for blah blah blah... [type] Parser [grammar] grammar T; a : ID* { }; ID : 'a'..'z'+; WS : (' '|'\n') -> skip; [start] a [input] a b c [output] """abc """ ``` The grammars are strings representing StringTemplates (`ST` objects) so `` will get replace when the unit test file is generated (`Test.java`, `Test.cs`, ...). The `writeln` template must be defined per target. Here are all the [Target templates for runtime tests](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates). Use triple-quotes `"""` when whitespace matters (usually input/output sections). ## Requirements In order to perform the tests on all target languages, the following tools should be installed: * dotnet * Node.js * Python 3 * Go * Swift * Clang (Linux, Mac) or MSBuild (Windows) for C++ * Dart * PHP To **install into local repository** `~/.m2/repository/org/antlr`, do this: ```bash $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux $ mvn install -DskipTests # make sure all artifacts are visible on this machine ``` ## Running the runtime tests A single test rig is sufficient to test all targets against all descriptors using the [junit dynamic tests](https://junit.org/junit5/docs/current/user-guide/#writing-tests-dynamic-tests) mechanism. But it's often convenient to test just a single target or perhaps even just a single test within a single group of a single target. IntelliJ automatically generates a bunch of [Target runtime test rigs](../runtime-testsuite/test/org/antlr/v4/test/runtime) that allows developers such flexibility. For example, here are the Python3 test rigs in IntelliJ: And the result of testing the entire subdirectory: All test are run in parallel both via maven and via IDE. In IntelliJ, it's very easy to go to source by right-clicking on any test and pressing `Jump to source` (F4). ## Running test subsets From the `runtime-testsuite` dir ### Run all tests for a single target ```bash $ cd runtime-testsuite $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux $ mvn -Dtest='java.**' test ------------------------------------------------------- T E S T S ------------------------------------------------------- [INFO] Running org.antlr.v4.test.runtime.java.TestIntegerList [INFO] Running org.antlr.v4.test.runtime.java.JavaRuntimeTests ... [INFO] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.023 s - in org.antlr.v4.test.runtime.java.TestIntegerList [INFO] Tests run: 348, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.269 s - in org.antlr.v4.test.runtime.java.JavaRuntimeTests ... ``` ## Adding a runtime test To add a new runtime test, first determine which [group (dir) of tests](../runtime-testsuite/resources/org/antlr/v4/test/runtime/descriptors) it belongs to. Then, add a new descriptor file implementation by filling in one of these (omitting unused sections): ``` [notes] [type] [grammar] [slaveGrammar] [start] [input] [output] [errors] [flags] [skip] ``` Your best bet is to find a similar test in the appropriate group and then copy and paste the descriptor file, creating a new file within the test group dir. Modify the sections to suit your new problem. ### Ignoring tests In order to turn off a test for a particular target, the `skip` section in the descriptor file should be used. For example, the following skips PHP and Dart targets: ``` [skip] PHP Dart ``` ### Target API/library testing Some parts of the runtime API need to be tested with code written specifically in the target language. For example, all the Java runtime API tests are placed here: [runtime-testsuite/test/org/antlr/v4/test/runtime/java/api](../runtime-testsuite/test/org/antlr/v4/test/runtime/java/api) Notice that it is under an `api` dir. The directory above is where all of the `*Test*` files go. ### Cross-language actions embedded within grammars To get: ``` System.out.println($set.stop); ``` Use instead the language-neutral: ``` ``` Template file [Java.test.stg](../runtime-testsuite/resources/org/antlr/v4/test/runtime/templates/Java.test.stg) has templates like: ``` writeln(s) ::= <);>> ``` that translate generic operations to target-specific language statements or expressions. ## Adding an ANTLR tool unit test Just go into the appropriate Java test class in dir [antlr4/tool-testsuite/test/org/antlr/v4/test/tool](../tool-testsuite/test/org/antlr/v4/test/tool) and add your unit test. --- ### Doc Building Antlr (`doc/building-antlr.md`) # Building ANTLR Most programmers do not need the information on this page because they will simply download the appropriate jar(s) or use ANTLR through maven (via ANTLR's antlr4-maven-plugin). If you would like to fork the project and fix bugs or tweak the runtime code generation, then you will almost certainly need to build ANTLR itself. There are two components: 1. the tool that compiles grammars down into parsers and lexers in one of the target languages 1. the runtime used by those generated parsers and lexers. I will assume that the root directory is `/tmp` for the purposes of explaining how to build ANTLR in this document. *As of 4.6, ANTLR tool and Java-target runtime requires Java 7. As of 4.10, we have verified that the tool itself builds with Java 8 and 11.* # Get the source The first step is to get the Java source code from the ANTLR 4 repository at github. You can download the repository from github, but the easiest thing to do is simply clone the repository on your local disk: ```bash $ cd /tmp /tmp $ git clone https://github.com/antlr/antlr4.git Cloning into 'antlr4'... remote: Counting objects: 61480, done. remote: Total 61480 (delta 0), reused 0 (delta 0), pack-reused 61480 Receiving objects: 100% (61480/61480), 31.24 MiB | 7.18 MiB/s, done. Resolving deltas: 100% (32970/32970), done. Checking connectivity... done. Checking out files: 100% (1427/1427), done. ``` # Check your environment If you are starting from a clean, minimum Ubuntu OS, check your environment. ```bash $ sudo apt-get update $ # Get Java $ java > /dev/null 2>&1 $ if [[ "$?" != "0" ]]; then sudo apt install -y openjdk-11-jre-headless; fi $ # Get Mvn $ mvn > /dev/null 2>&1 $ if [[ "$?" != "0" ]]; then sudo apt install -y maven; fi ``` # Compile The current maven build seems complicated to me because there is a dependency of the project on itself. The runtime tests naturally depend on the current version being available but it won't compile without the current version. Once you have the generated/installed jar, mvn builds but otherwise there's a dependency on what you are going to build. You will get this error when you try to clean but you can ignore it: ``` [INFO] ANTLR 4 Runtime Tests (4th generation) ............. FAILURE [ 0.073 s] ... [ERROR] Plugin org.antlr:antlr4-maven-plugin:4.10-SNAPSHOT or one of its dependencies could not be resolved: Could not find artifact org.antlr:antlr4-maven-plugin:jar:4.10-SNAPSHOT -> [Help 1] ``` To be super squeaky clean, you can wipe out the repository cache, then do the build: ``` $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux cd /tmp/antlr4 # or wherever you have the software rm -rf ~/.m2/repository/org/antlr* mvn clean mvn -DskipTests install ``` **NOTE:** We do `install` not `compile` as tool tests and such refer to modules that must be pulled from the maven install local cache. Once you have completed this process once and there is a jar hanging around in the repository cache. # Installing libs to mvn cache locally To skip the tests (which require all the target languages be installed) and **install into local repository** `~/.m2/repository/org/antlr`, do this: ```bash $ export MAVEN_OPTS="-Xmx1G" # don't forget this on linux $ mvn install -DskipTests # make sure all artifacts are visible on this machine ``` You should see these jars (when building 4.6-SNAPSHOT): ```bash /Users/parrt/.m2/repository/org/antlr $ find antlr4* -name '*.jar' antlr4-maven-plugin/4.6-SNAPSHOT/antlr4-maven-plugin-4.6-SNAPSHOT.jar antlr4-runtime-testsuite/4.6-SNAPSHOT/antlr4-runtime-testsuite-4.6-SNAPSHOT-tests.jar antlr4-runtime-testsuite/4.6-SNAPSHOT/antlr4-runtime-testsuite-4.6-SNAPSHOT.jar antlr4-runtime/4.6-SNAPSHOT/antlr4-runtime-4.6-SNAPSHOT.jar antlr4-tool-testsuite/4.6-SNAPSHOT/antlr4-tool-testsuite-4.6-SNAPSHOT.jar antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT-tests.jar antlr4/4.6-SNAPSHOT/antlr4-4.6-SNAPSHOT.jar ``` Note that ANTLR is written in itself, which is why maven downloads antlr4-4.5.jar for boostrapping 4.6-SNAPSHOT purposes. # Testing tool and targets See [ANTLR project unit tests](antlr-project-testing.md). # Building without testing To build without running the tests (saves a lot of time), do this: ```bash $ mvn -DskipTests install ``` ## Building ANTLR in Intellij IDE After download ANTLR source, just "import project from existing sources" and click on the "Maven Projects" tab in right gutter of IDE. It should build stuff in the background automatically and look like:

--- ### Doc Cpp Target (`doc/cpp-target.md`) # C++ The C++ target supports all platforms that can either run MS Visual Studio 2017 (or newer), XCode 7 (or newer) or CMake (C++17 required). All build tools can either create static or dynamic libraries, both as 64bit or 32bit arch. Additionally, XCode can create an iOS library. Also see [Antlr4 for C++ with CMake: A practical example](http://blorente.me/beyond-the-loop/Antlr-cpp-cmake/). ## How to create a C++ lexer or parser? This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example: ``` $ antlr4 -Dlanguage=Cpp MyGrammar.g4 ``` You will see that there are a whole bunch of files generated by this call. If visitor or listener are not suppressed (which is the default) you'll get: * MyGrammarLexer.h + MyGrammarLexer.cpp * MyGrammarParser.h + MyGrammarParser.cpp * MyGrammarVisitor.h + MyGrammarVisitor.cpp * MyGrammarBaseVisitor.h + MyGrammarBaseVisitor.cpp * MyGrammarListener.h + MyGrammarListener.cpp * MyGrammarBaseListener.h + MyGrammarBaseListener.cpp ## Where can I get the runtime? Once you've generated the lexer and/or parser code, you need to download or build the runtime. Prebuilt C++ runtime binaries for Windows (Visual Studio 2013/2015), OSX/macOS and iOS are available on the ANTLR web site: * http://www.antlr.org Use CMake to build a Linux library (works also on OSX, however not for the iOS library). Instead of downloading a prebuilt binary you can also easily build your own library on OSX or Windows. Just use the provided projects for XCode or Visual Studio and build it. Should work out of the box without any additional dependency. ## How do I run the generated lexer and/or parser? Putting it all together to get a working parser is really easy. Look in the [runtime/Cpp/demo](../runtime/Cpp/demo) folder for a simple example. The [README](../runtime/Cpp/demo/README.md) there describes shortly how to build and run the demo on OSX, Windows or Linux. ## How do I create and run a custom listener? The generation step above created a listener and base listener class for you. The listener class is an abstract interface, which declares enter and exit methods for each of your parser rules. The base listener implements all those abstract methods with an empty body, so you don't have to do it yourself if you just want to implement a single function. Hence use this base listener as the base class for your custom listener: ```c++ #include #include "antlr4-runtime.h" #include "MyGrammarLexer.h" #include "MyGrammarParser.h" #include "MyGrammarBaseListener.h" using namespace antlr4; class TreeShapeListener : public MyGrammarBaseListener { public: void enterKey(ParserRuleContext *ctx) override { // Do something when entering the key rule. } }; int main(int argc, const char* argv[]) { std::ifstream stream; stream.open(argv[1]); ANTLRInputStream input(stream); MyGrammarLexer lexer(&input); CommonTokenStream tokens(&lexer); MyGrammarParser parser(&tokens); tree::ParseTree *tree = parser.key(); TreeShapeListener listener; tree::ParseTreeWalker::DEFAULT.walk(&listener, tree); return 0; } ``` This example assumes your grammar contains a parser rule named `key` for which the `enterKey` function was generated. ## Special cases for this ANTLR target There are a couple of things that only the C++ ANTLR target has to deal with. They are described here. ### Code Generation Aspects The code generation (by running the ANTLR4 jar) allows to specify 2 values you might find useful for better integration of the generated files into your application (both are optional): * A **namespace**: use the **`-package`** parameter to specify the namespace you want. * An **export macro**: especially in VC++ extra work is required to export your classes from a DLL. This is usually accomplished by a macro that has different values depending on whether you are creating the DLL or import it. The ANTLR4 runtime itself also uses one for its classes: ```c++ #ifdef ANTLR4CPP_EXPORTS #define ANTLR4CPP_PUBLIC __declspec(dllexport) #else #ifdef ANTLR4CPP_STATIC #define ANTLR4CPP_PUBLIC #else #define ANTLR4CPP_PUBLIC __declspec(dllimport) #endif #endif ``` Just like the `ANTLR4CPP_PUBLIC` macro here you can specify your own one for the generated classes using the **`-DexportMacro=...`** command-line parameter or grammar option `options {exportMacro='...';}` in your grammar file. In order to create a static lib in Visual Studio define the `ANTLR4CPP_STATIC` macro in addition to the project settings that must be set for a static library (if you compile the runtime yourself). For gcc and clang it is possible to use the `-fvisibility=hidden` setting to hide all symbols except those that are made default-visible (which has been defined for all public classes in the runtime). ### Compile Aspects When compiling generated files, you can configure a compile option according to your needs (also optional): * A **thread local DFA macro**: Add `-DANTLR4_USE_THREAD_LOCAL_CACHE=1` to the compilation options will enable using thread local DFA cache (disabled by default), after that, each thread uses its own DFA. This will increase memory usage to store thread local DFAs and redundant computation to build thread local DFAs (not too much). The benefit is that it can improve the concurrent performance running with multiple threads. In other words, when you find your concurent throughput is not high enough, you should consider turning on this option. ### Memory Management Since C++ has no built-in memory management we need to take extra care. For that we rely mostly on smart pointers, which however might cause time penalties or memory side effects (like cyclic references) if not used with care. Currently however the memory household looks very stable. Generally, when you see a raw pointer in code consider this as being managed elsewhere. You should never try to manage such a pointer (delete, assign to smart pointer etc.). Accordingly a parse tree is only valid for the lifetime of its parser. The parser, in turn, is only valid for the lifetime of its token stream, and so on back to the original `ANTLRInputStream` (or equivalent). To retain a tree across function calls you'll need to create and store all of these and `delete` all but the tree when you no longer need it. ### Unicode Support Encoding is mostly an input issue, i.e. when the lexer converts text input into lexer tokens. The parser is completely encoding unaware. The C++ target always expects UTF-8 input (either in a string or stream) which is then converted to UTF-32 (a char32_t array) and fed to the lexer. ### Named Actions In order to help customizing the generated files there are a number of additional so-called **named actions**. These actions are tight to specific areas in the generated code and allow to add custom (target specific) code. All targets support these actions * @parser::header * @parser::members * @lexer::header * @lexer::members (and their scopeless alternatives `@header` and `@members`) where header doesn't mean a C/C++ header file, but the top of a code file. The content of the header action appears in all generated files at the first line. So it's good for things like license/copyright information. The content of a *members* action is placed in the public section of lexer or parser class declarations. Hence it can be used for public variables or predicate functions used in a grammar predicate. Since all targets support *header* + *members* they are the best place for stuff that should be available also in generated files for other languages. In addition to that the C++ target supports many more such named actions. Unfortunately, it's not possible to define new scopes (e.g. *listener* in addition to *parser*) so they had to be defined as part of the existing scopes (*lexer* or *parser*). The grammar in the demo application contains all of the named actions as well for reference. Here's the list: * **@lexer::preinclude** - Placed right before the first #include (e.g. good for headers that must appear first, for system headers etc.). Appears in both lexer h and cpp file. * **@lexer::postinclude** - Placed right after the last #include, but before any class code (e.g. for additional namespaces). Appears in both lexer h and cpp file. * **@lexer::context** - Placed right before the lexer class declaration. Use for e.g. additional types, aliases, forward declarations and the like. Appears in the lexer h file. * **@lexer::declarations** - Placed in the private section of the lexer declaration (generated sections in all classes strictly follow the pattern: public, protected, private, from top to bottom). Use this for private vars etc. * **@lexer::definitions** - Placed before other implementations in the cpp file (but after *@postinclude*). Use this to implement e.g. private types. For the parser there are the same actions as shown above for the lexer. In addition to that there are even more actions for visitor and listener classes: * **@parser::listenerpreinclude** * **@parser::listenerpostinclude** * **@parser::listenerdeclarations** * **@parser::listenermembers** * **@parser::listenerdefinitions** * * **@parser::baselistenerpreinclude** * **@parser::baselistenerpostinclude** * **@parser::baselistenerdeclarations** * **@parser::baselistenermembers** * **@parser::baselistenerdefinitions** * * **@parser::visitorpreinclude** * **@parser::visitorpostinclude** * **@parser::visitordeclarations** * **@parser::visitormembers** * **@parser::visitordefinitions** * * **@parser::basevisitorpreinclude** * **@parser::basevisitorpostinclude** * **@parser::basevisitordeclarations** * **@parser::basevisitormembers** * **@parser::basevisitordefinitions** and should be self explanatory now. Note: there is no *context* action for listeners or visitors, simply because they would be even less used than the other actions and there are so many already. --- ### Doc Creating A Language Target (`doc/creating-a-language-target.md`) # Creating an ANTLR Language Target This document describes how to make ANTLR generate parsers in a new language, *X*. ## Overview Creating a new target involves the following key elements: 1. For the tool, create class *X*Target as a subclass of class `Target` in package `org.antlr.v4.codegen.target`. This class describes language specific details about escape characters and strings and so on. There is very little to do here typically. 2. Create `*X*.stg` in directory `tool/resources/org/antlr/v4/tool/templates/codegen/*X*/*X*.stg`. This is a [StringTemplate](http://www.stringtemplate.org/) group file (`.stg`) that tells ANTLR how to express all the parsing elements needed to generate code. You will see templates called `ParserFile`, `Parser`, `Lexer`, `CodeBlockForAlt`, `AltBlock`, etc... Each of these must be described how to build the indicated chunk of code. Your best bet is to find the closest existing target, copy that template file, and tweak to suit. 3. Create a runtime library to support the parsers generated by ANTLR. Under directory `runtime/*X*`, you are in complete control of the directory structure as dictated by common usage of that target language. For example, Java has: `runtime/Java/lib` and `runtime/Java/src` directories. Under `src`, you will find a directory structure for package `org.antlr.v4.runtime` and below. 4. Create a template file for runtime tests. All you have to do is provide a few templates that indicate how to print values and declare variables. Our runtime test mechanism in dir `runtime-testsuite` will automatically generate code using these templates for each target and check the test results. It needs to know how to define various class fields, compare members and so on. You must create a `*X*.test.stg` file underneath [runtime-testsuite/resources/org/antlr/v4/test/runtime](../runtime-testsuite/resources/org/antlr/v4/test/runtime) and `Test.*x*.stg` underneath [runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers](../runtime-testsuite/resources/org/antlr/v4/test/runtime/helpers). Again, your best bet is to copy the templates from the closest language to your target and tweak it to suit. 6. Create test files under [/runtime-testsuite/test/org/antlr/v4/test/runtime](../runtime-testsuite/test/org/antlr/v4/test/runtime). They will load defined test cases in each test descriptor. Also add the `/runtime-testsuite/test/org/antlr/v4/test/runtime/X/BaseXTest.java` which defines how test cases will execute and output. 7. Create/edit shell scripts in [/.github](../.github) to run tests in CI pipelines. ## Getting started 1. Fork the `antlr/antlr4` repository at GitHub to your own user so that you have repository `username/antlr4`. 2. Clone `username/antlr4`, the forked repository, to your local disk. Your remote `origin` will be the forked repository on GitHub. Add a remote `upstream` to the original `antlr/antlr4` repository (URL `https://github.com/antlr/antlr4.git`). Changes that you would like to contribute back to the project are done with [pull requests](https://help.github.com/articles/using-pull-requests/). 3. Try to build it before doing anything ```bash $ mvn compile ``` That should proceed with success. See [Building ANTLR](building-antlr.md) for more details. ## Comparing your target's parsing decisionmaking with Java's ANTLR's power comes from it's dynamic parsing strategy, but that means each target must implement that complicated algorithm. You should compare your target's debug output for ParserATNSimulator with Java's. Run this so we get right jars before trying this script: ``` cd ANTLR-ROOT-DIR mvn install -DskipTests=true cd runtime-tests mvn install -DskipTests=true # yes do it again ``` Run the script from `runtime-tests` dir with ``` ../scripts/traceatn.sh /tmp/JSON.g4 json -target Go /tmp/foo.json ``` or whatever your test grammar, start rule, target, test input are. ### Debugging the PHP target Because the PHP target is hosted in a separate repository, you will need to clone the [antlr/php-antlr-runtime](https://github.com/antlr/antlr-php-runtime) repository into the `runtime/PHP` and install the dependencies with `composer install` before you can run the tests. ``` git clone -b dev https://github.com/antlr/antlr-php-runtime.git runtime/PHP cd runtime/PHP composer install ``` --- ### Doc Csharp Target (`doc/csharp-target.md`) # C♯ ## Which frameworks are supported? The C# runtime is CLS compliant, and only requires a corresponding 3.5 .Net framework. In practice, the runtime has been extensively tested against: * Microsoft .Net 3.5 framework * Mono .Net 3.5 framework No issue was found, so you should find that the runtime works pretty much against any recent .Net framework. ## How do I get started? You will find full instructions on the [Git repo page for ANTLR C# runtime](https://github.com/antlr/antlr4/tree/master/runtime/CSharp). ## How do I use the runtime from my project? (i.e., How do I run the generated lexer and/or parser?) Let's suppose that your grammar is named `MyGrammar`. The tool will generate for you the following files: * MyGrammarLexer.cs * MyGrammarParser.cs * MyGrammarListener.cs (if you have not activated the -no-listener option) * MyGrammarBaseListener.cs (if you have not activated the -no-listener option) * MyGrammarVisitor.cs (if you have activated the -visitor option) * MyGrammarBaseVisitor.cs (if you have activated the -visitor option) Now a fully functioning code might look like the following for start rule `StartRule`: ```csharp using Antlr4.Runtime; using Antlr4.Runtime.Tree; public void MyParseMethod() { String input = "your text to parse here"; ICharStream stream = CharStreams.fromString(input); ITokenSource lexer = new MyGrammarLexer(stream); ITokenStream tokens = new CommonTokenStream(lexer); MyGrammarParser parser = new MyGrammarParser(tokens); IParseTree tree = parser.StartRule(); } ``` This program will work. But it won't be useful unless you do one of the following: * you visit the parse tree using a custom listener * you visit the parse tree using a custom visitor * your grammar comprises production code (like AntLR3) (please note that production code is target specific, so you can't have multi target grammars that include production code) ## How do I create and run a custom listener? Let's suppose your MyGrammar grammar comprises 2 rules: "key" and "value". The antlr4 tool will have generated the following listener (only partial code shown here): ```csharp interface IMyGrammarParserListener : IParseTreeListener { void EnterKey (MyGrammarParser.KeyContext context); void ExitKey (MyGrammarParser.KeyContext context); void EnterValue (MyGrammarParser.ValueContext context); void ExitValue (MyGrammarParser.ValueContext context); } ``` In order to provide custom behavior, you might want to create the following class: ```csharp class KeyPrinter : MyGrammarBaseListener { // override default listener behavior void ExitKey (MyGrammarParser.KeyContext context) { Console.WriteLine("Oh, a key!"); } } ``` In order to execute this listener, you would simply add the following lines to the above code: ```csharp ... IParseTree tree = parser.StartRule() - only repeated here for reference KeyPrinter printer = new KeyPrinter(); ParseTreeWalker.Default.Walk(printer, tree); ``` Further information can be found from The Definitive ANTLR Reference book. The C# implementation of ANTLR is as close as possible to the Java one, so you shouldn't find it difficult to adapt the examples for C#. See also [Sam Harwell's alternative C# target](https://github.com/tunnelvisionlabs/antlr4cs) --- ### Doc Dart Target (`doc/dart-target.md`) # ANTLR4 Runtime for Dart From version 4.9 onwards antlr's dart generated code is null sound safety compatible and sets the minimum dart sdk version to 2.12.0. ### First steps #### 1. Install ANTLR4 [The getting started guide](https://github.com/antlr/antlr4/blob/master/doc/getting-started.md) should get you started. #### 2. Install the Dart ANTLR runtime Each target language for ANTLR has a runtime package for running parser generated by ANTLR4. The runtime provides a common set of tools for using your parser. Install the runtime with the same version as the main ANTLR tool: Add this to your package's pubspec.yaml file: ```yaml ... dependencies: antlr4: ... ``` #### 3. Generate your parser You use the ANTLR4 "tool" to generate a parser. These will reference the ANTLR runtime, installed above. Suppose you're using a UNIX system and have set up an alias for the ANTLR4 tool as described in [the getting started guide](https://github.com/antlr/antlr4/blob/master/doc/getting-started.md). To generate your Dart parser, run the following command: ```shell script antlr4 -Dlanguage=Dart MyGrammar.g4 ``` For a full list of antlr4 tool options, please visit the [tool documentation page](https://github.com/antlr/antlr4/blob/master/doc/tool-options.md). ### Complete example Suppose you're using the JSON grammar from https://github.com/antlr/grammars-v4/tree/master/json. Then, invoke `antlr4 -Dlanguage=Dart JSON.g4`. The result of this is a collection of `.dart` including: * JsonLexer.dart * JsonParser.dart * JsonBaseListener.dart * JsonListener.dart (if you have not activated the -no-listener option) * JsonVisitor.dart (if you have activated the -visitor option) We'll write a small main func to call the generated parser/lexer (assuming they are separate). This one writes out the encountered `ParseTreeContext`'s: ```dart import 'package:antlr4/antlr4.dart'; import 'package:my_project/JSONParser.dart'; import 'package:my_project/JSONLexer.dart'; class TreeShapeListener implements ParseTreeListener { @override void enterEveryRule(ParserRuleContext ctx) { print(ctx.text); } @override void exitEveryRule(ParserRuleContext node) { } @override void visitErrorNode(ErrorNode node) { } @override void visitTerminal(TerminalNode node) { } } void main(List args) async { JSONLexer.checkVersion(); JSONParser.checkVersion(); final input = await InputStream.fromPath(args[0]); final lexer = JSONLexer(input); final tokens = CommonTokenStream(lexer); final parser = JSONParser(tokens); parser.addErrorListener(DiagnosticErrorListener()); final tree = parser.json(); ParseTreeWalker.DEFAULT.walk(TreeShapeListener(), tree); } ``` Create a `example.json` file: ```json {"a":1} ``` Parse the input file: ```shell script dart bin/main.dart example.json ``` The expected output is: ``` {"a":1} {"a":1} {"a":1} "a":1 1 ``` ### Debug We have some logs in place that can ease the debugging process, in order to turn these logs on you can enable the following environment declarations: - ANTLR_LEXER_DEBUG - ANTLR_LEXER_DFA_DEBUG - ANTLR_PARSER_DEBUG - ANTLR_PARSER_LIST_ATN_DECISIONS_DEBUG - ANTLR_PARSER_DFA_DEBUG - ANTLR_PARSER_RETRY_DEBUG If you're using flutter, you can define these variables by adding an `--dart-define` arguments, eg. `flutter run --dart-define LEXER_DEBUG=false` --- ### Doc Go Changes (`doc/go-changes.md`) # Changes to the Go Runtime over time ## v4.12.0 to v4.13.0 Strictly speaking, if ANTLR was a go only project following [SemVer](https://semver.org/) release v4.13.0 would be at least a minor version change and arguably a bump to v5. However, we must follow the ANTLR conventions here or the release numbers would quickly become confusing. I apologize for being unable to follow the Go release rules absolutely to the letter. There are a lot of changes and improvements in this release, but only the change of repo holding the runtime code, and possibly the removal of interfaces will cause any code changes. There are no breaking changes to the runtime interfaces. ANTLR Go Maintainer: [Jim Idle](https://github.com/jimidle) - Email: [jimi@idle.ws](mailto:jimi@idle.ws) ### Code Relocation For complicated reasons, including not breaking the builds of some users who use a monorepo and eschew modules, as well as not making substantial changes to the internal test suite, the Go runtime code will continue to be maintained in the main ANTLR4 repo `antlr/antlr4`. If you wish to contribute changes to the Go runtime code, please continue to submit PRs to this main repo, against the `dev` branch. The code located in the main repo at about the depth of the Mariana Trench, means that the go tools cannot reconcile the module correctly. After some debate, it was decided that we would create a dedicated release repo for the Go runtime so that it will behave exactly as the Go tooling expects. This repo is auto-maintained and keeps both the dev and master branches up to date. Henceforth, all future projects using the ANTLR Go runtime, should import as follows: ```go import ( "github.com/antlr4-go/antlr/v4" ) ``` And use the command: ```shell go get github.com/antlr4-go/antlr ``` To get the module - `go mod tidy` is probably the best way once imports have been changed. Please note that there is no longer any source code kept in the ANTLR repo under `github.com/antlr/antlr4/runtime/Go/antlr`. If you are using the code without modules, then sync the code from the new release repo. ### Documentation Prior to this release, the godocs were essentially unusable as the go doc code was essentially copied without change, from teh Java runtime. The godocs are now properly formatted for Go and pkg.dev. Please feel free to raise an issue if you find any remaining mistakes. Or submit a PR (remember - not to the new repo). It is expected that it might take a few iterations to get the docs 100% squeaky clean. ### Removal of Unnecessary Interfaces The Go runtime was originally produced as almost a copy of the Java runtime but with go syntax. This meant that everything had an interface. There is no need to use interfaces in Go if there is only ever going to be one implementation of some struct and its methods. Interfaces cause an extra deference at runtime and are detrimental to performance if you are trying to squeeze out every last nanosecond, which some users will be trying to do. This is 99% an internal refactoring of the runtime with no outside effects to the user. ### Generated Recognizers Return *struct and not Interfaces The generated recognizer code generated an interface for the parsers and lexers. As they can only be implemented by the generated code, the interfaces were removed. This is possibly the only place you may need to make a code change to your driver code. If your code looked like this: ```go var lexer = parser.NewMySqlLexer(nil) var p = parser.NewMySqlParser(nil) ``` Or this: ```go lexer := parser.NewMySqlLexer(nil) p := parser.NewMySqlParser(nil) ``` Then no changes need to be made. However, fi you predeclared the parser and lexer variables with there type, such as like this: ```go var lexer parser.MySqlLexer var p parser.MySqlParser // ... lexer = parser.NewMySqlLexer(nil) p = parser.NewMySqlParser(nil) ``` You will need to change your variable declarations to pointers (note the introduction of the `*` below. ```go var lexer *parser.MySqlLexer var p *parser.MySqlParser // ... lexer = parser.NewMySqlLexer(nil) p = parser.NewMySqlParser(nil) ``` This is the only user facing change that I can see. This change though has a very beneficial side effect in that you no longer need to cast the interface into a struct so that you can access methods and data within it. Any code you had that needed to do that, will be cleaner and faster. The performance improvement is worth the change and there was no tidy way for me to avoid it. ### Parser Error Recovery Does Not Use Panic THe generated parser code was again essentially trying to be Java code in disguise. This meant that every parser rule executed a `defer {}` and a `recover()`, even if there wer no outstanding parser errors. Parser errors were issued by issuing a `panic()`! While some major work has been performed in the go compiler and runtime to make `defer {}` as fast as possible, `recover()` is (relatively) slow as it is not meant to be used as a general error mechanism, but to recover from say an internal library problem if that problem can be recovered to a known state. The generated code now stores a recognition error and a flag in the main parser struct and use `goto` to exit the rule instead of a `panic()`. As might be imagined, this is significantly faster through the happy path. It is also faster at generating errors. The ANTLR runtime tests do check error raising and recovery, but if you find any differences in the error handling behavior of your parsers, please raise an issue. ### Reduction in use of Pointers Certain internal structs, such as interval sets are small and immutable, but were being passed around as pointers anyway. These have been change to use copies, and resulted in significant performance increases in some cases. There is more work to come in this regard. ### ATN Deserialization When the ATN and associated structures are deserialized for the first time, there was a bug that caused a needed optimization to fail to be executed. This could have a significant performance effect on recognizers that were written in a suboptimal way (as in poorly formed grammars). This is now fixed. ### Prediction Context Caching was not Working This has a massive effect when reusing a parser for a second and subsequent run. The PredictionContextCache merely used memory but did not speed up subsequent executions. This is now fixed, and you should see a big difference in performance when reusing a parser. This single paragraph does not do this fix justice ;) ### Cumulative Performance Improvements Though too numerous to mention, there are a lot of small performance improvements, that add up in accumulation. Everything from improvements in collection performance to slightly better algorithms or specific non-generic algorithms. ### Cumulative Memory Improvements The real improvements in memory usage, allocation and garbage collection are saved for the next major release. However, if your grammar is well-formed and does not require almost infinite passes using ALL(*), then both memory and performance will be improved with this release. ### Bug Fixes Other small bug fixes have been addressed, such as potential panics in funcs that did not check input parameters. There are a lot of bug fixes in this release that most people were probably not aware of. All known bugs are fixed at the time of release preparation. ### A Note on Poorly Constructed Grammars Though I have made some significant strides on improving the performance of poorly formed grammars, those that are particularly bad will see much less of an incremental improvement compared to those that are fairly well-formed. This is deliberately so in this release as I felt that those people who have put in effort to optimize the form of their grammar are looking for performance, where those that have grammars that parser in seconds, tens of seconds or even minutes, are presumed to not care about performance. A particularly good (or bad) example is the MySQL grammar in the ANTLR grammar repository (apologies to the Author if you read this note - this isn't an attack). Although I have improved its runtime performance drastically in the Go runtime, it still takes about a minute to parse complex select statements. As it is constructed, there are no magic answers. I will look in more detail at improvements for such parsers, such as not freeing any memory until the parse is finished (improved 100x in experiments). The best advice I can give is to put some effort in to the actual grammar itself. well-formed grammars will potentially see some huge improvements with this release. Badly formed grammars, not so much. --- ### Doc Go Target (`doc/go-target.md`) # ANTLR4 Language Target, Runtime for Go ### Changes from ANTLR 4.12.0 Please see [Changes in ANTLR Go runtimes](go-changes.md), but in summary: - The Go runtime is now stored in the repo `antlr4-go/antlr` - change your import, remove the old location from `go.mod` and use `go get github.com/antlr4-go/antlr` - There are some new `@actions` for adding to the generated import statements and recognizer structure - The recognizer rules are no longer called via an interface, for performance reasons - Memory usage improvements - Performance improvements - Documentation in true Go format - Git tags now work correctly with go tools ### Removal of non v4 code Prior to the release of the v4 tagged runtime, the source code for the Go runtime module existed at `runtime/Go/antlr`, which is the pre-v4 version of the code, and also under `runtime/Go/antlr/v4`. If your project was not using modules, you could merely sync to the latest hash in the master branch and use the code. This has changed. As of the current release, the source code for the Go runtime module has been moved to its own repo in its own GitHub organization. As of now, you can still use the code without modules, but you must use the code in the repo at https://github.com/antlr4-go/antlr instead of the code in the main ANTLR repo. This is for historic reasons as the code was originally written before modules were a thing, and the go runtime source was - and the maintainer's version still is - a part of the monorepo that is `antlr/antlr4/...`. Note that I am unable to properly deprecate the go.mod in the non-V4 directory, for hte same reason that I cannot use tag the v4 module at this depth in the source tree. We strongly advise you to use modules, though it is not required. See below for more information. ANTLR Go Maintainer: [Jim Idle](https://github.com/jimidle) - Email: [jimi@idle.ws](mailto:jimi@idle.ws) ### First steps #### 1. Install ANTLR4 See: [The getting started guide](getting-started.md). #### 2. Get the Go ANTLR runtime Each target language for ANTLR has a runtime package for running a recognizer generated by ANTLR4. The runtime provides a common set of tools for using your parser/lexer. Note that if you have existing projects and have yet to replace the `v1.x.x` modules with the `v4` modules, then you can skip ahead to the section *Upgrading to v4 from earlier versions* The Go runtime uses modules and has a version path of `/v4` to stay in sync with the runtime versions of all the other runtimes and the tool itself. Setup is the same as any other module based project: ```bash $ cd mymodproject $ go mod init mymodproject ``` After which, you can use go get, to get the latest release version of the ANTLR v4 runtime using: ```bash go get github.com/antlr4-go/antlr ``` If your project was already using the v4 runtime from the main ANTLR repo, then you can upgrade to the latest release by removing the `github.com/antlr/antlr4/runtime/Go/antlr/v4` reference in your module, and changing the associated import in your project code. The following script may be useful in changing your imports: ```shell find . -type f \ -name '*.go' \ -exec sed -i -e 's,github.com/antlr/antlr4/runtime/Go/antlr/v4,github.com/antlr4-go/antlr/v4,g' {} \; ``` Note that the import package still imports with the final path as `antlr`, so only the import statement itself needs to change. If you are already using the repo and import `github.com/antlr4-go/antlr/v4` then you can upgrade to the latest version using the standard. ```shell go get -u github.com/antlr4-go/antlr ``` If you have not yet upgraded existing projects to the `/v4` module path, consult the section *Upgrading to v4 from earlier versions* The ANTLR runtime has only one external transient dependency, and that is part of the go system itself: ``` golang.org/x/exp ``` A complete list of releases can be found on [the release page](https://github.com/antlr/antlr4/releases). The Go runtime will be tagged using standard Go tags, so release 4.13.2 in the `antlr4-go/antlr` repo, will be tagged with `v4.13.2` and go get will pick that up from the ANTLR repo. #### 3. Configuring `go generate` in your project In order to promote the use of repeatable builds, it is often useful to add the latest tool jar to your project's repo and configure a `generate.sh` and `generate.go` file. You can of course globally alias the java command required to run the tool. Your own CI and dev environment will guide you. Here is how you can configure `go generate` for your project, assuming that you follow the general recommendation to place the ANTLR grammar files in their own package in your project structure. Here is a general template as a starting point: ``` . ├── myproject ├── parser │ ├── mygrammar.g4 │ ├── antlr-4.13.2-complete.jar │ ├── generate.go │ └── generate.sh ├── parsing # Generated code goes here │ └── error_listeners.go ├── go.mod ├── go.sum ├── main.go └── main_test.go ``` Make sure that the package statement in your grammar file(s) reflects the go package the go code will be generated in. The `generate.go` file then looks like this: ```golang package parser //go:generate ./generate.sh ``` And the `generate.sh` file will look similar to this: ```shell #!/bin/sh alias antlr4='java -Xmx500M -cp "./antlr-4.13.2-complete.jar:$CLASSPATH" org.antlr.v4.Tool' antlr4 -Dlanguage=Go -no-visitor -package parsing *.g4 ``` From the command line at the root of your package - the location of the `go.mod` file - you can then simply issue the command: ```shell go generate ./... ``` If you have not yet run a `go get`, you can now run `go mod tidy` and update your #### 4. Generate your parser manually You use the ANTLR4 "tool" to generate a parser. These will reference the ANTLR runtime, installed above. Suppose you're using a UNIX system and have set up an alias for the ANTLR4 tool as described in [the getting started guide](getting-started.md). To generate your go parser, you'll need to invoke: ```shell antlr4 -Dlanguage=Go MyGrammar.g4 ``` For a full list of antlr4 tool options, please visit the [tool documentation page](tool-options.md). ### Upgrading to `/v4` from the default path *NB: While switching to new module path would normally imply that the public interface for the runtime has changed, this is not actually the case - you will not need to change your existing code to upgrade. The main point of the repo change is so that git tagging works with the ANTLR Go runtime and the go tools.* Prior to release v4.11.0 the Go runtime shipped with a module but the module had no version path. This meant that the tags in the ANTLR repo did not work, as any tag above `v1` must refer to a matching module path. So the command `go get github.com/antlr/antlr4/runtime/Go/antlr` would just bring in whatever was the `HEAD` of the master branch. While this *kind of* worked, it is obviously subject to problems and does not fit properly with the idiomatic ways of Go. As of v4.13.0 the runtime code exists in its own repo, `github.com/antlr4-go/antlr`, and is correctly tagged. However, this means you need to perform a few simple actions in order to upgrade to the `/v4` path. - Firstly, make sure that you are using an ANTLR tool jar with a version number of 4.13.0 or greater. - Next you replace any mention of the old (default) path to ANTLR in your go source files. - If using modules, remove any existing reference to the ANTLR Go runtime - Now regenerate your grammar files either manually or using `go generate ./...` (see above) - Consider whether you can move to using modules in your project A quick way to replace the original module path references is to use this script from your module's base directory: ```shell find . -type f \ -name '*.go' \ -exec sed -i -e 's,github.com/antlr/antlr4/runtime/Go/antlr,github.com/antlr4-go/antlr/v4,g' {} \; ``` After performing the steps above, and you are using modules issuing: ```shell go mod tidy ``` Should fix up your `go.mod` file to reference only the `v4` version of the ANTLR Go runtime: ```shell require github.com/antlr/antlr4/runtime/Go/antlr/v4 v4.13.0 ``` From this point on, your go mod commands will work correctly with the ANTLR repo and upgrades and downgrades will work as you expect. As will branch version such as @dev ### Referencing the Go ANTLR runtime You can reference the go ANTLR runtime package like this: ```golang import "github.com/antlr4-go/antlr/v4" ``` ### Complete example Suppose you're using the JSON grammar from https://github.com/antlr/grammars-v4/tree/master/json placed in the parser directory and have initialized your `go mod` file. Then, invoke `antlr4 -Dlanguage=Go JSON.g4`. The result of this is a collection of .go files in the `parser` directory including: ``` json_parser.go json_base_listener.go json_lexer.go json_listener.go ``` Another common option to the ANTLR tool is `-visitor`, which generates a parse tree visitor, but we won't be doing that here. For a full list of antlr4 tool options, please visit the [tool documentation page](tool-options.md). We'll write a small main func to call the generated parser/lexer (assuming they are separate). This one writes out the encountered `ParseTreeContext`'s. Assuming the generated parser code is in the `parser` directory relative to this code: ```golang package main import ( "github.com/antlr4-go/antlr/v4" "./parser" // Note that with modules you may not be able to use a relative immport path "os" "fmt" ) type TreeShapeListener struct { *parser.BaseJSONListener } func NewTreeShapeListener() *TreeShapeListener { return new(TreeShapeListener) } func (this *TreeShapeListener) EnterEveryRule(ctx antlr.ParserRuleContext) { fmt.Println(ctx.GetText()) } func main() { input, _ := antlr.NewFileStream(os.Args[1]) lexer := parser.NewJSONLexer(input) stream := antlr.NewCommonTokenStream(lexer,0) p := parser.NewJSONParser(stream) p.AddErrorListener(antlr.NewDiagnosticErrorListener(true)) tree := p.Json() antlr.ParseTreeWalkerDefault.Walk(NewTreeShapeListener(), tree) } ``` Fix up your `go.mod` file: ```shell go mod tidy ``` This one expects the input to be passed on the command line: ``` go run test.go input ``` The output is: ``` {"a":1} {"a":1} "a":1 1 ``` --- ### Doc Grammars (`doc/grammars.md`) # Grammar Structure A grammar is essentially a grammar declaration followed by a list of rules, but has the general form: ``` /** Optional javadoc style comment */ grammar Name; ① options {...} import ... ; tokens {...} channels {...} // lexer only @actionName {...} rule1 // parser and lexer rules, possibly intermingled ... ruleN ``` The file name containing grammar `X` must be called `X.g4`. You can specify options, imports, token specifications, and actions in any order. There can be at most one each of options, imports, and token specifications. All of those elements are optional except for the header ① and at least one rule. Rules take the basic form: ``` ruleName : alternative1 | ... | alternativeN ; ``` Parser rule names must start with a lowercase letter and lexer rules must start with a capital letter. Grammars defined without a prefix on the `grammar` header are combined grammars that can contain both lexical and parser rules. To make a parser grammar that only allows parser rules, use the following header. ``` parser grammar Name; ... ``` And, naturally, a pure lexer grammar looks like this: ``` lexer grammar Name; ... ``` Only lexer grammars can contain `mode` specifications. Only lexer grammars can contain custom channels specifications ``` channels { WHITESPACE_CHANNEL, COMMENTS_CHANNEL } ``` Those channels can then be used like enums within lexer rules: ``` WS : [ \r\t\n]+ -> channel(WHITESPACE_CHANNEL) ; ``` Sections 15.5, [Lexer Rules](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference) and Section 15.3, [Parser Rules](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference) contain details on rule syntax. Section 15.8, Options describes grammar options and Section 15.4, Actions and Attributes has information on grammar-level actions. ## Grammar Imports Grammar `imports` let you break up a grammar into logical and reusable chunks, as we saw in [Importing Grammars](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference). ANTLR treats imported grammars very much like object-oriented programming languages treat superclasses. A grammar inherits all of the rules, tokens specifications, and named actions from the imported grammar. Rules in the “main grammar” override rules from imported grammars to implement inheritance. Think of `import` as more like a smart include statement (which does not include rules that are already defined). The result of all imports is a single combined grammar; the ANTLR code generator sees a complete grammar and has no idea there were imported grammars. To process a main grammar, the ANTLR tool loads all of the imported grammars into subordinate grammar objects. It then merges the rules, token types, and named actions from the imported grammars into the main grammar. In the diagram below, the grammar on the right illustrates the effect of grammar `MyELang` importing grammar `ELang`.

`MyELang` inherits rules `stat`, `WS`, and `ID`, but overrides rule `expr` and adds `INT`. Here’s a sample build and test run that shows `MyELang` can recognize integer expressions whereas the original `ELang` can’t. The third, erroneous input statement triggers an error message that also demonstrates the parser was looking for `MyELang`’s expr not `ELang`’s. ``` $ antlr4 MyELang.g4 $ javac MyELang*.java $ grun MyELang stat => 34; => a; => ; => EOF <= line 3:0 extraneous input ';' expecting {INT, ID} ``` If there are modes in the main grammar or any of the imported grammars then the import process will import those modes and merge their rules where they are not overridden. In the event any mode becomes empty as all its rules have been overridden by rules outside the mode this mode will be discarded. If there were any `tokens` specifications, the main grammar would merge the token sets. If there were any `channel` specifications, the main grammar would merge the channel sets. Any named actions such as `@members` would be merged. In general, you should avoid named actions and actions within rules in imported grammars since that limits their reuse. ANTLR also ignores any options in imported grammars. Imported grammars can also import other grammars. ANTLR pursues all imported grammars in a depth-first fashion. If two or more imported grammars define rule `r`, ANTLR chooses the first version of `r` it finds. In the following diagram, ANTLR examines grammars in the following order `Nested`, `G1`, `G3`, `G2`.

`Nested` includes the `r` rule from `G3` because it sees that version before the `r` in `G2`. Not every kind of grammar can import every other kind of grammar: * Lexer grammars can import lexers, including lexers containing modes. * Parsers can import parsers. * Combined grammars can import parsers or lexers without modes. ANTLR adds imported rules to the end of the rule list in a main lexer grammar. That means lexer rules in the main grammar get precedence over imported rules. For example, if a main grammar defines rule `IF : 'if' ;` and an imported grammar defines rule `ID : [a-z]+ ;` (which also recognizes `if`), the imported `ID` won’t hide the main grammar’s `IF` token definition. ## Tokens Section The purpose of the `tokens` section is to define token types needed by a grammar for which there is no associated lexical rule. The basic syntax is: ``` tokens { Token1, ..., TokenN } ``` Most of the time, the tokens section is used to define token types needed by actions in the grammar as shown in Section 10.3, [Recognizing Languages whose Keywords Aren’t Fixed](http://pragprog.com/book/tpantlr2/the-definitive-antlr-4-reference): ``` // explicitly define keyword token types to avoid implicit definition warnings tokens { BEGIN, END, IF, THEN, WHILE } @lexer::members { // keywords map used in lexer to assign token types Map keywords = new HashMap() {{ put("begin", KeywordsParser.BEGIN); put("end", KeywordsParser.END); ... }}; } ``` The `tokens` section really just defines a set of tokens to add to the overall set. ``` $ cat Tok.g4 grammar Tok; tokens { A, B, C } a : X ; $ antlr4 Tok.g4 warning(125): Tok.g4:3:4: implicit definition of token X in parser $ cat Tok.tokens A=1 B=2 C=3 X=4 ``` ## Actions at the Grammar Level Currently there are only two defined named actions (for the Java target) used outside of grammar rules: `header` and `members`. The former injects code into the generated recognizer class file, before the recognizer class definition, and the latter injects code into the recognizer class definition, as fields and methods. For combined grammars, ANTLR injects the actions into both the parser and the lexer. To restrict an action to the generated parser or lexer, use `@parser::name` or `@lexer::name`. Here’s an example where the grammar specifies a package for the generated code: ``` grammar Count; @header { package foo; } @members { int count = 0; } list @after {System.out.println(count+" ints");} : INT {count++;} (',' INT {count++;} )* ; INT : [0-9]+ ; WS : [ \r\t\n]+ -> skip ; ``` The grammar itself then should be in directory `foo` so that ANTLR generates code in that same `foo` directory (at least when not using the `-o` ANTLR tool option): ``` $ cd foo $ antlr4 Count.g4 # generates code in the current directory (foo) $ ls Count.g4 CountLexer.java CountParser.java Count.tokens CountLexer.tokens CountBaseListener.java CountListener.java $ javac *.java $ cd .. $ grun foo.Count list => 9, 10, 11 => EOF <= 3 ints ``` The Java compiler expects classes in package `foo` to be in directory `foo`. --- ### Doc Interpreters (`doc/interpreters.md`) # Parser and Lexer Interpreters *Since ANTLR 4.2* For small parsing tasks it is sometimes convenient to use ANTLR in interpreted mode, rather than generating a parser in a particular target, compiling it and running it as part of your application. Here's some sample code that creates lexer and parser Grammar objects and then creates interpreters. Once we have a ParserInterpreter, we can use it to parse starting in any rule we like, given a rule index (which the grammar + the parser can provide). ## Action Code Since interpreters don't use generated parsers + lexers they cannot execute any action code (including predicates). That means the interpreter runs as if there were no predicates at all. If your grammar requires action code in order to parse correctly you will not be able to test it using this approach. ## Java Target Interpreter Setup ```java LexerGrammar lg = new LexerGrammar( "lexer grammar L;\n" + "A : 'a' ;\n" + "B : 'b' ;\n" + "C : 'c' ;\n"); Grammar g = new Grammar( "parser grammar T;\n" + "s : (A|B)* C ;\n", lg); LexerInterpreter lexEngine = lg.createLexerInterpreter(new ANTLRInputStream(input)); CommonTokenStream tokens = new CommonTokenStream(lexEngine); ParserInterpreter parser = g.createParserInterpreter(tokens); ParseTree t = parser.parse(g.rules.get(startRule).index); ``` You can also load combined grammars from a file: ```java public static ParseTree parse(String fileName, String combinedGrammarFileName, String startRule) throws IOException { final Grammar g = Grammar.load(combinedGrammarFileName); LexerInterpreter lexEngine = g.createLexerInterpreter(CharStreams.fromPath(Paths.get(fileName))); CommonTokenStream tokens = new CommonTokenStream(lexEngine); ParserInterpreter parser = g.createParserInterpreter(tokens); ParseTree t = parser.parse(g.getRule(startRule).index); System.out.println("parse tree: "+t.toStringTree(parser)); return t; } ``` Then: ```java ParseTree t = parse("T.om", MantraGrammar, "compilationUnit"); ``` To load separate lexer/parser grammars, do this: ```java public static ParseTree parse(String fileNameToParse, String lexerGrammarFileName, String parserGrammarFileName, String startRule) throws IOException { final LexerGrammar lg = (LexerGrammar) Grammar.load(lexerGrammarFileName); final Grammar pg = Grammar.load(parserGrammarFileName, lg); CharStream input = CharStreams.fromPath(Paths.get(fileNameToParse)); LexerInterpreter lexEngine = lg.createLexerInterpreter(input); CommonTokenStream tokens = new CommonTokenStream(lexEngine); ParserInterpreter parser = pg.createParserInterpreter(tokens); ParseTree t = parser.parse(pg.getRule(startRule).index); System.out.println("parse tree: " + t.toStringTree(parser)); return t; } ``` Then: ```java ParseTree t = parse(fileName, XMLLexerGrammar, XMLParserGrammar, "document"); ``` This is also how we will integrate instantaneous parsing into ANTLRWorks2 and development environment plug-ins. See [TestParserInterpreter.java](../tool-testsuite/test/org/antlr/v4/test/tool/TestParserInterpreter.java). ## Non-Java Target Interpreter Setup The ANTLR4 runtimes do not contain any grammar parsing classes (they are in the ANTLR4 tool jar). Hence we cannot use `LexerGrammar` and `Grammar` to parse grammars for the interpreter. Instead we directly instantiate `LexerInterpreter` and `ParserInterpreter` objects. They require some data (namely symbol information and the ATNs) which only the ANTLR4 tool can give us. However, on each generation run ANTLR not only produces your parser + lexer files but also interpreter data files (*.interp) which contain all you need to feed the interpreters. A support class (`InterpreterDataReader`) is used to load the data for your convenience, which makes this very easy to use. Btw. even the Java target go this route instead of using the non-runtime classes `Grammar` and `LexerGrammar`. Sometimes it might not be feasible to use the tool jar for whatever reason. Here's how the setup looks like (C++ example): ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Doc Java Target (`doc/java-target.md`) # Java ## Development environments ### Intellij There is a very complete and useful plug-in for intellij 12-14, you can grab at the [download page](https://plugins.jetbrains.com/plugin/7358?pr=). Check the [plugin readme](https://github.com/antlr/intellij-plugin-v4) for feature set. Just go to the preferences and click on the "Install plug-in from disk..." button from this dialog box:

Select the intellij-plugin-1.x.zip (or whatever version) file and hit okay or apply. It will ask you to restart the IDE. If you look at the plug-ins again, you will see:

Also, I have prepared a [video](https://youtu.be/eW4WFgRtFeY) that will help you generate grammars and so on using ANTLR v4 in Intellij (w/o the plugin). ### Eclipse Edgar Espina has created an [eclipse plugin for ANTLR v4](https://marketplace.eclipse.org/content/antlr-ide). Features: Advanced Syntax Highlighting, Automatic Code Generation (on save), Manual Code Generation (through External Tools menu), Code Formatter (Ctrl+Shift+F), Syntax Diagrams, Advanced Rule Navigation between files (F3), Quick fixes. ### NetBeans Sam Harwell's [ANTLRWorks2](http://tunnelvisionlabs.com/products/demo/antlrworks) works also as a plug-in, not just a stand-alone tool built on top of NetBeans. ## Build systems ### ant ### mvn *Maven Plugin Reference* The reference pages for the latest version of the Maven plugin for ANTLR 4 can be found here: [http://www.antlr.org/api/maven-plugin/latest/index.html](http://www.antlr.org/api/maven-plugin/latest/index.html) *Walkthrough* This section describes how to create a simple Antlr 4 project and build it using maven. We are going to use the ArrayInit.g4 example from chapter 3 of the book, and bring it under maven. We will need to rename files and modify them. We will conclude by building a portable stand alone application. Generate the skeleton. To generate the maven skeleton, type these commands: ```bash mkdir SimpleAntlrMavenProject cd SimpleAntlrMavenProject mvn archetype:generate -DgroupId=org.abcd.examples -DartifactId=array-example -Dpackage=org.abcd.examples.ArrayInit -Dversion=1.0 # Accept all the default values cd array-example ``` Maven will ask a series of questions, simply accept the default answers by hitting enter. Move into the directory created by maven: ```bash cd array-example ``` We can use the find command to see the files created by maven: ```bash $ find . -type f ./pom.xml ./src/test/java/org/abcd/examples/ArrayInit/AppTest.java ./src/main/java/org/abcd/examples/ArrayInit/App.java ``` We need to edit the pom.xml file extensively. The App.java will be renamed to ArrayInit.java and will contain the main ANTLR java program which we will download from the book examples. The AppTest.java file will be renamed ArrayInitTest.java but will remain the empty test as created by maven. We will also be adding the grammar file ArrayInit.g4 from the book examples in there. Get the examples for the book and put them in the Downloads folder. To obtain the ArrayInit.g4 grammar from the book, simply download it: ```bash pushd ~/Downloads wget http://media.pragprog.com/titles/tpantlr2/code/tpantlr2-code.tgz tar xvfz tpantlr2-code.tgz popd ``` Copy the grammar to the maven project. The grammar file goes into a special folder under the src/ directory. The folder name must match the maven package name org.abcd.examples.ArrayInit. ```bash mkdir -p src/main/antlr4/org/abcd/examples/ArrayInit cp ~/Downloads/code/starter/ArrayInit.g4 src/main/antlr4/org/abcd/examples/ArrayInit ``` Copy the main program to the maven project. We replace the maven App.java file with the main java program from the book. In the book, that main program is called Test.java, we rename it to ArrayInit.java: ```bash # Remove the maven file rm ./src/main/java/org/abcd/examples/ArrayInit/App.java # Copy and rename the example from the book cp ~/Downloads/code/starter/Test.java ./src/main/java/org/abcd/examples/ArrayInit/ArrayInit.java ``` Spend a few minutes to read the main program. Notice that it reads the standard input stream. We need to remember this when we run the application. Edit the ArrayInit.java file. We need to add a package declaration and to rename the class. Edit the file ./src/main/java/org/abcd/examples/ArrayInit/ArrayInit.java in your favorite editor. The head of the file should look like this when you are done: ```java package org.abcd.examples.ArrayInit; import org.antlr.v4.runtime.*; import org.antlr.v4.runtime.tree.*; public class ArrayInit { ... ``` Edit the ArrayInitTest.java file. Maven creates a test file called AppTest.java, we need to rename it to match the name of our application: ```bash pushd src/test/java/org/abcd/examples/ArrayInit mv AppTest.java ArrayInitTest.java sed 's/App/ArrayInit/g' ArrayInitTest.java >ArrayInitTest.java.tmp mv ArrayInitTest.java.tmp ArrayInitTest.java popd ``` Edit the pom.xml file. Now we need to extensively modify the pom.xml file. The final product looks like this: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` This concludes the changes we had to make. We can look at the list of files we have with the find command: ```bash $ find . -type f ./pom.xml ./src/test/java/org/abcd/examples/ArrayInit/ArrayInitTest.java ./src/main/antlr4/org/abcd/examples/ArrayInit/ArrayInit.g4 ./src/main/java/org/abcd/examples/ArrayInit/ArrayInit.java ``` Building a stand alone application. With all the files now in place, we can ask maven to create a standalone application. The following command does this: ```bash mvn package ``` Maven creates a self-contained jar file called target/array-init-1.0-jar-with-dependencies.jar. We can execute the jar file, but remember that it expects some input on the command line, which means the command will hang on the command line until we feed it some input: ```bash java -cp target/array-init-1.0-jar-with-dependencies.jar org.abcd.examples.ArrayInit.ArrayInit ``` And let's feed it the following input: ```bash {1,2,3} ^D ``` The ^D signals the end of the input to the standard input stream and gets the rest of the application going. You should see the following output: ```bash (init { (value 1) , (value 2) , (value 3) }) ``` You can also build a jar file without the dependencies, and execute it with a maven command instead: ```bash mvn install mvn exec:java -Dexec.mainClass=org.abcd.examples.ArrayInit.ArrayInit {1,2,3} ^D ``` --- ### Doc Javascript Target (`doc/javascript-target.md`) # JavaScript ## Which browsers are supported? In theory, all browsers supporting ECMAScript 5.1. In practice, this target has been extensively tested against: * Firefox 34.0.5 * Safari 8.0.2 * Chrome 39.0.2171 * Explorer 11.0.3 The above tests were conducted using Selenium. No issue was found, so you should find that the runtime works pretty much against any recent JavaScript engine. ## Is NodeJS supported? The runtime has also been extensively tested against Node.js 14 LTS. No issue was found. NodeJS together with a packaging tool is now the preferred development path, developers are encouraged to follow it. ## What about modules? Starting with version 8.1, Antlr4 JavaScript runtime follows esm semantics (see https://tc39.es/ecma262/#sec-modules for details) Generated lexers, parsers, listeners and visitors also follow this new standard. If you have used previous versions of the runtime, you will need to migrate and make your parser a module. ## How to create a JavaScript lexer or parser? This is pretty much the same as creating a Java lexer or parser, except you need to specify the language target, for example: ```bash $ antlr4 -Dlanguage=JavaScript MyGrammar.g4 ``` For a full list of antlr4 tool options, please visit the [tool documentation page](tool-options.md). ## Where can I get the runtime? Once you've generated the lexer and/or parser code, you need to download the runtime. The JavaScript runtime is [available from npm](https://www.npmjs.com/package/antlr4). We will not document here how to refer to the runtime from your project, since this would differ a lot depending on your project type and IDE. ## How do I get the runtime in my browser? The runtime is quite big and is currently maintained in the form of around 50 scripts, which follow the same structure as the runtimes for other targets (Java, C#, Python...). This structure is key in keeping code maintainable and consistent across targets. However, it would be a bit of a problem when it comes to get it into a browser. Nobody wants to write 50 times: ```