### Doc/Design/Miniohm # Ohm and miniohm As part of the [WebAssembly MVP](https://github.com/ohmjs/ohm/issues/503), I've created a small Go library that takes a .wasm blob and provides a minimal interface for matching input and walking the resulting CST. I'm calling this interface "miniohm". There's also a JavaScript implementation, and eventually we'll have versions for other languages. Eventually, we want the WebAssembly-based matcher to be the core of `ohm-js`. Ideally we can do this by wrapping miniohm, so that `miniohm-js` would be both an implementation detail of `ohm-js`, _and_ a standalone library you could adopt. We'd also have `miniohm-go`, `miniohm-rs`, etc., but likely no "full" version of Ohm for those languages. Here's how I'm picturing this: --- ### Doc/Releases/Ohm Js 16.0 # Ohm v16.0 ## Upgrading ### Args to _\_iter_ and _\_nonterminal_ actions The [`_iter` and `_nonterminal` actions](../api-reference.md#special-actions) now take a variable number of arguments, rather than a single `Node[]` argument containing the child nodes. To make existing code work with Ohm v16, you should change the parameter to a [rest parameter](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/rest_parameters) β€” e.g., `_iter(children) { ... }` should be changed to `_iter(...children) { ... }`. You can easily find code that needs to change because `addOperation` and friends will now throw an exception if your actions have a single parameter which is _not_ a rest parameter. See [#324](https://github.com/ohmjs/ohm/issues/324) for the reasons behind this change. ### Default semantic actions In operations and attributes, if you haven't defined a semantic action for a particular rule application node, a default action will be used in some cases. For example, your grammar has an _AddExp_ rule but your action dictionary doesn't contain a semantic action named 'AddExp'. **In Ohm v16.0, there is no longer a default action for iteration nodes** β€” it is _only_ defined for non-terminal nodes with exactly one child. See [#309](https://github.com/ohmjs/ohm/issues/309) for context on this change. Semantic actions that worked in previous versions of Ohm may need to be modified to work with v16.0. You can identify the code that needs modification because they will now raise a "missing semantic action" error. The [recommended way of dealing with iteration nodes](https://github.com/ohmjs/ohm/blob/main/doc/patterns-and-pitfalls.md#iteration-nodes) is to use the `children` attribute to explicitly invoke the operation on each child, e.g. `iterNode.myOperation()` becomes `iterNode.children.map(c => c.myOperation())`. This way, it's clear to readers of the code that the result is an array. In some cases, it makes sense to write a generic _\_iter_ action that specifies the behaviour for all iteration nodes. This also makes it possible to replicate the old behaviour of the default action: ``` _iter(...children) { return children.map(c => c.myOperation()); } ``` ### grammarFromScriptElement / grammarsFromScriptElements The functions `grammarFromScriptElement` and `grammarsFromScriptElements` have been removed. When using Ohm in the browser, it's now recommended to put your grammar in a [template literal with String.raw](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/raw): ```js const myGrammar = ohm.grammar(String.raw` MyGrammar { greeting = "Hello" | "Hola" } `); ``` ### Type parameters for operations and actions (TypeScript only) On semantics objects, the methods `addOperation`, `extendOperation`, `addAttribute`, and `extendAttribute` now have a required type parameter, which is the return type of the operation (or the type of the attribute). The related types `Action` and `ActionDict` are similarly parameterized by the return type of the semantic actions. In other words, all of the semantic actions for a particular operation/attribute are now required to have a consistent return type. The simplest change to make your code compatible with Ohm v16 is to simply specify the type paramter. E.g., `semantics.addOperation('toString', { ... })` becomes `semantics.addOperation('toString', { ... })`. In some cases, it may be necessary to use `any`, which is the type that was assumed in previous versions of Ohm. ## Other changes - `Grammar.semantics` has been completely removed. Use `Grammar.createSemantics` instead. ## New deprecations - `Node.primitiveValue` is now deprecated in favour of `Node.sourceString`. --- ### Doc/Releases/Ohm Js 17.0 # Ohm v17.0 v17.0.0 is primarily a cleanup release in preparation for some new features that will be coming soon in minor releases. It contains a few breaking changes (as described below), but they are unlikely to affect the majority of Ohm users. This version also has experimental support for [indentation-sensitive grammars](#new-features). ## Upgrading ### `any` now consumes a full code point In JavaScript, a string is a sequence of 16-bit code units. Some Unicode characters, such as emoji, are encoded as pairs of 16-bit values. For example, the string 'πŸ˜†' has length 2, but contains a single Unicode code point. Previously, `any` matched a single 16-bit code unit β€” even if that unit was part of a surrogate pair. In v17, `any` now matches a full Unicode character. Old behaviour: ```js const g = ohm.grammar('OneChar { start = any }'); g.match('πŸ˜†').succeeded(); // false ``` New behaviour (Ohm v17+): ```js const g = ohm.grammar('OneChar { start = any }'); g.match('πŸ˜†').succeeded(); // true ``` ### Namespace helpers removed The top-level `namespace` and `extendNamespace` functions have been removed. They were never required β€” it was always possible to use a plain old object in any API that asked for a namespace. For example, take the following code (which no longer works in v17+): ```js const ns = ohm.createNamespace(); ns.G = ohm.grammar('G {}'); ns.G2 = ohm.grammar('G2 <: G {}', ns); ``` In Ohm v17 _and_ in previous versions, the code could instead be written like this: ```js const ns = {}; // <- Use a normal object literal ns.G = ohm.grammar('G {}'); ns.G2 = ohm.grammar('G2 <: G {}', ns); ``` ### Named exports only When used as ES module, the `ohm-js` and `ohm-js/extras` modules now have _only_ named exports, and no default export. This fixes some issues with the use of Ohm bundles (`.ohm-bundle` files) when used with JS module bundlers like Webpack and Rollup (e.g., [#377][issue-377]). As a result, the following code will no longer work (unless your bundler supports [synthetic default imports](https://www.typescriptlang.org/tsconfig#allowSyntheticDefaultImports)): ```js import ohm from 'ohm-js'; ``` In Ohm v17, it should be written like this: ```js import * as ohm from 'ohm-js'; ``` See [#386](https://github.com/ohmjs/ohm/issues/386) for more context on this change. ### `util` removed The `util` object has been removed from Ohm's public API. The two methods that were relevant to grammar authors β€” `getLineAndColumn` and `getLineAndColumnMessage` β€” have been moved into the `extras` module. Old code (no longer works): ```js ohm.util.getLineAndColumnMessage(str, [startIdx, endIdx]); ``` New code (works in Ohm v17+): ```js import {getLineAndColumnMessage} from 'ohm-js/extras'; // ... getLineAndColumnMessage(str, [startIdx, endIdx]); ``` By the way, the `Interval` class also has a `getLineAndColumnMessage` method, which can be useful inside of semantic actions. For example, if you are using this method with the `sourceString` attribute, like this: ```js getLineAndColumnMessage(aNode.sourceString, [startIdx, endIdx]); ``` ...you can write it this way instead: ```js aNode.source.getLineAndColumnMessage(); ``` ### `toAST` on built-in list rules Ohm v17 changed the default behavior of `toAST` for the built-in list rules (`ListOf` and friends). Both the syntactic (`ListOf`, ...) and lexical versions (`listOf`, ...) are now represented as arrays, with the separators _discarded_. Previously, the syntactic versions were represented by arrays, but with separators _included_, and the lexical versions were represented as strings (just like other lexical rules). See [#394](https://github.com/ohmjs/ohm/issues/394) for the reasoning behind this change. ## New features - **(EXPERIMENTAL)**: This release includes experimental support for indentation-sensitive grammars (e.g., Python, Elm, YAML). See the guide to [Parsing indentation sensitive languages](../indentation-sensitive.md). ## Other changes - The deprecated `grammarFromScriptElement` and `grammarsFromScriptElements` functions have been entirely removed. - The `primitiveValue` property of nodes, which was deprecated in Ohm v16, has now been completely removed β€” use `Node.sourceString` instead. --- ### Doc/Releases/Ohm Js 18.0 # Ohm v18.0 Beta β€” Migration Guide Ohm v18 compiles grammars to WebAssembly. The runtime is smaller and faster, but the API has changed significantly. This guide covers everything you need to migrate from v17. ## Installation ```bash npm install ohm-js # Runtime (production dependency) npm install --save-dev @ohm-js/compiler # Compiler (dev dependency) ``` ## Compiling grammars In v17, grammars were parsed at runtime. In v18, the recommendation is to compile them to `.wasm` at build time. ### Command line ```bash npx ohm2wasm my-grammar.ohm # writes my-grammar.wasm ``` ### Programmatic ```js import {compile, compileGrammars} from '@ohm-js/compiler'; const bytes = compile('MyGrammar { start = "hello" }'); // Multiple grammars: const bytesByName = compileGrammars(source); // Record ``` ## Loading and using grammars ```js import {Grammar} from 'ohm-js'; import fs from 'node:fs'; const g = await Grammar.instantiate(fs.readFileSync('my-grammar.wasm')); // Or from a fetch response: const g = await Grammar.instantiateStreaming(fetch('my-grammar.wasm')); ``` ## MatchResult lifecycle Parse results live in Wasm linear memory and must be explicitly disposed. ### `using` (recommended) ```js using result = g.match(input); if (result.succeeded()) { // ... use result ... } // Memory is automatically freed when `result` goes out of scope. ``` ### `.use()` callback In environments that don't support the `using` keyword, the `use()` callback should be used: ```js g.match(input).use(result => { if (result.succeeded()) { // ... use result ... } }); ``` ### Notes - Results must be disposed in LIFO order (most recent first). - Forgetting to dispose a result will prevent subsequent `match()` calls from succeeding. ## CST nodes In v17, you access CST nodes through Semantics wrappers. In v18, `MatchResult` gives you the CST directly β€” there is no Semantics layer (not yet, at least). ```js using result = g.match(input); if (result.succeeded()) { const cst = result.getCstRoot(); // NonterminalNode console.log(cst.ctorName); // rule name console.log(cst.sourceString); // matched text console.log(cst.children); // child nodes } ``` ### Node types All nodes share: `ctorName`, `sourceString`, `source` (with `startIdx`/`endIdx`), `children`. | Type | `ctorName` | Description | |------|-----------|-------------| | `NonterminalNode` | rule name | Has `.isSyntactic()`, `.isLexical()`, `.leadingSpaces` | | `TerminalNode` | `"_terminal"` | Has `.value` | | `ListNode` | `"_list"` | From `*` or `+`. Has `.collect(cb)` | | `OptNode` | `"_opt"` | From `?`. Has `.ifPresent(cb, orElse?)`, `.isPresent()`, `.isEmpty()` | | `SeqNode` | `"_seq"` | Grouped sequence. Has `.unpack(cb)` | Type guards: `node.isNonterminal()`, `node.isTerminal()`, `node.isList()`, `node.isOptional()`, `node.isSeq()`. ### Arity changes - Iter (`*`/`+`) and Opt (`?`) nodes are **no longer flattened**. In v17, `a b c*` would give a semantic action 4 arguments (a, b, c_1, c_2, ...); in v18, you get 3 children, where the third is a `ListNode`. - Positive lookahead (`&e`) **does not bind a node**. ### Working with ListNode ```js // .collect() maps over items, unpacking SeqNode children as arguments: const items = listNode.collect((name, sep, value) => { return {name: name.sourceString, value: value.sourceString}; }); ``` ### Working with OptNode ```js // .ifPresent() calls the callback if the option matched, with SeqNode unpacking: const val = optNode.ifPresent( child => child.sourceString, () => 'default' ); ``` ### Working with SeqNode ```js // .unpack() spreads children as callback arguments: seqNode.unpack((left, op, right) => { // ... }); ``` ## Error handling ```js using result = g.match(input); if (result.failed()) { console.log(result.message); // Full message with line/col and input excerpt console.log(result.shortMessage); // "Line 1, col 5: expected ..." console.log(result.getExpectedText()); // "letter or digit" console.log(result.getRightmostFailurePosition()); // number console.log(result.getRightmostFailures()); // Failure[] } ``` ## Breaking changes ### Syntactic vs. lexical rule classification `isSyntactic()` now looks at the first _letter_ in the rule name β€” if it's an upper case letter, the rule is syntactic, otherwise it's lexical. In v17, we compared `firstChar === firstChar.toUpperCase()`, which incorrectly classified rule names starting with non-letter characters (e.g. `_`, digits) as syntactic. This now means that a rule named `_ident` is now treated as lexical, not syntactic. ## Removed APIs The following v17 APIs do not exist in v18 (yet): - **Semantics** β€” `createSemantics()`, `addOperation()`, `addAttribute()`, `extendSemantics()`. Traverse the CST directly instead. - **Matcher** β€” `grammar.matcher()` and incremental parsing. - **Tracing** β€” `grammar.trace()`. - **Extras** β€” `ohm-js/extras` (including `toAST`). See `@ohm-js/to-ast-compat` below. - **Recipes** β€” `makeRecipe()`. - **PExprs** β€” The `pexprs` export. - **Grammar introspection** β€” `grammar.rules` (available via compat layer). ## Compat helpers ### `@ohm-js/compiler/compat` For incremental migration, `grammar()` and `grammars()` parse, compile, and instantiate in one step β€” matching the v17 API: ```js import {grammar} from '@ohm-js/compiler/compat'; const g = grammar('MyGrammar { start = "hello" }'); using result = g.match('hello'); ``` This compiles on every call. For production, compile to `.wasm` ahead of time. ### `@ohm-js/to-ast-compat` Replaces `toAST` from `ohm-js/extras`. ```bash npm install @ohm-js/to-ast-compat ``` Old (v17): ```js import {toAST} from 'ohm-js/extras'; const ast = toAST(match, { Equation: {content: 0}, AddExpr: {type: 'Expression', expr1: 0, op: 1, expr2: 2}, }); ``` New (v18): ```js import {createToAst} from '@ohm-js/to-ast-compat'; const toAST = createToAst({ Equation: {content: 0}, AddExpr: {type: 'Expression', expr1: 0, op: 1, expr2: 2}, }); // Later: const ast = toAST(matchResult); // accepts MatchResult or CstNode ``` #### Differences from v17's `toAST` - Mapping functions receive `CstNode` args (similar to v17's `Node`, but slightly different interface). - Recursive calls: use `toAST(someNode)` instead of `someNode.toAST(this.args.mapping)`. - The `AstBuilder` class is also exported for advanced use β€” it has a `currNode` property and a `toAst()` method. --- ### Doc/Api Reference # API Reference This page documents the API of Ohm/JS, a JavaScript library for working with grammars written in the Ohm language. For documentation on the Ohm language, see the [syntax reference](./syntax-reference.md). ## Instantiating Grammars **NOTE:** For grammars defined in a JavaScript string literal (i.e., not in a separate .ohm file), it's recommended to use a [template literal with the String.raw tag](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/raw). 
ohm.grammar(source: string, optNamespace?: object) → Grammar
 Instantiate the Grammar defined by `source`. If specified, `optNamespace` is an object in which references to other grammars should be resolved. For example, if the grammar source begins with an inheritance declaration like `MyGrammar <: OtherGrammar { ... }`, then `optNamespace` should have a property named `OtherGrammar`. 
ohm.grammars(source: string, optNamespace?: object) → object
 Create a new object containing Grammar instances for all of the grammars defined in `source`. As with `ohm.grammar`, if `optNamespace` is specified, it is an object in which references to other grammars should be resolved. Additionally, it will be the prototype of the returned object. Here is an example of instantiating a Grammar: ``` const parentDef = String.raw` Parent { start = "parent" } `; const parentGrammar = ohm.grammar(parentDef); ``` In the next example We instantiate a new grammar, *Child*, that inherits from our *Parent* grammar. We use the `ohm.grammars` method, which returns an object of our grammars: ``` const childDef = String.raw` Child <: Parent { start := "child" } `; const childGrammar = ohm.grammars(childDef, {Parent: parentGrammar}); console.log(Object.keys(childGrammar)); // > [ 'Child' ] ``` You could also concatenate the grammar definitions, and then instantiate them. This results in an object with both Grammars: ``` const combinedDef = parentDef.concat(childDef); const grammars = ohm.grammars(combinedDef); console.log(Object.keys(grammars)); // > [ 'Parent', 'Child' ] ``` ## Grammar objects A Grammar instance `g` has the following methods: 
g.match(str: string, optStartRule?: string) → MatchResult
 Try to match `str` against `g`, returning a [MatchResult](#matchresult-objects). If `optStartRule` is given, it specifies the rule on which to start matching. By default, the start rule is inherited from the supergrammar, or if there is no supergrammar specified, it is the first rule in `g`'s definition. 
g.matcher()
 Create a new [Matcher](#matcher-objects) object which supports incrementally matching `g` against a changing input string. 
g.trace(str: string, optStartRule?: string) → Trace
 Try to match `str` against `g`, returning a Trace object. `optStartRule` has the same meaning as in `g.match`. Trace objects have a `toString()` method, which returns a string which summarizes each parsing step (useful for debugging). 
g.createSemantics() → Semantics
 Create a new [Semantics](#semantics) object for `g`. 
g.extendSemantics(superSemantics: Semantics) → Semantics
 Create a new [Semantics](#semantics) object for `g` that inherits all of the operations and attributes in `superSemantics`. `g` must be a descendent of the grammar associated with `superSemantics`. ## Matcher objects Matcher objects can be used to incrementally match a changing input against the Matcher's grammar, e.g. in an editor or IDE. When a Matcher's input is modified via `replaceInputRange`, further calls to `match` will reuse the partial results of previous calls wherever possible. Generally, this means that small changes to the input will result in very short match times. A Matcher instance `m` has the following methods: 
m.getInput() → string
 Return the current input string. 
m.setInput(str: string)
 Set the input string to `str`. 
m.replaceInputRange(startIdx: number, endIdx: number, str: string)
 Edit the current input string, replacing the characters between `startIdx` and `endIdx` with `str`. 
m.match(optStartRule?: string) → MatchResult
 Like [Grammar's `match` method](#Grammar.match), but operates incrementally. 
m.trace(optStartRule?: string) → Trace
 Like [Grammar's `trace` method](#Grammar.trace), but operates incrementally. ## MatchResult objects Internally, a successful MatchResult contains a _parse tree_, which is made up of _parse nodes_. Parse trees are not directly exposed -- instead, they are inspected indirectly through _operations_ and _attributes_, which are described in the next section. A MatchResult instance `r` has the following methods: 
r.succeeded() → boolean
 Return `true` if the match succeeded, otherwise `false`. 
r.failed() → boolean
 Return `true` if the match failed, otherwise `false`. ### MatchFailure objects When `r.failed()` is `true`, `r` has the following additional properties and methods: 
r.message: string
 Contains a message indicating where and why the match failed. This message is suitable for end users of a language (i.e., people who do not have access to the grammar source). 
r.shortMessage: string
 Contains an abbreviated version of `r.message` that does not include an excerpt from the invalid input. 
r.getRightmostFailurePosition() → number
 Return the index in the input stream at which the match failed. 
r.getRightmostFailures() → Array
 Return an array of Failure objects describing the failures the occurred at the rightmost failure position.

Semantics, Operations, and Attributes

An Operation represents a function that can be applied to a successful match result. Like a [Visitor](http://en.wikipedia.org/wiki/Visitor_pattern), an operation is evaluated by recursively walking the parse tree, and at each node, invoking the matching semantic action from its _action dictionary_. An Attribute is an Operation whose result is memoized, i.e., it is evaluated at most once for any given node. A Semantics is a family of operations and/or attributes for a given grammar. A grammar may have any number of Semantics instances associated with it β€” this means that the clients of a grammar (even in the same program) never have to worry about operation/attribute name clashes. ### Semantics objects Operations and attributes are accessed by applying a semantics instance to a [MatchResult](#matchresult-objects). This returns a parse node, whose properties correspond to the operations and attributes of the semantics. For example, to invoke an operation named 'prettyPrint': `mySemantics(matchResult).prettyPrint()`. Attributes are accessed using property syntax β€” e.g., for an attribute named 'value': `mySemantics(matchResult).value`. A Semantics instance `s` has the following methods, which all return `this` so they can be chained: 
mySemantics.addOperation(nameOrSignature: string, actionDict: object) → Semantics
 Add a new Operation to this Semantics, using the [semantic actions](#semantic-actions) contained in `actionDict`. The first argument is either a name (e.g. `'prettyPrint'`) or a _signature_ which specifies the operation name and zero or more named parameters (e.g., `'prettyPrint()'`, `'prettyPrint(depth, strict)'`). It is an error if there is already an operation or attribute called `name` in this semantics. If the operation has arguments, they are accessible via `this.args` within a semantic action. For example, `this.args.depth` would hold the value of the `depth` argument for the current action. 
mySemantics.addAttribute(name: string, actionDict: object) → Semantics
 Exactly like `semantics.addOperation`, except it will add an Attribute to the semantics rather than an Operation. 
mySemantics.extendOperation(name: string, actionDict: object) → Semantics
 Extend the Operation named `name` with the semantic actions contained in `actionDict`. `name` must be the name of an operation in the super semantics β€” i.e., you must first extend the Semantics via [`extendSemantics`](#extendSemantics) before you can extend any of its operations. 
semantics.extendAttribute(name: string, actionDict: object) → Semantics
 Exactly like `semantics.extendOperation`, except it will extend an Attribute of the super semantics rather than an Operation. ### Semantic Actions A semantic action is a function that computes the value of an operation or attribute for a specific type of node in the parse tree. There are three different types of parse nodes: - _Rule application_, or _non-terminal_ nodes, which correspond to rule application expressions - _Terminal_ nodes, for string and number literals, and keyword expressions - _Iteration_ nodes, which are associated with expressions inside a [repetition operator](./syntax-reference.md#repetition-operators) (`*`, `+`, and `?`) Generally, you write a semantic action for each rule in your grammar, and store them together in an _action dictionary_. For example, given the following grammar: ``` Name { FullName = name name name = (letter | "-" | ".")+ } ``` A set of semantic actions for this grammar might look like this: ```js const actions = { FullName(firstName, lastName) { ... }, name(parts) { ... } }; ``` The value of an operation or attribute for a node is the result of invoking the node's matching semantic action. In the grammar above, the body of the `FullName` rule produces two values β€” one for each application of the `name` rule. The values are represented as parse nodes, which are passed as arguments when the semantic action is invoked. An error is thrown if the function arity does not match the number of values produced by the expression. The matching semantic action for a particular node is chosen as follows: - On a _rule application_ (non-terminal) node, first look for a semantic action with the same name as the rule (e.g., 'FullName'). If the action dictionary does not have a property with that name, use the action named `_nonterminal`, if it exists. If there is no `_nonterminal` action, and the node has exactly one child, then return the result of invoking the operation/attribute on the child node. - On a terminal node (e.g., a node produced by the parsing expression `"hello"`), use the semantic action named `_terminal`. - On an iteration node (e.g., a node produced by the parsing expression `letter+`), use the semantic action named `_iter`. The `_iter`, `_nonterminal`, and `_terminal` actions are sometimes called _special actions_. `_iter` and `_nonterminal` take a variable number of arguments, which are typically captured into an array using [rest parameter syntax](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Functions/rest_parameters) , e.g. `_iter(...children) { ... }`. The `_terminal` action takes no arguments. _**NOTE:** Versions of Ohm prior to v16.0 had slightly different behaviour with regards to default semantic actions. See [here](./releases/ohm-js-16.0.md#default-semantic-actions) for more details._ Note that you can also write semantic actions for built-in rules like `letter` or `digit`. For `ListOf`, please see the documentation on [asIteration](#asIteration) below. ### Parse Nodes Each parse node is associated with a particular _parsing expression_ (a fragment of an Ohm grammar), and the node captures any input that was successfully parsed by that expression. Unlike many parsing frameworks, Ohm does not have a syntax for binding/capturing -- every parsing expression captures all the input it consumes, and produces a fixed number of values. A node `n` has the following methods and properties: 
n.child(idx: number) → Node
 Get the child at index `idx`. 
n.isTerminal() → boolean
 `true` if the node is a terminal node, otherwise `false`. 
n.isIteration() → boolean
 `true` if the node is an iteration node (i.e., if it associated with a repetition operator in the grammar), otherwise `false`. 
n.children: Array
 An array containing the node's children. 
n.ctorName: string
 The name of grammar rule that created the node. 
n.source: Interval
 Captures the portion of the input that was consumed by the node. 
n.sourceString: string
 The substring of the input that was consumed by the node. Equivalent to `n.source.contents`. 
n.numChildren: number
 The number of child nodes that the node has. 
n.isOptional() → boolean
 `true` if the node is an iterator node having either one or no child (? operator), otherwise `false`. #### Operations and Attributes In addition to the properties listed above, within a given semantics, every node also has a method/property corresponding to each operation/attribute in the semantics. For example, in a semantics that has an operation named 'prettyPrint' and an attribute named 'freeVars', every node has a `prettyPrint()` method and a `freeVars` property. ## Built-in Operations ### asIteration The built-in `asIteration` operation offers a convenient way of handling _ListOf_ expressions, by adapting them to have the same interface as built-in iteration nodes. As an example, take the following grammar: ``` G { Start = ListOf } ``` ...and an operation defined as follows: ``` s.addOperation('upper()', { Start(list) { return list.asIteration().children.map(c => c.upper()); }, letter(l) { return this.sourceString.toUpperCase(); } }); ``` Then `s(g.match('a, b, c')).upper()` will return `['A', 'B', 'C']`. Note that calling `upper()` on the result of `asIteration` implicitly maps the `upper` operation over each element of the list. You can also extend the `asIteration` operation to handle other list-like rules in your own language. --- ### Doc/Contributor Guide # Contributing to Ohm Interested in contributing to Ohm? Great! Here are some tips to help you get started. ## Getting Started ### Prerequisites To develop Ohm, you'll need a recent version of [Node.js](https://nodejs.org). We support all [_Active LTS_ and _Maintenance LTS_ releases][nodejs-releases]. [nodejs-releases]: https://nodejs.dev/en/about/releases/ You also need to install [pnpm](https://pnpm.io/). ### Basic Setup First, clone the repository: git clone https://github.com/cdglabs/ohm.git Then, install the dev dependencies: cd ohm pnpm install _Note: the `postinstall` script (which is automatically run by `pnpm install`) will install a git pre-commit hook. See [here](#pre-commit-checks) for more information._ ### Useful Scripts The following scripts are useful when developing the main `ohm-js` package: - Use `pnpm test` to run the unit tests. - `pnpm run test-watch` re-runs the unit tests every time a file changes. - `pnpm build` builds `dist/ohm.js` and `dist/ohm.min.js`, which are stand-alone bundles that can be included in a webpage. - When editing Ohm's own grammar (in `src/ohm-grammar.ohm`), run `pnpm run bootstrap` to re-build Ohm and test your changes. ## Doing Development See our recommended [Git workflow](https://github.com/ohmjs/ohm/wiki/Git-Workflow). ### Pre-commit Checks When you run `pnpm install` in an Ohm checkout, it will automatically install a pre-commit hook into `.git/hooks/pre-commit`. Every time you commit to the repository, the pre-commit script checks that all tests pass, and that the code passes a lint check. We use [ESLint](http://eslint.org/), which helps prevent many common programming errors, and ensures that the code follows a consistent style. Usually, you should deal with a lint error by fixing the code so that ESLint no longer complains about it. However, there are a few cases where it makes sense to disable the error: - If you added new `console.log` statement, and **you are sure that it is actual useful**, you can disable the warning like this: ``` console.log('a useful message'); // eslint-disable-line no-console ``` - If you need to introduce a global variable (are you sure?), see the ESLint documentation on [specifying globals](http://eslint.org/docs/user-guide/configuring#specifying-globals). If you need to temporarily commit something that doesn't pass the checks, you can use `git commit --no-verify` -- but use this sparingly! ## Writing documentation - The documentation under doc/ is intended to work on both GitHub and [ohmjs.org](https://ohmjs.org). Since the prefix for GitHub docs is `doc/`, and the prefix on ohmjs.org is `docs/`, a link that includes one of those prefixes won't work in both places! - When linking to other documentation pages, use relative links. Examples: - `[Syntactic vs. Lexical Rules](#syntactic-lexical)` - `[negative lookahead](./syntax-reference.md#negative-lookahead-)` - When linking to source code, use an absolute link to GitHub. Example: - `The [operators example](https://github.com/ohmjs/ohm/tree/main/examples/operators)` - In the [changelog][], use absolute links to GitHub when linking to docs, as that file is intended to be read in a text editor or on GitHub. - Images go under `doc/images/` and should be linked with an absolute URL to ohmjs.org. Example: - `https://ohm.js.org/img/docs/matching.png` for `docs/images/matching.png`. [changelog]: https://github.com/ohmjs/ohm/blob/main/packages/ohm-js/CHANGELOG.md ## Publishing To version and publish the ohm-js package: 1. Update package.json manually to bump the version number. 2. Make sure the CHANGELOG.md is up to date. 3. Run `pnpm publish` 4. Run `git push && git push --tags` 5. Create a release on GitHub, pasting in the changelog contents. --- ### Doc/Docker # Using Ohm.js with Docker The `ohm:latest` image provides the Ohm compiler CLI in a self-contained environment β€” no local Node.js or pnpm installation required. ## Commands overview | Command | Description | |-------------------|-------------| | `compile` | Compile an `.ohm` grammar file to a `.wasm` module | | `generateBundles` | Generate grammar bundles (currently not working) | | `shell` | Open a bash shell inside the container | | `help` | Print usage information | Your current directory is mounted at `/local` inside the container, so relative paths to grammar files work as expected. --- ## 1. Running Pull and run the image directly from Docker Hub: ```sh docker run --rm -v $(pwd):/local ohmjs/ohm:latest compile my-grammar.ohm ``` ### `compile` usage ```sh docker run --rm -v $(pwd):/local ohmjs/ohm:latest compile [options] ``` Options: | Flag | Description | |-------------------------------|-------------| | `--debug` / `-d` | Enable debug output | | `--grammarName` / `-g ` | Override the grammar name | | `--output` / `-o ` | Write output to `` instead of `.wasm` | **Example** β€” compile `arithmetic.ohm` and write the result to `arithmetic.wasm`: ```sh docker run --rm -v $(pwd):/local ohmjs/ohm:latest compile -o arithmetic.wasm arithmetic.ohm ``` ### Getting help ```sh docker run --rm ohmjs/ohm:latest help ``` --- ## 2. Development ### Minimal Docker setup for macOS ```sh brew install colima docker docker-compose docker-credential-helper colima start ``` Add to `~/.docker/config.json`: ```json { "cliPluginsExtraDirs": [ "/opt/homebrew/lib/docker/cli-plugins" ] } ``` > **Note:** If you see `docker-credential-desktop` errors, remove `"credsStore": "desktop"` from `~/.docker/config.json` β€” it references Docker Desktop, which isn't needed with Colima. ### Building the image locally Clone the repository and build the production image with Docker Compose: ```sh git clone https://github.com/ohmjs/ohm.git cd ohm docker compose -f docker/docker-compose.yml build ``` This builds the `ohm:latest` image using the `dist` stage of the multi-stage `Dockerfile`, which produces a slim image containing only the compiled packages and their production dependencies. The `ohm:latest` image is 664 MB and is 99% space efficient (per `wagoodman/dive`). ### Building a development image The development image uses the `build` stage of the `Dockerfile`, which includes the full source tree, all dev dependencies, and the complete build output. This is useful for iterating on the compiler or debugging build issues. **Build:** ```sh TARGET=build docker compose -f docker/docker-compose.yml build ``` This produces the `ohmjs/ohm:development` image. **Run:** ```sh docker run -v $(pwd):/local -it --rm ohmjs/ohm:development shell # or docker run -v ${PWD}:/local -it --rm ohmjs/ohm:development shell ``` The `-v $(pwd):/local` mount makes your current directory available at `/local` inside the container. The `shell` command drops you into a bash session where you can inspect the built artifacts under `/ohm/` or run CLI commands directly. The `ohm-dev:latest` images is 1.62 GB and is 97% efficient with only 64 MB potentially wasted space. ### Publishing to Docker Hub Build and push a versioned image to Docker Hub using the git tag as the version: ```sh # if not set default to ohmjs (ie docker hub using the ohmjs org) export DOCKER_REPO= # if not set defaults to 'development' export VERSION=$(cat packages/runtime/package.json | jq -r '.version') # or export VERSION=$(git describe --tag --dirty) # # it might be necessary (particularly on osx) to create a new builder # # the default builder might not support multi-platform builds # docker buildx create --use --name ohmjs-builder # # might be needed # docker buildx inspect --bootstrap # # or if already created # docker buildx use ohmjs-builder # generate a person access token at https://app.docker.com/accounts/millergarym/settings/personal-access-tokens # assuming DHPAT contains your PAT echo $DHPAT | docker login -u --password-stdin cd docker docker buildx bake --allow=fs.read=.. --push ``` `git describe --tag --dirty` produces a version string based on the nearest git tag, appending commit info and a `-dirty` suffix if there are uncommitted changes. The defaults in `docker-compose.yml` are `DOCKER_REPO=ohmjs` and `VERSION=development`. See [docker-compose.yml](../docker/docker-compose.yml) for details. ## Tips and Tricks ### Image size To track down file which can be deleted. **From inside the container** ```sh apt-get update apt-get install -y ncdu ncdu / ``` **Analysing the images** ```sh alias dive="docker run -ti --rm -v /var/run/docker.sock:/var/run/docker.sock docker.io/wagoodman/dive" dive ohmjs/ohm:latest ``` --- ### Doc/Errors # Ohm Errors ## Instantiating Grammars These errors can occur when [instantiating a grammar](https://github.com/cdglabs/ohm/blob/main/doc/api-reference.md#instantiating-grammars) from a grammar definition. ### Grammar Syntax Error Example: Uncaught ohm.error.GrammarSyntaxError: Failed to parse grammar: Line 5, col 11: 4 | G { > 5 | start = *x ^ 6 | } Expected "~", "&", "#", an identifier, "\"", a number, "(", "[", "``", "{", "--", "|", or "}" Indicates that the grammar definition is not well-formed according to the syntax of the Ohm language. See the [syntax reference](./syntax-reference.md) for more details. ### Undeclared Grammar Example: Uncaught ohm.error.UndeclaredGrammar: Grammar Foo is not declared in namespace [object Object] Indicates that the grammar definition refers to another grammar by name, but that name does not refer to a grammar that Ohm knows about. This can happen when you instantiate a grammar that inherits from another grammar. Possible fixes: - If you're defining multiple grammars, ensure that the supergrammar appears first in the grammar definition. - If the supergrammar was defined elsewhere, you need to pass a namespace argument to the instantiation function (`ohm.grammar()`, etc.), and the namespace must reference the supergrammar with the appropriate property name. E.g.: ```js const ns = {}; ns.G1 = ohm.grammar('G1 {}'); const g2 = ohm.grammar('G2 <: G1 {}', ns); ``` ### Duplicate Grammar Declaration Example: Uncaught ohm.error.DuplicateGrammarDeclaration: Grammar G is already declared in namespace [object Object] Occurs when a grammar definition defines a grammar with the same name as an existing grammar in the same namespace. Possible fixes: - Make sure you're not accidentally instantiating the same grammar twice - Instantiate the second grammar in a different namespace, or without using a namespace. ### Undeclared Rule Example: Uncaught ohm.error.UndeclaredRule: Rule lettr is not declared in grammar G Occurs when the body of a rule refers to a another rule that is not defined in the grammar or in any of its supergrammars. ### Cannot Override Undeclared Rule Example: Uncaught ohm.error.CannotOverrideUndeclaredRule: Cannot override rule foo because it is not declared in BuiltInRules Occurs when a rule is being _overridden_ (using `:=`), but no rule with that name exists in the supergrammar. Learn more about defining, extending, and overriding rules in the [syntax reference](syntax-reference.md#defining-extending-and-overriding-rules). ### Cannot Extend Undeclared Rule Example: Uncaught ohm.error.CannotExtendUndeclaredRule: Cannot extend rule start foo it is not declared in BuiltInRules Occurs when a rule is being _extended_ (using `+=`), but no rule with that name exists in the supergrammar. Learn more about defining, extending, and overriding rules in the [syntax reference](syntax-reference.md#defining-extending-and-overriding-rules). ### Duplicate Rule Declaration Example: Uncaught ohm.error.DuplicateRuleDeclaration: Duplicate declaration for rule 'letter' in grammar 'G' (originally declared in grammar 'BuiltInRules') Occurs when a rule is being _defined_ (using `=`), but a rule with that name already exists in the grammar or supergrammar. If it exists in the supergrammar, you can only override (`:=`) or extend (`+=`) the rule. Learn more about defining, extending, and overriding rules in the [syntax reference](syntax-reference.md#defining-extending-and-overriding-rules). --- ### Doc/Extras # Ohm Extras Ohm comes with a few extras that are not part of Ohm's core but are related to grammars and semantics created with Ohm. Here is how you can access those extras: ```js const extras = require('ohm-js/extras'); // use something like extras.toAST(...); ``` In the future, the extras may contain helper methods, semantics and operations that are useful for many grammars but are not always wanted or necessary and would otherwise e.g. pollute the actions of a grammar. ## toAST(matchResult, [mapping]) `toAST()` is a helper method that takes a successful MatchResult and an optional mapping to convert the concrete syntax tree (**CST**) that Ohm creates when parsing input into an abstract syntax tree (**AST**). To do so, a generic operation is used that can be configure by the optional mapping. The resulting AST is inspired by the [ECMAScript Tree](https://github.com/estree/estree) format that is the output of popular JavaScript parsers like [acorn](https://github.com/ternjs/acorn) or [esprima](http://esprima.org/). **Example:** ```js const ohm = require('ohm-js'); const g = ohm.grammar(` G { Equation = AddExpr AddExpr = number "+" number number = digit+ } `); const match = g.match('24 + 6'); const toAST = require('ohm-js/extras').toAST; const ast = toAST(match); ``` will produce an AST like this: ```js { "type": "AddExpr", // 'type' automatically taken from the rule name "0": "24", // First part (position 0) matched by AddExpr rule "2": "6" // Third part (position 2) matched by AddExpr rule } ``` There are certain general assumptions for the CST-to-AST conversion. They are explained in the following section. ### General Assumptions By default: 1. Every node in an AST has a **'type' property** that is derived from the name of the matching rule (those rule names may consist of the original rule name and the case name (see [Inline Rule Declarations](syntax-reference.md#inline-rule-declarations))). 2. If a node's value is a **concrete value**, like the `"+"` in the example above, it will be omitted in the AST (not if there different possibilities though). 3. If a rule only has one child node, it is considered an **intermediate node** that does not add any value and is therefore omitted. 4. Possibly **repetitive** applications (`*` and `+` operator, `ListOf`) are represented as arrays of values, **optional** applications (`?` operator) are represented by their matched value or `null`. 5. The built-in list rules (`ListOf`/`listOf` and friends) are represented as arrays of values, with the separators discarded. All those defaults can be changed by the optional second parameter - _mapping_ - handed to `toAST()`. ### Configuration Using A Mapping The _mapping_ parameter is a JavaScript object that consists of key-value pairs where the key is name of the rule (including the case name) that needs a custom translation from CST to AST and it's value one of the possibly values below. **Customized Example (see [Example](#example)):** ```js // create ohm, g, match and toAST const ast = toAST(match, { Equation: {content: 0}, AddExpr: {type: 'Expression', expr1: 0, op: 1, expr2: 2}, }); ``` results in an AST like: ```js { "type": "Equation", // explicitly reintroduced node "content": { // "0" named "content" "type": "Expression", // rename type "expr1": "24", // "0" named "expr1" "op": "+", // explicitly reintroduced value "expr2": "6" // "2" named "expr2" } } ``` The following mapping options are possible **on the node-level** (values in the key-value map): - `an object`: A template for a node (see below: possible values _on the property-level_) - `a number`: Does not create a node for the corresponding rule but forwards to the _number_-th children of the rule (node omission). - `a function`: A [semantic action](api-reference.md#semantic-actions) for that rule that replaces the general `toAST()` action for this case. To call the default action on a child node, `[child].toAST(this.args.mapping)` can be called. Additionally, the following mapping options can be use **on the property-level** inside the node templates: - `a number`: Inserts the children's value return by `toAST()` as this property - `a string|a boolean|an object|null`: Values that is used as-is for this property - `a function`: A function similar to a [semantic action](api-reference.md#semantic-actions). Will only have one parameter that is a list of all the CST's child nodes. `[child].toAST(this.args.mapping)` can be called to convert a single child. **Note:** To use a primitive number as value for a property, the number has to be boxed, e.g. `new Number(12)` or `Object(12)`. --- ### Doc/Incremental Semantics # Incremental semantics Ohm supports incremental parsing, meaning that once an input is parsed, it can be quickly reparsed after each edit operation. Incremental _parsing_ is straightforward to use: you just need to use instantiate a [Matcher object](https://ohmjs.org/docs/api-reference#matcher-objects) rather than directly using your Grammar's `match` method. It's also possible to build fully incremental processing pipelines on top of Ohm's incremental parsing, but this is less straightforward. In this document we talk about some of the strategies for doing so. ## Understanding the _overlap rule_ TODO ## Defining an attribute The first step for building an incremental processing pipeline is to define an attribute. In Ohm, an _attribute_ is like an operation, but (a) it takes no arguments, and (b) it is memoized. The attribute's value for a given node will be recalculated whenever the edit may have affected that node. For nodes that are not affected by an edit, the attribute value is cached. So, the simplest kind of incremental processing pipeline you can build consists of a single attribute. For example, for an arithmetic grammar, you might define a `value` attribute for evaluating arithmetic expressions: ```js const semantics = grammar.createSemantics().addAttribute('value', { Exp: (addExp) => addExp.value, AddExp_plus: (left, _op, right) => left.value + right.value, AddExp_minus: (left, _op, right) => left.value - right.value, AddExp: (priExp) => priExp.value, PriExp_paren: (_open, exp, _close) => exp.value, PriExp: (number) => number.value, number(digits) { return parseInt(this.sourceString, 10); } }); ``` And suppose you used the grammar and a `Matcher` object to evaluate an expression, then make an edit, then re-parse and re-evaluate: ```js const m = grammar.matcher(); m.setInput('(1 + 2) + (3 - 4)'); assert.equal(semantics(m.match()).value, 2); m.replaceInputRange(1, 2, '0'); // Replace 1 with 0 assert.equal(semantics(m.match()).value, 1); ``` Note that the `AddExp_minus` action, which calculates the value of "3 - 4", will only run once. The edit does not affect that part of the parse result, so the attribute value is cached. Ohm's caching of attribute values is naive; notably, it does not track dependencies between attribute values or do any kind of [autotracking]([)](https://www.pzuraq.com/blog/what-is-reactivity) as seen in signals frameworks. If the naive caching is not sufficient, there is an **internal** operation named `_forgetMemoizedResultFor` TODO ## Building more complex pipelines You can leverage the caching of attributes to build more complex transformations. For example, if you define an `ast` attribute, you can rely on object identity to determine which AST nodes were affected by an edit: ```js const m = grammar.matcher(); m.setInput("(1 + 2) + (3 - 4)"); const seen = new Set(); // Could also use WeakSet here. let root = semantics(m.match()).ast; seen.add(root); seen.add(root.left); seen.add(root.right); m.replaceInputRange(1, 2, "0"); // Replace 1 with 0 root = semantics(m.match()).ast; // The root and left child are recreated; the right child is reused. assert(!seen.has(root)); assert(!seen.has(root.left)); assert(seen.has(root.right)); ``` You can then build additional transformations on top of the AST, using recursive tree-walking functions and memoization to avoid reprocessing unchanged subtrees. For example: ```js // Returns an array containing all the nubmers in the given subtree. function getNumbers(node, cache = new WeakMap()) { if (cache.has(node)) return cache.get(node); let result; if (node.type === "Number") { result = [node.value]; } else if (node.type === "BinaryOp") { result = [ ...getNumbers(node.left, cache), ...getNumbers(node.right, cache), ]; } else { throw new Error(`Unknown node type: ${node.type}`); } cache.set(node, result); return result; } ``` Rather than implementing the memoization yourself, you can also use an existing library like [memoizee](https://www.npmjs.com/package/memoizee), [micro-memoize](https://www.npmjs.com/package/micro-memoize), etc. ## Dependencies on siblings When an attribute value depends on the value of a sibling, you can: - Define an operation to compute the value, potentially taking one or more arguments with context information. - Define an attribute which caches the value of invoking the operation on all the node's children. This is an extremely useful pattern that can apply to many problems in text processing. For example, Ohm internally uses a version of this when calculating the absolute offset of nodes in the parse tree. Each node caches the relative offsets of all its children, so after an edit that affects _k_ nodes, the offsets can be updated in O(k) time. See [Zed Decoded: Rope & SumTree](https://zed.dev/blog/zed-decoded-rope-sumtree) for a discussion of how this same pattern is used in text editors. --- ### Doc/Indentation Sensitive # Parsing indentation sensitive languages As of v17, Ohm has experimental support for indentation-sensitive languages. This will make it possible to write Ohm grammars for languages like Python and YAML. **NOTE:** as this is API is experimental, it is outside the scope of semver β€”Β which means that we may make breaking changes to the API without a change to Ohm's major version number. ## Background The Ohm language is based on parsing expression grammars (PEGs), and pure PEGs can't express indentation sensitivity. The usual trick is to pre-process the input and insert explicit `indent` and `dedent` tokens, then parse the modified output. While this works, it has a few downsides: - Error messages can be confusing, as they refer to the modified input rather than the original input - You can't use the "real" syntax in the Ohm Editor β€”Β your examples inputs need to be pre-processed - The preprocessing step is trickier than you might think! For these reasons, we decided to add built-in support for indentation-sensitive languages. ## Making indentation-sensitive grammars To define an indentation-sensitive language, create a grammar that inherits from `ExperimentalIndentationSensitive`. For example, here is a grammar for language support nested lists of bullet points: ``` import * as ohm from 'ohm-js'; const outline = ohm.grammar( String.raw` Outline <: IndentationSensitive { Items = Item+ Item = "-" label indent Items dedent -- withChildren | "-" label -- leaf label = (~newline any)* eol eol = newline | end newline = "\r\n" | "\r" | "\n" spaces := (~newline space)* } `, {IndentationSensitive: ohm.ExperimentalIndentationSensitive} ); ``` ## Implementation details The `indent` and `dedent` rules are primitive rules defined by `ExperimentalIndentationSensitive`. You can think of them as special characters that automatically inserted at the appropriate points β€” except that they take up no width in the input stream. They are inserted immediately after the associated indentation characters at the beginning of the line. For example, here is some Python code, with comments indicating where the indents and dedents are inserted: ```python if col == 3: if row == 4: # # <~~ indent at position 2 on this line print("bingo") # <~~ indent at position 4 print("done") # <~~ dedent at position 2 ``` There is also final dedent at the end of the input. ## Examples See examples/indentation-sensitive for an example you can experiment with. ## Notes and open questions - The current implementation _only_ works with spaces (not tabs). Among the indentation-sensitive languages (Python, YAML, Elm, etc.) there are small difference in what characters they allow and how they treat them. We probably want to make this customizable in some way. --- ### Doc/Patterns And Pitfalls # Patterns and Pitfalls ## Grammars ### Dealing with greedy matching In Ohm, like other PEG-based tools, the [repetition operators](./syntax-reference.md#repetition-operators---) `*` and `+` are _greedy_, meaning they always consume as much input as possible. This is different than the way that `*` works in regular expressions. For example, the regular expression `/^a*a/` will successfully match `'aaa'`, whereas in Ohm, the equivalent parsing expression `"a"* "a"` can never match any input. You can use [negative lookahead](./syntax-reference.md#negative-lookahead-) (`~`) to prevent a repetition from matching too many characters. E.g., the following rule will match all but the last 'a' in a string: ``` allButLastA = (~("a" end) "a")* ``` The expression `"a" end` means "match an 'a' at the end of the input", and `~("a" end) "a"` means "match an 'a' only if it is not the last 'a' at the end of the input". For a more realistic example, see the next section on delimited strings. ### Delimited strings A common use for negative lookahead is implementing delimited strings and comments. For example, to support JavaScript-style multiline strings delimited by `` ` ``: ``` stringDelimiter = "`" string = stringDelimiter (~stringDelimiter any)* stringDelimiter ``` The expression `~stringDelimiter any` means "match any character not matched by _stringDelimiter_". ### Supporting comments In most languages, comments are treated as a form of whitespace. Ohm has implicit space skipping ([see Syntactic vs. Lexical rules](./syntax-reference.md#syntactic-vs-lexical-rules)), which is controlled by the _space_ rule. To add comments to your language, you first need to define a _comment_ rule. Here's an example of C-style (`/*`/`*/`-delimited) comments: ``` comment = "/*" (~"*/" any)* "*/" ``` Then, you need to extend the _space_ rule in your grammar so that Ohm will treat the comments as whitespace: ``` space += comment ``` ### Reserved words / keywords Many programming languages have the concept of [reserved words](https://en.wikipedia.org/wiki/Reserved_word) β€” identifiers that have a special meaning, and can't be used as the name of a variable, function, etc. In Ohm grammars, it's common to define a separate lexical rule for each reserved word. For example, here's the definition of [the `keyword` rule in our ES5 grammar](https://github.com/ohmjs/ohm/blob/c7dcbb6b97366daf54349ba8e5be9133978f5c83/examples/ecmascript/src/es5.ohm#L87): ``` keyword = break | do | instanceof | typeof | case | else | new | var | catch | finally | return | void | continue | for | switch | while | debugger | function | this | with | default | if | throw | delete | in | try ``` 🐍 There are a couple of things to watch out for: - One reserved word might be a prefix of another, e.g., `in` and `instanceof` in JavaScript. - Identifiers that begin with a reserved word shouldn't be disallowed, e.g. `className`. To prevent both of these potential problems, you can use [negative lookahead](./syntax-reference.md#negative-lookahead-) in the rules for your reserved words. For example: ``` in = "in" ~identifierPart ``` This ensures that (a) the `in` rule won't accidentally match the wrong keyword (like "instanceof"), and (b) it won't match a valid identifier like "inProgress". ### Matching exactly _n_ times Unlike regular expressions, Ohm does not support [quantifier](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Regular_Expressions/Quantifiers) syntax to indicate the number of times an expression should be matched. However, this can be implemented using a normal sequence: ``` zipCode = digit digit digit digit ``` ### Operator precedence The common way to handle operator precedence in Ohm is to use left-recursive rules which encode the precedence in the grammar structure. For example: ``` exp = addExp addExp = addExp "+" mulExp -- plus | addExp "-" mulExp -- minus | mulExp mulExp = mulExp "*" priExp -- times | mulExp "/" priExp -- divide | priExp ``` Note that the rule for the lower precedence operators (`+` and `-`) invokes the rule for the higher-precedence operators (`*`/`/`). This ensures that the higher-precedence operators "bind more tightly". See Ray Toal's [Operator Precedence and Associativity Examples](https://github.com/ohmjs/ohm/tree/main/examples/operators) for more. #### 🐍 Ambiguous recursion Notice that in the arithmetic grammar above, `mulExp` appears on the right hand side of all of `addExp`'s cases. Be careful that you don't write rules that are "ambiguously recursive", e.g. `addExp = addExp "+" addExp`. If you write your grammar like this, a reader can't tell whether `+` is left-associative or right-associative. (In Ohm, you will actually get a right-associative parse β€” see [#56](https://github.com/ohmjs/ohm/issues/56) for details.) ## Semantics ### Iteration nodes _Iteration nodes_ are associated with expressions inside a repetition operator (`*`, `+`, and `?`). E.g., for the grammar `G { letters = letter+ }`, the single argument to the _letters_ action will be an iteration node. There are two main ways to handle iteration nodes inside semantic actions: 1. Use array operations (`map`, `filter`, etc.) on the node's `children` attribute. For example, `iterNode.children.map(c => c.prettyPrint())` would invoke the `prettyPrint` operation on each child of the iteration node. 2. Define an _\_iter_ action for your operation, which allows you to write something like `iterNode.prettyPrint()`. If you have not defined an _\_iter_ action for the operation, this will result in a "missing semantic action" error. #### Optional nodes An optional node (associated with the `?` operator) is just an iteration node with at most one child. In modern JavaScript (EMCAScript 2020+) and TypeScript, the [optional chaining operator](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Optional_chaining) provides a convenient way to deal with optional nodes: ``` optNode.child(0)?.myOperation(); ``` This evaluates to either (a) `undefined`, if the node has no child, or (b) the result of calling `myOperation()` on the child. In older versions of JavaScript, you can achieve the same thing via `optNode.child(0) && optNode.child(0).myOperation()`. Another way to do the same thing is: `optNode.children.map(c => c.myOperation())[0]`. ### Handling the built-in list rules When using the built-in list rules (`listOf`, etc.) in your grammar, you usually don't need to write semantic actions for them. Instead, you can use the [built-in `asIteration` operation](./api-reference.md#asIteration). --- ### Doc/Philosophy # Ohm Philosophy One feature that distinguishes Ohm from OMeta and other parsing DSLs / frameworks is that Ohm completely separates grammars from semantic actions. In Ohm, a grammar defines a language, and semantic actions specify what to do with valid inputs in that language. The benefits of this separation of concerns include: - **Improved readability** - Grammars are _pure_, not cluttered with semantic actions. - The meaning of the semantic actions is also more obvious when they're seen in isolation, and not scattered throughout a grammar, amidst Kleene-\*s and other control structures. - **Better modularity** - Grammars and semantic actions can be extended independently, using familiar object-oriented mechanisms. - The same grammar can be used with different semantic actions to (for instance) parse, syntax highlight, and compile programs. (In fact, Ohm lets you do all of these things without taking multiple passes over the input.) - **Portability** - The same Ohm grammar can be used, without modification, with Ohm implementations that run on top of different languages, e.g., Ohm/JS and Ohm/Scheme. (Note that semantic actions are still language-dependent.) Another interesting aspect of Ohm is that it applies semantic actions _lazily_, i.e., a semantic action is only evaluated if and when its result is required. The benefits of laziness are: - **A more sensible programming model** - When writing semantic actions, programmers don't have to worry about backtracking. This is especially nice when you're writing side-effectful semantic actions. - **Semantic actions (not the grammar!) control the evaluation of sub-expressions** - Like the visitor design pattern, this enables programmers to specify what to do _before_ and _after_ the semantic actions of sub-epressions are evaluated. - Unlike visitors, laziness enables different semantic actions (i.e., different uses of the same grammar) to customize the order in which sub-expressions are evaluated to suit their own needs. - **Efficiency** - No semantic actions are evaluated if the input is invalid. - No semantic actions are evaluated for the failed branches of an alternation (`|`) expression, i.e., for anything that causes backtracking. - If a semantic action doesn't mention a particular sub-expression, that sub-expression's semantic actions will never be evaluated. --- ### Doc/Publishing Grammars # Publishing Grammars One of the core ideas behind Ohm is that _grammars are APIs_. If you have developed a grammar for a particular language and would like to share it for other people to use and build upon, here's what we suggest: ## Module Exports By convention, modules should export an object with a `grammar` property, and optionally a `semantics` property. For example, see the [ES5](https://github.com/ohmjs/ohm/blob/main/examples/ecmascript/src/es5.js) module in the `examples/` directory. If a single package contains multiple languages, each language should be exported as a separate object. E.g., a package supporting multiple versions of Python might be used like this: ```js const {python2} = require('./your-python-package'); const result = python2.grammar.match('print 3'); python2.semantics(result).eval(); ``` To package a single primary language along with other variants, you can expose the primary language from the top-level module, and the other languages as separate modules within the same package: ```js const smalltalk = require('./your-smalltalk-package'); const smalltalk72 = require('./your-smalltalk-package/smalltalk72'); ``` ## Package Naming To help users find your package, we suggest using a name of the form "ohm-grammar-someLanguage". E.g., the ES5 grammar in the `examples` directory is published on NPM as [ohm-grammar-ecmascript](https://www.npmjs.com/package/ohm-grammar-ecmascript). --- ### Doc/README # Ohm Documentation - [Ohm/JS API Reference](./api-reference.md) - [Ohm Syntax Reference](./syntax-reference.md) - Learn more about the [Ohm philosophy](./philosophy.md) - See [Patterns and Pitfalls](./patterns-and-pitfalls.md) for some common Ohm patterns and solutions to frequently-encountered difficulties. ## Examples Here are some quick samples of what it's like to work with Ohm. For more in-depth examples, see the [examples directory](../examples/), especially the [math example](../examples/math/index.html) which is extensively commented. ### Matching Strings Instantiate a grammar from a string using `ohm.grammar()`, and check inputs using the grammar's `match()` method: ```js const ohm = require('ohm-js'); const g = ohm.grammar(` Laugh { laugh = lol | "lmao" lol = "l" "o"+ "l" }`); assert(g.match('lol').succeeded()); assert(!g.match('lmao').failed()); assert(g.match('loooooool').succeeded()); ``` ### Implementing Semantics You can use _operations_ and _attributes_ to analyze and extract values from parsed data. For example, take the following grammar in `arithmetic.ohm`: ``` Arithmetic { Exp = AddExp AddExp = AddExp "+" PriExp -- plus | AddExp "-" PriExp -- minus | PriExp PriExp = "(" Exp ")" -- paren | number number = digit+ } ``` We can create an operation named 'eval' to evaluate arithmetic expressions that match the grammar: ```js // Instantiate the grammar. const fs = require('fs'); const g = ohm.grammar(fs.readFileSync('arithmetic.ohm')); // Create an operation that evaluates the expression. An operation always belongs to a Semantics, // which is a family of related operations and attributes for a particular grammar. const semantics = g.createSemantics().addOperation('eval', { Exp(e) { return e.eval(); }, AddExp(e) { return e.eval(); }, AddExp_plus(left, op, right) { return left.eval() + right.eval(); }, AddExp_minus(left, op, right) { return left.eval() - right.eval(); }, PriExp(e) { return e.eval(); }, PriExp_paren(open, exp, close) { return exp.eval(); }, number(chars) { return parseInt(this.sourceString, 10); }, }); const match = g.match('1 + (2 - 3) + 4'); assert.equal(semantics(match).eval(), 4); ``` You can learn more about semantics in the [API reference](./api-reference.md#semantics). --- ### Doc/Syntax Reference # Syntax Reference This document describes the syntax of the _Ohm language_, which is a variant of parsing expression grammars (PEGs). If you have experience with PEGs, the Ohm syntax will mostly look familiar, but there are a few important differences to note: - When naming rules, **case matters**: whitespace is implicitly skipped inside a rule application if the rule name begins with an uppercase letter. For further information, see [Syntactic vs. Lexical Rules](#syntactic-lexical). - Grammars are purely about recognition: they do not contain semantic actions (those are defined separately) or bindings. The separation of semantic actions is one of the defining features of Ohm β€” we believe that it improves modularity and makes both grammars and semantics easier to understand. - Alternation expressions support _case names_, which are used in [inline rule declarations](#inline-rule-declarations). This makes semantic actions for alternation expressions simpler and less error-prone. - Ohm does not (yet) support semantic predicates. Ohm is closely related to [OMeta](http://tinlizzie.org/ometa/), another PEG-based language for parsing and pattern matching. Like OMeta, Ohm supports a few features not supported by many PEG parsing frameworks: - [Rule applications](#rule-application) can accept parameters. This makes it possible to write higher-order rules, such as the built-in `ListOf` rule. - Grammars can be extended in an object-oriented way β€” see [Defining, Extending, and Overriding Rules](#defining-extending-and-overriding-rules). ## Terminology ``` Arithmetic { Expr = "1 + 1" } ``` This is a grammar named "Arithmetic", which has a single rule named "Expr". The right hand side of _Expr_ is known as a "rule body". A rule body may be any valid _parsing expression_. ## Parsing Expressions Here is a full list of the different kinds of parsing expressions supported by Ohm: ### Terminals "hello there" Matches exactly the characters contained inside the quotation marks. #### Special characters Special characters (`"`, `\`, and `'`) can be escaped with a backslash β€” e.g., `"\""` will match a literal quote character in the input stream. Other valid escape sequences include: `\b` (backspace), `\f` (form feed), `\n` (line feed), `\r` (carriage return), and `\t` (tab), as well as `\x` followed by 2 hex digits and `\u` followed by 4 hex digits, for matching characters by code point. The \u{hexDigits} escape sequence can be used to represent _any_ Unicode code point, including code points above `0xFFFF`. E.g., `"\u{1F639}"` will match `'😹'`. (_New in Ohm v16.3.0._) **NOTE:** For grammars defined in a JavaScript string literal (i.e., not in a separate .ohm file), it's recommended to use a [template literal with the String.raw tag](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/raw). Without `String.raw`, you'll need to use double-escaping β€” e.g., `\\n` rather than `\n`. ### Terminal Range 
start..end
Matches exactly one code point whose value is between _start_ and _end_ (inclusive). E.g., `"a".."c"` will match `'a'`, `'b'`, or `'c'`. Note: _start_ and _end_ must be [Terminal](#terminals) expressions containing a single character or code point. (_Note:_ Prior to Ohm v16.3.0, terminal ranges only supported code points up `0xFFFF`. As of v16.3.0, higher code points can be specified directly (e.g. `"πŸ˜‡".."😈"`) or with an escape code (`"\u{1F607}".."\u{1F608}"`). ### Rule Application 
ruleName
Matches the body of the rule named _ruleName_. For example, the built-in rule `letter` will parse a string of length 1 that is a letter. 
ruleName<expr>
Matches the body of the _parameterized rule_ named _ruleName_, substituting the parsing expression _expr_ as its first parameter. For parameterized rules with more than one parameter, the parameters are comma-separated, e.g. `ListOf`. ### Repetition operators: \*, +, ? 
expr *
Matches the expression _expr_ repeated 0 or more times. E.g., `"a"*` will match `''`, `'a'`, `'aa'`, ... Inside a _syntactic rule_ β€” any rule whose name begins with an upper-case letter β€” spaces before a match are automatically skipped. E.g., `"a"*` will match `" a a"` as well as `"aa"`. See the documentation on [syntactic and lexical rules](#syntactic-lexical) for more information. 
expr +
Matches the expression _expr_ repeated 1 or more times. E.g., `letter+` will match `'x'`, `'xA'`, ... As with the `*` operator, spaces are skipped when used in a [syntactic rule](#syntactic-lexical). 
expr ?
Tries to match the expression _expr_, succeeding whether it matches or not. No input is consumed if it does not match. ### Sequence 
expr1 expr2
Matches the expression `expr1` followed by `expr2`. E.g., `"grade" letter` will match `'gradeA'`, `'gradeB'`, ... As with the `*` and `+` operators, spaces are skipped when used in a [syntactic rule](#syntactic-lexical). E.g., `"grade" letter` will match `' grade A'` as well as `'gradeA'`. ### Alternation 
expr1 | expr2
Matches the expression `expr1`, and if that does not succeed, matches the expression `expr2`. E.g., `letter | digit` will match `'a'`, `'9'`, ... ### Lookahead: & 
& expr
Succeeds if the expression `expr` can be matched, but does not consume anything from the input stream. Usually used as part of a sequence, e.g. `letter &digit` will match `'a9'`, but only consume 'a'. `&"a" letter+` will match any string of letters that begins with 'a'. ### Negative Lookahead: ~ 
~ expr
Succeeds if the expression `expr` cannot be matched, and does not consume anything from the input stream. Usually used as part of a sequence, e.g., `~"\n" any` will consume any single character that is not a new line character. ### Lexification: # 
# expr
Matches _expr_ as if in a lexical context. This can be used to prevent whitespace skipping before an expression that appears in the body of a syntactic rule. For further information, see [Syntactic vs. Lexical Rules](#syntactic-lexical). ### Comment Inside an Ohm grammar, you can use both single-line (`//`) comments like ``` booleanLiteral = ("true" | "false") // TODO: Should we support "True"/"False" as well? ``` or ``` // For semantics on how decimal literals are constructed, see section 7.8.3 ``` as well as multiline (`/* */`) comments like: ``` /* Note: Punctuator and DivPunctuator (see https://es5.github.io/x7.html#x7.7) are not currently used by this grammar. */ ``` ## Built-in Rules (See [src/built-in-rules.ohm](https://github.com/ohmjs/ohm/blob/main/packages/ohm-js/src/built-in-rules.ohm).) `any`: Matches the next Unicode character β€” i.e., a single code point β€”Β in the input stream, if one exists. **NOTE:** A JavaScript string is a sequence of 16-bit _code units_. Some Unicode characters, such as emoji, are encoded as pairs of 16-bit values. For example, the string `'πŸ˜†'` has length 2, but contains a single Unicode code point. Prior to Ohm v17, `any` always consumed a single 16-bit code unit, rather than a full Unicode character. `letter`: Matches a single character which is a letter (either uppercase or lowercase). `lower`: Matches a single lowercase letter. `upper`: Matches a single uppercase letter. `digit`: Matches a single character which is a digit from 0 to 9. `hexDigit`: Matches a single character which is a either digit or a letter from A-F. `alnum`: Matches a single letter or digit; equivalent to `letter | digit`. `space`: Matches a single whitespace character (e.g., space, tab, newline, etc.) `end`: Matches the end of the input stream. Equivalent to `~any`. caseInsensitive<terminal>: Matches _terminal_, but ignoring any differences in casing (based on the simple, single-character Unicode case mappings). E.g., `caseInsensitive<"ohm">` will match `'Ohm'`, `'OHM'`, etc. ListOf<elem, sep>: Matches the expression _elem_ zero or more times, separated by something that matches the expression _sep_. E.g., `ListOf` will match `''`, `'a'`, and `'a, b, c'`. NonemptyListOf<elem, sep>: Like `ListOf`, but matches _elem_ at least one time. listOf<elem, sep>: Similar to `ListOf` but interpreted as [lexical rule](#syntactic-lexical). applySyntactic<ruleName>: Allows the syntactic rule _ruleName_ to be applied in a lexical context, which is otherwise not allowed. Spaces are skipped _before_ and _after_ the rule application. _New in Ohm v16.1.0._ unicodeChar<categoryOrProp>: matches a single Unicode code point from a given category, or with a given binary property. _categoryOrProp_ is a terminal that is either (a) a valid Unicode General_Category value (e.g. `"Zl"`), or (b) one of the following binary property names: `"XID_Start"`, `"XID_Continue"`, `"White_Space"`. ## Grammar Syntax ### Grammar Inheritance 
grammarName <: supergrammarName { ... }
Declares a grammar named `grammarName` which inherits from `supergrammarName`. ### Defining, Extending, and Overriding Rules In the three forms below, the rule body may optionally begin with a `|` character, which will be ignored. Also note that in rule names, [**case is significant**](#syntactic-lexical). 
ruleName = expr
Defines a new rule named `ruleName` in the grammar, with the parsing expression `expr` as the rule body. Throws an error if a rule with that name already exists in the grammar or one of its supergrammars. 
ruleName := expr
Defines a rule named `ruleName`, overriding a rule of the same name in a supergrammar. Throws an error if no rule with that name exists in a supergrammar. **New in 15.3.0:** The _super-splice_ operator (`...`) can be used to append and/or prepend cases to the supergrammar rule body. E.g., if the supergrammar defines `comment = multiLineComment`, then `comment := ... | singleLineComment` is equivalent to `comment := multiLineComment | singleLineComment`. 
ruleName += expr
Extends a supergrammar rule named `ruleName`, throwing an error if no rule with that name exists in a supergrammar. The rule body will effectively be expr | oldBody, where `oldBody` is the rule body as defined in the supergrammar. Note that as of v15.3.0, the super-splice operator (`...`) offers a more general form of rule extension. E.g., `keyword += "def"` can also be written `keyword := "def" | ...`. #### Parameterized Rules 
ruleName<arg1, ..., argN> = expr
Defines a new rule named `ruleName` which has _n_ parameters. In the rule body _expr_, the parameter names (e.g. _arg1_) may be used as rule applications. E.g., `Repeat = x x`. #### Rule Descriptions Rule declarations may optionally have a description, which is a parenthesized "comment" following the name of the rule in its declaration. Rule descriptions are used to produce better error messages for end users of a language when input is not recognized. For example: ``` ident (an identifier) = ~keyword name ``` #### Inline Rule Declarations 
expr -- caseName
When a parsing expression is followed by the characters `--` and a name, it signals an _inline rule declaration_. This is most commonly used in alternation expressions to ensure that each branch has the same arity. For example, the following declaration: ``` AddExp = AddExp "+" MulExp -- plus | MulExp ``` is equivalent to: ```ohm AddExp = AddExp_plus | MulExp AddExp_plus = AddExp "+" MulExp ```

Syntactic vs. Lexical Rules

A _syntactic rule_ is a rule whose name begins with an uppercase letter, and _lexical rule_ is one whose name begins with a lowercase letter. The difference between lexical and syntactic rules is that syntactic rules implicitly skip whitespace characters. The definition of "whitespace character" is anything that matches the grammar's `space` rule. The default implementation of `space` matches ' ', '\t', '\n', '\r', and any other character that is considered whitespace in the [ES5 spec](http://ecma-international.org/ecma-262/5.1/#sec-7.2). #### How space skipping works In the body of a syntactic rule, Ohm implicitly inserts applications of the `spaces` rule before each expression. (The `spaces` rule is defined as `spaces = space*`.) As an example, take this fragment of JSON grammar: ``` Array = "[" "]" -- empty | "[" Elements "]" -- nonEmpty Elements = Element ("," Element)* ``` `Array` and `Elements` are both syntactic rules, since their names begin with a capital letter. Here's what a lexical version of these rule would look like, with _explicit_ space skipping: ``` array = spaces "[" spaces "]" -- empty | spaces "[" spaces elements spaces "]" -- nonEmpty elements = spaces element (spaces "," spaces element)* ``` In terms of the language it accepts, this version of the rules β€” with explicit space skipping β€” is equivalent to the syntactic version above. A few other details that are helpful to know: 1. If the start rule is a syntactic rule, both leading and trailing spaces are skipped around the top-level application. 2. When the body of a rule contains a [repetition operator](#repetition-operators---) (e.g. `+` or `*`), spaces are skipped before each match. In other words, `Names = name+` is equivalent to `names = (spaces name)+`. 3. The [lexification operator (`#`)](#lexification-) can be used in the body of a syntactic rule to prevent space skipping in specific places. For example: ``` KeyAndValue = #(letter alnum+) ":" #(digit+) ``` is equivalent to: ``` keyAndValue = letter alnum+ spaces ":" digit+ ``` Note that no space skipping occurs _inside_ or _before_ the lexical context defined by the `#` character. That means that this rule will match `'count :33'`, but _not_ `'count: 33'`. --- ### Doc/Typescript # Using Ohm with TypeScript Out of the box, Ohm mostly "just works" with TypeScript. The the built-in type definitions cover the core API, though some less frequently used parts are not (yet) there. If you notice something that's missing, please feel free to [open an issue](https://github.com/ohmjs/ohm/issues/new) or submit a PR. You can see some projects that are using Ohm with TypeScript [on Sourcegraph](https://sourcegraph.com/search?q=context:global+ohm-js+lang:TypeScript+-repo:%5Egithub%5C.com/ohmjs/ohm%24+select:repo+&patternType=literal). ## Grammar-specific type definitions As of Ohm v16, it's also possible to generate type definitions that are specific to your grammar. The main advantages of this are: - the TypeScript compiler can check that your semantic actions have the correct number of arguments and a consistent return type. - IDEs such as [VS Code](https://code.visualstudio.com/) can autocomplete action names, show tooltips with argument types (`IterationNode`, `NonterminalNode`, or `TerminalNode`), etc. To enable grammar-specific type definitions: 1. Install the `@ohm-js/cli` package (e.g., `npm install -D @ohm-js/cli` to add it as a dev dependency). 2. Put your grammar in a separate .ohm file, if it isn't already. 3. Use the Ohm CLI to generate a bundle (.ohm-bundle.js) for your grammar, along with the associated type definitions (.d.ts). For example, if your grammar is in `src/my-grammar.ohm`: ``` npx ohm generateBundles --withTypes 'src/*.ohm' ``` ...will create `src/my-grammar.ohm-bundle.js` and `src/my-grammar.ohm-bundle.d.ts`. You can directly import the bundle like this: ``` import grammar from './my-grammar.ohm-bundle' ``` For more information, see the [Ohm TypeScript example](https://github.com/ohmjs/ohm/blob/main/examples/typescript/src/arithmetic.ts). --- ### Doc/Used By # Used by… Ohm has been around for over 10 years, and is used in a wide variety of interesting projects. Here are some of our favourites: - The [The ERC-funded OnePub project](https://cordis.europa.eu/project/id/101113339) is using Ohm in a collaborative, local-first editor for [Asciidoc](https://asciidoc.org/) documents. - [Ambsheets, a spreadsheet for exploring scenarios](https://www.inkandswitch.com/ambsheets/notebook/) by Ink & Switch. - [SchuBu](https://de.schubu.org/p768/programmieren), a free-to-use digital schoolbook in use by over 100.000 students in Austria, uses Ohm for the Gambu-Script language. - [Bruno](https://usebruno.com/) is an open source IDE for exploring and testing APIs. They used Ohm to develop [the Bru markup language](https://docs.usebruno.com/bru-lang/overview). - [WELLE](https://www.nime.org/proceedings/2020/nime2020_paper114.pdf) by Jens Vetter from KunstuniversitΓ€t Linz, is a web-based music environment for the blind. - [Wildcard](https://www.geoffreylitt.com/wildcard/), by Geoffrey Litt from MIT, is a browser extension that empowers anyone to modify websites to meet their own specific needs. It uses Ohm for its spreadsheet language. - [turtle.audio](http://turtle.audio) is an audio environment where simple text commands generate lines that can play music. - [Shadama](https://tinlizzie.org/~ohshima/shadama2/live2017/), by Yoshiki Ohshima et al, is a live programming environment for particle simulation. --- ### CONTRIBUTING # Contributing to Ohm Interested in contributing to Ohm? This document explains our guidelines for submitting pull requests to the [Ohm GitHub repository](https://github.com/ohmjs/ohm). For information on how to get started with Ohm development, please see the [contributor guide](doc/contributor-guide.md). ## Before submitting If you think you've found a bug in Ohm, please [file an issue describing the problem](https://github.com/ohmjs/ohm/issues) before submitting a PR. If the fix is non-trivial, it's usually best to wait for a response before putting a lot of work into a fix. For new features, it's also best to discuss the idea with us (via an issue or an email to the [mailing list](https://groups.google.com/a/ycr.org/forum/#!forum/ohm)) before submitting a pull request. Also, please ensure the tests pass (using `npm test`) and that there are no lint errors -- though the [pre-commit hook](doc/contributor-guide.md#pre-commit-checks) should do this automatically. ## Formatting To make the review process easier and maximize the chances that your pull request is accepted, here are some guidelines: - Use [informative commit messages](http://chris.beams.io/posts/git-commit/) that follow the standard [git commit template](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html). - Include only the changes that are relevant to your bug/feature. Before submitting, take a look over the diff to ensure that you haven't included any unrelated changes. - Try to keep your pull requests small -- anything more than a few hundred lines is too big. If necessary, break your changes up into several small PRs. ## Review Process When you submit a pull request, one of the Ohm developers will review it and (most likely) leave comments on it. Don't worry -- this is not about judging you, it's about improving the code and helping you get your pull request integrated into the project. A code review is a dialog; it shouldn't be seen as an inspection or approval process. Here are the steps involved in a typical review: - You submit your pull request, including a descriptive message describing its purpose. - An Ohm team member leaves comments on your code, with questions and suggestions. - You follow up on each comment by answering the question, incorporating the suggestion, or explaining why you disagree. - When the reviewer thinks your PR is ready to be merged, they will respond with a comment saying "LGTM", meaning "looks good to me". At this point, if you have permission, you should now merge your PR into main. If not, the reviewer will merge it for you. --- ### README # [Ohm](https://ohmjs.org/) · [](https://www.npmjs.com/package/ohm-js) [](https://github.com/ohmjs/ohm/actions/workflows/node.js.yml) [](https://discord.gg/KwxY5gegRQ) Ohm is a parsing toolkit consisting of a library and a domain-specific language. You can use it to parse custom file formats or quickly build parsers, interpreters, and compilers for programming languages. The _Ohm language_ is based on [parsing expression grammars](http://en.wikipedia.org/wiki/Parsing_expression_grammar) (PEGs), which are a formal way of describing syntax, similar to regular expressions and context-free grammars. The _Ohm library_ provides a JavaScript interface for creating parsers, interpreters, and more from the grammars you write. - **Full support for left-recursive rules** means that you can define left-associative operators in a natural way. - **Object-oriented grammar extension** makes it easy to extend an existing language with new syntax. - **Modular semantic actions.** Unlike many similar tools, Ohm completely separates grammars from semantic actions. This separation improves modularity and extensibility, and makes both grammars and semantic actions easier to read and understand. - **Online editor and visualizer.** The [Ohm Editor](https://ohmjs.org/editor/) provides instant feedback and an [interactive visualization](https://dubroy.com/blog/visualizing-packrat-parsing/) that makes the entire execution of the parser visible and tangible. It'll [make you feel like you have superpowers](https://twitter.com/kylestetz/status/1349770893120172036). πŸ’ͺ Some awesome things people have built using Ohm: - [Seymour](https://harc.github.io/seymour-live2017/), a live programming environment for the classroom. - [Shadama](https://tinlizzie.org/~ohshima/shadama2/live2017/), a particle simulation language designed for high-school science. - [turtle.audio](http://turtle.audio/), an audio environment where simple text commands generate lines that can play music. - A [browser-based tool](https://www.arthurcarabott.com/konnakkol/) that turns written _Konnakkol_ (a South Indian vocal percussion art) into audio. - [Wildcard](https://www.geoffreylitt.com/wildcard/), a browser extension that empowers anyone to modify websites to meet their own specific needs, uses Ohm for its spreadsheet formulas. ## Getting Started The easiest way to get started with Ohm is to use the [interactive editor](https://ohmjs.org/editor/). Alternatively, you can play with one of the following examples on JSFiddle: - [Basic parsing example](https://jsfiddle.net/pdubroy/p3b1v2xb/) - [Arithmetic example with semantics](https://jsfiddle.net/pdubroy/15k63qae/) ### Resources - Tutorial: [Ohm: Parsing Made Easy](https://nextjournal.com/dubroy/ohm-parsing-made-easy) - The [math example](https://github.com/ohmjs/ohm/tree/main/examples/math/index.html) is extensively commented and is a good way to dive deeper. - [Examples](https://github.com/ohmjs/ohm/tree/main/examples/) - [Documentation](doc/README.md) - For community support and discussion, join us on [Discord](https://discord.gg/KwxY5gegRQ), [GitHub Discussions](https://github.com/ohmjs/ohm/discussions), or the [ohm-discuss mailing list](https://groups.google.com/u/0/g/ohm-discuss). - For updates, follow on [Bluesky](hhttps://bsky.app/profile/ohmjs.org) or [Mastodon](https://hachyderm.io/@ohmjs). ### Docker You can use the Ohm CLI without a local Node.js installation via the Docker image: ```sh docker run --rm -v $(pwd):/local ohmjs/ohm:latest compile my-grammar.ohm ``` For full usage instructions, including how to build the image locally and set up a development container, see [doc/docker.md](doc/docker.md). ### Installation #### On a web page To use Ohm in the browser, just add a single ` ``` or ```html ``` This creates a global variable named `ohm`. #### Node.js First, install the `ohm-js` package with your package manager: - [npm](http://npmjs.org): `npm install ohm-js` - [Yarn](https://yarnpkg.com/): `yarn add ohm-js` - [pnpm](https://pnpm.io/): `pnpm add ohm-js` Then, you can use `require` to use Ohm in a script: ```js const ohm = require('ohm-js'); ``` Ohm can also be imported as an ES module: ```js import * as ohm from 'ohm-js'; ``` #### Deno To use Ohm from [Deno](https://deno.land/): ```js import * as ohm from 'https://unpkg.com/ohm-js@17'; ``` ### Basics #### Defining Grammars To use Ohm, you need a grammar that is written in the Ohm language. The grammar provides a formal definition of the language or data format that you want to parse. There are a few different ways you can define an Ohm grammar: - The simplest option is to define the grammar directly in a JavaScript string and instantiate it using `ohm.grammar()`. In most cases, you should use a [template literal with String.raw](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/raw): ```js const myGrammar = ohm.grammar(String.raw` MyGrammar { greeting = "Hello" | "Hola" } `); ``` - **In Node.js**, you can define the grammar in a separate file, and read the file's contents and instantiate it using `ohm.grammar(contents)`: In `myGrammar.ohm`: MyGrammar { greeting = "Hello" | "Hola" } In JavaScript: ```js const fs = require('fs'); const ohm = require('ohm-js'); const contents = fs.readFileSync('myGrammar.ohm', 'utf-8'); const myGrammar = ohm.grammar(contents); ``` For more information, see [Instantiating Grammars](doc/api-reference.md#instantiating-grammars) in the API reference. #### Using Grammars Once you've instantiated a grammar object, use the grammar's `match()` method to recognize input: ```js const userInput = 'Hello'; const m = myGrammar.match(userInput); if (m.succeeded()) { console.log('Greetings, human.'); } else { console.log("That's not a greeting!"); } ``` The result is a MatchResult object. You can use the `succeeded()` and `failed()` methods to see whether the input was recognized or not. For more information, see the [main documentation](doc/README.md). ### Debugging Ohm has two tools to help you debug grammars: a text trace, and a graphical visualizer. [](https://ohmjs.org/editor) You can [try the visualizer online](https://ohmjs.org/editor). To see the text trace for a grammar `g`, just use the [`g.trace()`](doc/api-reference.md#trace) method instead of `g.match`. It takes the same arguments, but instead of returning a MatchResult object, it returns a Trace object β€” calling its `toString` method returns a string describing all of the decisions the parser made when trying to match the input. For example, here is the result of `g.trace('ab').toString()` for the grammar `G { start = letter+ }`: ``` ab βœ“ start β‡’ "ab" ab βœ“ letter+ β‡’ "ab" ab βœ“ letter β‡’ "a" ab βœ“ lower β‡’ "a" ab βœ“ Unicode [Ll] character β‡’ "a" b βœ“ letter β‡’ "b" b βœ“ lower β‡’ "b" b βœ“ Unicode [Ll] character β‡’ "b" βœ— letter βœ— lower βœ— Unicode [Ll] character βœ— upper βœ— Unicode [Lu] character βœ— unicodeLtmo βœ— Unicode [Ltmo] character βœ“ end β‡’ "" ``` ## Publishing Grammars If you've written an Ohm grammar that you'd like to share with others, see our [suggestions for publishing grammars](./doc/publishing-grammars.md). ## Contributing to Ohm Interested in contributing to Ohm? Please read [CONTRIBUTING.md](./CONTRIBUTING.md) and the [Ohm Contributor Guide](doc/contributor-guide.md). ---