lark Documentation

Technical Documentation: lark-parser/lark

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Lark - a parsing toolkit for Python

Lark is a parsing toolkit for Python, built with a focus on ergonomics, performance and modularity.

Lark can parse all context-free languages. To put it simply, it means that it is capable of parsing almost any programming language out there, and to some degree most natural languages too.

Who is it for?

• Beginners: Lark is very friendly for experimentation. It can parse any grammar you throw at it, no matter how complicated or ambiguous, and do so efficiently. It also constructs an annotated parse-tree for you, using only the grammar and an input, and it gives you convenient and flexible tools to process that parse-tree.

• Experts: Lark implements both Earley(SPPF) and LALR(1), and several different lexers, so you can trade-off power and speed, according to your requirements. It also provides a variety of sophisticated features and utilities.

What can it do?

• Parse all context-free grammars, and handle any ambiguity gracefully
• Build an annotated parse-tree automagically, no construction code required.
• Provide first-rate performance in terms of both Big-O complexity and measured run-time (considering that this is Python ;)
• Run on every Python interpreter (it's pure-python)
• Generate a stand-alone parser (for LALR(1) grammars)

And many more features. Read ahead and find out!

Most importantly, Lark will save you time and prevent you from getting parsing headaches.

Quick links Copy Section

• Documentation @readthedocs
• Cheatsheet (PDF)
• Online IDE
• Tutorial for writing a JSON parser.
• Blog post: How to write a DSL with Lark
• Gitter chat

Install Lark Copy Section

$ pip install lark --upgrade

Lark has no dependencies.

Syntax Highlighting Copy Section

Lark provides syntax highlighting for its grammar files (*.lark):

• Sublime Text & TextMate
• vscode
• Intellij & PyCharm
• Vim
• Atom

Clones Copy Section

These are implementations of Lark in other languages. They accept Lark grammars, and provide similar utilities.

• Lerche (Julia) - an unofficial clone, written entirely in Julia.
• Lark.js (Javascript) - a port of the stand-alone LALR(1) parser generator to Javascsript.

Hello World Copy Section

Here is a little program to parse "Hello, World!" (Or any other similar phrase):

from lark import Lark

l = Lark('''start: WORD "," WORD "!"

            %import common.WORD   // imports from terminal library
            %ignore " "           // Disregard spaces in text
         ''')

print( l.parse("Hello, World!") )

And the output is:

Tree(start, [Token(WORD, 'Hello'), Token(WORD, 'World')])

Notice punctuation doesn't appear in the resulting tree. It's automatically filtered away by Lark.

Fruit flies like bananas Copy Section

Lark is great at handling ambiguity. Here is the result of parsing the phrase "fruit flies like bananas":

Read the code here, and see more examples here.

List of main features Copy Chapter

• Builds a parse-tree (AST) automagically, based on the structure of the grammar
• Earley parser
  • Can parse all context-free grammars
  • Full support for ambiguous grammars
• LALR(1) parser
  • Fast and light, competitive with PLY
  • Can generate a stand-alone parser (read more)
• EBNF grammar
• Unicode fully supported
• Automatic line & column tracking
• Interactive parser for advanced parsing flows and debugging
• Grammar composition - Import terminals and rules from other grammars
• Standard library of terminals (strings, numbers, names, etc.)
• Import grammars from Nearley.js (read more)
• Extensive test suite
• Type annotations (MyPy support)
• And much more!

See the full list of features here

Comparison to other libraries Copy Section

Performance comparison

Lark is fast and light (lower is better)

Check out the JSON tutorial for more details on how the comparison was made.

For thorough 3rd-party benchmarks, checkout the Python Parsing Benchmarks repo.

Feature comparison

(* PEGs cannot handle non-deterministic grammars. Also, according to Wikipedia, it remains unanswered whether PEGs can really parse all deterministic CFGs)

Projects using Lark Copy Section

• Poetry - A utility for dependency management and packaging
• Vyper - Pythonic Smart Contract Language for the EVM
• PyQuil - Python library for quantum programming using Quil
• Preql - An interpreted relational query language that compiles to SQL
• Hypothesis - Library for property-based testing
• mappyfile - a MapFile parser for working with MapServer configuration
• tartiflette - GraphQL server by Dailymotion
• synapse - an intelligence analysis platform
• Datacube-core - Open Data Cube analyses continental scale Earth Observation data through time
• SPFlow - Library for Sum-Product Networks
• Torchani - Accurate Neural Network Potential on PyTorch
• Command-Block-Assembly - An assembly language, and C compiler, for Minecraft commands
• EQL - Event Query Language
• Fabric-SDK-Py - Hyperledger fabric SDK with Python 3.x
• required - multi-field validation using docstrings
• miniwdl - A static analysis toolkit for the Workflow Description Language
• pytreeview - a lightweight tree-based grammar explorer
• harmalysis - A language for harmonic analysis and music theory
• gersemi - A CMake code formatter
• MistQL - A query language for JSON-like structures
• Outlines - Structured generation with Large Language Models

Full list

License Copy Chapter

Lark uses the MIT license.

(The standalone tool is under MPL2)

Contributors Copy Chapter

Lark accepts pull-requests. See How to develop Lark

Big thanks to everyone who contributed so far:

Sponsor Copy Chapter

If you like Lark, and want to see us grow, please consider sponsoring us!

Contact the author Copy Chapter

Questions about code are best asked on gitter or in the issues.

For anything else, I can be reached by email at erezshin at gmail com.

-- Erez

Classes (`docs/classes.rst`) Copy Section

API Reference

Lark Copy Chapter

.. autoclass:: lark.Lark
:members: open, parse, parse_interactive, scan, lex, save, load, get_terminal, open_from_package

Using Unicode character classes with regex
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Python's builtin re module has a few persistent known bugs and also won't parse
advanced regex features such as character classes.
With pip install lark[regex], the regex module will be
installed alongside lark and can act as a drop-in replacement to re.

Any instance of Lark instantiated with regex=True will use the regex module instead of re.

For example, we can use character classes to match PEP-3131 compliant Python identifiers:

::

from lark import Lark
>>> g = Lark(r"""
                    ?start: NAME
                    NAME: ID_START ID_CONTINUE*
                    ID_START: /[\p{Lu}\p{Ll}\p{Lt}\p{Lm}\p{Lo}\p{Nl}_]+/
                    ID_CONTINUE: ID_START | /[\p{Mn}\p{Mc}\p{Nd}\p{Pc}·]+/
                """, regex=True)

>>> g.parse('வணக்கம்')
'வணக்கம்'

Tree Copy Chapter

.. autoclass:: lark.Tree
:members: pretty, find_pred, find_data, iter_subtrees, scan_values,
iter_subtrees_topdown, rich

Token Copy Chapter

.. autoclass:: lark.Token

Transformer, Visitor & Interpreter Copy Chapter

See :doc:visitors.

ForestVisitor, ForestTransformer, & TreeForestTransformer Copy Chapter

See :doc:forest.

UnexpectedInput Copy Chapter

.. autoclass:: lark.exceptions.UnexpectedInput
:members: get_context, match_examples

.. autoclass:: lark.exceptions.UnexpectedToken

.. autoclass:: lark.exceptions.UnexpectedCharacters

.. autoclass:: lark.exceptions.UnexpectedEOF

InteractiveParser Copy Chapter

.. autoclass:: lark.parsers.lalr_interactive_parser.InteractiveParser
:members: choices, feed_token, copy, pretty, resume_parse, exhaust_lexer, accepts, as_immutable

.. autoclass:: lark.parsers.lalr_interactive_parser.ImmutableInteractiveParser
:members: choices, feed_token, copy, pretty, resume_parse, exhaust_lexer, accepts, as_mutable

ast_utils Copy Chapter

For an example of using ast_utils, see /examples/advanced/create_ast.py_

.. autoclass:: lark.ast_utils.Ast

.. autoclass:: lark.ast_utils.AsList

.. autofunction:: lark.ast_utils.create_transformer

.. _/examples/advanced/create_ast.py: examples/advanced/create_ast.html

Indenter Copy Chapter

.. autoclass:: lark.indenter.Indenter
:members: NL_type, OPEN_PAREN_types, CLOSE_PAREN_types, INDENT_type, DEDENT_type, tab_len
.. autoclass:: lark.indenter.PythonIndenter

TextSlice Copy Chapter

.. autoclass:: lark.utils.TextSlice

ScanMatch Copy Chapter

A single match yielded by :meth:Lark.scan.

.. autoclass:: lark.ScanMatch

Features (`docs/features.md`) Copy Section

Features

Main Features Copy Chapter

• Earley parser, capable of parsing any context-free grammar
  • Implements SPPF, for efficient parsing and storing of ambiguous grammars.
• LALR(1) parser, limited in power of expression, but very efficient in space and performance (O(n)).
  • Implements a parse-aware lexer that provides a better power of expression than traditional LALR implementations (such as ply).
• EBNF-inspired grammar, with extra features (See: Grammar Reference)
• Builds a parse-tree (AST) automagically based on the grammar
• Stand-alone parser generator - create a small independent parser to embed in your project. (read more)
• Flexible error handling by using an interactive parser interface (LALR only)
• Automatic line & column tracking (for both tokens and matched rules)
• Automatic terminal collision resolution
• Warns on regex collisions using the optional interegular library. (read more)
• Grammar composition - Import terminals and rules from other grammars (see example).
• Standard library of terminals (strings, numbers, names, etc.)
• Unicode fully supported
• Extensive test suite
• Type annotations (MyPy support)
• Pure-Python implementation

Read more about the parsers

Extra features Copy Chapter

• Lark.scan() for finding non-overlapping grammar matches embedded in arbitrary text (LALR only — see recipes)
• Support for external regex module (see here)
• Import grammars from Nearley.js (read more)
• CYK parser
• Visualize your parse trees as dot or png files (see_example)
• Automatic reconstruction of input from parse-tree (see example and another example)
• Use Lark grammars in Julia and Javascript.

Forest (`docs/forest.rst`) Copy Section

Working with the SPPF

When parsing with Earley, Lark provides the ambiguity='forest' option
to obtain the shared packed parse forest (SPPF) produced by the parser as
an alternative to it being automatically converted to a tree.

Lark provides a few tools to facilitate working with the SPPF. Here are some
things to consider when deciding whether or not to use the SPPF.

Pros

• Efficient storage of highly ambiguous parses
• Precise handling of ambiguities
• Custom rule prioritizers
• Ability to handle infinite ambiguities
• Directly transform forest -> object instead of forest -> tree -> object

Cons

• More complex than working with a tree

• SPPF may contain nodes corresponding to rules generated internally

• Loss of Lark grammar features:

  • Rules starting with '_' are not inlined in the SPPF
  • Rules starting with '?' are never inlined in the SPPF
  • All tokens will appear in the SPPF

SymbolNode Copy Chapter

.. autoclass:: lark.parsers.earley_forest.SymbolNode
:members: is_ambiguous, children

PackedNode Copy Chapter

.. autoclass:: lark.parsers.earley_forest.PackedNode
:members: children

ForestVisitor Copy Chapter

.. autoclass:: lark.parsers.earley_forest.ForestVisitor
:members: visit, visit_symbol_node_in, visit_symbol_node_out,
visit_packed_node_in, visit_packed_node_out,
visit_token_node, on_cycle, get_cycle_in_path

ForestTransformer Copy Chapter

.. autoclass:: lark.parsers.earley_forest.ForestTransformer
:members: transform, transform_symbol_node, transform_intermediate_node,
transform_packed_node, transform_token_node

TreeForestTransformer Copy Chapter

.. autoclass:: lark.parsers.earley_forest.TreeForestTransformer
:members: default, default_token, default_ambig

handles_ambiguity Copy Chapter

.. autofunction:: lark.parsers.earley_forest.handles_ambiguity

Grammar (`docs/grammar.md`) Copy Section

Grammar Reference

Definitions Copy Chapter

A grammar is a list of rules and terminals, that together define a language.

Terminals define the alphabet of the language, while rules define its structure.

In Lark, a terminal may be a string, a regular expression, or a concatenation of these and other terminals.

Each rule is a list of terminals and rules, whose location and nesting define the structure of the resulting parse-tree.

A parsing algorithm is an algorithm that takes a grammar definition and a sequence of symbols (members of the alphabet), and matches the entirety of the sequence by searching for a structure that is allowed by the grammar.

General Syntax and notes Copy Section

Grammars in Lark are based on EBNF syntax, with several enhancements.

EBNF is basically a short-hand for common BNF patterns.

Optionals are expanded:

a b? c    ->    (a c | a b c)

Repetition is extracted into a recursion:

a: b*    ->    a: _b_tag
                 _b_tag: (_b_tag b)?

And so on.

Lark grammars are composed of a list of definitions and directives, each on its own line. A definition is either a named rule, or a named terminal, with the following syntax, respectively:

rule: <EBNF-EXPRESSION>
      | etc.

  TERM: <EBNF-EXPRESSION>   // Rules aren't allowed

Comments start with
either // (C++ style) or # (Python style, since version 1.1.6)
and last to the end of the line.

Lark begins the parse with the rule 'start', unless specified otherwise in the options.

Names of rules are always in lowercase, while names of terminals are always in uppercase. This distinction has practical effects, for the shape of the generated parse-tree, and the automatic construction of the lexer (aka tokenizer, or scanner).

Terminals Copy Chapter

Terminals are used to match text into symbols. They can be defined as a combination of literals and other terminals.

Syntax:

<NAME> [. <priority>] : <literals-and-or-terminals>

Terminal names must be uppercase.

Literals can be one of:

• "string"
• /regular expression+/
• "case-insensitive string"i
• /re with flags/imulx
• Literal range: "a".."z", "1".."9", etc.

Terminals also support grammar operators, such as |, +, * and ?.

Terminals are a linear construct, and therefore may not contain themselves (recursion isn't allowed).

Templates Copy Section

Templates are expanded when preprocessing the grammar.

Definition syntax:

my_template{param1, param2, ...}: <EBNF EXPRESSION>

Use syntax:

some_rule: my_template{arg1, arg2, ...}

Example:

_separated{x, sep}: x (sep x)*  // Define a sequence of 'x sep x sep x ...'

num_list: "[" _separated{NUMBER, ","} "]"   // Will match "[1, 2, 3]" etc.

Priority Copy Section

Terminals can be assigned a priority to influence lexing. Terminal priorities
are signed integers with a default value of 0.

When using a lexer, the highest priority terminals are always matched first.

When using Earley's dynamic lexing, terminal priorities are used to prefer
certain lexings and resolve ambiguity.

Regexp Flags Copy Section

You can use flags on regexps and strings. For example:

SELECT: "select"i     //# Will ignore case, and match SELECT or Select, etc.
MULTILINE_TEXT: /.+/s
SIGNED_INTEGER: /
    [+-]?  # the sign
    (0|[1-9][0-9]*)  # the digits
 /x

Supported flags are one of: imslux. See Python's regex documentation for more details on each one.

Regexps/strings of different flags can only be concatenated in Python 3.6+

Notes for when using a lexer:

When using a lexer (basic or contextual), it is the grammar-author's responsibility to make sure the literals don't collide, or that if they do, they are matched in the desired order. Literals are matched according to the following precedence:

1. Highest priority first (priority is specified as: TERM.number: ...)
2. Length of match (for regexps, the longest theoretical match is used)
3. Length of literal / pattern definition
4. Name

Examples:

IF: "if"
INTEGER : /[0-9]+/
INTEGER2 : ("0".."9")+          //# Same as INTEGER
DECIMAL.2: INTEGER? "." INTEGER  //# Will be matched before INTEGER
WHITESPACE: (" " | /\t/ )+
SQL_SELECT: "select"i

Regular expressions & Ambiguity Copy Section

Each terminal is eventually compiled to a regular expression. All the operators and references inside it are mapped to their respective expressions.

For example, in the following grammar, A1 and A2, are equivalent:

A1: "a" | "b"
A2: /a|b/

This means that inside terminals, Lark cannot detect or resolve ambiguity, even when using Earley.

For example, for this grammar:

start           : (A | B)+
A               : "a" | "ab"
B               : "b"

We get only one possible derivation, instead of two:

>>> p = Lark(g, ambiguity="explicit")
>>> p.parse("ab")
Tree('start', [Token('A', 'ab')])

This is happening because Python's regex engine always returns the best matching option. There is no way to access the alternatives.

If you find yourself in this situation, the recommended solution is to either use the "dynamic_complete" lexer, or use rules instead.

Example using rules:

>>> p = Lark("""start: (a | b)+
...             !a: "a" | "ab"
...             !b: "b"
...             """, ambiguity="explicit")
>>> print(p.parse("ab").pretty())
_ambig
  start
    a   ab
  start
    a   a
    b   b

Example using dynamic-complete:

>>> g = """
...     start: (A | B)+
...     A    : "a" | "ab"
...     B    : "b"
... """
>>> p = Lark(g, ambiguity="explicit", lexer="dynamic_complete")
>>> rich.print(p.parse("ab"))
_ambig
├── start
│   └── ab
└── start
    ├── a
    └── b

(note: the dynamic-complete lexer can significantly affect the performance of the parser)

Rules Copy Chapter

Syntax:

<name> : <items-to-match>  [-> <alias> ]
       | ...

Names of rules and aliases are always in lowercase.

Rule definitions can be extended to the next line by using the OR operator (signified by a pipe: | ).

An alias is a name for the specific rule alternative. It affects tree construction.

Each item is one of:

• rule
• TERMINAL
• "string literal" or /regexp literal/
• (item item ..) - Group items
• [item item ..] - Maybe. Same as (item item ..)?, but when maybe_placeholders=True, generates None if there is no match.
• item? - Zero or one instances of item ("maybe")
• item* - Zero or more instances of item
• item+ - One or more instances of item
• item ~ n - Exactly n instances of item
• item ~ n..m - Between n to m instances of item (not recommended for wide ranges, due to performance issues)

Note that all operators, including ~ n, apply directly to the item on their left.
For instance a ~ 2 b ~ 3 is parsed as (a ~ 2) (b ~ 3).

Examples:

hello_world: "hello" "world"
mul: (mul "*")? number     //# Left-recursion is allowed and encouraged!
expr: expr operator expr
    | value               //# Multi-line, belongs to expr

four_words: word ~ 4

Priority Copy Section

Like terminals, rules can be assigned a priority. Rule priorities are signed
integers with a default value of 0.

When using LALR, the highest priority rules are used to resolve collision errors.

When using Earley, rule priorities are used to resolve ambiguity.

Directives Copy Chapter

%ignore Copy Section

All occurrences of the terminal will be ignored, and won't be part of the parse.

Using the %ignore directive results in a cleaner grammar.

It's especially important for the LALR(1) algorithm, because adding whitespace (or comments, or other extraneous elements) explicitly in the grammar, harms its predictive abilities, which are based on a lookahead of 1.

Syntax:

%ignore <TERMINAL>

Examples:

%ignore " "

COMMENT: "#" /[^\n]/*
%ignore COMMENT

%import Copy Section

Allows one to import terminals and rules from lark grammars.

When importing rules, all their dependencies will be imported into a namespace, to avoid collisions. To override any of their dependencies (e.g. like you would override methods when inheriting a class), use the %override directive.

Syntax:

%import <module>.<TERMINAL>
%import <module>.<rule>
%import <module>.<TERMINAL> -> <NEWTERMINAL>
%import <module>.<rule> -> <newrule>
%import <module> (<TERM1>, <TERM2>, <rule1>, <rule2>)

If the module path is absolute, Lark will attempt to load it from the built-in directory (which currently contains common.lark, lark.lark, python.lark, and unicode.lark).

If the module path is relative, such as .path.to.file, Lark will attempt to load it from the current working directory. Grammars must have the .lark extension.

The rule or terminal can be imported under another name (an alias) with the -> syntax.

Example:

%import common.NUMBER

%import .terminals_file (A, B, C)

%import .rules_file.rule_a -> rule_b

Note that %ignore directives cannot be imported. Imported rules will abide by the %ignore directives declared in the main grammar.

%declare Copy Section

Declare a terminal without defining it. Useful for plugins.

%override Copy Section

Override a rule or terminals, affecting all references to it, even in imported grammars.

Useful for implementing an inheritance pattern when importing grammars.

Example:

%import my_grammar (start, number, NUMBER)

// Add hex support to my_grammar
%override number: NUMBER | /0x\w+/

%extend Copy Section

Extend the definition of a rule or terminal, e.g. add a new option on what it can match, like when separated with |.

Useful for splitting up a definition of a complex rule with many different options over multiple files.

Can also be used to implement a plugin system where a core grammar is extended by others.

Example:

%import my_grammar (start, NUMBER)

// Add hex support to my_grammar
%extend NUMBER: /0x\w+/

For both %extend and %override, there is not requirement for a rule/terminal to come from another file, but that is probably the most common use-case.

How To Develop (`docs/how_to_develop.md`) Copy Section

How to develop Lark - Guide

There are many ways you can help the project:

• Help solve issues
• Improve the documentation
• Write new grammars for Lark's library
• Write a blog post introducing Lark to your audience
• Port Lark to another language
• Help with code development

If you're interested in taking one of these on, contact us on Gitter or Github Discussion, and we will provide more details and assist you in the process.

Code Style Copy Chapter

Lark does not follow a predefined code style.
We accept any code style that makes sense, as long as it's Pythonic and easy to read.

Unit Tests Copy Chapter

Lark comes with an extensive set of tests. Many of the tests will run several times, once for each parser configuration.

To run the tests, just go to the lark project root, and run the command:

python -m tests

or

pypy -m tests

For a list of supported interpreters, you can consult the tox.ini file.

You can also run a single unittest using its class and method name, for example:

##   test_package test_class_name.test_function_name
python -m tests TestLalrBasic.test_keep_all_tokens

tox Copy Section

To run all Unit Tests with tox,
install tox and Python 2.7 up to the latest python interpreter supported (consult the file tox.ini).
Then,
run the command tox on the root of this project (where the main setup.py file is on).

And, for example,
if you would like to only run the Unit Tests for Python version 2.7,
you can run the command tox -e py27

pytest Copy Section

You can also run the tests using pytest:

pytest tests

Using setup.py Copy Section

Another way to run the tests is using setup.py:

python setup.py test

Building the Documentation Copy Chapter

To build the documentation:

cd docs/
pip install -r requirements.txt
make html

To review the result, open the built HTML files under _build/html/ in your browser.

How To Use (`docs/how_to_use.md`) Copy Section

How To Use Lark - Guide

Work process Copy Chapter

This is the recommended process for working with Lark:

1. Collect or create input samples, that demonstrate key features or behaviors in the language you're trying to parse.

2. Write a grammar. Try to aim for a structure that is intuitive, and in a way that imitates how you would explain your language to a fellow human.

3. Try your grammar in Lark against each input sample. Make sure the resulting parse-trees make sense.

4. Use Lark's grammar features to shape the tree: Get rid of superfluous rules by inlining them, and use aliases when specific cases need clarification.

   You can perform steps 1-4 repeatedly, gradually growing your grammar to include more sentences.

5. Create a transformer to evaluate the parse-tree into a structure you'll be comfortable to work with. This may include evaluating literals, merging branches, or even converting the entire tree into your own set of AST classes.

Of course, some specific use-cases may deviate from this process. Feel free to suggest these cases, and I'll add them to this page.

Getting started Copy Chapter

Browse the Examples to find a template that suits your purposes.

Read the tutorials to get a better understanding of how everything works. (links in the main page)

Use the Cheatsheet (PDF) for quick reference.

Use the reference pages for more in-depth explanations. (links in the main page)

Debug Copy Chapter

Grammars may contain non-obvious bugs, usually caused by rules or terminals interfering with each other in subtle ways.

When trying to debug a misbehaving grammar, the following methodology is recommended:

1. Create a copy of the grammar, so you can change the parser/grammar without any worries
2. Find the minimal input that creates the error
3. Slowly remove rules from the grammar, while making sure the error still occurs.

Usually, by the time you get to a minimal grammar, the problem becomes clear.

But if it doesn't, feel free to ask us on gitter, or even open an issue. Post a reproducing code, with the minimal grammar and input, and we'll do our best to help.

Regex collisions Copy Section

A likely source of bugs occurs when two regexes in a grammar can match the same input. If both terminals have the same priority, most lexers would arbitrarily choose the first one that matches, which isn't always the desired one. (a notable exception is the dynamic_complete lexer, which always tries all variations. But its users pay for that with performance.)

These collisions can be hard to notice, and their effects can be difficult to debug, as they are subtle and sometimes hard to reproduce.

To help with these situations, Lark can utilize a new external library called interegular. If it is installed, Lark uses it to check for collisions, and warn about any conflicts that it can find:

import logging
from lark import Lark, logger

logger.setLevel(logging.WARN)

collision_grammar = '''
start: A | B
A: /a+/
B: /[ab]+/
'''
p = Lark(collision_grammar, parser='lalr')

# Output:
# Collision between Terminals B and A. The lexer will choose between them arbitrarily
# Example Collision: a

You can install interegular for Lark using pip install 'lark[interegular]'.

Note 1: Interegular currently only runs when the lexer is basic or contextual.

Note 2: Some advanced regex features, such as lookahead and lookbehind, may prevent interegular from detecting existing collisions.

Shift/Reduce collisions Copy Section

By default Lark automatically resolves Shift/Reduce conflicts as Shift. It produces notifications as debug messages.

when users pass debug=True, those notifications are written as warnings.

Either way, to get the messages printed you have to configure the logger beforehand. For example:

import logging
from lark import Lark, logger

logger.setLevel(logging.DEBUG)

collision_grammar = '''
start: as as
as: a*
a: "a"
'''
p = Lark(collision_grammar, parser='lalr', debug=True)
# Shift/Reduce conflict for terminal A: (resolving as shift)
#  * <as : >
# Shift/Reduce conflict for terminal A: (resolving as shift)
#  * <as : __as_star_0>

Strict-Mode Copy Section

Lark, by default, accepts grammars with unresolved Shift/Reduce collisions (which it always resolves to shift), and regex collisions.

Strict-mode allows users to validate that their grammars don't contain these collisions.

When Lark is initialized with strict=True, it raises an exception on any Shift/Reduce or regex collision.

If interegular isn't installed, an exception is thrown.

When using strict-mode, users will be expected to resolve their collisions manually:

• To resolve Shift/Reduce collisions, adjust the priority weights of the rules involved, until there are no more collisions.

• To resolve regex collisions, change the involved regexes so that they can no longer both match the same input (Lark provides an example).

Strict-mode only applies to LALR for now.

from lark import Lark

collision_grammar = '''
start: as as
as: a*
a: "a"
'''
p = Lark(collision_grammar, parser='lalr', strict=True)

# Traceback (most recent call last):
#   ...
# lark.exceptions.GrammarError: Shift/Reduce conflict for terminal A. [strict-mode]

Tools Copy Chapter

Stand-alone parser Copy Section

Lark can generate a stand-alone LALR(1) parser from a grammar.

The resulting module provides the same interface as Lark, but with a fixed grammar, and reduced functionality.

Run using:

python -m lark.tools.standalone

For a play-by-play, read the tutorial

Import Nearley.js grammars Copy Section

It is possible to import Nearley grammars into Lark. The Javascript code is translated using Js2Py.

See the tools page for more information.

Index (`docs/index.rst`) Copy Section

.. Lark documentation master file, created by
sphinx-quickstart on Sun Aug 16 13:09:41 2020.
You can adapt this file completely to your liking, but it should at least
contain the root toctree directive.

Welcome to Lark's documentation!

.. toctree::
:maxdepth: 2
:caption: Overview
:hidden:

philosophy
features
parsers

.. toctree::
:maxdepth: 2
:caption: Tutorials & Guides
:hidden:

json_tutorial
how_to_use
how_to_develop
pyparsing_porting
recipes
examples/index

.. toctree::
:maxdepth: 2
:caption: Reference
:hidden:

grammar
tree_construction
classes
visitors
forest
tools

Lark is a modern parsing library for Python. Lark can parse any context-free grammar.

Lark provides:

• Advanced grammar language, based on EBNF
• Three parsing algorithms to choose from: Earley, LALR(1) and CYK
• Automatic tree construction, inferred from your grammar
• Fast unicode lexer with regexp support, and automatic line-counting

Install Lark Copy Chapter

.. code:: bash

$ pip install lark

Syntax Highlighting Copy Chapter

• Sublime Text & TextMate_
• Visual Studio Code_ (Or install through the vscode plugin system)
• Intellij & PyCharm_
• Vim_
• Atom_

.. _Sublime Text & TextMate: https://github.com/lark-parser/lark_syntax
.. _Visual Studio Code: https://github.com/lark-parser/vscode-lark
.. _Intellij & PyCharm: https://github.com/lark-parser/intellij-syntax-highlighting
.. _Vim: https://github.com/lark-parser/vim-lark-syntax
.. _Atom: https://github.com/Alhadis/language-grammars

Resources Copy Chapter

• :doc:philosophy

• :doc:features

• Examples_

• Third-party examples_

• Online IDE_

• Tutorials

  • How to write a DSL_ - Implements a toy LOGO-like language with
    an interpreter

  • :doc:json_tutorial - Teaches you how to use Lark

  • Unofficial

    • Program Synthesis is Possible_ - Creates a DSL for Z3
    • Using Lark to Parse Text - Robin Reynolds-Haertle (PyCascades 2023) <https://www.youtube.com/watch?v=CeOtqlh0UuQ>_ (video presentation)

• Guides

  • :doc:how_to_use
  • :doc:how_to_develop
  • :doc:pyparsing_porting

• Reference

  • :doc:grammar
  • :doc:tree_construction
  • :doc:visitors
  • :doc:forest
  • :doc:classes
  • :doc:tools
  • Cheatsheet (PDF)_

• Discussion

  • Gitter_
  • Forum (Google Groups)_

.. _Examples: https://github.com/lark-parser/lark/tree/master/examples
.. _Third-party examples: https://github.com/ligurio/lark-grammars
.. _Online IDE: https://lark-parser.org/ide
.. _How to write a DSL: https://eshsoft.com/blog/write-dsl-in-python-with-lark
.. _Program Synthesis is Possible: https://www.cs.cornell.edu/~asampson/blog/minisynth.html
.. _Cheatsheet (PDF): _static/lark_cheatsheet.pdf
.. _Gitter: https://gitter.im/lark-parser/Lobby
.. _Forum (Google Groups): https://groups.google.com/forum/#!forum/lark-parser

Json Tutorial (`docs/json_tutorial.md`) Copy Section

JSON parser - Tutorial

Lark is a parser - a program that accepts a grammar and text, and produces a structured tree that represents that text.
In this tutorial we will write a JSON parser in Lark, and explore Lark's various features in the process.

It has 5 parts.

1. Writing the grammar
2. Creating the parser
3. Shaping the tree
4. Evaluating the tree
5. Optimizing

Knowledge assumed:

• Using Python
• A basic understanding of how to use regular expressions

Part 1 - The Grammar Copy Chapter

Lark accepts its grammars in a format called EBNF. It basically looks like this:

rule_name : list of rules and TERMINALS to match
          | another possible list of items
          | etc.

TERMINAL: "some text to match"

(a terminal is a string or a regular expression)

The parser will try to match each rule (left-part) by matching its items (right-part) sequentially, trying each alternative (In practice, the parser is predictive so we don't have to try every alternative).

How to structure those rules is beyond the scope of this tutorial, but often it's enough to follow one's intuition.

In the case of JSON, the structure is simple: A json document is either a list, or a dictionary, or a string/number/etc.

The dictionaries and lists are recursive, and contain other json documents (or "values").

Let's write this structure in EBNF form:

value: dict
         | list
         | STRING
         | NUMBER
         | "true" | "false" | "null"

    list : "[" [value ("," value)*] "]"

    dict : "{" [pair ("," pair)*] "}"
    pair : STRING ":" value

A quick explanation of the syntax:

• Parenthesis let us group rules together.
• rule* means any amount. That means, zero or more instances of that rule.
• [rule] means optional. That means zero or one instance of that rule.

Lark also supports the rule+ operator, meaning one or more instances. It also supports the rule? operator which is another way to say optional.

Of course, we still haven't defined "STRING" and "NUMBER". Luckily, both these literals are already defined in Lark's common library:

%import common.ESCAPED_STRING   -> STRING
    %import common.SIGNED_NUMBER    -> NUMBER

The arrow (->) renames the terminals. But that only adds obscurity in this case, so going forward we'll just use their original names.

We'll also take care of the white-space, which is part of the text, by simply matching and then throwing it away.

%import common.WS
    %ignore WS

We tell our parser to ignore whitespace. Otherwise, we'd have to fill our grammar with WS terminals.

By the way, if you're curious what these terminals signify, they are roughly equivalent to this:

NUMBER : /-?\d+(\.\d+)?([eE][+-]?\d+)?/
    STRING : /".*?(?<!\\)"/
    %ignore /[ \t\n\f\r]+/

Lark will accept this way of writing too, if you really want to complicate your life :)

You can find the original definitions in common.lark.
They don't strictly adhere to json.org - but our purpose here is to accept json, not validate it.

Notice that terminals are written in UPPER-CASE, while rules are written in lower-case.
I'll touch more on the differences between rules and terminals later.

Part 2 - Creating the Parser Copy Chapter

Once we have our grammar, creating the parser is very simple.

We simply instantiate Lark, and tell it to accept a "value":

from lark import Lark
json_parser = Lark(r"""
    value: dict
         | list
         | ESCAPED_STRING
         | SIGNED_NUMBER
         | "true" | "false" | "null"

    list : "[" [value ("," value)*] "]"

    dict : "{" [pair ("," pair)*] "}"
    pair : ESCAPED_STRING ":" value

    %import common.ESCAPED_STRING
    %import common.SIGNED_NUMBER
    %import common.WS
    %ignore WS

    """, start='value')

It's that simple! Let's test it out:

>>> text = '{"key": ["item0", "item1", 3.14]}'
>>> json_parser.parse(text)
Tree(value, [Tree(dict, [Tree(pair, [Token(STRING, "key"), Tree(value, [Tree(list, [Tree(value, [Token(STRING, "item0")]), Tree(value, [Token(STRING, "item1")]), Tree(value, [Token(NUMBER, 3.14)])])])])])])
>>> print( _.pretty() )
value
  dict
    pair
      "key"
      value
        list
          value	"item0"
          value	"item1"
          value	3.14

As promised, Lark automagically creates a tree that represents the parsed text.

But something is suspiciously missing from the tree. Where are the curly braces, the commas and all the other punctuation literals?

Lark automatically filters out literals from the tree, based on the following criteria:

• Filter out string literals without a name, or with a name that starts with an underscore.
• Keep regexps, even unnamed ones, unless their name starts with an underscore.

Unfortunately, this means that it will also filter out literals like "true" and "false", and we will lose that information. The next section, "Shaping the tree" deals with this issue, and others.

Part 3 - Shaping the Tree Copy Chapter

We now have a parser that can create a parse tree (or: AST), but the tree has some issues:

1. "true", "false" and "null" are filtered out (test it out yourself!)
2. Is has useless branches, like value, that clutter-up our view.

I'll present the solution, and then explain it:

?value: dict
          | list
          | string
          | SIGNED_NUMBER      -> number
          | "true"             -> true
          | "false"            -> false
          | "null"             -> null

    ...

    string : ESCAPED_STRING

1. Those little arrows signify aliases. An alias is a name for a specific part of the rule. In this case, we will name the true/false/null matches, and this way we won't lose the information. We also alias SIGNED_NUMBER to mark it for later processing.

2. The question-mark prefixing value ("?value") tells the tree-builder to inline this branch if it has only one member. In this case, value will always have only one member, and will always be inlined.

3. We turned the ESCAPED_STRING terminal into a rule. This way it will appear in the tree as a branch. This is equivalent to aliasing (like we did for the number), but now string can also be used elsewhere in the grammar (namely, in the pair rule).

Here is the new grammar:

from lark import Lark
json_parser = Lark(r"""
    ?value: dict
          | list
          | string
          | SIGNED_NUMBER      -> number
          | "true"             -> true
          | "false"            -> false
          | "null"             -> null

    list : "[" [value ("," value)*] "]"

    dict : "{" [pair ("," pair)*] "}"
    pair : string ":" value

    string : ESCAPED_STRING

    %import common.ESCAPED_STRING
    %import common.SIGNED_NUMBER
    %import common.WS
    %ignore WS

    """, start='value')

And let's test it out:

>>> text = '{"key": ["item0", "item1", 3.14, true]}'
>>> print( json_parser.parse(text).pretty() )
dict
  pair
    string	"key"
    list
      string	"item0"
      string	"item1"
      number	3.14
      true

Ah! That is much much nicer.

Part 4 - Evaluating the tree Copy Chapter

It's nice to have a tree, but what we really want is a JSON object.

The way to do it is to evaluate the tree, using a Transformer.

A transformer is a class with methods corresponding to branch names. For each branch, the appropriate method will be called with the children of the branch as its argument, and its return value will replace the branch in the tree.

So let's write a partial transformer, that handles lists and dictionaries:

from lark import Transformer

class MyTransformer(Transformer):
    def list(self, items):
        return list(items)
    def pair(self, key_value):
        k, v = key_value
        return k, v
    def dict(self, items):
        return dict(items)

And when we run it, we get this:

>>> tree = json_parser.parse(text)
>>> MyTransformer().transform(tree)
{Tree(string, [Token(ANONRE_1, "key")]): [Tree(string, [Token(ANONRE_1, "item0")]), Tree(string, [Token(ANONRE_1, "item1")]), Tree(number, [Token(ANONRE_0, 3.14)]), Tree(true, [])]}

This is pretty close. Let's write a full transformer that can handle the terminals too.

Also, our definitions of list and dict are a bit verbose. We can do better:

from lark import Transformer

class TreeToJson(Transformer):
    def string(self, s):
        (s,) = s
        return s[1:-1]
    def number(self, n):
        (n,) = n
        return float(n)

    list = list
    pair = tuple
    dict = dict

    null = lambda self, _: None
    true = lambda self, _: True
    false = lambda self, _: False

And when we run it:

>>> tree = json_parser.parse(text)
>>> TreeToJson().transform(tree)
{u'key': [u'item0', u'item1', 3.14, True]}

Magic!

Part 5 - Optimizing Copy Chapter

Step 1 - Benchmark Copy Section

By now, we have a fully working JSON parser, that can accept a string of JSON, and return its Pythonic representation.

But how fast is it?

Now, of course there are JSON libraries for Python written in C, and we can never compete with them. But since this is applicable to any parser you would write in Lark, let's see how far we can take this.

The first step for optimizing is to have a benchmark. For this benchmark I'm going to take data from json-generator.com/. I took their default suggestion and changed it to 5000 objects. The result is a 6.6MB sparse JSON file.

Our first program is going to be just a concatenation of everything we've done so far:

import sys
from lark import Lark, Transformer

json_grammar = r"""
    ?value: dict
          | list
          | string
          | SIGNED_NUMBER      -> number
          | "true"             -> true
          | "false"            -> false
          | "null"             -> null

    list : "[" [value ("," value)*] "]"

    dict : "{" [pair ("," pair)*] "}"
    pair : string ":" value

    string : ESCAPED_STRING

    %import common.ESCAPED_STRING
    %import common.SIGNED_NUMBER
    %import common.WS
    %ignore WS
    """

class TreeToJson(Transformer):
    def string(self, s):
        (s,) = s
        return s[1:-1]
    def number(self, n):
        (n,) = n
        return float(n)

    list = list
    pair = tuple
    dict = dict

    null = lambda self, _: None
    true = lambda self, _: True
    false = lambda self, _: False

json_parser = Lark(json_grammar, start='value', lexer='basic')

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        tree = json_parser.parse(f.read())
        print(TreeToJson().transform(tree))

We run it and get this:

$ time python tutorial_json.py json_data > /dev/null

real	0m36.257s
user	0m34.735s
sys         0m1.361s

That's unsatisfactory time for a 6MB file. Maybe if we were parsing configuration or a small DSL, but we're trying to handle large amount of data here.

Well, turns out there's quite a bit we can do about it!

Step 2 - LALR(1) Copy Section

So far we've been using the Earley algorithm, which is the default in Lark. Earley is powerful but slow. But it just so happens that our grammar is LR-compatible, and specifically LALR(1) compatible.

So let's switch to LALR(1) and see what happens:

json_parser = Lark(json_grammar, start='value', parser='lalr')

$ time python tutorial_json.py json_data > /dev/null

real        0m7.554s
user        0m7.352s
sys         0m0.148s

Ah, that's much better. The resulting JSON is of course exactly the same. You can run it for yourself and see.

It's important to note that not all grammars are LR-compatible, and so you can't always switch to LALR(1). But there's no harm in trying! If Lark lets you build the grammar, it means you're good to go.

Step 3 - Tree-less LALR(1) Copy Section

So far, we've built a full parse tree for our JSON, and then transformed it. It's a convenient method, but it's not the most efficient in terms of speed and memory. Luckily, Lark lets us avoid building the tree when parsing with LALR(1).

Here's the way to do it:

json_parser = Lark(json_grammar, start='value', parser='lalr', transformer=TreeToJson())

if __name__ == '__main__':
    with open(sys.argv[1]) as f:
        print( json_parser.parse(f.read()) )

We've used the transformer we've already written, but this time we plug it straight into the parser. Now it can avoid building the parse tree, and just send the data straight into our transformer. The parse() method now returns the transformed JSON, instead of a tree.

Let's benchmark it:

real	0m4.866s
user	0m4.722s
sys 	0m0.121s

That's a measurable improvement! Also, this way is more memory efficient. Check out the benchmark table at the end to see just how much.

As a general practice, it's recommended to work with parse trees, and only skip the tree-builder when your transformer is already working.

Step 4 - PyPy Copy Section

PyPy is a JIT engine for running Python, and it's designed to be a drop-in replacement.

Lark is written purely in Python, which makes it very suitable for PyPy.

Let's get some free performance:

$ time pypy tutorial_json.py json_data > /dev/null

real	0m1.397s
user	0m1.296s
sys 	0m0.083s

PyPy is awesome!

Conclusion Copy Section

We've brought the run-time down from 36 seconds to 1.1 seconds, in a series of small and simple steps.

Now let's compare the benchmarks in a nicely organized table.

I measured memory consumption using a little script called memusg

I added a few other parsers for comparison. PyParsing and funcparselib fair pretty well in their memory usage (they don't build a tree), but they can't compete with the run-time speed of LALR(1).

These benchmarks are for Lark's alpha version. I already have several optimizations planned that will significantly improve run-time speed.

Once again, shout-out to PyPy for being so effective.

Afterword Copy Chapter

This is the end of the tutorial. I hoped you liked it and learned a little about Lark.

To see what else you can do with Lark, check out the examples.

Read the documentation here: https://lark-parser.readthedocs.io/en/latest/

Parsers (`docs/parsers.md`) Copy Section

Parsers

Lark implements the following parsing algorithms: Earley, LALR(1), and CYK

Earley Copy Chapter

An Earley Parser is a chart parser capable of parsing any context-free grammar at O(n^3), and O(n^2) when the grammar is unambiguous. It can parse most LR grammars at O(n). Most programming languages are LR, and can be parsed at a linear time.

Lark's Earley implementation runs on top of a skipping chart parser, which allows it to use regular expressions, instead of matching characters one-by-one. This is a huge improvement to Earley that is unique to Lark. This feature is used by default, but can also be requested explicitly using lexer='dynamic'.

It's possible to bypass the dynamic lexing, and use the regular Earley parser with a basic lexer, that tokenizes as an independent first step. Doing so will provide a speed benefit, but will tokenize without using Earley's ambiguity-resolution ability. So choose this only if you know why! Activate with lexer='basic'

SPPF & Ambiguity resolution

Lark implements the Shared Packed Parse Forest data-structure for the Earley parser, in order to reduce the space and computation required to handle ambiguous grammars.

You can read more about SPPF here

As a result, Lark can efficiently parse and store every ambiguity in the grammar, when using Earley.

Lark provides the following options to combat ambiguity:

1. Lark will choose the best derivation for you (default). Users can choose between different disambiguation strategies, and can prioritize (or demote) individual rules over others, using the rule-priority syntax.

2. Users may choose to receive the set of all possible parse-trees (using ambiguity='explicit'), and choose the best derivation themselves. While simple and flexible, it comes at the cost of space and performance, and so it isn't recommended for highly ambiguous grammars, or very long inputs.

3. As an advanced feature, users may use specialized visitors to iterate the SPPF themselves. There is also a 3rd party utility for iterating over the SPPF.

lexer="dynamic_complete"

Earley's "dynamic" lexer uses regular expressions in order to tokenize the text. It tries every possible combination of terminals, but it matches each terminal exactly once, returning the longest possible match.

That means, for example, that when lexer="dynamic" (which is the default), the terminal /a+/, when given the text "aa", will return one result, aa, even though a would also be correct.

This behavior was chosen because it is much faster, and it is usually what you would expect.

Setting lexer="dynamic_complete" instructs the lexer to consider every possible regexp match. This ensures that the parser will consider and resolve every ambiguity, even inside the terminals themselves. This lexer provides the same capabilities as scannerless Earley, but with different performance tradeoffs.

Warning: This lexer can be much slower, especially for open-ended terminals such as /.*/

LALR(1) Copy Chapter

LALR(1) is a very efficient, true-and-tested parsing algorithm. It's incredibly fast and requires very little memory. It can parse most programming languages (For example: Python and Java).

LALR(1) stands for:

• Left-to-right parsing order

• Rightmost derivation, bottom-up

• Lookahead of 1 token

Lark comes with an efficient implementation that outperforms every other parsing library for Python (including PLY)

Lark extends the traditional YACC-based architecture with a contextual lexer, which processes feedback from the parser, making the LALR(1) algorithm stronger than ever.

The contextual lexer communicates with the parser, and uses the parser's lookahead prediction to narrow its choice of terminals. So at each point, the lexer only matches the subgroup of terminals that are legal at that parser state, instead of all of the terminals. It’s surprisingly effective at resolving common terminal collisions, and allows one to parse languages that LALR(1) was previously incapable of parsing.

(If you're familiar with YACC, you can think of it as automatic lexer-states)

This is an improvement to LALR(1) that is unique to Lark.

Grammar constraints in LALR(1) Copy Section

Due to having only a lookahead of one token, LALR is limited in its ability to choose between rules, when they both match the input.

Tips for writing a conforming grammar:

• Try to avoid writing different rules that can match the same sequence of characters.

• For the best performance, prefer left-recursion over right-recursion.

• Consider setting terminal priority only as a last resort.

For a better understanding of these constraints, it's recommended to learn how a SLR parser works. SLR is very similar to LALR but much simpler.

CYK Parser Copy Chapter

A CYK parser can parse any context-free grammar at O(n^3*|G|).

Its too slow to be practical for simple grammars, but it offers good performance for highly ambiguous grammars.

Philosophy (`docs/philosophy.md`) Copy Section

Philosophy

Parsers are innately complicated and confusing. They're difficult to understand, difficult to write, and difficult to use. Even experts on the subject can become baffled by the nuances of these complicated state-machines.

Lark's mission is to make the process of writing them as simple and abstract as possible, by following these design principles:

Design Principles Copy Chapter

1. Readability matters

2. Keep the grammar clean and simple

3. Don't force the user to decide on things that the parser can figure out on its own

4. Usability is more important than performance

5. Performance is still very important

6. Follow the Zen of Python, whenever possible and applicable

In accordance with these principles, I arrived at the following design choices:

Design Choices Copy Chapter

1. Separation of code and grammar Copy Section

Grammars are the de-facto reference for your language, and for the structure of your parse-tree. For any non-trivial language, the conflation of code and grammar always turns out convoluted and difficult to read.

The grammars in Lark are EBNF-inspired, so they are especially easy to read & work with.

2. Always build a parse-tree (unless told not to) Copy Section

Trees are always simpler to work with than state-machines.

1. Trees allow you to see the "state-machine" visually

2. Trees allow your computation to be aware of previous and future states

3. Trees allow you to process the parse in steps, instead of forcing you to do it all at once.

And anyway, every parse-tree can be replayed as a state-machine, so there is no loss of information.

See this answer in more detail here.

To improve performance, you can skip building the tree for LALR(1), by providing Lark with a transformer (see the JSON example).

3. Earley is the default Copy Section

The Earley algorithm can accept any context-free grammar you throw at it (i.e. any grammar you can write in EBNF, it can parse). That makes it extremely friendly to beginners, who are not aware of the strange and arbitrary restrictions that LALR(1) places on its grammars.

As the users grow to understand the structure of their grammar, the scope of their target language, and their performance requirements, they may choose to switch over to LALR(1) to gain a huge performance boost, possibly at the cost of some language features.

Both Earley and LALR(1) can use the same grammar, as long as all constraints are satisfied.

In short, "Premature optimization is the root of all evil."

Other design features Copy Section

• Automatically resolve terminal collisions whenever possible

• Automatically keep track of line & column numbers

Pyparsing Porting (`docs/pyparsing_porting.md`) Copy Section

Porting from pyparsing to Lark

This guide is for projects that already have a working parser built with
pyparsing and want to move it to
Lark incrementally.

The main change is architectural: pyparsing grammars are Python object graphs,
while Lark grammars are usually written as EBNF text. In return, Lark can build
a parse tree automatically, run different parser algorithms from the same
grammar, and keep grammar and tree-processing code separate.

One common early porting surprise is whitespace: pyparsing skips whitespace by
default, but Lark only ignores whitespace that the grammar explicitly declares
with %ignore.

Porting checklist Copy Chapter

1. Choose one representative input from the pyparsing parser's tests.
2. Write the Lark grammar for only that input shape, not the whole language.
3. Parse to a tree first, before adding a transformer.
4. Move parse actions to a Transformer once the tree shape is stable.
5. Add the rest of the syntax gradually, keeping each pyparsing test paired
   with an equivalent Lark test.

pyparsing concepts and Lark equivalents Copy Chapter

Unlike pyparsing's MatchFirst, Lark alternatives are not inherently ordered in
the same way: LALR resolves choices deterministically from the parse table, and
Earley uses ambiguity handling and priorities when needed. If a pyparsing grammar
relies on ordering to choose between overlapping expressions, make that choice
explicit with priorities or more specific rules when porting.

Minimal example Copy Chapter

A small pyparsing parser might look like this:

from pyparsing import Group, OneOrMore, Word, alphas, nums

assignment = Group(Word(alphas) + "=" + Word(nums))
parser = OneOrMore(assignment)

print(parser.parse_string("width=10 height=20").as_list())

A direct Lark version separates the grammar from the processing step:

from lark import Lark, Transformer, v_args

parser = Lark(r"""
    start: assignment+

    assignment: NAME "=" INT

    %import common.CNAME -> NAME
    %import common.INT
    %import common.WS
    %ignore WS
""", parser="lalr")

@v_args(inline=True)
class ToDict(Transformer):
    def assignment(self, name, value):
        return str(name), int(value)

    def start(self, *items):
        return dict(items)

print(ToDict().transform(parser.parse("width=10 height=20")))

Result:

{'width': 10, 'height': 20}

Port parse actions to transformers Copy Chapter

pyparsing parse actions run while parsing. Lark usually keeps parsing and tree
processing separate:

@v_args(inline=True)
class BuildConfig(Transformer):
    def assignment(self, name, value):
        return str(name), int(value)

This has two useful effects while porting:

• grammar bugs are easier to see, because you can inspect parser.parse(text).pretty();
• transformation bugs are isolated from parsing bugs.

If your pyparsing parse action validates or normalizes tokens, move that logic to
the matching transformer method. If it only discards punctuation, prefer handling
that in the grammar with literals or underscore-prefixed helper rules.

Replace `ignore()` with `%ignore` Copy Chapter

pyparsing commonly uses parser.ignore(...) for comments or whitespace. In Lark,
ignored text is part of the grammar:

parser = Lark(r"""
    start: assignment+
    assignment: NAME "=" INT

    COMMENT: /#[^\n]*/

    %import common.CNAME -> NAME
    %import common.INT
    %import common.WS
    %ignore WS
    %ignore COMMENT
""", parser="lalr")

Use %ignore for syntax that can appear between tokens. If comments are
meaningful to your application, do not ignore them; keep them as terminals and
process them with a transformer or lexer callback.

Port delimited lists Copy Chapter

For a pyparsing list such as:

from pyparsing import delimitedList, Word, alphas

names = delimitedList(Word(alphas))

Use a recursive or repeated Lark rule:

names: NAME ("," NAME)*
%import common.CNAME -> NAME
%ignore " "

If the comma should never appear in the final tree, keep it as a literal string.
If it carries meaning, promote it to a named terminal.

Port operator precedence Copy Chapter

pyparsing's infixNotation encodes precedence in a Python list. In Lark, spell
out precedence as layers of grammar rules:

?expr: sum
?sum: product
    | sum "+" product   -> add
    | sum "-" product   -> sub
?product: atom
    | product "*" atom  -> mul
    | product "/" atom  -> div
?atom: NUMBER           -> number
     | "(" expr ")"

%import common.NUMBER
%import common.WS
%ignore WS

This keeps the tree shape explicit: the aliases after -> become transformer
method names such as add, sub, and number.

Use `scan()` when pyparsing searched inside larger text Copy Chapter

Some pyparsing parsers are used with scan_string() to find matching snippets
inside text that is not fully parsed. For that use case, use Lark.scan():

from lark import Lark

parser = Lark(r"""
    start: assignment
    assignment: NAME "=" INT

    %import common.CNAME -> NAME
    %import common.INT
    %ignore " "
""", parser="lalr")

for match in parser.scan("before width=10, between height=20, after"):
    print(match.value)

scan() walks the input from left to right and yields matches for grammar
snippets, while skipping text that does not match. This is usually the closest
replacement for pyparsing code that intentionally searches inside log lines,
templates, prose, or mixed-format files.

Suggested migration workflow Copy Chapter

Keep the old and new parsers side by side until the Lark grammar covers the same
inputs:

def test_lark_matches_pyparsing_fixture():
    text = "width=10 height=20"
    assert parse_with_lark(text) == parse_with_pyparsing(text)

Good first fixtures are:

• the smallest valid input;
• a valid input with whitespace and comments;
• the most common real-world input;
• one invalid input that should produce a useful error.

Once these match, continue porting feature by feature. Avoid translating every
pyparsing helper mechanically; a shorter Lark grammar with a clear transformer is
often easier to maintain than a one-to-one rewrite.

Recipes (`docs/recipes.md`) Copy Section

Recipes

A collection of recipes to use Lark and its various features

Use a transformer to parse integer tokens Copy Chapter

Transformers are the common interface for processing matched rules and tokens.

They can be used during parsing for better performance.

from lark import Lark, Transformer

class T(Transformer):
    def INT(self, tok):
        "Convert the value of `tok` from string to int, while maintaining line number & column."
        return tok.update(value=int(tok))

parser = Lark("""
start: INT*
%import common.INT
%ignore " "
""", parser="lalr", transformer=T())

print(parser.parse('3 14 159'))

Prints out:

Tree(start, [Token(INT, 3), Token(INT, 14), Token(INT, 159)])

Collect all comments with lexer_callbacks Copy Chapter

lexer_callbacks can be used to interface with the lexer as it generates tokens.

It accepts a dictionary of the form

{TOKEN_TYPE: callback}

Where callback is of type f(Token) -> Token

It only works with the basic and contextual lexers.

This has the same effect of using a transformer, but can also process ignored tokens.

from lark import Lark

comments = []

parser = Lark("""
    start: INT*

    COMMENT: /#.*/

    %import common (INT, WS)
    %ignore COMMENT
    %ignore WS
""", parser="lalr", lexer_callbacks={'COMMENT': comments.append})

parser.parse("""
1 2 3  # hello
# world
4 5 6
""")

print(comments)

Prints out:

[Token(COMMENT, '# hello'), Token(COMMENT, '# world')]

Note: We don't have to return a token, because comments are ignored

CollapseAmbiguities Copy Chapter

Parsing ambiguous texts with earley and ambiguity='explicit' produces a single tree with _ambig nodes to mark where the ambiguity occurred.

However, it's sometimes more convenient instead to work with a list of all possible unambiguous trees.

Lark provides a utility transformer for that purpose:

from lark import Lark, Tree, Transformer
from lark.visitors import CollapseAmbiguities

grammar = """
    !start: x y

    !x: "a" "b"
      | "ab"
      | "abc"

    !y: "c" "d"
      | "cd"
      | "d"

"""
parser = Lark(grammar, ambiguity='explicit')

t = parser.parse('abcd')
for x in CollapseAmbiguities().transform(t):
    print(x.pretty())

This prints out:

start
x
    a
    b
y
    c
    d

start
x     ab
y     cd

start
x     abc
y     d

While convenient, this should be used carefully, as highly ambiguous trees will soon create an exponential explosion of such unambiguous derivations.

Keeping track of parents when visiting Copy Chapter

The following visitor assigns a parent attribute for every node in the tree.

If your tree nodes aren't unique (if there is a shared Tree instance), the assert will fail.

class Parent(Visitor):
    def __default__(self, tree):
        for subtree in tree.children:
            if isinstance(subtree, Tree):
                assert not hasattr(subtree, 'parent')
                subtree.parent = proxy(tree)

Unwinding VisitError after a transformer/visitor exception Copy Chapter

Errors that happen inside visitors and transformers get wrapped inside a VisitError exception.

This can often be inconvenient, if you wish the actual error to propagate upwards, or if you want to catch it.

But, it's easy to unwrap it at the point of calling the transformer, by catching it and raising the VisitError.orig_exc attribute.

For example:

from lark import Lark, Transformer
from lark.visitors import VisitError

tree = Lark('start: "a"').parse('a')

class T(Transformer):
    def start(self, x):
        raise KeyError("Original Exception")

t = T()
try:
    print( t.transform(tree))
except VisitError as e:
    raise e.orig_exc

Adding a Progress Bar to Parsing with tqdm Copy Chapter

Parsing large files can take a long time, even with the parser='lalr' option. To make this process more user-friendly, it's useful to add a progress bar. One way to achieve this is to use the InteractiveParser to display each token as it is processed. In this example, we use tqdm, but it should be easy to adapt to other kinds of progress bars.

from tqdm import tqdm

def parse_with_progress(parser: Lark, text: str, start=None):
    last = 0
    progress = tqdm(total=len(text))
    pi = parser.parse_interactive(text, start=start)
    for token in pi.iter_parse():
        if token.end_pos is not None:
            progress.update(token.end_pos - last)
            last = token.end_pos
    return pi.resume_parse()    # Finish up and get the result

Keep in mind that this implementation relies on the InteractiveParser and, therefore, only works with the LALR(1) parser, and not Earley.

Parsing a Language with Significant Indentation Copy Chapter

If your grammar needs to support significant indentation (e.g. Python, YAML), you will need to use
the Indenter class. Take a look at the indented tree example as well as the
Python grammar for inspiration.

Extract grammar matches from arbitrary text with `Lark.scan` Copy Chapter

Sometimes parsing the entire document isn't feasible. The scan method
lets you find every instance of a grammar that appears inside arbitrary text.
Things like dates in prose, JSON payloads in log lines, or configuration placeholders
in templates.

Lark.scan() walks the input from left to right, yielding a match per snippet that matches your grammar.
Text that doesn't match is silently skipped.

Example:

from lark import Lark

parser = Lark(r"""
    date: MONTH DAY "," YEAR
    MONTH: "January"|"February"|"March"|"April"|"May"|"June"
         |"July"|"August"|"September"|"October"|"November"|"December"
    DAY: /\d+/
    YEAR: /\d+/
    %ignore /\s+/
""", parser="lalr", start="date")

text = "Python 0.9.0 was released on February 20, 1991, and 1.0 on January 26, 1994."

for match in parser.scan(text):
    print("Found:", match)

Prints:

Found: ScanMatch(range=(29, 46), value=Tree(Token('RULE', 'date'), [Token('MONTH', 'February'), Token('DAY', '20'), Token('YEAR', '1991')]))
Found: ScanMatch(range=(59, 75), value=Tree(Token('RULE', 'date'), [Token('MONTH', 'January'), Token('DAY', '26'), Token('YEAR', '1994')]))

For an example scanning text for JSON, see examples/advanced/scan_json.py.

Requirements (`docs/requirements.txt`) Copy Section

https://docs.readthedocs.io/en/stable/guides/specifying-dependencies.html#specifying-a-requirements-file

pillow
recommonmark
requests==2.28.1
sphinx-gallery
sphinx_markdown_tables
sphinx_rtd_theme>=1.2

Tools (`docs/tools.md`) Copy Section

Tools (Stand-alone, Nearley)

Stand-alone parser Copy Chapter

Lark can generate a stand-alone LALR(1) parser from a grammar.

The resulting module provides the same interface as Lark, but with a fixed grammar, and reduced functionality.

Run using:

python -m lark.tools.standalone

For a play-by-play, read the tutorial

Importing grammars from Nearley.js Copy Chapter

Lark comes with a tool to convert grammars from Nearley, a popular Earley library for Javascript. It uses Js2Py to convert and run the Javascript postprocessing code segments.

Requirements

1. Install Lark with the nearley component:

pip install lark[nearley]

2. Acquire a copy of the Nearley codebase. This can be done using:

git clone https://github.com/Hardmath123/nearley

Usage

The tool can be run using:

python -m lark.tools.nearley <grammar.ne> <start_rule> <path_to_nearley_repo>

Here's an example of how to import nearley's calculator example into Lark:

git clone https://github.com/Hardmath123/nearley
python -m lark.tools.nearley nearley/examples/calculator/arithmetic.ne main ./nearley > ncalc.py

You can use the output as a regular python module:

>>> import ncalc
>>> ncalc.parse('sin(pi/4) ^ e')
0.38981434460254655

The Nearley converter also supports an experimental converter for newer JavaScript (ES6+), using the --es6 flag:

git clone https://github.com/Hardmath123/nearley
python -m lark.tools.nearley nearley/examples/calculator/arithmetic.ne main nearley --es6 > ncalc.py

Notes

• Lark currently cannot import templates from Nearley

• Lark currently cannot export grammars to Nearley

These might get added in the future, if enough users ask for them.

Tree Construction (`docs/tree_construction.md`) Copy Section

Tree Construction Reference

Lark builds a tree automatically based on the structure of the grammar, where each rule that is matched becomes a branch (node) in the tree, and its children are its matches, in the order of matching.

For example, the rule node: child1 child2 will create a tree node with two children. If it is matched as part of another rule (i.e. if it isn't the root), the new rule's tree node will become its parent.

Using item+ or item* will result in a list of items, equivalent to writing item item item ...

Using item? will return the item if it matched, or nothing.

If maybe_placeholders=True (the default), then using [item] will return the item if it matched, or the value None, if it didn't.

If maybe_placeholders=False, then [] behaves like ()?.

Terminals Copy Chapter

Terminals are always values in the tree, never branches.

Lark filters out certain types of terminals by default, considering them punctuation:

• Terminals that won't appear in the tree are:

  • Unnamed literals (like "keyword" or "+")
  • Terminals whose name starts with an underscore (like _DIGIT)

• Terminals that will appear in the tree are:

  • Unnamed regular expressions (like /[0-9]/)
  • Named terminals whose name starts with a letter (like DIGIT)

Note: Terminals composed of literals and other terminals always include the entire match without filtering any part.

Example:

start:  PNAME pname

PNAME:  "(" NAME ")"
pname:  "(" NAME ")"

NAME:   /\w+/
%ignore /\s+/

Lark will parse "(Hello) (World)" as:

start
    (Hello)
    pname World

Rules prefixed with ! will retain all their literals regardless.

Example:

expr: "(" expr ")"
        | NAME+

    NAME: /\w+/

    %ignore " "

Lark will parse "((hello world))" as:

expr
    expr
        expr
            "hello"
            "world"

The brackets do not appear in the tree by design. The words appear because they are matched by a named terminal.

Shaping the tree Copy Chapter

Users can alter the automatic construction of the tree using a collection of grammar features.

Inlining rules with `_` Copy Section

Rules whose name begins with an underscore will be inlined into their containing rule.

Example:

start: "(" _greet ")"
    _greet: /\w+/ /\w+/

Lark will parse "(hello world)" as:

start
    "hello"
    "world"

Conditionally inlining rules with `?` Copy Section

Rules that receive a question mark (?) at the beginning of their definition, will be inlined if they have a single child, after filtering.

Example:

start: greet greet
    ?greet: "(" /\w+/ ")"
          | /\w+/ /\w+/

Lark will parse "hello world (planet)" as:

start
    greet
        "hello"
        "world"
    "planet"

Pinning rule terminals with `!` Copy Section

Rules that begin with an exclamation mark will keep all their terminals (they won't get filtered).

!expr: "(" expr ")"
         | NAME+
    NAME: /\w+/
    %ignore " "

Will parse "((hello world))" as:

expr
  (
  expr
    (
    expr
      hello
      world
    )
  )

Using the ! prefix is usually a "code smell", and may point to a flaw in your grammar design.

Aliasing rules Copy Section

Aliases - options in a rule can receive an alias. It will be then used as the branch name for the option, instead of the rule name.

Example:

start: greet greet
    greet: "hello"
         | "world" -> planet

Lark will parse "hello world" as:

start
    greet
    planet

Visitors (`docs/visitors.rst`) Copy Section

Transformers & Visitors

Transformers & Visitors provide a convenient interface to process the
parse-trees that Lark returns.

They are used by inheriting from one of the classes described here,
and implementing methods corresponding to the rule you wish to process. Each
method accepts the children as an argument. That can be modified using the
v_args decorator, which allows one to inline the arguments (akin to *args),
or add the tree meta property as an argument.

See: visitors.py_

.. _visitors.py: https://github.com/lark-parser/lark/blob/master/lark/visitors.py

Visitor Copy Chapter

Visitors visit each node of the tree and run the appropriate method on it according to the node's data.
They can traverse top-down or bottom-up, depending on the class and method.

There are two classes that implement the visitor interface:

• Visitor: Visit every node (without recursion)
• Visitor_Recursive: Visit every node using recursion. Slightly faster.

Example:
::

class IncreaseAllNumbers(Visitor):
        def number(self, tree):
            assert tree.data == "number"
            tree.children[0] += 1

    IncreaseAllNumbers().visit(parse_tree)

.. autoclass:: lark.visitors.Visitor
:members: visit, visit_topdown, default

.. autoclass:: lark.visitors.Visitor_Recursive
:members: visit, visit_topdown, default

Interpreter Copy Chapter

Example:
::

from lark.visitors import Interpreter


    class IncreaseSomeOfTheNumbers(Interpreter):
        def number(self, tree):
            tree.children[0] += 1

        def skip(self, tree):
            # skip this subtree. don't change any number node inside it.
            pass


    IncreaseSomeOfTheNumbers().visit(parse_tree)

.. autoclass:: lark.visitors.Interpreter
:members: visit, visit_children, default

Transformer Copy Chapter

.. autoclass:: lark.visitors.Transformer
:members: transform, default, default_token, mul

Example:
::

from lark import Tree, Transformer

    class EvalExpressions(Transformer):
        def expr(self, args):
                return eval(args[0])

    t = Tree('a', [Tree('expr', ['1+2'])])
    print(EvalExpressions().transform( t ))

    # Prints: Tree(a, [3])

Example:
::

class T(Transformer):
        INT = int
        NUMBER = float
        def NAME(self, name):
            return lookup_dict.get(name, name)

    T(visit_tokens=True).transform(tree)

.. autoclass:: lark.visitors.Transformer_NonRecursive

.. autoclass:: lark.visitors.Transformer_InPlace

.. autoclass:: lark.visitors.Transformer_InPlaceRecursive

Useful Utilities Copy Chapter

.. autofunction:: lark.visitors.v_args
.. autofunction:: lark.visitors.visit_children_decor
.. autofunction:: lark.visitors.merge_transformers

Discard Copy Chapter

Discard is the singleton instance of _DiscardType.

.. autoclass:: lark.visitors._DiscardType

VisitError Copy Chapter

.. autoclass:: lark.exceptions.VisitError

Ide App Files.Json (`docs/ide/app/files.json`) Copy Section

[
"app.py",
"examples.py",
"html5.py",
"core.py",
"ext.py",
"ignite.py",
"utils.py"
]

--- METRICS ---

• Files Extracted: 18
• Estimated Token Budget: ~20950 tokens
• Canonical Reference: https://codewiki.google/github.com/lark-parser/lark