How To Use Lark - Guide¶
This is the recommended process for working with Lark:
Collect or create input samples, that demonstrate key features or behaviors in the language you’re trying to parse.
Write a grammar. Try to aim for a structure that is intuitive, and in a way that imitates how you would explain your language to a fellow human.
Try your grammar in Lark against each input sample. Make sure the resulting parse-trees make sense.
Use Lark’s grammar features to shape the tree: Get rid of superfluous rules by inlining them, and use aliases when specific cases need clarification.
You can perform steps 1-4 repeatedly, gradually growing your grammar to include more sentences.
Create a transformer to evaluate the parse-tree into a structure you’ll be comfortable to work with. This may include evaluating literals, merging branches, or even converting the entire tree into your own set of AST classes.
Of course, some specific use-cases may deviate from this process. Feel free to suggest these cases, and I’ll add them to this page.
Browse the Examples to find a template that suits your purposes.
Read the tutorials to get a better understanding of how everything works. (links in the main page)
Use the Cheatsheet (PDF) for quick reference.
Use the reference pages for more in-depth explanations. (links in the main page)
Grammars may contain non-obvious bugs, usually caused by rules or terminals interfering with each other in subtle ways.
When trying to debug a misbehaving grammar, the following methodology is recommended:
Create a copy of the grammar, so you can change the parser/grammar without any worries
Find the minimal input that creates the error
Slowly remove rules from the grammar, while making sure the error still occurs.
Usually, by the time you get to a minimal grammar, the problem becomes clear.
But if it doesn’t, feel free to ask us on gitter, or even open an issue. Post a reproducing code, with the minimal grammar and input, and we’ll do our best to help.
A likely source of bugs occurs when two regexes in a grammar can match the same input. If both terminals have the same priority, most lexers would arbitrarily choose the first one that matches, which isn’t always the desired one. (a notable exception is the
dynamic_complete lexer, which always tries all variations. But its users pay for that with performance.)
These collisions can be hard to notice, and their effects can be difficult to debug, as they are subtle and sometimes hard to reproduce.
To help with these situations, Lark can utilize a new external library called
interegular. If it is installed, Lark uses it to check for collisions, and warn about any conflicts that it can find:
import logging from lark import Lark, logger logger.setLevel(logging.WARN) collision_grammar = ''' start: A | B A: /a+/ B: /[ab]+/ ''' p = Lark(collision_grammar, parser='lalr') # Output: # Collision between Terminals B and A. The lexer will choose between them arbitrarily # Example Collision: a
You can install interegular for Lark using
pip install 'lark[interegular]'.
Note 1: Interegular currently only runs when the lexer is
Note 2: Some advanced regex features, such as lookahead and lookbehind, may prevent interegular from detecting existing collisions.
By default Lark automatically resolves Shift/Reduce conflicts as Shift. It produces notifications as debug messages.
when users pass
debug=True, those notifications are written as warnings.
Either way, to get the messages printed you have to configure the
logger beforehand. For example:
import logging from lark import Lark, logger logger.setLevel(logging.DEBUG) collision_grammar = ''' start: as as as: a* a: "a" ''' p = Lark(collision_grammar, parser='lalr', debug=True) # Shift/Reduce conflict for terminal A: (resolving as shift) # * <as : > # Shift/Reduce conflict for terminal A: (resolving as shift) # * <as : __as_star_0>
Lark, by default, accepts grammars with unresolved Shift/Reduce collisions (which it always resolves to shift), and regex collisions.
Strict-mode allows users to validate that their grammars don’t contain these collisions.
When Lark is initialized with
strict=True, it raises an exception on any Shift/Reduce or regex collision.
interegular isn’t installed, an exception is thrown.
When using strict-mode, users will be expected to resolve their collisions manually:
To resolve Shift/Reduce collisions, adjust the priority weights of the rules involved, until there are no more collisions.
To resolve regex collisions, change the involved regexes so that they can no longer both match the same input (Lark provides an example).
Strict-mode only applies to LALR for now.
from lark import Lark collision_grammar = ''' start: as as as: a* a: "a" ''' p = Lark(collision_grammar, parser='lalr', strict=True) # Traceback (most recent call last): # ... # lark.exceptions.GrammarError: Shift/Reduce conflict for terminal A. [strict-mode]
Lark can generate a stand-alone LALR(1) parser from a grammar.
The resulting module provides the same interface as Lark, but with a fixed grammar, and reduced functionality.
python -m lark.tools.standalone
For a play-by-play, read the tutorial
Import Nearley.js grammars¶
See the tools page for more information.