.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "examples/advanced/custom_lexer.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_examples_advanced_custom_lexer.py>`
        to download the full example code

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_examples_advanced_custom_lexer.py:


Custom lexer
============

Demonstrates using a custom lexer to parse a non-textual stream of data

You can use a custom lexer to tokenize text when the lexers offered by Lark
are too slow, or not flexible enough.

You can also use it (as shown in this example) to tokenize streams of objects.

.. GENERATED FROM PYTHON SOURCE LINES 12-58

.. code-block:: default

    from lark import Lark, Transformer, v_args
    from lark.lexer import Lexer, Token

    class TypeLexer(Lexer):
        def __init__(self, lexer_conf):
            pass

        def lex(self, data):
            for obj in data:
                if isinstance(obj, int):
                    yield Token('INT', obj)
                elif isinstance(obj, (type(''), type(u''))):
                    yield Token('STR', obj)
                else:
                    raise TypeError(obj)

    parser = Lark("""
            start: data_item+
            data_item: STR INT*

            %declare STR INT
            """, parser='lalr', lexer=TypeLexer)


    class ParseToDict(Transformer):
        @v_args(inline=True)
        def data_item(self, name, *numbers):
            return name.value, [n.value for n in numbers]

        start = dict


    def test():
        data = ['alice', 1, 27, 3, 'bob', 4, 'carrie', 'dan', 8, 6]

        print(data)

        tree = parser.parse(data)
        res = ParseToDict().transform(tree)

        print('-->')
        print(res) # prints {'alice': [1, 27, 3], 'bob': [4], 'carrie': [], 'dan': [8, 6]}


    if __name__ == '__main__':
        test()


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 0.000 seconds)


.. _sphx_glr_download_examples_advanced_custom_lexer.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example




    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: custom_lexer.py <custom_lexer.py>`

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: custom_lexer.ipynb <custom_lexer.ipynb>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_