lexer Documentation

Base library for a lexer that can be used in Top-Down, Recursive Descent Parsers.

11,140 stars PHP PHP 1 file · ~5,710 tokens
RAW Doc

File: README.md

Doctrine Lexer

Base library for a lexer that can be used in Top-Down, Recursive Descent Parsers.

This lexer is used in Doctrine Annotations and in Doctrine ORM (DQL).

https://www.doctrine-project.org/projects/lexer.html

File: composer.json

{
"name": "doctrine/lexer",
"description": "PHP Doctrine Lexer parser library that can be used in Top-Down, Recursive Descent Parsers.",
"license": "MIT",
"type": "library",
"keywords": [
"php",
"parser",
"lexer",
"annotations",
"docblock"
],
"authors": [
{
"name": "Guilherme Blanco",
"email": "[email protected]"
},
{
"name": "Roman Borschel",
"email": "[email protected]"
},
{
"name": "Johannes Schmitt",
"email": "[email protected]"
}
],
"homepage": "https://www.doctrine-project.org/projects/lexer.html",
"require": {
"php": "^8.1"
},
"require-dev": {
"doctrine/coding-standard": "^14",
"phpstan/phpstan": "^2",
"phpunit/phpunit": "^10.5.58 || ^12.5.4"
},
"autoload": {
"psr-4": {
"Doctrine\Common\Lexer\": "src"
}
},
"autoload-dev": {
"psr-4": {
"Doctrine\Tests\Common\Lexer\": "tests"
}
},
"config": {
"allow-plugins": {
"composer/package-versions-deprecated": true,
"dealerdirect/phpcodesniffer-composer-installer": true
},
"sort-packages": true
}
}

File: SECURITY.md

Security

If you find a Security bug in Doctrine, please follow our
Security reporting guidelines.

Security vulnerabilities should be reported to [email protected]
and not posted on the GitHub issue tracker of the project.

File: UPGRADE.md

Note about upgrading: Doctrine uses static and runtime mechanisms to raise
awareness about deprecated code.

  • Use of @deprecated docblock that is detected by IDEs (like PHPStorm) or
    Static Analysis tools (like Psalm, phpstan)
  • Use of our low-overhead runtime deprecation API, details:
    https://github.com/doctrine/deprecations/

Upgrade to 3.0.0

Doctrine\Common\Lexer\Token no longer implements ArrayAccess.
Parameter type declarations have been added to
Doctrine\Common\Lexer\AbstractLexer and Doctrine\Common\Lexer\Token.
You should add both parameter type declarations and return type declarations to
your lexers, based on the @return phpdoc.

Upgrade to 2.0.0

AbstractLexer::glimpse() and AbstractLexer::peek() now return
instances of Doctrine\Common\Lexer\Token, which is an array-like class
Using it as an array is deprecated in favor of using properties of that class.
Using count() on it is deprecated with no replacement.

File: .doctrine-project.json

{
"active": true,
"name": "Lexer",
"slug": "lexer",
"docsSlug": "doctrine-lexer",
"versions": [
{
"name": "3.1",
"branchName": "3.1.x",
"slug": "latest",
"upcoming": true
},
{
"name": "3.0",
"branchName": "3.0.x",
"slug": "3.1",
"current": true
},
{
"name": "2.1",
"branchName": "2.1.x",
"slug": "2.1",
"maintained": false
},
{
"name": "2.0",
"branchName": "2.0.x",
"slug": "2.0",
"maintained": false
},
{
"name": "1.2",
"branchName": "1.2.x",
"slug": "1.2",
"maintained": false
},
{
"name": "1.1",
"branchName": "1.1",
"slug": "1.1",
"maintained": false
},
{
"name": "1.0",
"branchName": "1.0",
"slug": "1.0",
"maintained": false
}
]
}

File: docs/en/dql-parser.rst

DQL Lexer

Here is a more complicated example from the Doctrine ORM project.
The Doctrine\ORM\Query\Lexer implementation for DQL looks something
like the following:

.. code-block:: php

text
<?php

use Doctrine\Common\Lexer\AbstractLexer;

class Lexer extends AbstractLexer
{
    // All tokens that are not valid identifiers must be < 100
    public const T_NONE              = 1;
    public const T_INTEGER           = 2;
    public const T_STRING            = 3;
    public const T_INPUT_PARAMETER   = 4;
    public const T_FLOAT             = 5;
    public const T_CLOSE_PARENTHESIS = 6;
    public const T_OPEN_PARENTHESIS  = 7;
    public const T_COMMA             = 8;
    public const T_DIVIDE            = 9;
    public const T_DOT               = 10;
    public const T_EQUALS            = 11;
    public const T_GREATER_THAN      = 12;
    public const T_LOWER_THAN        = 13;
    public const T_MINUS             = 14;
    public const T_MULTIPLY          = 15;
    public const T_NEGATE            = 16;
    public const T_PLUS              = 17;
    public const T_OPEN_CURLY_BRACE  = 18;
    public const T_CLOSE_CURLY_BRACE = 19;

    // All tokens that are identifiers or keywords that could be considered as identifiers should be >= 100
    public const T_ALIASED_NAME         = 100;
    public const T_FULLY_QUALIFIED_NAME = 101;
    public const T_IDENTIFIER           = 102;

    // All keyword tokens should be >= 200
    public const T_ALL      = 200;
    public const T_AND      = 201;
    public const T_ANY      = 202;
    public const T_AS       = 203;
    public const T_ASC      = 204;
    public const T_AVG      = 205;
    public const T_BETWEEN  = 206;
    public const T_BOTH     = 207;
    public const T_BY       = 208;
    public const T_CASE     = 209;
    public const T_COALESCE = 210;
    public const T_COUNT    = 211;
    public const T_DELETE   = 212;
    public const T_DESC     = 213;
    public const T_DISTINCT = 214;
    public const T_ELSE     = 215;
    public const T_EMPTY    = 216;
    public const T_END      = 217;
    public const T_ESCAPE   = 218;
    public const T_EXISTS   = 219;
    public const T_FALSE    = 220;
    public const T_FROM     = 221;
    public const T_GROUP    = 222;
    public const T_HAVING   = 223;
    public const T_HIDDEN   = 224;
    public const T_IN       = 225;
    public const T_INDEX    = 226;
    public const T_INNER    = 227;
    public const T_INSTANCE = 228;
    public const T_IS       = 229;
    public const T_JOIN     = 230;
    public const T_LEADING  = 231;
    public const T_LEFT     = 232;
    public const T_LIKE     = 233;
    public const T_MAX      = 234;
    public const T_MEMBER   = 235;
    public const T_MIN      = 236;
    public const T_NEW      = 237;
    public const T_NOT      = 238;
    public const T_NULL     = 239;
    public const T_NULLIF   = 240;
    public const T_OF       = 241;
    public const T_OR       = 242;
    public const T_ORDER    = 243;
    public const T_OUTER    = 244;
    public const T_PARTIAL  = 245;
    public const T_SELECT   = 246;
    public const T_SET      = 247;
    public const T_SOME     = 248;
    public const T_SUM      = 249;
    public const T_THEN     = 250;
    public const T_TRAILING = 251;
    public const T_TRUE     = 252;
    public const T_UPDATE   = 253;
    public const T_WHEN     = 254;
    public const T_WHERE    = 255;
    public const T_WITH     = 256;

    /**
     * Creates a new query scanner object.
     *
     * @param string $input A query string.
     */
    public function __construct(string $input)
    {
        $this->setInput($input);
    }

    /**
     * {@inheritdoc}
     */
    protected function getCatchablePatterns(): array
    {
        return [
            '[a-z_][a-z0-9_]*\:[a-z_][a-z0-9_]*(?:\\\[a-z_][a-z0-9_]*)*', // aliased name
            '[a-z_\\\][a-z0-9_]*(?:\\\[a-z_][a-z0-9_]*)*', // identifier or qualified name
            '(?:[0-9]+(?:[\.][0-9]+)*)(?:e[+-]?[0-9]+)?', // numbers
            "'(?:[^']|'')*'", // quoted strings
            '\?[0-9]*|:[a-z_][a-z0-9_]*', // parameters
        ];
    }

    /**
     * {@inheritdoc}
     */
    protected function getNonCatchablePatterns(): array
    {
        return ['\s+', '(.)'];
    }

    /**
     * {@inheritdoc}
     */
    protected function getType(&$value): int
    {
        $type = self::T_NONE;

        switch (true) {
            // Recognize numeric values
            case (is_numeric($value)):
                if (strpos($value, '.') !== false || stripos($value, 'e') !== false) {
                    return self::T_FLOAT;
                }

                return self::T_INTEGER;

            // Recognize quoted strings
            case ($value[0] === "'"):
                $value = str_replace("''", "'", substr($value, 1, strlen($value) - 2));

                return self::T_STRING;

            // Recognize identifiers, aliased or qualified names
            case (ctype_alpha($value[0]) || $value[0] === '_' || $value[0] === '\\'):
                $name = 'Doctrine\ORM\Query\Lexer::T_' . strtoupper($value);

                if (defined($name)) {
                    $type = constant($name);

                    if ($type > 100) {
                        return $type;
                    }
                }

                if (strpos($value, ':') !== false) {
                    return self::T_ALIASED_NAME;
                }

                if (strpos($value, '\\') !== false) {
                    return self::T_FULLY_QUALIFIED_NAME;
                }

                return self::T_IDENTIFIER;

            // Recognize input parameters
            case ($value[0] === '?' || $value[0] === ':'):
                return self::T_INPUT_PARAMETER;

            // Recognize symbols
            case ($value === '.'):
                return self::T_DOT;
            case ($value === ','):
                return self::T_COMMA;
            case ($value === '('):
                return self::T_OPEN_PARENTHESIS;
            case ($value === ')'):
                return self::T_CLOSE_PARENTHESIS;
            case ($value === '='):
                return self::T_EQUALS;
            case ($value === '>'):
                return self::T_GREATER_THAN;
            case ($value === '<'):
                return self::T_LOWER_THAN;
            case ($value === '+'):
                return self::T_PLUS;
            case ($value === '-'):
                return self::T_MINUS;
            case ($value === '*'):
                return self::T_MULTIPLY;
            case ($value === '/'):
                return self::T_DIVIDE;
            case ($value === '!'):
                return self::T_NEGATE;
            case ($value === '{'):
                return self::T_OPEN_CURLY_BRACE;
            case ($value === '}'):
                return self::T_CLOSE_CURLY_BRACE;

            // Default
            default:
                // Do nothing
        }

        return $type;
    }
}

This is roughly what the DQL Parser looks like that uses the above
Lexer implementation:

.. note::

text
You can see the full implementation `here <https://github.com/doctrine/doctrine2/blob/1.3.x/lib/Doctrine/ORM/Query/Parser.php>`_.

.. code-block:: php

text
<?php

class Parser
{
    private $lexer;

    public function __construct(string $dql)
    {
        $this->lexer = new Lexer($dql);
    }

    // ...

    public function getAST(): AST\SelectStatement|AST\UpdateStatement|AST\DeleteStatement
    {
        // Parse & build AST
        $AST = $this->QueryLanguage();

        // ...

        return $AST;
    }

    public function QueryLanguage(): AST\SelectStatement|AST\UpdateStatement|AST\DeleteStatement
    {
        $this->lexer->moveNext();

        switch ($this->lexer->lookahead->type) {
            case Lexer::T_SELECT:
                $statement = $this->SelectStatement();
                break;
            case Lexer::T_UPDATE:
                $statement = $this->UpdateStatement();
                break;
            case Lexer::T_DELETE:
                $statement = $this->DeleteStatement();
                break;
            default:
                $this->syntaxError('SELECT, UPDATE or DELETE');
        }

        // Check for end of string
        if ($this->lexer->lookahead !== null) {
            $this->syntaxError('end of string');
        }

        return $statement;
    }

    // ...
}

Now the AST is used to transform the DQL query in to portable SQL for whatever relational
database you are using!

.. code-block:: php

text
<?php

$parser = new Parser('SELECT u FROM User u');
$AST = $parser->getAST(); // returns \Doctrine\ORM\Query\AST\SelectStatement

What is an AST?

AST stands for Abstract syntax tree <http://en.wikipedia.org/wiki/Abstract_syntax_tree>_.
In computer science, an abstract syntax tree (AST), or just syntax tree, is a
tree representation of the abstract syntactic structure of source code written
in a programming language. Each node of the tree denotes a construct occurring in
the source code.

File: docs/en/index.rst

Introduction

Doctrine Lexer is a library that can be used in Top-Down, Recursive
Descent Parsers. This lexer is used in Doctrine Annotations and in
Doctrine ORM (DQL).

To write your own parser you just need to extend Doctrine\Common\Lexer\AbstractLexer
and implement the following three abstract methods.

.. code-block:: php

text
/**
 * Lexical catchable patterns.
 *
 * @return string[]
 */
abstract protected function getCatchablePatterns();

/**
 * Lexical non-catchable patterns.
 *
 * @return string[]
 */
abstract protected function getNonCatchablePatterns();

/** Retrieve token type. Also processes the token value if necessary. */
abstract protected function getType(string &$value): int;

These methods define the lexical <http://en.wikipedia.org/wiki/Lexical_analysis>_
catchable and non-catchable patterns and a method for returning the
type of a token and filtering the value if necessary.

The Lexer is responsible for giving you an API to walk across a
string one character at a time and analyze the type of each character, value and position of
each token in the string. The low level API of the lexer is pretty simple:

  • setInput($input) - Sets the input data to be tokenized. The Lexer is immediately reset and the new input tokenized.
  • reset() - Resets the lexer.
  • resetPeek() - Resets the peek pointer to 0.
  • resetPosition($position = 0) - Resets the lexer position on the input to the given position.
  • isNextToken($token) - Checks whether a given token matches the current lookahead.
  • isNextTokenAny(array $tokens) - Checks whether any of the given tokens matches the current lookahead.
  • moveNext() - Moves to the next token in the input string.
  • skipUntil($type) - Tells the lexer to skip input tokens until it sees a token with the given value.
  • isA($value, $token) - Checks if given value is identical to the given token.
  • peek() - Moves the lookahead token forward.
  • glimpse() - Peeks at the next token, returns it and immediately resets the peek.

File: docs/en/sidebar.rst

.. toctree::
:depth: 3

text
index
simple-parser-example
dql-parser

File: docs/en/simple-parser-example.rst

Simple Parser Example

Extend the Doctrine\Common\Lexer\AbstractLexer class and implement
the getCatchablePatterns, getNonCatchablePatterns, and getType
methods. Here is a very simple example lexer implementation named CharacterTypeLexer.
It tokenizes a string to T_UPPER, T_LOWER andT_NUMBER tokens:

.. code-block:: php
<?php

text
use Doctrine\Common\Lexer\AbstractLexer;

/**
 * @extends AbstractLexer<CharacterTypeLexer::T_*, string>
 */
class CharacterTypeLexer extends AbstractLexer
{
    const T_UPPER =  1;
    const T_LOWER =  2;
    const T_NUMBER = 3;

    protected function getCatchablePatterns(): array
    {
        return [
            '[a-zA-Z0-9]',
        ];
    }

    protected function getNonCatchablePatterns(): array
    {
        return [];
    }

    protected function getType(&$value): int
    {
        if (is_numeric($value)) {
            return self::T_NUMBER;
        }

        if (strtoupper($value) === $value) {
            return self::T_UPPER;
        }

        if (strtolower($value) === $value) {
            return self::T_LOWER;
        }
    }
}

Use CharacterTypeLexer to extract an array of upper case characters:

.. code-block:: php
<?php

text
class UpperCaseCharacterExtracter
{
    public function __construct(private CharacterTypeLexer $lexer)
    {
    }

    /** @return list<string> */
    public function getUpperCaseCharacters(string $string): array
    {
        $this->lexer->setInput($string);
        $this->lexer->moveNext();

        $upperCaseChars = [];
        while (true) {
            if (!$this->lexer->lookahead) {
                break;
            }

            $this->lexer->moveNext();

            if ($this->lexer->token->isA(CharacterTypeLexer::T_UPPER)) {
                $upperCaseChars[] = $this->lexer->token->value;
            }
        }

        return $upperCaseChars;
    }
}

$upperCaseCharacterExtractor = new UpperCaseCharacterExtracter(new CharacterTypeLexer());
$upperCaseCharacters = $upperCaseCharacterExtractor->getUpperCaseCharacters('1aBcdEfgHiJ12');

print_r($upperCaseCharacters);

The variable $upperCaseCharacters contains all of the upper case
characters:

.. code-block:: php
Array
(
[0] => B
[1] => E
[2] => H
[3] => J
)

This is a simple example but it should demonstrate the low level API
that can be used to build more complex parsers.

File: .github/dependabot.yml

version: 2
updates:

  • package-ecosystem: "github-actions"
    directory: "/"
    schedule:
    interval: "weekly"
    labels:
    • "CI"
      groups:
      doctrine:
      patterns:
      • "doctrine/*"

File: .github/FUNDING.yml

patreon: phpdoctrine
tidelift: packagist/doctrine%2Flexer
custom: https://www.doctrine-project.org/sponsorship.html

File: .github/workflows/coding-standards.yml

name: "Coding Standards"

on:
pull_request:
branches:
- ".x"
paths:
- .github/workflows/coding-standards.yml
- composer.
- src/**
- phpcs.xml.dist
- tests/**
push:
branches:
- ".x"
paths:
- .github/workflows/coding-standards.yml
- composer.
- src/**
- phpcs.xml.dist
- tests/**

jobs:
coding-standards:
name: "Coding Standards"
uses: "doctrine/.github/.github/workflows/[email protected]"

File: .github/workflows/composer-lint.yml

name: "Composer Lint"

on:
pull_request:
branches:
- ".x"
paths:
- .github/workflows/composer-lint.yml
- composer.json
push:
branches:
- ".x"
paths:
- .github/workflows/composer-lint.yml
- composer.json

jobs:
composer-lint:
name: "Composer Lint"
uses: "doctrine/.github/.github/workflows/[email protected]"

File: .github/workflows/continuous-integration.yml

name: "Continuous Integration"

on:
pull_request:
branches:
- ".x"
paths:
- .github/workflows/continuous-integration.yml
- composer.
- src/**
- phpunit.xml.dist
- tests/**
push:
branches:
- ".x"
paths:
- .github/workflows/continuous-integration.yml
- composer.
- src/**
- phpunit.xml.dist
- tests/**

jobs:
phpunit:
name: "PHPUnit"
uses: "doctrine/.github/.github/workflows/[email protected]"
with:
php-versions: '["8.1", "8.2", "8.3", "8.4", "8.5"]'
secrets:
CODECOV_TOKEN: "${{ secrets.CODECOV_TOKEN }}"

File: .github/workflows/release-on-milestone-closed.yml

name: "Automatic Releases"

on:
milestone:
types:
- "closed"

jobs:
release:
name: "Git tag, release & create merge-up PR"
uses: "doctrine/.github/.github/workflows/[email protected]"
secrets:
GIT_AUTHOR_EMAIL: ${{ secrets.GIT_AUTHOR_EMAIL }}
GIT_AUTHOR_NAME: ${{ secrets.GIT_AUTHOR_NAME }}
ORGANIZATION_ADMIN_TOKEN: ${{ secrets.ORGANIZATION_ADMIN_TOKEN }}
SIGNING_SECRET_KEY: ${{ secrets.SIGNING_SECRET_KEY }}

File: .github/workflows/static-analysis.yml

name: "Static Analysis"

on:
pull_request:
branches:
- ".x"
paths:
- .github/workflows/static-analysis.yml
- composer.
- src/**
- phpstan*
- tests/**
push:
branches:
- ".x"
paths:
- .github/workflows/static-analysis.yml
- composer.
- src/**
- phpstan*
- tests/**

jobs:
static-analysis:
name: "Static Analysis"
uses: "doctrine/.github/.github/workflows/[email protected]"

File: .github/workflows/website-schema.yml

name: "Website config validation"

on:
pull_request:
branches:
- ".x"
paths:
- ".doctrine-project.json"
- ".github/workflows/website-schema.yml"
push:
branches:
- ".x"
paths:
- ".doctrine-project.json"
- ".github/workflows/website-schema.yml"

jobs:
json-validate:
name: "Validate JSON schema"
uses: "doctrine/.github/.github/workflows/[email protected]"

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →