gritql Documentation

GritQL is a query language for searching, linting, and modifying code.

4,615 stars Rust Rust 20 files · ~6,785 tokens #ast#codemod#javascript#linter#refactoring#rust#search#tree-sitter
RAW Doc

Technical Documentation: biomejs/gritql

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md)

GritQL is a declarative query language for searching and modifying source code.

  • 📖 Start simply without learning AST details: any code snippet is a valid GritQL query
  • ⚡️ Use Rust and query optimization to scale up to 10M+ line repositories
  • 📦 Use Grit's built-in module system to reuse 200+ standard patterns or share your own
  • ♻️ Once you learn GritQL, you can use it to rewrite any target language: JavaScript/TypeScript, Python, JSON, Java, Terraform, Solidity, CSS, Markdown, YAML, Rust, Go, or SQL
  • 🔧 GritQL makes it easy to include auto-fix rules for faster remediation

Getting started

Read the documentation, interactive tutorial, or run grit --help.

Installation

Install the Grit CLI:

sh
curl -fsSL https://docs.grit.io/install | bash

Usage

Search for all your console.log calls by putting the desired pattern in backticks:

sh
grit apply '`console.log($_)`'

Replace console.log with winston.log, using => to create rewrites:

sh
grit apply '`console.log($msg)` => `winston.log($msg)`'

Save the pattern to a grit.yaml file and exclude test cases in a where clause:

sh
cat << 'EOF' > .grit/grit.yaml
patterns:
  - name: use_winston
    level: error
    body: |
      `console.log($msg)` => `winston.log($msg)` where {
        $msg <: not within or { `it($_, $_)`, `test($_, $_)`, `describe($_, $_)` }
      }
EOF
grit apply use_winston

Run grit check to enforce your patterns as custom lints.

sh
grit check

Examples

Remove all `console.log` calls, unless they are inside a try-catch block

grit
`console.log($log)` => . where {
  $log <: not within `try { $_ } catch { $_ }`
}

Replace a method call with a new method call

grit
`$instance.oldMethod($args)` => `$instance.newMethod($args)` where {
  $program <: contains `$instance = new TargetClass($_)`
}

More examples

Many more examples can be found in the GritQL standard library.

Patterns can be combined to create complex queries, including large refactors.

Why GritQL?

GritQL comes from our experiences with conducting large scale refactors and migrations.

Usually, migrations start with exploratory work to figure out the scope of the problem—often using simple grep searches. These are easy to start with, but most migrations end up accumulating additional requirements like ensuring the right packages are imported and excluding cases which don’t have a viable migration path.

Eventually, any complex migration ends up being a full codemod program written with a tool like jscodeshift. This comes with its own problems:

  • Most of the exploratory work has to be abandoned as you figure out how to represent your original regex search as an AST.
  • Reading/writing a codemod requires mentally translating from AST names back to what source code actually looks like.
  • Most frameworks are not composable, so you’re stuck copying patterns back and forth.
  • Performance is often an afterthought, so iterating on a large codemod can be painfully slow.
  • Codemod frameworks are language-specific, so if you’re hopping between multiple languages—or trying to migrate a shared API—you have to learn different frameworks.

GritQL is our attempt to develop a powerful middle ground:

  • Exploratory analysis is easy: just put a code snippet in backticks and use $metavariables for holes you want to represent.
  • Incrementally add complexity by introducing side conditions with where clauses.
  • Reuse named patterns to avoid rebuilding queries, and use shared patterns from our standard library for common tasks like ensuring modules are imported.
  • Written in Rust for maximum performance: rewrite millions of lines of code in seconds.

Acknowledgements

GritQL uses tree-sitter for all language parsers and benefits greatly from the Rust ecosystem.

GritQL is released under the MIT license.

Contributing

Contributions are welcome. To get started, check out the contributing guidelines.

You can also join us on Discord.

README (`docs/README.md`)

Docs App

This is a simple Next.js application hosting the GritQL docs.

Dev

shell
pnpm dev

Test

Install Playwright if you haven't already:

shell
pnpm playwright install

Run the tests using Playwright against your local instance:

shell
NEXT_PUBLIC_DOCS_URL=http://localhost:3200 pnpm test

Src Workers README (`docs/src/workers/README.md`)

Worker code

Next.js has built-in support for including worker code. Unfortunately, this does not traverse module boundaries effectively.

Therefore, we have a very hacky solution of writing the code for the workers here (in apps/docs/src/workers).

It is then copied into apps/web via a generate command.

Resources Language Metavariables Tree Sitter Elixir Docs Index (`resources/language-metavariables/tree-sitter-elixir/docs/index.md`)

Development notes

This documentation covers rationale behind some of the design and implementation decisions,
as well as basic Tree-sitter tips that are relevant.

Pages

Acknowledgements

While this parser is written from scratch, there were previous efforts that made
for a helpful reference:

In particular, some test cases were sourced from ananthakumaran/tree-sitter-elixir.

Resources Language Metavariables Tree Sitter Sql Docs Index (`resources/language-metavariables/tree-sitter-sql/docs/index.md`)

layout: default

tree-sitter-sql

The SQL grammar is defined at @ github://derekstride/tree-sitter-sql.
You can view the latest parser artifacts on site or on the gh-pages
branch.

Find previous artifacts in the history on GitHub.

The artifacts can also be found here:

{% for file in site.static_files %}
[{{ file.path }}](/tree-sitter-sql{{ file.path }})
{% endfor %}

Src Markdoc Partials Alpha (`docs/src/markdoc/partials/alpha.md`)

{% note type="warning" %}
Warning: This is an alpha feature in active development, without stability guarantees. Please contact us on Discord or Slack if you have any questions.
{% /note %}

Src Markdoc Partials Markdown Example (`docs/src/markdoc/partials/markdown_example.md`)

markdown {% filename=".grit/patterns/remove_console_log.md" %}
---
tags: [optional, tags, here]
---
# Remove console.log

Remove console.log in production code.

```grit
`console.log($_)` => .
```

## Test case one

This is the first test case. You can include an explanation of the case here.

```typescript
console.error("keep this");
console.log('remove this!');
```

It is fine to include additional descriptive text around the test cases.
This is often used to explain the context of the test case, or to explain a convention.

```typescript
console.error("keep this");

```

Src Markdoc Partials Markdown Rules (`docs/src/markdoc/partials/markdown_rules.md`)

  • The name of the file is used as the name of the pattern (without the .md extension).
  • The title of the pattern is retrieved from the first heading in the file. This can be overridden by adding a title field in the front matter.
  • The description of the pattern is retrieved from the first non-heading paragraph in the file.
  • The GritQL body is retrieved from the first fenced code block in the file.
  • Additional pattern metadata can be configured as YAML in the frontmatter of the markdown file, delineated between --- lines at the start of the document.
    {% partial file="pattern_shared_meta.md" /%}
  • Subheadings are used for each test case, following these conventions:
    • If a subheading has a single code block, it represents a test case that should be matched by the pattern. You don't need to provide a transformed example.
    • If a subheading has two code blocks, the first represents the input and the second represents the expected output.
    • A negative test case should have two identical code blocks.
    • Within the sample patterns, you can use // @filename: example.js to represent multiple input/output files that should be tested as a group - like this example.

Src Markdoc Partials Paid (`docs/src/markdoc/partials/paid.md`)

{% note type="info" %}
This is a paid feature. Please contact us on Discord or subscribe to use it.
{% /note %}

Src Markdoc Partials Pattern Shared Meta (`docs/src/markdoc/partials/pattern_shared_meta.md`)

  • level: (Optional, one of none, info, warn, error) The enforcement level of the pattern for running diagnostics via grit check. Defaults to info.
  • tags: (Optional, string[]) A list of tags which can be used to filter patterns.

Src Markdoc Partials Slash Continue (`docs/src/markdoc/partials/slash_continue.md`)

`/grit continue`

When Grit has paused work on a pull request (to allow developer edits), you can use /grit continue to transfer the pull request back to Grit's control.

Src Markdoc Partials Slash New (`docs/src/markdoc/partials/slash_new.md`)

`/grit new `

Create a new autoreview rule based on the comment contents. This should be used as an inline comment on a specific line of code.

Src Markdoc Partials Slash Rebase (`docs/src/markdoc/partials/slash_rebase.md`)

`/grit rebase`

Rebase the pull request onto the base branch. Grit will automatically attempt to resolve any merge conflicts and push the changes to the pull request.

Src Markdoc Partials Slash Reject (`docs/src/markdoc/partials/slash_reject.md`)

`/grit reject`

When replying to a code review from Grit, you can write /grit reject in a comment to reject the finding as a false positive.

Src Markdoc Partials Slash Report (`docs/src/markdoc/partials/slash_report.md`)

`/grit report `

If Grit fails to flag an issue with an existing rule, you can comment /grit report on the pull request to improve the Grit AI. This will attach the line comment to the linked rule.

Src Markdoc Partials Tutorial2 Transformed (`docs/src/markdoc/partials/tutorial2_transformed.md`)

js
import styled, { css } from 'styled-components';

export const Button = (({ $primary, ...props }) => {
  const primaryClasses = $primary ? 'bg-red text-black' : '';
  const className = twMerge(`bg-transparent rounded-md hover:brightness-85 ${primaryClasses}`);
  return (
    <a className={className} {...props} />
  );
});

Src Markdoc Partials Tutorial Snippet (`docs/src/markdoc/partials/tutorial_snippet.md`)

javascript
console.log("This message is different");

// Comment with console.log("Hello world!") in it

function main(me) {
  console.log(42)
  console.log("Hello world!")
  console.log("42")

  console.log({
    name: "John Doe",
    age: 42,
  })
  return me
};

console.error("This is an error message")
console.log(
  'Hello world!'
);
console.log("This is a user-facing message")

Src Markdoc Partials Tutorial Snippet2 (`docs/src/markdoc/partials/tutorial_snippet2.md`)

javascript
console.log(
  'This is a log message'
)

console.warn("This is a warning message")

console.error("This is an error message")

try {
  throw new Error("This is an error")
} catch (e) {
  console.log(e);
}

Resources Language Metavariables Tree Sitter Elixir Docs Highlighting (`resources/language-metavariables/tree-sitter-elixir/docs/highlighting.md`)

Syntax highlighting

For detailed introduction see the official guide on Syntax highlighting.

Briefly speaking, Tree-sitter uses the rules in queries/highlights.scm to annotate nodes
with specific tokens, then it maps those tokens to formatting style according to user-defined
theme.

To test highlighting using the CLI, you need to create local configuration.

shell
# Create the config file
npx tree-sitter init-config

The above command should print out the config location, so that you can further configure it.
Open the file and modify "parser-directories" to include the parent directory of tree-sitter-elixir.
Also, you can optionally customize the theme, here's a tiny subset of the One Dark theme:

json
{
  "number": {
    "color": "#61afef",
    "bold": true
  },
  "string": "#98c379",
  "string.escape": "#56b6c2",
  "string.special": "#61afef",
  "string.regexp": "#e06c75",
  "type": "#e06c75",
  "comment": {
    "color": "#5c6370",
    "italic": true
  },
  "punctuation": "#abb2bf",
  "punctuation.special": "#be5046",
  "operator": {
    "color": "#d19a66",
    "bold": true
  },
  "variable": "#abb2bf",
  "function": "#61afef",
  "constant": "#61afef",
  "constant.builtin": {
    "color": "#e06c75",
    "bold": true
  },
  "keyword": "#c678dd",
  "attribute": "#e06c75",
  "embedded": null
}

With this setup you can test highlighting on files using the Tree-sitter CLI.

shell
npx tree-sitter highlight tmp/test.ex

npx tree-sitter highlight test/highlight/**/*.ex

Resources Language Metavariables Tree Sitter Elixir Docs Parser (`resources/language-metavariables/tree-sitter-elixir/docs/parser.md`)

Parser

The AST

When it comes to AST Elixir is a rather specific language due to its macro system.
From the perspective of our parser, the important implication is that a seemingly
invalid code can be a valid syntax when used in a macro (or just put in the quote
expression). For example:

elixir
quote do
  def Bar.foo(x), definitely_not_do: 1
  %{a}
  */2
end

As opposed to other languages, core constructs like def, if and for are not
particularly special either, since they are itself regular functions (or macros rather).
As a result, these constructs can be used "improperly" in a quoted expression, as shown above.

Consequently, to correctly parse all Elixir code, we need the AST to closely match
the Elixir AST. See Elixir / Syntax reference
for more details.

Whenever possible, we try using a more specific nodes (like binary/unary operator), but only
to the extent that doesn't lose on generality. To get a sense of what the AST looks like, have
a look at the tests in test/corpus/.

Getting started with Tree-sitter

For detailed introduction see the official guide on Creating parsers.

Essentially, we define relevant language rules in grammar.js, based on which
Tree-sitter generates parser code (under src/). In some cases, we want to write
custom C code for tokenizing specific character sequences (in src/scanner.c).

The grammar rules may often conflict with each other, meaning that the given
sequence of tokens has multiple valid interpretations given one token of lookahead.
In many conflicts we always want to pick one interpretation over the other and we can
do this by assigning different precedence and associativity to relevant rules, which
tells the parser which way to go.

For example given expression1 * expression2 • * the next token we see ahead is *.
The parser needs to decide whether expression1 * expression2 is a complete binary operator
node, or if it should await the next expression and interpret it as expression1 * (expression2 * expression3).
Since the * operator is left-associative we can use prec.left on the corresponding
grammar rule, to inform the parser how to resolve this conflict.

However, in some cases looking at one token ahead isn't enough, in which case we can add
the conflicting rules to the conflicts list in the grammar. Whenever the parser stumbles
upon this conflict it uses its GLR algorithm, basically considering both interpretations
until one leads to parsing error. If both paths parse correctly (there's a genuine ambiguity)
we can use dynamic precedence (prec.dynamic) to decide on the preferred path.

Using the CLI

tree-sitter

shell
# See CLI usage
npx tree-sitter -h

# Generate the the parser code based on grammar.js
npx tree-sitter generate

# Run tests
npx tree-sitter test
npx tree-sitter test --filter "access syntax"

# Parse a specific file
npx tree-sitter parse tmp/test.ex
npx tree-sitter parse -x tmp/test.ex

# Parse codebase to verify syntax coverage
npx tree-sitter parse --quiet --stat 'tmp/elixir/**/*.ex*'

Whenever you make a change to grammar.js remember to run generate,
before verifying the result. To test custom code, create an Elixir file
like tmp/test.ex and use parse on it. The -x flag prints out the
source grouped into AST nodes as XML.

Additional scripts

shell
# Format the grammar.js file
npm run format

# Run parser against the given repository
scripts/parse_repo.sh elixir-lang/elixir

# Run parser against a predefined list of popular repositories
scripts/integration_test.sh

Implementation notes

This section covers some of the implementation decisions that have a more
elaborated rationale. The individual subsections are referenced in the code.

Ref 1. External scanner for quoted content

We want to scan quoted content as a single token, but it requires lookahead.
Specifically the # character may no longer be quoted content if followed by {.
Also, inside heredoc string tokenizing " (or ') requires lookahead to know
if it's already part of the end delimiter or not.

Since we need to use external scanner, we need to know the delimiter type.
One way to achieve this is using external scanner to scan the start delimiter
and then storing its type on the parser stack. This approach requires the parser
to allocate enough memory upfront and implement serialization/deserialization,
which ideally would be avoided. To avoid this, we use a different approach!
Instead of having a single quoted_content token, we have specific tokens for
each quoted content type, such as _quoted_content_i_single, _quoted_content_i_double.
Once the start delimiter is tokenized, we know which quoted content should be
tokenized next, and from the token we can infer the end delimiter and whether
it supports interpolation. In other words, we extract the information from the
parsing state, rather than maintaining custom parser state.

Ref 2. External scanner for newlines

Generally newlines may appear in the middle of expressions and we ignore them
as long as the expression is valid, that's why we list newline under extras.

When a newline follows a complete expression, most of the time it should be
treated as terminator. However, there are specific cases where the newline is
non-breaking and treated as if it was just a space. This cases are:

  • call followed by newline and a do end block
  • expression followed by newline and a binary operator

In both cases we want to tokenize the newline as non-breaking, so we use external
scanner for lookahead.

Note that the relevant rules already specify left/right associativity, so if we
simply added optional("\n") the conflicts would be resolved immediately rather
without using GLR.

Additionally, since comments may appear anywhere and don't change the context,
we also tokenize newlines before comments as non-breaking.

Ref 3. External scanner for unary + and -

Plus and minus are either binary or unary operators, depending on the context.
Consider the following variants

text
a + b
a+b
a+ b
a +b

In the first three expressions + is a binary operator, while in the last one
+ is an unary operator referring to local call argument.

To correctly tokenize all cases we use external scanner to tokenize a special empty
token (_before_unary_operator) when the spacing matches a +b, which forces the
parser to pick the unary operator path.

Ref 4. External scanner for `not in`

The not in operator may have an arbitrary inline whitespace between not and in.

We cannot use a regular expression like /not[ \t]+in/, because it would also match
in expressions like a not inn as the longest matching token.

A possible solution could be seq("not", "in") with dynamic conflict resolution, but
then we tokenize two separate tokens. Also to properly handle a not inn, we would need
keyword extraction, which causes problems in our case (https://github.com/tree-sitter/tree-sitter/issues/1404).

In the end it's easiest to use external scanner, so that we can skip inline whitespace
and ensure token ends after in.

Ref 5. External scanner for quoted atom start

For parsing quoted atom : we could make the " (or ') token immediate, however this
would require adding immediate rules for single/double quoted content and listing them
in relevant places. We could definitely do that, but using external scanner is actually
simpler.

Ref 6. Identifier pattern

See Elixir / Unicode Syntax for official
notes.

Tree-sitter already supports unicode properties in regular expressions, however character
class subtraction is not supported.

For the base <Start> and <Continue> we can use [\p{ID_Start}] and [\p{ID_Continue}]
respectively, since both are supported and according to the
Unicode Annex #31
they match the ranges listed in the Elixir docs.

For atoms this translates to a clean regular expression.

For variables however, we want to exclude uppercase (\p{Lu}) and titlecase (\p{Lt})
categories from \p{ID_Start}. As already mentioned, we cannot use group subtraction
in the regular expression, so instead we need to create a suitable group of characters
on our own.

After removing the uppercase/titlecase categories from [\p{ID_Start}], we obtain the
following group:

[\p{Ll}\p{Lm}\p{Lo}\p{Nl}\p{Other_ID_Start}-\p{Pattern_Syntax}-\p{Pattern_White_Space}]

At the time of writing the subtracted groups actually only remove a single character:

elixir
Mix.install([{:unicode_set, "~> 1.1"}])

Unicode.Set.to_utf8_char(
  "[[[:Ll:][:Lm:][:Lo:][:Nl:][:Other_ID_Start:]] & [[:Pattern_Syntax:][:Pattern_White_Space:]]]"
)
#=> {:ok, [11823]}

Consequently, by removing the subtraction we allow just one additional (not common) character,
which is perfectly acceptable.

It's important to note that JavaScript regular expressions don't support the \p{Other_ID_Start}
unicode category. Fortunately this category is a small set of characters introduces for
backward compatibility, so we can
enumerate it manually:

elixir
Mix.install([{:unicode_set, "~> 1.1"}])

Unicode.Set.to_utf8_char("[[[:Other_ID_Start:]] - [[:Pattern_Syntax:][:Pattern_White_Space:]]]")
|> elem(1)
|> Enum.flat_map(fn
  n when is_number(n) -> [n]
  range -> range
end)
|> Enum.map(&Integer.to_string(&1, 16))
#=> ["1885", "1886", "2118", "212E", "309B", "309C"]

Finally, we obtain this regular expression group for variable <Start>:

[\p{Ll}\p{Lm}\p{Lo}\p{Nl}\u1885\u1886\u2118\u212E\u309B\u309C]

Ref 7. Keyword token

We tokenize the whole keyword sequence like do: as a single token.
Ideally we wouldn't include the whitespace, but since we use token
it gets include. However, this is an intentionally accepted tradeoff,
because using token significantly simplifies the grammar and avoids
conflicts.

The alternative approach would be to define keyword as seq(alias(choice(...), $._keyword_literal), $._keyword_end),
where we list all other tokens that make for for valid keyword literal
and use custom scanner for _keyword_end to look ahead without tokenizing
the whitespace. However, this approach generates a number of conflicts
because : is tokenized separately and phrases like fun fun • do or
fun • {} are ambiguous (interpretation depends on whether : comes next).
Resolving some of these conflicts (for instance special keywords like {} or %{})
requires the use of external scanner. Given the complexities this approach
brings to the grammar, and consequently the parser, we stick to the simpler
approach.

Ref 8. Empty anonymous function

As opposed to the Elixir parser, we successfully parse anonymous functions with
no stab clauses, so this is valid:

text
x = fn

end

This code may appear if an editor extension automatically inserts end after
fn. We want it to parse as anonymous function node, so that functionality such
as indentation works as expected.

If we require at least one stab clause, the above would be parsed with an error,
where fn and end are both identifiers. That is not useful. Note that both
fn and end are reserved keywords, so there is no case where they would
actually be identifiers, hence no ambiguity.

Ideally, this would parse it as an anonymous function node with an error, however
that does not seem straightforward to achieve.

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#ast Browse ast catalog → #codemod Browse codemod catalog → #javascript Browse javascript catalog → #linter Browse linter catalog → #refactoring Browse refactoring catalog → #rust Browse rust catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →