## 1. Project Overview & Quickstart (pulsar-edit/package-preload)
# package-preload
Open-source repository pulsar-edit/package-preload
### Repository Details
- **Repository:** [pulsar-edit/package-preload](https://github.com/pulsar-edit/package-preload)
- **Primary Language:** Code
*Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.*
## 2. Official Technical Reference & Guides (pulsar-edit/docs)
## File: README.md
# Pulsar Documentation Eleventy
This new Pulsar Documentation website utilizes `eleventy`.
This project superseedes previous efforts utilizing `sloth` as it's been determined that `eleventy` is the more mature implementation of the same exact goals of `sloth` and it makes the most sense to utilize a framework that's more mature, and requires significantly less maintenance.
## Installation
```bash
npm install
git submodule init
git submodule update
```
## Running
* `npm run build`: Builds the site
* `npm run serve`: Creates a live version of the site at `:8080`
* `npm run build:debug`: Builds the site with debug flags set
* `npm run serve:debug`: Creates a live server with debug flags set
## Tool Translations from current Pulsar site to this one
While obviously nearly all text should be redone, there are also some technical changes that should occur:
### KDB
Switch uses of `Ctrl` to `[[Ctrl]]`
### Images
Switch links with `@images` to `/img`
## TODO's
The biggest aspects of this that still have to be completed:
- [X] Finalize CSS implementations
- [X] Create all EJS templates needed
- [X] Create collections and automatically create collection views on summary pages
- [X] Implement Dark Mode
- [X] Implement Light Mode
- [X] Implement Theme Switcher
- [X] Add all original Launch Manual Docs
- [X] Add all relevant documentation
- [X] Implement Pulsar API Docs
- [X] Implement Hovercards
- [ ] Update **ALL** Links!
- [ ] Ensure all sidebar descriptions are filled in
- [ ] Add support for syntax highlighting
- [ ] Design CSS to account for indented returns in the API view
- [X] Find a way to generate API docs from most recent Pulsar versions
- [X] GHA to generate new Pulsar Docs once above is done
- [ ] Wait for `pulsardoc` to support Coffee and enable coffee based repos to document
## Style Guide
Here will be the very short, non-complete style guide:
### Links to GitHub
If linking to a resource on GitHub, such as the `language-git` package, ensure to never include the branch.
By default if you copy paste the link to the `language-git` package it will look like:
`https://github.com/pulsar-edit/pulsar/tree/master/packages/language-git`
But in the rare event the branch name never changes, we can make simple changes now to avoid stress later, by replacing the `master` here with `HEAD`, such as:
`https://github.com/pulsar-edit/pulsar/tree/HEAD/packages/language-git`
### Heading Levels
Within each page of markdown you should never include an `` heading level (`#`) always start on `` (`##`) as the title of the page is extracted and used as a level one heading in all cases.
### Links between pages
For simplicity, and ability to find and replace links in the future, all links between pages should be from the root of the website.
Meaning start all links with `/` and direct to the page you are attempting to link to.
A safe exception to this rule is for anchor links on the same page, as those can just be the anchor itself.
But for all other links, please do follow this rule and do not use relative links.
Examples: When linking to `/example/sub/page` from `/example/sub/hello`
- Do: `/example/sub/page`
- Do **NOT**: `../page`
## Update Source Code Documentation
### Auto
Create an issue against this repository, with the title formatted strictly as the new Pulsar version number.
Such as `1.124.0`
Then once the issue is created, afterwards add the label `update-docs`
After a short time a PR will be created against the repository with the changes to go ahead and review.
### Manual
1) Update the JSON file `./pulsar-api/latest.json` with the newest Pulsar version.
2) Update all submodules to their latest commit, such as `git submodule update --remote --merge`
3) Run `npm run generate:docs` to trigger the documentation to be updated.
---
## File: docs/developing-for-pulsar/creating-a-fork-of-a-core-package.md
---
title: Creating a fork of a core package
layout: doc.ejs
---
Several of Pulsar’s core packages are maintained in the [`packages` directory of the `pulsar-edit/pulsar` repository](https://github.com/pulsar-edit/pulsar/tree/master/packages). If you would like to use one of these packages as a starting point for your own package, please follow the steps below.
::: tip Tip
In most cases, we recommend [generating a brand-new package](../developing-a-package/#package-generator) or a [brand-new theme](../developing-a-theme/#creating-a-syntax-theme) as the starting point for your creation. The guide below applies only to situations where you want to create a package that closely resembles a core Pulsar package.
:::
## Creating your new package
For the sake of this guide, let’s assume that you want to start with the current code in the [one-light-ui](https://github.com/pulsar-edit/pulsar/tree/master/packages/one-light-ui) package, make some customizations to it, and publish your new package under the name `one-light-ui-plus`.
1. Download the [current contents of the pulsar-edit/pulsar repository as a zip file](https://github.com/pulsar-edit/pulsar/archive/master.zip).
2. Unzip the file to a temporary location (for example, `/tmp/pulsar``C:\TEMP\pulsar`).
3. Copy the contents of the desired package into a working directory for your fork.
::: tabs#core-hacking
@tab Linux
```sh
$ cp -R /tmp/pulsar/packages/one-light-ui ~/src/one-light-ui-plus
$ cd ~/src/one-light-ui-plus
```
@tab macOS
```sh
$ cp -R /tmp/pulsar/packages/one-light-ui ~/src/one-light-ui-plus
$ cd ~/src/one-light-ui-plus
```
@tab Windows
```powershell
$ Copy-Item -Path "C:\TEMP\pulsar\packages\one-light-ui" -Destination "C:\src\one-light-ui-plus" -Recurse -Force
$ cd C:\src\one-light-ui-plus
```
:::
4. Create a local repository and commit the initial contents
```sh
$ git init
$ git commit -am "Import core Pulsar package"
```
5. Update the `name` property in `package.json` to give your package a unique name.
6. Make the other customizations that you have in mind.
7. Commit your changes.
```sh
$ git commit -am "Apply initial customizations"
```
8. [Create a public repository on github.com](https://help.github.com/articles/create-a-repo/) for your new package.
9. Follow the instructions in the github.com UI to push your code to your new online repository.
10. Follow the steps in the [Publishing guide](../publishing/) to publish your new package.
## Merging upstream changes into your package
The code in the original package will continue to evolve over time, either to fix bugs or to add new enhancements. You may want to incorporate some or all of those updates into your package. To do so, you can follow [these steps](../maintaining-a-fork-of-a-core-package/) for merging upstream changes into your package.
---
## File: docs/developing-for-pulsar/creating-a-grammar.md
---
title: Creating a grammar
layout: doc.ejs
---
Pulsar’s syntax highlighting and code folding are powered by packages called _grammars_. A grammar is something that knows how to analyze a certain language in order to provide syntax highlighting, code folding, and indentation hinting.
Pulsar grammars come in a few flavors:
* TextMate
* Modern Tree-sitter
* Legacy Tree-sitter (deprecated; see note below)
[Tree-sitter grammars](../creating-a-grammar-modern-tree-sitter/) are relatively new. Many languages in Pulsar are still supported by [TextMate grammars](../creating-a-grammar-textmate/) and TextMate grammars are easier to write from scratch, so they may still be the best option for more obscure languages. Tree-sitter grammars assume the existence of a [Tree-sitter parser](https://tree-sitter.github.io/tree-sitter/#parsers) having been written for a particular language. But if such a parser exists, a Tree-sitter grammar will be more performant and easier to author.
::: note Why two Tree-sitter systems?
Atom (now Pulsar) was the first editor to support Tree-sitter grammars, but the legacy implementation predated many of Tree-sitter’s current features. The need for Pulsar to switch to the `web-tree-sitter` bindings for compatibility reasons was an opportunity to update the architecture and the original Tree-sitter grammars.
The new “modern” Tree-sitter grammars should deliver better syntax highlighting, indentation, and code folding, while still offering better performance than TextMate-style grammars. “Legacy” Tree-sitter grammars are deprecated and will soon be removed from Pulsar.
:::
## The package
::: tip
Grammar package names should start with `language-`.
:::
A “grammar package” is just an ordinary Pulsar package that happens to contain one or more grammars. It can do anything that any other Pulsar package can do — run code, declare keybindings and styles, and so on — but usually doesn’t need to.
When Pulsar loads a package during startup, it will check for the existence of a `grammar` directory at the package root. This is where your grammars should be placed, with the format differing based on the type of grammar.
---
## File: docs/developing-for-pulsar/creating-a-grammar-converting-from-textmate.md
---
title: Creating a grammar (converting from TextMate)
layout: doc.ejs
---
It’s possible that you have themes or grammars from [TextMate](https://macromates.com) that you like and use and would like to convert to Pulsar. If so, you’re in luck because there are tools to help with the conversion.
## Converting a TextMate grammar bundle
Converting a TextMate bundle will allow you to use its editor preferences, snippets, and colorization inside Pulsar.
Let’s convert the TextMate bundle for the [R]() programming language. You can find other existing TextMate bundles [on GitHub](https://github.com/textmate).
You can convert the R bundle with the following command:
```command-line
$ pulsar -p init --package language-r --convert https://github.com/textmate/r.tmbundle
```
You can now change directory into `language-r` to see the converted bundle. Once you link your package with the `pulsar -p link` command, your new package is ready to use. Launch Pulsar and open a `.r` file in the editor to see it in action!
## Converting a TextMate syntax theme
This section will go over how to convert a [TextMate](https://macromates.com) theme to an Pulsar theme.
### Differences
TextMate themes use [plist](https://en.wikipedia.org/wiki/Property_list) files while Pulsar themes use [CSS](https://en.wikipedia.org/wiki/Cascading_Style_Sheets) or [Less](http://lesscss.org) to style the UI and syntax in the editor.
The utility that converts the theme first parses the theme’s plist file and then creates comparable CSS rules and properties that will style Pulsar similarly.
### Convert the theme
Download the theme you wish to convert.
Now, let’s say you’ve downloaded the theme to `~/Downloads/MyTheme.tmTheme`, you can convert the theme with the following command:
```sh
$ pulsar -p init --theme my-theme --convert ~/Downloads/MyTheme.tmTheme
```
You can then change directory to `my-theme` to see the converted theme.
### Activate the theme
Once your theme is installed, you can enable it by launching Pulsar and opening the Settings View with the _Edit > Preferences__Pulsar > Preferences__File > Preferences_ menu item. Then select the "Themes" tab on the left side navigation. Finally, choose "My Theme" from the "Syntax Theme" dropdown menu to enable your new theme.
Your theme is now enabled. Open an editor to see it in action!
---
## File: docs/developing-for-pulsar/creating-a-grammar-legacy-tree-sitter.md
---
title: Creating a grammar (legacy Tree-sitter)
layout: doc.ejs
---
Pulsar also has a syntax highlighting and code folding system powered by an earlier version of [Tree-sitter](http://tree-sitter.github.io/tree-sitter).
::: warning
Legacy Tree-sitter grammars are deprecated, and support for them **will be removed** in newer Pulsar versions. It is **not recommended** to develop any new packages in this format. As of version 1.114.0, legacy Tree-sitter grammars are not the default, and must be opted into.
:::
## Getting started
There are two components required to use Tree-sitter in Pulsar: a _parser_ and
a _grammar_ file.
## The parser
Tree-sitter generates parsers based on [context-free grammars](https://en.wikipedia.org/wiki/Context-free_grammar)
that are typically written in JavaScript. The generated parsers are C libraries
that can be used in other applications as well as Pulsar.
They can also be developed and tested at the command line, separately from
Pulsar. Tree-sitter has [its own documentation page](http://tree-sitter.github.io/tree-sitter/creating-parsers)
on how to create these parsers. The [Tree-sitter GitHub organization](https://github.com/tree-sitter)
also contains a lot of example parsers that you can learn from, each in its own
repository.
Once you have created a parser, you need to publish it to [the NPM registry](https://npmjs.com)
to use it in Pulsar. To do this, make sure you have a `name` and `version` in
your parser’s `package.json`:
```js
{
"name": "tree-sitter-mylanguage",
"version": "0.0.1",
// ...
}
```
then run the command `npm publish`.
## The package
Once you have a Tree-sitter parser that is available on npm, you can use it in
your Pulsar package. Packages with grammars are, by convention, always named
starting with _language_. You’ll need a folder with a `package.json`, a
`grammars` subdirectory, and a single `json` or `cson` file in the `grammars`
directory, which can be named anything.
```
language-mylanguage
├── LICENSE
├── README.md
├── grammars
│ └── mylanguage.cson
└── package.json
```
## The grammar file
The `mylanguage.cson` file specifies how Pulsar should use the parser you created.
## Basic fields
It starts with some required fields:
```coffee
name: 'My Language'
scopeName: 'mylanguage'
type: 'tree-sitter'
parser: 'tree-sitter-mylanguage'
```
- `scopeName` - A unique, stable identifier for the language. Pulsar users will
use this in configuration files if they want to specify custom configuration based on the language.
- `name` - A human readable name for the language.
- `parser` - The name of the parser node module that will be used for parsing.
This string will be passed directly to [`require()`](https://nodejs.org/api/modules.html#modules_require)
in order to load the parser.
- `type` - This should have the value `tree-sitter` to indicate to Pulsar that
this is a Tree-sitter grammar and not a [TextMate grammar](../creating-a-grammar-textmate/).
## Language recognition
Next, the file should contain some fields that indicate to Pulsar _when_ this
language should be used. These fields are all optional.
- `fileTypes` - An array of filename _suffixes_. The grammar will be used for
files whose names end with one of these suffixes. Note that the suffix may be
an entire filename.
- `firstLineRegex` - A regex pattern that will be tested against the first line
of the file. The grammar will be used if this regex matches.
- `contentRegex` - A regex pattern that will be tested against the contents of
the file in order to break ties in cases where _multiple_ grammars matched the
file using the above two criteria. If the `contentRegex` matches, this grammar
will be preferred over another grammar with no `contentRegex`. If the
`contentRegex` does _not_ match, a grammar with no `contentRegex` will be
preferred over this one.
## Syntax highlighting
The HTML classes that Pulsar uses for syntax highlighting do not correspond
directly to nodes in the syntax tree. Instead, Tree-sitter grammar files specify
_scope mappings_ that specify which classes should be applied to which syntax
nodes. The `scopes` object controls these scope mappings. Its keys are CSS
selectors that select nodes in the syntax tree. Its values can be of several
different types.
Here is a simple example:
```coffee
scopes:
'call_expression > identifier': 'entity.name.function'
```
This entry means that, in the syntax tree, any `identifier` node whose parent is
a `call_expression` should be highlighted using three classes: `syntax--entity`,
`syntax--name`, and `syntax--function`.
Note that in this selector, we’re using the [immediate child combinator](https://developer.mozilla.org/en-US/docs/Web/CSS/Child_selectors)
(`>`). Arbitrary descendant selectors without this combinator (for example
`'call_expression identifier'`, which would match any `identifier` occurring
anywhere within a `call_expression`) are currently not supported.
### Advanced selectors
The keys of the `scopes` object can also contain _multiple_ CSS selectors,
separated by commas, similar to CSS files. The triple-quote syntax in CSON makes
it convenient to write keys like this on multiple lines:
```coffee
scopes:
'''
function_declaration > identifier,
call_expression > identifier,
call_expression > field_expression > field_identifier
''': 'entity.name.function'
```
You can use the [`:nth-child` pseudo-class](https://developer.mozilla.org/en-US/docs/Web/CSS/:nth-child)
to select nodes based on their order within their parent. For example, this
example selects `identifier` nodes which are the fourth (zero-indexed) child of
a `singleton_method` node.
```coffee
scopes:
'singleton_method > identifier:nth-child(3)': 'entity.name.function'
```
Finally, you can use double-quoted strings in the selectors to select
_anonymous_ tokens in the syntax tree, like `(` and `:`. See
[the Tree-sitter documentation](http://tree-sitter.github.io/tree-sitter/using-parsers#named-vs-anonymous-nodes)
for more information about named vs anonymous tokens.
```coffee
scopes:
'''
"*",
"/",
"+",
"-"
''': 'keyword.operator'
```
### Text-based mappings
You can also apply different classes to a syntax node based on its text. Here
are some examples:
```coffee
scopes:
# Apply the classes `syntax--builtin` and `syntax--variable` to all
# `identifier` nodes whose text is `require`.
'identifier': {exact: 'require', scopes: 'builtin.variable'},
# Apply the classes `syntax--type` and `syntax--integer` to all
# `primitive_type` nodes whose text starts with `int` or `uint`.
'primitive_type': {match: /^u?int/, scopes: 'type.integer'},
# Apply the classes `syntax--builtin`, `syntax--class`, and
# `syntax--name` to `constant` nodes with the text `Array`,
# `Hash` and `String`. For all other `constant` nodes, just
# apply the classes `syntax--class` and `syntax--name`.
'constant': [
{match: '^(Array|Hash|String)$', scopes: 'builtin.class.name'},
'class.name'
]
```
In total there are four types of values that can be associated with selectors in
`scopes`:
- Strings - Each class name in the dot-separated string will be prefixed with
`syntax--` and applied to the selected node.
- Objects with the keys `exact` and `scopes` - If the node’s text equals the
`exact` string, the `scopes` string will be used as described above.
- Objects with the keys `match` and `scopes` - If the node’s text matches the
`match` regex pattern, the `scopes` string will be used as described above.
- Arrays - The elements of the array will be processed from beginning to end.
The first element that matches the selected node will be used as describe
above.
### Specificity
If multiple selectors in the `scopes` object match a node, the node’s classes
will be decided based on the [most specific](https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity)
selector. Note that the `exact` and `match` rules do _not_ affect specificity,
so you may need to supply the same `exact` or `match` rules for multiple
selectors to ensure that they take precedence over other selectors. You can use
the same selector multiple times in a scope mapping, within different
comma-separated keys:
```coffee
scopes:
'call_expression > identifier': 'entity.name.function'
# If we did not include the second selector here, then this rule
# would not apply to identifiers inside of call_expressions,
# because the selector `call_expression > identifier` is more
# specific than the selector `identifier`.
'identifier, call_expression > identifier': [
{exact: 'require', scopes: 'builtin.variable'},
{match: '^[A-Z]', scopes: 'constructor'},
]
```
## Language injection
Sometimes, a source file can contain code written in several different
languages. Tree-sitter grammars support this situation using a two-part process
called _language injection_. First, an 'outer' language must define an
_injection point_ - a set of syntax nodes whose text can be parsed using a
different language, along with some logic for guessing the _name_ of the other
language that should be used. Second, an 'inner' language must define an
`injectionRegex` - a regex pattern that will be tested against the language name
provided by the injection point.
For example, in JavaScript, [tagged template literals](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#Tagged_templates)
sometimes contain code written in a different language, and the name of the
language is often used in the 'tag' function, as shown in this example:
```js
// HTML in a template literal
const htmlContent = html`Hello ${name}`;
```
The `tree-sitter-javascript` parser parses this tagged template literal as a
`call_expression` with two children: an `identifier` and a `template_literal`:
```
(call_expression
(identifier)
(template_literal
(interpolation
(identifier))))
```
Here is an injection point that would allow syntax highlighting inside of
template literals:
```js
atom.grammars.addInjectionPoint("source.js", {
type: "call_expression",
language(callExpression) {
const { firstChild } = callExpression;
if (firstChild.type === "identifier") {
return firstChild.text;
}
},
content(callExpression) {
const { lastChild } = callExpression;
if (lastChild.type === "template_string") {
return lastChild;
}
},
});
```
The `language` callback would then be called with every `call_expression` node
in the syntax tree. In the example above, it would retrieve the first child of
the `call_expression`, which is an `identifier` with the name "html". The
callback would then return the string "html".
The `content` callback would then be called with the same `call_expression` node
and return the `template_string` node within the `call_expression` node.
In order to parse the HTML within the template string, the HTML grammar file
would need to specify an `injectionRegex`:
```coffee
injectionRegex: 'html|HTML'
```
## Code folding
The next field in the grammar file, `folds`, controls code folding. Its value is
an array of _fold pattern_ objects. Fold patterns are used to decide whether or
not a syntax node can be folded, and if so, where the fold should start and end.
Here are some example fold patterns:
```coffee
folds: [
# All `comment` nodes are foldable. By default, the fold starts at
# the end of the node's first line, and ends at the beginning
# of the node's last line.
{
type: 'comment'
}
# `if_statement` nodes are foldable if they contain an anonymous
# "then" token and either an `elif_clause` or `else_clause` node.
# The fold starts at the end of the "then" token and ends at the
# `elif_clause` or `else_clause`.
{
type: 'if_statement',
start: {type: '"then"'}
end: {type: ['elif_clause', 'else_clause']}
}
# Any node that starts with an anonymous "(" token and ends with
# an anonymous ")" token is foldable. The fold starts after the
# "(" and ends before the ")".
{
start: {type: '"("', index: 0},
end: {type: '")"', index: -1}
}
]
```
Fold patterns can have one or more of the following fields:
- `type` - A string or array of strings. In order to be foldable according to
this pattern, a syntax node’s type must match one of these strings.
- `start` - An object that is used to identify a _child_ node after which the
fold should start. The object can have one or both of the following fields:
- `type` - A string or array of strings. To start a fold, a child node’s type
must match one of these strings.
- `index` - a number that’s used to select a specific child according to its
index. Negative values are interpreted as indices relative the last child,
so that `-1` means the last child.
- `end` - An object that is used to identify a _child_ node before which the
fold should end. It has the same structure as the `start` object.
## Comments
The last field in the grammar file, `comments`, controls the behavior of
Pulsar’s **Editor: Toggle Line Comments** command. Its value is an object with a
`start` field and an optional `end` field. The start field is a string that
should be prepended to or removed from lines in order to comment or uncomment
them.
In JavaScript, it looks like this:
```coffee
comments:
start: '// '
```
The `end` field should be used for languages that only support block comments, not line comments. If present, it will be appended to or removed from the end of the last selected line in order to comment or un-comment the selection.
In CSS, it would look like this:
```coffee
comments:
start: '/* '
end: ' */'
```
## Example Packages
More examples of all of these features can be found in the Tree-sitter grammars bundled with Pulsar:
- [Bash](https://github.com/pulsar-edit/pulsar/tree/HEAD/packages/language-shellscript)
- [C](https://github.com/pulsar-edit/pulsar/tree/HEAD/packages/language-c)
- [Go](https://github.com/pulsar-edit/pulsar/tree/HEAD/packages/language-go)
- [HTML](https://github.com/pulsar-edit/pulsar/tree/HEAD/packages/language-html)
- [JavaScript](https://github.com/pulsar-edit/pulsar/tree/HEAD/packages/language-javascript)
- [Python](https://github.com/pulsar-edit/pulsar/tree/HEAD/packages/language-python)
- [Ruby](https://github.com/pulsar-edit/pulsar/tree/HEAD/packages/language-ruby)
- [TypeScript](https://github.com/pulsar-edit/pulsar/tree/HEAD/packages/language-typescript)
---
## File: docs/developing-for-pulsar/creating-a-grammar-modern-tree-sitter.md
---
title: Creating a grammar (modern Tree-sitter)
layout: doc.ejs
---
Pulsar’s modern syntax highlighting and code folding system is powered by [Tree-sitter](http://tree-sitter.github.io/tree-sitter). Tree-sitter parsers create and maintain full [syntax trees](https://en.wikipedia.org/wiki/Abstract_syntax_tree) representing your code.
Modeling the buffer as a syntax tree gives Pulsar a comprehensive understanding of the structure of your code, which has several benefits:
1. Syntax highlighting will not break because of formatting changes.
2. Code folding will work regardless of how your code is indented.
3. Editor features can operate on the syntax tree. For instance, the **Select Larger Syntax Node** and **Select Smaller Syntax Node** commands allow you to select conceptually larger and smaller chunks of your code.
4. Community packages can use the syntax tree to understand and manipulate code more intelligently.
## Getting started
There are three components required to use Tree-sitter in Pulsar: a parser, a grammar file, and a handful of query files.
:::info
In most of this example code, we’ll assume you’re creating a grammar for a fictional language creatively titled MyLanguage. Your package will thus be named `language-mylanguage`, and we assume the existence of a project called `tree-sitter-mylanguage` that implements a Tree-sitter parser for MyLanguage.
:::
## The parser
Tree-sitter generates parsers based on [context-free grammars](https://en.wikipedia.org/wiki/Context-free_grammar) that are typically written in JavaScript. The generated parsers are C libraries that can be used in other applications as well as Pulsar.
They can also be developed and tested on the command line, separately from Pulsar. Tree-sitter has [its own documentation page](http://tree-sitter.github.io/tree-sitter/creating-parsers) on how to create these parsers. The [Tree-sitter GitHub organization](https://github.com/tree-sitter) also contains a lot of example parsers that you can learn from, each in its own repository.
Pulsar uses `web-tree-sitter` — the WebAssembly bindings to tree-sitter. That means that you’ll have to build a WASM file for your parser before it can be used.
:::warning Tree-sitter versions are important
Currently, Pulsar builds its own custom copy of `web-tree-sitter` for reasons explained [in this README](https://github.com/pulsar-edit/pulsar/tree/master/vendor/web-tree-sitter). In the future, to know which version of `web-tree-sitter` Pulsar is using, please visit that folder on GitHub and read the commit message of the commit that most recently touched `tree-sitter.js`.
You do not have to use the exact same version of Tree-sitter to _build_ your parser as the version of Tree-sitter that will _consume_ it. In general, though, it’s a good idea not to allow _too much_ variance between Tree-sitter versions, as it introduces a new uncertainty into the mix.
At time of writing this document, Pulsar was using `web-tree-sitter` version 0.25.3, which means you may want to generate your WASM file using an `0.25.x` version of `web-tree-sitter`.
Building WASM files from the `tree-sitter` CLI requires either a local installation of [Emscripten](https://emscripten.org/) or use of a Docker image. See [this reference](https://github.com/sogaiu/ts-questions/blob/master/questions/which-version-of-emscripten-should-be-used-for-the-playground/README.md) for details.
:::
If you want to use an existing parser, you’ll probably be able to find it [in this list](https://github.com/tree-sitter/tree-sitter/wiki/List-of-parsers) on the Tree-sitter wiki. (If you’ve written your own parser, it’s a good idea to reach out to the Tree-sitter maintainers and get your parser added to this wiki page.)
Because the WASM file is self-contained, the source code for the Tree-sitter parser _should not_ be part of your package, since it’d just increase the file size of the package for no good reason.
You can then go into the directory for your parser and use [the Tree-sitter CLI](https://github.com/tree-sitter/tree-sitter-cli) to build the WASM file:
```bash
npm install
cd node_modules/tree-sitter-mylanguage
npx tree-sitter build --wasm
```
This command will produce a `tree-sitter-mylanguage.wasm` file in the working directory; you should add it to your `language-mylanguage` project.
## The package
Once you have a WASM file, you can use it in your Pulsar package. Packages with grammars are, by convention, always named starting with `language-`. You’ll need a folder with a `package.json`, a `grammars` subdirectory, and a single JSON or CSON file in the `grammars` directory, which can be named anything.
We’ve also decided to put our WASM file in the `grammars/tree-sitter` subdirectory, though this is just a convention. The SCM files alongside our WASM file will be explained in a moment.
```
language-mylanguage
├── LICENSE
├── README.md
├── grammars
│ ├── mylanguage.cson
│ └── tree-sitter
│ ├── grammar.wasm
| ├── folds.scm
| ├── highlights.scm
| ├── indents.scm
| └── tags.scm
└── package.json
```
## The grammar file
The `mylanguage.cson` file specifies how Pulsar should use the parser you created.
### Basic fields
It starts with some required fields:
```coffeescript
name: 'My Language'
scopeName: 'source.mylanguage'
type: 'modern-tree-sitter'
parser: 'tree-sitter-mylanguage'
```
* `scopeName` - A unique, stable identifier for the language. Pulsar users will use this identifier in configuration files if they want to specify custom configuration based on the language. Examples: `source.js`, `text.html.basic`. It will also be the root scope name used in syntax highlighing.
* `name` - A human-readable name for the language.
* `parser` - The name of the parser that will be used for parsing. In theory, this should point to an NPM package from which the WASM file was built; in practice it can just use the name of the project that contains the lanugage’s Tree-sitter parser, whether or not it exists on NPM. (This value is currently **unused**, but is required as a way of future-proofing in case Pulsar should migrate to a different Tree-sitter binding in the future.)
* `type` - This should have the value `modern-tree-sitter` to indicate to Pulsar that this is a modern Tree-sitter grammar, as opposed to a legacy Tree-sitter grammar (soon to be removed from Pulsar) or a [TextMate grammar](https://pulsar-edit.dev/docs/launch-manual/sections/core-hacking/#creating-a-legacy-textmate-grammar).
### Tree-sitter fields
The `treeSitter` configuration key holds the fields that specify the paths on disk to the grammar and its query files:
```coffeescript
treeSitter:
grammar: 'tree-sitter/grammar.wasm'
highlightsQuery: 'tree-sitter/highlights.scm'
foldsQuery: 'tree-sitter/folds.scm'
indentsQuery: 'tree-sitter/indents.scm'
```
All values are paths that will be resolved relative to the directory in which the grammar configuration file itself lives. Of these, `grammar` is the only required field.
* `grammar` — The relative path to the WASM file you generated earlier.
* `highlightsQuery` — The relative path to a file (canonically called `highlights.scm`) that will tell Pulsar how to highlight the code in this language. (Most Tree-sitter repositories include a `highlights.scm` file that can be useful to consult, but _should not_ be used in Pulsar, because its naming conventions are different from Pulsar’s.)
* `foldsQuery` — The relative path to a file (canonically called `folds.scm`) that will tell Pulsar which ranges of a buffer can be folded.
* `indentsQuery` — The relative path to a file (canonically called `indents.scm`) that will tell Pulsar when it should indent or dedent lines of code in this language.
* `tagsQuery` — The relative path to a file (canonically called `tags.scm`) that will identify the important symbols in the document (class names, function names, and so on) along with their locations. If present, Pulsar will use this query file for symbol navigation. (Most Tree-sitter repositories include a `tags.scm` file that _can be understood as-is_ by Pulsar and is a good starting point.)
Each of the settings ending in `Query` is optional. You can skip `indentsQuery` if your language doesn’t need indentation hinting, `foldsQuery` if it doesn’t need code folding, or even `highlightsQuery` in the unlikely event that your language does not need syntax highlighting.
Any of the settings that end in `Query` can also accept an array of relative paths, instead of just a single relative path. At initialization time, the grammar will concatenate each file’s contents into a single query file. This isn’t a common need, but is explained further below in the [Sharing query files](#sharing-query-files) section.
### Language recognition
Next, the file should contain some fields that indicate to Pulsar when this language should be chosen for a given file. These fields are all optional and are listed in the order that Pulsar consults them when making its decision.
* `fileTypes` - An array of filename suffixes. The grammar will be used for files whose names end with one of these suffixes. Note that the suffix may be an entire filename, like `Makefile` or `.eslintrc`. If no grammars match (or more than one grammar matches) for a given file extension, ties are broken according to…
* `firstLineRegex` - A regex pattern (written as a string) that will be tested against the first line of the file. The grammar will be used if this regex matches. If no grammars match (or more than one grammar matches) for a given `firstLineRegex`, ties are broken according to…
* `contentRegex` - A regex pattern (written as a string) that will be tested against the contents of the file. If the `contentRegex` matches, this grammar will be preferred over another grammar with no `contentRegex`. If the `contentRegex` does not match, a grammar with no `contentRegex` will be preferred over this one.
### Comments
The last field in the grammar file, `comments`, tells Pulsar what sort of comment syntax this language uses. This helps the editor deal with comments for purposes such as the **Editor: Toggle Line Comments** command and for snippets that use magic comment-delimiter variables so that the snippet works equally well across more than one language.
Its value is an object with values as follows:
* `start`: The delimiter that should be added to the beginning of a line to mark a comment. If your language supports line comments, specify the line comment delimiter here and skip the `end` value. This value will be used by the **Editor: Toggle Line Comments** command.
* `end`: The delimiter that should be added to the end of a line to mark a comment. Specify `end` _only_ if your language supports only block comments (for example, CSS). If present, this value will be used by the **Editor: Toggle Line Comments** command.
* `line`: The delimiter that marks a line comment. Regardless of what is defined in `start` or `end`, `line` refers to the line comment delimiter. If your language doesn’t support line comments, omit this field. This value is used by snippets that want to insert comment delimiters in a language-agnostic fashion.
* `block`: A two item array containing the starting and ending delimiters of a block comment. If your language doesn’t support block comments, omit this field. These values are used by snippets that want to insert comment delimiters in a language-agnostic fashion.
:::note
This may seem redundant. It’s because the **Editor: Toggle Line Comments** command uses _either_ a line comment _or_ a block comment to comment out a line, but expects you to define only one kind as needed.
On the other hand, the `snippets` package needs to know about _both_ kinds of comments if your language supports more than one type.
:::
JavaScript has both line and block comments, so the `comments` field might look like this:
```coffeescript
comments:
start: '// '
block: ['/*', '*/']
line: '//'
```
CSS has only block comments, so it’d look like this:
```coffeescript
comments:
start: '/* '
end: ' */'
block: ['/*', '*/']
```
And Python has only line comments, so it’d look like this:
```coffeescript
comments:
start: '# ',
line: '#'
```
:::tip
When present, Pulsar prefers for these values to be specified in CSON files in the `settings` directory of your package. That approach has the advantage of allowing you to customize comment delimiters according to the surroundings of the file.
If this isn’t something your language needs — and most don’t — then you can ignore this tip. But it’s occasionally crucial. For instance, comment syntax within JSX blocks (used in React and other framewokrs) is different from ordinary JavaScript comment syntax. [The settings file for `language-javascript`](https://github.com/pulsar-edit/pulsar/blob/master/packages/language-javascript/settings/language-javascript.cson) may help illustrate.
When looking for settings instead of grammar metadata, **Editor: Toggle Line Comments** looks for the `commentStart` and/or `commentEnd` keys, whereas the `snippets` package looks for the `commentDelimiters` key.
:::
## Syntax highlighting
Tree-sitter analyzes your source code and turns it into a syntax tree in which each node has a descriptive name. However, HTML classes that Pulsar uses for syntax highlighting do not correspond directly to nodes in the syntax tree.
Instead, Pulsar queries the tree using a file called `highlights.scm` and written using Tree-sitter’s own [query language](https://tree-sitter.github.io/tree-sitter/using-parsers#pattern-matching-with-queries).
Here is a simple example:
```scm
(call_expression
(identifier) @support.other.function)
```
This entry means that, in the syntax tree, any `identifier` node whose parent is a `call_expression` should be given the scope name `support.other.function`. In the editor, such an identifier will be wrapped in a `span` tag with three classes applied to it: `syntax--support`, `syntax--other`, and `syntax--function`. Syntax themes can hook into these class names to [style source code](../developing-a-theme) via CSS or LESS files.
Some queries will be quite easy to express, but some others will be highly contextual. Consult some built-in grammars’ `highlights.scm` files for examples.
### Scope tests: advanced querying
Tree-sitter supports additional matching criteria for queries called _predicates_. For instance, here’s how we can distinguish a block comment from a line comment in JavaScript:
```scm
((comment) @comment.block.js
(#match? @comment.block.js "^/\\*"))
((comment) @comment.line.js
(#match? @comment.line.js "^//"))
```
We’re using the built-in `#match?` predicate, along with a regular expression, to search the text within the `comment` node. Our regexes are anchored to the beginning of the string and test whether the opening delimiter signifies a block comment (`/* like this */`) or a line comment (`// like this`). In the block comment’s case, we don’t have to attempt to match the ending (`*/`) delimiter — we _know_ it must be present, or else the Tree-sitter parser wouldn’t have classified it as a `comment` node in the first place.
Unfortunately, there aren’t many built-in predicates in `web-tree-sitter` alongside `#match?` and `#eq?` (which tests for _exact_ equality). But the ones that are present — `#set!`, `#is?`, and `#is-not?` — allow us to associate arbitrary key/value pairs with a specific capture. Pulsar uses these to define its own _custom_ predicates.
For instance, you may want to highlight things differently based on their position among siblings:
```scm
(string
"\"" @punctuation.definition.string.begin.js
(#is? test.first))
(string
"\"" @punctuation.definition.string.end.js
(#is? test.last))
```
In most Tree-sitter languages, a `string` node’s first child will be its opening delimiter, and its last child will be its closing delimiter. To add two different scopes to these quotation marks, we can use the `test.first` and `test.last` custom predicates to distinguish these two nodes from one another.
### Prioritizing scopes
It’s common to want to add one scope to something if it passes a certain test, but a different scope if it fails the test.
```scm
; Scope this like a built-in function if we recognize the name…
(call_expression (identifier) @support.function.builtin.js
(#match? @support.function.builtin.js "^(isFinite|isNaN|parseFloat|parseInt)$"))
; …or as a user-defined function if we don't.
(call_expression (identifier) @support.other.function.js)
```
This doesn’t do what you might expect because a given buffer range can have any number of scopes applied to it. That means the "parseFloat" in `parseFloat(foo)` will be given _both_ of these scope names, since it matches both of these captures.
How can we get around this? One option is the `#not-match?` predicate — the negation of `#match?` — to ensure that anything which passes the first test will fail the second, and vice versa:
```scm
; Scope this like a built-in function if we recognize the name…
(call_expression (identifier) @support.function.builtin.js
(#match? @support.function.builtin.js "^(isFinite|isNaN|parseFloat|parseInt)$"))
; …or as a user-defined function if we don't.
(call_expression (identifier) @support.other.function.js
(#not-match? @support.function.builtin.js "^(isFinite|isNaN|parseFloat|parseInt)$"))
```
This is a fine solution for our oversimplified example, but would get pretty complicated if there were more than one fallback.
Another approach is to use Pulsar’s custom predicates called `capture.final` and `capture.shy`. For instance, we could use `capture.final` on the first capture to “claim” it:
```scm
; Scope this like a built-in function if we recognize the name…
(call_expression (identifier) @support.function.builtin.js
(#match? @support.function.builtin.js "^(isFinite|isNaN|parseFloat|parseInt)$")
(#set! capture.final))
; …or as a user-defined function if we don't.
(call_expression (identifier) @support.other.function.js)
```
The `capture.final` predicate means that the first capture will apply its own scope name, then _prevent_ all further attempts to add a scope to the same buffer range. This works because two different captures for the same node will be processed in the order in which their queries are defined in the SCM file — so if a token were to match both of these captures, it’s guaranteed that the first capture would be processed before the second.
Another option would be to use `capture.shy` on the second capture:
```scm
; Scope this like a built-in function if we recognize the name…
(call_expression (identifier) @support.function.builtin.js
(#match? @support.function.builtin.js "^(isFinite|isNaN|parseFloat|parseInt)$"))
; …or as a user-defined function if we don't.
(call_expression (identifier) @support.other.function.js
(#set! capture.shy))
```
The `capture.shy` predicate creates a true fallback option; it only applies its scope if _no_ other scope — not just one that uses `capture.final` — has previously been applied for the same buffer range. But it doesn’t “lock down” its buffer range the way that `capture.final` does, so a later capture could add another scope to the same range.
There’s one caveat to mention. Consider this query:
```scm
(call_expression (identifier) @support.other.function.js @meta.something-else.js
(#set! capture.final))
```
This is a valid Tree-sitter query; you can assign up to three capture names at once. But the outcome might be surprising: `support.other.function.js` will be applied, and `meta.something-else.js` will not. This happens because these two capture names aren’t processed simultaneously; they’re processed in sequence. So the `capture.final` predicate will act after the first capture name and prevent the second from being applied.
Here’s one way to rewrite this to have the intended effect:
```scm
; Apply each capture in its own query…
(call_expression (identifier) @support.other.function.js)
; …and use capture.final only on the second capture name.
(call_expression (identifier) @meta.something-else.js
(#set! capture.final))
```
### Other scope tests
Pulsar defines many custom predicates, otherwise known as **scope tests**, to help grammar authors define accurate syntax highlighting. All scope tests are prefaced with `test.` in `#is?` and `#is-not?` predicates.
You’ve already seen an example with `test.first` and `test.last`. Other examples include:
* `test.firstOfType` and `test.lastOfType`, for matching only the first or last node of a certain type among siblings
* `test.ancestorOfType` and `test.descendantOfType`, for testing whether a node contains, or is contained by, a node of a certain type
* `test.config`, for capturing certain nodes conditionally based on the user’s configuration
Some tests take a single argument…
```scm
((identifier) @foo
(#is? test.descendantOfType "function"))
```
…but any test that doesn’t requre an argument can be expressed without one.
```scm
((identifier) @foo
(#is? test.lastOfType))
```
Consult the [`ScopeResolver` API documentation](https://github.com/pulsar-edit/pulsar/blob/df0050f7b2f47448061895d14269ff97c4bae29f/src/scope-resolver.js#L608-L632) for a full list.
### Scope adjustments: tweaking buffer ranges
There may be times when the range you want to highlight doesn’t correspond exactly with the range of a single node in the syntax tree. For those situations, you can use **scope adjustments** to tweak the range:
```scm
; Scope the `//` in a line comment as punctuation.
((comment) @punctuation.definition.comment.js
(#match? @punctuation.definition.comment.js "^//")
(#set! adjust.startAndEndAroundFirstMatchOf "^//"))
```
Some adjustments move the boundaries of the scope based on pattern matches inside a node’s text, like in the example above. Others may move the boundaries based on a _node position descriptor_ — a string like `lastChild.startPosition` that points to a specific position in a tree relative to another node — so they can wrap a single scope around two or three adjacent sibling nodes.
There’s only one catch: **adjustments can only _narrow_ the range of a capture, not expand it**. That’s important because Pulsar depends on Tree-sitter to tell it when certain regions of the buffer are affected by edits and need to be re-highlighted. That system won’t work correctly if a capture that starts and ends on row 1 can stretch itself to add a scope to something on row 100.
Consult the [`ScopeResolver` API documentation](https://github.com/pulsar-edit/pulsar/blob/df0050f7b2f47448061895d14269ff97c4bae29f/src/scope-resolver.js#L608-L632) for a full list.
### Sharing query files
You may find it appropriate for two different grammars to share a query file. This can work when one Tree-sitter parser builds upon the work of another; for instance, the `tree-sitter-tsx` parser is basically the `tree-sitter-typescript` parser with JSX additions, so it makes sense for the two grammars to share most of their query files.
For this reason, each of the fields that ends in `Query` in a grammar definition file can accept an array of paths instead of a single path. Consider a hypothetical grammar for TypeScript-plus-JSX:
```coffeescript
treeSitter:
grammar: 'tree-sitter-tsx/grammar.wasm'
languageSegment: 'ts.tsx'
highlightsQuery: [
'highlights-common.scm'
'tree-sitter-tsx/highlights.scm'
]
foldsQuery: 'tree-sitter-tsx/folds.scm'
indentsQuery: 'tree-sitter-tsx/indents.scm'
```
The highlights query loads two different files: one that is common to `tree-sitter-typescript` and `tree-sitter-tsx`, and one that is unique to `tree-sitter-tsx`. The latter file would contain queries that deal with JSX and anything else that would not be understood by a `tree-sitter-typescript` parser.
You might also notice a new key: `languageSegment`. This optional property allows one to write a shared query file generically…
```scm
(class_declaration
name: (type_identifier) @entity.name.type.class._LANG_)
```
…while retaining the ability to add a grammar-specific scope segment at the end of a capture. At initialization time, all `_LANG_` segments in this SCM file would be dynamically replaced with `ts.tsx`, and the capture name above would become `@entity.name.type.class.ts.tsx`. In the ordinary TypeScript grammar, specifying a `languageSegment` of `ts` would allow that grammar to define a capture name of `@entity.name.type.class.ts` instead.
## Language injection
Often, a source file will contain code written in several different languages. An HTML file, for instance, may need to highlight JavaScript (if the file has an inline `script` element) or CSS (if the file has an inline `style` element).
Tree-sitter grammars support this situation using a two-part process called _language injection_. First, an “outer” language must define an injection point - a set of syntax nodes whose text can be parsed using a different language, along with some logic for guessing the name of the other language that should be used. Second, an “inner” language must define an `injectionRegex` - a regex pattern that will be tested against the language name provided by the injection point.
This allows for loose coupling. One grammar can say “these sorts of nodes need a parser that understands `html`”; another can say “hey, I know what `html` is! I can be that grammar!”
The inner language can, in turn, define any injection points _it_ may need, such that different grammars can be “nested” inside of a buffer to an arbitrary depth.
If your language package needs to inject other languages, it can define injection points in JavaScript via the {GrammarRegistry::addInjectionPoint "atom.grammars.addInjectionPoint()"} API.
For instance, you could define `lib/main.js` within your `language-mylanguage` package. That file should export a function called `activate` that defines the injection points:
```js
exports.activate = () => {
atom.grammars.addInjectionPoint(/* … */);
};
```
Be sure to include a corresponding `main` field in the package’s `package.json` that points to this file:
```json
"main": "lib/main"
```
### Using `addInjectionPoint`
In JavaScript, [tagged template literals](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals#Tagged_templates) sometimes contain code written in a different language, and the tag’s name tends to hint at the language being used inside the template string:
```js
// HTML in a template literal
const htmlContent = html`Hello, ${name}`;
// CSS in a template literal
const styles = styled.a`
border: 2px solid #000;
color: #fff;
`
```
The `tree-sitter-javascript` parser parses the first tagged template literal as a `call_expression` with two children: an `identifier` and a `template_literal`:
```scm
(call_expression
function: (identifier)
arguments: (template_string
(template_substitution
(identifier))))
```
So here’s how we might allow syntax highlighting inside of template literals:
```js
atom.grammars.addInjectionPoint("source.js", {
type: "call_expression",
language(callExpression) {
const { firstChild, lastChild } = callExpression;
if (firstChild?.type === "identifier" && lastChild?.type === "template_string") {
return firstChild.text;
}
},
content(callExpression) {
return callExpression?.lastChild;
}
});
```
So what happens when we use an `html` tagged template literal, as in the first example above?
1. Every `call_expression` node in the tree would be assessed as a possible candidate for injection.
2. Each of those nodes would be passed into our `language` callback to see if it can be matched to an injection language. In our example, first we’d inspect the tree to make sure this is a tagged template literal; then we’d return the text of the `identifier` node — `html` in this example. If that string can be matched with a known grammar, the injection can proceed.
3. The `content` callback would then be called with the same `call_expression` node and return the last child of the `call_expression` node — which we’ve already proven is of the type `template_string` — since that node precisely describes the content that should be parsed as HTML. That node exists in our example, so the injection can proceed.
We skipped something important in step 2: _how_ do we turn the string `html` into the HTML grammar? The HTML grammar file would need to specify an `injectionRegex`, so that the string `html` returned from the `language` callback can match itself to the right grammar:
```coffeescript
injectionRegex: 'html|HTML'
```
If more than one grammar’s `injectionRegex` matches the string in question, Pulsar will pick the grammar whose `injectionRegex` produced the longest string match.
When defining your own `injectionRegex`, consider how specific you want your pattern to be. In our example, it’s a safe bet that any template literal tag or heredoc string delimiter that even _contains_ the string `HTML` is describing content that should be highlighted like HTML. But a language like C, for example, might want to define a much more restrictive pattern that won’t get matched for an identifier like `coffeescript` or `c-sharp`:
```coffeescript
injectionRegex: '^(c|C)$'
```
The `injectionRegex` property is only required for grammars that expect to be injected into other grammars. If this doesn’t apply to your grammar, you can omit `injectionRegex` altogether.
### Advanced injection scenarios
Each individual injection point understands that it might have to operate on disjoint ranges of the buffer, ignoring certain ranges of content in between. Let’s look at our example code again:
```js
// HTML in a template literal
const htmlContent = html`Hello, ${name}`;
```
Here, `${name}` is a JavaScript template string interpolation, and it has no special meaning in HTML. But an interpolation could include arbitrary JavaScript, including content that would flummox a parser designed to interpret HTML.
It’s for this reason that, by default, injection points ignore the _descendants_ of their content nodes. When an injection determines which ranges in the buffer it should parse, it takes the ranges described by its content nodes and _subtracts_ the ranges of their children, unless instructed otherwise. So our injection will cover the range of the `template_string` _minus_ the ranges of any interpolations (the `template_substitution` child nodes).
When the injection content is parsed, Tree-sitter will look at only the ranges Pulsar tells it to; anything outside those ranges will be invisible to the parser.
So our example above would look like this to the HTML grammar:
```html
Hello,
```
Ignoring child nodes is the correct decision for scenarios like tagged template literals and heredoc strings, but it might not be the correct decision for other injections, so this behavior is configurable.
Here are some other things you can do with injections, if needed:
* block the parent grammar from highlighting anything in the injection’s content ranges
* include the injected language’s base scope name (`text.html.basic` in our case), include a different base scope name instead, or omit it altogether
* consolidate ranges that are separated only by whitespace
* include newline characters when they appear between disjoint content ranges so that the injection’s parser doesn’t think those ranges are part of the same line
For more information on these features, read the API documentation for {GrammarRegistry::addInjectionPoint}.
## Code folding
[Code folding](/core-packages-and-features/folding/) can only happen if the grammar helps Pulsar to understand which ranges of the buffer represent logical sections that can be collapsed. A Tree-sitter grammar does this via `folds.scm` — a query file whose only purpose is to mark “foldable” sections of the buffer.
### Simple folds
The complexity of a `folds.scm` will vary based on the language. At its simplest, it will look like the following:
```scm
(block) @fold
```
Believe it or not, that’s the _entire contents_ of the `folds.scm` file inside the `language-css` package. Because CSS’s syntax is very regular, the `block` node can handle all situations where content is enclosed in a pair of curly braces.
The `@fold` capture is called a **simple fold**. It’s the easiest kind of fold to describe because Tree-sitter has done most of the work simply by identifying the regions to be folded. By default, here’s how Pulsar turns that capture into a foldable range:
1. It will inspect the `block` node to find out the buffer rows it starts and ends on. If the `block` node starts on row `X`, the fold will begin at the very end of row `X`.
2. It assumes the very last child of `block` is its closing delimiter (because that’s usually true) and sets the ending boundary of the fold to be just _before_ that closing delimiter, so that both the opening and closing delimiters are visible when the range is folded.
3. If the two ends of the fold are on different rows, the fold is valid, and will be indicated by a chevron in the gutter of row `X`. If the would-be fold range starts and ends on the _same_ row, the fold is invalid and therefore ignored.
Sometimes simple folds need tweaking — for instance, in the case of a multi-line `if/else-if/else` construction. And in some languages, like Python, there _aren’t_ any ending delimiters, so this logic won’t work out of the box.
That’s why Pulsar lets you customize the ends of simple folds — for instance, by specifying a _different_ ending position in the tree relative to the starting node. Or by altering a position an arbitrary amount — nudging it a few characters in either direction, or moving it to the end of the previous line.
Consult the API documentation for more information.
### Divided folds
There’s a different kind of fold, called a _divided fold_, that can be used when simple folds aren’t an option. They use the capture names `@fold.start` and `@fold.end` to specify the boundaries of the fold in two separate captures.
Divided folds are needed when the region to be folded isn’t represented by a single node, or by some predictable path from one node to another. Examples include preprocessor definitions in C/C++ files and sections inside Markdown files.
But the best example might be complex conditionals in shell scripts. In most other built-in Tree-sitter grammars, these conditionals can be handled with simple folds. But shell scripts are a bit “messy,” and the parsed tree reflects that:
```scm
(if_statement "then" @fold.start)
(elif_clause) @fold.end
(elif_clause "then" @fold.start)
(else_clause) @fold.end @fold.start
"fi" @fold.end
```
A `@fold.start` capture on a node means that a fold will start at the end of that node’s starting row. A `@fold.end` capture on a node means that a fold will end at the end of the row **before** that node’s starting row.
This behavior allows a given node to be captured with _both_ `@fold.start` _and_ `@fold.end`, as in the case of the `else_clause` above. If we see an `else` on row 10, it means that one fold has ended at the end of row 9, and another one will begin at the end of row 10.
Divided folds need to pair up, and Pulsar pairs them up by starting with a `@fold.start` capture and looking for a _balanced_ occurrence of `@fold.end`, keeping in mind that folds can be nested inside other folds.
There are good reasons to prefer simple folds wherever possible, and to use divided folds only when there isn’t another option. For one thing, it becomes the grammar author’s responsibility to ensure that `@fold.start` and `@fold.end` are captured in equal numbers, and that each `@fold.start` matches up with its intended `@fold.end`.
You can read more about folds in the API documentation.
## Indents
The third sort of query file is typically called `indents.scm`, and its purpose is to identify items in the tree that hint at indents or dedents.
To oversimplify, here’s how indentation typically works in Pulsar, regardless of which sort of grammar is used:
* If the user is typing on row 9, then presses Enter, we’ll decide whether to indent row 10 based on what’s present on line 9. For example, if row 9 ends with an opening curly brace (`{`), that’s a clear sign that row 10 should start with the cursor one level deeper than row 9. Therefore: **to decide whether to _indent_ a row, we usually examine the row above it.**
* When the user starts typing on row 10, we might decide that row 10 shouldn’t be indented after all. For instance, if the first character the user typed is a closing curly brace (`}`), then Pulsar will immediately decrease the indent level of that line by one level. Therefore: **to decide whether to _dedent_ a row, we usually examine the content of the row itself.**
In TextMate grammars, the decisions to indent and dedent are made by comparing the contents of lines to regular expressions. In Tree-sitter grammars, the decisions are made by through query captures — typically captures named `@indent` and `@dedent`. Indentation hinting is a two-phase process that corresponds to the bullet points above.
This is a good starting point for an `indents.scm` for a C-like language:
```scm
["{" "[" "("] @indent
["}" "]" ")"] @dedent
```
The fact that Tree-sitter grammars expose their delimiters in the tree as [anonymous nodes](http://tree-sitter.github.io/tree-sitter/using-parsers#named-vs-anonymous-nodes) makes it very easy to interpret indentation hints. You are encouraged to capture anonymous nodes in your `indents.scm` when possible — because (a) they’re usually the best signifiers of when indentation needs to happen, and (b) they tend to be present in a tree even when the tree is in an error state (like when the user is in the middle of typing a line).
Here’s how we’d use these queries to make indentation decisions:
* Starting with an empty JavaScript file, a user types `if (foo) {` and presses Enter. Pulsar runs an indent query on row 1, gets a match for `@indent`, and responds by increasing the indent level on row 2. Phase one has increased the indent by one level; since the line has no content, phase two does not alter the indentation level further.
* The user types a placeholder comment like `// TODO implement later`. While the user types, Pulsar checks to see if any of the new content on row 2 should affect its indentation level, but it doesn’t.
* The user presses Enter again. A query runs against row 2, finds no matches, and therefore decides that row 3 should maintain row 2’s indentation level of `1`.
* Finally, the user types `}`. After that keystroke, Pulsar runs an indent query on row 3, finds that the row now starts with a `@dedent` capture, and responds by dedenting row 3 by one level immediately. Phase one produced an indent delta of `0`, but phase two produced a delta of `-1`.
Thus you can see that `@indent` means “indent the next line,” while `@dedent` typically means “dedent the current line.” But keep this in mind as well:
* If the user had instead typed `if (foo) { /* TODO */ }` on row 1 and pressed Enter, we’d need to be smart enough to know that the `{` that signals an indent was “cancelled out” by the `}` that came after it. That’s the second purpose of `@dedent`: to balance out `@indent` captures when deciding whether to indent the next line.
* A `@dedent` capture typically results in a dedent _only_ when it’s the first non-whitespace content on the row. And if the row to be dedented is the one being typed on, Pulsar will trigger a dedent _exactly once_, rather than after each character typed on the row.
Why? Because if you don’t want the row to be dedented after all, this behavior allows you to re-indent the row the way you want and continue typing without Pulsar stubbornly trying to re-dedent the row over and over in a perverse game of tug-of-war.
Some other languages don’t have it so easy. Ruby, for instance, doesn’t open its logical blocks with one consistent delimiter…
```rb
if foo
bar
end
while x < y
x += 2
end
```
…which means that its `indents.scm` looks more complex.
```scm
[
"class" "def" "module" "if" "elsif" "else" "unless" "case" "when" "while"
"until" "for" "begin" "do" "rescue" "ensure" "(" "{" "["
] @indent
[
"end" ")" "}" "]" "when" "elsif" "else" "rescue" "ensure"
] @dedent
```
### Advanced indents
Indent captures can use the same set of scope tests that were described earlier for syntax highlighting, because sometimes a node should only hint at an indent in certain situations.
For instance, we can handle “hanging” indents like this one…
```js
return this.somewhatLongMethodName() ||
this.somehowAnEvenLongerMethodName();
```
…because we understand that `||` can’t possibly terminate a JavaScript statement, so the next line must be a logical continuation of the statement.
```scm
(["||" "&&"] @indent
(#is? test.lastTextOnRow))
```
(Indent captures can also use some specialized tests that apply only to indentation queries: `indent.matchesComparisonRow` and `indent.matchesCurrentRow`. You’ll see an example of this below.)
`@indent` and `@dedent` are often the only captures you need. But for unusual situations, Pulsar allows for other sorts of captures:
* `@dedent.next` can be used for the situation where something in row `X` hints that row `X + 1` should be dedented _no matter what_ its content is.
One example would be a conditional statement without braces…
```js
if (!e.shiftKey)
return e.preventDefault();
```
…because the line immediately after this code should always be dedented one level from the `return` statement.
* `@match` is a powerful capture that can accept configuration. When a `@match` capture is present, it will set the indent level of the current row to equal the level of a specific earlier row. For instance, consider one way to indent a `switch` statement:
```js
switch (job) {
case 'lint':
lintFile();
case 'fix':
fixFile();
default:
console.warn("Unknown job");
}
```
This indentation style means that the closing brace (`}`) should be dedented _two_ levels from the previous line. A `@match` capture can handle this as follows…
```scm
((switch_body "}" @match
(#set! indent.match parent.startPosition)))
```
…because this capture tells Pulsar to set the closing brace’s row to match the indent level of the row where the `switch_body` itself starts. Pulsar therefore sets row 8’s level to match row 1’s.
`@match` captures can also define an offset — for scenarios where they want to indent themselves some number of levels _more_ or _less_ than a reference row.
`@match` captures apply to the current row, so they are considered in phase two of indentation hinting. But they end up overriding anything that has happened in phase 1.
* `@match.next` is the counterpart to `@match`, but it acts in phase one. But it also works like `@dedent.next` in that it allows you to set a “baseline” indentation level for a row without even knowing what its content will be.
We’ve seen that Pulsar’s default behavior is to maintain the previous row’s indentation level on the next row. But what if you wanted to ignore the previous row’s indentation level in “hanging indent” scenarios?
```js
let result = createNewObject("foo", "bar", "baz", "thud",
{ save: true, notifyObservers: false });
```
In this example, the author has chosen a slightly unusual way to handle the indentation of this awkward parameter list: with a hanging indent. Hence the indentation on row 2 is meant to be a “one-off” exception rather than to set a new level for future lines. When the cursor is at the end of row 2 and we press [[Enter]], we know we should move the indentation level back to `0` on row 3 without even waiting to see what the user types.
A `@match.next` capture can handle this as follows:
```scm
(
[
(lexical_declaration)
; (and other types, but this is a simplified example)
] @match.next
; This test means “if the lexical_declaration ends on the comparison row.”
(#if? indent.matchesComparisonRow endPosition)
; The new row’s indent level should match that of the lexical_declaration’s
; starting row.
(#set! indent.match startPosition)
)
```
The logic for resolving a `@match.next` capture is identical to that of resolving a `@match` capture. The differences between them are that
* a `@match.next` capture works in phase one, not phase two — hence it supersedes other captures that are considered in phase one, like `@indent` and `@dedent.next`;
* unlike `@match`, `@match.next` does not immediately return a result — it instead proceeds to phase two and can still have its suggestion altered by a `@dedent` (or overridden entirely by `@match`).
Read the full indent query documentation to learn the details.
## Tags
The fourth sort of query file is typically called `tags.scm`, and its purpose is to identify “tags” or “symbols” — nodes in the tree that contain the names of important things.
:::info
Tree-sitter and other tools call these things “tags,” hence `tags.scm`; but Pulsar calls them “symbols” — hence the {symbols-view} package.
:::
Pulsar’s knowlege of symbols is what allows you to type Cmd+RCtrl+R and navigate a source code file by function name, or a CSS file by selector, or a Markdown file by heading name.
The `tags.scm` file present in most Tree-sitter repositories goes into a level of detail far greater than what Pulsar needs, but that file will nonetheless work very well as-is, and can almost always be copied directly.
If your Tree-sitter parser does not include a `tags.scm`, or if you want to explore customization options, you can read the README for the built-in {symbol-provider-tree-sitter} package.
---
## File: docs/developing-for-pulsar/creating-a-grammar-textmate.md
---
title: Creating a grammar (TextMate)
layout: doc.ejs
---
This section describes Pulsar’s legacy support for TextMate grammars.
:::warning
If you’re adding support for a new language, the preferred way is to [create a modern Tree-sitter grammar](../creating-a-grammar-modern-tree-sitter/). Tree-sitter grammars have better performance and provide support for more editor features, such as the **Editor: Select Larger Syntax Node** command.
But don’t worry; TextMate grammars aren’t going anywhere. They’re probably the right choice for simpler languages.
:::
TextMate grammars are supported by several popular text editors. They provide a set of _regex_ (regular expression) patterns which are assigned _scopes_. These scopes are then turned into the CSS classes that you can target in syntax themes.
## Getting started
TextMate grammars depend heavily on regexes, and you should be comfortable with interpreting and writing regexes before continuing. Note that Pulsar uses the Oniguruma engine, which is very similar to the PCRE or Perl regex engines. Here are some resources to help you out:
- [https://www.regular-expressions.info/tutorial.html](https://www.regular-expressions.info/tutorial.html) provides a comprehensive regex tutorial
- [https://www.rexegg.com/regex-quickstart.html](https://www.rexegg.com/regex-quickstart.html) contains a cheat sheet for various regex expressions
- [https://regex101.com/](https://regex101.com/) or [https://regexr.com/](https://regexr.com/) allows live prototyping
- [https://github.com/kkos/oniguruma/blob/master/doc/RE](https://github.com/kkos/oniguruma/blob/master/doc/RE) are the docs for the Oniguruma regex engine
Grammar files are written in the [CSON](https://github.com/bevry/cson#what-is-cson) or [JSON](https://www.json.org/) format. Whichever one you decide to use is up to you, but this tutorial will be written in CSON.
## Create the package
To get started, press Ctrl+Shift+P Cmd+Shift+P and start typing “Generate Package” to generate a new grammar package. Select “Package Generator: Generate Package,” and you’ll be asked for the path where your package will be created. Let’s call ours `language-flight-manual`.
::: tip Tip
Grammar packages should start with `language-`.
:::
The default package template creates a lot of folders that aren’t needed for grammar packages. Go ahead and delete the `keymaps`, `lib`, `menus`, and `styles` folders. Furthermore, in `package.json`, remove the `activationCommands` section. Now create a new folder called `grammars`, and inside that a file called `flight-manual.cson`. This is the main file that we will be working with; start by populating it with a [boilerplate template](https://gist.github.com/DamnedScholar/622926bcd222eb1ddc483d12103fd315). Now let’s go over what each key means.
`scopeName` is the root _scope_ of your package. This should generally describe what language your grammar package is highlighting; for example, `language-javascript`’s `scopeName` is `source.js` and `language-html`’s is `text.html.basic`. Name it `source.flight-manual` for now.
`name` is the user-friendly name that is displayed in places like the status bar or the grammar selector. Again, this name should describe what the grammar package is highlighting. Rename it to `Flight Manual`.
`fileTypes` is an array of filetypes that `language-flight-manual` should highlight. We’re interested in highlighting the Flight Manual’s Markdown files, so add the `md` extension to the list and remove the others.
`patterns` contains the array of regex patterns that will determine how the file is tokenized.
## Adding patterns
To start, let’s add a basic pattern to tokenize the words `Flight Manual` whenever they show up. Your regex should look like `\bFlight Manual\b`. Here’s what your `patterns` block should look like:
```coffee
'patterns': [
{
'match': '\\bFlight Manual\\b'
'name': 'entity.other.flight-manual'
}
]
```
`match` is where your regex is contained, and `name` is the scope name that is to be applied to the entirety of the match. More information about scope names can be found in [Section 12.4 of the TextMate Manual](https://manual.macromates.com/en/language_grammars).
::: tip Tip
All scopes should end with the portion of the root `scopeName` after the leading `source` or `text`. In our case, all scopes should end with `flight-manual`.
:::
::: note Note
Astute readers may have noticed that the `\b` was changed to `\\b` with two backslashes and not one. This is because CSON processes the regex string before handing it to Oniguruma, so all backslashes need to be escaped twice.
:::
But what if we wanted to apply different scopes to `Flight` and `Manual`? This is possible by adding capture groups to the regex and then referencing those capture groups in a new `capture` property. For example:
```coffee
'match': '\\b(Flight) (Manual)\\b'
'name': 'entity.other.flight-manual'
'captures':
'1':
'name': 'keyword.other.flight.flight-manual'
'2':
'name': 'keyword.other.manual.flight-manual'
```
This will assign the scope `keyword.other.flight.flight-manual` to `Flight`, `keyword.other.manual.flight-manual` to `Manual`, and `entity.other.flight-manual` to the overarching `Flight Manual`.
## Begin/end patterns
Now let’s say we want to tokenize the `{{#note}}` blocks that occur in Flight Manual files. Our previous two examples used `match`, but one limit of `match` is that it can only match single lines. `{{#note}}` blocks, on the other hand, can span multiple lines. For these cases, you can use the `begin`/`end` keys. Once the regex in the `begin` key is matched, tokenization will continue until the `end` pattern is reached.
```coffee
'begin': '({{)(#note)(}})'
'beginCaptures':
'0': # The 0 capture contains the entire match
'name': 'meta.block.start.flight-manual'
'1':
'name': 'punctuation.definition.block.flight-manual'
'2':
'name': 'keyword.note.flight-manual'
'3':
'name': 'punctuation.definition.block.flight-manual'
'end': '({{)(/note)(}})'
'endCaptures':
'0':
'name': 'meta.block.end.flight-manual'
'1':
'name': 'punctuation.definition.block.flight-manual'
'2':
'name': 'keyword.note.flight-manual'
'3':
'name': 'punctuation.definition.block.flight-manual'
'name': 'meta.block.note.flight-manual'
```
::: tip Tip
Get into the habit of providing punctuation scopes early on. It’s much less effort than having to go back and rewriting all your patterns to support punctuation scopes when your grammar starts to get a bit longer!
:::
Awesome, we have our first multiline pattern! However, if you’ve been following along and playing around in your own `.md` file, you may have noticed that `Flight Manual` doesn’t receive any scopes inside a note block. A begin/end block is essentially a subgrammar of its own: once it starts matching, it will only match its own subpatterns until the end pattern is reached. Since we haven’t defined any subpatterns, then clearly nothing will be matched inside of a note block. Let’s fix that!
```coffee
'begin': '({{)(#note)(}})'
'beginCaptures':
'0': # The 0 capture contains the entire match
'name': 'meta.block.start.flight-manual'
'1':
'name': 'punctuation.definition.block.flight-manual'
'2':
'name': 'keyword.note.flight-manual'
'3':
'name': 'punctuation.definition.block.flight-manual'
'end': '({{)(/note)(}})'
'endCaptures':
'0':
'name': 'meta.block.end.flight-manual'
'1':
'name': 'punctuation.definition.block.flight-manual'
'2':
'name': 'keyword.note.flight-manual'
'3':
'name': 'punctuation.definition.block.flight-manual'
'name': 'meta.block.note.flight-manual'
'patterns': [
{
'match': '\\b(Flight) (Manual)\\b'
'name': 'entity.other.flight-manual'
'captures':
'1':
'name': 'keyword.other.flight.flight-manual'
'2':
'name': 'keyword.other.manual.flight-manual'
}
]
```
There. With the patterns block, `Flight Manual` should now receive the proper scopes.
## Repositories and the `include` keyword (or: how to avoid duplication)
At this point, note blocks are looking pretty nice, as is the `Flight Manual` keyword, but the rest of the file is noticeably lacking any form of Markdown syntax highlighting. Is there a way to include the GitHub-Flavored Markdown grammar without copying and pasting everything over? This is where the `include` keyword comes in. `include` allows you to _include_ other patterns, even from other grammars! `language-gfm`’s `scopeName` is `source.gfm`, so let’s include that. Our `patterns` block should now look like the following:
```coffee
'patterns': [
{
'include': 'source.gfm'
}
{
# Flight Manual pattern
}
{
# Note begin/end pattern
}
]
```
However, including `source.gfm` has led to another problem: note blocks still don’t have any Markdown highlighting! The quick fix would be to add the include pattern to the note’s pattern block as well, but now we’re duplicating two patterns. You can imagine that as this grammar grows it’ll quickly become inefficient to keep copying each new global pattern over to the `note` pattern as well. Therefore, `include` helpfully recognizes the special `$self` scope. `$self` automatically includes all the top-level patterns of the current grammar. The `note` block can then be simplified to the following:
```coffee
'begin': '({{)(#note)(}})'
# beginCaptures
'end': '({{)(/note)(}})'
# endCaptures
'name': 'meta.block.note.flight-manual'
'patterns': [
{
'include': '$self'
}
]
```
## Where to go from here
There are several good resources out there that help when writing a grammar. The following is a list of some particularly useful ones (some have been linked to in the sections above as well).
- [DamnedScholar’s Gist](https://gist.github.com/DamnedScholar/622926bcd222eb1ddc483d12103fd315). Provides a template of most keys, each with a short comment explaining their function.
- [Aerijo’s Gist](https://gist.github.com/Aerijo/b8c82d647db783187804e86fa0a604a1). Another guide that attempts to fully explain making a grammar package for users of all levels.
- [http://www.apeth.com/nonblog/stories/textmatebundle.html](http://www.apeth.com/nonblog/stories/textmatebundle.html). A blog of a programmer’s experience writing a grammar package for TextMate.
- [Oniguruma docs](https://github.com/kkos/oniguruma/blob/master/doc/RE). The documentation for the regex engine Pulsar uses.
- [TextMate Section 12](http://manual.macromates.com/en/language_grammars.html). Pulsar uses the same principles as laid out here, including the list of acceptable scopes.
- [`second-mate`](https://github.com/pulsar-edit/second-mate). Not necessary to write a grammar, but a good technical reference for what Pulsar is doing behind the scenes.
- Look at any existing packages, such as the ones for [Python](https://github.com/pulsar-edit/pulsar/tree/master/packages/language-python), [JavaScript](https://github.com/pulsar-edit/pulsar/tree/master/packages/language-javascript), [HTML](https://github.com/pulsar-edit/pulsar/tree/master/packages/language-html), [and more](https://github.com/pulsar-edit/pulsar/tree/master/packages).
---
## File: docs/developing-for-pulsar/developing-a-package.md
---
title: Developing a package
layout: doc.ejs
---
Let’s get started by creating and looking at the general layout of a package, and how it should look.
## Package generator
The simplest way to start a package is to use the built-in package generator that ships with Pulsar. This generator is implemented as a separate package called {package-generator}.
You can run the generator by invoking the command palette and searching for "Generate Package". A dialog will appear asking you to name your new project. Whatever name you choose Pulsar will then create that directory and fill it out with a skeleton project and link it into your `~/.pulsar/packages` `%USERPROFILE%\.pulsar\packages` directory so it’s loaded when you launch your editor next time.
::: note Note
You may encounter a situation where your package is not loaded.
Make sure the package name you choose isn’t an existing bundled package name or one hosted on [pulsar-edit.dev](https://packages.pulsar-edit.dev/packages).
:::
Once this command has run you’ll see that Pulsar has created about a dozen files that make up the package. Let’s take a look at each of them to get an idea of how a package is structured.
```
my-package/
├─ grammars/
├─ keymaps/
├─ lib/
├─ menus/
├─ spec/
├─ snippets/
├─ styles/
├─ index.js
└─ package.json
```
Not every package will have (or need) all of these directories and the package generator doesn’t create `snippets` or `grammars`. Let’s see what some of these are so we can start messing with them.
### `package.json`
Similar to [Node modules](https://en.wikipedia.org/wiki/Npm_(software)), Pulsar packages contain a `package.json` file in their top-level directory. This file contains metadata about the package, such as the path to its "main" module, library dependencies, and manifests specifying the order in which its resources should be loaded.
In addition to some of the regular [Node `package.json` keys](https://docs.npmjs.com/files/package.json) available, Pulsar `package.json` files have their own additions.
- `main`: the path to the JavaScript file that’s the entry point to your package. If this is missing, Pulsar will default to looking for an `index.js` or `index.coffee`.
- `styleSheets`: an array of strings identifying the order of the style sheets your package needs to load. If not specified, style sheets in the `styles` directory are added alphabetically.
- `keymaps`: an array of strings identifying the order of the key mappings your package needs to load. If not specified, mappings in the `keymaps` directory are added alphabetically.
- `menus`: an array of strings identifying the order of the menu mappings your package needs to load. If not specified, mappings in the `menus` directory are added alphabetically.
- `snippets`: an array of strings identifying the order of the snippets your package needs to load. If not specified, snippets in the `snippets` directory are added alphabetically.
- `activationCommands`: an object identifying commands that trigger your package’s activation. The keys are CSS selectors; the values are arrays of strings identifying the command. The loading of your package is **delayed** until one of these events is triggered within the associated scope defined by the CSS selector. If not specified, the `activate()` method of your main export will be called when your package is loaded.
- `activationHooks`: an array of strings identifying hooks that trigger your package’s activation. If `activationHooks` is present, the loading of your package is **delayed** until one of these hooks is triggered. Activation hooks include:
- `core:loaded-shell-environment` for when Pulsar has finished loading the shell environment variables
- `scope.name:root-scope-used` for when a file first is opened from the specified language (e.g. `source.ruby:root-scope-used`)
- `language-package-name:grammar-used` for when a specific language package is used (e.g., `my-special-language-javascript:grammar-used`)
- `...:uri-opened` for when any valid URI has been opened. Change the `...` to the URI the package should be triggered by. (e.g. to activate when change log is opened `atom://welcome/changeLog:uri-opened`)
- `...:file-name-opened` for when any specific file has been opened. Change the `...` to the full filename the package should be triggered by. (e.g. to activate when `robots.txt` is opened `robots.txt:file-name-opened`)
- `workspaceOpeners`: An array of strings identifying URIs that trigger your package’s activation. For example, suppose your package registers a custom opener for `atom://my-custom-panel`; by including that string in `workspaceOpeners`, your package will defer its activation until that URI is opened.
The `package.json` made by the package generator looks like:
```json
{
"name": "your-package-name",
"main": "./lib/your-package-name",
"version": "0.0.0",
"description": "A short description of your package",
"activationCommands": {
"atom-workspace": "your-package-name:toggle"
},
"repository": "https://github.com/pulsar-edit/your-package-name",
"license": "MIT",
"engines": {
"atom": ">=1.0.0 <2.0.0"
},
"dependencies": {}
}
```
If we wanted `activationHooks`, we might add:
```json
{
"name": "your-package-name",
"main": "./lib/your-package-name",
"version": "0.0.0",
"description": "A short description of your package",
"activationHooks": [
"language-javascript:grammar-used",
"language-coffee-script:grammar-used"
],
"repository": "https://github.com/pulsar-edit/your-package-name",
"license": "MIT",
"engines": {
"atom": ">=1.0.0 <2.0.0"
},
"dependencies": {}
}
```
One of the first things you should do is ensure that this information is filled out; `name`, `description`, `repository` URL, and `license` can all often be filled out immediately.
::: warning Warning
Do not forget to update the repository URL! The one generated for you is invalid by design and will prevent you from publishing your package until updated.
:::
### Source code
If you want to extend Pulsar’s behavior, your package should contain a single top-level module which you export from whichever file is indicated by the `main` key in your `package.json` file. In the package we just generated, the main package file is `lib/your-package-name.js`. The remainder of your code should be placed in the `lib` directory, and required from your top-level file. If the `main` key is not in your `package.json` file, it will look for `index.js` or `index.coffee` as the main entry point.
Your package’s top-level module is a singleton object that manages the lifecycle of your extensions to Pulsar. Even if your package creates ten different views and appends them to different parts of the DOM, it’s all managed from your top-level object.
Your package’s top-level module can implement the following basic methods:
- `activate(state)`: This **optional** method is called when your package is activated. It is passed the state data from the last time the window was serialized if your module implements the `serialize()` method. Use this to do initialization work when your package is started (like setting up DOM elements or binding events). If this method returns a promise the package will be considered loading until the promise resolves (or rejects).
- `initialize(state)`: This **optional** method is similar to `activate()` but is called earlier. Whereas activation occurs after the workspace has been deserialized (and can therefore happen after [your package’s deserializers](/infrastructure/serialization-in-pulsar/#serialization-methods) have been called), `initialize()` is guaranteed to be called before everything. Use `activate()` if you want to be sure that the workspace is ready; use `initialize()` if you need to do some setup prior to your deserializers or view providers being invoked.
- `serialize()`: This **optional** method is called when the window is shutting down, allowing you to return JSON to represent the state of your component. When the window is later restored, the data you returned is passed to your module’s `activate` method so you can restore your view to where the user left off.
- `deactivate()`: This **optional** method is called when the window is shutting down and when the package is disabled. If your package is watching any files or holding external resources in any other way, release them here. You should also dispose of all subscriptions you’re holding on to.
### Style sheets
Style sheets for your package should be placed in the `styles` directory. Any style sheets in this directory will be loaded and attached to the DOM when your package is activated. Style sheets can be written as CSS or [Less](http://lesscss.org), but Less is recommended.
Ideally, you won’t need much in the way of styling. Pulsar provides a standard set of components which define both the colors and UI elements for any package that fits into Pulsar seamlessly. You can view all of Pulsar’s UI components by opening the styleguide: open the command palette Ctrl+Shift+P Cmd+Shift+P and search for `styleguide`, or type Ctrl+Shift+G Cmd+Ctrl+Shift+G.
If you _do_ need special styling, try to keep only structural styles in the package style sheets. If you _must_ specify colors and sizing, these should be taken from the active theme’s [ui-variables.less](https://github.com/pulsar-edit/atom-dark-ui/blob/master/styles/ui-variables.less).
An optional `styleSheets` array in your `package.json` can list the style sheets by name to specify a loading order; otherwise, style sheets are loaded alphabetically.
### Keymaps
You can provide key bindings for commonly used actions for your extension, especially if you’re also adding a new command. In our new package, we have a keymap filled in for us already in the `keymaps/your-package-name.json` file:
```js
{
"atom-workspace": {
"ctrl-alt-o": "your-package-name:toggle"
}
}
```
This means that if you press Alt+Ctrl+O, our package will run the `your-package-name:toggle` command. We’ll look at that code later, but if you want to change the default key mapping, you can do that in this file.
Keymaps are placed in the `keymaps` subdirectory. By default, all keymaps are loaded in alphabetical order. An optional `keymaps` array in your `package.json` can specify which keymaps to load and in what order.
Keybindings are executed by determining which element the keypress occurred on. In the example above, the `your-package-name:toggle` command is executed when pressing Alt+Ctrl+O on the `atom-workspace` element. Because the `atom-workspace` element is the parent of the entire Pulsar UI, this means the key combination will work anywhere in the application.
We’ll cover more advanced keybinding stuff a bit later in [Keymaps in-depth](/infrastructure/keymaps-in-depth).
### Menus
Menus are placed in the `menus` subdirectory. This defines menu elements like what pops up when you right click a context-menu or would go in the application menu to trigger functionality in your package.
By default, all menus are loaded in alphabetical order. An optional `menus` array in your `package.json` can specify which menus to load and in what order.
#### Application menu
It’s recommended that you create an application menu item under the _Packages_ menu for common actions with your package that aren’t tied to a specific element. If we look in the `menus/your-package-name.json` file that was generated for us, we’ll see a section that looks like this:
```js
"menu": [
{
"label": "Packages",
"submenu": [
{
"label": "Your Package Name",
"submenu": [
{
"label": "Toggle",
"command": "your-package-name:toggle"
}
]
}
]
}
]
```
This section puts a "Toggle" menu item under a menu group named "Your Package Name" in the "Packages" menu.
When you select that menu item, it will run the `your-package-name:toggle` command, which we’ll look at in a bit.
The menu templates you specify are merged with all other templates provided by other packages in the order which they were loaded.
#### Context menu
It’s recommended to specify a context menu item for commands that are linked to specific parts of the interface. In our `menus/your-package-name.json` file, we can see an auto-generated section that looks like this:
```js
"context-menu": {
"atom-text-editor": [
{
"label": "Toggle your-package-name",
"command": "your-package-name:toggle"
}
]
}
```
This adds a "Toggle your-package-name" menu option to the menu that pops up when you right-click in an Pulsar text editor pane.
When you click that it will again run the `your-package-name:toggle` method in your code.
Context menus are created by determining which element was selected and then adding all of the menu items whose selectors match that element (in the order which they were loaded). The process is then repeated for the elements until reaching the top of the DOM tree.
You can also add separators and submenus to your context menus. To add a submenu, provide a `submenu` key instead of a command. To add a separator, add an item with a single `type: 'separator'` key/value pair. For instance, you could do something like this:
```js
{
"context-menu": {
"atom-workspace": [
{
"label": "Text",
"submenu": [
{
"label": "Inspect Element",
"command": "core:inspect"
},
{
"type": "separator"
},
{
"label": "Select All",
"command": "core:select-all"
},
{
"type": "separator"
},
{
"label": "Delete Selected Text",
"command": "core:delete"
}
]
}
]
}
}
```
## Basic debugging
One of the cool things about Pulsar being built on Chromium is that you can use some of the same debugging tools available to you that you have when doing web development.
To open up the Developer Console, press Ctrl+Shift+I Alt+Cmd+I or choose the menu option _View > Developer > Toggle Developer Tools_.
From here you can inspect objects, run code and view console output just as though you were debugging a web site.
## Testing
Your package should have tests, and if they’re placed in the `spec` directory, they can be run by Pulsar.
Under the hood, [Jasmine v1.3](https://jasmine.github.io/archives/1.3/introduction) executes your tests, so you can assume that any DSL available there is also available to your package.
### Running tests
Once you’ve got your test suite written, you can run it by pressing Ctrl+Shift+YAlt+Cmd+Ctrl+P or via the _View > Developer > Run Package Specs_ menu. Our generated package comes with an example test suite, so you can run this right away to see what happens.
You can also use the `pulsar --test spec` command to run them from the command line. It prints the test output and results to the console and returns the proper status code depending on whether the tests passed or failed.
---
## File: docs/developing-for-pulsar/developing-a-theme.md
---
title: Developing a theme
layout: doc.ejs
---
Pulsar’s interface is rendered using HTML, and it’s styled via [Less](http://lesscss.org/) which is a superset of CSS. Don’t worry if you haven’t heard of Less before; it’s just like CSS, but with a few handy extensions.
Pulsar supports two types of themes: _UI_ and _Syntax_. UI themes style elements such as the tree view, the tabs, drop-down lists, and the status bar. Syntax themes style the code, gutter and other elements inside the editor view.
Themes can be installed and changed from the settings view — which you can open by selecting the _Edit > Preferences_ _Pulsar > Preferences_ _File > Preferences_ menu, and clicking the "Install" or "Themes" tab on the left-hand navigation.
## Getting started
Themes are pretty straightforward, but it’s still helpful to be familiar with a few things before starting:
- Less is a superset of CSS, but it has some really handy features like
variables. If you aren’t familiar with its syntax, take a few minutes to
[familiarize yourself](https://speakerdeck.com/danmatthews/less-css).
- You may also want to review the concept of a `package.json` (as covered in
[Pulsar `package.json`](../developing-a-package/#package.json)). This file is used to help distribute your theme to Pulsar users.
- Your theme’s `package.json` must contain a `theme` key with a value of `ui`
or `syntax` for Pulsar to recognize and load it as a theme.
- You can find existing themes to install or fork in
[Pulsar Package Registry](https://packages.pulsar-edit.dev/packages).
## Creating a syntax theme
Let’s create your first theme.
To get started, press Ctrl+Shift+P Cmd+Shift+P and start typing `Generate Syntax Theme` to generate a new theme package. Select **Package Generator: Generate Syntax Theme**, and you’ll be asked for the path where your theme will be created. Let’s call ours `motif-syntax`.
::: tip Tip
Syntax themes should end with `-syntax` and UI themes should end with `-ui`.
:::
Pulsar will display a new window, showing the `motif-syntax` theme, with a default set of folders and files created for us. If you open the Settings View with Ctrl+,Cmd+, and click the **Themes** tab on the left, you’ll see the "Motif" theme listed in the "Syntax Theme" drop-down. Select it from the menu to activate it. Now, when you open an editor, you should see your new motif-syntax theme in action.
Open up `styles/colors.less` to change the various color variables which have already been defined. For example, turn `@red` into `#f4c2c1`.
Then open `styles/base.less` and modify the various selectors that have already been defined. These selectors style different parts of code in the editor such as comments, strings and the line numbers in the gutter.
As an example, let’s make the `.gutter` `background-color` into `@red`.
Reload Pulsar by pressing Alt+Ctrl+RAlt+Ctrl+Cmd+L (or running the **Window: Reload** command) to see the changes you made reflected in your Pulsar window. Pretty neat!
::: tip Tip
You can avoid reloading to see changes you make by opening an Pulsar window in Dev Mode. To open a Dev Mode Pulsar window, run `pulsar --dev .` in the terminal or use the _View > Developer > Open in Dev Mode_ menu. When you edit your theme, changes will instantly be reflected!
:::
::: note Note
It’s advised to _not_ specify a `font-family` in your syntax theme because it will override the Font Family field in Pulsar’s settings. Users will reasonably expect to use their preferred font for coding with your syntax theme. If you’d still like to recommend a font that goes well with your theme, we suggest you do so in your README.
Because typefaces are a user interface concern, and because there is no general setting that controls the `font-family` of the Pulsar UI, you may specify a `font-family` in your _UI_ theme if you like.
:::
## Creating a UI theme
To create a UI theme, do the following:
1. Fork the [ui-theme-template](https://github.com/pulsar-edit/ui-theme-template)
2. Clone the forked repository to the local filesystem
3. Open a terminal in the forked theme’s directory
4. Open your new theme in a dev mode Pulsar window: run `pulsar --dev .` in the
terminal or use the _View > Developer > Open in Dev Mode_ menu
5. Change the name of the theme in the theme’s `package.json` file
6. Name your theme end with a `-ui`, for example `super-white-ui`
7. Run `pulsar -p link --dev` to symlink your repository to `~/.pulsar/dev/packages``%USERPROFILE%\.pulsar`
8. Reload Pulsar using Alt+Ctrl+R Alt+Cmd+Ctrl+L
9. Enable the theme via the “UI Theme” drop-down menu in the “Themes” tab of the
settings view
10. Make changes! Since you opened the theme in a dev mode window, changes will
be instantly reflected in the editor without having to reload.
::: tip Tip
Because we used `pulsar -p link --dev` in the above instructions, if you break anything you can always close your project window and relaunch Pulsar normally (without dev mode) to force the window to the default theme. This allows you to continue working on your theme even if something goes catastrophically wrong.
:::
## Theme variables
UI themes **must** provide a `ui-variables.less` file; syntax themes **must** provide a `syntax-variables.less` file. These files contain predefined variables that packages use so that useful values can be exposed.
These files in the Pulsar repository define default values for certain variables:
- [ui-variables.less](https://github.com/pulsar-edit/pulsar/blob/master/static/variables/ui-variables.less)
- [syntax-variables.less](https://github.com/pulsar-edit/pulsar/blob/master/static/variables/syntax-variables.less)
These default values will be used as fallbacks in case a theme doesn’t define its own variables.
### Use in user customizations
Suppose you want the `async` modifier on JavaScript functions to be colored like a keyword. You could edit your user stylesheet as follows:
```less
@import "syntax-variables";
.syntax--source.syntax--js {
.syntax--storage.syntax--modifier.syntax--async {
color: @syntax-color-keyword;
}
}
```
You don’t have to look up the specific color that your syntax theme uses for keywords! You can use this variable because it’s part of the `syntax-variables.less` contract. And it’s designed to be portable so that you won’t have to change the value if you change syntax themes.
### Use in packages
Community packages can also use these values to deliver theme-appropriate editor decorations and user interfaces. In any of your package’s `.less` files, you can access the theme variables by importing the `ui-variables` or `syntax-variables` file from Pulsar.
If you’re writing a community package that _isn’t_ a theme, you might have to use CSS to lay out some interface elements. It’s fine to use structural styling (flexboxes, grids, positioning, et cetera), but your package should avoid specifying colors, padding sizes, or any measurements in absolute pixels. Why? Because those are the concerns of a UI theme!
Likewise, if your package adds decorations to source code within a text editor, you don’t want to hard-code any elements that will clash with the user’s syntax theme.
Instead, your package should **use the theme variables**. If you follow this guideline, your package will look good out of the box with any theme!
Here are two example `.less` files illustrating how a package can use `ui-variables` and `syntax-variables`:
```less
@import "ui-variables";
.my-selector {
background-color: @base-background-color;
padding: @component-padding;
}
```
```less
@import "syntax-variables";
.my-selector {
background-color: @syntax-background-color;
}
```
## Development workflow
There are a few tools to help make theme development faster and easier.
### Live reload
Reloading by pressing Alt+Ctrl+RAlt+Cmd+Ctrl+L after you make changes to your theme is less than ideal. That’s why Pulsar supports live updating of styles on Pulsar windows in dev mode through the {dev-live-reload} package.
To launch a dev mode window:
- open your theme directory in a dev window by selecting the _View > Developer > Open in Dev Mode_ menu item; or
- launch Pulsar from the terminal with `pulsar --dev`.
If you’d like to reload all the styles at any time, you can use the shortcut Alt+Ctrl+RAlt+Cmd+Ctrl+L.
### Developer tools
Pulsar is based on the Chromium browser and supports its developer tools. You can open them by selecting the _View > Developer > Toggle Developer Tools_ menu, or by using the Ctrl+Shift+IAlt+Cmd+I.
The dev tools allow you to inspect elements and take a look at their CSS properties.
Check out Google’s [extensive tutorial](https://developer.chrome.com/devtools/docs/dom-and-styles) for an introduction.
### Pulsar styleguide
If you are creating an UI theme, you’ll want a way to see how your theme changes affect all the components in the system. The {styleguide} is a page that renders every component Pulsar supports.
To open the Styleguide, open the command palette with Ctrl+Shift+PCmd+Shift+P and find the **Styleguide: Show** command, or use the shortcut Ctrl+Shift+GCmd+Ctrl+Shift+G.
If you’re editing your UI theme in dev mode, the elements shown in the styleguide will update in real time when you make changes, just like all other UI elements in the window.
### Side by side
Sometimes when creating a theme (or package) things can go wrong and the editor becomes unusable. For instance: the text and background might accidentally use the same color, or a critical element might get hidden or pushed out of sight.
To avoid having to open Pulsar in “normal” mode to fix the issue, it’s advised to open **two** Pulsar windows: one for making changes and one in dev mode to see the changes getting applied.
> Make changes on the **left**, see the changes getting applied in dev mode
> on the **right**.
Now if you mess up something, only the window in dev mode will be affected and you can easily correct the mistake in your “normal” window.
## Publish your theme
Once you’re happy with your theme and would like to share it with other Pulsar users, it’s time to publish it.
Follow the steps on the [Publishing](../publishing/) page. The example used is for the Word Count package, but publishing a theme works exactly the same.
---
## File: docs/developing-for-pulsar/handling-uris.md
---
title: Handling URIs
layout: doc.ejs
---
Packages have the ability to handle special URIs triggered from the system; for example, a package named `my-package` can register itself to handle any URI starting with `atom://my-package/`.
::: note
Every URI still begins with `atom://` to keep backwards compatibility with all existing URIs out there. The `pulsar://` URL scheme **does not exist** and will not work.
:::
::: warning Warning
Handling URIs triggered from other applications, like a web browser, is a powerful tool, but also one that can be jarring. You should shape your package’s user experience to handle this well. In general, you should **avoid taking direct action on behalf of a user**. For example, a URI handler that immediately installs a package is too invasive, but a URI handler that shows the package’s pane in the settings view is useful. A URI handler that begins to clone a repo is overly aggressive, but a URI handler that _prompts_ the user to clone a repo is okay.
Any package with a URI handler that we feel violates this guideline is subject to removal from the Pulsar package registry at our discretion.
:::
## Modifying your `package.json`
The first step to handling URIs from your package is to modify its `package.json` file. You should add a new key called `uriHandler`, and its value should be an object.
The `uriHandler` object _must_ contain a key called `method` with a string value that tells Pulsar which method in your package to call when a URI needs to be handled. The object can _optionally_ include a key called `deferActivation` which can be set to the boolean `false` to prevent Pulsar from deferring activation of your package — see more below.
For example, if we want our package `my-package` to handle URIs with a method on our package’s main module called `handleURI`, we could add the following to our `package.json`:
```json
"uriHandler": {
"method": "handleURI"
}
```
## Modifying your main module
Now that we’ve told Pulsar that we want our package to handle URIs beginning with `atom://my-package/` via our `handleURI` method, we need to actually write this method. Pulsar passes two arguments to your URI handler method; the first one is the fully-parsed URI plus query string, [parsed with Node’s `url.parse(uri, true)`](https://nodejs.org/api/url.html#url_url_parse_urlstring_parsequerystring_slashesdenotehost). The second argument is the raw, string URI; this is normally not needed since the first argument gives you structured information about the URI.
Here’s a sample package, written in JavaScript, that handles URIs with the `package.json` configuration we saw above.
```js
export default {
activate() {
// normal activation code here
},
handleURI(parsedUri) {
console.log(parsedUri);
},
};
```
When Pulsar handles, for example, the URI `atom://my-package/my/test/url?value=42&other=false`, the package would log out something like the following:
```js
{
protocol: 'atom:',
slashes: true,
auth: null,
host: 'my-package',
port: null,
hostname: 'my-package',
hash: null,
search: '?value=true&other=false',
query: { value: '42', other: 'false' },
pathname: '/my/test/url',
path: '/my/test/url?value=true&other=false',
href: 'atom://my-package/my/test/url?value=true&other=false'
}
```
Notice that the query string arguments are available in the `query` property, but are strings — you’ll have to convert to other native types yourself.
## Controlling activation deferral
For performance reasons, adding a `uriHandler` entry to your package’s `package.json` will enable _deferred activation_. This means that Pulsar will not activate your package until it has a URI for it to handle — at which point it will activate your package and then immediately call the URI handler method. If you want to disable the deferred activation, ensuring your package is activated upon startup, you can add `"deferActivation": false` to the URI handler config:
```json
"uriHandler": {
"method": "handleURI",
"deferActivation": false
}
```
Before doing this, make sure your package actually needs to be activated immediately. Disabling deferred activation means Pulsar takes longer to start, since it has to activate one additional package during startup.
## Linux support
Because URI handling is different across operating systems and distributions, there is no built-in URI handler support for Pulsar on Linux. If you want to configure URI handling on your system yourself, then you should configure `atom:` protocol URIs to trigger Pulsar with the `--uri-handler` flag; for example, the URI `atom://test/uri` should launch Atom via `atom --uri-handler atom://test/uri`.
## Core URIs
Pulsar provides a core URI to handle opening files with the syntax `atom://core/open/file?filename=&line=&column=`.