difftastic Documentation

Technical Documentation: Wilfred/difftastic

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Difftastic is a structural diff tool that compares files based on
their syntax.

For installation instructions, see
Installation in
the manual.

Examples Copy Chapter

^ Difftastic understands exactly which pieces of syntax have changed,
and can highlight them in context.

^ Difftastic understands when whitespace matters, and when it's just
an indentation change.

^ Difftastic is not line-oriented. If you reformat your code and it's
now split over multiple lines, difftastic will show you what's
actually changed.

^ Difftastic is compatible with git (see the configuration
instructions), as well as
many other version control systems.

Languages Supported Copy Chapter

Difftastic supports over 30 programming languages, see the
manual for the full list.

If a file has an unrecognised extension, difftastic uses a
line-oriented diff with word highlighting.

Known Issues Copy Chapter

Performance. Difftastic scales relatively poorly on files with a large
number of changes, and can use a lot of memory.

Display. Difftastic has a side-by-side display which usually works well, but can
be confusing.

Robustness. Difftastic regularly has releases that fix crashes.

Non-goals Copy Chapter

Patching. Difftastic output is intended for human consumption, and it
does not generate patches that you can apply later. Use diff if you
need a patch.

(Patch files are also line-oriented, which is too limited for
difftastic. Difftastic might find additions and removals on the same
line, and it tracks the relationship between line numbers in the old
and new file.)

Merging. AST merging is a hard problem that difftastic does not
address. You might be interested in the mergiraf
tool ("merge giraffe"), which does do AST
merging.

FAQ Copy Chapter

Can I use difftastic with git? Copy Section

You can! The difftastic manual includes instructions for git
usage. You can also use it
with mercurial.

If you're an Emacs user, check out this blog
post
showing one way to use difftastic with magit, as well as
difftastic.el.

Does difftastic integrate with my favourite tool? Copy Section

Probably not. Difftastic is young. Consider writing a plugin for your
favourite tool, and I will link it in the README!

What about parse errors? Copy Section

By default, difftastic falls back to a line-oriented diff whenever
parse errors are encountered.

This is a conservative choice to ensure that difftastic never claims
that two syntactically different files are the same.

Parse errors can occur if the file uses language features that the
parser does not understand, if the language relies on a preprocessor
before parsing (e.g. C++), or if the file has genuine syntactic
mistakes.

In practice, difftastic virtually always produces a good result when
there are a few minor parse errors. Consider allowing a small number
of parse errors when using difftastic.

$ export DFT_PARSE_ERROR_LIMIT=20
$ difft foo1.c foo2.c

Can difftastic help me with merge conflicts? Copy Section

Yes! As of version 0.50 (released 2023-08-16), difftastic understands merge conflict markers
(i.e. <<<<<<<, ======= and >>>>>>>).

Pass your file with conflicts as a single argument to
difftastic. Difftastic will construct the two conflicting files and
diff those.

$ difft file_with_conflicts.js

Can difftastic do merges? Copy Section

No. AST merging is a hard problem that difftastic does not address.

AST diffing is a lossy process from the perspective of a text
diff. Difftastic will ignore whitespace that isn't syntactically
significant, but merging requires tracking whitespace.

The mergiraf tool does offer merges based on
a tree-sitter AST however.

Can difftastic ignore reordering? Copy Section

No. Difftastic always considers order to be important, so diffing
e.g. set(1, 2) and set(2, 1) will show changes.

If you're diffing JSON, consider sorting the keys before passing them
to difftastic.

$ difft <(jq --sort-keys < file_1.json) <(jq --sort-keys < file_2.json)

See also Tricky Cases: Unordered Data
Types
in the manual.

Can I use difftastic to check for syntactic changes without diffing? Copy Section

Yes. Difftastic can check if the two files have the same AST, without
calculating a diff. This is much faster than normal diffing, and
useful for building tools that check for changes.

For example:

$ difft --check-only --exit-code before.js after.js

This will set the exit code to 0 if there are no syntactic changes, or
1 if there are changes found.

Why aren't colours appearing in my terminal? Copy Section

Difftastic uses ANSI bright colours by default, but some terminal
themes show bright colours as grey. Solarized is a popular theme that
does this.

If you're a Solarized user, use export DFT_BACKGROUND=light to
disable bright colours, or try a different terminal colour scheme.

How does it work? Copy Section

Difftastic treats structural diffing as a graph problem, and uses
Dijkstra's algorithm.

My blog
post
describes the design, and there is also an internals section in the
manual.

Translation Copy Chapter

• Chinese

License Copy Chapter

Difftastic is open source under the MIT license, see LICENSE for more
details.

This repository also includes tree-sitter parsers by other authors in
the vendored_parsers/ directory. These are a mix of the MIT license and the
Apache license. See vendored_parsers/*/LICENSE for more details.

Files in sample_files/ are also under the MIT license unless stated
otherwise in their headers.

Overview (`vendored_parsers/tree-sitter-janet-simple/README.md`) Copy Section

tree-sitter-janet-simple

Status Copy Chapter

Subject to change, grammar still evolving.

Coincidentally, it appears another effort by GrayJack was started at about the same time.

The main difference between these two are that GrayJack's grammar supports higher level constructs (e.g. def is recognized by the grammar).

There might end up being different trade-offs in either approach and my belief is that there is room in the world for multiple attempts (especially for lisp-like languages).

Prerequisites Copy Chapter

• emsdk -- emscripten via homebrew seems to work for macos
• node >= 12 (nvm recommended) -- recently tested 12.9.1, 12,16,1

Fine Print Copy Chapter

• The instructions below assume emsdk has been installed, but emcc (tool that can be used to compile to wasm) is not necessarily on one's PATH. If an appropriate emcc is on one's PATH (e.g. emscripten installed via homebrew), the emsdk steps (e.g. source ~/src/emsdk/emsdk_env.sh) below may be ignored.

• node-gyp (tool for compiling native addon modules for Node.js) may fail on machines upgraded to macos Catalina. This document may help cope with such a situation.

Initial Setup Copy Chapter

Suppose typical development sources are stored under ~/src.

# clone repository
cd ~/src
git clone https://github.com/sogaiu/tree-sitter-janet-simple
cd tree-sitter-janet-simple

# create / populate
#   `node_modules` with dependencies
#   `src` with tree-sitter .c goodness
#   `build`
#   `build/Release` and build `tree_sitter_janet_simple_binding.node`
npm install

# included in previous command
#npx tree-sitter generate
#npx node-gyp configure
#npx node-gyp rebuild

Grammar Development Copy Chapter

Hack on grammar and interactively test.

# prepare emsdk (specifically emcc) for building .wasm
source ~/src/emsdk/emsdk_env.sh

# edit grammar.js using some editor

# rebuild tree-sitter stuff and invoke web-ui for interactive testing
npx tree-sitter generate && \
npx node-gyp rebuild && \
npx tree-sitter build-wasm && \
npx tree-sitter web-ui

# in appropriate browser window, paste code in left pane

# examine results in right pane -- can even click on nodes

# find errors and loop back to edit step above...

Parse individual files.

# create and populate sample code file for parsing named `sample.janet`

# parse sample file
npx tree-sitter parse sample.janet

# examine output similar to web-ui, but less convenient

Measure Performance Copy Chapter

# single measurement
npx tree-sitter parse --time sample.janet

# mutliple measurements with `multitime`
multitime -n10 -s1 npx tree-sitter parse --time --quiet sample.janet

Build .wasm Copy Chapter

Assuming emsdk is installed appropriately under ~/src/emsdk.

# prepare emsdk (specifically emcc) for use
source ~/src/emsdk/emsdk_env.sh

# create `tree-sitter-janet-simple.wasm`
npx tree-sitter build-wasm

Resources Copy Chapter

• Guide to your first Tree-sitter grammar
• tree-sitter

Acknowledgments Copy Chapter

• 314eter - handling null characters
• Aerijo - Guide to your first Tree-sitter grammar
• bakpakin - janet
• GrayJack - tree-sitter-janet
• maxbrunsfeld - tree-sitter and related

Overview (`vendored_parsers/tree-sitter-latex/README.md`) Copy Section

tree-sitter-latex

This repo provides a LaTeX grammar for the tree-sitter parser generator.

Originally, this grammar is based off the parser of the texlab language server and primarily focuses on the constructs that are relevant for the language server.

This repository does not provide the accompanying highlighting definitions, instead they live in the nvim-treesitter repository.

To generate the parser, run

npx tree-sitter generate

after checking out the repository.

Limitations Copy Chapter

As widely known, parsing LaTeX is Turing complete so there is no way to handle every construct in a tree-sitter grammar. Instead, the grammar relies on a best effort approach while focusing on the LaTeX specific constructs like environments or sections instead of dealing with TeX internals like catcode.

--- METRICS ---

• Files Extracted: 3
• Estimated Token Budget: ~2888 tokens
• Canonical Reference: https://codewiki.google/github.com/Wilfred/difftastic