xformers Documentation

Technical Documentation: facebookresearch/xformers

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

xFormers - Toolbox to Accelerate Research on Transformers

xFormers is:

Customizable building blocks: Independent/customizable building blocks that can be used without boilerplate code. The components are domain-agnostic and xFormers is used by researchers in vision, NLP and more.
Research first: xFormers contains bleeding-edge components, that are not yet available in mainstream libraries like PyTorch.
Built with efficiency in mind: Because speed of iteration matters, components are as fast and memory-efficient as possible. xFormers contains its own CUDA kernels, but dispatches to other libraries when relevant.

Installing xFormers

(RECOMMENDED, linux & win) Install latest stable with pip: Requires PyTorch 2.10.0

bash

# [linux & win] cuda 12.6 version
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu126
# [linux & win] cuda 12.8 version
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu128
# [linux & win] cuda 13.0 version
pip3 install -U xformers --index-url https://download.pytorch.org/whl/cu130
# [linux only] (EXPERIMENTAL) rocm 7.1 version
pip3 install -U xformers --index-url https://download.pytorch.org/whl/rocm7.1

Development binaries:

bash

# Same requirements as for the stable version above
pip install --pre -U xformers

Install from source: If you want to use with another version of PyTorch for instance (including nightly-releases)

bash

# (Optional) Makes the build much faster
pip install ninja
# Set TORCH_CUDA_ARCH_LIST if running and building on different GPU types
# NOTE: pytorch must already be installed!
pip install -v --no-build-isolation -U git+https://github.com/facebookresearch/xformers.git@main#egg=xformers
# (this can take dozens of minutes)

Benchmarks

Memory-efficient MHA

Setup: A100 on f16, measured total time for a forward+backward pass

Note that this is exact attention, not an approximation, just by calling xformers.ops.memory_efficient_attention

More benchmarks

xFormers provides many components, and more benchmarks are available in BENCHMARKS.md.

(Optional) Testing the installation

This command will provide information on an xFormers installation, and what kernels are built/available:

python

python -m xformers.info

Using xFormers

Key Features

Optimized building blocks, beyond PyTorch primitives
1. Memory-efficient exact attention - up to 10x faster
2. sparse attention
3. block-sparse attention
4. fused softmax
5. fused linear layer
6. fused layer norm
7. fused dropout(activation(x+bias))
8. fused SwiGLU

Install troubleshooting

NVCC and the current CUDA runtime match. Depending on your setup, you may be able to change the CUDA runtime with module unload cuda; module load cuda/xx.x, possibly also nvcc
the version of GCC that you're using matches the current NVCC capabilities
the TORCH_CUDA_ARCH_LIST env variable is set to the architectures that you want to support. A suggested setup (slow to build but comprehensive) is export TORCH_CUDA_ARCH_LIST="6.0;6.1;6.2;7.0;7.2;7.5;8.0;8.6"
If the build from source OOMs, it's possible to reduce the parallelism of ninja with MAX_JOBS (eg MAX_JOBS=2)
If getting error message Filename longer than 260 characters on Windows, make sure long paths are enabled at OS level, and also execute the command git config --global core.longpaths true

License

xFormers has a BSD-style license, as found in the LICENSE file.
It includes code from the triton-lang/kernels repo.

Citing xFormers

If you use xFormers in your publication, please cite it by using the following BibTeX entry.

bibtex

@Misc{xFormers2022,
  author =       {Benjamin Lefaudeux and Francisco Massa and Diana Liskovich and Wenhan Xiong and Vittorio Caggiano and Sean Naren and Min Xu and Jieru Hu and Marta Tintore and Susan Zhang and Patrick Labatut and Daniel Haziza and Luca Wehrstedt and Jeremy Reizenstein and Grigory Sizov},
  title =        {xFormers: A modular and hackable Transformer modelling library},
  howpublished = {\url{https://github.com/facebookresearch/xformers}},
  year =         {2022}
}

Credits

The following repositories are used in xFormers, either in close to original form or as an inspiration:

Overview (`examples/llama_inference/README.md`)

Llama inference

This example showcases how to use xformers kernels and cuda graphs to generate efficiently from large language models. The generation code works with both Llama2 and Code Llama (2023) models, but be aware that generating from large models will require more than a single GPU and a nightly build of PyTorch.

Example runs:

console

$ python -m generate --ckpt_dir models/CodeLlama-7b-Instruct/
loaded SentencePiece model: #words: 32016 - bos id: 1 - eos id: 2
loaded model in 12.36 seconds
> [INST]abc[/INST]
I'm not sure I understand what you are saying with "abc". Could you explain?
---------------
> [INST]can you write a hello world program in C#[/INST]
 Certainly! Here is a simple "Hello, World!" program in C#:
` ` `
using System;

class HelloWorld
{
    static void Main(string[] args)
    {
        Console.WriteLine("Hello, World!");
    }
}

[...]

$ # run a Code Llama model with model parallelism > 1, assuming you have two GPUs
$ torchrun --nnodes 1 --nproc-per-node 2 -m generate --ckpt_dir models/CodeLlama-13b-Instruct/
[...]

Requirements (`docs/requirements.txt`)

recommonmark==0.5.0
docutils==0.17.1
sphinx==5.0.0
git+https://github.com/pytorch/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme
torch>=1.6.0
numpy>=1.19.5
pyre-extensions==0.0.29
jinja2==3.1.6
einops

Source Index (`docs/source/index.rst`)

Welcome to xFormers's documentation!

xFormers is a PyTorch based library which hosts flexible Transformers parts.
They are interoperable and optimized building blocks, which can optionally be combined
to create some state of the art models.

.. toctree::
:maxdepth: 2
:caption: Index
:hidden:

what_is_xformers

|
|

.. toctree::
:maxdepth: 2
:caption: Components Documentation

components/index

|
|

Source What Is Xformers (`docs/source/what_is_xformers.rst`)

What is xFormers?

Flexible Transformers, defined by interoperable and optimized building blocks.

.. image:: _static/logo.png
:width: 700px
:height: 205px
:align: center

xFormers is focused on the following values

Field agnostic. This library is not focused on any given field, by design.
Composable. Ideally, break all the Transformer inspired models into a block zoo, which allows you to compose reference models but also study ablations or architecture search.
Extensible. xFormers aims at being easy to extend locally, so that one can focus on a specific improvement, and easily compare it against the state of the art.
Optimized. Reusing building blocks across domains means that engineering efforts can be more valued. And since you cannot improve what you cannot measure, xFormers is benchmark-heavy.
Tested. Each and every of the variant in the repo is tested, alone and composed with the other relevant blocks. This happens automatically anytime somebody proposes a new variant through a PR.
Crowd Sourced. PRs are really welcome, the state of the art is moving too fast for anything but a crowd sourced effort.

Source Components Index (`docs/source/components/index.rst`)

API Reference

.. toctree::
:maxdepth: 2

ops

Source Components Ops (`docs/source/components/ops.rst`)

xFormers optimized operators

Memory-efficient attention

.. automodule:: xformers.ops
:members: memory_efficient_attention, AttentionOpBase
:show-inheritance:
:imported-members:

Available implementations

text

.. automodule:: xformers.ops.fmha.cutlass
    :members: FwOp, BwOp
    :member-order: bysource

.. automodule:: xformers.ops.fmha.flash
    :members: FwOp, BwOp
    :member-order: bysource

.. automodule:: xformers.ops.fmha.small_k
    :members: FwOp, BwOp
    :member-order: bysource

.. automodule:: xformers.ops.fmha.ck
    :members: FwOp, BwOp
    :member-order: bysource

.. automodule:: xformers.ops.fmha.ck_decoder
    :members: FwOp
    :member-order: bysource

.. automodule:: xformers.ops.fmha.ck_splitk
    :members: FwOp
    :member-order: bysource

Attention biases
~~~~~~~~~~~~~~~~~~~~

.. automodule:: xformers.ops.fmha.attn_bias
    :members:
    :show-inheritance:
    :member-order: bysource

Partial Attention
~~~~~~~~~~~~~~~~~~~~

.. automodule:: xformers.ops.fmha
    :members: memory_efficient_attention_partial, merge_attentions
    :member-order: bysource


Non-autograd implementations

.. automodule:: xformers.ops.fmha
:members: memory_efficient_attention_forward, memory_efficient_attention_forward_requires_grad, memory_efficient_attention_backward
:show-inheritance:
:imported-members:
:member-order: bysource

--- METRICS ---

Files Extracted: 7
Estimated Token Budget: ~2629 tokens
Canonical Reference: https://codewiki.google/github.com/facebookresearch/xformers

Technical Documentation: facebookresearch/xformers

Overview (`README.md`) Copy Section

xFormers - Toolbox to Accelerate Research on Transformers Copy Chapter

Installing xFormers Copy Chapter

Benchmarks Copy Chapter

(Optional) Testing the installation Copy Section

Using xFormers Copy Chapter

Key Features Copy Section

Install troubleshooting Copy Section

License Copy Section

Citing xFormers Copy Chapter

Credits Copy Chapter

Overview (`examples/llama_inference/README.md`) Copy Section

Llama inference

Requirements (`docs/requirements.txt`) Copy Section

Source Index (`docs/source/index.rst`) Copy Section

Welcome to xFormers's documentation!

Source What Is Xformers (`docs/source/what_is_xformers.rst`) Copy Section

What is xFormers?

Source Components Index (`docs/source/components/index.rst`) Copy Section

API Reference

Source Components Ops (`docs/source/components/ops.rst`) Copy Section

xFormers optimized operators

Memory-efficient attention Copy Chapter

Explore Related Technologies & Topics

Overview (`README.md`)

xFormers - Toolbox to Accelerate Research on Transformers

Installing xFormers

Benchmarks

(Optional) Testing the installation

Using xFormers

Key Features

Install troubleshooting

License

Citing xFormers

Credits

Overview (`examples/llama_inference/README.md`)

Requirements (`docs/requirements.txt`)

Source Index (`docs/source/index.rst`)

Source What Is Xformers (`docs/source/what_is_xformers.rst`)

Source Components Index (`docs/source/components/index.rst`)

Source Components Ops (`docs/source/components/ops.rst`)

Memory-efficient attention