## File: README.md
# **👋 Welcome**
Composer is an open-source deep learning training library by [MosaicML](https://www.mosaicml.com/). Built on top of PyTorch, the Composer library makes it easier to implement distributed training workflows on large-scale clusters.
We built Composer to be **optimized for scalability and usability**, integrating best practices for efficient, multi-node training. By abstracting away low-level complexities like parallelism techniques, distributed data loading, and memory optimization, you can focus on training modern ML models and running experiments without slowing down.
We recommend using Composer to speedup your experimentation workflow if you’re training neural networks of any size, including:
- Large Language Models (LLMs)
- Diffusion models
- Embedding models (e.g. BERT)
- Transformer-based models
- Convolutional Neural Networks (CNNs)
Composer is heavily used by the MosaicML research team to train state-of-the-art models like MPT, and we open-sourced this library to enable the ML community to do the same. This framework is used by organizations in both the tech industry and the academic sphere and is continually updated with new features, bug fixes, and stability improvements for production workloads.
# **🔑 Key Features**
We designed Composer from the ground up for modern deep learning workloads. Gone are the days of AlexNet and ResNet, when state-of-the-art models could be trained on a couple of desktop GPUs. Today, developing the latest and greatest deep learning models often requires cluster-scale hardware — but with Composer’s help, you’ll hardly notice the difference.
The heart of Composer is our Trainer abstraction: a highly optimized PyTorch training loop designed to allow both you and your model to iterate faster. Our trainer has simple ways for you to configure your parallelization scheme, data loaders, metrics, loggers, and more.
## Scalability
Whether you’re training on 1 GPU or 512 GPUs, 50MB or 10TB of data - Composer is built to keep your workflow simple.
- [**FSDP**](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html#fullyshardeddataparallel-fsdp): For large models that are too large to fit on GPUs, Composer has integrated PyTorch [FullyShardedDataParallelism](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html#fullyshardeddataparallel-fsdp) into our trainer and made it simple to efficiently parallelize custom models. We’ve found FSDP is competitive performance-wise with much more complex parallelism strategies. Alternatively, Composer also supports standard PyTorch distributed data parallelism (DDP) execution.
- [**Elastic sharded checkpointing**](https://docs.mosaicml.com/projects/composer/en/stable/notes/distributed_training.html#saving-and-loading-sharded-checkpoints-with-fsdp): Save on eight GPUs, resume on sixteen. Composer supports elastic sharded checkpointing, so you never have to worry if your sharded saved state is compatible with your new hardware setup.
- **Data streaming:** Working with large datasets? Download datasets from cloud blob storage on the fly by integrating with MosaicML [StreamingDataset](https://github.com/mosaicml/streaming) during model training.
## Customizability
Other high-level deep learning trainers provide simplicity at the cost of rigidity. When you want to add your own features, their abstractions get in your way. Composer, on the other hand, provides simple ways for you to customize our Trainer to your needs.
***Fig. 1:** Composer’s training loop has a series of events that occur at each stage in the training process. Callbacks are functions that users write to run at specific events. For example, our [Learning Rate Monitor Callback](https://docs.mosaicml.com/projects/composer/en/stable/api_reference/generated/composer.callbacks.LRMonitor.html#composer.callbacks.LRMonitor) logs the learning rate at every BATCH_END event.*
- [**Callbacks**](https://docs.mosaicml.com/projects/composer/en/stable/trainer/callbacks.html): Composer’s callback system allows you to insert custom logic at any point in the training loop. We’ve written callbacks to monitor memory usage, log and visualize images, and estimate your model’s remaining training time, to name a few. This feature is popular among researchers who want to implement and experiment with custom training techniques.
- [**Speedup algorithms**](https://docs.mosaicml.com/projects/composer/en/stable/examples/custom_speedup_methods.html): We draw from the latest research to create a collection of algorithmic speedups. Stack these speedups into MosaicML recipes to boost your training speeds. Our team has open-sourced the optimal combinations of speedups for different types of models.
- **8x speedup: Stable Diffusion**
- $200k original SD2 cost —> $50k ([Blog](https://www.mosaicml.com/blog/diffusion))
- **7x speedup: ResNet-50 on ImageNet**
- 3h33m —> 25m on 8xA100 ([Blog](https://www.mosaicml.com/blog/mosaic-resnet))
- **8.8x speedup: BERT-Base Pretraining**
- 10h —> 1.13h on 8xA100 ([Blog](https://www.mosaicml.com/blog/mosaicbert))
- **5.4x speedup: DeepLab v3 on ADE20K**
- 3h30m —> 39m on 8xA100 ([Blog](https://www.mosaicml.com/blog/behind-the-scenes))
## Better workflows
Composer is built to automate away low-level pain points and headaches so you can focus on the important (and fun) parts of deep learning and iterate faster.
- [**Auto-resumption**](https://docs.mosaicml.com/projects/composer/en/stable/notes/resumption.html): Failed training run? Have no fear — just re-run your code, and Composer will automatically resume from your latest saved checkpoint.
- [**CUDA OOM Prevention**](https://docs.mosaicml.com/projects/composer/en/stable/examples/auto_microbatching.html): Say goodbye to out-of-memory errors. Set your microbatch size to “auto”, and Composer will automatically select the biggest one that fits on your GPUs.
- **[Time Abstractions](https://docs.mosaicml.com/projects/composer/en/latest/trainer/time.html):** Ever messed up your conversion between update steps, epochs, samples, and tokens? Specify your training duration with custom units (epochs, batches, samples, and tokens) in your training loop with our `Time` class.
## Integrations
Integrate with the tools you know and love for experiment tracking and data streaming.
- **Cloud integrations**: Our Checkpointing and logging features have first-class support for remote storage and loading from Cloud bucket (OCI, GCP, AWS S3).
- **********Experiment tracking:********** Weights and Biases, MLFlow, CometML, and neptune.ai — the choice is yours, easily log your data to your favorite platform.
# **🚀 Getting Started**
## **📍**Prerequisites
Composer is designed for users who are comfortable with Python and have basic familiarity with deep learning fundamentals and PyTorch.
**********************************************Software requirements:********************************************** A recent version of PyTorch.
**********************************************Hardware requirements:********************************************** System with CUDA-compatible GPUs (AMD + RoCM coming soon!). Composer can run on CPUs, but for full benefits, we recommend using it on hardware accelerators.
## **💾 Installation**
Composer can be installed with `pip`:
```bash
pip install mosaicml
```
To simplify the environment setup for Composer, we also provide a set of [pre-built Docker images](https://docs.mosaicml.com/projects/composer/en/stable/getting_started/installation.html#docker). We *highly recommend* you use our Docker images.
## **🏁 Quick Start**
Here is a code snippet demonstrating our Trainer on the MNIST dataset.
```python
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
from composer import Trainer
from composer.models import ComposerClassifier
from composer.algorithms import LabelSmoothing, CutMix, ChannelsLast
class Model(nn.Module):
"""Toy convolutional neural network architecture in pytorch for MNIST."""
def __init__(self, num_classes: int = 10):
super().__init__()
self.num_classes = num_classes
self.conv1 = nn.Conv2d(1, 16, (3, 3), padding=0)
self.conv2 = nn.Conv2d(16, 32, (3, 3), padding=0)
self.bn = nn.BatchNorm2d(32)
self.fc1 = nn.Linear(32 * 16, 32)
self.fc2 = nn.Linear(32, num_classes)
def forward(self, x):
out = self.conv1(x)
out = F.relu(out)
out = self.conv2(out)
out = self.bn(out)
out = F.relu(out)
out = F.adaptive_avg_pool2d(out, (4, 4))
out = torch.flatten(out, 1, -1)
out = self.fc1(out)
out = F.relu(out)
return self.fc2(out)
transform = transforms.Compose([transforms.ToTensor()])
dataset = datasets.MNIST("data", train=True, download=True, transform=transform)
train_dataloader = DataLoader(dataset, batch_size=128)
trainer = Trainer(
model=ComposerClassifier(module=Model(), num_classes=10),
train_dataloader=train_dataloader,
max_duration="2ep",
algorithms=[
LabelSmoothing(smoothing=0.1),
CutMix(alpha=1.0),
ChannelsLast(),
],
)
trainer.fit()
```
Next, check out our [Getting Started Colab](https://colab.research.google.com/github/mosaicml/composer/blob/9f594876f957c912758e540598ac9f47a468c39d/examples/getting_started.ipynb) for a walk-through of Composer’s main features. In this tutorial, we will cover the basics of the Composer Trainer:
- Dataloader
- Trainer
- Optimizer and Scheduler
- Logging
- Training a baseline model
- Speeding up training
## **📚 Learn more**
Once you’ve completed the Quick Start, you can go through the below tutorials or our [documentation](https://docs.mosaicml.com/projects/composer/en/stable/) to further familiarize yourself with Composer.
If you have any questions, please feel free to reach out to us on our [Community Slack](https://dub.sh/mcomm)!
Here are some resources actively maintained by the Composer community to help you get started:
| **Resource** | **Details** |
| --- | --- |
| [Training BERTs with Composer and 🤗 ](https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/finetune_huggingface.ipynb) | A Colab Notebook showing how to train BERT models with Composer and 🤗! |
| [Pretraining and Finetuning an LLM Tutorial](https://github.com/mosaicml/llm-foundry/blob/main/TUTORIAL.md) | A tutorial from MosaicML’s LLM Foundry, using MosaicML Composer, StreamingDataset, and MCLI on training and evaluating LLMs. |
| [Migrating from PyTorch Lightning](https://docs.mosaicml.com/projects/composer/en/stable/examples/migrate_from_ptl.html) | A tutorial is to illustrating a path from working in PyTorch Lightning to working in Composer. |
| [Finetuning and Pretraining HuggingFace Models](https://docs.mosaicml.com/projects/composer/en/stable/examples/finetune_huggingface.html) | Want to use Hugging Face models with Composer? No problem. Here, we’ll walk through using Composer to fine-tune a pretrained Hugging Face BERT model. |
| [Building Speedup Methods](https://colab.research.google.com/github/mosaicml/composer/blob/dev/examples/custom_speedup_methods.ipynb) | A Colab Notebook showing how to build new training modifications on top of Composer |
# 🛠️ For Best Results, Use within the Databricks & MosaicML Ecosystem
Composer can be used on its own, but for the smoothest experience we recommend using it in combination with other components of the MosaicML ecosystem:
- [**Mosaic AI training**](https://www.databricks.com/product/machine-learning/mosaic-ai-training) (MCLI)- Our proprietary Command Line Interface (CLI) and Python SDK for orchestrating, scaling, and monitoring the GPU nodes and container images executing training and deployment. Used by our customers for training their own Generative AI models.
- **To get started, [reach out here](https://www.databricks.com/company/contact) and check out our [Training](https://www.databricks.com/product/machine-learning/mosaic-ai-training) product pages**
- [**MosaicML LLM Foundry**](https://github.com/mosaicml/llm-foundry) - This open source repository contains code for training, finetuning, evaluating, and preparing LLMs for inference with [Composer](https://github.com/mosaicml/composer). Designed to be easy to use, efficient and flexible, this codebase is designed to enable rapid experimentation with the latest techniques.
- [**MosaicML StreamingDataset**](https://github.com/mosaicml/streaming) - Open-source library for fast, accurate streaming from cloud storage.
- [**MosaicML Diffusion**](https://github.com/mosaicml/diffusion) - Open-source code to train your own Stable Diffusion model on your own data. Learn more via our blogs: ([Results](https://www.mosaicml.com/blog/stable-diffusion-2) , [Speedup Details](https://www.mosaicml.com/blog/diffusion))
# **🏆 Project Showcase**
Here are some projects and experiments that used Composer. Got something to add? Share in our [Community Slack](https://dub.sh/mcomm)!
- [**MPT Foundation Series:**](https://www.mosaicml.com/mpt) Commercially usable open source LLMs, optimized for fast training and inference and trained with Composer.
- [MPT-7B Blog](https://www.mosaicml.com/blog/mpt-7b)
- [MPT-7B-8k Blog](https://www.mosaicml.com/blog/long-context-mpt-7b-8k)
- [MPT-30B Blog](https://www.mosaicml.com/blog/mpt-30b)
- [**Mosaic Diffusion Models**](https://www.mosaicml.com/blog/training-stable-diffusion-from-scratch-costs-160k): see how we trained a stable diffusion model from scratch for <$50k
- [**replit-code-v1-3b**](https://huggingface.co/replit/replit-code-v1-3b): A 2.7B Causal Language Model focused on **Code Completion,** trained by Replit on Mosaic AI training in 10 days.
- **BabyLLM:** the first LLM to support both Arabic and English. This 7B model was trained by MetaDialog on the world’s largest Arabic/English dataset to improve customer support workflows ([Blog](https://blogs.nvidia.com/blog/2023/08/31/generative-ai-startups-africa-middle-east/))
- [**BioMedLM**](https://www.mosaicml.com/blog/introducing-pubmed-gpt): a domain-specific LLM for Bio Medicine built by MosaicML and [Stanford CRFM](https://crfm.stanford.edu/)
# 💫 Contributors
Composer is part of the broader Machine Learning community, and we welcome any contributions, pull requests, or issues!
To start contributing, see our [Contributing](https://github.com/mosaicml/composer/blob/dev/CONTRIBUTING.md) page.
P.S.: [We're hiring](https://www.databricks.com/company/careers/open-positions?department=Mosaic%20AI&location=all)!
# ❓FAQ
- **What is the best tech stack you recommend when training large models?**
- We recommend that users combine components of the MosaicML ecosystem for the smoothest experience:
- Composer
- [StreamingDataset](https://github.com/mosaicml/streaming)
- [MCLI](https://www.databricks.com/product/machine-learning/mosaic-ai-training) (Databricks Mosaic AI Training)
- **How can I get community support for using Composer?**
- You can join our [Community Slack](https://dub.sh/mcomm)!
- **How does Composer compare to other trainers like NeMo Megatron and PyTorch Lightning?**
- We built Composer to be optimized for both simplicity and efficiency. Community users have shared that they enjoy Composer for its capabilities and ease of use compared to alternative libraries.
- **How do I use Composer to train graph neural networks (GNNs), or Generative Adversarial Networks (GANs), or models for reinforcement learning (RL)?**
- We recommend you use alternative libraries for if you want to train these types of models - a lot of assumptions we made when designing Composer are suboptimal for GNNs, RL, and GANs
- **How can I speed up HuggingFace downloads?
- You can use hf transfer (`pip install hf-transfer`) and set the environment variable `HF_HUB_ENABLE_HF_TRANSFER=1`
# ✍️ Citation
```
@misc{mosaicml2022composer,
author = {The Mosaic ML Team},
title = {composer},
year = {2021},
howpublished = {\url{https://github.com/mosaicml/composer/}},
}
```
---
## File: composer/algorithms/alibi/README.md
# 🥸 ALiBi
[\[How to Use\]](#how-to-use) - [\[Suggested Hyperparameters\]](#suggested-hyperparameters) - [\[Technical Details\]](#technical-details) - [\[Attribution\]](#attribution) - [\[API Reference\]](#api-reference)
`Natural Language Processing`
ALiBi (Attention with Linear Biases) dispenses with position embeddings for tokens in transformer-based NLP models, instead encoding position information by biasing the query-key attention scores proportionally to each token pair’s distance. ALiBi yields excellent extrapolation to unseen sequence lengths compared to other position embedding schemes. We leverage this extrapolation capability by training with shorter sequence lengths, which reduces the memory and computation load.
| |
|:--:
|*The matrix on the left depicts the attention score for each key-query token pair. The matrix on the right depicts the distance between each query-key token pair. m is a head-specific scalar that is fixed during training. Figure from [Press et al., 2021](https://openreview.net/forum?id=R8sQPpGCv0).*|
## How to Use
### Functional Interface
```python
# Run the ALiBi algorithm directly on the model using the Composer functional API
import torch
import composer.functional as cf
def training_loop(model, train_loader):
cf.apply_alibi(
model=model,
max_sequence_length=1024,
)
opt = torch.optim.Adam(model.parameters())
loss_fn = F.cross_entropy
model.train()
for epoch in range(num_epochs):
for X, y in train_loader:
y_hat = model(X)
loss = loss_fn(y_hat, y)
loss.backward()
opt.step()
opt.zero_grad()
```
### Composer Trainer
```python
# Instantiate the algorithm and pass it into the Trainer
# The trainer will automatically run it at the appropriate points in the training loop
from composer.algorithms import Alibi
from composer.trainer import Trainer
alibi = Alibi(
max_sequence_length=1024,
train_sequence_length_scaling=0.25,
)
trainer = Trainer(
model=model,
train_dataloader=train_dataloader,
eval_dataloader=eval_dataloader,
max_duration='1ep',
algorithms=[alibi]
)
trainer.fit()
```
### Implementation Details
ALiBi is implemented as follows. On `Event.INIT`:
1. The model's position embeddings are expanded to accommodate sequences of up to length `max_sequence_length` and then "bypassed" by setting them to zero and freezing them.
2. The attribute that computes the self-attention in the model's self-attention modules is replaced with an ALiBi-enabled self-attention method using graph surgery. Our implementation builds a registry that maps module types to their graph surgery functions. **Note:** you may need to add to the registry if your model's self-attention and embedding modules are not included in the registry (see "Supported Models" below).
3. On `Event.AFTER_DATALOADER`, the length of training data sequences in a batch are scaled by `train_sequence_length_scaling` by reshaping the data tensors.
## Supported Models
Our current implementation of ALiBi provides out-of-the-box support for HuggingFace BERT (and RoBERTa) and HuggingFace GPT2 models. This support extends to any models that are superclasses of these models (e.g., the BERT and GPT2 models that can be created using `composer.models.create_bert_mlm` and `composer.models.create_gpt2`, respectively, both of which return instances of a `composer.models.huggingface.HuggingFaceModel` class).
Please be aware that if you use ALiBi with an unsupported model type a logger warning will be generated but no error will be raised. Make sure to check the logs if you are uncertain whether your model is supported by ALiBi.
You can add your own ALiBi implementation by extending the `policy_registry` that defines the "policy" that maps source module types (e.g., `transformers.models.bert.BertSelfAttention`) to their respective "replacement functions" (functions that modify instances of the source module or return new modules to replace them). Please see the documentation for `composer.algorithms.alibi.attention_surgery_functions.utils.PolicyRegistry` for details on the requirements of such implementations and specific examples. The following example demonstrates the basic pattern:
```python
# Example for adding Alibi surgery functions to support a custom transformer model
import torch
import composer.functional as cf
# MyTransformer is the model class. Its modules include instances of EmbeddingLayer and AttentionLayer
from my_custom_models import MyTransformer, EmbeddingLayer, AttentionLayer
# Import the alibi policy registry
from composer.algorithms.alibi.attention_surgery_functions import policy_registry
# Register a surgery function for handling position embeddings in EmbeddingLayer instances
@policy_registry.register(EmbeddingLayer)
def embedding_surgery(module: torch.nn.Module, module_index: int, max_sequence_length: int) -> torch.nn.Module:
# Define code here that augments `module` (an instance of EmbeddingLayer) such that:
# - position embeddings are not used
# - any sequence-sized buffers are rebuilt to support max_sequence_length sized sequences
...
return module # Return the augmented `module`
# Register a surgery function for using Alibi biases to attention in AttentionLayer instances
@policy_registry.register(AttentionLayer)
def attention_surgery(module: torch.nn.Module, module_index: int, max_sequence_length: int) -> torch.nn.Module:
# Define code here that augments `module` (an instance of AttentionLayer) such that:
# - the module has a buffer of Alibi-style attention biases, which are added to the attention logits before applying softmax
# - any sequence-sized buffers are rebuilt to support max_sequence_length sized sequences
...
return module # Return the augmented `module`
# Alibi will now support MyTransformer (and any other model that uses the EmbeddingLayer and AttentionLayer modules)
model = MyTransformer()
cf.apply_alibi(model, max_sequence_length=1024)
```
## Suggested Hyperparameters
We found that `train_sequence_length_scaling=0.25` (sequence length 256) provides appreciable speed and accuracy gains for models evaluated with sequence length 1024.
We observe that performance significantly degrades for ALiBi models trained on sequence lengths ≤128.
As such, we do not recommend training models with sequence lengths ≤256 or `train_sequence_length_scaling≤0.03125`, whichever is larger.
## Technical Details
ALiBi dispenses with traditional position embeddings and instead adds a static, non-learned bias to the query-key attention scores (or attention weights). This bias is proportional to the distance between the query and key tokens that comprise each attention score. The distances are scaled by *m*, a head-specific scalar that is fixed during training.
Press et al. found that learning *m* did not lead to strong extrapolation. They instead set *m = (4(log2 H + 3)-1)-h* where *H* is the number of attention heads in a layer and *h* is the index of the current head.
Press et al. report that models trained with ALiBi maintain similar performance even when tested on sequences 5-10x longer than they were trained on. ALiBi’s extrapolation capabilities can be leveraged to train on shorter sequences. This is desirable because the number of operations required to compute self-attention and the GPU memory usage required to store the resulting representations both increase with the square of the sequence length. In one example scenario, Press et al. reported training to equal perplexity 90% of the time and utilizing 90% of the GPU memory compared to a baseline model with sinusoidal position embeddings. Our experiments show that ALiBi can reduce perplexity by 0.2-0.6, train models 1.15x faster, and utilize 1.2x less GPU memory compared to baseline models (see below).
> ✅ ALiBi Improves the Tradeoff Between Quality and Training Speed
>
> In our experiments, ALiBi improves the attainable tradeoffs between training speed and the final quality of the trained model.
> We recommend ALiBi for training language models.
We conducted experiments on the GPT-2 model family trained on OpenWebText on 8x NVIDIA A100-40GBs. We compared baseline models with learned position embeddings and training sequence length 1024 to models using ALiBi with `train_sequence_length_scaling=0.25` (i.e., train sequence length 256). We found that `train_sequence_length_scaling=0.25` (sequence length 256) provides appreciable speed and accuracy gains for models evaluated at sequence length 1024. Our results are shown in the table below.
|Name|Perplexity| Δ|Train Time (s)|Speedup|GPU Memory|Reduction|
|:-|:-:|:-:|:-:|:-:|:-:|:-:|
|GPT2-52M|30.78||9801||92.91%||
|GPT2-52M ALiBi 0.25x|30.54|-0.24|8411|1.16x|79.79|1.16x|
|GPT2-83M|26.57||17412||97.04||
|GPT2-83M ALiBi 0.25x|26.19|-0.38|14733|1.18x|80.97|1.20x|
|GPT2-125M|24.11||30176||95.96||
|GPT2-125M ALiBi 0.25x|23.49|-0.63|25280|1.19x|74.83|1.28x|
> ❗ Don't Set the Sequence Length Too Short
>
>We observed that performance significantly degraded for ALiBi models trained on sequence lengths ≤128, implying that very short sequences (≤128 tokens) may be irreconcilably out-of-distribution with regard to longer sequences. Considering our results together with those of Press et al. leads us to suggest that models with ALiBi should not be trained on sequences ≤256 or `train_sequence_length_scaling≤0.03125`, whichever is larger.
## Attribution
[*Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation*](https://openreview.net/forum?id=R8sQPpGCv0) by Ofir Press, Noah A. Smith, and Mike Lewis. Published in ICLR 2022.
*The Composer implementation of this method and the accompanying documentation were produced by Matthew Leavitt and Alex Trott at MosaicML.*
## API Reference
**Algorithm class:** {class}`composer.algorithms.Alibi`
**Functional:** {func}`composer.functional.apply_alibi`
---
## File: composer/algorithms/augmix/README.md
# 🎨 AugMix
[\[How to Use\]](#how-to-use) - [\[Suggested Hyperparameters\]](#suggested-hyperparameters) - [\[Technical Details\]](#technical-details) - [\[Attribution\]](#attribution) - [\[API Reference\]](#api-reference)
`Computer Vision`
For each data sample, AugMix creates an _augmentation chain_ by sampling `depth` image augmentations from a set (e.g. translation, shear, contrast).
It then applies these augmentations sequentially with randomly sampled intensity.
This is repeated `width` times in parallel to create `width` different augmented images.
The augmented images are then combined via a random convex combination to yield a single augmented image, which is in turn combined via a random convex combination sampled from a Beta(`alpha`, `alpha`) distribution with the original image.
Training in this fashion regularizes the network and can improve generalization performance.
| |
|:--:
|*An image of a turtle that undergoes three different augmentation chains is pieced together using a convex combination and combined with the original image. [Figure 4 from Hendrycks et al. (2020)](https://arxiv.org/abs/1912.02781).*|
## How to Use
### Functional Interface
```python
# Run augmix on the image to produce a new augmixed image
from typing import Union
import torch
from PIL.Image import Image as PillowImage
import composer.functional as cf
from composer.algorithms.utils import augmentation_sets
def augmix_image(image: Union[PillowImage, torch.Tensor]):
augmixed_image = cf.augmix_image(
img=image,
severity=3,
width=3,
depth=-1,
alpha=1.0,
augmentation_set=augmentation_sets["all"]
)
return augmixed_image
```
### Torchvision Transform
```python
# Create a callable for AugmentAndMix which can be composed with other image augmentations
import torchvision.transforms as transforms
from torchvision.datasets.vision import VisionDataset
from composer.algorithms.augmix import AugmentAndMixTransform
augmix_transform = AugmentAndMixTransform(severity=3,
width=3,
depth=-1,
alpha=1.0,
augmentation_set="all")
composed = transforms.Compose([augmix_transform, transforms.RandomHorizontalFlip()])
dataset = VisionDataset(data_path, transform=composed)
```
### Composer Trainer
```python
# Instantiate the algorithm and pass it into the Trainer
# The trainer will automatically run it at the appropriate points in the training loop
from composer.algorithms import AugMix
from composer.trainer import Trainer
augmix_algorithm = AugMix(severity=3,
width=3,
depth=-1,
alpha=1.0,
augmentation_set="all")
trainer = Trainer(
model=model,
train_dataloader=train_dataloader,
eval_dataloader=eval_dataloader,
max_duration="1ep",
algorithms=[augmix_algorithm],
)
trainer.fit()
```
### Implementation Details
AugMix leverages `torchvision.transforms` to add a transformation to the dataset which will be applied per image on the CPU. The transformation takes in a `PIL.Image` and outputs a `PIL.Image` with AugMix applied.
The functional form of AugMix (`augmix_image()`) requires AugMix hyperparameters when it is called.
The Torchvision transform form of AugMix (`AugmentAndMixTransform`) is composable with other dataset transformations via `torchvision.transforms.Compose`.
The class form of AugMix runs on `Event.FIT_START` and inserts `AugmentAndMixTransform` into the set of transforms in a `torchvision.datasets.VisionDataset` dataset.
## Suggested Hyperparameters
[As per Hendrycks et al. (2020)](https://arxiv.org/abs/1912.02781), we found that `width=3`, `depth=-1`, (`depth=-1` means that depth will be randomly sampled from the uniform distribution {1, 2, 3} for each data sample), `severity=3` (out of a maximum possible value of 10), and `alpha=1` (i.e., performing no mixing with the original image) worked well for different models of the ResNet family. We used `augmentation_set=all`.
> ❗ Potential CPU Bottleneck
>
> Further increasing `width` or `depth` significantly decreases throughput when training ResNet-50 on ImageNet due to bottlenecks in performing data augmentation on the CPU.
## Technical Details
AugMix randomly samples `depth` image augmentations (with replacement) from the set of {`translate_x`, `translate_y`, `shear_x`, `shear_y`, `rotate`, `solarize`, `equalize`, `posterize`, `autocontrast`, `color`, `brightness`, `contrast`, `sharpness`}.
The augmentations use the PILLOW Image library (specifically Pillow-SIMD); we found that OpenCV-based augmentations result in similar or worse performance.
AugMix is applied after "standard" image transformations, such as resizing and cropping, and before normalization.
Each augmentation is applied with an intensity that is randomly sampled uniformly from \[0.1-`severity`\] (`severity` ≤ 10). where `severity` is a unit-free upper bound on the intensity of an augmentation and is mapped to the unit specific for each augmentation. For example, `severity` would be mapped to degrees for the rotation augmentation with `severity=10` corresponding to 30°.
Hendrycks et al.’s original implementation of AugMix also includes a custom loss function computed across three samples (an image and two AugMix’d versions of that image).
We omit this custom loss function from our AugMix implementation because it effectively triples the number of samples required for a parameter update step, imposing a significant computational burden.
Our implementation, which consists only of the augmentation component of AugMix, is referred to by Hendrycks et al. as "AugmentAndMix."
> ❗ AugMix Provided Limited Benefits in Our Experiments
>
> We found that using AugMix with the hyperparameters recommended by Hendrycks et al. can increase the data augmentation load on the CPU so much that it bottlenecks training.
> Depending on the hardware configuration and model, we found that those hyperparameters increase training time by 1.1x-10x.
Hendrycks et al. report a 13.8% accuracy improvement on CIFAR-10C (a benchmark for corruption robustness) over baseline for a 40-2 Wide ResNet and a 1.5-10% improvement over other augmentation schemes.
Hendrycks et al. also report a 1.5% improvement over a baseline ResNet-50 on ImageNet, but this result uses AugMix in combination with the aforementioned custom loss function.
When omitting the custom loss function and using the AugMix augmentation scheme alone, we observe an accuracy gain of about 0.5% over a baseline ResNet-50 on ImageNet.
However, the increased CPU load imposed by AugMix substantially reduces throughput.
AugMix will be more useful in overparameterized regimes (i.e. larger models) and for longer training runs.
Larger models typically take longer to run on a deep learning accelerator (e.g., a GPU), meaning there is more headroom to perform work on the CPU before augmentation becomes a bottleneck.
In addition, AugMix is a regularization technique, meaning it makes training more difficult.
Doing so can allow models to reach higher quality, but this typically requires (1) larger models with more capacity to perform this more difficult learning and (2) longer training runs to allow these models time to learn.
> 🚧 AugMix May Reduce Quality for Smaller Models and Shorter Training Runs
>
> AugMix is a regularization technique that makes training more difficult for the model.
> Because AugMix is a regularization technique, it can allow models to reach higher quality for
>
> (1) longer training runs and...
>
> (2) overparameterized models
>
> However, for shorter training runs or smaller models it may reduce quality.
> 🚧 Composing Regularization Methods
>
> As general rule, composing regularization methods may lead to diminishing returns in quality improvements while increasing the risk of creating a CPU bottleneck.
> ❗ CIFAR-10C and ImageNet-C are no longer out-of-distribution
>
> [CIFAR-10C and ImageNet-C](https://github.com/hendrycks/robustness) are test sets created to evaluate the ability of models to generalize to images that are corrupted in various ways (i.e., images that are _out-of-distribution_ with respect to the standard CIFAR-10 and ImageNet training sets).
> These images were corrupted using some of the augmentation techniques in `augmentation_set=all`.
> If you use `augmentation_set=all`, these images are therefore no longer out-of-distribution.
## Attribution
[*AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty*](https://arxiv.org/abs/1912.02781) by Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Published in ICLR 2020.
*The Composer implementation of this method and the accompanying documentation were produced by Matthew Leavitt at MosaicML.*
## API Reference
**Algorithm class:** {class}`composer.algorithms.AugMix`, {class}`composer.algorithms.AugmentAndMixTransform`
**Functional:** {func}`composer.functional.augmix_image`
---
## File: composer/algorithms/blurpool/README.md
# 🏊 BlurPool
[\[How to Use\]](#how-to-use) - [\[Suggested
Hyperparameters\]](#suggested-hyperparameters) - [\[Technical
Details\]](#technical-details) - [\[Attribution\]](#attribution) - [\[API Reference\]](#api-reference)
`Computer Vision`
BlurPool increases the accuracy of convolutional neural networks for computer vision, while maintaining
nearly the same speed, by applying a spatial low-pass filter before pooling operations and strided convolutions.
Doing so reduces [aliasing](https://en.wikipedia.org/wiki/Aliasing) when performing these operations.
| |
|:--:
|*A diagram of the BlurPool replacements (bottom row) for typical pooling and downsampling operations (top row) in convolutional neural networks. In each case, BlurPool applies a low-pass filter before the spatial downsampling to avoid aliasing. This image is Figure 2 in [Zhang (2019)](https://proceedings.mlr.press/v97/zhang19a.html).*|
## How to Use
### Functional Interface
```python
# Run the Blurpool algorithm directly on the model using the Composer functional API
import composer.functional as cf
import torch
import torch.nn.functional as F
def training_loop(model, train_loader):
opt = torch.optim.Adam(model.parameters())
# only need to pass in opt if apply_blurpool is used after optimizer
# creation; otherwise only the model needs to be passed in
cf.apply_blurpool(
model,
optimizers=opt,
replace_convs=True,
replace_maxpools=True,
blur_first=True
)
loss_fn = F.cross_entropy
model.train()
for epoch in range(10):
for X, y in train_loader:
y_hat = model(X)
loss = loss_fn(y_hat, y)
loss.backward()
opt.step()
opt.zero_grad()
```
### Composer Trainer
```python
# Instantiate the algorithm and pass it into the Trainer
# The trainer will automatically run it at the appropriate point in the training loop
from composer.algorithms import BlurPool
from composer.trainer import Trainer
blurpool = BlurPool(replace_convs=True,
replace_maxpools=True)
trainer = Trainer(model=model,
train_dataloader=train_dataloader,
eval_dataloader=eval_dataloader,
max_duration='1ep',
algorithms=[blurpool])
trainer.fit()
```
### Implementation Details
The Composer implementation of BlurPool uses model surgery to replace instances of pooling and downsampling operations with the BlurPool equivalents.
For max pooling, it replaces `torch.nn.MaxPool2d` instances with instances of a custom `nn.Module` subclass that decouples the computation of the max within a given spatial window from the pooling and adds a spatial low-pass filter in between. This change roughly doubles the data movement required for the op, although it shouldn’t add significant overhead unless there are many maxpools in the network. For convolutions, it replaces strided `torch.nn.Conv2d` instances (i.e., those where the stride is larger than 1) with a custom module class that 1) applies a low-pass filter to the input, and then 2) applies a copy of the original convolution operation.
🚧 Implementation Note
>
> Blurpool does not replace strided convolutions with fewer than `min_channels` input channels, which by default is set to `16`. This is a heuristic used to avoid blurpooling the network's input. Doing so is undesirable since it amounts to downsampling the input by more than the amount specified in the preprocessing pipeline.
## Suggested Hyperparameters
We suggest setting `blur_first=True` to avoid unnecessarily increasing computational cost.
We also suggest setting `blur_maxpools=True` to match the configuration in the original paper, but we haven’t observed a major effect of setting this to either `True` or `False`. We recommend always setting `blur_convs=True` since blurring strided convolutions seems to matter more than blurring maxpools. Note, however, that some models (such as ResNet-20 and other CIFAR ResNets) have no strided convolutions, so this argument may have no effect.
## Technical Details
The possible effects of BlurPool can be understood in several different ways:
1. It improves the network’s invariance to small spatial shifts.
2. It reduces aliasing in the downsampling operations.
3. It adds a structural bias towards preserving low-spatial-frequency components of neural network activations.
Consequently, it is likely to be useful on natural images or other inputs that change slowly over their spatial/time dimension(s).
Zhang (2019) showed that BlurPool improves accuracy by 0.5-1% on ImageNet for various networks.
[A follow-up paper by Zou et al.](https://maureenzou.github.io/ddac/) demonstrated similar improvements for ImageNet as well as significant improvements on instance segmentation on MS COCO and semantic segmentation metrics on PASCAL VOC2012 and Cityscapes.
[Lee et al.](https://arxiv.org/abs/2001.06268) also reproduced ImageNet accuracy improvements, especially when applying BlurPool only to strided convolutions.
Depending on the value of the `blur_first` parameter, the strided low-pass filtering can happen either before or after the convolution.
Setting `blur_first=True` (i.e., performing low-pass filtering before the convolution) keeps the number of multiply-add operations in the convolution itself constant, adding only the overhead of the low-pass filtering.
Setting `blur_first=False` (i.e., performing low-pass filtering after the convolution) increases the number of multiply-add operations by a factor of `np.prod(conv.stride)` (e.g., 4 for a stride of `(2, 2)`). This more closely matches the approach used in the paper. Anecdotally, we’ve observed this version yielding a roughly 0.1% larger accuracy gain on ResNet-50 + ImageNet in exchange for a ~10% slowdown. Having `blur_first=False` is not as well characterized in our experiments as `blur_first=True`.
> 🚧 Quality/Speed Tradeoff
>
> BlurPool leads to accurracy improvements but also slightly increases training time due to the additional operations it performs.
> On ResNet-50 on ImageNet, we found this tradeoff to be worthwhile: it is a pareto improvement over the standard versions of those benchmarks.
> We also found it to be worthwhile in composition with other methods.
> We recommend that you carefully evaluate whether BlurPool is also a pareto improvement in the context of your application.
Our implementation deviates from the original paper in that we apply the low-pass filter and pooling before the nonlinearity instead of after. This is because we have no reliable way of adding it after the nonlinearity in an architecture-agnostic way.
BlurPool tends to compose well with other methods. We are not aware of an example of its effects changing significantly as a result of other methods being present.
## Attribution
[*Making Convolutional Networks Shift-Invariant Again*](https://proceedings.mlr.press/v97/zhang19a.html) by Richard Zhang in ICML 2019.
*The Composer implementation of this method and the accompanying documentation were produced by Davis Blalock at MosaicML. We thank Richard Zhang for helpful discussion.*
## API Reference
**Algorithm class:** {class}`composer.algorithms.BlurPool`
**Functional:** {func}`composer.functional.apply_blurpool`
---
## File: composer/algorithms/channels_last/README.md
# 📺 Channels Last
[\[How to Use\]](#how-to-use) - [\[Suggested Hyperparameters\]](#suggested-hyperparameters) - [\[Technical Details\]](#technical-details) - [\[Attribution\]](#attribution) - [\[API Reference\]](#api-reference)
`Computer Vision`, `Math Equivalent`
Channels Last improves the throughput of convolution operations in networks for computer vision by changing the memory format of activation and weight tensors to contain channels as their last dimension (i.e., NHWC format) rather than the default format in which the height and width are the last dimensions (i.e., NCHW format).
NVIDIA GPUs natively perform convolution operations in NHWC format, so storing the tensors this way eliminates transpositions that would otherwise need to take place, increasing throughput.
This is a systems-level method that does not change the math or outcome of training in any way.
| |
|:--:
|*A diagram of a convolutional layer using the standard NCHW tensor memory layout (left) and the NHWC tensor memory layout (right). Fewer operations take place in NHWC format because the convolution operation is natively performed in NHWC format (right); in contrast, the NCHW tensor must be transposed to NHWC before the convolution and transposed back to NCHW after (right). This digram is from [NVIDIA](https://developer.nvidia.com/blog/tensor-core-ai-performance-milestones/).*|
## How to Use
### Functional Interface
```python
# Run the Channels Last algorithm directly on the model using the Composer functional API
import composer.functional as cf
def training_loop(model, train_loader):
cf.apply_channels_last(model)
opt = torch.optim.Adam(model.parameters())
loss_fn = F.cross_entropy
model.train()
for epoch in range(num_epochs):
for X, y in train_loader:
y_hat = model(X)
loss = loss_fn(y_hat, y)
loss.backward()
opt.step()
opt.zero_grad()
```
### Composer Trainer
```python
# Instantiate the algorithm and pass it into the Trainer
# The trainer will automatically run it at the appropriate points in the training loop
from composer.algorithms import ChannelsLast
from composer.trainer import Trainer
channels_last = ChannelsLast()
trainer = Trainer(
model=model,
train_dataloader=train_dataloader,
eval_dataloader=eval_dataloader,
max_duration='1ep',
algorithms=[channels_last]
)
trainer.fit()
```
### Implementation Details
Channels Last is implemented by converting the entire model to the channels last memory format at the beginning of training using `model.to(memory_format=torch.channels_last)`.
## Suggested Hyperparameters
Channels Last does not have any hyperparameters.
## Technical Details
At a high level, NVIDIA tensor cores require tensors to be in NHWC format in order to get the best performance, but PyTorch creates tensors in NCHW format.
Every time a convolution operation is called by a layer like `torch.nn.Conv2D`, the cuDNN library performs a transpose operation to convert the tensor into NHWC format. This transpose introduces overhead.
If the model weights are instead initialized in NHWC format, PyTorch will automatically convert the first input activation tensor to NHWC to match, and it will persist the memory format across all subsequent activations and gradients. This means that convolution operations no longer need to perform transposes, speeding up training.
We currently implement this method by casting the user’s model to channels-last format (no changes to the dataloader are necessary). When the first convolution operation receives its input activation, it will automatically convert it to NHWC format, after which the memory format will persist for the remainder of the network (or until it reaches a layer that cannot support having channels last).
> ✅ Channels Last Improves Training Speed
>
> In our experiments, Channels Last improves the attainable tradeoffs between training speed and the final quality of the trained model.
> We recommend Channels Last for training convolutional networks.
> ❗ Overhead from Operations Incompatible with Channels Last Memory Format
>
> If a model has layers that cannot support the channels last memory format, there will be overhead due to PyTorch switching activation tensors back and forth between NCHW and NHWC memory formats. We believe this problem currently affects placing channels last on UNet.
## Attribution
*The Composer implementation of this method and the accompanying documentation were produced by Abhi Venigalla at MosaicML.*
## API Reference
**Algorithm class:** {class}`composer.algorithms.ChannelsLast`
**Functional:** {func}`composer.functional.apply_channels_last`
---
## File: composer/algorithms/colout/README.md
# 🏛️ ColOut
[\[How to Use\]](#how-to-use) - [\[Suggested Hyperparameters\]](#suggested-hyperparameters) - [\[Technical Details\]](#technical-details) - [\[Attribution\]](#attribution) - [\[API Reference\]](#api-reference)
`Computer Vision`
ColOut is a data augmentation technique that drops a fraction of the rows or columns of an input image for a computer vision model.
If the fraction of rows/columns dropped isn't too large, the image content is not significantly altered but the image size is reduced, speeding up training.
This modification modestly reduces accuracy, but it is a worthwhile tradeoff for the increased speed.
| |
|:--:
|*Several instances of an image of an apple from the CIFAR-100 dataset with ColOut applied. ColOut randomly removes different rows and columns each time it is applied.*|
## How to Use
### Functional Interface
```python
# Run colout on a batch of images to produce a new batch
from typing import Union
import torch
from PIL.Image import Image as PillowImage
import composer.functional as cf
from composer.algorithms.utils import augmentation_sets
def colout_batch(image: Union[PillowImage, torch.Tensor]):
colout_batch = cf.colout_batch(
img=image,
p_row=0.15,
p_col=0.15
)
return colout_batch
```
### Torchvision Transform
Create a callable for ColOut which can be composed with other image augmentations
```python
from torchvision import transforms
from torchvision.datasets import VisionDataset
from composer.algorithms.colout import ColOutTransform
colout_transform = ColOutTransform(p_row=0.15, p_col=0.15)
composed = transforms.Compose([colout_transform, transforms.ToTensor()])
dataset = VisionDataset("data_path", transform=composed)
```
### Composer Trainer
```python
# Instantiate the algorithm and pass it into the Trainer
# The trainer will automatically run it at the appropriate points in the training loop
from composer.algorithms import ColOut
from composer.trainer import Trainer
colout = ColOut(
p_row=0.15,
p_col=0.15,
batch=True
)
trainer = Trainer(
model=model,
train_dataloader=train_dataloader,
eval_dataloader=eval_dataloader,
max_duration='1ep',
algorithms=[colout]
)
trainer.fit()
```
### Implementation Details
ColOut currently has two implementations.
One implementation, accessed by passing `batch=False`, acts as an additional data augmentation for use in PyTorch dataloaders. It runs on the CPU and applies ColOut independently to each training example.
A second implementation, accessed by passing `batch=True`, runs immediately before the training example is provided to the model. It runs on the GPU and drops the same rows and columns for all training examples in a mini-batch.
## Suggested Hyperparameters
We found that setting `p_row = 0.15` and `p_col = 0.15` strike a good balance between improving training throughput and limiting the negative impact on model accuracy. Setting `batch = True` also yields slightly lower accuracy, but we found that - in contexts that were CPU-bottlenecked - this reduction was offset by a large increase in throughput (~11% for ResNet-50 on ImageNet) because ColOut is only called once per batch and its operations are offloaded onto the GPU.
## Technical Details
ColOut reduces the size of images, reducing the number of operations per training step and consequently the total time to train the network.
The variability induced by randomly dropping rows and columns can negatively affect generalization performance. In our testing, we saw a decrease in accuracy of ~0.2% in some models on ImageNet and a decrease in accuracy of ~1% on CIFAR-10.
> 🚧 Quality/Speed Tradeoff
>
> In our experiments, ColOut presents a tradeoff in that it increases training speed at the cost of lower model quality.
> On ResNet-50 applied to ImageNet and ResNet-56 applied to CIFAR-10, we found this tradeoff to be worthwhile: it is a pareto improvement over the standard versions of those benchmarks.
> We also found it to be worthwhile in composition with other methods.
> We recommend that you carefully evaluate whether ColOut is also a pareto improvement in the context of your application.
> 🚧 CPU/GPU Tradeoff
>
> If the workload is CPU heavy, it may make sense to run ColOut batch-wise on GPU so that it does not bottleneck training on the CPU. If the workload is GPU-bottlenecked, it will make sense to run ColOut sample-wise on the CPU, avoiding the accuracy reduction of running it batch-wise and improving GPU throughput.
ColOut will show diminishing returns when composed with other methods that change the size of images, such as Progressive Resizing and Selective Backdrop with downsampling. In addition, to the extent that ColOut serves as a form of regularization, combining regularization-based methods can lead to sublinear improvements in accuracy.
## Attribution
*This method and the accompanying documentation were created and implemented by Cory Stephenson at MosaicML.*
## API Reference
**Algorithm class:** {class}`composer.algorithms.ColOut`, {class}`composer.algorithms.ColOutTransform`
**Functional:** {func}`composer.functional.colout_batch`
---
## File: composer/algorithms/cutmix/README.md
# ✂️ CutMix
[\[How to Use\]](#how-to-use) - [\[Suggested Hyperparameters\]](#suggested-hyperparameters) - [\[Technical Details\]](#technical-details) - [\[Attribution\]](#attribution) - [\[API Reference\]](#api-reference)
`Computer Vision`
CutMix is a data augmentation technique that modifies images by cutting out a small patch and replacing it with a different image.
It is a regularization technique that can improve the generalization accuracy of computer
vision models.
| |
|:--:
|*An image with CutMix applied. A picture of a cat has been placed over the top left corner of a picture of a dog. This image is taken from [Figure 1 from Yun et al. (2019)](https://arxiv.org/abs/1905.04899).*|
## How to Use
### Functional Interface
Here we run `CutMix` using index labels and interpolating the loss (a trick when using cross entropy).
```python
# Run the CutMix algorithm directly on the batch data using the Composer functional API
import torch
import torch.nn.functional as F
import composer.functional as cf
def training_loop(model, train_loader):
opt = torch.optim.Adam(model.parameters())
loss_fn = F.cross_entropy
model.train()
for epoch in range(num_epochs):
for X, y in train_loader:
X_cutmix, y_perm, area, _ = cf.cutmix_batch(X, y, alpha=0.2)
y_hat = model(X_cutmix)
loss = area * loss_fn(y_hat, y) + (1 - area) * loss_fn(y_hat, y_perm)
loss.backward()
opt.step()
opt.zero_grad()
```
### Composer Trainer
```python
# Instantiate the algorithm and pass it into the Trainer
# The trainer will automatically run it at the appropriate points in the training loop
from composer.algorithms import CutMix
from composer.trainer import Trainer
cutmix = CutMix(alpha=1.0)
trainer = Trainer(
model=model,
train_dataloader=train_dataloader,
eval_dataloader=eval_dataloader,
max_duration='1ep',
algorithms=[cutmix]
)
trainer.fit()
```
### Implementation Details
CutMix is implemented following the [original paper](https://arxiv.org/abs/1905.04899). This means CutMix runs immediately before the training example is provided to the model and on the GPU, if one is being used.
The construction of the bounding box for the mixed region follows the [paper's implementation](https://github.com/clovaai/CutMix-PyTorch) which selects the center pixel of the bounding box uniformly at random from all locations in the image and clips the bounding box to fit. This implies that the size of the region mixed by CutMix is not always square, and the area is not directly drawn from a beta distribution. It also implies that not all regions are equally likely to lie inside the bounding box.
## Suggested Hyperparameters
Setting `alpha=1` is a standard choice. This produces a uniform distribution, meaning the interpolation between the labels of the two sets of examples is selected uniformly between 0 and 1.
## Technical Details
CutMix works by creating a new mini-batch of inputs to the network by operating on a batch `(X1, y1)` of (inputs, targets) together with version `(X2, y2)` with the same examples but where the ordering of examples has been shuffled.
The final set of inputs `X` is created by choosing a rectangular box within each example `x1` in `X1` and filling it with the data from the same region from the corresponding example `x2` in `X2`.
The final set of targets `y` is created by sampling a value `interpolation` (between 0.0 and 1.0) from the Beta distribution parameterized by `alpha` and interpolating between the targets `y1` and `y2`.
> ❗ CutMix Produces a Full Distribution, Not a Target Index
>
> Many classification tasks represent the target value using the index of the target value rather than the full, one-hot encoding of the label value.
> Since CutMix interpolates between two target values for each example, it must represent the final targets as a dense distribution.
> Our implementation of CutMix turns each label into a dense distribution (if it has not already been converted into a distribution).
> The loss function used for the model must be able to accept this dense distribution as the target.
CutMix is intended to improve generalization performance, and we empirically found this to be the case in our image classification settings. The original paper also reports improvements in object localization and robustness.
> 🚧 Composing Regularization Methods
>
> As general rule, composing regularization methods may lead to diminishing returns in quality improvements. CutMix is one such regularization method.
Data augmentation techniques can sometimes put additional load on the CPU, potentially to the point where the CPU becomes a bottleneck for training.
To prevent this from happening, our implementation of CutMix (1) takes place on the GPU and (2) uses the same patch and interpolation for all examples in the minibatch.
Doing so avoids putting additional work on the CPU (since augmentation occurs on the GPU) and minimizes the additional work on the GPU (since all images are handled uniformly within a batch).
> 🚧 CutMix Requires a Small Amount of Additional GPU Compute and Memory
>
> CutMix requires a small amount of additional GPU compute and memory to produce the mixed-up batch.
> In our experiments, we have found these additional resource requirements to be negligible.
## Attribution
[*CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features*](https://arxiv.org/abs/1905.04899) by Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Published in ICCV 2019.
*This Composer implementation of this method and the accompanying documentation were produced by Cory Stephenson at MosaicML.*
## API Reference
**Algorithm class:** {class}`composer.algorithms.CutMix`
**Functional:** {func}`composer.functional.cutmix_batch`
---
## File: composer/algorithms/cutout/README.md
# 🎃 Cutout
[\[How to Use\]](#how-to-use) - [\[Suggested Hyperparameters\]](#suggested-hyperparameters) - [\[Technical Details\]](#technical-details) - [\[Attribution\]](#attribution) - [\[API Reference\]](#api-reference)
`Computer Vision`
Cutout is a data augmentation technique that masks one or more square regions of an input image, replacing them with gray boxes.
It is a regularization technique that improves the accuracy of models for computer vision.
| |
|:--:
|*Several images from the CIFAR-10 dataset with Cutout applied. Cutout adds a gray box that occludes a portion of each image. This is [Figure 1 from DeVries & Taylor (2017)](https://arxiv.org/abs/1708.04552).*|
## How to Use
### Functional Interface
```python
# Run the CutOut algorithm directly on the batch data using the Composer functional API
import torch
import torch.nn.functional as F
from composer import functional as cf
def training_loop(model, train_loader):
opt = torch.optim.Adam(model.parameters())
loss_fn = F.cross_entropy
model.train()
for epoch in range(num_epochs):
for X, y in train_loader:
X_cutout = cf.cutout_batch(X,
num_holes=1,
length=0.5)
y_hat = model(X_cutout)
loss = loss_fn(y_hat, y)
loss.backward()
opt.step()
opt.zero_grad()
```
### Composer Trainer
```python
# Instantiate the algorithm and pass it into the Trainer
# The trainer will automatically run it at the appropriate points in the training loop
from composer.algorithms import CutOut
from composer.trainer import Trainer
cutout = CutOut(num_holes=1, length=0.5)
trainer = Trainer(
model=model,
train_dataloader=train_dataloader,
eval_dataloader=eval_dataloader,
max_duration='1ep',
algorithms=[cutout]
)
trainer.fit()
```
### Implementation Details
CutOut randomly selects `num_holes` square regions (which are possibly overlapping) with side length `length` and uses them to generate a binary mask for the image where the points within any hole are set to 0 and the remaining points are set to 1.
This mask is then multiplied element-wise with the image in order to set the pixel value of any pixel value within a hole to 0.
CutOut is implemented following the [original paper](https://arxiv.org/abs/1708.04552). However, our implementation currently differs in that CutOut operates on a batch of data and runs on device to avoid potential CPU bottlenecks.
This means the same bounding box is used for all examples in a batch, which can have either a positive or negative effect on accuracy.
The construction of the bounding box for the mixed region follows the [paper's implementation](https://github.com/uoguelph-mlrg/Cutout) which selects the center pixel of the bounding box uniformly at random from all locations in the image and clips the bounding box to fit. This implies that the size of the region masked by CutOut is not always square and that the area is not always as large as suggested by the `length` parameter. It also implies that not all regions are equally likely to lie inside the bounding box.
## Suggested Hyperparameters
We found that setting `num_holes=1` (adding a single gray patch) to the image gives good results. We also found that setting `length = 0.5`, indicating that the masked region should have height and width half as large as the image, produces good results. However, in some scenarios this value may be too large, obstructing a quarter of the total area of the image; if so, setting `length` to a number of pixels equivalent to a quarter of the image width or height may be better.
## Technical Details
Cutout works by randomly choosing one or more square regions from an input image and replacing them with the mean value over the dataset.
Since it is common to normalize image data based on the dataset mean and variance, the mean value is typically 0.
To ease implementation, we went with a simple binary mask, in which the regions to be cut out are set to pixel value zero and the remainder of the image stays the same.
We found Cutout to be an effective way of improving accuracy for ResNets trained on CIFAR-10 and ImageNet in the absence of robust hyperparameter tuning and other regularizers.
As we improved our training methodology through improved hyperparameters and by adding other regularization techniques, the benefits of Cutout diminished to the point of becoming negligible.
> ❗ Cutout Provided Limited Benefits in Our Experiments
>
> In our experiments on ResNets for CIFAR-10 and ImageNet, Cutout provided little or no improvements in accuracy when the models were well-tuned and when we combined it with other regularization methods.
> It is possible that Cutout may still be helpful for other models and tasks and in settings that are less well-tuned.
Because Cutout is a regularizer, it may improve or degrade accuracy, depending on the setting.
Regularization methods reduce overfitting, potentially allowing models to reach higher quality.
However this typically requires (1) larger models with more capacity to perform this more difficult learning and (2) longer training runs to allow these models time to learn.
> 🚧 Composing Regularization Methods
>
> As general rule, composing regularization methods may lead to diminishing returns in quality improvements. Cutout is one such regularization method. We do not see improvements when combining Cutout with other regularization and augmentation methods such as Mixup and Label Smoothing.
Data augmentation techniques can sometimes put additional load on the CPU, potentially to the point where the CPU becomes a bottleneck for training.
To prevent this from happening for Cutout, our implementation of Cutout (1) takes place on the GPU and (2) occludes the same patch on each image in a minibatch.
Doing so avoids putting additional work on the CPU (since augmentation occurs on the GPU) and avoids putting additional work on the GPU (since all images are handled uniformly within a batch).
> ❗ Cutout Increases Memory Requirements
>
> Since Cutout runs on GPU by default and uses some extra memory to construct the mask, out of memory errors may occur if GPU memory is severely limited.
Since Cutout masks a portion of the input, this can alter the inherent shape/texture bias of the model. For an example, see [Hermann et al. (2020)](https://arxiv.org/abs/1911.09071).
Although our implementation of Cutout is designed for computer vision tasks, variants of Cutout have been shown to be useful in other settings, for example, audio processing ([Cances et al., 2021](https://arxiv.org/abs/2102.08183)).
The implementation in Composer currently only supports computer vision.
## Attribution
[*Improved Regularization of Convolutional Neural Networks with Cutout*](https://arxiv.org/abs/1708.04552) by Terrance DeVries and Graham W. Taylor. Posted to arXiv in 2017.
*This Composer implementation of this method and the accompanying documentation were produced by Cory Stephenson at MosaicML.*
## API Reference
**Algorithm class:** {class}`composer.algorithms.CutOut`
**Functional:** {func}`composer.functional.cutout_batch`
---
## File: composer/algorithms/ema/README.md
# 🚚 EMA
[\[How to Use\]](#how-to-use) - [\[Suggested Hyperparameters\]](#suggested-hyperparameters) - [\[Technical Details\]](#technical-details) - [\[Attribution\]](#attribution) - [\[API Reference\]](#api-reference)
Exponential Moving Average (EMA) is a model averaging technique that maintains an exponentially weighted moving average of the model parameters during training. The averaged parameters are used for model evaluation. EMA typically results in less noisy validation metrics over the course of training, and sometimes increased generalization.
## How to Use
### Functional Interface
```python
# Run the EMA algorithm directly on the batch data using the Composer functional API
import copy
import composer.functional as cf
def training_loop(model, train_loader):
opt = torch.optim.Adam(model.parameters())
loss_fn = F.cross_entropy
ema_model = copy.deepcopy(model)
model.train()
for epoch in range(num_epochs):
for X, y in train_loader:
y_hat = model(X)
loss = loss_fn(y_hat, y)
loss.backward()
opt.step()
opt.zero_grad()
cf.compute_ema(model, ema_model, smoothing=0.99)
```
### Composer Trainer
```python
# Instantiate the algorithm and pass it into the Trainer
# The trainer will automatically run it at the appropriate points in the training loop
from composer.algorithms import EMA
from composer.trainer import Trainer
ema = EMA(half_life='50ba')
trainer = Trainer(model=model,
train_dataloader=train_dataloader,
max_duration='1ep',
algorithms=[ema])
trainer.fit()
model = ema.ema_model
```
### Implementation Details
Because EMA needs to maintain a copy of the model's (averaged) weights, it requires a bit more on-device memory. The amount of extra memory used is equal to the size of the model's trainable parameters and buffers. In practice, the extra memory used is small relative to the total amount of memory used, as activations and optimizer state are not duplicated.
EMA also uses a bit of extra compute to calculate the moving average. This can lead to a small slowdown. The extra compute can be reduced by not computing the moving average every iteration. In the composer trainer implementation this can be done by using a larger `update_interval`. In practice we find that as long as `half_life` is much larger than `update_interval`, increasing `update_interval` does not have much effect on generalization performance.
## Suggested Hyperparameters
The Composer Trainer implementation of EMA has several hyperparameters:
- `half_life` - The half life for terms in the average. A longer half life means old information is remembered longer, a shorter half life means old information is discared sooner. Defaults to `'1000ba'`
- `update_interval` - The period at which updates to the moving average are computed. A longer update interval means that updates are computed less frequently. If left unspecified, this defaults to `1` in the units of `half_life`, or `1ba` if using `smoothing`.
- `ema_start` - The amount of training completed before SWA is applied. The default value is `'0.0dur'` which starts EMA at the start of training.
A good typical starting value for `half_life` is `half_life="1000ba"`, for a half life of 1000 batches. At the same time, `update_interval` can be left unspecified which will default to `update_interval="1ba"`, or set to a larger value such as `update_interval="10ba"` to improve runtime. Shorter update intervals typically result in better generalization performance at the cost of somewhat increased runtime.
For compatibility with other implementations, there is also an option to specify the value of `smoothing` directly.
- `smoothing` - The coefficient representing the degree to which older observations are kept. The default (unspecified) value is `None`. Should only be used if `half_life` is not used
To use this, `half_life` should be set to `half_life=None`, and the value of smoothing given instead. This value is not modified when `update_interval` is changed, and so changes to `update_interval` when using `smoothing` will result in changes to the time scale of the average.
## Technical Details
> ✅ EMA Improves the Tradeoff Between Quality and Training Speed
>
> In our experiments, EMA improves the attainable tradeoffs between training speed and the final quality of the trained model.
> We recommend EMA for training convolutional networks.
> ✅ EMA should result in less noisy validation metrics during training
>
> If evalutation metrics are computed over the course of training, EMA should result in these metrics being smoother and less noisy due to averaging.
> 🚧 Composing Model-Averaging Methods
>
> As a general rule, model-averaging methods do not compose well. We recommend using one
> of EMA or SWA, but not both.
> ❗ EMA increases memory consumption
>
> Because EMA needs to maintain a copy of the model's (averaged) weights, it requires a bit more on device memory. In practice, the extra memory used is small relative to the total amount of memory used, as activations and optimizer state are not duplicated.
> ❗ EMA uses some extra compute
>
>This can lead to a small slowdown. The extra compute can be reduced by not computing the moving average every iteration. In the composer trainer implementation this can be done by using a larger `update_interval`.
> ❗ Evaluation should not be done with the training model
>
> Evaluation should be done with the `ema_model` in the functional impementation as this is the model containing the averaged parameters. The ema model can be accessed after training from the `EMA` object via `model = ema.get_ema_model(model)` in the composer trainer implementation. This replaces the parameters of the supplied model with the ema_weights unless composer's model already contains them. Similarly, the model without ema applied (the training model) can be accessed via `model=ema.get_training_model(model)`. By default, when saving checkpoints with the `CheckpointSaver` callback or through trainer arguments the weights saved will be the ema model weights. An exception is if saving is done by explicitly calling `trainer.save_checkpoint()` which will result in the training model weights being saved as `state.model`.
## Attribution
Our implementation of EMA was inspired by [Tensorflow's Exponential Moving Average](https://www.tensorflow.org/api_docs/python/tf/train/ExponentialMovingAverage)
*This Composer implementation of this method and the accompanying documentation were produced by Cory Stephenson at MosaicML.*
## API Reference
**Algorithm class:** {class}`composer.algorithms.EMA`
**Functional:** {func}`composer.functional.compute_ema`
---
## File: composer/algorithms/factorize/README.md
# ➗ Factorize
[\[How to Use\]](#how-to-use) - [\[Suggested Hyperparameters\]](#suggested-hyperparameters) - [\[Technical Details\]](#technical-details) - [\[Attribution\]](#attribution) - [\[API Reference\]](#api-reference)
`Computer Vision`, `Natural Language Processing`
Factorize splits a large linear or convolutional layer into two smaller ones that compute a similar function.
This can be applied to models for both computer vision and natural language processing.
| |
|:--:
|*Figure 1 of [Zhang et al. (2015)](https://ieeexplore.ieee.org/abstract/document/7332968). (a) The weights `W` of a 2D convolutional layer with `k x k` filters, `c` input channels, and `d` output channels are factorized into two smaller convolutions (b) with weights `W'` and `P` with `d'` intermediate channels. The first convolution uses the original filter size but produces only `d'` channels. The second convolution has `1 x 1` filters and produces the original `d` output channels but has only `d'` input channels. This changes the complexity per spatial position from $O(k^2cd)$ to $O(k^2cd') + O(d'd)$.*|
## How to Use
### Functional Interface
```python
# Run the Factorization algorithm directly on the model using the Composer functional API
import torch
import torch.nn.functional as F
import composer.functional as cf
def training_loop(model, train_loader):
opt = torch.optim.Adam(model.parameters())
# only need to pass in opt if apply_factorization is used after optimizer
# creation; otherwise, only the model needs to be passed in.
cf.apply_factorization(
model,
factorize_convs=True,
factorize_linears=True,
min_channels=512,
latent_channels=0.25,
min_features=512,
latent_features=0.25,
optimizers=opt
)
loss_fn = F.cross_entropy
model.train()
for epoch in range(1):
for X, y in train_loader:
y_hat = model(X)
loss = loss_fn(y_hat, y)
loss.backward()
opt.step()
opt.zero_grad()
```
### Composer Trainer
```python
# Instantiate the algorithm and pass it into the Trainer
# The trainer will automatically run it at the appropriate point in the training loop
from composer.algorithms import Factorize
from composer.trainer import Trainer
factorize = Factorize(
factorize_convs=True,
factorize_linears=True,
min_channels=256,
latent_channels=0.25,
min_features=256,
latent_features=128
)
trainer = Trainer(
model=model,
train_dataloader=train_dataloader,
eval_dataloader=eval_dataloader,
max_duration='10ep',
algorithms=[factorize]
)
trainer.fit()
```
## Suggested Hyperparameters
For Factorize to have any effect on Linear modules, it is necessary to have `factorize_linears=True` and `min_features` small enough that at least one Linear module has at least this many input and output features.
Similarly, for Factorize to have any effect on Conv2d modules, it is necessary to have `factorize_convs=True`, and `min_channels` small enough that at least one Conv2d module has at least this many input and output features. This is most likely to be an issue with CIFAR-10 ResNets such as ResNet-20 and ResNet-56, which have at most 64 channels.
While factorizing with `latent_{features,channels} < 0.5` always reduces the number of FLOPs needed by a module, factorizing small modules is unlikely to result in a speedup. This is because small operations are limited by memory bandwidth, not computation. Since factorization increases memory bandwidth usage in order to save compute, it is not helpful in this regime.
We suggest setting `min_channels >= 512`, `min_features >= 512`, `latent_channels <= 0.25`, and `latent_features <= 0.25` to obtain any speedup.
## Technical Details
Based on ResNet-50 experiments, we have not observed Factorize to ever be helpful.
Even with conservative settings like `min_channels=512`, `latent_channels=128`, we observe over a 1% accuracy loss and only a small (<5%) throughput increase.
We have provided this implementation and method card for informational purposes, since factorization is a popular technique in the research literature.
> ❗ Factorize Did Not Improve Efficiency in Our Experiments
>
> Factorize provided no improvements in (and often decreased) accuracy, and provided very modest throughput increases in our experiments.
> It is possible that Factorize may still be helpful in other settings.
At present, only factorization before training is supported. This is because of limitations of PyTorch Distributed Data Parallel.
We hope to allow factorization during training in the future.
This might allow more intelligent allocation of factorization to different layers based on how well they can be approximated.
To work around this limitation, one can save the model, stop training, load and alter the model, and then restart training.
Factorize can be applied to any model with linear or convolutional layers but is most likely to be useful for large models with many channels or large hidden layer sizes. However, factorization may not work with your model if it makes special assumptions about linear layers and their attributes. For example, factorization will not work with `torch.nn.MultiHeadAttention` modules, because MultiHeadAttention expects its `linear` submodule to have a `weight` attribute, and `FactorizedLinear` does not have this attribute.
At present, only factorizing `linear` and `conv2d` modules is supported (i.e., factorizing `conv1d` and `conv3d` modules is not supported).
> ❗ Only Linear and 2D Convolution Modules are Supported
>
> Factorization does not currently support other kinds of layers, for example 1D and 3D convolutions.
## Attribution
Factorizing convolution kernels dates back to at least [Gotsman 1994](https://onlinelibrary.wiley.com/doi/abs/10.1111/1467-8659.1320153). To the best of our knowledge, the first papers to apply factorization to modern neural networks were:
* [*Speeding up convolutional neural networks with low rank expansions*](https://arxiv.org/abs/1405.3866) by Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Published in the British Machine Vision Conference in 2014.
* [*Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation*](https://arxiv.org/abs/1404.0736) by Emily Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Published in NeurIPS 2014.
Our factorization structure most closely matches that in:
* [*Accelerating Very Deep Convolutional Networks for Classification and Detection*](https://ieeexplore.ieee.org/abstract/document/7332968) by Xiangyu Zhang, Jianhua Zou, Kaiming He, and Jian Sun. Published in IEEE TPAMI in 2016.
*The Composer implementation of this method and the accompanying documentation were produced by Davis Blalock at MosaicML.*
## API Reference
**Algorithm class:** {class}`composer.algorithms.Factorize`
**Functional:** {func}`composer.functional.apply_factorization`