examples Documentation

A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

24,076 stars Python Python 42 files · ~17,435 tokens
RAW Doc

Technical Documentation: pytorch/examples

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

PyTorch Examples

https://pytorch.org/examples/

pytorch/examples is a repository showcasing examples of using PyTorch. The goal is to have curated, short, few/no dependencies high quality examples that are substantially different from each other that can be emulated in your existing work.

Available models

Additionally, a list of good examples hosted in their own repositories:

Contributing

If you'd like to contribute your own example or fix a bug please make sure to take a look at CONTRIBUTING.md.

Overview (`dcgan/README.md`)

Deep Convolution Generative Adversarial Networks

This example implements the paper Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

The implementation is very close to the Torch implementation dcgan.torch

After every 100 training iterations, the files real_samples.png and fake_samples.png are written to disk
with the samples from the generative model.

After every epoch, models are saved to: netG_epoch_%d.pth and netD_epoch_%d.pth

Downloading the dataset

You can download the LSUN dataset by cloning this repo and running

text
python download.py -c bedroom

Usage

text
/* Detailed source-code truncated for AI context efficiency. */

Overview (`fast_neural_style/README.md`)

fast-neural-style :city_sunrise: :rocket:

This repository contains a pytorch implementation of an algorithm for artistic style transfer. The algorithm can be used to mix the content of an image with the style of another image. For example, here is a photograph of a door arch rendered in the style of a stained glass painting.

The model uses the method described in Perceptual Losses for Real-Time Style Transfer and Super-Resolution along with Instance Normalization. The saved-models for examples shown in the README can be downloaded from here.

Requirements

The program is written in Python, and uses pytorch, scipy. A GPU is not necessary, but can provide a significant speed up especially for training a new model. Regular sized images can be styled on a laptop or desktop using saved models.

Usage

Stylize image

text
python neural_style/neural_style.py eval --content-image </path/to/content/image> --model </path/to/saved/model> --output-image </path/to/output/image> --accel
  • --content-image: path to content image you want to stylize.
  • --model: saved model to be used for stylizing the image (eg: mosaic.pth)
  • --output-image: path for saving the output image.
  • --content-scale: factor for scaling down the content image if memory is an issue (eg: value of 2 will halve the height and width of content-image)
  • --accel: use accelerator

Train model

bash
python neural_style/neural_style.py train --dataset </path/to/train-dataset> --style-image </path/to/style/image> --save-model-dir </path/to/save-model/folder> --epochs 2 --accel

There are several command line arguments, the important ones are listed below

  • --dataset: path to training dataset, the path should point to a folder containing another folder with all the training images. I used COCO 2014 Training images dataset [80K/13GB] (download).
  • --style-image: path to style-image.
  • --save-model-dir: path to folder where trained model will be saved.
  • --accel: use accelerator.

If --accel argument is given, pytorch will search for available hardware acceleration device and attempt to use it. This example is known to work on CUDA, MPS and XPU devices.

Refer to neural_style/neural_style.py for other command line arguments. For training new models you might have to tune the values of --content-weight and --style-weight. The mosaic style model shown above was trained with --content-weight 1e5 and --style-weight 1e10. The remaining 3 models were also trained with similar order of weight parameters with slight variation in the --style-weight (5e10 or 1e11).

Models

Models for the examples shown below can be downloaded from here or by running the script download_saved_models.py.

Overview (`fx/README.md`)

FX Examples

This folder contains several examples of program transformations implemented using torch.fx. More information about FX can be found in the documentation.

Note that all examples should be runnable as standalone Python files. In the case of an exception, the example will appear in a subfolder with a README.md file explaining how to run the example.

As FX is currently in a Beta release, the API or these examples are subject to change.

Overview (`gat/README.md`)

Graph Attention Network

This repository contains a PyTorch implementation of the Graph Attention Networks (GAT) based on the paper "Graph Attention Network" by Velickovic et al.

The Graph Attention Network is a powerful graph neural network model for learning represtations on graph-structured data, which has shown excellent performance in various tasks such as node classification, link prediction, and graph classification.

Overview

The Graph Attention Network (GAT) is a graph neural network architecture designed specifically for handling graph-structured data. It leverages multi-head attention mechanism to capture the information of neighboring nodes in an attentive manner to learn represtations for each node. This attention mechanism allows the model to focus on relevant nodes and adaptively weight their contributions during message passing.

Check out the following resources for more ino on GATs:

This repository provides a clean and short implementation of the official GAT model using PyTorch. The code is well-documented and easy to understand, making it a valuable resource for researchers and practitioners interested in graph deep learning.

Key Features

  • GAT Model: Implementation of the Graph Attention Network model with multi-head attention based on the paper "Graph Attention Network" by Velickovic et al.
  • Graph Attention Layers: Implementation of graph convolutional layers that aggregate information from neighboring nodes using a self-attention mechanisms to learn node importance weights.
  • Training and Evaluation: Code for training GAT models on graph-structured data and evaluating their performance on node classification tasks on the Cora benchmark dataset.

Requirements

  • Python 3.7 or higher
  • PyTorch 2.0 or higher
  • Requests 2.31 or higher
  • NumPy 1.24 or higher

Dataset

The implementation includes support for the Cora dataset, a standard benchmark dataset for graph-based machine learning tasks. The Cora dataset consists of scientific publications, where nodes represent papers and edges represent citation relationships. Each paper is associated with a binary label indicating one of seven classes. The dataset is downloaded, preprocessed and ready to use.

Model Architecture

The official architecture (used in this project) proposed in the paper "Graph Attention Network" by Velickovic et al. consists of two graph attention layers which incorporates the multi-head attention mechanisms during its message trasformation and aggregation. Each graph attention layer applies a shared self-attention mechanism to every node in the graph, allowing them to learn different representations based on the importance of their neighbors.

In terms of activation functions, the GAT model employs both the Exponential Linear Unit (ELU) and the Leaky Rectified Linear Unit (LeakyReLU) activations, which introduce non-linearity to the model. ELU is used as the activation function for the hidden layers, while LeakyReLU is applied to the attention coefficients to ensure non-zero gradients for negative values.

Following the official implementation, the first GAT layer consists of K = 8 attention heads computing F' = 8 features each (for a total of 64 features) followed by an exponential linear unit (ELU) activation on the layer outputs. The second GAT layer is used for classification: a single attention head that computes C features (where C is the number of classes), followed by a softmax activation for probablisitic outputs. (we use log-softmax instead for computational convenience with using NLLLoss)

Note that due to being an educational example, this implementation uses the full dense form of the adjacency matrix of the graph, and not the sparse form of the matrix. Thus all the operations in the model implemeation is done in a non-sparse from. This will not affect the model's performance accuracy-wise. However an sparse-friendly implementation will help with the efficiency in the use of resources, storage, and speed.

Usage

Training and evaluating the GAT model on the Cora dataset can be done through running the main.py script as follows:

  1. Clone the PyTorch examples repository:
text
git clone https://github.com/pytorch/examples.git
cd examples/gat
  1. Install the required dependencies:
text
pip install -r requirements.txt
  1. Train the GAT model by running the main.py script as follows:: (Example using the default parameters)
bash
python main.py --epochs 300 --lr 0.005 --l2 5e-4 --dropout-p 0.6 --num-heads 8 --hidden-dim 64 --val-every 20

In more detail, the main.py script recieves following arguments:

text
usage: main.py [-h] [--epochs EPOCHS] [--lr LR] [--l2 L2] [--dropout-p DROPOUT_P] [--hidden-dim HIDDEN_DIM] [--num-heads NUM_HEADS] [--concat-heads] [--val-every VAL_EVERY]
               [--no-cuda] [--no-mps] [--dry-run] [--seed S]

PyTorch Graph Attention Network

options:
  -h, --help            show this help message and exit
  --epochs EPOCHS       number of epochs to train (default: 300)
  --lr LR               learning rate (default: 0.005)
  --l2 L2               weight decay (default: 6e-4)
  --dropout-p DROPOUT_P
                        dropout probability (default: 0.6)
  --hidden-dim HIDDEN_DIM
                        dimension of the hidden representation (default: 64)
  --num-heads NUM_HEADS
                        number of the attention heads (default: 4)
  --concat-heads        wether to concatinate attention heads, or average over them (default: False)
  --val-every VAL_EVERY
                        epochs to wait for print training and validation evaluation (default: 20)
  --no-accel            disables accelerator
  --dry-run             quickly check a single pass
  --seed S              random seed (default: 13)

Results

After training for 300 epochs with default hyperparameters on random train/val/test data splits, the GAT model achieves around %81.25 classification accuracy on the test split. This result is comparable to the performance reported in the original paper. However, the results can vary due to the randomness of the train/val/test split.

Reference

text
@article{
  velickovic2018graph,
  title="{Graph Attention Networks}",
  author={Veli{\v{c}}kovi{\'{c}}, Petar and Cucurull, Guillem and Casanova, Arantxa and Romero, Adriana and Li{\`{o}}, Pietro and Bengio, Yoshua},
  journal={International Conference on Learning Representations},
  year={2018},
  url={https://openreview.net/forum?id=rJXMpikCZ},
}

Overview (`gcn/README.md`)

Graph Convolutional Network

This repository contains an implementation of Graph Convolutional Networks (GCN) based on the paper "Semi-Supervised Classification with Graph Convolutional Networks" by Thomas N. Kipf and Max Welling.

Overview

This project implements the GCN model proposed in the paper for semi-supervised node classification on graph-structured data. GCN leverages graph convolutions to aggregate information from neighboring nodes and learn node representations for downstream tasks. The implementation provides a flexible and efficient GCN model for graph-based machine learning tasks.

Requirements

bash
pip install -r requirements.txt

Usage

bash
python main.py --epochs 200 --lr 0.01 --l2 5e-4 --dropout-p 0.5 --hidden-dim 16 --val-every 20 --include-bias

Dataset

The implementation includes support for the Cora dataset, a standard benchmark dataset for graph-based machine learning tasks. The Cora dataset consists of scientific publications, where nodes represent papers and edges represent citation relationships. Each paper is associated with a binary label indicating one of seven classes. The dataset is downloaded, preprocessed and ready to use.

Model Architecture

The GCN model architecture follows the details provided in the paper. It consists of multiple graph convolutional layers with ReLU activation, followed by a final softmax layer for classification. The implementation supports customizable hyperparameters such as the number of hidden units, the number of layers, and dropout rate.

Results

The model achieves a classification accuracy of 82.5% on the test set of the Cora dataset after 200 epochs of training. This result is comparable to the performance reported in the original paper. However, the results can vary due to the randomness of the train/val/test split.

References
Thomas N. Kipf and Max Welling. "Semi-Supervised Classification with Graph Convolutional Networks." Link to the paper

Original paper repository: https://github.com/tkipf/gcn

Overview (`imagenet/README.md`)

ImageNet training in PyTorch

This implements training of popular model architectures, such as ResNet, AlexNet, and VGG on the ImageNet dataset.

Requirements

Training

To train a model, run main.py with the desired model architecture and the path to the ImageNet dataset:

bash
python main.py -a resnet18 [imagenet-folder with train and val folders]

The default learning rate schedule starts at 0.1 and decays by a factor of 10 every 30 epochs. This is appropriate for ResNet and models with batch normalization, but too high for AlexNet and VGG. Use 0.01 as the initial learning rate for AlexNet or VGG:

bash
python main.py -a alexnet --lr 0.01 [imagenet-folder with train and val folders]

Use Dummy Data

ImageNet dataset is large and time-consuming to download. To get started quickly, run main.py using dummy data by "--dummy". It's also useful for training speed benchmark. Note that the loss or accuracy is useless in this case.

bash
python main.py -a resnet18 --dummy

Multi-processing Distributed Data Parallel Training

If running on CUDA, you should always use the NCCL backend for multi-processing distributed training since it currently provides the best distributed training performance.

For XPU multiprocessing is not supported as of PyTorch 2.6.

Single node, multiple GPUs:

bash
python main.py -a resnet50 --dist-url 'tcp://127.0.0.1:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0 [imagenet-folder with train and val folders]

Multiple nodes:

Node 0:

bash
python main.py -a resnet50 --dist-url 'tcp://IP_OF_NODE0:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 2 --rank 0 [imagenet-folder with train and val folders]

Node 1:

bash
python main.py -a resnet50 --dist-url 'tcp://IP_OF_NODE0:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 2 --rank 1 [imagenet-folder with train and val folders]

Usage

text
/* Detailed source-code truncated for AI context efficiency. */

Overview (`language_translation/README.md`)

Language Translation

This example shows how one might use transformers for language translation. In particular, this implementation is loosely based on the Attention is All You Need paper.

Requirements

We will need a tokenizer for our languages. Torchtext does include a tokenizer for English, but unfortunately, we will need more languages then that. We can get these tokenizers via spacy

bash
python3 -m spacy download <language>
python3 -m spacy download en
python3 -m spacy download de

Spacy supports many languages. For a full accounting of supported languages, please look here. This example will default from German to English.

Torchtext is also required:

bash
pip install torchtext

Just running these commands will get you started:

bash
pip install -r requirements.txt
python3 -m spacy download <language-you-want>

Usage

This example contains a lot of flags that you can set to change the behavior / training of the module. You can see all of them by running:

bash
python3 main.py -h

But in general, all of the settings have "sensible" defaults; however, the default translation is to translate from German to English. To train the model, you only need to run the following command, but there is also an example for how to use any language you want:

bash
python3 main.py
python3 main.py --src en --tgt fr # For english to french translation

For model inference, you can use this command:

bash
python3 main.py --inference --model_path <path-to-model>

After some loading time, this will open an interactive interface where you can type in whatever sentence you are interested in translating.

Overview (`mnist/README.md`)

Basic MNIST Example

bash
pip install -r requirements.txt
python main.py
# CUDA_VISIBLE_DEVICES=2 python main.py  # to specify GPU id to ex. 2

Overview (`mnist_forward_forward/README.md`)

Basic Forward-Forward Example

This example implements the paper The Forward-Forward Algorithm: Some Preliminary Investigations by Geoffrey Hinton.

the aim of this paper is to introduce a new learning procedure for neural networks. the forward and backward passes of backpropagation by two forward passes.

bash
pip install -r requirements.txt
python main.py

The main.py script accepts the following arguments:

bash
optional arguments:
  -h, --help            show this help message and exit
  --epochs EPOCHS       number of epochs to train (default: 1000)
  --lr LR               learning rate (default: 0.03)
  --no_accel            disables accelerator
  --seed SEED           random seed (default: 1)
  --save_model          For saving the current Model
  --train_size TRAIN_SIZE
                        size of training set
  --threshold THRESHOLD
                        threshold for training
  --test_size TEST_SIZE
                        size of test set
  --save-model          For Saving the current Model
  --log-interval LOG_INTERVAL
                        logging training status interval

Overview (`mnist_hogwild/README.md`)

MNIST Hogwild Example

bash
pip install -r requirements.txt
python main.py

The main.py script accepts the following arguments:

bash
optional arguments:
  -h, --help            show this help message and exit
  --batch-size          input batch_size for training (default: 64)
  --test-batch-size     input batch size for testing (default: 1000)
  --epochs EPOCHS       number of epochs to train (default: 10)
  --lr LR               learning rate (default: 0.01)
  --momentum            SGD momentum (default: 0.5)
  --seed SEED           random seed (default: 1)
  --save_model          save the trained model to state_dict
  --log-interval        how many batches to wait before logging training status (default: 10)
  --num-processes       how many training processes to use (default: 2)
  --cuda                enables CUDA training
  --mps                 enables macos GPU training
  --dry-run             quickly check a single pass

Overview (`mnist_rnn/README.md`)

Example of MNIST using RNN

Motivation

Create pytorch example similar to Official Tensorflow Keras RNN example using MNIST here

bash
pip install -r requirements.txt
python main.py
# CUDA_VISIBLE_DEVICES=2 python main.py  # to specify GPU id to ex. 2
bash
optional arguments:
  -h, --help            show this help message and exit
  --batch_size          input batch_size for training (default:64)
  --testing_batch_size  input batch size for testing (default: 1000)
  --epochs EPOCHS       number of epochs to train (default: 14)
  --lr LR               learning rate (default: 0.1)
  --gamma               learning rate step gamma (default: 0.7)
  --accel               enables accelerator
  --seed SEED           random seed (default: 1)
  --save_model          For saving the current Model
  --log_interval        how many batches to wait before logging training status
  --dry-run             quickly check a single pass

Overview (`regression/README.md`)

Linear regression example

Trains a single fully-connected layer to fit a 4th degree polynomial.

Overview (`reinforcement_learning/README.md`)

Reinforcement learning training example

bash
pip install -r requirements.txt
# For REINFORCE:
python reinforce.py
# For actor critic:
python actor_critic.py

Overview (`siamese_network/README.md`)

Siamese Network Example

Siamese network for image similarity estimation.
The network is composed of two identical networks, one for each input.
The output of each network is concatenated and passed to a linear layer.
The output of the linear layer passed through a sigmoid function.
FaceNet is a variant of the Siamese network.
This implementation varies from FaceNet as we use the ResNet-18 model from
Deep Residual Learning for Image Recognition as our feature extractor.
In addition, we aren't using TripletLoss as the MNIST dataset is simple, so BCELoss can do the trick.

Usage

Install the required dependencies:

bash
pip install -r requirements.txt

To run the example, execute:

bahs
python main.py
# CUDA_VISIBLE_DEVICES=2 python main.py  # to specify GPU id to ex. 2

If a hardware accelerator device is detected, the example will execute on the accelerator; otherwise, it will run on the CPU.

To force execution on the CPU, use --no-accel command line argument:

bash
python main.py --no-accel

Optionally, you can add the following arguments to customize your execution.

bash
--batch-size            input batch size for training (default: 64)
--test-batch-size       input batch size for testing (default: 1000)
--epochs                number of epochs to train (default: 14)
--lr                    learning rate (default: 1.0)
--gamma                 learning rate step gamma (default: 0.7)
--no-accel              disables accelerator
--dry-run               quickly check a single pass
--seed                  random seed (default: 1)
--log-interval          how many batches to wait before logging training status
--save-model            Saving the current Model

Overview (`super_resolution/README.md`)

Superresolution using an efficient sub-pixel convolutional neural network

This example illustrates how to use the efficient sub-pixel convolution layer described in "Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network" - Shi et al. for increasing spatial resolution within your network for tasks such as superresolution.

text
usage: main.py [-h] --upscale_factor UPSCALE_FACTOR [--batchSize BATCHSIZE]
               [--testBatchSize TESTBATCHSIZE] [--nEpochs NEPOCHS] [--lr LR]
               [--accel] [--threads THREADS] [--seed SEED]

PyTorch Super Res Example

optional arguments:
  -h, --help            show this help message and exit
  --upscale_factor      super resolution upscale factor
  --batchSize           training batch size
  --testBatchSize       testing batch size
  --nEpochs             number of epochs to train for
  --lr                  Learning Rate. Default=0.01
  --accel                use accelerator
  --threads             number of threads for data loader to use Default=4
  --seed                random seed to use. Default=123

This example trains a super-resolution network on the BSD300 dataset, using crops from the 200 training images, and evaluating on crops of the 100 test images. A snapshot of the model after every epoch with filename model_epoch_<epoch_number>.pth.

Example Usage:

Train

bash
python main.py --upscale_factor 3 --batchSize 4 --testBatchSize 100 --nEpochs 30 --lr 0.001 --accel

Super Resolve

bash
python super_resolve.py --input_image dataset/BSDS300/images/test/16077.jpg --model model_epoch_30.pth --output_filename out.png --accel

Overview (`time_sequence_prediction/README.md`)

Time Sequence Prediction

This is a toy example for beginners to start with. It helps learn both PyTorch and time sequence prediction. Two LSTMCell units are used in this example to learn some sine wave signals starting at different phases. After learning the sine waves, the network tries to predict the signal values in the future. The results are shown in the picture below.

Usage

text
python generate_sine_wave.py
python train.py

Result

The initial signal and the predicted results are shown in the image. We first give some initial signals (full line). The network will subsequently give some predicted results (dash line). It can be concluded that the network can generate new sine waves.

Overview (`vae/README.md`)

Basic VAE Example

This is an improved implementation of the paper Auto-Encoding Variational Bayes by Kingma and Welling.
It uses ReLUs and the adam optimizer, instead of sigmoids and adagrad. These changes make the network converge much faster.

Usage

Install the required dependencies:

bash
pip install -r requirements.txt

To run the example, execute:

bash
python main.py

If a hardware accelerator device is detected, the example will execute on the accelerator; otherwise, it will run on the CPU.

To force execution on the CPU, use --no-accel command line argument:

bash
python main.py --no-accel

The main.py script accepts the following optional arguments:

bash
--batch-size            input batch size for training (default: 128)
--epochs                number of epochs to train (default: 10)
--no-accel              disables accelerator
--seed                  random seed (default: 1)
--log-interval	        how many batches to wait before logging training status

Overview (`word_language_model/README.md`)

Word-level Language Modeling using RNN and Transformer

This example trains a multi-layer RNN (Elman, GRU, or LSTM) or Transformer on a language modeling task. By default, the training script uses the Wikitext-2 dataset, provided.
The trained model can then be used by the generate script to generate new text.

bash
python main.py --accel --epochs 6           # Train a LSTM on Wikitext-2.
python main.py --accel --epochs 6 --tied    # Train a tied LSTM on Wikitext-2.
python main.py --accel --tied               # Train a tied LSTM on Wikitext-2for 40 epochs.
python main.py --accel --epochs 6 --model Transformer --lr 5
                                            # Train a Transformer model on Wikitext-2.
python main.py --accel --epochs 6 --model Transformer --use-optimizer --lr 0.001
                                            # Train a Transformer model with AdamW optimizer on Wikitext-2.

python generate.py --accel                  # Generate samples from the default model checkpoint.

[!NOTE]
Example supports running on acceleration devices (CUDA, MPS, XPU)

The model uses the nn.RNN module (and its sister modules nn.GRU and nn.LSTM) or Transformer module (nn.TransformerEncoder and nn.TransformerEncoderLayer) which will automatically use the cuDNN backend if run on CUDA with cuDNN installed.

During training, if a keyboard interrupt (Ctrl-C) is received, training is stopped and the current model is evaluated against the test dataset.

The main.py script accepts the following arguments:

bash
optional arguments:
  -h, --help            show this help message and exit
  --data DATA           location of the data corpus
  --model MODEL         type of network (RNN_TANH, RNN_RELU, LSTM, GRU, Transformer)
  --emsize EMSIZE       size of word embeddings
  --nhid NHID           number of hidden units per layer
  --nlayers NLAYERS     number of layers
  --lr LR               initial learning rate
  --clip CLIP           gradient clipping
  --epochs EPOCHS       upper epoch limit
  --batch_size N        batch size
  --bptt BPTT           sequence length
  --dropout DROPOUT     dropout applied to layers (0 = no dropout)
  --tied                tie the word embedding and softmax weights
  --seed SEED           random seed
  --accel               use accelerator
  --log-interval N      report interval
  --save SAVE           path to save the final model
  --onnx-export ONNX_EXPORT
                        path to export the final model in onnx format
  --nhead NHEAD         the number of heads in the encoder/decoder of the transformer model
  --dry-run             verify the code and the model
  --use-optimizer       specify whether to use an AdamW optimizer

With these arguments, a variety of models can be tested.
As an example, the following arguments produce slower but better models:

bash
python main.py --accel --emsize 650 --nhid 650 --dropout 0.5 --epochs 40
python main.py --accel --emsize 650 --nhid 650 --dropout 0.5 --epochs 40 --tied
python main.py --accel --emsize 1500 --nhid 1500 --dropout 0.65 --epochs 40
python main.py --accel --emsize 1500 --nhid 1500 --dropout 0.65 --epochs 40 --tied

Overview (`cpp/autograd/README.md`)

C++ autograd example

autograd.cpp contains several examples of doing autograd in PyTorch C++ frontend.

To build the code, run the following commands from your terminal:

shell
$ cd autograd
$ mkdir build
$ cd build
$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..
$ make

where /path/to/libtorch should be the path to the unzipped LibTorch
distribution, which you can get from the PyTorch
homepage.

Execute the compiled binary to run:

shell
$ ./autograd
====== Running: "Basic autograd operations" ======
 1  1
 1  1
[ CPUFloatType{2,2} ]
 3  3
 3  3
[ CPUFloatType{2,2} ]
AddBackward1
 27  27
 27  27
[ CPUFloatType{2,2} ]
MulBackward1
27
[ CPUFloatType{} ]
MeanBackward0
false
true
SumBackward0
 4.5000  4.5000
 4.5000  4.5000
[ CPUFloatType{2,2} ]
  813.6625
 1015.0142
 -664.8849
[ CPUFloatType{3} ]
MulBackward1
  204.8000
 2048.0000
    0.2048
[ CPUFloatType{3} ]
true
true
false
true
false
true

====== Running "Computing higher-order gradients in C++" ======
 0.0025  0.0946  0.1474  0.1387
 0.0238 -0.0018  0.0259  0.0094
 0.0513 -0.0549 -0.0604  0.0210
[ CPUFloatType{3,4} ]

====== Running "Using custom autograd function in C++" ======
-3.5513  3.7160  3.6477
-3.5513  3.7160  3.6477
[ CPUFloatType{2,3} ]
 0.3095  1.4035 -0.0349
 0.3095  1.4035 -0.0349
 0.3095  1.4035 -0.0349
 0.3095  1.4035 -0.0349
[ CPUFloatType{4,3} ]
 5.5000
 5.5000
[ CPUFloatType{2} ]

Overview (`cpp/custom-dataset/README.md`)

Custom Dataset Example with the PyTorch C++ Frontend

This folder contains an example of loading a custom image dataset with OpenCV and training a model to label images, using the PyTorch C++ frontend.

The dataset used here is Caltech 101 dataset.

The entire training code is contained in custom-data.cpp.

You can find instructions on how to install OpenCV here.

To build the code, run the following commands from your terminal:

shell
$ cd custom-dataset
$ mkdir build
$ cd build
$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..
$ make

where /path/to/libtorch should be the path to the unzipped LibTorch distribution, which you can get from the PyTorch homepage.

If you see an error like undefined reference to cv::imread(std::string const&, int) when running the make command, you should build LibTorch from source using the instructions here, and then set CMAKE_PREFIX_PATH to that PyTorch source directory. An alternative solution is to use libtorch-cxx11-abi-shared-with-deps instead of libtorch-shared-with-deps as the latter is not compatible with openCV (reported here).

The build directory should look like this:

text
.
├── custom-dataset
├── dataset
│   ├── accordion
│   │   ├── image_0001.jpg
│   │   ├── ...
│   ├── airplanes
│   │   ├── ...
│   ├── ...
├── info.txt
└── Makefile
└── ...

info.txt file gets copied from source directory during build.

Execute the compiled binary to train the model:

shell
./custom-dataset
Running on: CUDA
Train Epoch: 1 16/7281	Loss: 0.314655	Acc: 0
Train Epoch: 1 176/7281	Loss: 0.532111	Acc: 0.0681818
Train Epoch: 1 336/7281	Loss: 0.538482	Acc: 0.0714286
Train Epoch: 1 496/7281	Loss: 0.535302	Acc: 0.0705645
Train Epoch: 1 656/7281	Loss: 0.536113	Acc: 0.0716463
Train Epoch: 1 816/7281	Loss: 0.537626	Acc: 0.0784314
Train Epoch: 1 976/7281	Loss: 0.537055	Acc: 0.079918
...

Overview (`cpp/dcgan/README.md`)

DCGAN Example with the PyTorch C++ Frontend

This folder contains an example of training a DCGAN to generate MNIST digits
with the PyTorch C++ frontend.

The entire training code is contained in dcgan.cpp.

You can find the commands to install argparse here.

To build the code, run the following commands from your terminal:

shell
$ cd dcgan
$ mkdir build
$ cd build
$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..
$ make

where /path/to/libtorch should be the path to the unzipped LibTorch
distribution, which you can get from the PyTorch
homepage.

Execute the compiled binary to train the model:

shell
$ ./dcgan
[ 1/30][200/938] D_loss: 0.4953 | G_loss: 4.0195
-> checkpoint 1
[ 1/30][400/938] D_loss: 0.3610 | G_loss: 4.8148
-> checkpoint 2
[ 1/30][600/938] D_loss: 0.4072 | G_loss: 4.36760
-> checkpoint 3
[ 1/30][800/938] D_loss: 0.4444 | G_loss: 4.0250
-> checkpoint 4
[ 2/30][200/938] D_loss: 0.3761 | G_loss: 3.8790
-> checkpoint 5
[ 2/30][400/938] D_loss: 0.3977 | G_loss: 3.3315
-> checkpoint 6
[ 2/30][600/938] D_loss: 0.3815 | G_loss: 3.5696
-> checkpoint 7
[ 2/30][800/938] D_loss: 0.4039 | G_loss: 3.2759
-> checkpoint 8
[ 3/30][200/938] D_loss: 0.4236 | G_loss: 4.5132
-> checkpoint 9
[ 3/30][400/938] D_loss: 0.3645 | G_loss: 3.9759
-> checkpoint 10
...

We can also specify the --epochs to change the number of epochs to train as follows:

shell
$ ./dcgan --epochs 10

Without specifying the --epochs flag, the default number of epochs to train is 30.

The training script periodically generates image samples. Use the
display_samples.py script situated in this folder to generate a plot image.
For example:

shell
$ python display_samples.py -i dcgan-sample-10.pt
Saved out.png

Overview (`cpp/distributed/README.md`)

Distributed Training on MNIST using PyTorch C++ Frontend (Libtorch)

This folder contains an example of data-parallel training of a convolutional neural network on the MNIST dataset. For parallelization, Message Passing Interface (MPI) is used.

The entire code is contained in dist-mnist.cpp

You can find instructions on how to install MPI [here] (https://www.open-mpi.org/faq/?category=building). This code was tested on Open MPI but it should run on other MPI distributions as well such as MPICH, MVAPICH, etc.

To build the code, run the following commands from the terminal:

shell
$ cd distributed
$ mkdir build
$ cd build
$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..
$ make

where /path/to/libtorch should be the path to the unzipped LibTorch distribution. Note that the LibTorch from the [PyTorch homepage] ((https://pytorch.org/get-started/locally/) does not include MPI headers and cannot be used for this example. You have to compile LibTorch manually - a set of guidelines is provided [here] (https://gist.github.com/lasagnaphil/3e0099816837318e8e8bcab7edcfd5d9), however this may vary for different systems.

To run the code,

shell
mpirun -np {NUM-PROCS} ./dist-mnist

Overview (`cpp/mnist/README.md`)

MNIST Example with the PyTorch C++ Frontend

This folder contains an example of training a computer vision model to recognize
digits in images from the MNIST dataset, using the PyTorch C++ frontend.

The entire training code is contained in mnist.cpp.

To build the code, run the following commands from your terminal:

shell
$ cd mnist
$ mkdir build
$ cd build
$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..
$ make

where /path/to/libtorch should be the path to the unzipped LibTorch
distribution, which you can get from the PyTorch
homepage.

Execute the compiled binary to train the model:

shell
$ ./mnist
Train Epoch: 1 [59584/60000] Loss: 0.4232
Test set: Average loss: 0.1989 | Accuracy: 0.940
Train Epoch: 2 [59584/60000] Loss: 0.1926
Test set: Average loss: 0.1338 | Accuracy: 0.959
Train Epoch: 3 [59584/60000] Loss: 0.1390
Test set: Average loss: 0.0997 | Accuracy: 0.969
Train Epoch: 4 [59584/60000] Loss: 0.1239
Test set: Average loss: 0.0875 | Accuracy: 0.972
...

Overview (`cpp/regression/README.md`)

Linear regression example

Trains a single fully-connected layer to fit a 4th degree polynomial.

To build the code, run the following commands from your terminal:

shell
$ cd regression
$ mkdir build
$ cd build
$ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch ..
$ make

where /path/to/libtorch should be the path to the unzipped LibTorch
distribution, which you can get from the PyTorch
homepage.

Execute the compiled binary to run:

shell
$ ./regression
Loss: 0.000301158 after 584 batches
==> Learned function:	y = 11.6441 x^4 -3.10164 x^3 2.19786 x^2 -3.83606 x^1 + 4.37066
==> Actual function:	y = 11.669 x^4 -3.16023 x^3 2.19182 x^2 -3.81505 x^1 + 4.38219
...

Overview (`cpp/transfer-learning/README.md`)

Transfer Learning on Dogs vs Cats Dataset using Libtorch and OpenCV

Transfer Learning on Dogs vs Cats dataset using PyTorch C++ API.

Usage

For training:

  1. Remove final layer of ResNet18 pre-trained model and convert to torch.jit module: python3 convert.py.
  2. Create build directory: mkdir build && cd build
  3. cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..
  4. make
  5. Run training code: ./example

For prediction:

  1. cd build
  2. ./classify <model_linear_trained> : ./classify ../resnet18_without_last_layer.pt model_linear.pt

Detailed blog on applying Transfer Learning using Libtorch: https://krshrimali.github.io/Applying-Transfer-Learning-Dogs-Cats/.

Overview (`distributed/FSDP/README.md`)

Note: FSDP1 is deprecated. Please follow FSDP2 tutorial and code examples.

FSDP1 T5

To run the T5 example with FSDP1 for text summarization:

Get the wikihow dataset

bash
sh download_dataset.sh

Install the requirements:

text
pip install -r requirements.txt

Ensure you are running a recent version of PyTorch:

see https://pytorch.org to install at least 1.12 and ideally a current nightly build.

Start the training with Torchrun (adjust nproc_per_node to your GPU count):

text
torchrun --nnodes 1 --nproc_per_node 4  T5_training.py

Overview (`distributed/FSDP2/README.md`)

FSDP2

To run FSDP2 on transformer model:

text
cd distributed/FSDP2
pip install -r requirements.txt
torchrun --nproc_per_node 2 example.py
  • For 1st time, it creates a "checkpoints" folder and saves state dicts there
  • For 2nd time, it loads from previous checkpoints

To enable explicit prefetching

text
torchrun --nproc_per_node 2 example.py --explicit-prefetch

To enable mixed precision

text
torchrun --nproc_per_node 2 example.py --mixed-precision

To showcase DCP API

text
torchrun --nproc_per_node 2 example.py --dcp-api

Ensure you are running a recent version of PyTorch:

see https://pytorch.org/get-started/locally/ to install at least 2.5 and ideally a current nightly build.

Overview (`distributed/ddp-tutorial-series/README.md`)

distributed-pytorch

Code for the DDP tutorial series at https://pytorch.org/tutorials/beginner/ddp_series_intro.html

Each code file extends upon the previous one. The series starts with a non-distributed script that runs on a single GPU and incrementally updates to end with multinode training on a Slurm cluster.

Files

Overview (`distributed/ddp/README.md`)

Distributed Data Parallel (DDP) Applications with PyTorch

This guide demonstrates how to structure a distributed model training application for convenient multi-node launches using torchrun.

Prerequisites

You should be familiar with:

This tutorial uses the torch.nn.parallel.DistributedDataParallel (DDP) class for data parallel training: multiple workers train the same global model on different data shards, compute local gradients, and synchronize them using AllReduce. In High-Performance Computing (HCP), this is called Single Program Multiple Data (SPMD).

Application Process Topologies

A Distributed Data Parallel (DDP) application can be executed on
multiple nodes where each node can consist of multiple GPU
devices. Each node in turn can run multiple copies of the DDP
application, each of which processes its models on multiple GPUs.

Let:

  • N = number of nodes
  • G = number of GPUs per node
  • W = World Size = total number of processes
  • L = Local World Size = processes per node

Each process has:

  • Local rank: in [0, L-1]
  • Global rank: in [0, W-1]

Example:
If you launch a DDP app on 2 nodes, each with 4 GPUs, and want each process to span 2 GPUs, the mapping is as follows:

While there are quite a few ways to map processes to nodes, a good rule of thumb is to have one process span a single GPU. This enables the DDP application to have as many parallel reader streams as there are GPUs and in practice provides a good balance between I/O and computational costs. In the rest of this tutorial, we assume that the application follows this heuristic.

Preparing and launching a DDP application

Independent of how a DDP application is launched, each process needs a mechanism to know its global and local ranks. Once this is known, all processes create a ProcessGroup that enables them to participate in collective communication operations such as AllReduce.

A convenient way to start multiple DDP processes and initialize all values needed to create a ProcessGroup is to use the torchrun script provided with PyTorch.

Sample Application

This example is based on the "Hello, World" DDP tutorial.

The application calls the spmd_main entrypoint:

python
if __name__ == "__main__":
    spmd_main()

In spmd_main, the process group is initialized using the Accelerator API. The rest of the rendezvous information comes from environment variables set by torchrun:

python
def spmd_main():
    # These are the parameters used to initialize the process group
    env_dict = {
        key: os.environ[key]
        for key in ("MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE")
    }
    rank = int(env_dict['RANK'])
    local_rank = int(env_dict['LOCAL_RANK'])
    local_world_size = int(env_dict['LOCAL_WORLD_SIZE'])

    print(f"[{os.getpid()}] Initializing process group with: {env_dict}")
    acc = torch.accelerator.current_accelerator()
    vendor_backend = torch.distributed.get_default_backend_for_device(acc)
    torch.accelerator.set_device_index(rank)
    torch.distributed.init_process_group(backend=vendor_backend)

    demo_basic(rank)

    # Tear down the process group
    torch.distributed.destroy_process_group()

Key points:

  • Each process reads its rank and world size from environment variables.
  • The process group is initialized for distributed communication.

The training function, demo_basic, initializes the DDP model on the appropriate GPU:

python
def demo_basic(rank):
    print(
        f"[{os.getpid()}] rank = {torch.distributed.get_rank()}, "
        + f"world_size = {torch.distributed.get_world_size()}"
    )

    model = ToyModel().to(rank)
    ddp_model = DDP(model, device_ids=[rank])

    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    optimizer.zero_grad()
    outputs = ddp_model(torch.randn(20, 10))
    labels = torch.randn(20, 5).to(rank)
    loss_fn(outputs, labels).backward()
    optimizer.step()

Launching the Application

sh
torchrun --nnodes=1 --nproc_per_node=8 example.py

Example Output

Expected output:

text
/* Detailed source-code truncated for AI context efficiency. */

Conclusions

As the author of a distributed data parallel application, your code needs to be aware of two types of resources: compute nodes and the GPUs within each node. The process of setting up bookkeeping to track how the set of GPUs is mapped to the processes of your application can be tedious and error-prone. We hope that by structuring your application as shown in this example and using torchrun, the mechanics of setting up distributed training can be significantly simplified.

Overview (`distributed/minGPT-ddp/README.md`)

minGPT-DDP

Code accompanying the tutorial at https://pytorch.org/tutorials/intermediate/ddp_series_minGPT.html for training a GPT-like model with Distributed Data Parallel (DDP) in PyTorch.

Files marked with an asterisk (*) are adapted from the minGPT repo (https://github.com/karpathy/minGPT).

  • trainer.py includes the Trainer class that runs the distributed training iterations on the model with the provided dataset.
  • model.py * defines the model architecture.
  • char_dataset.py * contains the Datasetclass for a character-level dataset.
  • gpt2_train_cfg.yaml contains the configurations for data, model, optimizer and training run.
  • main.py is the entry point to the trainig job. It sets up the DDP process group, reads all the configurations and runs the training job.
  • slurm/ contains files for setting up an AWS cluster and the slurm script to run multinode training.

Overview (`distributed/tensor_parallelism/README.md`)

PyTorch native Tensor Parallel for distributed training

This example demonstrates SPMD Megatron-LM style Tensor Parallel by using
PyTorch native Tensor Parallel APIs, which include:

  1. Simple module-level Tensor Parallelism on a dummy MLP model.
  2. Simple module-level Tensor Parallelism with Sequence Parallel inputs/outputs on a dummy MLP model.
  3. A E2E demo of Fully Sharded Data Parallel + Tensor Parallel (with Sequence Parallel) on a example Llama2 model.

More details about the PyTorch native Tensor Parallel APIs, please see PyTorch docs:
https://pytorch.org/docs/stable/distributed.tensor.parallel.html

Installation

bash
pip install -r requirements.txt

Running Examples

You can run the examples using torchrun to launch distributed training:

bash
# Simple Tensor Parallel example
torchrun --nnodes=1 --nproc_per_node=4 tensor_parallel_example.py

# Tensor Parallel with Sequence Parallel
torchrun --nnodes=1 --nproc_per_node=4 sequence_parallel_example.py

# FSDP + Tensor Parallel with Llama2 model
torchrun --nnodes=1 --nproc_per_node=4 fsdp_tp_example.py

For more details, check the run_examples.sh script.

Overview (`fx/native_interpreter/README.md`)

Converting PyTorch Code to a Native Runtime With FX and TorchScript Custom Classes

In this example, we are going to build a pipeline that does the following things:

  1. Converts (or “lowers”) code in a PyTorch module into another representation (we will define the representation within the example)
  2. Registers an interpreter for that code representation that can be used in TorchScript or Python
  3. Wrap the converted code into a format that can still be used in TorchScript compilation.

We are going to build up a trivial interpreter for this example, but you can imagine extending the same process to work with more sophisticated backends, ones which may do code optimization or offloading to an accelerator.

We will be using TorchScript custom classes to expose this Interpreter to Python and TorchScript. You may want to review that tutorial and documentation before reading this example project.

Defining the Interpreter

We define the interpreter in interpreter.cpp. This interpreter is very limited: it only supports two element-wise operations (add and mul) and it only supports Tensor values. When this interpreter runs code, it iterates through the list of instructions and simply calls the appropriate PyTorch operator from C++.

To build the interpreter into a shared-object file to be loaded in for use, use the following commands from this example’s root:

text
$ mkdir build
$ cd build
$ cmake -DCMAKE_PREFIX_PATH="$(python -c 'import torch.utils; print(torch.utils.cmake_prefix_path)')" ..
$ make -j

After the build finishes, you should see build/libinterpreter.so (or with a different extension depending on your OS). We will use this dynamic library next when we load it up into a process to be used in execution.

Defining the Transformation

We define the code that transforms a PyTorch module to the format the interpreter understands in use_interpreter.py. Note that that file loads in the shared object we built in the previous step via a torch.classes.load_library call. use_interpreter.py contains driver code and the end that can be directly run to test the lowering transformation.

Questions, Comments, Feedback

Please direct questions and discussion to the PyTorch forums. To report any issues with PyTorch (including FX and custom classes), please use the issue tracker.

Overview (`legacy/snli/README.md`)

PyTorch-based NLI Training with SNLI

📝 Overview

This repository contains Python scripts to train a Natural Language Inference (NLI) model, specifically the SNLIClassifier, using the Stanford Natural Language Inference (SNLI) corpus. The trained model predicts textual entailment, identifying if a statement is entailed, contradicted, or neither by another statement.

⚙️ Dependencies

Install the necessary Python libraries with:

bash
pip install -r requirements.txt

The requirements.txt file includes:

text
torch
torchtext
spacy

💻 Usage

Start the training process with:

bash
python train.py --lower --word-vectors [PATH_TO_WORD_VECTORS] --vector-cache [PATH_TO_VECTOR_CACHE] --epochs [NUMBER_OF_EPOCHS] --batch-size [BATCH_SIZE] --save-path [PATH_TO_SAVE_MODEL] --gpu [GPU_NUMBER]

🏋️‍♀️ Training

The script trains the model on mini-batches of data across a specified number of epochs. It saves the best-performing model on the validation set as a .pt file in the specified directory.

📚 Scripts

  • model.py: Defines the SNLIClassifier model and auxiliary classes.
  • util.py: Contains utility functions for directory creation and command-line argument parsing.

📣 Note

Ensure the model.py and util.py scripts are available in your working directory.

Overview (`distributed/rpc/batch/README.md`)

Examples For Asynchronous RPC User Functions

This folder contains two examples for @rpc.functions.async_execution:

  1. Synchronized Batch Update Parameter Server: uses @rpc.functions.async_execution
    for parameter update and retrieving. This serves as a simple starter example
    for batch RPC.
    text
    pip install -r requirements.txt
python parameter_server.py
  2. Multi-Observer with Batch-Processing Agent: uses @rpc.functions.async_execution
    to run multiple observed states through the policy to get actions.
    text
    pip install -r requirements.txt
python reinforce.py

Overview (`distributed/rpc/ddp_rpc/README.md`)

Distributed DataParallel + Distributed RPC Framework Example

This example demonstrates how to combine Distributed DataParallel (DDP) with the Distributed RPC Framework. It requires two trainer nodes (each with a GPU), one master node, and one parameter server.

The master node initializes an embedding table on the parameter server and orchestrates the training loop across the trainers. The model is composed of a dense component (nn.Linear), which is replicated on the trainers using DDP, and a sparse component (nn.EmbeddingBag), which resides on the parameter server.

Each trainer performs embedding lookups on the parameter server via RPC, then processes the results through its local nn.Linear module. During the backward pass, DDP aggregates gradients for the dense part using allreduce, while the distributed backward pass updates the embedding table parameters on the parameter server.

text
pip install -r requirements.txt
python main.py

Overview (`distributed/rpc/parameter_server/README.md`)

RPC-based distributed training

This is a basic example of RPC-based training that uses several trainers remotely train a model hosted on a server.

To run the example locally, run the following command worker for the server and each worker you wish to spawn, in separate terminal windows:
python rpc_parameter_server.py --world_size=WORLD_SIZE --rank=RANK. For example, for a master node with world size of 2, the command would be python rpc_parameter_server.py --world_size=2 --rank=0. The trainer can then be launched with the command python rpc_parameter_server.py --world_size=2 --rank=1 in a separate window, and this will begin training with one server and a single trainer.

Note that for demonstration purposes, this example supports only between 0-2 GPUs, although the pattern can be extended to make use of additional GPUs. To configure the number of GPUs, pass in --num_gpus=N to your training command.

You can pass in the command line arguments --master_addr=<address> and master_port=PORT to indicate the address:port that the master worker is listening on. All workers will contact the master for rendezvous during worker discovery. By default, master_addr will be localhost and master_port will be 29500.

Overview (`distributed/rpc/pipeline/README.md`)

Distributed Pipeline Parallel Example

This example shows how to distribute a ResNet50 model on two RPC workers and
then implement distributed pipeline parallelism using RPC. With pipeline
parallelism, every input batch is divided into micro-batches and thse
micro-batches are feed into the model in a pipelined fashion to increase the
amortized device utilization. Note that this example only parallelizes the
forward pass which can be viewed as the distributed counterpart of the
single machine pipeline parallel
example.

text
pip install -r requirements.txt
python main.py

Overview (`distributed/rpc/rl/README.md`)

Distributed Multi-Observer Single-Agent Reinforcement Learning Example

This example demonstrates torch.distributed.rpc API using an CartPole
reinforcement learning example. Please note that the goal is to present the RPC
API instead of building the best CartPole solver.

text
pip install -r requirements.txt
python main.py

Overview (`distributed/rpc/rnn/README.md`)

Distributed RNN Model Parallel Example

This example shows how to build an RNN model using RPC where different
components of the RNN model can be placed on different workers.

text
pip install -r requirements.txt
python main.py

Requirements (`docs/requirements.txt`)

sphinx==5.3.0
#Pinned versions: 5.3.0
-e git+https://github.com/pytorch/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme
sphinx-panels

Source Index (`docs/source/index.rst`)

PyTorch Examples

This pages lists various PyTorch examples that you can use to learn and
experiment with PyTorch.

.. panels::

text
Image Classification Using ConvNets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This example demonstrates how to run image classification
with `Convolutional Neural Networks ConvNets <https://cs231n.github.io/convolutional-networks/>`__
on the `MNIST <https://en.wikipedia.org/wiki/MNIST_database>`__ database.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/mnist>`__ :opticon:`link-external` 

---

Measuring Similarity using Siamese Network
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This example demonstrates how to measure similarity between two images
using `Siamese network <https://en.wikipedia.org/wiki/Siamese_neural_network>`__
on the `MNIST <https://en.wikipedia.org/wiki/MNIST_database>`__ database.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/siamese_network>`__ :opticon:`link-external` 

---

Word-level Language Modeling using RNN and Transformer
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This example demonstrates how to train a multi-layer `recurrent neural
network (RNN) <https://en.wikipedia.org/wiki/Recurrent_neural_network>`__,
such as Elman, GRU, or LSTM, or Transformer on a language
modeling task by using the Wikitext-2 dataset. 

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/word_language_model>`__ :opticon:`link-external`
---

Training ImageNet Classifiers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This example demonstrates how you can train some of the most popular
model architectures, including `ResNet <https://en.wikipedia.org/wiki/Residual_neural_network>`__, 
`AlexNet <https://en.wikipedia.org/wiki/AlexNet>`__, and `VGG <https://arxiv.org/pdf/1409.1556.pdf>`__
on the `ImageNet <https://image-net.org/>`__ dataset.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/imagenet>`__ :opticon:`link-external`
---

Generative Adversarial Networks (DCGAN)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This example implements the `Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks  <https://arxiv.org/abs/1511.06434>`__ paper.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/dcgan>`__ :opticon:`link-external`
---
 
Variational Auto-Encoders
^^^^^^^^^^^^^^^^^^^^^^^^^

This example implements the `Auto-Encoding Variational Bayes <https://arxiv.org/abs/1312.6114>`__ paper
with `ReLUs <https://en.wikipedia.org/wiki/Rectifier_(neural_networks)>`__ and the Adam optimizer.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/vae>`__ :opticon:`link-external`
---

Super-resolution Using an Efficient Sub-Pixel CNN
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This example demonstrates how to use the sub-pixel convolution layer
described in `Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network <https://arxiv.org/abs/1609.05158>`__ paper. This example trains a super-resolution
network on the `BSD300 dataset <https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/>`__. 

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/super_resolution>`__ :opticon:`link-external`

---
HOGWILD! Training of Shared ConvNets
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

`HOGWILD! <https://arxiv.org/abs/1106.5730>`__ is a scheme that allows
Stochastic Gradient Descent (SGD)
parallelization without memory locking. This example demonstrates how
to perform HOGWILD! training of shared ConvNets on MNIST.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/mnist_hogwild>`__ :opticon:`link-external`

---
Training a CartPole to balance with actor-critic
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This reinforcement learning tutorial demonstrates how to train a
CartPole to balance
in the `Gymnasium <https://gymnasium.farama.org/>`__ toolkit by using the
`Actor-Critic <https://proceedings.neurips.cc/paper/1999/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf>`__ method.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/reinforcement_learning>`__ :opticon:`link-external`
---

Time Sequence Prediction
^^^^^^^^^^^^^^^^^^^^^^^^

This beginner example demonstrates how to use LSTMCell to
learn sine wave signals to predict the signal values in the future.

`GO TO EXAMPLE <https://github.com/pytorch/examples/tree/main/time_sequence_prediction>`__ :opticon:`link-external`

---

Implement the Neural Style Transfer algorithm on images
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This tutorial demonstrates how you can use PyTorch's implementation
of the `Neural Style Transfer (NST) <https://en.wikipedia.org/wiki/Neural_style_transfer>`__
algorithm on images.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/fast_neural_style>`__ :opticon:`link-external`
---

PyTorch Module Transformations using fx
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This set of examples demonstrates the torch.fx toolkit. For more
information about `torch.fx`, see
`torch.fx Overview <https://pytorch.org/docs/master/fx.html>`__.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/fx>`__ :opticon:`link-external`
---

Distributed PyTorch
^^^^^^^^^^^^^^^^^^^

This set of examples demonstrates `Distributed Data Parallel (DDP) <https://pytorch.org/tutorials/intermediate/ddp_tutorial.html>`__ and `Distributed RPC framework <https://pytorch.org/docs/stable/rpc.html>`__. 
Includes the code used in the `DDP tutorial series <https://pytorch.org/tutorials/beginner/ddp_series_intro.html>`__.

`GO TO EXAMPLES <https://github.com/pytorch/examples/tree/main/distributed>`__ :opticon:`link-external`

---

C++ Frontend
^^^^^^^^^^^^

The PyTorch C++ frontend is a C++14 library for CPU and GPU tensor computation.
This set of examples includes a linear regression, autograd, image recognition
(MNIST), and other useful examples using PyTorch C++ frontend.

`GO TO EXAMPLES <https://github.com/pytorch/examples/tree/main/cpp>`__ :opticon:`link-external`

---

Image Classification Using Forward-Forward Algorithm
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This example implements the paper `The Forward-Forward Algorithm: Some Preliminary Investigations <https://arxiv.org/pdf/2212.13345.pdf>`__ by Geoffrey Hinton.
on the `MNIST <https://en.wikipedia.org/wiki/MNIST_database>`__ database.
It is an introductory example to the Forward-Forward algorithm.

`GO TO EXAMPLE <https://github.com/pytorch/examples/tree/main/mnist_forward_forward>`__ :opticon:`link-external` 

---

Graph Convolutional Network
^^^^^^^^^^^^^^^^^^^^^^^^^^^

This example implements the `Semi-Supervised Classification with Graph Convolutional Networks <https://arxiv.org/pdf/1609.02907.pdf>`__ paper on the CORA database.

`GO TO EXAMPLE <https://github.com/pytorch/examples/blob/main/gcn>`__ :opticon:`link-external`

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →