# Technical Documentation: pytorch/examples > ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/pytorch/examples) ### Overview (`README.md`) # PyTorch Examples https://pytorch.org/examples/ `pytorch/examples` is a repository showcasing examples of using [PyTorch](https://github.com/pytorch/pytorch). The goal is to have curated, short, few/no dependencies _high quality_ examples that are substantially different from each other that can be emulated in your existing work. - For tutorials: https://github.com/pytorch/tutorials - For changes to pytorch.org: https://github.com/pytorch/pytorch.github.io - For a general model hub: https://pytorch.org/hub/ or https://huggingface.co/models - For recipes on how to run PyTorch in production: https://github.com/facebookresearch/recipes - For general Q&A and support: https://discuss.pytorch.org/ ## Available models - [Image classification (MNIST) using Convnets](./mnist/README.md) - [Word-level Language Modeling using RNN and Transformer](./word_language_model/README.md) - [Training Imagenet Classifiers with Popular Networks](./imagenet/README.md) - [Generative Adversarial Networks (DCGAN)](./dcgan/README.md) - [Variational Auto-Encoders](./vae/README.md) - [Superresolution using an efficient sub-pixel convolutional neural network](./super_resolution/README.md) - [Hogwild training of shared ConvNets across multiple processes on MNIST](mnist_hogwild) - [Training a CartPole to balance with actor-critic](./reinforcement_learning/README.md) - [Natural Language Inference (SNLI) with GloVe vectors, LSTMs, and torchtext](snli) - [Time sequence prediction - use an LSTM to learn Sine waves](./time_sequence_prediction/README.md) - [Implement the Neural Style Transfer algorithm on images](./fast_neural_style/README.md) - [Reinforcement Learning with Actor Critic and REINFORCE algorithms on OpenAI gym](./reinforcement_learning/README.md) - [PyTorch Module Transformations using fx](./fx/README.md) - Distributed PyTorch examples with [Distributed Data Parallel](./distributed/ddp/README.md) and [RPC](./distributed/rpc) - [Several examples illustrating the C++ Frontend](cpp) - [Image Classification Using Forward-Forward](./mnist_forward_forward/README.md) - [Language Translation using Transformers](./language_translation/README.md) Additionally, a list of good examples hosted in their own repositories: - [Neural Machine Translation using sequence-to-sequence RNN with attention (OpenNMT)](https://github.com/OpenNMT/OpenNMT-py) ## Contributing If you'd like to contribute your own example or fix a bug please make sure to take a look at [CONTRIBUTING.md](CONTRIBUTING.md). --- ### Overview (`dcgan/README.md`) # Deep Convolution Generative Adversarial Networks This example implements the paper [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](http://arxiv.org/abs/1511.06434) The implementation is very close to the Torch implementation [dcgan.torch](https://github.com/soumith/dcgan.torch) After every 100 training iterations, the files `real_samples.png` and `fake_samples.png` are written to disk with the samples from the generative model. After every epoch, models are saved to: `netG_epoch_%d.pth` and `netD_epoch_%d.pth` ## Downloading the dataset You can download the LSUN dataset by cloning [this repo](https://github.com/fyu/lsun) and running ``` python download.py -c bedroom ``` ## Usage ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Overview (`fast_neural_style/README.md`) # fast-neural-style :city_sunrise: :rocket: This repository contains a pytorch implementation of an algorithm for artistic style transfer. The algorithm can be used to mix the content of an image with the style of another image. For example, here is a photograph of a door arch rendered in the style of a stained glass painting. The model uses the method described in [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://arxiv.org/abs/1603.08155) along with [Instance Normalization](https://arxiv.org/pdf/1607.08022.pdf). The saved-models for examples shown in the README can be downloaded from [here](https://www.dropbox.com/s/lrvwfehqdcxoza8/saved_models.zip?dl=0).

## Requirements The program is written in Python, and uses [pytorch](http://pytorch.org/), [scipy](https://www.scipy.org). A GPU is not necessary, but can provide a significant speed up especially for training a new model. Regular sized images can be styled on a laptop or desktop using saved models. ## Usage Stylize image ``` python neural_style/neural_style.py eval --content-image --model --output-image --accel ``` - `--content-image`: path to content image you want to stylize. - `--model`: saved model to be used for stylizing the image (eg: `mosaic.pth`) - `--output-image`: path for saving the output image. - `--content-scale`: factor for scaling down the content image if memory is an issue (eg: value of 2 will halve the height and width of content-image) - `--accel`: use accelerator Train model ```bash python neural_style/neural_style.py train --dataset --style-image --save-model-dir --epochs 2 --accel ``` There are several command line arguments, the important ones are listed below - `--dataset`: path to training dataset, the path should point to a folder containing another folder with all the training images. I used COCO 2014 Training images dataset [80K/13GB] [(download)](https://cocodataset.org/#download). - `--style-image`: path to style-image. - `--save-model-dir`: path to folder where trained model will be saved. - `--accel`: use accelerator. If `--accel` argument is given, pytorch will search for available hardware acceleration device and attempt to use it. This example is known to work on CUDA, MPS and XPU devices. Refer to `neural_style/neural_style.py` for other command line arguments. For training new models you might have to tune the values of `--content-weight` and `--style-weight`. The mosaic style model shown above was trained with `--content-weight 1e5` and `--style-weight 1e10`. The remaining 3 models were also trained with similar order of weight parameters with slight variation in the `--style-weight` (`5e10` or `1e11`). ## Models Models for the examples shown below can be downloaded from [here](https://www.dropbox.com/s/lrvwfehqdcxoza8/saved_models.zip?dl=0) or by running the script `download_saved_models.py`.

--- ### Overview (`fx/README.md`) # FX Examples This folder contains several examples of program transformations implemented using `torch.fx`. More information about FX can be found in the [documentation](https://pytorch.org/docs/master/fx.html). Note that all examples should be runnable as standalone Python files. In the case of an exception, the example will appear in a subfolder with a `README.md` file explaining how to run the example. As FX is currently in a Beta release, the API or these examples are subject to change. --- ### Overview (`gat/README.md`) # Graph Attention Network This repository contains a PyTorch implementation of the **Graph Attention Networks (GAT)** based on the paper ["Graph Attention Network" by Velickovic et al](https://arxiv.org/abs/1710.10903v3). The Graph Attention Network is a powerful graph neural network model for learning represtations on graph-structured data, which has shown excellent performance in various tasks such as node classification, link prediction, and graph classification. ## Overview The Graph Attention Network (GAT) is a graph neural network architecture designed specifically for handling graph-structured data. It leverages multi-head attention mechanism to capture the information of neighboring nodes in an attentive manner to learn represtations for each node. This attention mechanism allows the model to focus on relevant nodes and adaptively weight their contributions during message passing. Check out the following resources for more ino on GATs: - [Blog post by the main auther, Petar Velickovic](https://petar-v.com/GAT/) - [Main paper](https://doi.org/10.48550/arXiv.1710.10903) This repository provides a clean and short implementation of the official GAT model using PyTorch. The code is well-documented and easy to understand, making it a valuable resource for researchers and practitioners interested in graph deep learning. ## Key Features - **GAT Model**: Implementation of the Graph Attention Network model with multi-head attention based on the paper "Graph Attention Network" by Velickovic et al. - **Graph Attention Layers**: Implementation of graph convolutional layers that aggregate information from neighboring nodes using a self-attention mechanisms to learn node importance weights. - **Training and Evaluation**: Code for training GAT models on graph-structured data and evaluating their performance on node classification tasks on the *Cora* benchmark dataset. --- # Requirements - Python 3.7 or higher - PyTorch 2.0 or higher - Requests 2.31 or higher - NumPy 1.24 or higher # Dataset The implementation includes support for the Cora dataset, a standard benchmark dataset for graph-based machine learning tasks. The Cora dataset consists of scientific publications, where nodes represent papers and edges represent citation relationships. Each paper is associated with a binary label indicating one of seven classes. The dataset is downloaded, preprocessed and ready to use. # Model Architecture The official architecture (used in this project) proposed in the paper "Graph Attention Network" by Velickovic et al. consists of two graph attention layers which incorporates the multi-head attention mechanisms during its message trasformation and aggregation. Each graph attention layer applies a shared self-attention mechanism to every node in the graph, allowing them to learn different representations based on the importance of their neighbors. In terms of activation functions, the GAT model employs both the **Exponential Linear Unit (ELU)** and the **Leaky Rectified Linear Unit (LeakyReLU)** activations, which introduce non-linearity to the model. ELU is used as the activation function for the **hidden layers**, while LeakyReLU is applied to the **attention coefficients** to ensure non-zero gradients for negative values. Following the official implementation, the first GAT layer consists of **K = 8 attention heads** computing **F' = 8 features** each (for a **total of 64 features**) followed by an exponential linear unit (ELU) activation on the layer outputs. The second GAT layer is used for classification: a **single attention head** that computes C features (where C is the number of classes), followed by a softmax activation for probablisitic outputs. (we use log-softmax instead for computational convenience with using NLLLoss) *Note that due to being an educational example, this implementation uses the full dense form of the adjacency matrix of the graph, and not the sparse form of the matrix. Thus all the operations in the model implemeation is done in a non-sparse from. This will not affect the model's performance accuracy-wise. However an sparse-friendly implementation will help with the efficiency in the use of resources, storage, and speed.* # Usage Training and evaluating the GAT model on the Cora dataset can be done through running the `main.py` script as follows: 1. Clone the PyTorch examples repository: ``` git clone https://github.com/pytorch/examples.git cd examples/gat ``` 2. Install the required dependencies: ``` pip install -r requirements.txt ``` 3. Train the GAT model by running the `main.py` script as follows:: (Example using the default parameters) ```bash python main.py --epochs 300 --lr 0.005 --l2 5e-4 --dropout-p 0.6 --num-heads 8 --hidden-dim 64 --val-every 20 ``` In more detail, the `main.py` script recieves following arguments: ``` usage: main.py [-h] [--epochs EPOCHS] [--lr LR] [--l2 L2] [--dropout-p DROPOUT_P] [--hidden-dim HIDDEN_DIM] [--num-heads NUM_HEADS] [--concat-heads] [--val-every VAL_EVERY] [--no-cuda] [--no-mps] [--dry-run] [--seed S] PyTorch Graph Attention Network options: -h, --help show this help message and exit --epochs EPOCHS number of epochs to train (default: 300) --lr LR learning rate (default: 0.005) --l2 L2 weight decay (default: 6e-4) --dropout-p DROPOUT_P dropout probability (default: 0.6) --hidden-dim HIDDEN_DIM dimension of the hidden representation (default: 64) --num-heads NUM_HEADS number of the attention heads (default: 4) --concat-heads wether to concatinate attention heads, or average over them (default: False) --val-every VAL_EVERY epochs to wait for print training and validation evaluation (default: 20) --no-accel disables accelerator --dry-run quickly check a single pass --seed S random seed (default: 13) ``` # Results After training for **300 epochs** with default hyperparameters on random train/val/test data splits, the GAT model achieves around **%81.25** classification accuracy on the test split. This result is comparable to the performance reported in the original paper. However, the results can vary due to the randomness of the train/val/test split. # Reference ``` @article{ velickovic2018graph, title="{Graph Attention Networks}", author={Veli{\v{c}}kovi{\'{c}}, Petar and Cucurull, Guillem and Casanova, Arantxa and Romero, Adriana and Li{\`{o}}, Pietro and Bengio, Yoshua}, journal={International Conference on Learning Representations}, year={2018}, url={https://openreview.net/forum?id=rJXMpikCZ}, } ``` - Paper on arxiv: [arXiv:1710.10903v3](https://doi.org/10.48550/arXiv.1710.10903) - Original paper repository: [https://github.com/PetarV-/GAT](https://github.com/PetarV-/GAT) --- ### Overview (`gcn/README.md`) # Graph Convolutional Network This repository contains an implementation of Graph Convolutional Networks (GCN) based on the paper "Semi-Supervised Classification with Graph Convolutional Networks" by Thomas N. Kipf and Max Welling. ## Overview This project implements the GCN model proposed in the paper for semi-supervised node classification on graph-structured data. GCN leverages graph convolutions to aggregate information from neighboring nodes and learn node representations for downstream tasks. The implementation provides a flexible and efficient GCN model for graph-based machine learning tasks. ## Requirements ```bash pip install -r requirements.txt ``` # Usage ```bash python main.py --epochs 200 --lr 0.01 --l2 5e-4 --dropout-p 0.5 --hidden-dim 16 --val-every 20 --include-bias ``` # Dataset The implementation includes support for the Cora dataset, a standard benchmark dataset for graph-based machine learning tasks. The Cora dataset consists of scientific publications, where nodes represent papers and edges represent citation relationships. Each paper is associated with a binary label indicating one of seven classes. The dataset is downloaded, preprocessed and ready to use. ## Model Architecture The GCN model architecture follows the details provided in the paper. It consists of multiple graph convolutional layers with ReLU activation, followed by a final softmax layer for classification. The implementation supports customizable hyperparameters such as the number of hidden units, the number of layers, and dropout rate. # Results The model achieves a classification accuracy of 82.5% on the test set of the Cora dataset after 200 epochs of training. This result is comparable to the performance reported in the original paper. However, the results can vary due to the randomness of the train/val/test split. References Thomas N. Kipf and Max Welling. "Semi-Supervised Classification with Graph Convolutional Networks." Link to the paper Original paper repository: [https://github.com/tkipf/gcn](https://github.com/tkipf/gcn) --- ### Overview (`imagenet/README.md`) # ImageNet training in PyTorch This implements training of popular model architectures, such as ResNet, AlexNet, and VGG on the ImageNet dataset. ## Requirements - Install PyTorch ([pytorch.org](http://pytorch.org)) - `pip install -r requirements.txt` - Download the ImageNet dataset from http://www.image-net.org/ - Then, move and extract the training and validation images to labeled subfolders, using [the following shell script](extract_ILSVRC.sh) ## Training To train a model, run `main.py` with the desired model architecture and the path to the ImageNet dataset: ```bash python main.py -a resnet18 [imagenet-folder with train and val folders] ``` The default learning rate schedule starts at 0.1 and decays by a factor of 10 every 30 epochs. This is appropriate for ResNet and models with batch normalization, but too high for AlexNet and VGG. Use 0.01 as the initial learning rate for AlexNet or VGG: ```bash python main.py -a alexnet --lr 0.01 [imagenet-folder with train and val folders] ``` ## Use Dummy Data ImageNet dataset is large and time-consuming to download. To get started quickly, run `main.py` using dummy data by "--dummy". It's also useful for training speed benchmark. Note that the loss or accuracy is useless in this case. ```bash python main.py -a resnet18 --dummy ``` ## Multi-processing Distributed Data Parallel Training If running on CUDA, you should always use the NCCL backend for multi-processing distributed training since it currently provides the best distributed training performance. For XPU multiprocessing is not supported as of PyTorch 2.6. ### Single node, multiple GPUs: ```bash python main.py -a resnet50 --dist-url 'tcp://127.0.0.1:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 1 --rank 0 [imagenet-folder with train and val folders] ``` ### Multiple nodes: Node 0: ```bash python main.py -a resnet50 --dist-url 'tcp://IP_OF_NODE0:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 2 --rank 0 [imagenet-folder with train and val folders] ``` Node 1: ```bash python main.py -a resnet50 --dist-url 'tcp://IP_OF_NODE0:FREEPORT' --dist-backend 'nccl' --multiprocessing-distributed --world-size 2 --rank 1 [imagenet-folder with train and val folders] ``` ## Usage ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ### Overview (`language_translation/README.md`) # Language Translation This example shows how one might use transformers for language translation. In particular, this implementation is loosely based on the [Attention is All You Need paper](https://arxiv.org/abs/1706.03762). ## Requirements We will need a tokenizer for our languages. Torchtext does include a tokenizer for English, but unfortunately, we will need more languages then that. We can get these tokenizers via ```spacy``` ```bash python3 -m spacy download python3 -m spacy download en python3 -m spacy download de ``` Spacy supports many languages. For a full accounting of supported languages, please look [here](https://spacy.io/usage/models). This example will default from German to English. Torchtext is also required: ```bash pip install torchtext ``` Just running these commands will get you started: ```bash pip install -r requirements.txt python3 -m spacy download ``` ## Usage This example contains a lot of flags that you can set to change the behavior / training of the module. You can see all of them by running: ```bash python3 main.py -h ``` But in general, all of the settings have "sensible" defaults; however, the default translation is to translate from German to English. To *train* the model, you only need to run the following command, but there is also an example for how to use any language you want: ```bash python3 main.py python3 main.py --src en --tgt fr # For english to french translation ``` For model inference, you can use this command: ```bash python3 main.py --inference --model_path ``` After some loading time, this will open an interactive interface where you can type in whatever sentence you are interested in translating. --- ### Overview (`mnist/README.md`) # Basic MNIST Example ```bash pip install -r requirements.txt python main.py # CUDA_VISIBLE_DEVICES=2 python main.py # to specify GPU id to ex. 2 ``` --- ### Overview (`mnist_forward_forward/README.md`) # Basic Forward-Forward Example This example implements the paper [The Forward-Forward Algorithm: Some Preliminary Investigations](https://arxiv.org/abs/2212.13345) by Geoffrey Hinton. the aim of this paper is to introduce a new learning procedure for neural networks. the forward and backward passes of backpropagation by two forward passes. ```bash pip install -r requirements.txt python main.py ``` The main.py script accepts the following arguments: ```bash optional arguments: -h, --help show this help message and exit --epochs EPOCHS number of epochs to train (default: 1000) --lr LR learning rate (default: 0.03) --no_accel disables accelerator --seed SEED random seed (default: 1) --save_model For saving the current Model --train_size TRAIN_SIZE size of training set --threshold THRESHOLD threshold for training --test_size TEST_SIZE size of test set --save-model For Saving the current Model --log-interval LOG_INTERVAL logging training status interval ``` --- ### Overview (`mnist_hogwild/README.md`) # MNIST Hogwild Example ```bash pip install -r requirements.txt python main.py ``` The main.py script accepts the following arguments: ```bash optional arguments: -h, --help show this help message and exit --batch-size input batch_size for training (default: 64) --test-batch-size input batch size for testing (default: 1000) --epochs EPOCHS number of epochs to train (default: 10) --lr LR learning rate (default: 0.01) --momentum SGD momentum (default: 0.5) --seed SEED random seed (default: 1) --save_model save the trained model to state_dict --log-interval how many batches to wait before logging training status (default: 10) --num-processes how many training processes to use (default: 2) --cuda enables CUDA training --mps enables macos GPU training --dry-run quickly check a single pass ``` --- ### Overview (`mnist_rnn/README.md`) # Example of MNIST using RNN ## Motivation Create pytorch example similar to Official Tensorflow Keras RNN example using MNIST [here](https://www.tensorflow.org/guide/keras/rnn) ```bash pip install -r requirements.txt python main.py # CUDA_VISIBLE_DEVICES=2 python main.py # to specify GPU id to ex. 2 ``` ```bash optional arguments: -h, --help show this help message and exit --batch_size input batch_size for training (default:64) --testing_batch_size input batch size for testing (default: 1000) --epochs EPOCHS number of epochs to train (default: 14) --lr LR learning rate (default: 0.1) --gamma learning rate step gamma (default: 0.7) --accel enables accelerator --seed SEED random seed (default: 1) --save_model For saving the current Model --log_interval how many batches to wait before logging training status --dry-run quickly check a single pass ``` --- ### Overview (`regression/README.md`) # Linear regression example Trains a single fully-connected layer to fit a 4th degree polynomial. --- ### Overview (`reinforcement_learning/README.md`) # Reinforcement learning training example ```bash pip install -r requirements.txt # For REINFORCE: python reinforce.py # For actor critic: python actor_critic.py ``` --- ### Overview (`siamese_network/README.md`) # Siamese Network Example Siamese network for image similarity estimation. The network is composed of two identical networks, one for each input. The output of each network is concatenated and passed to a linear layer. The output of the linear layer passed through a sigmoid function. [FaceNet](https://arxiv.org/pdf/1503.03832.pdf) is a variant of the Siamese network. This implementation varies from FaceNet as we use the `ResNet-18` model from [Deep Residual Learning for Image Recognition](https://arxiv.org/pdf/1512.03385.pdf) as our feature extractor. In addition, we aren't using `TripletLoss` as the MNIST dataset is simple, so `BCELoss` can do the trick. ### Usage Install the required dependencies: ```bash pip install -r requirements.txt ``` To run the example, execute: ```bahs python main.py # CUDA_VISIBLE_DEVICES=2 python main.py # to specify GPU id to ex. 2 ``` If a hardware accelerator device is detected, the example will execute on the accelerator; otherwise, it will run on the CPU. To force execution on the CPU, use `--no-accel` command line argument: ```bash python main.py --no-accel ``` Optionally, you can add the following arguments to customize your execution. ```bash --batch-size input batch size for training (default: 64) --test-batch-size input batch size for testing (default: 1000) --epochs number of epochs to train (default: 14) --lr learning rate (default: 1.0) --gamma learning rate step gamma (default: 0.7) --no-accel disables accelerator --dry-run quickly check a single pass --seed random seed (default: 1) --log-interval how many batches to wait before logging training status --save-model Saving the current Model ``` --- ### Overview (`super_resolution/README.md`) # Superresolution using an efficient sub-pixel convolutional neural network This example illustrates how to use the efficient sub-pixel convolution layer described in ["Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network" - Shi et al.](https://arxiv.org/abs/1609.05158) for increasing spatial resolution within your network for tasks such as superresolution. ``` usage: main.py [-h] --upscale_factor UPSCALE_FACTOR [--batchSize BATCHSIZE] [--testBatchSize TESTBATCHSIZE] [--nEpochs NEPOCHS] [--lr LR] [--accel] [--threads THREADS] [--seed SEED] PyTorch Super Res Example optional arguments: -h, --help show this help message and exit --upscale_factor super resolution upscale factor --batchSize training batch size --testBatchSize testing batch size --nEpochs number of epochs to train for --lr Learning Rate. Default=0.01 --accel use accelerator --threads number of threads for data loader to use Default=4 --seed random seed to use. Default=123 ``` This example trains a super-resolution network on the [BSD300 dataset](https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/), using crops from the 200 training images, and evaluating on crops of the 100 test images. A snapshot of the model after every epoch with filename `model_epoch_.pth`. ## Example Usage: ### Train ```bash python main.py --upscale_factor 3 --batchSize 4 --testBatchSize 100 --nEpochs 30 --lr 0.001 --accel ``` ### Super Resolve ```bash python super_resolve.py --input_image dataset/BSDS300/images/test/16077.jpg --model model_epoch_30.pth --output_filename out.png --accel ``` --- ### Overview (`time_sequence_prediction/README.md`) # Time Sequence Prediction This is a toy example for beginners to start with. It helps learn both PyTorch and time sequence prediction. Two LSTMCell units are used in this example to learn some sine wave signals starting at different phases. After learning the sine waves, the network tries to predict the signal values in the future. The results are shown in the picture below. ## Usage ``` python generate_sine_wave.py python train.py ``` ## Result The initial signal and the predicted results are shown in the image. We first give some initial signals (full line). The network will subsequently give some predicted results (dash line). It can be concluded that the network can generate new sine waves. --- ### Overview (`vae/README.md`) # Basic VAE Example This is an improved implementation of the paper [Auto-Encoding Variational Bayes](http://arxiv.org/abs/1312.6114) by Kingma and Welling. It uses ReLUs and the adam optimizer, instead of sigmoids and adagrad. These changes make the network converge much faster. ### Usage Install the required dependencies: ```bash pip install -r requirements.txt ``` To run the example, execute: ```bash python main.py ``` If a hardware accelerator device is detected, the example will execute on the accelerator; otherwise, it will run on the CPU. To force execution on the CPU, use `--no-accel` command line argument: ```bash python main.py --no-accel ``` The main.py script accepts the following optional arguments: ```bash --batch-size input batch size for training (default: 128) --epochs number of epochs to train (default: 10) --no-accel disables accelerator --seed random seed (default: 1) --log-interval how many batches to wait before logging training status ``` --- ### Overview (`word_language_model/README.md`) # Word-level Language Modeling using RNN and Transformer This example trains a multi-layer RNN (Elman, GRU, or LSTM) or Transformer on a language modeling task. By default, the training script uses the Wikitext-2 dataset, provided. The trained model can then be used by the generate script to generate new text. ```bash python main.py --accel --epochs 6 # Train a LSTM on Wikitext-2. python main.py --accel --epochs 6 --tied # Train a tied LSTM on Wikitext-2. python main.py --accel --tied # Train a tied LSTM on Wikitext-2for 40 epochs. python main.py --accel --epochs 6 --model Transformer --lr 5 # Train a Transformer model on Wikitext-2. python main.py --accel --epochs 6 --model Transformer --use-optimizer --lr 0.001 # Train a Transformer model with AdamW optimizer on Wikitext-2. python generate.py --accel # Generate samples from the default model checkpoint. ``` > [!NOTE] > Example supports running on acceleration devices (CUDA, MPS, XPU) The model uses the `nn.RNN` module (and its sister modules `nn.GRU` and `nn.LSTM`) or Transformer module (`nn.TransformerEncoder` and `nn.TransformerEncoderLayer`) which will automatically use the cuDNN backend if run on CUDA with cuDNN installed. During training, if a keyboard interrupt (Ctrl-C) is received, training is stopped and the current model is evaluated against the test dataset. The `main.py` script accepts the following arguments: ```bash optional arguments: -h, --help show this help message and exit --data DATA location of the data corpus --model MODEL type of network (RNN_TANH, RNN_RELU, LSTM, GRU, Transformer) --emsize EMSIZE size of word embeddings --nhid NHID number of hidden units per layer --nlayers NLAYERS number of layers --lr LR initial learning rate --clip CLIP gradient clipping --epochs EPOCHS upper epoch limit --batch_size N batch size --bptt BPTT sequence length --dropout DROPOUT dropout applied to layers (0 = no dropout) --tied tie the word embedding and softmax weights --seed SEED random seed --accel use accelerator --log-interval N report interval --save SAVE path to save the final model --onnx-export ONNX_EXPORT path to export the final model in onnx format --nhead NHEAD the number of heads in the encoder/decoder of the transformer model --dry-run verify the code and the model --use-optimizer specify whether to use an AdamW optimizer ``` With these arguments, a variety of models can be tested. As an example, the following arguments produce slower but better models: ```bash python main.py --accel --emsize 650 --nhid 650 --dropout 0.5 --epochs 40 python main.py --accel --emsize 650 --nhid 650 --dropout 0.5 --epochs 40 --tied python main.py --accel --emsize 1500 --nhid 1500 --dropout 0.65 --epochs 40 python main.py --accel --emsize 1500 --nhid 1500 --dropout 0.65 --epochs 40 --tied ``` --- ### Overview (`cpp/autograd/README.md`) # C++ autograd example `autograd.cpp` contains several examples of doing autograd in PyTorch C++ frontend. To build the code, run the following commands from your terminal: ```shell $ cd autograd $ mkdir build $ cd build $ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch .. $ make ``` where `/path/to/libtorch` should be the path to the unzipped _LibTorch_ distribution, which you can get from the [PyTorch homepage](https://pytorch.org/get-started/locally/). Execute the compiled binary to run: ```shell $ ./autograd ====== Running: "Basic autograd operations" ====== 1 1 1 1 [ CPUFloatType{2,2} ] 3 3 3 3 [ CPUFloatType{2,2} ] AddBackward1 27 27 27 27 [ CPUFloatType{2,2} ] MulBackward1 27 [ CPUFloatType{} ] MeanBackward0 false true SumBackward0 4.5000 4.5000 4.5000 4.5000 [ CPUFloatType{2,2} ] 813.6625 1015.0142 -664.8849 [ CPUFloatType{3} ] MulBackward1 204.8000 2048.0000 0.2048 [ CPUFloatType{3} ] true true false true false true ====== Running "Computing higher-order gradients in C++" ====== 0.0025 0.0946 0.1474 0.1387 0.0238 -0.0018 0.0259 0.0094 0.0513 -0.0549 -0.0604 0.0210 [ CPUFloatType{3,4} ] ====== Running "Using custom autograd function in C++" ====== -3.5513 3.7160 3.6477 -3.5513 3.7160 3.6477 [ CPUFloatType{2,3} ] 0.3095 1.4035 -0.0349 0.3095 1.4035 -0.0349 0.3095 1.4035 -0.0349 0.3095 1.4035 -0.0349 [ CPUFloatType{4,3} ] 5.5000 5.5000 [ CPUFloatType{2} ] ``` --- ### Overview (`cpp/custom-dataset/README.md`) # Custom Dataset Example with the PyTorch C++ Frontend This folder contains an example of loading a custom image dataset with OpenCV and training a model to label images, using the PyTorch C++ frontend. The dataset used here is [Caltech 101](https://data.caltech.edu/records/mzrjq-6wc02) dataset. The entire training code is contained in custom-data.cpp. You can find instructions on how to install OpenCV [here](../tools/InstallingOpenCV.md). To build the code, run the following commands from your terminal: ```shell $ cd custom-dataset $ mkdir build $ cd build $ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch .. $ make ``` where /path/to/libtorch should be the path to the unzipped LibTorch distribution, which you can get from the [PyTorch homepage](https://pytorch.org/get-started/locally/). If you see an error like `undefined reference to cv::imread(std::string const&, int)` when running the `make` command, you should build LibTorch from source using the instructions [here](https://github.com/pytorch/pytorch#from-source), and then set `CMAKE_PREFIX_PATH` to that PyTorch source directory. An alternative solution is to use `libtorch-cxx11-abi-shared-with-deps` instead of `libtorch-shared-with-deps` as the latter is not compatible with openCV (reported [here](https://discuss.pytorch.org/t/library-conflict-between-libtorch-and-opencv/64489)). The build directory should look like this: ``` . ├── custom-dataset ├── dataset │   ├── accordion │   │   ├── image_0001.jpg │   │   ├── ... │   ├── airplanes │   │   ├── ... │   ├── ... ├── info.txt └── Makefile └── ... ``` `info.txt` file gets copied from source directory during build. Execute the compiled binary to train the model: ```shell ./custom-dataset Running on: CUDA Train Epoch: 1 16/7281 Loss: 0.314655 Acc: 0 Train Epoch: 1 176/7281 Loss: 0.532111 Acc: 0.0681818 Train Epoch: 1 336/7281 Loss: 0.538482 Acc: 0.0714286 Train Epoch: 1 496/7281 Loss: 0.535302 Acc: 0.0705645 Train Epoch: 1 656/7281 Loss: 0.536113 Acc: 0.0716463 Train Epoch: 1 816/7281 Loss: 0.537626 Acc: 0.0784314 Train Epoch: 1 976/7281 Loss: 0.537055 Acc: 0.079918 ... ``` --- ### Overview (`cpp/dcgan/README.md`) # DCGAN Example with the PyTorch C++ Frontend This folder contains an example of training a DCGAN to generate MNIST digits with the PyTorch C++ frontend. The entire training code is contained in `dcgan.cpp`. You can find the commands to install argparse [here](https://github.com/pytorch/examples/blob/main/.github/workflows/main_cpp.yml#L34). To build the code, run the following commands from your terminal: ```shell $ cd dcgan $ mkdir build $ cd build $ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch .. $ make ``` where `/path/to/libtorch` should be the path to the unzipped _LibTorch_ distribution, which you can get from the [PyTorch homepage](https://pytorch.org/get-started/locally/). Execute the compiled binary to train the model: ```shell $ ./dcgan [ 1/30][200/938] D_loss: 0.4953 | G_loss: 4.0195 -> checkpoint 1 [ 1/30][400/938] D_loss: 0.3610 | G_loss: 4.8148 -> checkpoint 2 [ 1/30][600/938] D_loss: 0.4072 | G_loss: 4.36760 -> checkpoint 3 [ 1/30][800/938] D_loss: 0.4444 | G_loss: 4.0250 -> checkpoint 4 [ 2/30][200/938] D_loss: 0.3761 | G_loss: 3.8790 -> checkpoint 5 [ 2/30][400/938] D_loss: 0.3977 | G_loss: 3.3315 -> checkpoint 6 [ 2/30][600/938] D_loss: 0.3815 | G_loss: 3.5696 -> checkpoint 7 [ 2/30][800/938] D_loss: 0.4039 | G_loss: 3.2759 -> checkpoint 8 [ 3/30][200/938] D_loss: 0.4236 | G_loss: 4.5132 -> checkpoint 9 [ 3/30][400/938] D_loss: 0.3645 | G_loss: 3.9759 -> checkpoint 10 ... ``` We can also specify the `--epochs` to change the number of epochs to train as follows: ```shell $ ./dcgan --epochs 10 ``` Without specifying the `--epochs` flag, the default number of epochs to train is 30. The training script periodically generates image samples. Use the `display_samples.py` script situated in this folder to generate a plot image. For example: ```shell $ python display_samples.py -i dcgan-sample-10.pt Saved out.png ``` --- ### Overview (`cpp/distributed/README.md`) # Distributed Training on MNIST using PyTorch C++ Frontend (Libtorch) This folder contains an example of data-parallel training of a convolutional neural network on the MNIST dataset. For parallelization, Message Passing Interface (MPI) is used. The entire code is contained in dist-mnist.cpp You can find instructions on how to install MPI [here] (https://www.open-mpi.org/faq/?category=building). This code was tested on Open MPI but it should run on other MPI distributions as well such as MPICH, MVAPICH, etc. To build the code, run the following commands from the terminal: ```shell $ cd distributed $ mkdir build $ cd build $ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch .. $ make ``` where /path/to/libtorch should be the path to the unzipped LibTorch distribution. Note that the LibTorch from the [PyTorch homepage] ((https://pytorch.org/get-started/locally/) does not include MPI headers and cannot be used for this example. You have to compile LibTorch manually - a set of guidelines is provided [here] (https://gist.github.com/lasagnaphil/3e0099816837318e8e8bcab7edcfd5d9), however this may vary for different systems. To run the code, ```shell mpirun -np {NUM-PROCS} ./dist-mnist ``` --- ### Overview (`cpp/mnist/README.md`) # MNIST Example with the PyTorch C++ Frontend This folder contains an example of training a computer vision model to recognize digits in images from the MNIST dataset, using the PyTorch C++ frontend. The entire training code is contained in `mnist.cpp`. To build the code, run the following commands from your terminal: ```shell $ cd mnist $ mkdir build $ cd build $ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch .. $ make ``` where `/path/to/libtorch` should be the path to the unzipped _LibTorch_ distribution, which you can get from the [PyTorch homepage](https://pytorch.org/get-started/locally/). Execute the compiled binary to train the model: ```shell $ ./mnist Train Epoch: 1 [59584/60000] Loss: 0.4232 Test set: Average loss: 0.1989 | Accuracy: 0.940 Train Epoch: 2 [59584/60000] Loss: 0.1926 Test set: Average loss: 0.1338 | Accuracy: 0.959 Train Epoch: 3 [59584/60000] Loss: 0.1390 Test set: Average loss: 0.0997 | Accuracy: 0.969 Train Epoch: 4 [59584/60000] Loss: 0.1239 Test set: Average loss: 0.0875 | Accuracy: 0.972 ... ``` --- ### Overview (`cpp/regression/README.md`) # Linear regression example Trains a single fully-connected layer to fit a 4th degree polynomial. To build the code, run the following commands from your terminal: ```shell $ cd regression $ mkdir build $ cd build $ cmake -DCMAKE_PREFIX_PATH=/path/to/libtorch .. $ make ``` where `/path/to/libtorch` should be the path to the unzipped _LibTorch_ distribution, which you can get from the [PyTorch homepage](https://pytorch.org/get-started/locally/). Execute the compiled binary to run: ```shell $ ./regression Loss: 0.000301158 after 584 batches ==> Learned function: y = 11.6441 x^4 -3.10164 x^3 2.19786 x^2 -3.83606 x^1 + 4.37066 ==> Actual function: y = 11.669 x^4 -3.16023 x^3 2.19182 x^2 -3.81505 x^1 + 4.38219 ... ``` --- ### Overview (`cpp/transfer-learning/README.md`) # Transfer Learning on Dogs vs Cats Dataset using Libtorch and OpenCV Transfer Learning on Dogs vs Cats dataset using PyTorch C++ API. ## Usage For **training**: 1. Remove final layer of `ResNet18` pre-trained model and convert to `torch.jit` module: `python3 convert.py`. 2. Create build directory: `mkdir build && cd build` 3. `cmake -DCMAKE_PREFIX_PATH=/absolute/path/to/libtorch ..` 4. `make` 5. Run training code: `./example ` For **prediction**: 1. `cd build` 2. `./classify ` : `./classify ../resnet18_without_last_layer.pt model_linear.pt` Detailed blog on applying Transfer Learning using Libtorch: https://krshrimali.github.io/Applying-Transfer-Learning-Dogs-Cats/. --- ### Overview (`distributed/FSDP/README.md`) Note: FSDP1 is deprecated. Please follow [FSDP2 tutorial](https://docs.pytorch.org/tutorials/intermediate/FSDP_tutorial.html) and [code examples](https://github.com/pytorch/examples/tree/main/distributed/FSDP2). ## FSDP1 T5 To run the T5 example with FSDP1 for text summarization: ## Get the wikihow dataset ```bash sh download_dataset.sh ``` ## Install the requirements: ~~~ pip install -r requirements.txt ~~~ ## Ensure you are running a recent version of PyTorch: see https://pytorch.org to install at least 1.12 and ideally a current nightly build. Start the training with Torchrun (adjust nproc_per_node to your GPU count): ``` torchrun --nnodes 1 --nproc_per_node 4 T5_training.py ``` --- ### Overview (`distributed/FSDP2/README.md`) ## FSDP2 To run FSDP2 on transformer model: ``` cd distributed/FSDP2 pip install -r requirements.txt torchrun --nproc_per_node 2 example.py ``` * For 1st time, it creates a "checkpoints" folder and saves state dicts there * For 2nd time, it loads from previous checkpoints To enable explicit prefetching ``` torchrun --nproc_per_node 2 example.py --explicit-prefetch ``` To enable mixed precision ``` torchrun --nproc_per_node 2 example.py --mixed-precision ``` To showcase DCP API ``` torchrun --nproc_per_node 2 example.py --dcp-api ``` ## Ensure you are running a recent version of PyTorch: see https://pytorch.org/get-started/locally/ to install at least 2.5 and ideally a current nightly build. --- ### Overview (`distributed/ddp-tutorial-series/README.md`) # distributed-pytorch Code for the DDP tutorial series at https://pytorch.org/tutorials/beginner/ddp_series_intro.html Each code file extends upon the previous one. The series starts with a non-distributed script that runs on a single GPU and incrementally updates to end with multinode training on a Slurm cluster. ## Files * [single_gpu.py](single_gpu.py): Non-distributed training script * [multigpu.py](multigpu.py): DDP on a single node * [multigpu_torchrun.py](multigpu_torchrun.py): DDP on a single node using Torchrun * [multinode.py](multinode.py): DDP on multiple nodes using Torchrun (and optionally Slurm) * [slurm/setup_pcluster_slurm.md](slurm/setup_pcluster_slurm.md): instructions to set up an AWS cluster * [slurm/config.yaml.template](slurm/config.yaml.template): configuration to set up an AWS cluster * [slurm/sbatch_run.sh](slurm/sbatch_run.sh): slurm script to launch the training job --- ### Overview (`distributed/ddp/README.md`) # Distributed Data Parallel (DDP) Applications with PyTorch This guide demonstrates how to structure a distributed model training application for convenient multi-node launches using `torchrun`. --- ## Prerequisites You should be familiar with: - [PyTorch basics](https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html) - [Writing distributed applications](https://pytorch.org/tutorials/intermediate/dist_tuto.html) - [Distributed model training](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html) This tutorial uses the [`torch.nn.parallel.DistributedDataParallel`](https://docs.pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel) (DDP) class for data parallel training: multiple workers train the same global model on different data shards, compute local gradients, and synchronize them using AllReduce. In High-Performance Computing (HCP), this is called _Single Program Multiple Data_ (SPMD). --- ## Application Process Topologies A Distributed Data Parallel (DDP) application can be executed on multiple nodes where each node can consist of multiple GPU devices. Each node in turn can run multiple copies of the DDP application, each of which processes its models on multiple GPUs. Let: - _N_ = number of nodes - _G_ = number of GPUs per node - _W_ = **World Size** = total number of processes - _L_ = **Local World Size** = processes per node Each process has: - **Local rank**: in `[0, L-1]` - **Global rank**: in `[0, W-1]` **Example:** If you launch a DDP app on 2 nodes, each with 4 GPUs, and want each process to span 2 GPUs, the mapping is as follows: While there are quite a few ways to map processes to nodes, a good rule of thumb is to have one process span a single GPU. This enables the DDP application to have as many parallel reader streams as there are GPUs and in practice provides a good balance between I/O and computational costs. In the rest of this tutorial, we assume that the application follows this heuristic. # Preparing and launching a DDP application Independent of how a DDP application is launched, each process needs a mechanism to know its global and local ranks. Once this is known, all processes create a `ProcessGroup` that enables them to participate in collective communication operations such as AllReduce. A convenient way to start multiple DDP processes and initialize all values needed to create a `ProcessGroup` is to use the [`torchrun`](https://docs.pytorch.org/docs/stable/elastic/run.html) script provided with PyTorch. --- ## Sample Application This example is based on the ["Hello, World" DDP tutorial](https://pytorch.org/tutorials/intermediate/ddp_tutorial.html). The application calls the `spmd_main` entrypoint: ```python if __name__ == "__main__": spmd_main() ``` In `spmd_main`, the process group is initialized using the Accelerator API. The rest of the rendezvous information comes from environment variables set by `torchrun`: ```python def spmd_main(): # These are the parameters used to initialize the process group env_dict = { key: os.environ[key] for key in ("MASTER_ADDR", "MASTER_PORT", "RANK", "WORLD_SIZE") } rank = int(env_dict['RANK']) local_rank = int(env_dict['LOCAL_RANK']) local_world_size = int(env_dict['LOCAL_WORLD_SIZE']) print(f"[{os.getpid()}] Initializing process group with: {env_dict}") acc = torch.accelerator.current_accelerator() vendor_backend = torch.distributed.get_default_backend_for_device(acc) torch.accelerator.set_device_index(rank) torch.distributed.init_process_group(backend=vendor_backend) demo_basic(rank) # Tear down the process group torch.distributed.destroy_process_group() ``` **Key points:** - Each process reads its rank and world size from environment variables. - The process group is initialized for distributed communication. The training function, `demo_basic`, initializes the DDP model on the appropriate GPU: ```python def demo_basic(rank): print( f"[{os.getpid()}] rank = {torch.distributed.get_rank()}, " + f"world_size = {torch.distributed.get_world_size()}" ) model = ToyModel().to(rank) ddp_model = DDP(model, device_ids=[rank]) loss_fn = nn.MSELoss() optimizer = optim.SGD(ddp_model.parameters(), lr=0.001) optimizer.zero_grad() outputs = ddp_model(torch.randn(20, 10)) labels = torch.randn(20, 5).to(rank) loss_fn(outputs, labels).backward() optimizer.step() ``` --- ## Launching the Application ```sh torchrun --nnodes=1 --nproc_per_node=8 example.py ``` --- ## Example Output Expected output: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` # Conclusions As the author of a distributed data parallel application, your code needs to be aware of two types of resources: compute nodes and the GPUs within each node. The process of setting up bookkeeping to track how the set of GPUs is mapped to the processes of your application can be tedious and error-prone. We hope that by structuring your application as shown in this example and using `torchrun`, the mechanics of setting up distributed training can be significantly simplified. --- ### Overview (`distributed/minGPT-ddp/README.md`) # minGPT-DDP Code accompanying the tutorial at https://pytorch.org/tutorials/intermediate/ddp_series_minGPT.html for training a GPT-like model with Distributed Data Parallel (DDP) in PyTorch. Files marked with an asterisk (*) are adapted from the minGPT repo (https://github.com/karpathy/minGPT). - [trainer.py](mingpt/trainer.py) includes the Trainer class that runs the distributed training iterations on the model with the provided dataset. - [model.py *](mingpt/model.py) defines the model architecture. - [char_dataset.py *](mingpt/char_dataset.py) contains the `Dataset`class for a character-level dataset. - [gpt2_train_cfg.yaml](mingpt/gpt2_train_cfg.yaml) contains the configurations for data, model, optimizer and training run. - [main.py](mingpt/main.py) is the entry point to the trainig job. It sets up the DDP process group, reads all the configurations and runs the training job. - [slurm/](mingpt/slurm) contains files for setting up an AWS cluster and the slurm script to run multinode training. --- ### Overview (`distributed/tensor_parallelism/README.md`) # PyTorch native Tensor Parallel for distributed training This example demonstrates SPMD Megatron-LM style Tensor Parallel by using PyTorch native Tensor Parallel APIs, which include: 1. Simple module-level Tensor Parallelism on a dummy MLP model. 2. Simple module-level Tensor Parallelism with Sequence Parallel inputs/outputs on a dummy MLP model. 3. A E2E demo of Fully Sharded Data Parallel + Tensor Parallel (with Sequence Parallel) on a example Llama2 model. More details about the PyTorch native Tensor Parallel APIs, please see PyTorch docs: https://pytorch.org/docs/stable/distributed.tensor.parallel.html ## Installation ```bash pip install -r requirements.txt ``` ## Running Examples You can run the examples using `torchrun` to launch distributed training: ```bash # Simple Tensor Parallel example torchrun --nnodes=1 --nproc_per_node=4 tensor_parallel_example.py # Tensor Parallel with Sequence Parallel torchrun --nnodes=1 --nproc_per_node=4 sequence_parallel_example.py # FSDP + Tensor Parallel with Llama2 model torchrun --nnodes=1 --nproc_per_node=4 fsdp_tp_example.py ``` For more details, check the `run_examples.sh` script. --- ### Overview (`fx/native_interpreter/README.md`) # Converting PyTorch Code to a Native Runtime With FX and TorchScript Custom Classes In this example, we are going to build a pipeline that does the following things: 1. Converts (or “lowers”) code in a PyTorch module into another representation (we will define the representation within the example) 2. Registers an interpreter for that code representation that can be used in TorchScript or Python 3. Wrap the converted code into a format that can still be used in TorchScript compilation. We are going to build up a trivial interpreter for this example, but you can imagine extending the same process to work with more sophisticated backends, ones which may do code optimization or offloading to an accelerator. We will be using [TorchScript custom classes](https://pytorch.org/tutorials/advanced/torch_script_custom_classes.html) to expose this Interpreter to Python and TorchScript. You may want to review that tutorial and documentation before reading this example project. ### Defining the Interpreter We define the interpreter in `interpreter.cpp`. This interpreter is very limited: it only supports two element-wise operations (`add` and `mul`) and it only supports `Tensor` values. When this interpreter runs code, it iterates through the list of instructions and simply calls the appropriate PyTorch operator from C++. To build the interpreter into a shared-object file to be loaded in for use, use the following commands from this example’s root: ``` $ mkdir build $ cd build $ cmake -DCMAKE_PREFIX_PATH="$(python -c 'import torch.utils; print(torch.utils.cmake_prefix_path)')" .. $ make -j ``` After the build finishes, you should see `build/libinterpreter.so` (or with a different extension depending on your OS). We will use this dynamic library next when we load it up into a process to be used in execution. ### Defining the Transformation We define the code that transforms a `PyTorch` module to the format the interpreter understands in `use_interpreter.py`. Note that that file loads in the shared object we built in the previous step via a `torch.classes.load_library` call. `use_interpreter.py` contains driver code and the end that can be directly run to test the lowering transformation. ### Questions, Comments, Feedback Please direct questions and discussion to the [PyTorch forums](https://discuss.pytorch.org/). To report any issues with PyTorch (including FX and custom classes), please use the [issue tracker](https://github.com/pytorch/pytorch/issues). --- ### Overview (`legacy/snli/README.md`) # PyTorch-based NLI Training with SNLI ## 📝 Overview This repository contains Python scripts to train a Natural Language Inference (NLI) model, specifically the `SNLIClassifier`, using the Stanford Natural Language Inference (SNLI) corpus. The trained model predicts textual entailment, identifying if a statement is entailed, contradicted, or neither by another statement. ## ⚙️ Dependencies Install the necessary Python libraries with: ```bash pip install -r requirements.txt ``` The `requirements.txt` file includes: ``` torch torchtext spacy ``` ## 💻 Usage Start the training process with: ```bash python train.py --lower --word-vectors [PATH_TO_WORD_VECTORS] --vector-cache [PATH_TO_VECTOR_CACHE] --epochs [NUMBER_OF_EPOCHS] --batch-size [BATCH_SIZE] --save-path [PATH_TO_SAVE_MODEL] --gpu [GPU_NUMBER] ``` ## 🏋️‍♀️ Training The script trains the model on mini-batches of data across a specified number of epochs. It saves the best-performing model on the validation set as a `.pt` file in the specified directory. ## 📚 Scripts - `model.py`: Defines the `SNLIClassifier` model and auxiliary classes. - `util.py`: Contains utility functions for directory creation and command-line argument parsing. ## 📣 Note Ensure the `model.py` and `util.py` scripts are available in your working directory. --- ### Overview (`distributed/rpc/batch/README.md`) # Examples For Asynchronous RPC User Functions This folder contains two examples for [`@rpc.functions.async_execution`](https://pytorch.org/docs/master/rpc.html#torch.distributed.rpc.functions.async_execution): 1. Synchronized Batch Update Parameter Server: uses `@rpc.functions.async_execution` for parameter update and retrieving. This serves as a simple starter example for batch RPC. ``` pip install -r requirements.txt python parameter_server.py ``` 2. Multi-Observer with Batch-Processing Agent: uses `@rpc.functions.async_execution` to run multiple observed states through the policy to get actions. ``` pip install -r requirements.txt python reinforce.py ``` --- ### Overview (`distributed/rpc/ddp_rpc/README.md`) Distributed DataParallel + Distributed RPC Framework Example This example demonstrates how to combine Distributed DataParallel (DDP) with the Distributed RPC Framework. It requires two trainer nodes (each with a GPU), one master node, and one parameter server. The master node initializes an embedding table on the parameter server and orchestrates the training loop across the trainers. The model is composed of a dense component (`nn.Linear`), which is replicated on the trainers using DDP, and a sparse component (`nn.EmbeddingBag`), which resides on the parameter server. Each trainer performs embedding lookups on the parameter server via RPC, then processes the results through its local `nn.Linear` module. During the backward pass, DDP aggregates gradients for the dense part using allreduce, while the distributed backward pass updates the embedding table parameters on the parameter server. ``` pip install -r requirements.txt python main.py ``` --- ### Overview (`distributed/rpc/parameter_server/README.md`) ### RPC-based distributed training This is a basic example of RPC-based training that uses several trainers remotely train a model hosted on a server. To run the example locally, run the following command worker for the server and each worker you wish to spawn, in separate terminal windows: `python rpc_parameter_server.py --world_size=WORLD_SIZE --rank=RANK`. For example, for a master node with world size of 2, the command would be `python rpc_parameter_server.py --world_size=2 --rank=0`. The trainer can then be launched with the command `python rpc_parameter_server.py --world_size=2 --rank=1` in a separate window, and this will begin training with one server and a single trainer. Note that for demonstration purposes, this example supports only between 0-2 GPUs, although the pattern can be extended to make use of additional GPUs. To configure the number of GPUs, pass in `--num_gpus=N` to your training command. You can pass in the command line arguments `--master_addr=
` and `master_port=PORT` to indicate the address:port that the master worker is listening on. All workers will contact the master for rendezvous during worker discovery. By default, `master_addr` will be `localhost` and `master_port` will be 29500. --- ### Overview (`distributed/rpc/pipeline/README.md`) Distributed Pipeline Parallel Example This example shows how to distribute a ResNet50 model on two RPC workers and then implement distributed pipeline parallelism using RPC. With pipeline parallelism, every input batch is divided into micro-batches and thse micro-batches are feed into the model in a pipelined fashion to increase the amortized device utilization. Note that this example only parallelizes the forward pass which can be viewed as the distributed counterpart of the [single machine pipeline parallel](https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html#speed-up-by-pipelining-inputs) example. ``` pip install -r requirements.txt python main.py ``` --- ### Overview (`distributed/rpc/rl/README.md`) Distributed Multi-Observer Single-Agent Reinforcement Learning Example This example demonstrates `torch.distributed.rpc` API using an CartPole reinforcement learning example. Please note that the goal is to present the RPC API instead of building the best CartPole solver. ``` pip install -r requirements.txt python main.py ``` --- ### Overview (`distributed/rpc/rnn/README.md`) Distributed RNN Model Parallel Example This example shows how to build an RNN model using RPC where different components of the RNN model can be placed on different workers. ``` pip install -r requirements.txt python main.py ``` --- ### Requirements (`docs/requirements.txt`) sphinx==5.3.0 #Pinned versions: 5.3.0 -e git+https://github.com/pytorch/pytorch_sphinx_theme.git#egg=pytorch_sphinx_theme sphinx-panels --- ### Source Index (`docs/source/index.rst`) PyTorch Examples ================ This pages lists various PyTorch examples that you can use to learn and experiment with PyTorch. .. panels:: Image Classification Using ConvNets ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This example demonstrates how to run image classification with `Convolutional Neural Networks ConvNets `__ on the `MNIST `__ database. `GO TO EXAMPLE `__ :opticon:`link-external` --- Measuring Similarity using Siamese Network ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This example demonstrates how to measure similarity between two images using `Siamese network `__ on the `MNIST `__ database. `GO TO EXAMPLE `__ :opticon:`link-external` --- Word-level Language Modeling using RNN and Transformer ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This example demonstrates how to train a multi-layer `recurrent neural network (RNN) `__, such as Elman, GRU, or LSTM, or Transformer on a language modeling task by using the Wikitext-2 dataset. `GO TO EXAMPLE `__ :opticon:`link-external` --- Training ImageNet Classifiers ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This example demonstrates how you can train some of the most popular model architectures, including `ResNet `__, `AlexNet `__, and `VGG `__ on the `ImageNet `__ dataset. `GO TO EXAMPLE `__ :opticon:`link-external` --- Generative Adversarial Networks (DCGAN) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This example implements the `Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks `__ paper. `GO TO EXAMPLE `__ :opticon:`link-external` --- Variational Auto-Encoders ^^^^^^^^^^^^^^^^^^^^^^^^^ This example implements the `Auto-Encoding Variational Bayes `__ paper with `ReLUs `__ and the Adam optimizer. `GO TO EXAMPLE `__ :opticon:`link-external` --- Super-resolution Using an Efficient Sub-Pixel CNN ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This example demonstrates how to use the sub-pixel convolution layer described in `Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network `__ paper. This example trains a super-resolution network on the `BSD300 dataset `__. `GO TO EXAMPLE `__ :opticon:`link-external` --- HOGWILD! Training of Shared ConvNets ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `HOGWILD! `__ is a scheme that allows Stochastic Gradient Descent (SGD) parallelization without memory locking. This example demonstrates how to perform HOGWILD! training of shared ConvNets on MNIST. `GO TO EXAMPLE `__ :opticon:`link-external` --- Training a CartPole to balance with actor-critic ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This reinforcement learning tutorial demonstrates how to train a CartPole to balance in the `Gymnasium `__ toolkit by using the `Actor-Critic `__ method. `GO TO EXAMPLE `__ :opticon:`link-external` --- Time Sequence Prediction ^^^^^^^^^^^^^^^^^^^^^^^^ This beginner example demonstrates how to use LSTMCell to learn sine wave signals to predict the signal values in the future. `GO TO EXAMPLE `__ :opticon:`link-external` --- Implement the Neural Style Transfer algorithm on images ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This tutorial demonstrates how you can use PyTorch's implementation of the `Neural Style Transfer (NST) `__ algorithm on images. `GO TO EXAMPLE `__ :opticon:`link-external` --- PyTorch Module Transformations using fx ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This set of examples demonstrates the torch.fx toolkit. For more information about `torch.fx`, see `torch.fx Overview `__. `GO TO EXAMPLE `__ :opticon:`link-external` --- Distributed PyTorch ^^^^^^^^^^^^^^^^^^^ This set of examples demonstrates `Distributed Data Parallel (DDP) `__ and `Distributed RPC framework `__. Includes the code used in the `DDP tutorial series `__. `GO TO EXAMPLES `__ :opticon:`link-external` --- C++ Frontend ^^^^^^^^^^^^ The PyTorch C++ frontend is a C++14 library for CPU and GPU tensor computation. This set of examples includes a linear regression, autograd, image recognition (MNIST), and other useful examples using PyTorch C++ frontend. `GO TO EXAMPLES `__ :opticon:`link-external` --- Image Classification Using Forward-Forward Algorithm ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This example implements the paper `The Forward-Forward Algorithm: Some Preliminary Investigations `__ by Geoffrey Hinton. on the `MNIST `__ database. It is an introductory example to the Forward-Forward algorithm. `GO TO EXAMPLE `__ :opticon:`link-external` --- Graph Convolutional Network ^^^^^^^^^^^^^^^^^^^^^^^^^^^ This example implements the `Semi-Supervised Classification with Graph Convolutional Networks `__ paper on the CORA database. `GO TO EXAMPLE `__ :opticon:`link-external` --- --- METRICS --- - Files Extracted: 42 - Estimated Token Budget: ~17355 tokens - Canonical Reference: https://codewiki.google/github.com/pytorch/examples