{"owner":"pytorch","repo":"vision","hasSkills":true,"totalSkillsCount":7,"totalTokensCount":13196,"categories":["anthropic-skill"],"hasMcp":false,"mcpConfig":null,"found":["references/classification/README.md","references/optical_flow/README.md","references/segmentation/README.md","references/similarity/README.md","references/video_classification/README.md","references/detection/README.md","references/depth/stereo/README.md"],"skills":{"references/classification/README.md":"# Image classification reference training scripts\n\nThis folder contains reference training scripts for image classification.\nThey serve as a log of how to train specific models, as provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nExcept otherwise noted, all models have been trained on 8x V100 GPUs with \nthe following parameters:\n\n| Parameter                | value  |\n| ------------------------ | ------ |\n| `--batch_size`           | `32`   |\n| `--epochs`               | `90`   |\n| `--lr`                   | `0.1`  |\n| `--momentum`             | `0.9`  |\n| `--wd`, `--weight-decay` | `1e-4` |\n| `--lr-step-size`         | `30`   |\n| `--lr-gamma`             | `0.1`  |\n\n### AlexNet and VGG\n\nSince `AlexNet` and the original `VGG` architectures do not include batch \nnormalization, the default initial learning rate `--lr 0.1` is too high.\n\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model $MODEL --lr 1e-2\n```\n\nHere `$MODEL` is one of `alexnet`, `vgg11`, `vgg13`, `vgg16` or `vgg19`. Note\nthat `vgg11_bn`, `vgg13_bn`, `vgg16_bn`, and `vgg19_bn` include batch\nnormalization and thus are trained with the default parameters.\n\n### GoogLeNet\n\nThe weights of the GoogLeNet model are ported from the original paper rather than trained from scratch.\n\n### Inception V3\n\nThe weights of the Inception V3 model are ported from the original paper rather than trained from scratch.\n\nSince it expects tensors with a size of N x 3 x 299 x 299, to validate the model use the following command:\n\n```\ntorchrun --nproc_per_node=8 train.py --model inception_v3\\\n      --test-only --weights Inception_V3_Weights.IMAGENET1K_V1\n```\n\n### ResNet\n```\ntorchrun --nproc_per_node=8 train.py --model $MODEL\n```\n\nHere `$MODEL` is one of `resnet18`, `resnet34`, `resnet50`, `resnet101` or `resnet152`.\n\n### ResNext\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model $MODEL --epochs 100\n```\n\nHere `$MODEL` is one of `resnext50_32x4d` or `resnext101_32x8d`.\nNote that the above command corresponds to a single node with 8 GPUs. If you use\na different number of GPUs and/or a different batch size, then the learning rate\nshould be scaled accordingly. For example, the pretrained model provided by\n`torchvision` was trained on 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nwith `--batch_size 16` and `--lr 0.4`, instead of the current defaults\nwhich are respectively batch_size=32 and lr=0.1\n\n### MobileNetV2\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model mobilenet_v2 --epochs 300 --lr 0.045 --wd 0.00004\\\n     --lr-step-size 1 --lr-gamma 0.98\n```\n\n\n### MobileNetV3 Large & Small\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 600 --opt rmsprop --batch-size 128 --lr 0.064\\ \n     --wd 0.00001 --lr-step-size 2 --lr-gamma 0.973 --auto-augment imagenet --random-erase 0.2\n```\n\nHere `$MODEL` is one of `mobilenet_v3_large` or `mobilenet_v3_small`.\n\nThen we averaged the parameters of the last 3 checkpoints that improved the Acc@1. See [#3182](https://github.com/pytorch/vision/pull/3182) \nand [#3354](https://github.com/pytorch/vision/pull/3354) for details.\n\n\n### EfficientNet-V1\n\nThe weights of the B0-B4 variants are ported from Ross Wightman's [timm repo](https://github.com/rwightman/pytorch-image-models/blob/01cb46a9a50e3ba4be167965b5764e9702f09b30/timm/models/efficientnet.py#L95-L108).\n\nThe weights of the B5-B7 variants are ported from Luke Melas' [EfficientNet-PyTorch repo](https://github.com/lukemelas/EfficientNet-PyTorch/blob/1039e009545d9329ea026c9f7541341439712b96/efficientnet_pytorch/utils.py#L562-L564).\n\nAll models were trained using Bicubic interpolation and each have custom crop and resize sizes. To validate the models use the following commands:\n```\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b0 --test-only --weights EfficientNet_B0_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b1 --test-only --weights EfficientNet_B1_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b2 --test-only --weights EfficientNet_B2_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b3 --test-only --weights EfficientNet_B3_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b4 --test-only --weights EfficientNet_B4_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b5 --test-only --weights EfficientNet_B5_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b6 --test-only --weights EfficientNet_B6_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b7 --test-only --weights EfficientNet_B7_Weights.IMAGENET1K_V1\n```\n\n\n### EfficientNet-V2\n```\ntorchrun --nproc_per_node=8 train.py \\\n--model $MODEL --batch-size 128 --lr 0.5 --lr-scheduler cosineannealinglr \\\n--lr-warmup-epochs 5 --lr-warmup-method linear --auto-augment ta_wide --epochs 600 --random-erase 0.1 \\\n--label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --weight-decay 0.00002 --norm-weight-decay 0.0 \\\n--train-crop-size $TRAIN_SIZE --model-ema --val-crop-size $EVAL_SIZE --val-resize-size $EVAL_SIZE \\\n--ra-sampler --ra-reps 4\n```\nHere `$MODEL` is one of `efficientnet_v2_s` and `efficientnet_v2_m`. \nNote that the Small variant had a `$TRAIN_SIZE` of `300` and a `$EVAL_SIZE` of `384`, while the Medium `384` and `480` respectively.\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 4 nodes, each with 8 GPUs (for a total of 32 GPUs),\nand `--batch_size 32`.\n\nThe weights of the Large variant are ported from the original paper rather than trained from scratch. See the `EfficientNet_V2_L_Weights` entry for their exact preprocessing transforms.\n\n\n### RegNet\n\n#### Small models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 128 --wd 0.00005 --lr=0.8\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_400mf`, `regnet_x_800mf`, `regnet_x_1_6gf`, `regnet_y_400mf`, `regnet_y_800mf` and `regnet_y_1_6gf`. Please note we used learning rate 0.4 for `regent_y_400mf` to get the same Acc@1 as [the paper](https://arxiv.org/abs/2003.13678).\n\n#### Medium models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 64 --wd 0.00005 --lr=0.4\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_3_2gf`, `regnet_x_8gf`, `regnet_x_16gf`, `regnet_y_3_2gf` and `regnet_y_8gf`.\n\n#### Large models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 32 --wd 0.00005 --lr=0.2\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_32gf`, `regnet_y_16gf` and `regnet_y_32gf`.\n\n### Vision Transformer\n\n#### vit_b_16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_b_16 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nand `--batch_size 64`.\n\n#### vit_b_32\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_b_32 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment imagenet\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 256`.\n\n#### vit_l_16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_l_16 --epochs 600 --batch-size 128 --lr 0.5 --lr-scheduler cosineannealinglr\\\n    --lr-warmup-method linear --lr-warmup-epochs 5 --label-smoothing 0.1 --mixup-alpha 0.2\\\n    --auto-augment ta_wide --random-erase 0.1 --weight-decay 0.00002 --norm-weight-decay 0.0\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema --val-resize-size 232\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 64`.\n\n#### vit_l_32\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_l_32 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nand `--batch_size 64`.\n\n\n### ConvNeXt\n```\ntorchrun --nproc_per_node=8 train.py\\ \n--model $MODEL --batch-size 128 --opt adamw --lr 1e-3 --lr-scheduler cosineannealinglr \\ \n--lr-warmup-epochs 5 --lr-warmup-method linear --auto-augment ta_wide --epochs 600 --random-erase 0.1 \\ \n--label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --weight-decay 0.05 --norm-weight-decay 0.0 \\\n--train-crop-size 176 --model-ema --val-resize-size 232 --ra-sampler --ra-reps 4\n```\nHere `$MODEL` is one of `convnext_tiny`, `convnext_small`, `convnext_base` and `convnext_large`. Note that each variant had its `--val-resize-size` optimized in a post-training step, see their `Weights` entry for their exact value.\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 64`.\n\n\n### SwinTransformer\n```\ntorchrun --nproc_per_node=8 train.py\\ \n--model $MODEL --epochs 300 --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler --ra-reps 4  --val-resize-size 224\n```\nHere `$MODEL` is one of `swin_t`, `swin_s` or `swin_b`.\nNote that `--val-resize-size` was optimized in a post-training step, see their `Weights` entry for the exact value.\n\n\n\n\n### SwinTransformer V2\n```\ntorchrun --nproc_per_node=8 train.py\\\n--model $MODEL --epochs 300 --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler --ra-reps 4  --val-resize-size 256 --val-crop-size 256 --train-crop-size 256 \n```\nHere `$MODEL` is one of `swin_v2_t`, `swin_v2_s` or `swin_v2_b`.\nNote that `--val-resize-size` was optimized in a post-training step, see their `Weights` entry for the exact value.\n\n\n### MaxViT\n```\ntorchrun --nproc_per_node=8 --n_nodes=4 train.py\\\n--model $MODEL --epochs 400 --batch-size 128 --opt adamw --lr 3e-3 --weight-decay 0.05 --lr-scheduler cosineannealinglr --lr-min 1e-5 --lr-warmup-method linear  --lr-warmup-epochs 32  --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 1.0 --interpolation bicubic --auto-augment ta_wide --policy-magnitude 15 --model-ema --val-resize-size 224\\\n--val-crop-size 224 --train-crop-size 224 --amp  --model-ema-steps 32 --transformer-embedding-decay 0 --sync-bn\n```\nHere `$MODEL` is `maxvit_t`.\nNote that `--val-resize-size` was not optimized in a post-training step.\n\n\n### ShuffleNet V2\n```\ntorchrun --nproc_per_node=8 train.py \\\n--batch-size=128 \\\n--lr=0.5 --lr-scheduler=cosineannealinglr --lr-warmup-epochs=5 --lr-warmup-method=linear \\\n--auto-augment=ta_wide --epochs=600 --random-erase=0.1 --weight-decay=0.00002 \\\n--norm-weight-decay=0.0 --label-smoothing=0.1 --mixup-alpha=0.2 --cutmix-alpha=1.0 \\\n--train-crop-size=176 --model-ema --val-resize-size=232 --ra-sampler --ra-reps=4\n```\nHere `$MODEL` is either `shufflenet_v2_x1_5` or `shufflenet_v2_x2_0`.\n\nThe models `shufflenet_v2_x0_5` and `shufflenet_v2_x1_0` were contributed by the community. See [PR-849](https://github.com/pytorch/vision/pull/849#issuecomment-483391686) for details.\n\n\n## Mixed precision training\nAutomatic Mixed Precision (AMP) training on GPU for Pytorch can be enabled with the [torch.cuda.amp](https://pytorch.org/docs/stable/amp.html?highlight=amp#module-torch.cuda.amp).\n\nMixed precision training makes use of both FP32 and FP16 precisions where appropriate. FP16 operations can leverage the Tensor cores on NVIDIA GPUs (Volta, Turing or newer architectures) for improved throughput, generally without loss in model accuracy. Mixed precision training also often allows larger batch sizes. GPU automatic mixed precision training for Pytorch Vision can be enabled via the flag value `--amp=True`.\n\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model resnext50_32x4d --epochs 100 --amp\n```\n\n## Quantized\n\n### Post training quantized models\n\nFor all post training quantized models, the settings are:\n\n1. num_calibration_batches: 32\n2. num_workers: 16\n3. batch_size: 32\n4. eval_batch_size: 128\n5. qbackend: 'fbgemm'\n\n```\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' --model='$MODEL'\n```\nHere `$MODEL` is one of `googlenet`, `inception_v3`, `resnet18`, `resnet50`, `resnext101_32x8d`, `shufflenet_v2_x0_5` and `shufflenet_v2_x1_0`.\n\n### Quantized ShuffleNet V2\n\nHere are commands that we use to quantize the `shufflenet_v2_x1_5` and `shufflenet_v2_x2_0` models.\n```\n# For shufflenet_v2_x1_5\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' \\\n    --model=shufflenet_v2_x1_5 --weights=\"ShuffleNet_V2_X1_5_Weights.IMAGENET1K_V1\" \\\n    --train-crop-size 176 --val-resize-size 232 --data-path /datasets01_ontap/imagenet_full_size/061417/\n\n# For shufflenet_v2_x2_0\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' \\\n    --model=shufflenet_v2_x2_0 --weights=\"ShuffleNet_V2_X2_0_Weights.IMAGENET1K_V1\" \\\n    --train-crop-size 176 --val-resize-size 232 --data-path /datasets01_ontap/imagenet_full_size/061417/\n```\n\n### QAT MobileNetV2\n\nFor Mobilenet-v2, the model was trained with quantization aware training, the settings used are:\n1. num_workers: 16\n2. batch_size: 32\n3. eval_batch_size: 128\n4. qbackend: 'qnnpack'\n5. learning-rate: 0.0001\n6. num_epochs: 90\n7. num_observer_update_epochs:4\n8. num_batch_norm_update_epochs:3\n9. momentum: 0.9\n10. lr_step_size:30\n11. lr_gamma: 0.1\n12. weight-decay: 0.0001\n\n```\ntorchrun --nproc_per_node=8 train_quantization.py --model='mobilenet_v2'\n```\n\nTraining converges at about 10 epochs.\n\n### QAT MobileNetV3\n\nFor Mobilenet-v3 Large, the model was trained with quantization aware training, the settings used are:\n1. num_workers: 16\n2. batch_size: 32\n3. eval_batch_size: 128\n4. qbackend: 'qnnpack'\n5. learning-rate: 0.001\n6. num_epochs: 90\n7. num_observer_update_epochs:4\n8. num_batch_norm_update_epochs:3\n9. momentum: 0.9\n10. lr_step_size:30\n11. lr_gamma: 0.1\n12. weight-decay: 0.00001\n\n```\ntorchrun --nproc_per_node=8 train_quantization.py --model='mobilenet_v3_large' \\\n    --wd 0.00001 --lr 0.001\n```\n\nFor post training quant, device is set to CPU. For training, the device is set to CUDA.\n\n### Command to evaluate quantized models using the pre-trained weights:\n\n```\npython train_quantization.py --device='cpu' --test-only --qbackend='<qbackend>' --model='<model_name>'\n```\n\nFor inception_v3 you need to pass the following extra parameters:\n```\n--val-resize-size 342 --val-crop-size 299 --train-crop-size 299\n```\n","references/optical_flow/README.md":"# Optical flow reference training scripts\n\nThis folder contains reference training scripts for optical flow.\nThey serve as a log of how to train specific models, so as to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\n\n### RAFT Large\n\nThe RAFT large model was trained on Flying Chairs and then on Flying Things.\nBoth used 8 A100 GPUs and a batch size of 2 (so effective batch size is 16). The\nrest of the hyper-parameters are exactly the same as the original RAFT training\nrecipe from https://github.com/princeton-vl/RAFT. The original recipe trains for\n100000 updates (or steps) on each dataset - this corresponds to about 72 and 20\nepochs on Chairs and Things respectively:\n\n```\nnum_epochs = ceil(num_steps / number_of_steps_per_epoch)\n           = ceil(num_steps / (num_samples / effective_batch_size))\n```\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_chairs \\\n    --model raft_large \\\n    --train-dataset chairs \\\n    --batch-size 2 \\\n    --lr 0.0004 \\\n    --weight-decay 0.0001 \\\n    --epochs 72 \\\n    --output-dir $chairs_dir\n```\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_things \\\n    --model raft_large \\\n    --train-dataset things \\\n    --batch-size 2 \\\n    --lr 0.000125 \\\n    --weight-decay 0.0001 \\\n    --epochs 20 \\\n    --freeze-batch-norm \\\n    --output-dir $things_dir\\\n    --resume $chairs_dir/$name_chairs.pth\n```\n\n\n### Evaluation\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 train.py --val-dataset sintel --batch-size 1 --dataset-root $dataset_root --model raft_large --weights Raft_Large_Weights.C_T_SKHT_V2\n```\n\nThis should give an epe of about 1.3822 on the clean pass and 2.7161 on the\nfinal pass of Sintel-train. Results may vary slightly depending on the batch\nsize and the number of GPUs. For the most accurate results use 1 GPU and\n`--batch-size 1`:\n\n```\nSintel val clean epe: 1.3822\t1px: 0.9028\t3px: 0.9573\t5px: 0.9697\tper_image_epe: 1.3822\tf1: 4.0248\nSintel val final epe: 2.7161\t1px: 0.8528\t3px: 0.9204\t5px: 0.9392\tper_image_epe: 2.7161\tf1: 7.5964\n```\n\nYou can also evaluate on Kitti train:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 train.py --val-dataset kitti --batch-size 1 --dataset-root $dataset_root --model raft_large --weights Raft_Large_Weights.C_T_SKHT_V2\nKitti val epe: 4.7968\t1px: 0.6388\t3px: 0.8197\t5px: 0.8661\tper_image_epe: 4.5118\tf1: 16.0679\n```\n","references/segmentation/README.md":"# Semantic segmentation reference training scripts\n\nThis folder contains reference training scripts for semantic segmentation.\nThey serve as a log of how to train specific models and provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nAll models have been trained on 8x V100 GPUs.\n\nYou must modify the following flags:\n\n`--data-path=/path/to/dataset`\n\n`--nproc_per_node=<number_of_gpus_available>`\n\n## fcn_resnet50\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model fcn_resnet50 --aux-loss --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n## fcn_resnet101\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model fcn_resnet101 --aux-loss --weights-backbone ResNet101_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_resnet50\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model deeplabv3_resnet50 --aux-loss --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_resnet101\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model deeplabv3_resnet101 --aux-loss --weights-backbone ResNet101_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_mobilenet_v3_large\n```\ntorchrun --nproc_per_node=8 train.py --dataset coco -b 4 --model deeplabv3_mobilenet_v3_large --aux-loss --wd 0.000001 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n## lraspp_mobilenet_v3_large\n```\ntorchrun --nproc_per_node=8 train.py --dataset coco -b 4 --model lraspp_mobilenet_v3_large --wd 0.000001 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n","references/similarity/README.md":"# Similarity Learning Using Triplet Loss #\n\nIn this reference, we use triplet loss to learn embeddings which can be used to differentiate images. This learning technique was popularized by [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/abs/1503.03832) and has been quite effective in learning embeddings to differentiate between faces.\n\nThis reference can be directly applied to the following use cases:\n\n* You have an unknown number of classes and would like to train a model to learn how to differentiate between them.\n* You want to train a model to learn a distance-based metric between samples. For example, learning a distance-based similarity measure between faces.\n\n### Training ###\nBy default, the training script trains ResNet50 on the FashionMNIST Dataset to learn image embeddings which can be used to differentiate between images by measuring the euclidean distance between embeddings. This can be changed as per your requirements.\n\nImage embeddings of the same class should be 'close' to each other, while image embeddings between different classes should be 'far' away.\n\nTo run the training script:\n\n```bash\npython train.py -h    # Lists all optional arguments\npython train.py \t\t\t# Runs training script with default args\n```\n\nRunning the training script as is should yield 97% accuracy on the FMNIST test set within 10 epochs.\n\n### Loss ###\n`TripletMarginLoss` is a loss function which takes in a triplet of samples. A valid triplet has an:\n\n1. Anchor: a sample from the dataset\n2. Positive: another sample with the same label/group as the anchor (Generally, positive != anchor)\n3. Negative: a sample with a different label/group from the anchor\n\n`TripletMarginLoss` (refer to `loss.py`) does the following:\n\n```python\nloss = max(dist(anchor, positive) - dist(anchor, negative) + margin, 0)\n```\nWhere `dist` is a distance function. Minimizing this function effectively leads to minimizing `dist(anchor, positive)` and maximizing `dist(anchor, negative)`.\n\nThe FaceNet paper describe this loss in more detail.\n\n### Sampler ###\n\nIn order to generate valid triplets from a batch of samples, we need to make sure that each batch has multiple samples with the same label. We do this using `PKSampler` (refer to `sampler.py`), which ensures that each batch of size `p * k` will have samples from exactly `p` classes and `k` samples per class.\n\n### Triplet Mining ###\n\n`TripletMarginLoss` currently supports the following mining techniques:\n\n* `batch_all`: Generates all possible triplets from a batch and excludes the triplets which are 'easy' (which have `loss = 0`) before passing it through the loss function.\n* `batch_hard`: For every anchor, `batch_hard` creates a triplet with the 'hardest' positive (farthest positive) and negative (closest negative).\n\nThese mining strategies usually speed up training.\n\nThis [webpage](https://omoindrot.github.io/triplet-loss) describes the sampling and mining strategies in more detail. \n","references/video_classification/README.md":"# Video Classification\n\nWe present a simple training script that can be used for replicating the result of [resenet-based video models](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf). All models are trained on [Kinetics400 dataset](https://deepmind.com/research/open-source/kinetics), a benchmark dataset for human-action recognition. The accuracy is reported on the traditional validation split.\n\n## Data preparation\n\nIf you already have downloaded [Kinetics400 dataset](https://deepmind.com/research/open-source/kinetics), \nplease proceed directly to the next section.\n\nTo download videos, one can use https://github.com/Showmax/kinetics-downloader. Please note that the dataset can take up upwards of 400GB, depending on the quality setting during download.\n\n## Training\n\nWe assume the training and validation AVI videos are stored at `/data/kinectics400/train` and \n`/data/kinectics400/val`. For training we suggest starting with the hyperparameters reported in the [paper](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf), in order to match the performance of said models. Clip sampling strategy is a particularly important parameter during training, and we suggest using random temporal jittering during training - in other words sampling multiple training clips from each video with random start times during at every epoch. This functionality is built into our training script, and optimal hyperparameters are set by default.  \n\n### Multiple GPUs\n\nRun the training on a single node with 8 GPUs:\n```bash\ntorchrun --nproc_per_node=8 train.py --data-path=/data/kinectics400 --kinetics-version=\"400\" --lr 0.08 --cache-dataset --sync-bn --amp \n```\n\n**Note:** all our models were trained on 8 nodes with 8 V100 GPUs each for a total of 64 GPUs. Expected training time for 64 GPUs is 24 hours, depending on the storage solution.\n**Note 2:** hyperparameters for exact replication of our training can be found on the section below. Some hyperparameters such as learning rate must be scaled linearly in proportion to the number of GPUs. The default values assume 64 GPUs.\n\n### Single GPU \n\n**Note:** training on a single gpu can be extremely slow. \n\n\n```bash\npython train.py --data-path=/data/kinectics400 --kinetics-version=\"400\" --batch-size=8 --cache-dataset\n```\n\n\n### Additional Kinetics versions\n\nSince the original release, additional versions of Kinetics dataset became available (Kinetics 600).\nOur training scripts support these versions of dataset as well by setting the `--kinetics-version` parameter to `\"600\"`.\n\n**Note:** training on Kinetics 600 requires a different set of hyperparameters for optimal performance. We do not provide Kinetics 600 pretrained models.\n\n\n## Video classification models\n\nStarting with version `0.4.0` we have introduced support for basic video tasks and video classification modelling.\nFor more information about the available models check [here](https://pytorch.org/docs/stable/torchvision/models.html#video-classification). \n\n### Video ResNet models\n\nSee reference training script [here](https://github.com/pytorch/vision/blob/main/references/video_classification/train.py):\n\n- input space: RGB\n- resize size: [128, 171]\n- crop size: [112, 112]\n- mean: [0.43216, 0.394666, 0.37645]\n- std: [0.22803, 0.22145, 0.216989]\n- number of classes: 400\n\nInput data augmentations at training time (with optional parameters):\n\n1. ConvertImageDtype\n2. Resize (resize size value above)\n3. Random horizontal flip (0.5)\n4. Normalization (mean, std, see values above)\n5. Random Crop (crop size value above)\n6. Convert BCHW to CBHW\n\nInput data augmentations at validation time (with optional parameters):\n\n1. ConvertImageDtype\n2. Resize (resize size value above)\n3. Normalization (mean, std, see values above)\n4. Center Crop (crop size value above)\n5. Convert BCHW to CBHW\n\nThis translates in the following set of command-line arguments. Please note that `--batch-size` parameter controls the\nbatch size per GPU. Moreover, note that our default `--lr` is configured for 64 GPUs which is how many we used for the \nVideo resnet models:\n```\n# number of frames per clip\n--clip_len 16 \\ \n--frame-rate 15 \\\n# allow for temporal jittering\n--clips_per_video 5 \\\n--batch-size 24 \\\n--epochs 45 \\\n--lr 0.64 \\\n# we use 10 epochs for linear warmup\n--lr-warmup-epochs 10 \\\n# learning rate is decayed at 20, 30, and 40 epoch by a factor of 10\n--lr-milestones 20, 30, 40 \\\n--lr-gamma 0.1 \\\n--train-resize-size 128 171 \\\n--train-crop-size 112 112 \\\n--val-resize-size 128 171 \\\n--val-crop-size 112 112\n```\n\n### S3D\n\nThe S3D model was trained similarly to the above but with the following changes on the default configuration:\n```\n--batch-size=12 --lr 0.2 --clip-len 64 --clips-per-video 5 --sync-bn \\\n--train-resize-size 256 256 --train-crop-size 224 224 --val-resize-size 256 256 --val-crop-size 224 224\n```\n\nWe used 64 GPUs to train the architecture. \n\nTo estimate the validation statistics of the model, we run the reference script with the following configuration:\n```\n--batch-size=16 --test-only --clip-len 128 --clips-per-video 1 \n```\n\n### Additional video modelling resources\n\n- [Video Model Zoo](https://github.com/facebookresearch/VMZ)\n- [PySlowFast](https://github.com/facebookresearch/SlowFast)\n\n### References\n\n[0] _D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri_: A Closer Look at Spatiotemporal Convolutions for Action Recognition. _CVPR 2018_ ([paper](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf))\n\n[1] _W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, A. Zisserman_: The Kinetics Human Action Video Dataset ([paper](https://arxiv.org/abs/1705.06950))\n","references/detection/README.md":"# Object detection reference training scripts\n\nThis folder contains reference training scripts for object detection.\nThey serve as a log of how to train specific models, to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nTo execute the example commands below you must install the following:\n\n```\ncython\npycocotools\nmatplotlib\n```\n\nYou must modify the following flags:\n\n`--data-path=/path/to/coco/dataset`\n\n`--nproc_per_node=<number_of_gpus_available>`\n\nExcept otherwise noted, all models have been trained on 8x V100 GPUs. \n\n### Faster R-CNN ResNet-50 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### Faster R-CNN MobileNetV3-Large FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_mobilenet_v3_large_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n### Faster R-CNN MobileNetV3-Large 320 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_mobilenet_v3_large_320_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n### FCOS ResNet-50 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fcos_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3  --lr 0.01 --amp --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### RetinaNet\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model retinanet_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --lr 0.01 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### SSD300 VGG16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model ssd300_vgg16 --epochs 120\\\n    --lr-steps 80 110 --aspect-ratio-group-factor 3 --lr 0.002 --batch-size 4\\\n    --weight-decay 0.0005 --data-augmentation ssd --weights-backbone VGG16_Weights.IMAGENET1K_FEATURES\n```\n\n### SSDlite320 MobileNetV3-Large\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model ssdlite320_mobilenet_v3_large --epochs 660\\\n    --aspect-ratio-group-factor 3 --lr-scheduler cosineannealinglr --lr 0.15 --batch-size 24\\\n    --weight-decay 0.00004 --data-augmentation ssdlite\n```\n\n\n### Mask R-CNN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model maskrcnn_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n\n### Keypoint R-CNN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco_kp --model keypointrcnn_resnet50_fpn --epochs 46\\\n    --lr-steps 36 43 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n","references/depth/stereo/README.md":"# Stereo Matching reference training scripts\n\nThis folder contains reference training scripts for Stereo Matching.\nThey serve as a log of how to train specific models, so as to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nNote: this depends on the `prototype` folder which was removed, the latest\ncommit with the prototype folder is 8b6ba92ce5c48d6db596f6d415a90c128bc5f77b.\n\n\n### CREStereo\n\nThe CREStereo model was trained on a dataset mixture between **CREStereo**, **ETH3D** and the additional split from **Middlebury2014**.\nA ratio of **88-6-6** was used in order to train a baseline weight set. We provide multi-set variant as well.\nBoth used 8 A100 GPUs and a batch size of 2 (so effective batch size is 16). The\nrest of the hyper-parameters loosely follow the recipe from https://github.com/megvii-research/CREStereo.\nThe original recipe trains for **300000** updates (or steps) on the dataset mixture. We modify the learning rate\nschedule to one that starts decaying the weight much sooner. Throughout the experiments we found that this reduces \noverfitting during evaluation time and gradient clip help stabilize the loss during a pre-mature learning rate change.\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_cre \\\n    --model crestereo_base \\\n    --train-datasets crestereo eth3d-train middlebury2014-other \\\n    --dataset-steps 264000 18000 18000\n    --batch-size 2 \\\n    --lr 0.0004 \\\n    --min-lr 0.00002 \\\n    --lr-decay-method cosine \\\n    --warmup-steps 6000 \\\n    --decay-after-steps 30000 \\\n    --clip-grad-norm 1.0 \\\n```\n\nWe employ a multi-set fine-tuning stage where we uniformly sample from multiple datasets. Given hat some of these datasets have extremely large images (``2048x2048`` or more) we opt for a very aggressive scale-range ``[0.2 - 0.8]`` such that as much of the original frame composition is captured inside the ``384x512`` crop.\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_things \\\n    --model crestereo_base \\\n    --train-datasets crestereo eth3d-train middlebury2014-other instereo2k fallingthings carla-highres sintel sceneflow-monkaa sceneflow-driving \\\n    --dataset-steps 12000 12000 12000 12000 12000 12000 12000 12000 12000\n    --batch-size 2 \\\n    --scale-range 0.2 0.8 \\\n    --lr 0.0004 \\\n    --lr-decay-method cosine \\\n    --decay-after-steps 0 \\\n    --warmup-steps 0 \\\n    --min-lr 0.00002 \\\n    --resume-path $checkpoint_dir/$name_cre.pth\n```\n\n\n### Evaluation\n\nEvaluating the base weights\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.CRESTEREO_ETH_MBL_V1\n```\n\nThis should give an **mae of about 1.416** on the train set of `Middlebury2014`. Results may vary slightly depending on the batch size and the number of GPUs. For the most accurate results use 1 GPU and `--batch-size 1`. The created log file should look like this, where the first key is the number of cascades and the nested key is the number of recursive iterations:\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 2.363, 'rmse': 4.352, '1px': 0.611, '3px': 0.828, '5px': 0.891, 'relepe': 0.176, 'fl-all': 64.511}\n\t\t5: {'mae': 1.618, 'rmse': 3.71, '1px': 0.761, '3px': 0.879, '5px': 0.918, 'relepe': 0.154, 'fl-all': 77.128}\n\t\t10: {'mae': 1.416, 'rmse': 3.53, '1px': 0.777, '3px': 0.896, '5px': 0.933, 'relepe': 0.148, 'fl-all': 78.388}\n\t\t20: {'mae': 1.448, 'rmse': 3.583, '1px': 0.771, '3px': 0.893, '5px': 0.931, 'relepe': 0.145, 'fl-all': 77.7}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.972, 'rmse': 4.125, '1px': 0.73, '3px': 0.865, '5px': 0.908, 'relepe': 0.169, 'fl-all': 74.396}\n\t\t5: {'mae': 1.403, 'rmse': 3.448, '1px': 0.793, '3px': 0.905, '5px': 0.937, 'relepe': 0.151, 'fl-all': 80.186}\n\t\t10: {'mae': 1.312, 'rmse': 3.368, '1px': 0.799, '3px': 0.912, '5px': 0.943, 'relepe': 0.148, 'fl-all': 80.379}\n\t\t20: {'mae': 1.376, 'rmse': 3.542, '1px': 0.796, '3px': 0.91, '5px': 0.942, 'relepe': 0.149, 'fl-all': 80.054}\n\t},\n}\n```\n\nYou can also evaluate the Finetuned weights:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.CRESTEREO_FINETUNE_MULTI_V1\n```\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 1.85, 'rmse': 3.797, '1px': 0.673, '3px': 0.862, '5px': 0.917, 'relepe': 0.171, 'fl-all': 69.736}\n\t\t5: {'mae': 1.111, 'rmse': 3.166, '1px': 0.838, '3px': 0.93, '5px': 0.957, 'relepe': 0.134, 'fl-all': 84.596}\n\t\t10: {'mae': 1.02, 'rmse': 3.073, '1px': 0.854, '3px': 0.938, '5px': 0.96, 'relepe': 0.129, 'fl-all': 86.042}\n\t\t20: {'mae': 0.993, 'rmse': 3.059, '1px': 0.855, '3px': 0.942, '5px': 0.967, 'relepe': 0.126, 'fl-all': 85.784}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.667, 'rmse': 3.867, '1px': 0.78, '3px': 0.891, '5px': 0.922, 'relepe': 0.165, 'fl-all': 78.89}\n\t\t5: {'mae': 1.158, 'rmse': 3.278, '1px': 0.843, '3px': 0.926, '5px': 0.955, 'relepe': 0.135, 'fl-all': 84.556}\n\t\t10: {'mae': 1.046, 'rmse': 3.13, '1px': 0.85, '3px': 0.934, '5px': 0.96, 'relepe': 0.13, 'fl-all': 85.464}\n\t\t20: {'mae': 1.021, 'rmse': 3.102, '1px': 0.85, '3px': 0.935, '5px': 0.963, 'relepe': 0.129, 'fl-all': 85.417}\n\t},\n}\n```\n\nEvaluating the author provided weights:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.MEGVII_V1\n```\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 1.704, 'rmse': 3.738, '1px': 0.738, '3px': 0.896, '5px': 0.933, 'relepe': 0.157, 'fl-all': 76.464}\n\t\t5: {'mae': 0.956, 'rmse': 2.963, '1px': 0.88, '3px': 0.948, '5px': 0.965, 'relepe': 0.124, 'fl-all': 88.186}\n\t\t10: {'mae': 0.792, 'rmse': 2.765, '1px': 0.905, '3px': 0.958, '5px': 0.97, 'relepe': 0.114, 'fl-all': 90.429}\n\t\t20: {'mae': 0.749, 'rmse': 2.706, '1px': 0.907, '3px': 0.961, '5px': 0.972, 'relepe': 0.113, 'fl-all': 90.807}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.702, 'rmse': 3.784, '1px': 0.784, '3px': 0.894, '5px': 0.924, 'relepe': 0.172, 'fl-all': 80.313}\n\t\t5: {'mae': 0.932, 'rmse': 2.907, '1px': 0.877, '3px': 0.944, '5px': 0.963, 'relepe': 0.125, 'fl-all': 87.979}\n\t\t10: {'mae': 0.773, 'rmse': 2.768, '1px': 0.901, '3px': 0.958, '5px': 0.972, 'relepe': 0.117, 'fl-all': 90.43}\n\t\t20: {'mae': 0.854, 'rmse': 2.971, '1px': 0.9, '3px': 0.957, '5px': 0.97, 'relepe': 0.122, 'fl-all': 90.269}\n\t},\n}\n```\n\n# Concerns when training\n\nWe encourage users to be aware of the **aspect-ratio** and **disparity scale** they are targeting when doing any sort of training or fine-tuning. The model is highly sensitive to these two factors, as a consequence of naive multi-set fine-tuning one can achieve `0.2 mae` relatively fast. We recommend that users pay close attention to how they **balance dataset sizing** when training such networks.\n\n Ideally, dataset scaling should be trated at an individual level and a thorough **EDA** of the disparity distribution in random crops at the desired training / inference size should be performed prior to any large compute investments.\n\n### Disparity scaling\n\n##### Sample A\n The top row contains a sample from `Sintel` whereas the bottom row one from `Middlebury`.\n\n![Disparity1](assets/disparity-domain-drift.jpg)\n\nFrom left to right (`left_image`, `right_image`, `valid_mask`, `valid_mask & ground_truth`, `prediction`). **Darker is further away, lighter is closer**. In the case of `Sintel` which is more closely aligned to the original distribution of `CREStereo` we notice that the model accurately predicts the background scale whereas in the case of `Middlebury2014` it cannot correctly estimate the continuous disparity. Notice that the frame composition is similar for both examples. The blue skybox in the `Sintel` scene behaves similarly to the `Middlebury` black background. However, because the `Middlebury` samples comes from an extremely large scene the crop size of `384x512` does not correctly capture the general training distribution.\n\n\n\n\n##### Sample B\n\nThe top row contains a scene from `Sceneflow` using the `Monkaa` split whilst the bottom row is a scene from `Middlebury`. This sample exhibits the same issues when it comes to **background estimation**. Given the exaggerated size of the `Middlebury` samples the model **colapses the smooth background** of the sample to what it considers to be a mean background disparity value.\n\n![Disparity2](assets/disparity-background-mode-collapse.jpg)\n\n\nFor more detail on why this behaviour occurs based on the training distribution proportions you can read more about the network at: https://github.com/pytorch/vision/pull/6629#discussion_r978160493\n\n\n### Metric overfitting\n\n##### Learning is critical in the beginning\n\nWe also advise users to make user of faster training schedules, as the performance gain over long periods time is marginal. Here we exhibit a difference between a faster decay schedule and later decay schedule.\n\n![Loss1](assets/Loss.jpg)\n\nIn **grey** we set the lr decay to begin after `30000` steps whilst in **orange** we opt for a very late learning rate decay at around `180000` steps. Although exhibiting stronger variance, we can notice that unfreezing the learning rate earlier whilst employing `gradient-norm` out-performs the default configuration.\n\n##### Gradient norm saves time\n\n![Loss2](assets/gradient-norm-removal.jpg)\n\nIn **grey** we keep ``gradient norm`` enabled whilst in **orange** we do not. We can notice that remvoing the gradient norm exacerbates the performance decrease in the early stages whilst also showcasing an almost complete collapse around the `60000` steps mark where we started decaying the lr for **orange**.\n\nAlthough both runs ahieve an improvement of about ``0.1`` mae after the lr decay start, the benefits of it are observable much faster when ``gradient norm`` is employed as the recovery period is no longer accounted for.\n"},"files":{"references/classification/README.md":"# Image classification reference training scripts\n\nThis folder contains reference training scripts for image classification.\nThey serve as a log of how to train specific models, as provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nExcept otherwise noted, all models have been trained on 8x V100 GPUs with \nthe following parameters:\n\n| Parameter                | value  |\n| ------------------------ | ------ |\n| `--batch_size`           | `32`   |\n| `--epochs`               | `90`   |\n| `--lr`                   | `0.1`  |\n| `--momentum`             | `0.9`  |\n| `--wd`, `--weight-decay` | `1e-4` |\n| `--lr-step-size`         | `30`   |\n| `--lr-gamma`             | `0.1`  |\n\n### AlexNet and VGG\n\nSince `AlexNet` and the original `VGG` architectures do not include batch \nnormalization, the default initial learning rate `--lr 0.1` is too high.\n\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model $MODEL --lr 1e-2\n```\n\nHere `$MODEL` is one of `alexnet`, `vgg11`, `vgg13`, `vgg16` or `vgg19`. Note\nthat `vgg11_bn`, `vgg13_bn`, `vgg16_bn`, and `vgg19_bn` include batch\nnormalization and thus are trained with the default parameters.\n\n### GoogLeNet\n\nThe weights of the GoogLeNet model are ported from the original paper rather than trained from scratch.\n\n### Inception V3\n\nThe weights of the Inception V3 model are ported from the original paper rather than trained from scratch.\n\nSince it expects tensors with a size of N x 3 x 299 x 299, to validate the model use the following command:\n\n```\ntorchrun --nproc_per_node=8 train.py --model inception_v3\\\n      --test-only --weights Inception_V3_Weights.IMAGENET1K_V1\n```\n\n### ResNet\n```\ntorchrun --nproc_per_node=8 train.py --model $MODEL\n```\n\nHere `$MODEL` is one of `resnet18`, `resnet34`, `resnet50`, `resnet101` or `resnet152`.\n\n### ResNext\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model $MODEL --epochs 100\n```\n\nHere `$MODEL` is one of `resnext50_32x4d` or `resnext101_32x8d`.\nNote that the above command corresponds to a single node with 8 GPUs. If you use\na different number of GPUs and/or a different batch size, then the learning rate\nshould be scaled accordingly. For example, the pretrained model provided by\n`torchvision` was trained on 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nwith `--batch_size 16` and `--lr 0.4`, instead of the current defaults\nwhich are respectively batch_size=32 and lr=0.1\n\n### MobileNetV2\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model mobilenet_v2 --epochs 300 --lr 0.045 --wd 0.00004\\\n     --lr-step-size 1 --lr-gamma 0.98\n```\n\n\n### MobileNetV3 Large & Small\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 600 --opt rmsprop --batch-size 128 --lr 0.064\\ \n     --wd 0.00001 --lr-step-size 2 --lr-gamma 0.973 --auto-augment imagenet --random-erase 0.2\n```\n\nHere `$MODEL` is one of `mobilenet_v3_large` or `mobilenet_v3_small`.\n\nThen we averaged the parameters of the last 3 checkpoints that improved the Acc@1. See [#3182](https://github.com/pytorch/vision/pull/3182) \nand [#3354](https://github.com/pytorch/vision/pull/3354) for details.\n\n\n### EfficientNet-V1\n\nThe weights of the B0-B4 variants are ported from Ross Wightman's [timm repo](https://github.com/rwightman/pytorch-image-models/blob/01cb46a9a50e3ba4be167965b5764e9702f09b30/timm/models/efficientnet.py#L95-L108).\n\nThe weights of the B5-B7 variants are ported from Luke Melas' [EfficientNet-PyTorch repo](https://github.com/lukemelas/EfficientNet-PyTorch/blob/1039e009545d9329ea026c9f7541341439712b96/efficientnet_pytorch/utils.py#L562-L564).\n\nAll models were trained using Bicubic interpolation and each have custom crop and resize sizes. To validate the models use the following commands:\n```\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b0 --test-only --weights EfficientNet_B0_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b1 --test-only --weights EfficientNet_B1_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b2 --test-only --weights EfficientNet_B2_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b3 --test-only --weights EfficientNet_B3_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b4 --test-only --weights EfficientNet_B4_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b5 --test-only --weights EfficientNet_B5_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b6 --test-only --weights EfficientNet_B6_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b7 --test-only --weights EfficientNet_B7_Weights.IMAGENET1K_V1\n```\n\n\n### EfficientNet-V2\n```\ntorchrun --nproc_per_node=8 train.py \\\n--model $MODEL --batch-size 128 --lr 0.5 --lr-scheduler cosineannealinglr \\\n--lr-warmup-epochs 5 --lr-warmup-method linear --auto-augment ta_wide --epochs 600 --random-erase 0.1 \\\n--label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --weight-decay 0.00002 --norm-weight-decay 0.0 \\\n--train-crop-size $TRAIN_SIZE --model-ema --val-crop-size $EVAL_SIZE --val-resize-size $EVAL_SIZE \\\n--ra-sampler --ra-reps 4\n```\nHere `$MODEL` is one of `efficientnet_v2_s` and `efficientnet_v2_m`. \nNote that the Small variant had a `$TRAIN_SIZE` of `300` and a `$EVAL_SIZE` of `384`, while the Medium `384` and `480` respectively.\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 4 nodes, each with 8 GPUs (for a total of 32 GPUs),\nand `--batch_size 32`.\n\nThe weights of the Large variant are ported from the original paper rather than trained from scratch. See the `EfficientNet_V2_L_Weights` entry for their exact preprocessing transforms.\n\n\n### RegNet\n\n#### Small models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 128 --wd 0.00005 --lr=0.8\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_400mf`, `regnet_x_800mf`, `regnet_x_1_6gf`, `regnet_y_400mf`, `regnet_y_800mf` and `regnet_y_1_6gf`. Please note we used learning rate 0.4 for `regent_y_400mf` to get the same Acc@1 as [the paper](https://arxiv.org/abs/2003.13678).\n\n#### Medium models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 64 --wd 0.00005 --lr=0.4\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_3_2gf`, `regnet_x_8gf`, `regnet_x_16gf`, `regnet_y_3_2gf` and `regnet_y_8gf`.\n\n#### Large models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 32 --wd 0.00005 --lr=0.2\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_32gf`, `regnet_y_16gf` and `regnet_y_32gf`.\n\n### Vision Transformer\n\n#### vit_b_16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_b_16 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nand `--batch_size 64`.\n\n#### vit_b_32\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_b_32 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment imagenet\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 256`.\n\n#### vit_l_16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_l_16 --epochs 600 --batch-size 128 --lr 0.5 --lr-scheduler cosineannealinglr\\\n    --lr-warmup-method linear --lr-warmup-epochs 5 --label-smoothing 0.1 --mixup-alpha 0.2\\\n    --auto-augment ta_wide --random-erase 0.1 --weight-decay 0.00002 --norm-weight-decay 0.0\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema --val-resize-size 232\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 64`.\n\n#### vit_l_32\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_l_32 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nand `--batch_size 64`.\n\n\n### ConvNeXt\n```\ntorchrun --nproc_per_node=8 train.py\\ \n--model $MODEL --batch-size 128 --opt adamw --lr 1e-3 --lr-scheduler cosineannealinglr \\ \n--lr-warmup-epochs 5 --lr-warmup-method linear --auto-augment ta_wide --epochs 600 --random-erase 0.1 \\ \n--label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --weight-decay 0.05 --norm-weight-decay 0.0 \\\n--train-crop-size 176 --model-ema --val-resize-size 232 --ra-sampler --ra-reps 4\n```\nHere `$MODEL` is one of `convnext_tiny`, `convnext_small`, `convnext_base` and `convnext_large`. Note that each variant had its `--val-resize-size` optimized in a post-training step, see their `Weights` entry for their exact value.\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 64`.\n\n\n### SwinTransformer\n```\ntorchrun --nproc_per_node=8 train.py\\ \n--model $MODEL --epochs 300 --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler --ra-reps 4  --val-resize-size 224\n```\nHere `$MODEL` is one of `swin_t`, `swin_s` or `swin_b`.\nNote that `--val-resize-size` was optimized in a post-training step, see their `Weights` entry for the exact value.\n\n\n\n\n### SwinTransformer V2\n```\ntorchrun --nproc_per_node=8 train.py\\\n--model $MODEL --epochs 300 --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler --ra-reps 4  --val-resize-size 256 --val-crop-size 256 --train-crop-size 256 \n```\nHere `$MODEL` is one of `swin_v2_t`, `swin_v2_s` or `swin_v2_b`.\nNote that `--val-resize-size` was optimized in a post-training step, see their `Weights` entry for the exact value.\n\n\n### MaxViT\n```\ntorchrun --nproc_per_node=8 --n_nodes=4 train.py\\\n--model $MODEL --epochs 400 --batch-size 128 --opt adamw --lr 3e-3 --weight-decay 0.05 --lr-scheduler cosineannealinglr --lr-min 1e-5 --lr-warmup-method linear  --lr-warmup-epochs 32  --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 1.0 --interpolation bicubic --auto-augment ta_wide --policy-magnitude 15 --model-ema --val-resize-size 224\\\n--val-crop-size 224 --train-crop-size 224 --amp  --model-ema-steps 32 --transformer-embedding-decay 0 --sync-bn\n```\nHere `$MODEL` is `maxvit_t`.\nNote that `--val-resize-size` was not optimized in a post-training step.\n\n\n### ShuffleNet V2\n```\ntorchrun --nproc_per_node=8 train.py \\\n--batch-size=128 \\\n--lr=0.5 --lr-scheduler=cosineannealinglr --lr-warmup-epochs=5 --lr-warmup-method=linear \\\n--auto-augment=ta_wide --epochs=600 --random-erase=0.1 --weight-decay=0.00002 \\\n--norm-weight-decay=0.0 --label-smoothing=0.1 --mixup-alpha=0.2 --cutmix-alpha=1.0 \\\n--train-crop-size=176 --model-ema --val-resize-size=232 --ra-sampler --ra-reps=4\n```\nHere `$MODEL` is either `shufflenet_v2_x1_5` or `shufflenet_v2_x2_0`.\n\nThe models `shufflenet_v2_x0_5` and `shufflenet_v2_x1_0` were contributed by the community. See [PR-849](https://github.com/pytorch/vision/pull/849#issuecomment-483391686) for details.\n\n\n## Mixed precision training\nAutomatic Mixed Precision (AMP) training on GPU for Pytorch can be enabled with the [torch.cuda.amp](https://pytorch.org/docs/stable/amp.html?highlight=amp#module-torch.cuda.amp).\n\nMixed precision training makes use of both FP32 and FP16 precisions where appropriate. FP16 operations can leverage the Tensor cores on NVIDIA GPUs (Volta, Turing or newer architectures) for improved throughput, generally without loss in model accuracy. Mixed precision training also often allows larger batch sizes. GPU automatic mixed precision training for Pytorch Vision can be enabled via the flag value `--amp=True`.\n\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model resnext50_32x4d --epochs 100 --amp\n```\n\n## Quantized\n\n### Post training quantized models\n\nFor all post training quantized models, the settings are:\n\n1. num_calibration_batches: 32\n2. num_workers: 16\n3. batch_size: 32\n4. eval_batch_size: 128\n5. qbackend: 'fbgemm'\n\n```\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' --model='$MODEL'\n```\nHere `$MODEL` is one of `googlenet`, `inception_v3`, `resnet18`, `resnet50`, `resnext101_32x8d`, `shufflenet_v2_x0_5` and `shufflenet_v2_x1_0`.\n\n### Quantized ShuffleNet V2\n\nHere are commands that we use to quantize the `shufflenet_v2_x1_5` and `shufflenet_v2_x2_0` models.\n```\n# For shufflenet_v2_x1_5\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' \\\n    --model=shufflenet_v2_x1_5 --weights=\"ShuffleNet_V2_X1_5_Weights.IMAGENET1K_V1\" \\\n    --train-crop-size 176 --val-resize-size 232 --data-path /datasets01_ontap/imagenet_full_size/061417/\n\n# For shufflenet_v2_x2_0\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' \\\n    --model=shufflenet_v2_x2_0 --weights=\"ShuffleNet_V2_X2_0_Weights.IMAGENET1K_V1\" \\\n    --train-crop-size 176 --val-resize-size 232 --data-path /datasets01_ontap/imagenet_full_size/061417/\n```\n\n### QAT MobileNetV2\n\nFor Mobilenet-v2, the model was trained with quantization aware training, the settings used are:\n1. num_workers: 16\n2. batch_size: 32\n3. eval_batch_size: 128\n4. qbackend: 'qnnpack'\n5. learning-rate: 0.0001\n6. num_epochs: 90\n7. num_observer_update_epochs:4\n8. num_batch_norm_update_epochs:3\n9. momentum: 0.9\n10. lr_step_size:30\n11. lr_gamma: 0.1\n12. weight-decay: 0.0001\n\n```\ntorchrun --nproc_per_node=8 train_quantization.py --model='mobilenet_v2'\n```\n\nTraining converges at about 10 epochs.\n\n### QAT MobileNetV3\n\nFor Mobilenet-v3 Large, the model was trained with quantization aware training, the settings used are:\n1. num_workers: 16\n2. batch_size: 32\n3. eval_batch_size: 128\n4. qbackend: 'qnnpack'\n5. learning-rate: 0.001\n6. num_epochs: 90\n7. num_observer_update_epochs:4\n8. num_batch_norm_update_epochs:3\n9. momentum: 0.9\n10. lr_step_size:30\n11. lr_gamma: 0.1\n12. weight-decay: 0.00001\n\n```\ntorchrun --nproc_per_node=8 train_quantization.py --model='mobilenet_v3_large' \\\n    --wd 0.00001 --lr 0.001\n```\n\nFor post training quant, device is set to CPU. For training, the device is set to CUDA.\n\n### Command to evaluate quantized models using the pre-trained weights:\n\n```\npython train_quantization.py --device='cpu' --test-only --qbackend='<qbackend>' --model='<model_name>'\n```\n\nFor inception_v3 you need to pass the following extra parameters:\n```\n--val-resize-size 342 --val-crop-size 299 --train-crop-size 299\n```\n","references/optical_flow/README.md":"# Optical flow reference training scripts\n\nThis folder contains reference training scripts for optical flow.\nThey serve as a log of how to train specific models, so as to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\n\n### RAFT Large\n\nThe RAFT large model was trained on Flying Chairs and then on Flying Things.\nBoth used 8 A100 GPUs and a batch size of 2 (so effective batch size is 16). The\nrest of the hyper-parameters are exactly the same as the original RAFT training\nrecipe from https://github.com/princeton-vl/RAFT. The original recipe trains for\n100000 updates (or steps) on each dataset - this corresponds to about 72 and 20\nepochs on Chairs and Things respectively:\n\n```\nnum_epochs = ceil(num_steps / number_of_steps_per_epoch)\n           = ceil(num_steps / (num_samples / effective_batch_size))\n```\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_chairs \\\n    --model raft_large \\\n    --train-dataset chairs \\\n    --batch-size 2 \\\n    --lr 0.0004 \\\n    --weight-decay 0.0001 \\\n    --epochs 72 \\\n    --output-dir $chairs_dir\n```\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_things \\\n    --model raft_large \\\n    --train-dataset things \\\n    --batch-size 2 \\\n    --lr 0.000125 \\\n    --weight-decay 0.0001 \\\n    --epochs 20 \\\n    --freeze-batch-norm \\\n    --output-dir $things_dir\\\n    --resume $chairs_dir/$name_chairs.pth\n```\n\n\n### Evaluation\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 train.py --val-dataset sintel --batch-size 1 --dataset-root $dataset_root --model raft_large --weights Raft_Large_Weights.C_T_SKHT_V2\n```\n\nThis should give an epe of about 1.3822 on the clean pass and 2.7161 on the\nfinal pass of Sintel-train. Results may vary slightly depending on the batch\nsize and the number of GPUs. For the most accurate results use 1 GPU and\n`--batch-size 1`:\n\n```\nSintel val clean epe: 1.3822\t1px: 0.9028\t3px: 0.9573\t5px: 0.9697\tper_image_epe: 1.3822\tf1: 4.0248\nSintel val final epe: 2.7161\t1px: 0.8528\t3px: 0.9204\t5px: 0.9392\tper_image_epe: 2.7161\tf1: 7.5964\n```\n\nYou can also evaluate on Kitti train:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 train.py --val-dataset kitti --batch-size 1 --dataset-root $dataset_root --model raft_large --weights Raft_Large_Weights.C_T_SKHT_V2\nKitti val epe: 4.7968\t1px: 0.6388\t3px: 0.8197\t5px: 0.8661\tper_image_epe: 4.5118\tf1: 16.0679\n```\n","references/segmentation/README.md":"# Semantic segmentation reference training scripts\n\nThis folder contains reference training scripts for semantic segmentation.\nThey serve as a log of how to train specific models and provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nAll models have been trained on 8x V100 GPUs.\n\nYou must modify the following flags:\n\n`--data-path=/path/to/dataset`\n\n`--nproc_per_node=<number_of_gpus_available>`\n\n## fcn_resnet50\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model fcn_resnet50 --aux-loss --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n## fcn_resnet101\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model fcn_resnet101 --aux-loss --weights-backbone ResNet101_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_resnet50\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model deeplabv3_resnet50 --aux-loss --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_resnet101\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model deeplabv3_resnet101 --aux-loss --weights-backbone ResNet101_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_mobilenet_v3_large\n```\ntorchrun --nproc_per_node=8 train.py --dataset coco -b 4 --model deeplabv3_mobilenet_v3_large --aux-loss --wd 0.000001 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n## lraspp_mobilenet_v3_large\n```\ntorchrun --nproc_per_node=8 train.py --dataset coco -b 4 --model lraspp_mobilenet_v3_large --wd 0.000001 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n","references/similarity/README.md":"# Similarity Learning Using Triplet Loss #\n\nIn this reference, we use triplet loss to learn embeddings which can be used to differentiate images. This learning technique was popularized by [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/abs/1503.03832) and has been quite effective in learning embeddings to differentiate between faces.\n\nThis reference can be directly applied to the following use cases:\n\n* You have an unknown number of classes and would like to train a model to learn how to differentiate between them.\n* You want to train a model to learn a distance-based metric between samples. For example, learning a distance-based similarity measure between faces.\n\n### Training ###\nBy default, the training script trains ResNet50 on the FashionMNIST Dataset to learn image embeddings which can be used to differentiate between images by measuring the euclidean distance between embeddings. This can be changed as per your requirements.\n\nImage embeddings of the same class should be 'close' to each other, while image embeddings between different classes should be 'far' away.\n\nTo run the training script:\n\n```bash\npython train.py -h    # Lists all optional arguments\npython train.py \t\t\t# Runs training script with default args\n```\n\nRunning the training script as is should yield 97% accuracy on the FMNIST test set within 10 epochs.\n\n### Loss ###\n`TripletMarginLoss` is a loss function which takes in a triplet of samples. A valid triplet has an:\n\n1. Anchor: a sample from the dataset\n2. Positive: another sample with the same label/group as the anchor (Generally, positive != anchor)\n3. Negative: a sample with a different label/group from the anchor\n\n`TripletMarginLoss` (refer to `loss.py`) does the following:\n\n```python\nloss = max(dist(anchor, positive) - dist(anchor, negative) + margin, 0)\n```\nWhere `dist` is a distance function. Minimizing this function effectively leads to minimizing `dist(anchor, positive)` and maximizing `dist(anchor, negative)`.\n\nThe FaceNet paper describe this loss in more detail.\n\n### Sampler ###\n\nIn order to generate valid triplets from a batch of samples, we need to make sure that each batch has multiple samples with the same label. We do this using `PKSampler` (refer to `sampler.py`), which ensures that each batch of size `p * k` will have samples from exactly `p` classes and `k` samples per class.\n\n### Triplet Mining ###\n\n`TripletMarginLoss` currently supports the following mining techniques:\n\n* `batch_all`: Generates all possible triplets from a batch and excludes the triplets which are 'easy' (which have `loss = 0`) before passing it through the loss function.\n* `batch_hard`: For every anchor, `batch_hard` creates a triplet with the 'hardest' positive (farthest positive) and negative (closest negative).\n\nThese mining strategies usually speed up training.\n\nThis [webpage](https://omoindrot.github.io/triplet-loss) describes the sampling and mining strategies in more detail. \n","references/video_classification/README.md":"# Video Classification\n\nWe present a simple training script that can be used for replicating the result of [resenet-based video models](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf). All models are trained on [Kinetics400 dataset](https://deepmind.com/research/open-source/kinetics), a benchmark dataset for human-action recognition. The accuracy is reported on the traditional validation split.\n\n## Data preparation\n\nIf you already have downloaded [Kinetics400 dataset](https://deepmind.com/research/open-source/kinetics), \nplease proceed directly to the next section.\n\nTo download videos, one can use https://github.com/Showmax/kinetics-downloader. Please note that the dataset can take up upwards of 400GB, depending on the quality setting during download.\n\n## Training\n\nWe assume the training and validation AVI videos are stored at `/data/kinectics400/train` and \n`/data/kinectics400/val`. For training we suggest starting with the hyperparameters reported in the [paper](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf), in order to match the performance of said models. Clip sampling strategy is a particularly important parameter during training, and we suggest using random temporal jittering during training - in other words sampling multiple training clips from each video with random start times during at every epoch. This functionality is built into our training script, and optimal hyperparameters are set by default.  \n\n### Multiple GPUs\n\nRun the training on a single node with 8 GPUs:\n```bash\ntorchrun --nproc_per_node=8 train.py --data-path=/data/kinectics400 --kinetics-version=\"400\" --lr 0.08 --cache-dataset --sync-bn --amp \n```\n\n**Note:** all our models were trained on 8 nodes with 8 V100 GPUs each for a total of 64 GPUs. Expected training time for 64 GPUs is 24 hours, depending on the storage solution.\n**Note 2:** hyperparameters for exact replication of our training can be found on the section below. Some hyperparameters such as learning rate must be scaled linearly in proportion to the number of GPUs. The default values assume 64 GPUs.\n\n### Single GPU \n\n**Note:** training on a single gpu can be extremely slow. \n\n\n```bash\npython train.py --data-path=/data/kinectics400 --kinetics-version=\"400\" --batch-size=8 --cache-dataset\n```\n\n\n### Additional Kinetics versions\n\nSince the original release, additional versions of Kinetics dataset became available (Kinetics 600).\nOur training scripts support these versions of dataset as well by setting the `--kinetics-version` parameter to `\"600\"`.\n\n**Note:** training on Kinetics 600 requires a different set of hyperparameters for optimal performance. We do not provide Kinetics 600 pretrained models.\n\n\n## Video classification models\n\nStarting with version `0.4.0` we have introduced support for basic video tasks and video classification modelling.\nFor more information about the available models check [here](https://pytorch.org/docs/stable/torchvision/models.html#video-classification). \n\n### Video ResNet models\n\nSee reference training script [here](https://github.com/pytorch/vision/blob/main/references/video_classification/train.py):\n\n- input space: RGB\n- resize size: [128, 171]\n- crop size: [112, 112]\n- mean: [0.43216, 0.394666, 0.37645]\n- std: [0.22803, 0.22145, 0.216989]\n- number of classes: 400\n\nInput data augmentations at training time (with optional parameters):\n\n1. ConvertImageDtype\n2. Resize (resize size value above)\n3. Random horizontal flip (0.5)\n4. Normalization (mean, std, see values above)\n5. Random Crop (crop size value above)\n6. Convert BCHW to CBHW\n\nInput data augmentations at validation time (with optional parameters):\n\n1. ConvertImageDtype\n2. Resize (resize size value above)\n3. Normalization (mean, std, see values above)\n4. Center Crop (crop size value above)\n5. Convert BCHW to CBHW\n\nThis translates in the following set of command-line arguments. Please note that `--batch-size` parameter controls the\nbatch size per GPU. Moreover, note that our default `--lr` is configured for 64 GPUs which is how many we used for the \nVideo resnet models:\n```\n# number of frames per clip\n--clip_len 16 \\ \n--frame-rate 15 \\\n# allow for temporal jittering\n--clips_per_video 5 \\\n--batch-size 24 \\\n--epochs 45 \\\n--lr 0.64 \\\n# we use 10 epochs for linear warmup\n--lr-warmup-epochs 10 \\\n# learning rate is decayed at 20, 30, and 40 epoch by a factor of 10\n--lr-milestones 20, 30, 40 \\\n--lr-gamma 0.1 \\\n--train-resize-size 128 171 \\\n--train-crop-size 112 112 \\\n--val-resize-size 128 171 \\\n--val-crop-size 112 112\n```\n\n### S3D\n\nThe S3D model was trained similarly to the above but with the following changes on the default configuration:\n```\n--batch-size=12 --lr 0.2 --clip-len 64 --clips-per-video 5 --sync-bn \\\n--train-resize-size 256 256 --train-crop-size 224 224 --val-resize-size 256 256 --val-crop-size 224 224\n```\n\nWe used 64 GPUs to train the architecture. \n\nTo estimate the validation statistics of the model, we run the reference script with the following configuration:\n```\n--batch-size=16 --test-only --clip-len 128 --clips-per-video 1 \n```\n\n### Additional video modelling resources\n\n- [Video Model Zoo](https://github.com/facebookresearch/VMZ)\n- [PySlowFast](https://github.com/facebookresearch/SlowFast)\n\n### References\n\n[0] _D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri_: A Closer Look at Spatiotemporal Convolutions for Action Recognition. _CVPR 2018_ ([paper](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf))\n\n[1] _W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, A. Zisserman_: The Kinetics Human Action Video Dataset ([paper](https://arxiv.org/abs/1705.06950))\n","references/detection/README.md":"# Object detection reference training scripts\n\nThis folder contains reference training scripts for object detection.\nThey serve as a log of how to train specific models, to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nTo execute the example commands below you must install the following:\n\n```\ncython\npycocotools\nmatplotlib\n```\n\nYou must modify the following flags:\n\n`--data-path=/path/to/coco/dataset`\n\n`--nproc_per_node=<number_of_gpus_available>`\n\nExcept otherwise noted, all models have been trained on 8x V100 GPUs. \n\n### Faster R-CNN ResNet-50 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### Faster R-CNN MobileNetV3-Large FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_mobilenet_v3_large_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n### Faster R-CNN MobileNetV3-Large 320 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_mobilenet_v3_large_320_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n### FCOS ResNet-50 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fcos_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3  --lr 0.01 --amp --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### RetinaNet\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model retinanet_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --lr 0.01 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### SSD300 VGG16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model ssd300_vgg16 --epochs 120\\\n    --lr-steps 80 110 --aspect-ratio-group-factor 3 --lr 0.002 --batch-size 4\\\n    --weight-decay 0.0005 --data-augmentation ssd --weights-backbone VGG16_Weights.IMAGENET1K_FEATURES\n```\n\n### SSDlite320 MobileNetV3-Large\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model ssdlite320_mobilenet_v3_large --epochs 660\\\n    --aspect-ratio-group-factor 3 --lr-scheduler cosineannealinglr --lr 0.15 --batch-size 24\\\n    --weight-decay 0.00004 --data-augmentation ssdlite\n```\n\n\n### Mask R-CNN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model maskrcnn_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n\n### Keypoint R-CNN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco_kp --model keypointrcnn_resnet50_fpn --epochs 46\\\n    --lr-steps 36 43 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n","references/depth/stereo/README.md":"# Stereo Matching reference training scripts\n\nThis folder contains reference training scripts for Stereo Matching.\nThey serve as a log of how to train specific models, so as to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nNote: this depends on the `prototype` folder which was removed, the latest\ncommit with the prototype folder is 8b6ba92ce5c48d6db596f6d415a90c128bc5f77b.\n\n\n### CREStereo\n\nThe CREStereo model was trained on a dataset mixture between **CREStereo**, **ETH3D** and the additional split from **Middlebury2014**.\nA ratio of **88-6-6** was used in order to train a baseline weight set. We provide multi-set variant as well.\nBoth used 8 A100 GPUs and a batch size of 2 (so effective batch size is 16). The\nrest of the hyper-parameters loosely follow the recipe from https://github.com/megvii-research/CREStereo.\nThe original recipe trains for **300000** updates (or steps) on the dataset mixture. We modify the learning rate\nschedule to one that starts decaying the weight much sooner. Throughout the experiments we found that this reduces \noverfitting during evaluation time and gradient clip help stabilize the loss during a pre-mature learning rate change.\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_cre \\\n    --model crestereo_base \\\n    --train-datasets crestereo eth3d-train middlebury2014-other \\\n    --dataset-steps 264000 18000 18000\n    --batch-size 2 \\\n    --lr 0.0004 \\\n    --min-lr 0.00002 \\\n    --lr-decay-method cosine \\\n    --warmup-steps 6000 \\\n    --decay-after-steps 30000 \\\n    --clip-grad-norm 1.0 \\\n```\n\nWe employ a multi-set fine-tuning stage where we uniformly sample from multiple datasets. Given hat some of these datasets have extremely large images (``2048x2048`` or more) we opt for a very aggressive scale-range ``[0.2 - 0.8]`` such that as much of the original frame composition is captured inside the ``384x512`` crop.\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_things \\\n    --model crestereo_base \\\n    --train-datasets crestereo eth3d-train middlebury2014-other instereo2k fallingthings carla-highres sintel sceneflow-monkaa sceneflow-driving \\\n    --dataset-steps 12000 12000 12000 12000 12000 12000 12000 12000 12000\n    --batch-size 2 \\\n    --scale-range 0.2 0.8 \\\n    --lr 0.0004 \\\n    --lr-decay-method cosine \\\n    --decay-after-steps 0 \\\n    --warmup-steps 0 \\\n    --min-lr 0.00002 \\\n    --resume-path $checkpoint_dir/$name_cre.pth\n```\n\n\n### Evaluation\n\nEvaluating the base weights\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.CRESTEREO_ETH_MBL_V1\n```\n\nThis should give an **mae of about 1.416** on the train set of `Middlebury2014`. Results may vary slightly depending on the batch size and the number of GPUs. For the most accurate results use 1 GPU and `--batch-size 1`. The created log file should look like this, where the first key is the number of cascades and the nested key is the number of recursive iterations:\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 2.363, 'rmse': 4.352, '1px': 0.611, '3px': 0.828, '5px': 0.891, 'relepe': 0.176, 'fl-all': 64.511}\n\t\t5: {'mae': 1.618, 'rmse': 3.71, '1px': 0.761, '3px': 0.879, '5px': 0.918, 'relepe': 0.154, 'fl-all': 77.128}\n\t\t10: {'mae': 1.416, 'rmse': 3.53, '1px': 0.777, '3px': 0.896, '5px': 0.933, 'relepe': 0.148, 'fl-all': 78.388}\n\t\t20: {'mae': 1.448, 'rmse': 3.583, '1px': 0.771, '3px': 0.893, '5px': 0.931, 'relepe': 0.145, 'fl-all': 77.7}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.972, 'rmse': 4.125, '1px': 0.73, '3px': 0.865, '5px': 0.908, 'relepe': 0.169, 'fl-all': 74.396}\n\t\t5: {'mae': 1.403, 'rmse': 3.448, '1px': 0.793, '3px': 0.905, '5px': 0.937, 'relepe': 0.151, 'fl-all': 80.186}\n\t\t10: {'mae': 1.312, 'rmse': 3.368, '1px': 0.799, '3px': 0.912, '5px': 0.943, 'relepe': 0.148, 'fl-all': 80.379}\n\t\t20: {'mae': 1.376, 'rmse': 3.542, '1px': 0.796, '3px': 0.91, '5px': 0.942, 'relepe': 0.149, 'fl-all': 80.054}\n\t},\n}\n```\n\nYou can also evaluate the Finetuned weights:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.CRESTEREO_FINETUNE_MULTI_V1\n```\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 1.85, 'rmse': 3.797, '1px': 0.673, '3px': 0.862, '5px': 0.917, 'relepe': 0.171, 'fl-all': 69.736}\n\t\t5: {'mae': 1.111, 'rmse': 3.166, '1px': 0.838, '3px': 0.93, '5px': 0.957, 'relepe': 0.134, 'fl-all': 84.596}\n\t\t10: {'mae': 1.02, 'rmse': 3.073, '1px': 0.854, '3px': 0.938, '5px': 0.96, 'relepe': 0.129, 'fl-all': 86.042}\n\t\t20: {'mae': 0.993, 'rmse': 3.059, '1px': 0.855, '3px': 0.942, '5px': 0.967, 'relepe': 0.126, 'fl-all': 85.784}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.667, 'rmse': 3.867, '1px': 0.78, '3px': 0.891, '5px': 0.922, 'relepe': 0.165, 'fl-all': 78.89}\n\t\t5: {'mae': 1.158, 'rmse': 3.278, '1px': 0.843, '3px': 0.926, '5px': 0.955, 'relepe': 0.135, 'fl-all': 84.556}\n\t\t10: {'mae': 1.046, 'rmse': 3.13, '1px': 0.85, '3px': 0.934, '5px': 0.96, 'relepe': 0.13, 'fl-all': 85.464}\n\t\t20: {'mae': 1.021, 'rmse': 3.102, '1px': 0.85, '3px': 0.935, '5px': 0.963, 'relepe': 0.129, 'fl-all': 85.417}\n\t},\n}\n```\n\nEvaluating the author provided weights:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.MEGVII_V1\n```\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 1.704, 'rmse': 3.738, '1px': 0.738, '3px': 0.896, '5px': 0.933, 'relepe': 0.157, 'fl-all': 76.464}\n\t\t5: {'mae': 0.956, 'rmse': 2.963, '1px': 0.88, '3px': 0.948, '5px': 0.965, 'relepe': 0.124, 'fl-all': 88.186}\n\t\t10: {'mae': 0.792, 'rmse': 2.765, '1px': 0.905, '3px': 0.958, '5px': 0.97, 'relepe': 0.114, 'fl-all': 90.429}\n\t\t20: {'mae': 0.749, 'rmse': 2.706, '1px': 0.907, '3px': 0.961, '5px': 0.972, 'relepe': 0.113, 'fl-all': 90.807}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.702, 'rmse': 3.784, '1px': 0.784, '3px': 0.894, '5px': 0.924, 'relepe': 0.172, 'fl-all': 80.313}\n\t\t5: {'mae': 0.932, 'rmse': 2.907, '1px': 0.877, '3px': 0.944, '5px': 0.963, 'relepe': 0.125, 'fl-all': 87.979}\n\t\t10: {'mae': 0.773, 'rmse': 2.768, '1px': 0.901, '3px': 0.958, '5px': 0.972, 'relepe': 0.117, 'fl-all': 90.43}\n\t\t20: {'mae': 0.854, 'rmse': 2.971, '1px': 0.9, '3px': 0.957, '5px': 0.97, 'relepe': 0.122, 'fl-all': 90.269}\n\t},\n}\n```\n\n# Concerns when training\n\nWe encourage users to be aware of the **aspect-ratio** and **disparity scale** they are targeting when doing any sort of training or fine-tuning. The model is highly sensitive to these two factors, as a consequence of naive multi-set fine-tuning one can achieve `0.2 mae` relatively fast. We recommend that users pay close attention to how they **balance dataset sizing** when training such networks.\n\n Ideally, dataset scaling should be trated at an individual level and a thorough **EDA** of the disparity distribution in random crops at the desired training / inference size should be performed prior to any large compute investments.\n\n### Disparity scaling\n\n##### Sample A\n The top row contains a sample from `Sintel` whereas the bottom row one from `Middlebury`.\n\n![Disparity1](assets/disparity-domain-drift.jpg)\n\nFrom left to right (`left_image`, `right_image`, `valid_mask`, `valid_mask & ground_truth`, `prediction`). **Darker is further away, lighter is closer**. In the case of `Sintel` which is more closely aligned to the original distribution of `CREStereo` we notice that the model accurately predicts the background scale whereas in the case of `Middlebury2014` it cannot correctly estimate the continuous disparity. Notice that the frame composition is similar for both examples. The blue skybox in the `Sintel` scene behaves similarly to the `Middlebury` black background. However, because the `Middlebury` samples comes from an extremely large scene the crop size of `384x512` does not correctly capture the general training distribution.\n\n\n\n\n##### Sample B\n\nThe top row contains a scene from `Sceneflow` using the `Monkaa` split whilst the bottom row is a scene from `Middlebury`. This sample exhibits the same issues when it comes to **background estimation**. Given the exaggerated size of the `Middlebury` samples the model **colapses the smooth background** of the sample to what it considers to be a mean background disparity value.\n\n![Disparity2](assets/disparity-background-mode-collapse.jpg)\n\n\nFor more detail on why this behaviour occurs based on the training distribution proportions you can read more about the network at: https://github.com/pytorch/vision/pull/6629#discussion_r978160493\n\n\n### Metric overfitting\n\n##### Learning is critical in the beginning\n\nWe also advise users to make user of faster training schedules, as the performance gain over long periods time is marginal. Here we exhibit a difference between a faster decay schedule and later decay schedule.\n\n![Loss1](assets/Loss.jpg)\n\nIn **grey** we set the lr decay to begin after `30000` steps whilst in **orange** we opt for a very late learning rate decay at around `180000` steps. Although exhibiting stronger variance, we can notice that unfreezing the learning rate earlier whilst employing `gradient-norm` out-performs the default configuration.\n\n##### Gradient norm saves time\n\n![Loss2](assets/gradient-norm-removal.jpg)\n\nIn **grey** we keep ``gradient norm`` enabled whilst in **orange** we do not. We can notice that remvoing the gradient norm exacerbates the performance decrease in the early stages whilst also showcasing an almost complete collapse around the `60000` steps mark where we started decaying the lr for **orange**.\n\nAlthough both runs ahieve an improvement of about ``0.1`` mae after the lr decay start, the benefits of it are observable much faster when ``gradient norm`` is employed as the recovery period is no longer accounted for.\n"},"items":[{"name":"README.md","path":"references/classification/README.md","rawUrl":"https://raw.githubusercontent.com/pytorch/vision/HEAD/references/classification/README.md","title":"Classification Skill","category":"anthropic-skill","format":"markdown","content":"# Image classification reference training scripts\n\nThis folder contains reference training scripts for image classification.\nThey serve as a log of how to train specific models, as provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nExcept otherwise noted, all models have been trained on 8x V100 GPUs with \nthe following parameters:\n\n| Parameter                | value  |\n| ------------------------ | ------ |\n| `--batch_size`           | `32`   |\n| `--epochs`               | `90`   |\n| `--lr`                   | `0.1`  |\n| `--momentum`             | `0.9`  |\n| `--wd`, `--weight-decay` | `1e-4` |\n| `--lr-step-size`         | `30`   |\n| `--lr-gamma`             | `0.1`  |\n\n### AlexNet and VGG\n\nSince `AlexNet` and the original `VGG` architectures do not include batch \nnormalization, the default initial learning rate `--lr 0.1` is too high.\n\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model $MODEL --lr 1e-2\n```\n\nHere `$MODEL` is one of `alexnet`, `vgg11`, `vgg13`, `vgg16` or `vgg19`. Note\nthat `vgg11_bn`, `vgg13_bn`, `vgg16_bn`, and `vgg19_bn` include batch\nnormalization and thus are trained with the default parameters.\n\n### GoogLeNet\n\nThe weights of the GoogLeNet model are ported from the original paper rather than trained from scratch.\n\n### Inception V3\n\nThe weights of the Inception V3 model are ported from the original paper rather than trained from scratch.\n\nSince it expects tensors with a size of N x 3 x 299 x 299, to validate the model use the following command:\n\n```\ntorchrun --nproc_per_node=8 train.py --model inception_v3\\\n      --test-only --weights Inception_V3_Weights.IMAGENET1K_V1\n```\n\n### ResNet\n```\ntorchrun --nproc_per_node=8 train.py --model $MODEL\n```\n\nHere `$MODEL` is one of `resnet18`, `resnet34`, `resnet50`, `resnet101` or `resnet152`.\n\n### ResNext\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model $MODEL --epochs 100\n```\n\nHere `$MODEL` is one of `resnext50_32x4d` or `resnext101_32x8d`.\nNote that the above command corresponds to a single node with 8 GPUs. If you use\na different number of GPUs and/or a different batch size, then the learning rate\nshould be scaled accordingly. For example, the pretrained model provided by\n`torchvision` was trained on 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nwith `--batch_size 16` and `--lr 0.4`, instead of the current defaults\nwhich are respectively batch_size=32 and lr=0.1\n\n### MobileNetV2\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model mobilenet_v2 --epochs 300 --lr 0.045 --wd 0.00004\\\n     --lr-step-size 1 --lr-gamma 0.98\n```\n\n\n### MobileNetV3 Large & Small\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 600 --opt rmsprop --batch-size 128 --lr 0.064\\ \n     --wd 0.00001 --lr-step-size 2 --lr-gamma 0.973 --auto-augment imagenet --random-erase 0.2\n```\n\nHere `$MODEL` is one of `mobilenet_v3_large` or `mobilenet_v3_small`.\n\nThen we averaged the parameters of the last 3 checkpoints that improved the Acc@1. See [#3182](https://github.com/pytorch/vision/pull/3182) \nand [#3354](https://github.com/pytorch/vision/pull/3354) for details.\n\n\n### EfficientNet-V1\n\nThe weights of the B0-B4 variants are ported from Ross Wightman's [timm repo](https://github.com/rwightman/pytorch-image-models/blob/01cb46a9a50e3ba4be167965b5764e9702f09b30/timm/models/efficientnet.py#L95-L108).\n\nThe weights of the B5-B7 variants are ported from Luke Melas' [EfficientNet-PyTorch repo](https://github.com/lukemelas/EfficientNet-PyTorch/blob/1039e009545d9329ea026c9f7541341439712b96/efficientnet_pytorch/utils.py#L562-L564).\n\nAll models were trained using Bicubic interpolation and each have custom crop and resize sizes. To validate the models use the following commands:\n```\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b0 --test-only --weights EfficientNet_B0_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b1 --test-only --weights EfficientNet_B1_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b2 --test-only --weights EfficientNet_B2_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b3 --test-only --weights EfficientNet_B3_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b4 --test-only --weights EfficientNet_B4_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b5 --test-only --weights EfficientNet_B5_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b6 --test-only --weights EfficientNet_B6_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b7 --test-only --weights EfficientNet_B7_Weights.IMAGENET1K_V1\n```\n\n\n### EfficientNet-V2\n```\ntorchrun --nproc_per_node=8 train.py \\\n--model $MODEL --batch-size 128 --lr 0.5 --lr-scheduler cosineannealinglr \\\n--lr-warmup-epochs 5 --lr-warmup-method linear --auto-augment ta_wide --epochs 600 --random-erase 0.1 \\\n--label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --weight-decay 0.00002 --norm-weight-decay 0.0 \\\n--train-crop-size $TRAIN_SIZE --model-ema --val-crop-size $EVAL_SIZE --val-resize-size $EVAL_SIZE \\\n--ra-sampler --ra-reps 4\n```\nHere `$MODEL` is one of `efficientnet_v2_s` and `efficientnet_v2_m`. \nNote that the Small variant had a `$TRAIN_SIZE` of `300` and a `$EVAL_SIZE` of `384`, while the Medium `384` and `480` respectively.\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 4 nodes, each with 8 GPUs (for a total of 32 GPUs),\nand `--batch_size 32`.\n\nThe weights of the Large variant are ported from the original paper rather than trained from scratch. See the `EfficientNet_V2_L_Weights` entry for their exact preprocessing transforms.\n\n\n### RegNet\n\n#### Small models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 128 --wd 0.00005 --lr=0.8\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_400mf`, `regnet_x_800mf`, `regnet_x_1_6gf`, `regnet_y_400mf`, `regnet_y_800mf` and `regnet_y_1_6gf`. Please note we used learning rate 0.4 for `regent_y_400mf` to get the same Acc@1 as [the paper](https://arxiv.org/abs/2003.13678).\n\n#### Medium models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 64 --wd 0.00005 --lr=0.4\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_3_2gf`, `regnet_x_8gf`, `regnet_x_16gf`, `regnet_y_3_2gf` and `regnet_y_8gf`.\n\n#### Large models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 32 --wd 0.00005 --lr=0.2\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_32gf`, `regnet_y_16gf` and `regnet_y_32gf`.\n\n### Vision Transformer\n\n#### vit_b_16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_b_16 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nand `--batch_size 64`.\n\n#### vit_b_32\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_b_32 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment imagenet\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 256`.\n\n#### vit_l_16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_l_16 --epochs 600 --batch-size 128 --lr 0.5 --lr-scheduler cosineannealinglr\\\n    --lr-warmup-method linear --lr-warmup-epochs 5 --label-smoothing 0.1 --mixup-alpha 0.2\\\n    --auto-augment ta_wide --random-erase 0.1 --weight-decay 0.00002 --norm-weight-decay 0.0\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema --val-resize-size 232\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 64`.\n\n#### vit_l_32\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_l_32 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nand `--batch_size 64`.\n\n\n### ConvNeXt\n```\ntorchrun --nproc_per_node=8 train.py\\ \n--model $MODEL --batch-size 128 --opt adamw --lr 1e-3 --lr-scheduler cosineannealinglr \\ \n--lr-warmup-epochs 5 --lr-warmup-method linear --auto-augment ta_wide --epochs 600 --random-erase 0.1 \\ \n--label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --weight-decay 0.05 --norm-weight-decay 0.0 \\\n--train-crop-size 176 --model-ema --val-resize-size 232 --ra-sampler --ra-reps 4\n```\nHere `$MODEL` is one of `convnext_tiny`, `convnext_small`, `convnext_base` and `convnext_large`. Note that each variant had its `--val-resize-size` optimized in a post-training step, see their `Weights` entry for their exact value.\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 64`.\n\n\n### SwinTransformer\n```\ntorchrun --nproc_per_node=8 train.py\\ \n--model $MODEL --epochs 300 --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler --ra-reps 4  --val-resize-size 224\n```\nHere `$MODEL` is one of `swin_t`, `swin_s` or `swin_b`.\nNote that `--val-resize-size` was optimized in a post-training step, see their `Weights` entry for the exact value.\n\n\n\n\n### SwinTransformer V2\n```\ntorchrun --nproc_per_node=8 train.py\\\n--model $MODEL --epochs 300 --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler --ra-reps 4  --val-resize-size 256 --val-crop-size 256 --train-crop-size 256 \n```\nHere `$MODEL` is one of `swin_v2_t`, `swin_v2_s` or `swin_v2_b`.\nNote that `--val-resize-size` was optimized in a post-training step, see their `Weights` entry for the exact value.\n\n\n### MaxViT\n```\ntorchrun --nproc_per_node=8 --n_nodes=4 train.py\\\n--model $MODEL --epochs 400 --batch-size 128 --opt adamw --lr 3e-3 --weight-decay 0.05 --lr-scheduler cosineannealinglr --lr-min 1e-5 --lr-warmup-method linear  --lr-warmup-epochs 32  --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 1.0 --interpolation bicubic --auto-augment ta_wide --policy-magnitude 15 --model-ema --val-resize-size 224\\\n--val-crop-size 224 --train-crop-size 224 --amp  --model-ema-steps 32 --transformer-embedding-decay 0 --sync-bn\n```\nHere `$MODEL` is `maxvit_t`.\nNote that `--val-resize-size` was not optimized in a post-training step.\n\n\n### ShuffleNet V2\n```\ntorchrun --nproc_per_node=8 train.py \\\n--batch-size=128 \\\n--lr=0.5 --lr-scheduler=cosineannealinglr --lr-warmup-epochs=5 --lr-warmup-method=linear \\\n--auto-augment=ta_wide --epochs=600 --random-erase=0.1 --weight-decay=0.00002 \\\n--norm-weight-decay=0.0 --label-smoothing=0.1 --mixup-alpha=0.2 --cutmix-alpha=1.0 \\\n--train-crop-size=176 --model-ema --val-resize-size=232 --ra-sampler --ra-reps=4\n```\nHere `$MODEL` is either `shufflenet_v2_x1_5` or `shufflenet_v2_x2_0`.\n\nThe models `shufflenet_v2_x0_5` and `shufflenet_v2_x1_0` were contributed by the community. See [PR-849](https://github.com/pytorch/vision/pull/849#issuecomment-483391686) for details.\n\n\n## Mixed precision training\nAutomatic Mixed Precision (AMP) training on GPU for Pytorch can be enabled with the [torch.cuda.amp](https://pytorch.org/docs/stable/amp.html?highlight=amp#module-torch.cuda.amp).\n\nMixed precision training makes use of both FP32 and FP16 precisions where appropriate. FP16 operations can leverage the Tensor cores on NVIDIA GPUs (Volta, Turing or newer architectures) for improved throughput, generally without loss in model accuracy. Mixed precision training also often allows larger batch sizes. GPU automatic mixed precision training for Pytorch Vision can be enabled via the flag value `--amp=True`.\n\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model resnext50_32x4d --epochs 100 --amp\n```\n\n## Quantized\n\n### Post training quantized models\n\nFor all post training quantized models, the settings are:\n\n1. num_calibration_batches: 32\n2. num_workers: 16\n3. batch_size: 32\n4. eval_batch_size: 128\n5. qbackend: 'fbgemm'\n\n```\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' --model='$MODEL'\n```\nHere `$MODEL` is one of `googlenet`, `inception_v3`, `resnet18`, `resnet50`, `resnext101_32x8d`, `shufflenet_v2_x0_5` and `shufflenet_v2_x1_0`.\n\n### Quantized ShuffleNet V2\n\nHere are commands that we use to quantize the `shufflenet_v2_x1_5` and `shufflenet_v2_x2_0` models.\n```\n# For shufflenet_v2_x1_5\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' \\\n    --model=shufflenet_v2_x1_5 --weights=\"ShuffleNet_V2_X1_5_Weights.IMAGENET1K_V1\" \\\n    --train-crop-size 176 --val-resize-size 232 --data-path /datasets01_ontap/imagenet_full_size/061417/\n\n# For shufflenet_v2_x2_0\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' \\\n    --model=shufflenet_v2_x2_0 --weights=\"ShuffleNet_V2_X2_0_Weights.IMAGENET1K_V1\" \\\n    --train-crop-size 176 --val-resize-size 232 --data-path /datasets01_ontap/imagenet_full_size/061417/\n```\n\n### QAT MobileNetV2\n\nFor Mobilenet-v2, the model was trained with quantization aware training, the settings used are:\n1. num_workers: 16\n2. batch_size: 32\n3. eval_batch_size: 128\n4. qbackend: 'qnnpack'\n5. learning-rate: 0.0001\n6. num_epochs: 90\n7. num_observer_update_epochs:4\n8. num_batch_norm_update_epochs:3\n9. momentum: 0.9\n10. lr_step_size:30\n11. lr_gamma: 0.1\n12. weight-decay: 0.0001\n\n```\ntorchrun --nproc_per_node=8 train_quantization.py --model='mobilenet_v2'\n```\n\nTraining converges at about 10 epochs.\n\n### QAT MobileNetV3\n\nFor Mobilenet-v3 Large, the model was trained with quantization aware training, the settings used are:\n1. num_workers: 16\n2. batch_size: 32\n3. eval_batch_size: 128\n4. qbackend: 'qnnpack'\n5. learning-rate: 0.001\n6. num_epochs: 90\n7. num_observer_update_epochs:4\n8. num_batch_norm_update_epochs:3\n9. momentum: 0.9\n10. lr_step_size:30\n11. lr_gamma: 0.1\n12. weight-decay: 0.00001\n\n```\ntorchrun --nproc_per_node=8 train_quantization.py --model='mobilenet_v3_large' \\\n    --wd 0.00001 --lr 0.001\n```\n\nFor post training quant, device is set to CPU. For training, the device is set to CUDA.\n\n### Command to evaluate quantized models using the pre-trained weights:\n\n```\npython train_quantization.py --device='cpu' --test-only --qbackend='<qbackend>' --model='<model_name>'\n```\n\nFor inception_v3 you need to pass the following extra parameters:\n```\n--val-resize-size 342 --val-crop-size 299 --train-crop-size 299\n```\n","isInternal":false,"tokens":5359,"sizeBytes":16653},{"name":"README.md","path":"references/depth/stereo/README.md","rawUrl":"https://raw.githubusercontent.com/pytorch/vision/HEAD/references/depth/stereo/README.md","title":"Stereo Skill","category":"anthropic-skill","format":"markdown","content":"# Stereo Matching reference training scripts\n\nThis folder contains reference training scripts for Stereo Matching.\nThey serve as a log of how to train specific models, so as to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nNote: this depends on the `prototype` folder which was removed, the latest\ncommit with the prototype folder is 8b6ba92ce5c48d6db596f6d415a90c128bc5f77b.\n\n\n### CREStereo\n\nThe CREStereo model was trained on a dataset mixture between **CREStereo**, **ETH3D** and the additional split from **Middlebury2014**.\nA ratio of **88-6-6** was used in order to train a baseline weight set. We provide multi-set variant as well.\nBoth used 8 A100 GPUs and a batch size of 2 (so effective batch size is 16). The\nrest of the hyper-parameters loosely follow the recipe from https://github.com/megvii-research/CREStereo.\nThe original recipe trains for **300000** updates (or steps) on the dataset mixture. We modify the learning rate\nschedule to one that starts decaying the weight much sooner. Throughout the experiments we found that this reduces \noverfitting during evaluation time and gradient clip help stabilize the loss during a pre-mature learning rate change.\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_cre \\\n    --model crestereo_base \\\n    --train-datasets crestereo eth3d-train middlebury2014-other \\\n    --dataset-steps 264000 18000 18000\n    --batch-size 2 \\\n    --lr 0.0004 \\\n    --min-lr 0.00002 \\\n    --lr-decay-method cosine \\\n    --warmup-steps 6000 \\\n    --decay-after-steps 30000 \\\n    --clip-grad-norm 1.0 \\\n```\n\nWe employ a multi-set fine-tuning stage where we uniformly sample from multiple datasets. Given hat some of these datasets have extremely large images (``2048x2048`` or more) we opt for a very aggressive scale-range ``[0.2 - 0.8]`` such that as much of the original frame composition is captured inside the ``384x512`` crop.\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_things \\\n    --model crestereo_base \\\n    --train-datasets crestereo eth3d-train middlebury2014-other instereo2k fallingthings carla-highres sintel sceneflow-monkaa sceneflow-driving \\\n    --dataset-steps 12000 12000 12000 12000 12000 12000 12000 12000 12000\n    --batch-size 2 \\\n    --scale-range 0.2 0.8 \\\n    --lr 0.0004 \\\n    --lr-decay-method cosine \\\n    --decay-after-steps 0 \\\n    --warmup-steps 0 \\\n    --min-lr 0.00002 \\\n    --resume-path $checkpoint_dir/$name_cre.pth\n```\n\n\n### Evaluation\n\nEvaluating the base weights\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.CRESTEREO_ETH_MBL_V1\n```\n\nThis should give an **mae of about 1.416** on the train set of `Middlebury2014`. Results may vary slightly depending on the batch size and the number of GPUs. For the most accurate results use 1 GPU and `--batch-size 1`. The created log file should look like this, where the first key is the number of cascades and the nested key is the number of recursive iterations:\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 2.363, 'rmse': 4.352, '1px': 0.611, '3px': 0.828, '5px': 0.891, 'relepe': 0.176, 'fl-all': 64.511}\n\t\t5: {'mae': 1.618, 'rmse': 3.71, '1px': 0.761, '3px': 0.879, '5px': 0.918, 'relepe': 0.154, 'fl-all': 77.128}\n\t\t10: {'mae': 1.416, 'rmse': 3.53, '1px': 0.777, '3px': 0.896, '5px': 0.933, 'relepe': 0.148, 'fl-all': 78.388}\n\t\t20: {'mae': 1.448, 'rmse': 3.583, '1px': 0.771, '3px': 0.893, '5px': 0.931, 'relepe': 0.145, 'fl-all': 77.7}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.972, 'rmse': 4.125, '1px': 0.73, '3px': 0.865, '5px': 0.908, 'relepe': 0.169, 'fl-all': 74.396}\n\t\t5: {'mae': 1.403, 'rmse': 3.448, '1px': 0.793, '3px': 0.905, '5px': 0.937, 'relepe': 0.151, 'fl-all': 80.186}\n\t\t10: {'mae': 1.312, 'rmse': 3.368, '1px': 0.799, '3px': 0.912, '5px': 0.943, 'relepe': 0.148, 'fl-all': 80.379}\n\t\t20: {'mae': 1.376, 'rmse': 3.542, '1px': 0.796, '3px': 0.91, '5px': 0.942, 'relepe': 0.149, 'fl-all': 80.054}\n\t},\n}\n```\n\nYou can also evaluate the Finetuned weights:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.CRESTEREO_FINETUNE_MULTI_V1\n```\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 1.85, 'rmse': 3.797, '1px': 0.673, '3px': 0.862, '5px': 0.917, 'relepe': 0.171, 'fl-all': 69.736}\n\t\t5: {'mae': 1.111, 'rmse': 3.166, '1px': 0.838, '3px': 0.93, '5px': 0.957, 'relepe': 0.134, 'fl-all': 84.596}\n\t\t10: {'mae': 1.02, 'rmse': 3.073, '1px': 0.854, '3px': 0.938, '5px': 0.96, 'relepe': 0.129, 'fl-all': 86.042}\n\t\t20: {'mae': 0.993, 'rmse': 3.059, '1px': 0.855, '3px': 0.942, '5px': 0.967, 'relepe': 0.126, 'fl-all': 85.784}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.667, 'rmse': 3.867, '1px': 0.78, '3px': 0.891, '5px': 0.922, 'relepe': 0.165, 'fl-all': 78.89}\n\t\t5: {'mae': 1.158, 'rmse': 3.278, '1px': 0.843, '3px': 0.926, '5px': 0.955, 'relepe': 0.135, 'fl-all': 84.556}\n\t\t10: {'mae': 1.046, 'rmse': 3.13, '1px': 0.85, '3px': 0.934, '5px': 0.96, 'relepe': 0.13, 'fl-all': 85.464}\n\t\t20: {'mae': 1.021, 'rmse': 3.102, '1px': 0.85, '3px': 0.935, '5px': 0.963, 'relepe': 0.129, 'fl-all': 85.417}\n\t},\n}\n```\n\nEvaluating the author provided weights:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.MEGVII_V1\n```\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 1.704, 'rmse': 3.738, '1px': 0.738, '3px': 0.896, '5px': 0.933, 'relepe': 0.157, 'fl-all': 76.464}\n\t\t5: {'mae': 0.956, 'rmse': 2.963, '1px': 0.88, '3px': 0.948, '5px': 0.965, 'relepe': 0.124, 'fl-all': 88.186}\n\t\t10: {'mae': 0.792, 'rmse': 2.765, '1px': 0.905, '3px': 0.958, '5px': 0.97, 'relepe': 0.114, 'fl-all': 90.429}\n\t\t20: {'mae': 0.749, 'rmse': 2.706, '1px': 0.907, '3px': 0.961, '5px': 0.972, 'relepe': 0.113, 'fl-all': 90.807}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.702, 'rmse': 3.784, '1px': 0.784, '3px': 0.894, '5px': 0.924, 'relepe': 0.172, 'fl-all': 80.313}\n\t\t5: {'mae': 0.932, 'rmse': 2.907, '1px': 0.877, '3px': 0.944, '5px': 0.963, 'relepe': 0.125, 'fl-all': 87.979}\n\t\t10: {'mae': 0.773, 'rmse': 2.768, '1px': 0.901, '3px': 0.958, '5px': 0.972, 'relepe': 0.117, 'fl-all': 90.43}\n\t\t20: {'mae': 0.854, 'rmse': 2.971, '1px': 0.9, '3px': 0.957, '5px': 0.97, 'relepe': 0.122, 'fl-all': 90.269}\n\t},\n}\n```\n\n# Concerns when training\n\nWe encourage users to be aware of the **aspect-ratio** and **disparity scale** they are targeting when doing any sort of training or fine-tuning. The model is highly sensitive to these two factors, as a consequence of naive multi-set fine-tuning one can achieve `0.2 mae` relatively fast. We recommend that users pay close attention to how they **balance dataset sizing** when training such networks.\n\n Ideally, dataset scaling should be trated at an individual level and a thorough **EDA** of the disparity distribution in random crops at the desired training / inference size should be performed prior to any large compute investments.\n\n### Disparity scaling\n\n##### Sample A\n The top row contains a sample from `Sintel` whereas the bottom row one from `Middlebury`.\n\n![Disparity1](assets/disparity-domain-drift.jpg)\n\nFrom left to right (`left_image`, `right_image`, `valid_mask`, `valid_mask & ground_truth`, `prediction`). **Darker is further away, lighter is closer**. In the case of `Sintel` which is more closely aligned to the original distribution of `CREStereo` we notice that the model accurately predicts the background scale whereas in the case of `Middlebury2014` it cannot correctly estimate the continuous disparity. Notice that the frame composition is similar for both examples. The blue skybox in the `Sintel` scene behaves similarly to the `Middlebury` black background. However, because the `Middlebury` samples comes from an extremely large scene the crop size of `384x512` does not correctly capture the general training distribution.\n\n\n\n\n##### Sample B\n\nThe top row contains a scene from `Sceneflow` using the `Monkaa` split whilst the bottom row is a scene from `Middlebury`. This sample exhibits the same issues when it comes to **background estimation**. Given the exaggerated size of the `Middlebury` samples the model **colapses the smooth background** of the sample to what it considers to be a mean background disparity value.\n\n![Disparity2](assets/disparity-background-mode-collapse.jpg)\n\n\nFor more detail on why this behaviour occurs based on the training distribution proportions you can read more about the network at: https://github.com/pytorch/vision/pull/6629#discussion_r978160493\n\n\n### Metric overfitting\n\n##### Learning is critical in the beginning\n\nWe also advise users to make user of faster training schedules, as the performance gain over long periods time is marginal. Here we exhibit a difference between a faster decay schedule and later decay schedule.\n\n![Loss1](assets/Loss.jpg)\n\nIn **grey** we set the lr decay to begin after `30000` steps whilst in **orange** we opt for a very late learning rate decay at around `180000` steps. Although exhibiting stronger variance, we can notice that unfreezing the learning rate earlier whilst employing `gradient-norm` out-performs the default configuration.\n\n##### Gradient norm saves time\n\n![Loss2](assets/gradient-norm-removal.jpg)\n\nIn **grey** we keep ``gradient norm`` enabled whilst in **orange** we do not. We can notice that remvoing the gradient norm exacerbates the performance decrease in the early stages whilst also showcasing an almost complete collapse around the `60000` steps mark where we started decaying the lr for **orange**.\n\nAlthough both runs ahieve an improvement of about ``0.1`` mae after the lr decay start, the benefits of it are observable much faster when ``gradient norm`` is employed as the recovery period is no longer accounted for.\n","isInternal":false,"tokens":3490,"sizeBytes":10138},{"name":"README.md","path":"references/detection/README.md","rawUrl":"https://raw.githubusercontent.com/pytorch/vision/HEAD/references/detection/README.md","title":"Detection Skill","category":"anthropic-skill","format":"markdown","content":"# Object detection reference training scripts\n\nThis folder contains reference training scripts for object detection.\nThey serve as a log of how to train specific models, to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nTo execute the example commands below you must install the following:\n\n```\ncython\npycocotools\nmatplotlib\n```\n\nYou must modify the following flags:\n\n`--data-path=/path/to/coco/dataset`\n\n`--nproc_per_node=<number_of_gpus_available>`\n\nExcept otherwise noted, all models have been trained on 8x V100 GPUs. \n\n### Faster R-CNN ResNet-50 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### Faster R-CNN MobileNetV3-Large FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_mobilenet_v3_large_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n### Faster R-CNN MobileNetV3-Large 320 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_mobilenet_v3_large_320_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n### FCOS ResNet-50 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fcos_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3  --lr 0.01 --amp --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### RetinaNet\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model retinanet_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --lr 0.01 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### SSD300 VGG16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model ssd300_vgg16 --epochs 120\\\n    --lr-steps 80 110 --aspect-ratio-group-factor 3 --lr 0.002 --batch-size 4\\\n    --weight-decay 0.0005 --data-augmentation ssd --weights-backbone VGG16_Weights.IMAGENET1K_FEATURES\n```\n\n### SSDlite320 MobileNetV3-Large\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model ssdlite320_mobilenet_v3_large --epochs 660\\\n    --aspect-ratio-group-factor 3 --lr-scheduler cosineannealinglr --lr 0.15 --batch-size 24\\\n    --weight-decay 0.00004 --data-augmentation ssdlite\n```\n\n\n### Mask R-CNN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model maskrcnn_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n\n### Keypoint R-CNN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco_kp --model keypointrcnn_resnet50_fpn --epochs 46\\\n    --lr-steps 36 43 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n","isInternal":false,"tokens":918,"sizeBytes":2894},{"name":"README.md","path":"references/optical_flow/README.md","rawUrl":"https://raw.githubusercontent.com/pytorch/vision/HEAD/references/optical_flow/README.md","title":"Optical_flow Skill","category":"anthropic-skill","format":"markdown","content":"# Optical flow reference training scripts\n\nThis folder contains reference training scripts for optical flow.\nThey serve as a log of how to train specific models, so as to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\n\n### RAFT Large\n\nThe RAFT large model was trained on Flying Chairs and then on Flying Things.\nBoth used 8 A100 GPUs and a batch size of 2 (so effective batch size is 16). The\nrest of the hyper-parameters are exactly the same as the original RAFT training\nrecipe from https://github.com/princeton-vl/RAFT. The original recipe trains for\n100000 updates (or steps) on each dataset - this corresponds to about 72 and 20\nepochs on Chairs and Things respectively:\n\n```\nnum_epochs = ceil(num_steps / number_of_steps_per_epoch)\n           = ceil(num_steps / (num_samples / effective_batch_size))\n```\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_chairs \\\n    --model raft_large \\\n    --train-dataset chairs \\\n    --batch-size 2 \\\n    --lr 0.0004 \\\n    --weight-decay 0.0001 \\\n    --epochs 72 \\\n    --output-dir $chairs_dir\n```\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_things \\\n    --model raft_large \\\n    --train-dataset things \\\n    --batch-size 2 \\\n    --lr 0.000125 \\\n    --weight-decay 0.0001 \\\n    --epochs 20 \\\n    --freeze-batch-norm \\\n    --output-dir $things_dir\\\n    --resume $chairs_dir/$name_chairs.pth\n```\n\n\n### Evaluation\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 train.py --val-dataset sintel --batch-size 1 --dataset-root $dataset_root --model raft_large --weights Raft_Large_Weights.C_T_SKHT_V2\n```\n\nThis should give an epe of about 1.3822 on the clean pass and 2.7161 on the\nfinal pass of Sintel-train. Results may vary slightly depending on the batch\nsize and the number of GPUs. For the most accurate results use 1 GPU and\n`--batch-size 1`:\n\n```\nSintel val clean epe: 1.3822\t1px: 0.9028\t3px: 0.9573\t5px: 0.9697\tper_image_epe: 1.3822\tf1: 4.0248\nSintel val final epe: 2.7161\t1px: 0.8528\t3px: 0.9204\t5px: 0.9392\tper_image_epe: 2.7161\tf1: 7.5964\n```\n\nYou can also evaluate on Kitti train:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 train.py --val-dataset kitti --batch-size 1 --dataset-root $dataset_root --model raft_large --weights Raft_Large_Weights.C_T_SKHT_V2\nKitti val epe: 4.7968\t1px: 0.6388\t3px: 0.8197\t5px: 0.8661\tper_image_epe: 4.5118\tf1: 16.0679\n```\n","isInternal":false,"tokens":764,"sizeBytes":2452},{"name":"README.md","path":"references/segmentation/README.md","rawUrl":"https://raw.githubusercontent.com/pytorch/vision/HEAD/references/segmentation/README.md","title":"Segmentation Skill","category":"anthropic-skill","format":"markdown","content":"# Semantic segmentation reference training scripts\n\nThis folder contains reference training scripts for semantic segmentation.\nThey serve as a log of how to train specific models and provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nAll models have been trained on 8x V100 GPUs.\n\nYou must modify the following flags:\n\n`--data-path=/path/to/dataset`\n\n`--nproc_per_node=<number_of_gpus_available>`\n\n## fcn_resnet50\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model fcn_resnet50 --aux-loss --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n## fcn_resnet101\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model fcn_resnet101 --aux-loss --weights-backbone ResNet101_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_resnet50\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model deeplabv3_resnet50 --aux-loss --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_resnet101\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model deeplabv3_resnet101 --aux-loss --weights-backbone ResNet101_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_mobilenet_v3_large\n```\ntorchrun --nproc_per_node=8 train.py --dataset coco -b 4 --model deeplabv3_mobilenet_v3_large --aux-loss --wd 0.000001 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n## lraspp_mobilenet_v3_large\n```\ntorchrun --nproc_per_node=8 train.py --dataset coco -b 4 --model lraspp_mobilenet_v3_large --wd 0.000001 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n","isInternal":false,"tokens":500,"sizeBytes":1576},{"name":"README.md","path":"references/similarity/README.md","rawUrl":"https://raw.githubusercontent.com/pytorch/vision/HEAD/references/similarity/README.md","title":"Similarity Skill","category":"anthropic-skill","format":"markdown","content":"# Similarity Learning Using Triplet Loss #\n\nIn this reference, we use triplet loss to learn embeddings which can be used to differentiate images. This learning technique was popularized by [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/abs/1503.03832) and has been quite effective in learning embeddings to differentiate between faces.\n\nThis reference can be directly applied to the following use cases:\n\n* You have an unknown number of classes and would like to train a model to learn how to differentiate between them.\n* You want to train a model to learn a distance-based metric between samples. For example, learning a distance-based similarity measure between faces.\n\n### Training ###\nBy default, the training script trains ResNet50 on the FashionMNIST Dataset to learn image embeddings which can be used to differentiate between images by measuring the euclidean distance between embeddings. This can be changed as per your requirements.\n\nImage embeddings of the same class should be 'close' to each other, while image embeddings between different classes should be 'far' away.\n\nTo run the training script:\n\n```bash\npython train.py -h    # Lists all optional arguments\npython train.py \t\t\t# Runs training script with default args\n```\n\nRunning the training script as is should yield 97% accuracy on the FMNIST test set within 10 epochs.\n\n### Loss ###\n`TripletMarginLoss` is a loss function which takes in a triplet of samples. A valid triplet has an:\n\n1. Anchor: a sample from the dataset\n2. Positive: another sample with the same label/group as the anchor (Generally, positive != anchor)\n3. Negative: a sample with a different label/group from the anchor\n\n`TripletMarginLoss` (refer to `loss.py`) does the following:\n\n```python\nloss = max(dist(anchor, positive) - dist(anchor, negative) + margin, 0)\n```\nWhere `dist` is a distance function. Minimizing this function effectively leads to minimizing `dist(anchor, positive)` and maximizing `dist(anchor, negative)`.\n\nThe FaceNet paper describe this loss in more detail.\n\n### Sampler ###\n\nIn order to generate valid triplets from a batch of samples, we need to make sure that each batch has multiple samples with the same label. We do this using `PKSampler` (refer to `sampler.py`), which ensures that each batch of size `p * k` will have samples from exactly `p` classes and `k` samples per class.\n\n### Triplet Mining ###\n\n`TripletMarginLoss` currently supports the following mining techniques:\n\n* `batch_all`: Generates all possible triplets from a batch and excludes the triplets which are 'easy' (which have `loss = 0`) before passing it through the loss function.\n* `batch_hard`: For every anchor, `batch_hard` creates a triplet with the 'hardest' positive (farthest positive) and negative (closest negative).\n\nThese mining strategies usually speed up training.\n\nThis [webpage](https://omoindrot.github.io/triplet-loss) describes the sampling and mining strategies in more detail. \n","isInternal":false,"tokens":644,"sizeBytes":2977},{"name":"README.md","path":"references/video_classification/README.md","rawUrl":"https://raw.githubusercontent.com/pytorch/vision/HEAD/references/video_classification/README.md","title":"Video_classification Skill","category":"anthropic-skill","format":"markdown","content":"# Video Classification\n\nWe present a simple training script that can be used for replicating the result of [resenet-based video models](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf). All models are trained on [Kinetics400 dataset](https://deepmind.com/research/open-source/kinetics), a benchmark dataset for human-action recognition. The accuracy is reported on the traditional validation split.\n\n## Data preparation\n\nIf you already have downloaded [Kinetics400 dataset](https://deepmind.com/research/open-source/kinetics), \nplease proceed directly to the next section.\n\nTo download videos, one can use https://github.com/Showmax/kinetics-downloader. Please note that the dataset can take up upwards of 400GB, depending on the quality setting during download.\n\n## Training\n\nWe assume the training and validation AVI videos are stored at `/data/kinectics400/train` and \n`/data/kinectics400/val`. For training we suggest starting with the hyperparameters reported in the [paper](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf), in order to match the performance of said models. Clip sampling strategy is a particularly important parameter during training, and we suggest using random temporal jittering during training - in other words sampling multiple training clips from each video with random start times during at every epoch. This functionality is built into our training script, and optimal hyperparameters are set by default.  \n\n### Multiple GPUs\n\nRun the training on a single node with 8 GPUs:\n```bash\ntorchrun --nproc_per_node=8 train.py --data-path=/data/kinectics400 --kinetics-version=\"400\" --lr 0.08 --cache-dataset --sync-bn --amp \n```\n\n**Note:** all our models were trained on 8 nodes with 8 V100 GPUs each for a total of 64 GPUs. Expected training time for 64 GPUs is 24 hours, depending on the storage solution.\n**Note 2:** hyperparameters for exact replication of our training can be found on the section below. Some hyperparameters such as learning rate must be scaled linearly in proportion to the number of GPUs. The default values assume 64 GPUs.\n\n### Single GPU \n\n**Note:** training on a single gpu can be extremely slow. \n\n\n```bash\npython train.py --data-path=/data/kinectics400 --kinetics-version=\"400\" --batch-size=8 --cache-dataset\n```\n\n\n### Additional Kinetics versions\n\nSince the original release, additional versions of Kinetics dataset became available (Kinetics 600).\nOur training scripts support these versions of dataset as well by setting the `--kinetics-version` parameter to `\"600\"`.\n\n**Note:** training on Kinetics 600 requires a different set of hyperparameters for optimal performance. We do not provide Kinetics 600 pretrained models.\n\n\n## Video classification models\n\nStarting with version `0.4.0` we have introduced support for basic video tasks and video classification modelling.\nFor more information about the available models check [here](https://pytorch.org/docs/stable/torchvision/models.html#video-classification). \n\n### Video ResNet models\n\nSee reference training script [here](https://github.com/pytorch/vision/blob/main/references/video_classification/train.py):\n\n- input space: RGB\n- resize size: [128, 171]\n- crop size: [112, 112]\n- mean: [0.43216, 0.394666, 0.37645]\n- std: [0.22803, 0.22145, 0.216989]\n- number of classes: 400\n\nInput data augmentations at training time (with optional parameters):\n\n1. ConvertImageDtype\n2. Resize (resize size value above)\n3. Random horizontal flip (0.5)\n4. Normalization (mean, std, see values above)\n5. Random Crop (crop size value above)\n6. Convert BCHW to CBHW\n\nInput data augmentations at validation time (with optional parameters):\n\n1. ConvertImageDtype\n2. Resize (resize size value above)\n3. Normalization (mean, std, see values above)\n4. Center Crop (crop size value above)\n5. Convert BCHW to CBHW\n\nThis translates in the following set of command-line arguments. Please note that `--batch-size` parameter controls the\nbatch size per GPU. Moreover, note that our default `--lr` is configured for 64 GPUs which is how many we used for the \nVideo resnet models:\n```\n# number of frames per clip\n--clip_len 16 \\ \n--frame-rate 15 \\\n# allow for temporal jittering\n--clips_per_video 5 \\\n--batch-size 24 \\\n--epochs 45 \\\n--lr 0.64 \\\n# we use 10 epochs for linear warmup\n--lr-warmup-epochs 10 \\\n# learning rate is decayed at 20, 30, and 40 epoch by a factor of 10\n--lr-milestones 20, 30, 40 \\\n--lr-gamma 0.1 \\\n--train-resize-size 128 171 \\\n--train-crop-size 112 112 \\\n--val-resize-size 128 171 \\\n--val-crop-size 112 112\n```\n\n### S3D\n\nThe S3D model was trained similarly to the above but with the following changes on the default configuration:\n```\n--batch-size=12 --lr 0.2 --clip-len 64 --clips-per-video 5 --sync-bn \\\n--train-resize-size 256 256 --train-crop-size 224 224 --val-resize-size 256 256 --val-crop-size 224 224\n```\n\nWe used 64 GPUs to train the architecture. \n\nTo estimate the validation statistics of the model, we run the reference script with the following configuration:\n```\n--batch-size=16 --test-only --clip-len 128 --clips-per-video 1 \n```\n\n### Additional video modelling resources\n\n- [Video Model Zoo](https://github.com/facebookresearch/VMZ)\n- [PySlowFast](https://github.com/facebookresearch/SlowFast)\n\n### References\n\n[0] _D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri_: A Closer Look at Spatiotemporal Convolutions for Action Recognition. _CVPR 2018_ ([paper](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf))\n\n[1] _W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, A. Zisserman_: The Kinetics Human Action Video Dataset ([paper](https://arxiv.org/abs/1705.06950))\n","isInternal":false,"tokens":1521,"sizeBytes":5901}],"systemPromptSnippet":"<yakaai_skills repo=\"pytorch/vision\">\n<!-- File: references/classification/README.md (Tokens: ~5359 | Category: anthropic-skill) -->\n# Image classification reference training scripts\n\nThis folder contains reference training scripts for image classification.\nThey serve as a log of how to train specific models, as provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nExcept otherwise noted, all models have been trained on 8x V100 GPUs with \nthe following parameters:\n\n| Parameter                | value  |\n| ------------------------ | ------ |\n| `--batch_size`           | `32`   |\n| `--epochs`               | `90`   |\n| `--lr`                   | `0.1`  |\n| `--momentum`             | `0.9`  |\n| `--wd`, `--weight-decay` | `1e-4` |\n| `--lr-step-size`         | `30`   |\n| `--lr-gamma`             | `0.1`  |\n\n### AlexNet and VGG\n\nSince `AlexNet` and the original `VGG` architectures do not include batch \nnormalization, the default initial learning rate `--lr 0.1` is too high.\n\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model $MODEL --lr 1e-2\n```\n\nHere `$MODEL` is one of `alexnet`, `vgg11`, `vgg13`, `vgg16` or `vgg19`. Note\nthat `vgg11_bn`, `vgg13_bn`, `vgg16_bn`, and `vgg19_bn` include batch\nnormalization and thus are trained with the default parameters.\n\n### GoogLeNet\n\nThe weights of the GoogLeNet model are ported from the original paper rather than trained from scratch.\n\n### Inception V3\n\nThe weights of the Inception V3 model are ported from the original paper rather than trained from scratch.\n\nSince it expects tensors with a size of N x 3 x 299 x 299, to validate the model use the following command:\n\n```\ntorchrun --nproc_per_node=8 train.py --model inception_v3\\\n      --test-only --weights Inception_V3_Weights.IMAGENET1K_V1\n```\n\n### ResNet\n```\ntorchrun --nproc_per_node=8 train.py --model $MODEL\n```\n\nHere `$MODEL` is one of `resnet18`, `resnet34`, `resnet50`, `resnet101` or `resnet152`.\n\n### ResNext\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model $MODEL --epochs 100\n```\n\nHere `$MODEL` is one of `resnext50_32x4d` or `resnext101_32x8d`.\nNote that the above command corresponds to a single node with 8 GPUs. If you use\na different number of GPUs and/or a different batch size, then the learning rate\nshould be scaled accordingly. For example, the pretrained model provided by\n`torchvision` was trained on 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nwith `--batch_size 16` and `--lr 0.4`, instead of the current defaults\nwhich are respectively batch_size=32 and lr=0.1\n\n### MobileNetV2\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model mobilenet_v2 --epochs 300 --lr 0.045 --wd 0.00004\\\n     --lr-step-size 1 --lr-gamma 0.98\n```\n\n\n### MobileNetV3 Large & Small\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 600 --opt rmsprop --batch-size 128 --lr 0.064\\ \n     --wd 0.00001 --lr-step-size 2 --lr-gamma 0.973 --auto-augment imagenet --random-erase 0.2\n```\n\nHere `$MODEL` is one of `mobilenet_v3_large` or `mobilenet_v3_small`.\n\nThen we averaged the parameters of the last 3 checkpoints that improved the Acc@1. See [#3182](https://github.com/pytorch/vision/pull/3182) \nand [#3354](https://github.com/pytorch/vision/pull/3354) for details.\n\n\n### EfficientNet-V1\n\nThe weights of the B0-B4 variants are ported from Ross Wightman's [timm repo](https://github.com/rwightman/pytorch-image-models/blob/01cb46a9a50e3ba4be167965b5764e9702f09b30/timm/models/efficientnet.py#L95-L108).\n\nThe weights of the B5-B7 variants are ported from Luke Melas' [EfficientNet-PyTorch repo](https://github.com/lukemelas/EfficientNet-PyTorch/blob/1039e009545d9329ea026c9f7541341439712b96/efficientnet_pytorch/utils.py#L562-L564).\n\nAll models were trained using Bicubic interpolation and each have custom crop and resize sizes. To validate the models use the following commands:\n```\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b0 --test-only --weights EfficientNet_B0_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b1 --test-only --weights EfficientNet_B1_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b2 --test-only --weights EfficientNet_B2_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b3 --test-only --weights EfficientNet_B3_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b4 --test-only --weights EfficientNet_B4_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b5 --test-only --weights EfficientNet_B5_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b6 --test-only --weights EfficientNet_B6_Weights.IMAGENET1K_V1\ntorchrun --nproc_per_node=8 train.py --model efficientnet_b7 --test-only --weights EfficientNet_B7_Weights.IMAGENET1K_V1\n```\n\n\n### EfficientNet-V2\n```\ntorchrun --nproc_per_node=8 train.py \\\n--model $MODEL --batch-size 128 --lr 0.5 --lr-scheduler cosineannealinglr \\\n--lr-warmup-epochs 5 --lr-warmup-method linear --auto-augment ta_wide --epochs 600 --random-erase 0.1 \\\n--label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --weight-decay 0.00002 --norm-weight-decay 0.0 \\\n--train-crop-size $TRAIN_SIZE --model-ema --val-crop-size $EVAL_SIZE --val-resize-size $EVAL_SIZE \\\n--ra-sampler --ra-reps 4\n```\nHere `$MODEL` is one of `efficientnet_v2_s` and `efficientnet_v2_m`. \nNote that the Small variant had a `$TRAIN_SIZE` of `300` and a `$EVAL_SIZE` of `384`, while the Medium `384` and `480` respectively.\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 4 nodes, each with 8 GPUs (for a total of 32 GPUs),\nand `--batch_size 32`.\n\nThe weights of the Large variant are ported from the original paper rather than trained from scratch. See the `EfficientNet_V2_L_Weights` entry for their exact preprocessing transforms.\n\n\n### RegNet\n\n#### Small models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 128 --wd 0.00005 --lr=0.8\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_400mf`, `regnet_x_800mf`, `regnet_x_1_6gf`, `regnet_y_400mf`, `regnet_y_800mf` and `regnet_y_1_6gf`. Please note we used learning rate 0.4 for `regent_y_400mf` to get the same Acc@1 as [the paper](https://arxiv.org/abs/2003.13678).\n\n#### Medium models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 64 --wd 0.00005 --lr=0.4\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_3_2gf`, `regnet_x_8gf`, `regnet_x_16gf`, `regnet_y_3_2gf` and `regnet_y_8gf`.\n\n#### Large models\n```\ntorchrun --nproc_per_node=8 train.py\\\n     --model $MODEL --epochs 100 --batch-size 32 --wd 0.00005 --lr=0.2\\\n     --lr-scheduler=cosineannealinglr --lr-warmup-method=linear\\\n     --lr-warmup-epochs=5 --lr-warmup-decay=0.1\n```\nHere `$MODEL` is one of `regnet_x_32gf`, `regnet_y_16gf` and `regnet_y_32gf`.\n\n### Vision Transformer\n\n#### vit_b_16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_b_16 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nand `--batch_size 64`.\n\n#### vit_b_32\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_b_32 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment imagenet\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 256`.\n\n#### vit_l_16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_l_16 --epochs 600 --batch-size 128 --lr 0.5 --lr-scheduler cosineannealinglr\\\n    --lr-warmup-method linear --lr-warmup-epochs 5 --label-smoothing 0.1 --mixup-alpha 0.2\\\n    --auto-augment ta_wide --random-erase 0.1 --weight-decay 0.00002 --norm-weight-decay 0.0\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema --val-resize-size 232\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 64`.\n\n#### vit_l_32\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model vit_l_32 --epochs 300 --batch-size 512 --opt adamw --lr 0.003 --wd 0.3\\\n    --lr-scheduler cosineannealinglr --lr-warmup-method linear --lr-warmup-epochs 30\\\n    --lr-warmup-decay 0.033 --amp --label-smoothing 0.11 --mixup-alpha 0.2 --auto-augment ra\\\n    --clip-grad-norm 1 --ra-sampler --cutmix-alpha 1.0 --model-ema\n```\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 8 nodes, each with 8 GPUs (for a total of 64 GPUs),\nand `--batch_size 64`.\n\n\n### ConvNeXt\n```\ntorchrun --nproc_per_node=8 train.py\\ \n--model $MODEL --batch-size 128 --opt adamw --lr 1e-3 --lr-scheduler cosineannealinglr \\ \n--lr-warmup-epochs 5 --lr-warmup-method linear --auto-augment ta_wide --epochs 600 --random-erase 0.1 \\ \n--label-smoothing 0.1 --mixup-alpha 0.2 --cutmix-alpha 1.0 --weight-decay 0.05 --norm-weight-decay 0.0 \\\n--train-crop-size 176 --model-ema --val-resize-size 232 --ra-sampler --ra-reps 4\n```\nHere `$MODEL` is one of `convnext_tiny`, `convnext_small`, `convnext_base` and `convnext_large`. Note that each variant had its `--val-resize-size` optimized in a post-training step, see their `Weights` entry for their exact value.\n\nNote that the above command corresponds to training on a single node with 8 GPUs.\nFor generating the pre-trained weights, we trained with 2 nodes, each with 8 GPUs (for a total of 16 GPUs),\nand `--batch_size 64`.\n\n\n### SwinTransformer\n```\ntorchrun --nproc_per_node=8 train.py\\ \n--model $MODEL --epochs 300 --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler --ra-reps 4  --val-resize-size 224\n```\nHere `$MODEL` is one of `swin_t`, `swin_s` or `swin_b`.\nNote that `--val-resize-size` was optimized in a post-training step, see their `Weights` entry for the exact value.\n\n\n\n\n### SwinTransformer V2\n```\ntorchrun --nproc_per_node=8 train.py\\\n--model $MODEL --epochs 300 --batch-size 128 --opt adamw --lr 0.001 --weight-decay 0.05 --norm-weight-decay 0.0  --bias-weight-decay 0.0 --transformer-embedding-decay 0.0 --lr-scheduler cosineannealinglr --lr-min 0.00001 --lr-warmup-method linear  --lr-warmup-epochs 20 --lr-warmup-decay 0.01 --amp --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 5.0 --cutmix-alpha 1.0 --random-erase 0.25 --interpolation bicubic --auto-augment ta_wide --model-ema --ra-sampler --ra-reps 4  --val-resize-size 256 --val-crop-size 256 --train-crop-size 256 \n```\nHere `$MODEL` is one of `swin_v2_t`, `swin_v2_s` or `swin_v2_b`.\nNote that `--val-resize-size` was optimized in a post-training step, see their `Weights` entry for the exact value.\n\n\n### MaxViT\n```\ntorchrun --nproc_per_node=8 --n_nodes=4 train.py\\\n--model $MODEL --epochs 400 --batch-size 128 --opt adamw --lr 3e-3 --weight-decay 0.05 --lr-scheduler cosineannealinglr --lr-min 1e-5 --lr-warmup-method linear  --lr-warmup-epochs 32  --label-smoothing 0.1 --mixup-alpha 0.8 --clip-grad-norm 1.0 --interpolation bicubic --auto-augment ta_wide --policy-magnitude 15 --model-ema --val-resize-size 224\\\n--val-crop-size 224 --train-crop-size 224 --amp  --model-ema-steps 32 --transformer-embedding-decay 0 --sync-bn\n```\nHere `$MODEL` is `maxvit_t`.\nNote that `--val-resize-size` was not optimized in a post-training step.\n\n\n### ShuffleNet V2\n```\ntorchrun --nproc_per_node=8 train.py \\\n--batch-size=128 \\\n--lr=0.5 --lr-scheduler=cosineannealinglr --lr-warmup-epochs=5 --lr-warmup-method=linear \\\n--auto-augment=ta_wide --epochs=600 --random-erase=0.1 --weight-decay=0.00002 \\\n--norm-weight-decay=0.0 --label-smoothing=0.1 --mixup-alpha=0.2 --cutmix-alpha=1.0 \\\n--train-crop-size=176 --model-ema --val-resize-size=232 --ra-sampler --ra-reps=4\n```\nHere `$MODEL` is either `shufflenet_v2_x1_5` or `shufflenet_v2_x2_0`.\n\nThe models `shufflenet_v2_x0_5` and `shufflenet_v2_x1_0` were contributed by the community. See [PR-849](https://github.com/pytorch/vision/pull/849#issuecomment-483391686) for details.\n\n\n## Mixed precision training\nAutomatic Mixed Precision (AMP) training on GPU for Pytorch can be enabled with the [torch.cuda.amp](https://pytorch.org/docs/stable/amp.html?highlight=amp#module-torch.cuda.amp).\n\nMixed precision training makes use of both FP32 and FP16 precisions where appropriate. FP16 operations can leverage the Tensor cores on NVIDIA GPUs (Volta, Turing or newer architectures) for improved throughput, generally without loss in model accuracy. Mixed precision training also often allows larger batch sizes. GPU automatic mixed precision training for Pytorch Vision can be enabled via the flag value `--amp=True`.\n\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --model resnext50_32x4d --epochs 100 --amp\n```\n\n## Quantized\n\n### Post training quantized models\n\nFor all post training quantized models, the settings are:\n\n1. num_calibration_batches: 32\n2. num_workers: 16\n3. batch_size: 32\n4. eval_batch_size: 128\n5. qbackend: 'fbgemm'\n\n```\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' --model='$MODEL'\n```\nHere `$MODEL` is one of `googlenet`, `inception_v3`, `resnet18`, `resnet50`, `resnext101_32x8d`, `shufflenet_v2_x0_5` and `shufflenet_v2_x1_0`.\n\n### Quantized ShuffleNet V2\n\nHere are commands that we use to quantize the `shufflenet_v2_x1_5` and `shufflenet_v2_x2_0` models.\n```\n# For shufflenet_v2_x1_5\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' \\\n    --model=shufflenet_v2_x1_5 --weights=\"ShuffleNet_V2_X1_5_Weights.IMAGENET1K_V1\" \\\n    --train-crop-size 176 --val-resize-size 232 --data-path /datasets01_ontap/imagenet_full_size/061417/\n\n# For shufflenet_v2_x2_0\npython train_quantization.py --device='cpu' --post-training-quantize --qbackend='fbgemm' \\\n    --model=shufflenet_v2_x2_0 --weights=\"ShuffleNet_V2_X2_0_Weights.IMAGENET1K_V1\" \\\n    --train-crop-size 176 --val-resize-size 232 --data-path /datasets01_ontap/imagenet_full_size/061417/\n```\n\n### QAT MobileNetV2\n\nFor Mobilenet-v2, the model was trained with quantization aware training, the settings used are:\n1. num_workers: 16\n2. batch_size: 32\n3. eval_batch_size: 128\n4. qbackend: 'qnnpack'\n5. learning-rate: 0.0001\n6. num_epochs: 90\n7. num_observer_update_epochs:4\n8. num_batch_norm_update_epochs:3\n9. momentum: 0.9\n10. lr_step_size:30\n11. lr_gamma: 0.1\n12. weight-decay: 0.0001\n\n```\ntorchrun --nproc_per_node=8 train_quantization.py --model='mobilenet_v2'\n```\n\nTraining converges at about 10 epochs.\n\n### QAT MobileNetV3\n\nFor Mobilenet-v3 Large, the model was trained with quantization aware training, the settings used are:\n1. num_workers: 16\n2. batch_size: 32\n3. eval_batch_size: 128\n4. qbackend: 'qnnpack'\n5. learning-rate: 0.001\n6. num_epochs: 90\n7. num_observer_update_epochs:4\n8. num_batch_norm_update_epochs:3\n9. momentum: 0.9\n10. lr_step_size:30\n11. lr_gamma: 0.1\n12. weight-decay: 0.00001\n\n```\ntorchrun --nproc_per_node=8 train_quantization.py --model='mobilenet_v3_large' \\\n    --wd 0.00001 --lr 0.001\n```\n\nFor post training quant, device is set to CPU. For training, the device is set to CUDA.\n\n### Command to evaluate quantized models using the pre-trained weights:\n\n```\npython train_quantization.py --device='cpu' --test-only --qbackend='<qbackend>' --model='<model_name>'\n```\n\nFor inception_v3 you need to pass the following extra parameters:\n```\n--val-resize-size 342 --val-crop-size 299 --train-crop-size 299\n```\n\n<!-- File: references/depth/stereo/README.md (Tokens: ~3490 | Category: anthropic-skill) -->\n# Stereo Matching reference training scripts\n\nThis folder contains reference training scripts for Stereo Matching.\nThey serve as a log of how to train specific models, so as to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nNote: this depends on the `prototype` folder which was removed, the latest\ncommit with the prototype folder is 8b6ba92ce5c48d6db596f6d415a90c128bc5f77b.\n\n\n### CREStereo\n\nThe CREStereo model was trained on a dataset mixture between **CREStereo**, **ETH3D** and the additional split from **Middlebury2014**.\nA ratio of **88-6-6** was used in order to train a baseline weight set. We provide multi-set variant as well.\nBoth used 8 A100 GPUs and a batch size of 2 (so effective batch size is 16). The\nrest of the hyper-parameters loosely follow the recipe from https://github.com/megvii-research/CREStereo.\nThe original recipe trains for **300000** updates (or steps) on the dataset mixture. We modify the learning rate\nschedule to one that starts decaying the weight much sooner. Throughout the experiments we found that this reduces \noverfitting during evaluation time and gradient clip help stabilize the loss during a pre-mature learning rate change.\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_cre \\\n    --model crestereo_base \\\n    --train-datasets crestereo eth3d-train middlebury2014-other \\\n    --dataset-steps 264000 18000 18000\n    --batch-size 2 \\\n    --lr 0.0004 \\\n    --min-lr 0.00002 \\\n    --lr-decay-method cosine \\\n    --warmup-steps 6000 \\\n    --decay-after-steps 30000 \\\n    --clip-grad-norm 1.0 \\\n```\n\nWe employ a multi-set fine-tuning stage where we uniformly sample from multiple datasets. Given hat some of these datasets have extremely large images (``2048x2048`` or more) we opt for a very aggressive scale-range ``[0.2 - 0.8]`` such that as much of the original frame composition is captured inside the ``384x512`` crop.\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_things \\\n    --model crestereo_base \\\n    --train-datasets crestereo eth3d-train middlebury2014-other instereo2k fallingthings carla-highres sintel sceneflow-monkaa sceneflow-driving \\\n    --dataset-steps 12000 12000 12000 12000 12000 12000 12000 12000 12000\n    --batch-size 2 \\\n    --scale-range 0.2 0.8 \\\n    --lr 0.0004 \\\n    --lr-decay-method cosine \\\n    --decay-after-steps 0 \\\n    --warmup-steps 0 \\\n    --min-lr 0.00002 \\\n    --resume-path $checkpoint_dir/$name_cre.pth\n```\n\n\n### Evaluation\n\nEvaluating the base weights\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.CRESTEREO_ETH_MBL_V1\n```\n\nThis should give an **mae of about 1.416** on the train set of `Middlebury2014`. Results may vary slightly depending on the batch size and the number of GPUs. For the most accurate results use 1 GPU and `--batch-size 1`. The created log file should look like this, where the first key is the number of cascades and the nested key is the number of recursive iterations:\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 2.363, 'rmse': 4.352, '1px': 0.611, '3px': 0.828, '5px': 0.891, 'relepe': 0.176, 'fl-all': 64.511}\n\t\t5: {'mae': 1.618, 'rmse': 3.71, '1px': 0.761, '3px': 0.879, '5px': 0.918, 'relepe': 0.154, 'fl-all': 77.128}\n\t\t10: {'mae': 1.416, 'rmse': 3.53, '1px': 0.777, '3px': 0.896, '5px': 0.933, 'relepe': 0.148, 'fl-all': 78.388}\n\t\t20: {'mae': 1.448, 'rmse': 3.583, '1px': 0.771, '3px': 0.893, '5px': 0.931, 'relepe': 0.145, 'fl-all': 77.7}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.972, 'rmse': 4.125, '1px': 0.73, '3px': 0.865, '5px': 0.908, 'relepe': 0.169, 'fl-all': 74.396}\n\t\t5: {'mae': 1.403, 'rmse': 3.448, '1px': 0.793, '3px': 0.905, '5px': 0.937, 'relepe': 0.151, 'fl-all': 80.186}\n\t\t10: {'mae': 1.312, 'rmse': 3.368, '1px': 0.799, '3px': 0.912, '5px': 0.943, 'relepe': 0.148, 'fl-all': 80.379}\n\t\t20: {'mae': 1.376, 'rmse': 3.542, '1px': 0.796, '3px': 0.91, '5px': 0.942, 'relepe': 0.149, 'fl-all': 80.054}\n\t},\n}\n```\n\nYou can also evaluate the Finetuned weights:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.CRESTEREO_FINETUNE_MULTI_V1\n```\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 1.85, 'rmse': 3.797, '1px': 0.673, '3px': 0.862, '5px': 0.917, 'relepe': 0.171, 'fl-all': 69.736}\n\t\t5: {'mae': 1.111, 'rmse': 3.166, '1px': 0.838, '3px': 0.93, '5px': 0.957, 'relepe': 0.134, 'fl-all': 84.596}\n\t\t10: {'mae': 1.02, 'rmse': 3.073, '1px': 0.854, '3px': 0.938, '5px': 0.96, 'relepe': 0.129, 'fl-all': 86.042}\n\t\t20: {'mae': 0.993, 'rmse': 3.059, '1px': 0.855, '3px': 0.942, '5px': 0.967, 'relepe': 0.126, 'fl-all': 85.784}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.667, 'rmse': 3.867, '1px': 0.78, '3px': 0.891, '5px': 0.922, 'relepe': 0.165, 'fl-all': 78.89}\n\t\t5: {'mae': 1.158, 'rmse': 3.278, '1px': 0.843, '3px': 0.926, '5px': 0.955, 'relepe': 0.135, 'fl-all': 84.556}\n\t\t10: {'mae': 1.046, 'rmse': 3.13, '1px': 0.85, '3px': 0.934, '5px': 0.96, 'relepe': 0.13, 'fl-all': 85.464}\n\t\t20: {'mae': 1.021, 'rmse': 3.102, '1px': 0.85, '3px': 0.935, '5px': 0.963, 'relepe': 0.129, 'fl-all': 85.417}\n\t},\n}\n```\n\nEvaluating the author provided weights:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 cascade_evaluation.py --dataset middlebury2014-train --batch-size 1 --dataset-root $dataset_root --model crestereo_base --weights CREStereo_Base_Weights.MEGVII_V1\n```\n\n```\nDataset: middlebury2014-train @size: [384, 512]:\n{\n\t1: {\n\t\t2: {'mae': 1.704, 'rmse': 3.738, '1px': 0.738, '3px': 0.896, '5px': 0.933, 'relepe': 0.157, 'fl-all': 76.464}\n\t\t5: {'mae': 0.956, 'rmse': 2.963, '1px': 0.88, '3px': 0.948, '5px': 0.965, 'relepe': 0.124, 'fl-all': 88.186}\n\t\t10: {'mae': 0.792, 'rmse': 2.765, '1px': 0.905, '3px': 0.958, '5px': 0.97, 'relepe': 0.114, 'fl-all': 90.429}\n\t\t20: {'mae': 0.749, 'rmse': 2.706, '1px': 0.907, '3px': 0.961, '5px': 0.972, 'relepe': 0.113, 'fl-all': 90.807}\n\t},\n}\n{\n\t2: {\n\t\t2: {'mae': 1.702, 'rmse': 3.784, '1px': 0.784, '3px': 0.894, '5px': 0.924, 'relepe': 0.172, 'fl-all': 80.313}\n\t\t5: {'mae': 0.932, 'rmse': 2.907, '1px': 0.877, '3px': 0.944, '5px': 0.963, 'relepe': 0.125, 'fl-all': 87.979}\n\t\t10: {'mae': 0.773, 'rmse': 2.768, '1px': 0.901, '3px': 0.958, '5px': 0.972, 'relepe': 0.117, 'fl-all': 90.43}\n\t\t20: {'mae': 0.854, 'rmse': 2.971, '1px': 0.9, '3px': 0.957, '5px': 0.97, 'relepe': 0.122, 'fl-all': 90.269}\n\t},\n}\n```\n\n# Concerns when training\n\nWe encourage users to be aware of the **aspect-ratio** and **disparity scale** they are targeting when doing any sort of training or fine-tuning. The model is highly sensitive to these two factors, as a consequence of naive multi-set fine-tuning one can achieve `0.2 mae` relatively fast. We recommend that users pay close attention to how they **balance dataset sizing** when training such networks.\n\n Ideally, dataset scaling should be trated at an individual level and a thorough **EDA** of the disparity distribution in random crops at the desired training / inference size should be performed prior to any large compute investments.\n\n### Disparity scaling\n\n##### Sample A\n The top row contains a sample from `Sintel` whereas the bottom row one from `Middlebury`.\n\n![Disparity1](assets/disparity-domain-drift.jpg)\n\nFrom left to right (`left_image`, `right_image`, `valid_mask`, `valid_mask & ground_truth`, `prediction`). **Darker is further away, lighter is closer**. In the case of `Sintel` which is more closely aligned to the original distribution of `CREStereo` we notice that the model accurately predicts the background scale whereas in the case of `Middlebury2014` it cannot correctly estimate the continuous disparity. Notice that the frame composition is similar for both examples. The blue skybox in the `Sintel` scene behaves similarly to the `Middlebury` black background. However, because the `Middlebury` samples comes from an extremely large scene the crop size of `384x512` does not correctly capture the general training distribution.\n\n\n\n\n##### Sample B\n\nThe top row contains a scene from `Sceneflow` using the `Monkaa` split whilst the bottom row is a scene from `Middlebury`. This sample exhibits the same issues when it comes to **background estimation**. Given the exaggerated size of the `Middlebury` samples the model **colapses the smooth background** of the sample to what it considers to be a mean background disparity value.\n\n![Disparity2](assets/disparity-background-mode-collapse.jpg)\n\n\nFor more detail on why this behaviour occurs based on the training distribution proportions you can read more about the network at: https://github.com/pytorch/vision/pull/6629#discussion_r978160493\n\n\n### Metric overfitting\n\n##### Learning is critical in the beginning\n\nWe also advise users to make user of faster training schedules, as the performance gain over long periods time is marginal. Here we exhibit a difference between a faster decay schedule and later decay schedule.\n\n![Loss1](assets/Loss.jpg)\n\nIn **grey** we set the lr decay to begin after `30000` steps whilst in **orange** we opt for a very late learning rate decay at around `180000` steps. Although exhibiting stronger variance, we can notice that unfreezing the learning rate earlier whilst employing `gradient-norm` out-performs the default configuration.\n\n##### Gradient norm saves time\n\n![Loss2](assets/gradient-norm-removal.jpg)\n\nIn **grey** we keep ``gradient norm`` enabled whilst in **orange** we do not. We can notice that remvoing the gradient norm exacerbates the performance decrease in the early stages whilst also showcasing an almost complete collapse around the `60000` steps mark where we started decaying the lr for **orange**.\n\nAlthough both runs ahieve an improvement of about ``0.1`` mae after the lr decay start, the benefits of it are observable much faster when ``gradient norm`` is employed as the recovery period is no longer accounted for.\n\n<!-- File: references/detection/README.md (Tokens: ~918 | Category: anthropic-skill) -->\n# Object detection reference training scripts\n\nThis folder contains reference training scripts for object detection.\nThey serve as a log of how to train specific models, to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nTo execute the example commands below you must install the following:\n\n```\ncython\npycocotools\nmatplotlib\n```\n\nYou must modify the following flags:\n\n`--data-path=/path/to/coco/dataset`\n\n`--nproc_per_node=<number_of_gpus_available>`\n\nExcept otherwise noted, all models have been trained on 8x V100 GPUs. \n\n### Faster R-CNN ResNet-50 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### Faster R-CNN MobileNetV3-Large FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_mobilenet_v3_large_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n### Faster R-CNN MobileNetV3-Large 320 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fasterrcnn_mobilenet_v3_large_320_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n### FCOS ResNet-50 FPN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model fcos_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3  --lr 0.01 --amp --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### RetinaNet\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model retinanet_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --lr 0.01 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n### SSD300 VGG16\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model ssd300_vgg16 --epochs 120\\\n    --lr-steps 80 110 --aspect-ratio-group-factor 3 --lr 0.002 --batch-size 4\\\n    --weight-decay 0.0005 --data-augmentation ssd --weights-backbone VGG16_Weights.IMAGENET1K_FEATURES\n```\n\n### SSDlite320 MobileNetV3-Large\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model ssdlite320_mobilenet_v3_large --epochs 660\\\n    --aspect-ratio-group-factor 3 --lr-scheduler cosineannealinglr --lr 0.15 --batch-size 24\\\n    --weight-decay 0.00004 --data-augmentation ssdlite\n```\n\n\n### Mask R-CNN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco --model maskrcnn_resnet50_fpn --epochs 26\\\n    --lr-steps 16 22 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n\n### Keypoint R-CNN\n```\ntorchrun --nproc_per_node=8 train.py\\\n    --dataset coco_kp --model keypointrcnn_resnet50_fpn --epochs 46\\\n    --lr-steps 36 43 --aspect-ratio-group-factor 3 --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n<!-- File: references/optical_flow/README.md (Tokens: ~764 | Category: anthropic-skill) -->\n# Optical flow reference training scripts\n\nThis folder contains reference training scripts for optical flow.\nThey serve as a log of how to train specific models, so as to provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\n\n### RAFT Large\n\nThe RAFT large model was trained on Flying Chairs and then on Flying Things.\nBoth used 8 A100 GPUs and a batch size of 2 (so effective batch size is 16). The\nrest of the hyper-parameters are exactly the same as the original RAFT training\nrecipe from https://github.com/princeton-vl/RAFT. The original recipe trains for\n100000 updates (or steps) on each dataset - this corresponds to about 72 and 20\nepochs on Chairs and Things respectively:\n\n```\nnum_epochs = ceil(num_steps / number_of_steps_per_epoch)\n           = ceil(num_steps / (num_samples / effective_batch_size))\n```\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_chairs \\\n    --model raft_large \\\n    --train-dataset chairs \\\n    --batch-size 2 \\\n    --lr 0.0004 \\\n    --weight-decay 0.0001 \\\n    --epochs 72 \\\n    --output-dir $chairs_dir\n```\n\n```\ntorchrun --nproc_per_node 8 --nnodes 1 train.py \\\n    --dataset-root $dataset_root \\\n    --name $name_things \\\n    --model raft_large \\\n    --train-dataset things \\\n    --batch-size 2 \\\n    --lr 0.000125 \\\n    --weight-decay 0.0001 \\\n    --epochs 20 \\\n    --freeze-batch-norm \\\n    --output-dir $things_dir\\\n    --resume $chairs_dir/$name_chairs.pth\n```\n\n\n### Evaluation\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 train.py --val-dataset sintel --batch-size 1 --dataset-root $dataset_root --model raft_large --weights Raft_Large_Weights.C_T_SKHT_V2\n```\n\nThis should give an epe of about 1.3822 on the clean pass and 2.7161 on the\nfinal pass of Sintel-train. Results may vary slightly depending on the batch\nsize and the number of GPUs. For the most accurate results use 1 GPU and\n`--batch-size 1`:\n\n```\nSintel val clean epe: 1.3822\t1px: 0.9028\t3px: 0.9573\t5px: 0.9697\tper_image_epe: 1.3822\tf1: 4.0248\nSintel val final epe: 2.7161\t1px: 0.8528\t3px: 0.9204\t5px: 0.9392\tper_image_epe: 2.7161\tf1: 7.5964\n```\n\nYou can also evaluate on Kitti train:\n\n```\ntorchrun --nproc_per_node 1 --nnodes 1 train.py --val-dataset kitti --batch-size 1 --dataset-root $dataset_root --model raft_large --weights Raft_Large_Weights.C_T_SKHT_V2\nKitti val epe: 4.7968\t1px: 0.6388\t3px: 0.8197\t5px: 0.8661\tper_image_epe: 4.5118\tf1: 16.0679\n```\n\n<!-- File: references/segmentation/README.md (Tokens: ~500 | Category: anthropic-skill) -->\n# Semantic segmentation reference training scripts\n\nThis folder contains reference training scripts for semantic segmentation.\nThey serve as a log of how to train specific models and provide baseline\ntraining and evaluation scripts to quickly bootstrap research.\n\nAll models have been trained on 8x V100 GPUs.\n\nYou must modify the following flags:\n\n`--data-path=/path/to/dataset`\n\n`--nproc_per_node=<number_of_gpus_available>`\n\n## fcn_resnet50\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model fcn_resnet50 --aux-loss --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n## fcn_resnet101\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model fcn_resnet101 --aux-loss --weights-backbone ResNet101_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_resnet50\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model deeplabv3_resnet50 --aux-loss --weights-backbone ResNet50_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_resnet101\n```\ntorchrun --nproc_per_node=8 train.py --lr 0.02 --dataset coco -b 4 --model deeplabv3_resnet101 --aux-loss --weights-backbone ResNet101_Weights.IMAGENET1K_V1\n```\n\n## deeplabv3_mobilenet_v3_large\n```\ntorchrun --nproc_per_node=8 train.py --dataset coco -b 4 --model deeplabv3_mobilenet_v3_large --aux-loss --wd 0.000001 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n## lraspp_mobilenet_v3_large\n```\ntorchrun --nproc_per_node=8 train.py --dataset coco -b 4 --model lraspp_mobilenet_v3_large --wd 0.000001 --weights-backbone MobileNet_V3_Large_Weights.IMAGENET1K_V1\n```\n\n<!-- File: references/similarity/README.md (Tokens: ~644 | Category: anthropic-skill) -->\n# Similarity Learning Using Triplet Loss #\n\nIn this reference, we use triplet loss to learn embeddings which can be used to differentiate images. This learning technique was popularized by [FaceNet: A Unified Embedding for Face Recognition and Clustering](https://arxiv.org/abs/1503.03832) and has been quite effective in learning embeddings to differentiate between faces.\n\nThis reference can be directly applied to the following use cases:\n\n* You have an unknown number of classes and would like to train a model to learn how to differentiate between them.\n* You want to train a model to learn a distance-based metric between samples. For example, learning a distance-based similarity measure between faces.\n\n### Training ###\nBy default, the training script trains ResNet50 on the FashionMNIST Dataset to learn image embeddings which can be used to differentiate between images by measuring the euclidean distance between embeddings. This can be changed as per your requirements.\n\nImage embeddings of the same class should be 'close' to each other, while image embeddings between different classes should be 'far' away.\n\nTo run the training script:\n\n```bash\npython train.py -h    # Lists all optional arguments\npython train.py \t\t\t# Runs training script with default args\n```\n\nRunning the training script as is should yield 97% accuracy on the FMNIST test set within 10 epochs.\n\n### Loss ###\n`TripletMarginLoss` is a loss function which takes in a triplet of samples. A valid triplet has an:\n\n1. Anchor: a sample from the dataset\n2. Positive: another sample with the same label/group as the anchor (Generally, positive != anchor)\n3. Negative: a sample with a different label/group from the anchor\n\n`TripletMarginLoss` (refer to `loss.py`) does the following:\n\n```python\nloss = max(dist(anchor, positive) - dist(anchor, negative) + margin, 0)\n```\nWhere `dist` is a distance function. Minimizing this function effectively leads to minimizing `dist(anchor, positive)` and maximizing `dist(anchor, negative)`.\n\nThe FaceNet paper describe this loss in more detail.\n\n### Sampler ###\n\nIn order to generate valid triplets from a batch of samples, we need to make sure that each batch has multiple samples with the same label. We do this using `PKSampler` (refer to `sampler.py`), which ensures that each batch of size `p * k` will have samples from exactly `p` classes and `k` samples per class.\n\n### Triplet Mining ###\n\n`TripletMarginLoss` currently supports the following mining techniques:\n\n* `batch_all`: Generates all possible triplets from a batch and excludes the triplets which are 'easy' (which have `loss = 0`) before passing it through the loss function.\n* `batch_hard`: For every anchor, `batch_hard` creates a triplet with the 'hardest' positive (farthest positive) and negative (closest negative).\n\nThese mining strategies usually speed up training.\n\nThis [webpage](https://omoindrot.github.io/triplet-loss) describes the sampling and mining strategies in more detail.\n\n<!-- File: references/video_classification/README.md (Tokens: ~1521 | Category: anthropic-skill) -->\n# Video Classification\n\nWe present a simple training script that can be used for replicating the result of [resenet-based video models](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf). All models are trained on [Kinetics400 dataset](https://deepmind.com/research/open-source/kinetics), a benchmark dataset for human-action recognition. The accuracy is reported on the traditional validation split.\n\n## Data preparation\n\nIf you already have downloaded [Kinetics400 dataset](https://deepmind.com/research/open-source/kinetics), \nplease proceed directly to the next section.\n\nTo download videos, one can use https://github.com/Showmax/kinetics-downloader. Please note that the dataset can take up upwards of 400GB, depending on the quality setting during download.\n\n## Training\n\nWe assume the training and validation AVI videos are stored at `/data/kinectics400/train` and \n`/data/kinectics400/val`. For training we suggest starting with the hyperparameters reported in the [paper](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf), in order to match the performance of said models. Clip sampling strategy is a particularly important parameter during training, and we suggest using random temporal jittering during training - in other words sampling multiple training clips from each video with random start times during at every epoch. This functionality is built into our training script, and optimal hyperparameters are set by default.  \n\n### Multiple GPUs\n\nRun the training on a single node with 8 GPUs:\n```bash\ntorchrun --nproc_per_node=8 train.py --data-path=/data/kinectics400 --kinetics-version=\"400\" --lr 0.08 --cache-dataset --sync-bn --amp \n```\n\n**Note:** all our models were trained on 8 nodes with 8 V100 GPUs each for a total of 64 GPUs. Expected training time for 64 GPUs is 24 hours, depending on the storage solution.\n**Note 2:** hyperparameters for exact replication of our training can be found on the section below. Some hyperparameters such as learning rate must be scaled linearly in proportion to the number of GPUs. The default values assume 64 GPUs.\n\n### Single GPU \n\n**Note:** training on a single gpu can be extremely slow. \n\n\n```bash\npython train.py --data-path=/data/kinectics400 --kinetics-version=\"400\" --batch-size=8 --cache-dataset\n```\n\n\n### Additional Kinetics versions\n\nSince the original release, additional versions of Kinetics dataset became available (Kinetics 600).\nOur training scripts support these versions of dataset as well by setting the `--kinetics-version` parameter to `\"600\"`.\n\n**Note:** training on Kinetics 600 requires a different set of hyperparameters for optimal performance. We do not provide Kinetics 600 pretrained models.\n\n\n## Video classification models\n\nStarting with version `0.4.0` we have introduced support for basic video tasks and video classification modelling.\nFor more information about the available models check [here](https://pytorch.org/docs/stable/torchvision/models.html#video-classification). \n\n### Video ResNet models\n\nSee reference training script [here](https://github.com/pytorch/vision/blob/main/references/video_classification/train.py):\n\n- input space: RGB\n- resize size: [128, 171]\n- crop size: [112, 112]\n- mean: [0.43216, 0.394666, 0.37645]\n- std: [0.22803, 0.22145, 0.216989]\n- number of classes: 400\n\nInput data augmentations at training time (with optional parameters):\n\n1. ConvertImageDtype\n2. Resize (resize size value above)\n3. Random horizontal flip (0.5)\n4. Normalization (mean, std, see values above)\n5. Random Crop (crop size value above)\n6. Convert BCHW to CBHW\n\nInput data augmentations at validation time (with optional parameters):\n\n1. ConvertImageDtype\n2. Resize (resize size value above)\n3. Normalization (mean, std, see values above)\n4. Center Crop (crop size value above)\n5. Convert BCHW to CBHW\n\nThis translates in the following set of command-line arguments. Please note that `--batch-size` parameter controls the\nbatch size per GPU. Moreover, note that our default `--lr` is configured for 64 GPUs which is how many we used for the \nVideo resnet models:\n```\n# number of frames per clip\n--clip_len 16 \\ \n--frame-rate 15 \\\n# allow for temporal jittering\n--clips_per_video 5 \\\n--batch-size 24 \\\n--epochs 45 \\\n--lr 0.64 \\\n# we use 10 epochs for linear warmup\n--lr-warmup-epochs 10 \\\n# learning rate is decayed at 20, 30, and 40 epoch by a factor of 10\n--lr-milestones 20, 30, 40 \\\n--lr-gamma 0.1 \\\n--train-resize-size 128 171 \\\n--train-crop-size 112 112 \\\n--val-resize-size 128 171 \\\n--val-crop-size 112 112\n```\n\n### S3D\n\nThe S3D model was trained similarly to the above but with the following changes on the default configuration:\n```\n--batch-size=12 --lr 0.2 --clip-len 64 --clips-per-video 5 --sync-bn \\\n--train-resize-size 256 256 --train-crop-size 224 224 --val-resize-size 256 256 --val-crop-size 224 224\n```\n\nWe used 64 GPUs to train the architecture. \n\nTo estimate the validation statistics of the model, we run the reference script with the following configuration:\n```\n--batch-size=16 --test-only --clip-len 128 --clips-per-video 1 \n```\n\n### Additional video modelling resources\n\n- [Video Model Zoo](https://github.com/facebookresearch/VMZ)\n- [PySlowFast](https://github.com/facebookresearch/SlowFast)\n\n### References\n\n[0] _D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun and M. Paluri_: A Closer Look at Spatiotemporal Convolutions for Action Recognition. _CVPR 2018_ ([paper](https://research.fb.com/wp-content/uploads/2018/04/a-closer-look-at-spatiotemporal-convolutions-for-action-recognition.pdf))\n\n[1] _W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, A. Zisserman_: The Kinetics Human Action Video Dataset ([paper](https://arxiv.org/abs/1705.06950))\n</yakaai_skills>"}