README (README.md) Copy Section

Cloud TPUs #

This repository is a collection of reference models and tools used with
Cloud TPUs.

The fastest way to get started training a model on a Cloud TPU is by following
the tutorial. Click the button below to launch the tutorial using Google Cloud
Shell.

[](https://console.cloud.google.com/cloudshell/open?git_repo=https%3A%2F%2Fgithub.com%2Ftensorflow%2Ftpu&page=shell&tutorial=tools%2Fctpu%2Ftutorial.md)

_Note:_ This repository is a public mirror, pull requests will not be accepted.
Please file an issue if you have a feature or bug request.

Running Models Copy Chapter

To run models in the models subdirectory, you may need to add the top-level
/models folder to the Python path with the command:

export PYTHONPATH="$PYTHONPATH:/path/to/models"

---

Benchmarks/ResNet 50 V1.5 Performance Comparison TensorFlow 1.12 GCP (benchmarks/ResNet-50_v1.5_Performance_Comparison_TensorFlow_1.12_GCP.md) Copy Section

Methodology for ResNet-50 v1.5 Performance Comparison on Cloud TPUs and Google Cloud GPUs

Frank Chen, Toby Boyd, Jing Li (Google Brain)

Our approach to performance measurement Copy Chapter

Great care is required to construct performance benchmarks that fairly and reproducibly compare machine learning (ML) training performance across an increasing variety of different hardware configurations and software frameworks.

For this initial performance comparison, we chose to focus on two top-of-the-line hardware accelerators that are currently available on Google Cloud: NVIDIA’s V100 GPU and Google’s Cloud TPU v2 Pod. We ran our analysis on Google Cloud Platform (GCP) and used well-optimized, open-source TensorFlow 1.12 implementations to collect all performance measurements. To maximize performance, we use bfloat16 (a half-precision, 16-bit data type explicitly designed for ML) on the Cloud TPUs and use mixed-precision float16 to maximize the utilization of Tensor Cores on the NVIDIA Tesla V100s.

The scale of the largest ML training runs has increased rapidly over the past few years, and we expect this trend to continue; we also expect rapid continued improvements in accelerator performance and capabilities.

Model architecture: ResNet-50 v1.5 Copy Chapter

We chose to focus on training the ResNet-50 image recognition model on the ImageNet dataset because it is well-known and has been well-optimized on many platforms. There are actually several different variants of the ResNet-50 architecture and training procedure that have vastly different computational profiles and achieve different trained accuracies. In this study, we choose a variant of ResNet-50 that we informally call “ResNet-50 v1.5.”

ResNet-50 v1.5 is almost the same model architecture described by He, et. al. in the original ResNet paper, “Deep Residual Learning for Image Recognition” (arXiv:1512.03385v1). However, stride 2 is used in the first 3x3 convolution of each block instead of in the first 1x1 convolution. This variation can be found in the code corresponding to the paper "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour" (arXiv:1706.02677v2). We use the same input size (224x224) as the original ResNet paper.

Implementation Details Copy Section

We use the tf_cnn_benchmarks implementation of ResNet-50 v1.5 training for the GPU benchmark. This version of ResNet-50 utilizes mixed-precision FP16 to maximize the utilization of Tensor Cores on the NVIDIA Tesla V100. XLA was used to optimize the graph for GPU execution to further improve the performance of the V100 GPUs.

We use the standard Cloud TPU reference model implementation of ResNet-50 v1.5 for Cloud TPU Pods. This implementation includes minor optimizations specific to TPUs (including using bfloat16 numerics on more variables, and transposing NCHW-formatted data to NHWC before sending it to the TPU for better performance).

A variety of alternative ResNet-50 training protocols have recently emerged that can accelerate convergence by reducing the amount of computation required for training. For example, progressively scaling up image sizes as training progresses and setting more aggressive learning rate schedules empirically lead to faster convergence on ImageNet. However, for the purposes of this benchmark comparison, we have chosen to stick with the most standard ResNet-50 training protocol, and we hold the amount of computation fixed and then compare the performance of different systems as they carry out the same logical operations.

Target accuracy and reproducibility: 76% Top-1 accuracy across 5 runs Copy Section

We set a top-1 accuracy target of 76.0% on the ImageNet dataset as we believe this is near the top of the achievable range for ResNet-50 v1.5 with the standard training protocol. When training ML models, there are many ways to increase training throughput by sacrificing accuracy. However, those last few percentage points of accuracy are often the most valuable ones in real-world ML applications, so we chose the challenge of training a well-known model to the highest-achievable accuracy.

To make sure that our training results are reproducible, we performed five separate training runs for each hardware configuration, and we certified that all runs achieved at least 76.0% top-1 accuracy on the ImageNet validation dataset with no blacklists.

Training epochs Copy Section

Similar to the “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour” paper, we measure the performance of training 90 epochs on each hardware configuration. Before beginning our measurements, we run a single “warm-up epoch” to exclude one-time setup costs and ensure that data caches are fully filled.

As training time decreases to the order of minutes on our largest Cloud TPU Pod configurations, initialization and compiler overhead becomes significant, per Amdahl’s law. While software optimizations will continue to reduce these overheads, we find that Cloud TPU Pod customers such as eBay typically train on much larger datasets than ImageNet, in which case these overheads are no longer significant, which is why we choose to exclude them here.

We determine the training time for 90 epochs by examining the TensorFlow summary file in the model directory after each training run. Specifically, we measure the training duration as the time between TensorFlow logging the training loss for the last step of the warmup epoch and the last step of the entire training run. We believe that this accurately captures the time taken to train ImageNet for 90 epochs and excludes overheads and cache warming latencies for both GPUs and Cloud TPUs.

Optimizations for large-batch training: LARS and label smoothing Copy Section

To enable efficient training on our largest Cloud TPU Pod configurations using batch sizes of 16,384 and larger, our open-source ResNet-50 implementation includes the following optimizations:

1. At large batch sizes, the Cloud TPU implementation switches to the Layer-wise Adaptive Rate Scaling (LARS) optimizer presented by You et al. in “Large Batch Training of Convolutional Neural Networks” (arXiv:1708.03888) rather than the conventional stochastic gradient descent optimizer with momentum.

2. At large batch sizes, the Cloud TPU implementation enables “label smoothing” as described in the TensorFlow documentation. Label smoothing becomes more important as the total number of gradient updates per training run decreases.

Detailed Instructions to Reproduce All Experiments Copy Chapter

Training ResNet-50 v1.5 on V100 GPUs on GCP Copy Section

#### ImageNet Data Preparation for GPUs

Instructions for generating the ImageNet image set. These instructions result in the data being uploaded to a Google Storage Bucket.

1. Sign up for the ImageNet image database (image-net.org) and obtain a username and access key to download the training and evaluation data.

2. Utilize instructions for the imagenet_to_gcs.py tool to process the data and upload it to a Google Storage Bucket.

#### VM Sizes for V100 GPUs on GCP

GCP makes it possible to attach GPUs to VMs of different sizes. Since all input pipeline stages (such as JPEG decoding and image pre-processing) happen on the VM CPUs, it is important to rent a large enough VM to keep up with the attached GPUs. Our experiments show that each V100 GPU requires approximately 8 virtual CPU threads for full utilization when training ResNet-50 v1.5, so we chose the following configurations for our experiments to minimize total training costs while maximizing performance:

|Number of GPUs|GCE Machine Size|
|--------------|----------------|
|1 x V100|n1-standard-8|
|4 x V100|n1-standard-32|
|8 x V100|n1-standard-64|

#### GPU training instructions

1. Start the instance type to use for training based on the Google Deep Learning Images optimized for TensorFlow.

/ Detailed source-code truncated for AI context efficiency. /

2. Set up local data storage using Google Cloud local solid-state drives (SSD).

gcloud compute ssh $INSTANCE_NAME
Instructions for 1, 4, and 8 GPUs are different.  Copy Section

The difference is each setup has a different number of nvme drives.  Copy Section

When more than one drive exist RAID is used to create a single drive.  Copy Section
4 and 8 GPU instances  Copy Chapter

Installs raid management tool.

sudo apt-get update && sudo apt-get install mdadm --no-install-recommends
Only run for 8 GPUs with 4x local nvme drives.

sudo mdadm --create /dev/md0 --level=0 --raid-devices=4 \
/dev/nvme0n1 /dev/nvme0n2 /dev/nvme0n3 /dev/nvme0n4 
Only run for 4 GPUs with 2x local nvme drives.

sudo mdadm --create /dev/md0 --level=0 --raid-devices=2 \
/dev/nvme0n1 /dev/nvme0n2
Formats and mounts the array.

sudo mkfs.ext4 -F /dev/md0
sudo mkdir -p /data && sudo mount /dev/md0 /data
1 GPU instances.  Copy Chapter

sudo mkfs.ext4 -F /dev/nvme0n1
sudo mkdir -p /data && sudo mount /dev/nvme0n1 /data

3. Copy data from your GCS bucket created earlier to the local drive.

Copies data from your GCS bucket created earlier to the local drive.  Copy Section

sudo mkdir -p /data/imagenet && sudo chmod -R 777 /data
gcloud storage cp --recursive gs://<your bucket with imagenet>/imagenet/* /data/imagenet/

4. Install TensorFlow 1.12 compiled with CUDA 10.0, cuDNN 7.3, and AVX2.

Install custom TensorFlow build.  Copy Section

pip install --upgrade --force-reinstall \
https://storage.googleapis.com/tf-performance/tf_binary/tensorflow-1.12.0.a6d8ffa.AVX2.CUDA10-cp27-cp27mu-linux_x86_64.whl

5. Start training with the commands below. For this test, only the loss and the learning rate are recorded with their timestamps to calculate elapsed training time. Summaries are recorded to disk asynchronously and have not shown to have a performance impact.

/ Detailed source-code truncated for AI context efficiency. /

6. After training is complete, execute the evaluation with one of the commands below:

4 and 8 GPUs

python scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--batch_size=250 \
--model=resnet50_v1.5 \
--variable_update=replicated \
--num_gpus=1 \
--num_batches=200 \
--use_fp16 \
--data_dir=/data/imagenet/validation \
--train_dir=$HOME/test00 \
--eval=True \
--xla_compile=True
1 GPU

python scripts/tf_cnn_benchmarks/tf_cnn_benchmarks.py \
--batch_size=250 \
--model=resnet50_v1.5 \
--nodistortions \
--num_gpus=1 \
--num_batches=200 \
--use_fp16 \
--data_dir=/data/imagenet/validation \
--train_dir=$HOME/test00 \
--eval=True \
--xla_compile=True

7. At the end of the evaluation, you should see the evaluation results printed out in the following format:

Accuracy @ 1 = 0.7649 Accuracy @ 5 = 0.9309 [50000 examples]

Calculate the training time.

Get script to read event log

git clone https://github.com/tensorflow/tpu.git
cd tpu/models/official/resnet/benchmark
8 GPUs

python read_training_time.py --model_dir=$HOME/test00/  \
--warmup_steps=513 \
--end_step=46710  \
--event_name=base_loss
4 GPUs

python read_training_time.py --model_dir=$HOME/test00/  \
--warmup_steps=1026 \
--end_step=93419  \
--event_name=base_loss
1 GPU

python read_training_time.py --model_dir=$HOME/test00/  \
--warmup_steps=4107 \
--end_step=373674  \
--event_name=base_loss

Training ResNet-50 v1.5 on Cloud TPUs on GCP Copy Section

#### ImageNet Data Preparation for Cloud TPUs

1. Create a new GCS bucket, making sure to select the “Regional” storage class and choose a region that is the same as the desired location of your Cloud TPUs.

2. Sign up for the ImageNet image database (image-net.org) and obtain a username and access key to download the training and evaluation data.

3. Run the imagenet_to_gcs.py on a GCE VM or your local desktop to download, process, and re-upload the data into your GCS bucket.

#### VM Size for Cloud TPUs on GCP

Multiple Cloud TPU hardware configurations are available on GCP, and each one includes a balanced combination of host machines and TPU accelerators. These combinations are constructed automatically and made available over the network. In this study, training data was stored in Google Cloud Storage (GCS) and all input preprocessing happens on the Cloud TPU server, so only a tiny VM is required to orchestrate the computation. We used an n1-standard-2 in all of our Cloud TPU experiments.

#### Cloud TPU Training Instructions

1. Create a Cloud TPU or Cloud TPU Pod slice with TensorFlow version 1.12 via the Google Cloud console.

2. Start a new n1-standard-2 GCE VM with Ubuntu 16.04 LTS and the “Allow full access to all Cloud APIs option”, and then install TensorFlow 1.12 on the VM.

3. Clone the Cloud TPU GitHub repository located at https://github.com/tensorflow/tpu using the command git clone https://github.com/tensorflow/tpu.

4. Go to the local copy of the repository and check out the r1.12 branch using git checkout r1.12.

5. Start ResNet training using the following command line in the GCE VM. We recommend that you start a screen session so that training will be uninterrupted even if the SSH connection to your VM is temporarily lost.

export PYTHONPATH="$PYTHONPATH:~/tpu/models"
python tpu/models/official/resnet/resnet_main.py \
  --tpu=MY_TPU_NAME --tpu_zone=MY_TPU_ZONE --num_cores=TPU_CORES \
  --data_dir=gs://IMAGENET_DATA_BUCKET/DIRECTORY \
  --model_dir=gs://RESNET_CHECKPOINT_BUCKET/DIRECTORY \
  --train_batch_size=BATCH_SIZE --iterations_per_loop=ITERATIONS \
  --train_steps=ITERATIONS \
  --mode=train --eval_batch_size=1000

Use the following parameters for various Cloud TPU system sizes:

|TPU Type |TPU_CORES|BATCH_SIZE|ITERATIONS|
|---------------------------|---------|----------|----------|
|Cloud TPU v2 |8 |1024 |113854 |
|1/16 Cloud TPU Pod (v2-32) |32 |4096 |28464 |
|1/4 Cloud TPU Pod (v2-128) |128 |16384 |7116 |
|1/2 Cloud TPU Pod (v2-256) |256 |32768 |3558 |
|Full Cloud TPU Pod (v2-512)|512 |32768 |3558 |

In addition, add the following additional parameters for batch sizes >= 16384 to enable the Layer-wise Adaptive Rate Scaling optimizer and label smoothing changes needed for large-batch training: --enable_lars=True --label_smoothing=0.1.

This script runs a total of 91 epochs (one warm-up epoch and 90 training epochs). For each result, we run five complete training runs of the script and report the median elapsed time. All of the runs using the above BATCH_SIZE and ITERATIONS parameters were observed to reach 76% top-1 accuracy.

After training completes, we then measure and report the training time using the method specified in the “Training epochs” section of this methodology. To implement the methodology, we have provided a script for use here. The warmup_steps parameter (corresponding to one training epoch) used are as follows:

|TPU Type |warmup_steps|
|---------------------------|--------------|
|Cloud TPU v2 |1251 |
|1/16 Cloud TPU Pod (v2-32) |313 |
|1/4 Cloud TPU Pod (v2-128) |78 |
|1/2 Cloud TPU Pod (v2-256) |39 |
|Full Cloud TPU Pod (v2-512)|39 |

A sample command is as follows:

python ~/tpu/models/official/resnet/benchmark/read_training_time.py \
  --model_dir=gs://RESNET_CHECKPOINT_BUCKET/DIRECTORY
  --warmup_steps=WARMUP_STEPS
  --tpu=True

6. Start ResNet evaluation using the following command line on the GCE VM. Note that evaluation is only supported on a single Cloud TPU at present.

python ~/tpu/models/official/resnet/resnet_main.py \
  --tpu=MY_TPU_NAME --tpu_zone=MY_TPU_ZONE --num_cores=8 \
  --data_dir=gs://IMAGENET_DATA_BUCKET/DIRECTORY \
  --model_dir=gs://RESNET_CHECKPOINT_BUCKET/DIRECTORY \
  --train_batch_size=BATCH_SIZE --iterations_per_loop=ITERATIONS \
  --train_steps=ITERATIONS \
  --mode=eval --eval_batch_size=1000

At the end of the evaluation, you should see the evaluation results printed out in the following format:

Eval results: {'loss': 2.2301836, 'top_1_accuracy': 0.76658, 'global_step': 28151, 'top_5_accuracy': 0.93422}. Elapsed seconds: 33

---

Benchmarks/ShapeMask Performance Comparison TensorFlow 1.14 GCP (benchmarks/ShapeMask_Performance_Comparison_TensorFlow_1.14_GCP.md) Copy Section

Methodology for ShapeMask Performance Benchmark on Cloud TPUs

Weicheng Kuo, Anelia Angelova, Pengchong Jin, Zak Stone, Omkar Pathak, Tsungyi Lin (Google Brain) (order TBD).

Performance Measurement Copy Chapter

Implementation Details Copy Section

All experiments are performed on Google’s Cloud TPU v3-8 device (batch size = 64) and larger slices of Cloud TPU v3 pods (batch_size > 64). TPU v3 offers significant speedup over v2, so we use it to demonstrate the speed of our system. This implementation uses bfloat16 numerics and input data transpose to improve TPU performance.

Benchmark Results Copy Chapter

|Model|Batch Size|Number of Cores|Mask AP|Box AP|Training Time (mins)|
|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|
|Mask R-CNN|64|8|37.3|42.1|730|
|ShapeMask|64|8|38.0|41.6|485|
|ShapeMask|256|32|37.9|41.5|187|
|ShapeMask|1024|128|35.1|37.9|51|
|ShapeMask|2048|256|34.7|37.1|36|

Benchmark Configurations Copy Section

|Model|Batch Size|Number of Cores|Total Steps|Warmup Steps|Initial Learning Rate|First Decay Step|Second Decay Step|
|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|
|Mask R-CNN|64|8|45000|500|0.08|30000|40000|
|ShapeMask|64|8|45000|500|0.08|30000|40000|
|ShapeMask|256|32|11250|1600|0.24|7500|10000|
|ShapeMask|1024|128|2813|1093|0.64|2188|2656|
|ShapeMask|2048|256|1800|600|0.64|1200|1600|

Commands Copy Section

Download code and dependencies:

Install packages

sudo apt-get install -y python-tk && \
pip install --user Cython matplotlib opencv-python-headless pyyaml Pillow && \
pip install --user 'git+https://github.com/cocodataset/cocoapi#egg=pycocotools&subdirectory=PythonAPI'
Download the code base.

git clone https://github.com/tensorflow/tpu/

Download and prepare data:

export USER=weicheng  # Your user name.
Export the bucket path to env.

export STORAGE_BUCKET=gs://${USER}-data
Create storage bucket.

gcloud storage buckets create $STORAGE_BUCKET
Download COCO data.

mkdir ~/data
mkdir ~/data/coco
cd ~/tpu/tools/datasets
bash download_and_preprocess_coco.sh ~/data/coco
Create coco directory under the bucket.

mkdir coco
touch coco/empty.txt
gcloud storage cp --recursive coco $STORAGE_BUCKET
Move data over to bucket.

gcloud storage cp data/coco/*.tfrecord gs://${USER}-data/coco
gcloud storage cp data/coco/raw-data/annotations/*.json gs://${USER}-data/coco
Create shapemask directory under the bucket.

mkdir shapemask_exp
touch shapemask_exp/empty.txt
gcloud storage cp --recursive shapemask_exp gs://${USER}-data/
Back to home directory.

cd ~

Setup environment variables:

export TPU_NAME=''  # Your tpu name.
export EVAL_TPU_NAME=''  # Your evaluation tpu name. tf.Estimator only supports 2x2 at the moment.
export EXP_NAME=shapemask_demo_run  # Your experiment name.
export MODEL_DIR=${STORAGE_BUCKET}/shapemask_exp/${EXP_NAME};  # You must have created shapemask directory under the bucket.
export RESNET_CHECKPOINT=gs://cloud-tpu-checkpoints/shapemask/retinanet/resnet101-checkpoint-2018-02-24;
export TRAIN_FILE_PATTERN=${STORAGE_BUCKET}/coco/train-*;  # Make sure coco directory exists under your bucket.
export EVAL_FILE_PATTERN=${STORAGE_BUCKET}/coco/val-*;
export VAL_JSON_FILE=${STORAGE_BUCKET}/coco/instances_val2017.json;
export SHAPE_PRIOR_PATH=gs://cloud-tpu-checkpoints/shapemask/kmeans_class_priors_91x20x32x32.npy
export PYTHONPATH="/home/${USER}/tpu/models"

Training commands:

/ Detailed source-code truncated for AI context efficiency. /

Evaluation Commands:

python ~/tpu/models/official/detection/main.py \
--model shapemask --use_tpu=True --tpu=${EVAL_TPU_NAME} \
--num_cores=8 --model_dir="${MODEL_DIR}" --mode="eval" \
--params_override="{resnet: {resnet_depth: 101}, \
eval: { val_json_file: ${VAL_JSON_FILE}, eval_file_pattern: ${EVAL_FILE_PATTERN}, eval_samples: 5000 }, \
shapemask_head: {use_category_for_mask: true, shape_prior_path: ${SHAPE_PRIOR_PATH}}, \
shapemask_parser: {output_size: [1024, 1024]}}"

At the end of the evaluation, you should see the evaluation results printed out in the following format:

/ Detailed source-code truncated for AI context efficiency. /

---

Models/Experimental/Cifar Keras/README (models/experimental/cifar_keras/README.md) Copy Section

Cifar Keras #

This directory contains an example using the Keras layers API inside an
Estimator/TPUEstimator. If you have a complete Keras model already built,
consider the new experimental Cloud TPU-Keras integration available since TF
1.9. For examples, see models/experimental/keras

---

Models/Experimental/Dcgan/README (models/experimental/dcgan/README.md) Copy Section

Overview Copy Chapter

This example uses a DCGAN architecture to learn to produce MNIST digits and
CIFAR10 images. It trains on Google Cloud TPUs. It uses an open source library
called TF-GAN to abstract away many of the GAN and TPU infrastructure details.

To run this example, be sure to install TF-GAN with:

pip install tensorflow-gan

---

Models/Experimental/Deeplab/README (models/experimental/deeplab/README.md) Copy Section

Deeplab on TPU

Prerequisites Copy Chapter

Setup a Google Cloud project Copy Section

Follow the instructions at the Quickstart Guide
to get a GCE VM with access to Cloud TPU.

To run this model, you will need:

* A GCE VM instance with an associated Cloud TPU resource
* A GCS bucket to store your training checkpoints
* A GCS bucket to store your training and evaluation data.

Setup Deeplab under tensorflow/models Copy Section

Deeplab on Cloud TPU depends on Deeplab under tensorflow/models. Please follow the instructions to add the library to PYTHONPATH and test the installation.

You can use their script to download PASCAL VOC 2012 semantic segmentation dataset and convert it to TFRecord.

You can download their pretrained checkpoints. In particular, we use a modified resnet 101 pretrained on ImageNet below.

Train and Eval Copy Chapter

python main.py \
--mode='train' \
--num_shards=8 \
--train_split='train' \
--alsologtostderr=true \
--model_dir=${MODEL_DIR} \
--dataset_dir=${DATASET_DIR} \
--init_checkpoint=${INIT_CHECKPOINT} \
--model_variant=resnet_v1_101_beta \
--image_pyramid=1. \
--aspp_with_separable_conv=false \
--multi_grid=1,2,4 \
--decoder_use_separable_conv=false

---

Models/Experimental/Densenet Keras/README (models/experimental/densenet_keras/README.md) Copy Section

Cloud TPU Port of DenseNet using Keras API

This folder contains an implementation of the DenseNet
image classification model using the Keras API.

Running the model on ImageNet Copy Chapter

The process for running on ImageNet is similar, just specify the directory
containing your converted tfrecord files:

python densenet_keras_imagenet.py\
  --alsologtostderr\
  --num_shards=8\
  --batch_size=1024\
  --master=grpc://$TPU_WORKER:8470\
  --use_tpu=1\
  --model_dir=gs://my-cloud-bucket/models/densenet-keras-imagenet/0\
  --data_dir=gs://my-cloud-bucket/data/imagenet
---
Models/Experimental/Embedding/README (models/experimental/embedding/README.md)  Copy Section
TPU Embedding example model
Prerequisites  Copy Chapter
Setup a Google Cloud project  Copy Section
Follow the instructions at the Quickstart Guide
to get a GCE VM with access to Cloud TPU.
To run this model, you will need:
* A GCE VM instance with an associated Cloud TPU resource. It might be helpful if the VM has a large number of CPUs and large memory as it is used for generating training and evaluation data.
* A GCS bucket to store data.
Setup Model  Copy Chapter

Clone the tpu respository and move to the example directory:

Setup a Google Cloud Bucket for your training data and model storage:

Create a new embedding subdirectory in your bucket.
Run the training data generator  Copy Chapter

Train and Eval  Copy Chapter

---
Models/Experimental/Inception/Inception V3 K8s.Yaml (models/experimental/inception/inception_v3_k8s.yaml)  Copy Section
Train Inception v3 with fake ImageNet dataset using Cloud TPU and Google

Kubernetes Engine.

#
[Training Data]

  In this example, we use the randomly generated fake ImageNet dataset at

  gs://cloud-tpu-test-datasets/fake_imagenet as the training data.

#
[Instructions]

  1. Follow the instructions on https://cloud.google.com/tpu/docs/kubernetes-engine-setup

     to create a Kubernetes Engine cluster.

  2. Change the environment variable MODEL_BUCKET in the Job spec to the

     Google Cloud Storage location where you want to store the output model.

  3. Run kubectl create -f inception_v3_k8s.yaml.
apiVersion: batch/v1
kind: Job
metadata:
  name: inception-v3-tpu
spec:
  template:
    metadata:
      annotations:
        # The Cloud TPUs that will be created for this Job must support
        # TensorFlow 1.11. This version MUST match the TensorFlow version that
        # your model is built on.
        tf-version.cloud-tpus.google.com: "1.11"
    spec:
      restartPolicy: Never
      containers:
      - name: inception-v3-tpu
        # The official TensorFlow 1.11 TPU model image built from https://github.com/tensorflow/tpu/blob/r1.11/tools/docker/Dockerfile.
        image: gcr.io/tensorflow/tpu-models:r1.11
        command:
          - python
          - /tensorflow_tpu_models/models/experimental/inception/inception_v3.py
          - --learning_rate=0.165
          - --train_steps=250000
          - --iterations=500
          - --use_data=real
          - --mode=train_and_eval
          - --train_steps_per_eval=2000
          - --data_dir=$(DATA_BUCKET)
          - --model_dir=$(MODEL_BUCKET)
        env:
          # The Google Cloud Storage location where the fake ImageNet dataset is
          # stored.
        - name: DATA_BUCKET
          value: "gs://cloud-tpu-test-datasets/fake_imagenet"
          # [REQUIRED] Must specify the Google Cloud Storage location where your
          # output model will be stored.
        - name: MODEL_BUCKET
          value: "gs://<my-model-bucket>/inception_v3"
        resources:
          limits:
            # Request a single v2-8 Cloud TPU device to train the model.
            # A single v2-8 Cloud TPU device consists of 4 chips, each of which
            # has 2 cores, so there are 8 cores in total.
            cloud-tpus.google.com/v2: 8
---
Models/Experimental/Inference/Api Config.Yaml (models/experimental/inference/api_config.yaml)  Copy Section
Copyright 2019 Google Inc. All Rights Reserved.

#
Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

#
    http://www.apache.org/licenses/LICENSE-2.0

#
Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.
#
A TF Serving API configuration.

#
Below, replace MY_PROJECT_ID with your Google Cloud Project ID.

#
The configuration schema is defined by service.proto file

https://github.com/googleapis/googleapis/blob/master/google/api/service.proto

type: google.api.Service
config_version: 3
#
Name of the service configuration.

'tf-tpu-serving' is the endpoint name. Users are free to rename it.

#
name: tf-tpu-serving.endpoints.<MY_PROJECT_ID>.cloud.goog
#
API title to appear in the user interface (Google Cloud Console).

#
title: tf-tpu-serving
apis:
- name: tensorflow.serving.PredictionService
#
API usage restrictions.

#
usage:
  rules:
  - selector: tensorflow.serving.PredictionService.Predict
    allow_unregistered_calls: true
---
Models/Experimental/Inference/Openapi.Yaml (models/experimental/inference/openapi.yaml)  Copy Section
Copyright 2019 The TensorFlow Authors. All Rights Reserved.

#
Licensed under the Apache License, Version 2.0 (the "License");

you may not use this file except in compliance with the License.

You may obtain a copy of the License at

#
    http://www.apache.org/licenses/LICENSE-2.0

#
Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

==============================================================================

---
swagger: '2.0'
info:
  title: <ENDPOINT_NAME>-<MODEL_NAME>-<MY_PROJECT_ID>
  version: 0.0.1
host: <ENDPOINT_NAME>.endpoints.<MY_PROJECT_ID>.cloud.goog
schemes:
- http
- https
consumes:
- application/json
produces:
- application/json
paths:
  "/v1/models/<MODEL_NAME>:predict":
    post:
      summary: Predict -- provides access to loaded TensorFlow model.
      operationId: Predict-REST
      responses:
        '200':
          description: ''
          schema:
            "$ref": "#/definitions/servingPredictResponse"
      parameters:
      - name: body
        in: body
        required: true
        schema:
          "$ref": "#/definitions/predictRequest"
      tags:
      - PredictionService
definitions:
  TensorShapeProtoDim:
    type: object
    properties:
      size:
        type: string
        format: int64
        description: |-
          Size of the tensor in that dimension.
      name:
        type: string
        description: Optional name of the tensor dimension.
    description: One dimension of the tensor.
  predictRequest:
    type: object
    properties:
      signature_name:
        type: string
        description: signature for inference
      inputs:
        type: array
        items:
          "$ref": "#/definitions/encodedImage"
    description: |-
      PredictRequest specifies which TensorFlow model to run.
  encodedImage:
    type: object
    additionalProperties:
      type: string
  servingPredictResponse:
    type: object
    properties:
      outputs:
        type: object
        additionalProperties:
          "$ref": "#/definitions/tensorflowTensorProto"
        description: Output tensors.
    description: Response for PredictRequest on successful run.
  tensorflowDataType:
    type: string
    enum:
    - DT_INVALID
    - DT_FLOAT
    - DT_DOUBLE
    - DT_INT32
    - DT_UINT8
    - DT_INT16
    - DT_INT8
    - DT_STRING
    - DT_COMPLEX64
    - DT_INT64
    - DT_BOOL
    - DT_QINT8
    - DT_QUINT8
    - DT_QINT32
    - DT_BFLOAT16
    - DT_QINT16
    - DT_QUINT16
    - DT_UINT16
    - DT_COMPLEX128
    - DT_HALF
    - DT_RESOURCE
    - DT_FLOAT_REF
    - DT_DOUBLE_REF
    - DT_INT32_REF
    - DT_UINT8_REF
    - DT_INT16_REF
    - DT_INT8_REF
    - DT_STRING_REF
    - DT_COMPLEX64_REF
    - DT_INT64_REF
    - DT_BOOL_REF
    - DT_QINT8_REF
    - DT_QUINT8_REF
    - DT_QINT32_REF
    - DT_BFLOAT16_REF
    - DT_QINT16_REF
    - DT_QUINT16_REF
    - DT_UINT16_REF
    - DT_COMPLEX128_REF
    - DT_HALF_REF
    - DT_RESOURCE_REF
    default: DT_INVALID
    description: |-
      - DT_INVALID: Not a legal value for DataType.  Used to indicate a DataType field
      has not been set.
       - DT_FLOAT: Data types that all computation devices are expected to be
      capable to support.
       - DT_FLOAT_REF: Only for parameters.
    title: LINT.IfChange
  tensorflowResourceHandle:
    type: object
    properties:
      device:
        type: string
        description: Unique name for the device containing the resource.
      container:
        type: string
        description: Container in which this resource is placed.
      name:
        type: string
        description: Unique name of this resource.
      hash_code:
        type: string
        format: uint64
        description: |-
          Hash code for the type of the resource. Is only valid in the same device
          and in the same execution.
      maybe_type_name:
        type: string
        description: |-
          For debug-only, the name of the type pointed to by this handle, if
          available.
    description: |-
      Protocol buffer representing a handle to a tensorflow resource.
  tensorflowTensorProto:
    type: object
    properties:
      dtype:
        "$ref": "#/definitions/tensorflowDataType"
      tensor_shape:
        "$ref": "#/definitions/tensorflowTensorShapeProto"
        description: 'Shape of the tensor.'
      version_number:
        type: integer
        format: int32
        description: |-
          Version number.
      tensor_content:
        type: string
        format: byte
        description: |-
          Serialized raw tensor content from either Tensor::AsProtoTensorContent or
          memcpy in tensorflow::grpc::EncodeTensorToByteBuffer.
      half_val:
        type: array
        items:
          type: integer
          format: int32
        description: |-
          DT_HALF. Note that since protobuf has no int16 type, we'll have some
          pointless zero padding for each value here.
      float_val:
        type: array
        items:
          type: number
          format: float
        description: DT_FLOAT.
      double_val:
        type: array
        items:
          type: number
          format: double
        description: DT_DOUBLE.
      int_val:
        type: array
        items:
          type: integer
          format: int32
        description: DT_INT32, DT_INT16, DT_INT8, DT_UINT8.
      string_val:
        type: array
        items:
          type: string
          format: byte
        title: DT_STRING
      scomplex_val:
        type: array
        items:
          type: number
          format: float
        description: |-
          DT_COMPLEX64. scomplex_val(2i) and scomplex_val(2i+1) are real
          and imaginary parts of i-th single precision complex.
      int64_val:
        type: array
        items:
          type: string
          format: int64
        title: DT_INT64
      bool_val:
        type: array
        items:
          type: boolean
          format: boolean
        title: DT_BOOL
      dcomplex_val:
        type: array
        items:
          type: number
          format: double
        description: |-
          DT_COMPLEX128. dcomplex_val(2i) and dcomplex_val(2i+1) are real
          and imaginary parts of i-th double precision complex.
      resource_handle_val:
        type: array
        items:
          "$ref": "#/definitions/tensorflowResourceHandle"
        title: DT_RESOURCE
    description: Protocol buffer representing a tensor.
  tensorflowTensorShapeProto:
    type: object
    properties:
      dim:
        type: array
        items:
          "$ref": "#/definitions/TensorShapeProtoDim"
        description: |-
          Dimensions of the tensor, such as {"input", 30}, {"output", 40}
          for a 30 x 40 2D tensor.  If an entry has size -1, this
          corresponds to a dimension of unknown size. The names are
          optional.
          The order of entries in "dim" matters: It indicates the layout of the
          values in the tensor in-memory representation.
          The first entry in "dim" is the outermost dimension used to layout the
          values, the last entry is the innermost dimension.  This matches the
          in-memory layout of RowMajor Eigen tensors.
          If "dim.size()" > 0, "unknown_rank" must be false.
      unknown_rank:
        type: boolean
        format: boolean
        description: |-
          If true, the number of dimensions in the shape is unknown.
          If true, "dim.size()" must be 0.
    description: Dimensions of a tensor.
---
Models/Experimental/Inference/Docker/README (models/experimental/inference/docker/README.md)  Copy Section
TensorFlow Serving with TPU VM Example
This contains an experimental fork of TensorFlow Serving Dockerfiles specifically for usage with TPU VM.
You can use TensorFlow Serving with TPU VMs in the same way as you can use TensorFlow serving with CPU/GPU VMs.
This document assumes all commands are being run on a TPU VM, e.g. created with:

For more information about using TensorFlow Serving with Docker, please refer to TensorFlow Serving with Docker.
For more information about TPU VMs and Cloud TPUs, please refer to the official Cloud TPU Documentation.

Example Usage
The following instructions demonstrate how you can use the provided Dockerfiles to
create your own model server running on TPU VMs.
Set sample environment variables  Copy Chapter

Build TensorFlow Serving for TPU VM  Copy Chapter
Start by building the base Docker image for TF serving.

Next, build the model server container.

* Note: this uses a version of TensorFlow that is fixed at a known stable commit.
Start the model server  Copy Chapter

Make sure you set MODEL_NAME.

---
Models/Experimental/Inference/Load Test/README (models/experimental/inference/load_test/README.md)  Copy Section
MLPerf inference benchmark for Vertex Prediction
This folder containers a tool for benchmarking models deployed on
Vertex Prediction or running locally on TensorFlow Model Server using
MLPerf inference loadgen.
Example Usage  Copy Chapter
The following instructions demonstrate how to run MLPerf load test against a Vertex AI Endpoint.
Deploy model to Vertex AI  Copy Section
In order to deploy the NLP model used in the MLPerf NLP benchmark
please follow the official documentation.
Run the benchmark from Docker container  Copy Section
Running the benchmark involves building a docker image and executing the load test from it.

Build the load test image

Start the container in interactive mode

After docker container is built, you can run benchmark from docker container.
Run the benchmark locally  Copy Section
Alternatively you can manually setup your environment and run benchmark from
the environment you are using. This is more convenient if you want to run
benchmark from Colab or Jupyter Notebook.

Install dependencies.

Download and build MLPerf loadgen.

See https://github.com/mlcommons/inference/tree/master/loadgen/demos/lon for details.

Download benchmark tool

#### Run benchmark
The commands to run benchmarks from docker container or from local environment
are same.

Obtain GCP user credentials. Follow the instructions on the screen.

You might not need to run this if you are running from Colab or Jupyter Notebook

that is already configured to use your project.

Set parameters.

Run the benchmark against Vertex AI Endpoint.

The gRPC protocol will only work with private endpoints. Please follow Setup private endpoint for online prediction section from Vertex AI Samples to set up a private endpoint.
See examples/loadgen_vertex_main.py for all available flags.
<sub>Readme author: cezarym@</sub>
---
Models/Experimental/Keras Colab/README (models/experimental/keras_colab/README.md)  Copy Section
Keras Colab #
This directory contains an example of using the experimental Cloud TPU-Keras
integration that was added in TF 1.9 in an interactive collaboratory
environment. To learn more about this new integration,
check out the documentation (coming soon!).
---
Models/Experimental/Ncf/README (models/experimental/ncf/README.md)  Copy Section
Neural Collaborative Filtering (NCF) on TPU
Prerequisites  Copy Chapter
Setup a Google Cloud project  Copy Section
Follow the instructions at the Quickstart Guide
to get a GCE VM with access to Cloud TPU.
To run this model, you will need:
* A GCE VM instance with an associated Cloud TPU resource. It might be helpful if the VM has a large number of CPUs and large memory as it is used for generating training and evaluation data. TF nightly is required.
* A GCS bucket to store data. To avoid downloading MovieLens dataset, you can copy it from gs://ncf/data_dir.
Setup NCF from tensorflow/models  Copy Section
Neural collaborative filtering on Cloud TPU depends on the same model under tensorflow/models. In your working directory, run git clone https://github.com/tensorflow/models.git, and add models/ to your python path by running export PYTHONPATH=$PYTHONPATH:/your/working/directory/models/.
Setup NCF  Copy Chapter

Copy ./ncf_main.py to your working directory.

Setup a Google Cloud Bucket for your training data and model storage:

Run the training data generator  Copy Chapter
From the models/ directory run the command:

This will download an preprocess your data and take several minutes to process
the data.
NOTE The pre-batch sizes must be the same as the --batch_size and
--eval_batch_size passed to ncf_main.py divided by the value of
--num_tpu_shards (the number of TPU cores being trained on). By default this
model trains on a single host with 8 TPU cores, giving the pre-batch sizes
above.
Train and Eval  Copy Chapter

Most of the time, the hit rate metric (HR) reaches 0.635 in around 10 epochs.
---
Models/Experimental/Resnet50 Keras/README (models/experimental/resnet50_keras/README.md)  Copy Section
ResNet-50 #
This directory contains an example of using the experimental Cloud TPU-Keras
integration that was added in TF 1.9. ResNet-50 is a commonly used convolutional
neural network used for image classification. The ResNet-family of models were
introduced for the ImageNet 2015 competition and performed very well there.
---
Models/Official/Amoeba Net/README (models/official/amoeba_net/README.md)  Copy Section
AmoebaNet-D on TPU
This code was implemented based on results in the AmoebaNet paper, which should
be cited as: Real, E., Aggarwal, A., Huang, Y. and Le, Q.V., 2018. Regularized
Evolution for Image Classifier Architecture Search. arXiv preprint
arXiv:1802.01548.

Acknowledgements  Copy Chapter
The starting point for this code was branched from the implementation for NASNet
in https://github.com/tensorflow/models/tree/master/research/slim/nets/nasnet
and from image processing code in
https://github.com/tensorflow/tpu/blob/master/models/experimental/inception/inception_preprocessing.py.

Prerequisites  Copy Chapter
Setup a Google Cloud project  Copy Section
Follow the instructions at the Quickstart Guide
to get a GCE VM with access to Cloud TPU.
To run this model, you will need:
* A GCE VM instance with an associated Cloud TPU resource
* A GCS bucket to store your training checkpoints
* (Optional): The ImageNet training and validation data preprocessed into
  TFRecord format, and stored in GCS.
Installing extra packages  Copy Section
The AmoebaNet trainer uses a few extra packages.  We can install them now:

Formatting the data  Copy Section
The data is expected to be formatted in TFRecord format, as generated by this
script.
If you do not have ImageNet dataset prepared, you can use a randomly generated
fake dataset to test the model. It is located at
gs://cloud-tpu-test-datasets/fake_imagenet.

Training the model  Copy Chapter
Train the model by executing the following command (substituting the appropriate
values):

If you are not running this script on a GCE VM in the same project and zone as
your Cloud TPU, you will need to add the --project and --zone flags
specifying the corresponding values for the Cloud TPU you'd like to use.
This will train an AmoebaNet-D model on ImageNet with 256 batch size on a
single Cloud TPU. With the default flags on everything, the model should train
to above 80% accuracy in under 48 hours (including evaluation time every
few epochs).
You can launch TensorBoard (e.g. tensorboard -logdir=$MODEL_DIR) to view loss
curves and other metadata regarding your training run. (Note: if you launch
on your VM, be sure to configure ssh port forwarding or the GCE firewall rules
appropriately.)
You can also train the AmoebaNet-D model to 93% top-5 accuracy in under 7.5
hours using the following command:

Understanding the code  Copy Chapter
For more detailed information, read the documentation within each file.
Additional notes  Copy Chapter
About the model and training regime  Copy Section
The model is the result of evolutionary neural architecture search presented
in Regularized Evolution for Image Classifier
Search.
TODO: give some more details
---
Models/Official/Amoeba Net/Amoeba Net K8s.Yaml (models/official/amoeba_net/amoeba_net_k8s.yaml)  Copy Section
Train AmoebaNet-D with fake ImageNet dataset using Cloud TPU and Google

Kubernetes Engine.

#
[Training Data]

  In this example, we use the randomly generated fake ImageNet dataset at

  gs://cloud-tpu-test-datasets/fake_imagenet as the training data.

#
[Instructions]

  1. Follow the instructions on https://cloud.google.com/tpu/docs/kubernetes-engine-setup

     to create a Kubernetes Engine cluster.

     Note: Use a base machine type with more memory than the default n1-standard-1.

  2. Change the environment variable MODEL_BUCKET in the Job spec to the

     Google Cloud Storage location where you want to store the output model.

  3. Run kubectl create -f amoeba_net_k8s.yaml.
apiVersion: batch/v1
kind: Job
metadata:
  name: amoeba-net-tpu
spec:
  template:
    metadata:
      annotations:
        # The Cloud TPUs that will be created for this Job must support
        # TensorFlow 1.11. This version MUST match the TensorFlow version that
        # your model is built on.
        tf-version.cloud-tpus.google.com: "1.11"
    spec:
      restartPolicy: Never
      containers:
      - name: amoeba-net-tpu
        # The official TensorFlow 1.11 TPU model image built from https://github.com/tensorflow/tpu/blob/r1.11/tools/docker/Dockerfile.
        image: gcr.io/tensorflow/tpu-models:r1.11
        command:
          - python
          - /tensorflow_tpu_models/models/official/amoeba_net/amoeba_net.py
          - --data_dir=$(DATA_BUCKET)
          - --model_dir=$(MODEL_BUCKET)
        env:
          # The Google Cloud Storage location where the fake ImageNet dataset is
          # stored.
        - name: DATA_BUCKET
          value: "gs://cloud-tpu-test-datasets/fake_imagenet"
          # [REQUIRED] Must specify the Google Cloud Storage location where your
          # output model will be stored.
        - name: MODEL_BUCKET
          value: "gs://<my-model-bucket>/amoeba_net"
        resources:
          limits:
            # Request a single v2-8 Cloud TPU device to train the model.
            # A single v2-8 Cloud TPU device consists of 4 chips, each of which
            # has 2 cores, so there are 8 cores in total.
            cloud-tpus.google.com/v2: 8
---
Models/Official/Bert/README (models/official/bert/README.md)  Copy Section
See https://github.com/google-research/bert/blob/master/README.md
---
Models/Official/Densenet/README (models/official/densenet/README.md)  Copy Section
Cloud TPU Port of DenseNet
This folder contains an implementation of the DenseNet
image classification model.
Running the model on ImageNet  Copy Chapter
The process for running on ImageNet is similar, just specify the directory
containing your converted tfrecord files:

---

Models/Official/Detection/README (models/official/detection/README.md) Copy Section

TPU Object Detection and Segmentation Framework

TPU Object Detection and Segmentation Framework provides implementations of
common image classification, object detection and instance segmentation models in Tensorflow.
Our models produce the competitive results,
can be trained on multiple platforms including GPU and TPUs,
and have been highly optimized for TPU performance.
It also features latest research including
Auto-Augument,
NAS-FPN,
ShapeMask, and
SpineNet.

Instance segmentation results of our Mask R-CNN model.

Updates Copy Chapter

Major Features Copy Chapter

* Tasks:
- Image classification
- Object detection
- Instance segmentation
* Meta-architectures:
- RetinaNet
- Faster / Mask R-CNN
- ShapeMask
* Backbones:
- ResNet
- SpineNet
* Feature pyramids:
- FPN
- NAS-FPN
* Other model features:
- Auto-Augment
* Training platforms:
- Single machine GPUs
- Cloud TPU
- Cloud TPU Pods

Model Zoo Copy Chapter

MODEL_ZOO.md
provides a large collection of baselines and checkpoints for object detection, instance segmentation, and image classification.

Get started Copy Chapter

Please follow the instructions in GETTING_STARTED.md.

---

Models/Official/Detection/GETTING STARTED (models/official/detection/GETTING_STARTED.md) Copy Section

Getting started

Installation Copy Chapter

To get started, make sure you install Tensorflow 1.15+.

* For GPU training, make sure it has the GPU support. See the guideline by Tensorflow.

pip3 install tensorflow-gpu==1.15  # GPU

* For Cloud TPU / TPU Pods training, make sure Tensorflow 1.15+ is pre-installed in your Google Cloud VM.

Also, there are a few packages that you need to install.

sudo apt-get install -y python-tk && \
pip3 install --user Cython matplotlib opencv-python-headless pyyaml Pillow && \
pip3 install --user 'git+https://github.com/cocodataset/cocoapi#egg=pycocotools&subdirectory=PythonAPI'

Dataset download and convesion Copy Chapter

Next, download the latest code from tpu github repository.

git clone https://github.com/tensorflow/tpu/

The training expects the data in TFExample format stored in TFRecord.
Tools and scripts are provided to download and convert datasets.

| Dataset | Tool |
|:---------:|:-------------:|
| ImageNet | instructions |
| COCO | instructions |

Model Training Copy Chapter

We support both GPU training on a single machine, and Cloud TPU / TPU Pods training.
Below we provide sample commands to launch RetinaNet training on different platforms.

GPU training on a single machine Copy Section

MODEL_DIR="<path to the directory to store model files>"
TRAIN_FILE_PATTERN="<path to the TFRecord training data>"
EVAL_FILE_PATTERN="<path to the TFRecord validation data>"
VAL_JSON_FILE="<path to the validation annotation JSON file>"
RESNET_CHECKPOINT="gs://cloud-tpu-artifacts/resnet/resnet-nhwc-2018-10-14/model.ckpt-112602"
python ~/tpu/models/official/detection/main.py \
  --model="retinanet" \
  --model_dir="${MODEL_DIR?}" \
  --mode=train \
  --eval_after_training=True \
  --use_tpu=False \
  --params_override="{ train: { checkpoint: { path: ${RESNET_CHECKPOINT?}, prefix: resnet50/ }, train_file_pattern: ${TRAIN_FILE_PATTERN?} }, eval: { val_json_file: ${VAL_JSON_FILE?}, eval_file_pattern: ${EVAL_FILE_PATTERN?} } }"

Training on Cloud TPU Copy Section

To train this model on Cloud TPU, you will need:

* A GCE VM instance with an associated Cloud TPU resource.
* A GCS bucket to store your training checkpoints (the --model_dir flag).
* Install TensorFlow 1.15+ for both GCE VM and Cloud TPU instances.

See the RetinaNet tutorial
for more instructuions about TPU training.

TPU_NAME="<your GCP TPU name>"
MODEL_DIR="<path to the directory to store model files>"
TRAIN_FILE_PATTERN="<path to the TFRecord training data>"
EVAL_FILE_PATTERN="<path to the TFRecord validation data>"
VAL_JSON_FILE="<path to the validation annotation JSON file>"
RESNET_CHECKPOINT="gs://cloud-tpu-artifacts/resnet/resnet-nhwc-2018-10-14/model.ckpt-112602"
python ~/tpu/models/official/detection/main.py \
  --model="retinanet" \
  --model_dir="${MODEL_DIR?}" \
  --use_tpu=True \
  --tpu="${TPU_NAME?}" \
  --num_cores=8 \
  --mode=train \
  --eval_after_training=True \
  --params_override="{ train: { checkpoint: { path: ${RESNET_CHECKPOINT?}, prefix: resnet50/ }, train_file_pattern: ${TRAIN_FILE_PATTERN?} }, eval: { val_json_file: ${VAL_JSON_FILE?}, eval_file_pattern: ${EVAL_FILE_PATTERN?} } }"

Training on Cloud TPU Pods Copy Section

You can leverage large Cloud TPU Pods
in Google Cloud to significantly improve the training performance.

TPU_POD_NAME="<your GCP TPU name>"
NUM_CORES=<num cores in TPU pod>  # e.g. v3-32 offers 32 cores.
MODEL_DIR="<path to the directory to store model files>"
TRAIN_FILE_PATTERN="<path to the TFRecord training data>"
EVAL_FILE_PATTERN="<path to the TFRecord validation data>"
VAL_JSON_FILE="<path to the validation annotation JSON file>"
RESNET_CHECKPOINT="gs://cloud-tpu-artifacts/resnet/resnet-nhwc-2018-10-14/model.ckpt-112602"
CONFIG=""
python ~/tpu/models/official/detection/main.py \
  --model="retinanet" \
  --model_dir="${MODEL_DIR?}" \
  --use_tpu=True \
  --tpu="${TPU_POD_NAME?}" \
  --num_cores=${NUM_CORES} \
  --mode=train \
  --config_file="" \
  --params_override="{ train: { checkpoint: { path: ${RESNET_CHECKPOINT?}, prefix: resnet50/ }, train_file_pattern: ${TRAIN_FILE_PATTERN?} }, eval: { val_json_file: ${VAL_JSON_FILE?}, eval_file_pattern: ${EVAL_FILE_PATTERN?} } }"

Customize configurations Copy Section

The framework supports three levels of parameter overrides to accommodate different use cases.

1. <xxx>_config.py under ./configs directory.

This defines and sets the default values of all the parameters required by the particular model.

2. <xxx>.yaml and override through the --config_file flag.

This provides the first level override on top of the default defined by <xxx>_config.py.
One can use it to define a controlled experiment by
first defining a .yaml file as the template and passing to the --config_file flag
and then changing only one or two parameters using the --params_override flag.

3. parameters in JSON string and override through the --params_override flag.

This provides the final override on top of 1 and 2.

#### Example: Train RetinaNet using customized configurations.

First, create a YAML config file, e.g. my_retinanet.yaml,
to define training / evaluation dataset.

my_retinanet.yaml

type: 'retinanet'
train:
  train_file_pattern: <path to the TFRecord training data>
eval:
  eval_file_pattern: <path to the TFRecord validation data>
  val_json_file: <path to the validation annotation JSON file>

Override learning rate hyper-parameter via --params_override in the launch command.

python ~/tpu/models/official/detection/main.py \
  ... \
  --config_file="my_retinanet.yaml" \
  --params_override="{ train: { learnin_rate: { init_learning_rate: 0.2 } } }"

Model Export Copy Chapter

Export to SavedModel Copy Section

Given the checkpoint, one can easily export the SavedModel for serving using the following command.

EXPORT_DIR="<path to the directory to store the exported model>"
CHECKPOINT_PATH="<path to the checkpoint>"
PARAMS_OVERRIDE=""  # if any.
BATCH_SIZE=1
INPUT_TYPE="image_bytes"
INPUT_NAME="input"
INPUT_IMAGE_SIZE="640,640"
python ~/tpu/models/official/detection/export_saved_model.py \
  --export_dir="${EXPORT_DIR?}" \
  --checkpoint_path="${CHECKPOINT_PATH?}" \
  --params_override="${PARAMS_OVERRIDE?}" \
  --batch_size=${BATCH_SIZE?} \
  --input_type="${INPUT_TYPE?}" \
  --input_name="${INPUT_NAME?}" \
  --input_image_size="${INPUT_IMAGE_SIZE?}" \

Export to TF-lite Copy Section

Given the exported SavedModel, one can further convert it to the TF-lite format that can be deployed on mobile platform.

SAVED_MODEL_DIR="<path to the SavedModel directory>"
OUTPUT_DIR="<path to the directory to store the tflite model>"
python ~/tpu/models/official/detection/export_tflite_model.py \
  --saved_model_dir="${SAVED_MODEL_DIR?}" \
  --output_dir="${OUTPUT_DIR?}" \

Export to TensorRT Copy Section

Given the exported SavedModel, one can further convert it to the TensoRT format that can be deployed on GPU platform.

SAVED_MODEL_DIR="<path to the SavedModel directory>"
OUTPUT_DIR="<path to the output TensorRT SavedModel directory>"
python ~/tpu/models/official/detection/export_tensorrt_model.py \
  --saved_model_dir="${SAVED_MODEL_DIR?}" \
  --output_dir="${OUTPUT_DIR?}" \

Model Inference Copy Chapter

Use checkpoint Copy Section

Given the checkpoint, one can easily run the model inference using the following command.

MODEL="retinanet"
IMAGE_SIZE=640
CHECKPOINT_PATH="<path to the checkpoint>"
PARAMS_OVERRIDE=""  # if any.
LABEL_MAP_FILE="~/tpu/models/official/detection/datasets/coco_label_map.csv"
IMAGE_FILE_PATTERN="<path to the JPEG image that you want to run inference on>"
OUTPUT_HTML="./test.html"
python ~/tpu/models/official/detection/inference.py \
  --model="${MODEL?}" \
  --image_size=${IMAGE_SIZE?} \
  --checkpoint_path="${CHECKPOINT_PATH?}" \
  --label_map_file="${LABEL_MAP_FILE?}" \
  --image_file_pattern="${IMAGE_FILE_PATTERN?}" \
  --output_html="${OUTPUT_HTML?}" \
  --max_boxes_to_draw=10 \
  --min_score_threshold=0.05

Use SavedModel Copy Section

One can also use the exported SavedModel, which a bundle of model weights and
graph computation, to run inference.

SAVED_MODEL_DIR="<path to the SavedModel>"
LABEL_MAP_FILE="~/tpu/models/official/detection/datasets/coco_label_map.csv"
IMAGE_FILE_PATTERN="<path to the JPEG image that you want to run inference on>"
OUTPUT_HTML="./test.html"
python ~/tpu/models/detection/inference_saved_model \
  --saved_model_dir="${SAVED_MODEL_DIR?}" \
  --label_map_file="${LABEL_MAP_FILE?}" \
  --image_file_pattern="${IMAGE_FILE_PATTERN?}" \
  --output_html="${OUTPUT_HTML?}" \
  --max_boxes_to_draw=10 \
  --min_score_threshold=0.05

---

Models/Official/Detection/MODEL ZOO (models/official/detection/MODEL_ZOO.md) Copy Section

TPU Object Detection and Segmentation Model Zoo

Introduction Copy Chapter

Object Detection and Instance Segmentation Copy Chapter

Common Settings and Notes Copy Section

COCO Object Detection Baselines Copy Section

#### RetinaNet (Trained from scratch)
| model | resolution | epochs | FLOPs (B) | params (M) | V100 / P100 <br /> lat (ms/im) | box AP | download |
| ------------ |:-------------:| ---------:|-----------:|--------:|------:|---------:|-----------:|
| R50-FPN | 640x640 | 350 | 97.0 | 34.0 | 23 / 37 |40.4 |ckpt \| config|
| R101-FPN | 1024x1024 | 350 | 326.3 | 53.1 | 55 / 95 | 43.9 | ckpt \| config |
| R152-FPN | 1280x1280 | 350 | 630.5 | 68.7 | 100 / 167 |45.2 | ckpt \| config |
| R50-NAS-FPN | 640x640 | 72 | 140.6 | 60.3 | 29 / 48 |37.3 | N/A |
| R50-NAS-FPN | 640x640 | 350 | 140.6 | 60.3 | 29 / 48 |42.4 |ckpt \| config|
| SpineNet-49 | 640x640 | 72 | 85.4| 28.5 | 24 / 38 |37.7| N/A |
| SpineNet-49 | 640x640 | 350 | 85.4| 28.5 | 24 /38 |42.8|ckpt \| config |
| SpineNet-49S | 640x640 | 350 | 33.8 | 11.9 | 19 / 26|39.5 | ckpt \| config |
| SpineNet-96 | 1024x1024 | 350 | 265.4 | 43.0 | 53 / 87 |46.7 | ckpt \| config |
| SpineNet-143 | 1280x1280 | 350 | 524.0 | 67.0 |97 / 159 |48.0 | ckpt \| config |

SpineNet models trained with stochastic depth and swish activation for a longer shedule:
| model | resolution | epochs | FLOPs (B) | params (M) | box AP | download |
| ------------ |:-------------:| ---------:|-----------:|--------:|--------:|-----------:|
| SpineNet-49S | 640x640 | 500 | 33.8 | 11.9 |41.5 | ckpt \| config |
| SpineNet-49 | 640x640 | 500 | 85.4 | 28.5 |44.3 | ckpt \| config |
| SpineNet-96 | 1024x1024 | 500 | 265.4 | 43.0 | 48.5 | ckpt \| config |
| SpineNet-143 | 1280x1280 | 500 | 524.0 | 67.0 | 50.6 | ckpt \| config |
| SpineNet-190 | 1280x1280 | 400 | 1885.0 | 163.6 | 52.0 | ckpt \| config |

#### Mobile RetinaNet (Trained from scratch)
| model | resolution | epochs | FLOPs (B) | params (M) | box AP | download |
| --------------- |:-------------:| ----------:|-----------:|--------:|--------:|-----------:|
| SpineNetMB-49 | 384x384 | 600 | 1.0 | 2.34 | 28.6 | ckpt \| config |

Instance Segmentation Baselines Copy Section

#### Mask R-CNN (Trained from scratch)
| model | resolution | epochs | FLOPs (B) | params (M) | box AP | mask AP | download |
| ------------ |:-------------:| ---------:|-----------:|--------:|--------:|-----------:|-----------:|
| SpineNet-49 | 640x640 | 350 | 215.7 | 40.8 | 42.8 | 37.8 | ckpt \| config |
| SpineNet-96 | 1024x1024 | 350 | 314.6 | 55.2 | 46.8 | 41.2 | ckpt \| config |
| SpineNet-143 | 1280x1280 | 350 | 498.4 | 79.2 | 48.7 | 42.6 | ckpt \| config |

SpineNet-190 trained with stochastic depth and swish activation for a longer shedule:
| model | resolution | epochs | FLOPs (B) | params (M) | box AP | mask AP | download |
| ------------ |:-------------:| ---------:|-----------:|--------:|--------:|-----------:|-----------:|
| SpineNet-190 | 1536x1536 | 400 | 1685.7 | 168.2 | 52.0 | 45.9 | ckpt \| config |

Image Classification Copy Chapter

Common Settings and Notes Copy Section

ImageNet Baselines Copy Section

SpineNet models trained with stochastic depth, swish activation, and label smoothing:
| model | resolution | epochs | FLOPs (B) | params (M) | Top-1 | Top-5 | download |
| ------------ |:-------------:| ---------:|-----------:|--------:|--------:|---------:|-----------:|
| SpineNet-49 | 224x224 | 200 | 3.5 | 22.1 | 78.1 | 94.0 | ckpt \| config |
| SpineNet-96 | 224x224 | 200 | 5.7 | 36.5 | 79.4 | 94.6 | ckpt\| config|
| SpineNet-143 | 224x224 | 200 | 9.1 | 60.5 | 80.1 | 95.0 | ckpt \| config |
| SpineNet-190 | 224x224 | 200 | 19.1 | 127.1 | 80.8 | 95.3 | ckpt \| config |

iNaturalist-2017 Baselines Copy Section

SpineNet models trained with stochastic depth, swish activation, and label smoothing:
| model | resolution | epochs | FLOPs (B) | params (M) | Top-1 | Top-5 |
| ------------ |:-------------:| ---------:|-----------:|--------:|--------:|---------:|
| SpineNet-49 | 224x224 | 200 | 3.5 | 23.1 | 63.3 | 85.1 |
| SpineNet-96 | 224x224 | 200 | 5.7 | 37.6 | 64.7 | 85.9 |
| SpineNet-143 | 224x224 | 200 | 9.1 | 61.6 | 66.7 | 87.1 |
| SpineNet-190 | 224x224 | 200 | 19.1 | 129.2 | 67.6 | 87.4 |

---

Models/Official/Detection/Configs/Spinenet/Spinenet Mbconv49 Retinanet.Yaml (models/official/detection/configs/spinenet/spinenet-mbconv49_retinanet.yaml) Copy Section

SpineNet49-MBConv + RetinaNet with swish. 28.5% mAP.

---

Models/Official/Detection/Configs/Spinenet/Spinenet143 Classification.Yaml (models/official/detection/configs/spinenet/spinenet143_classification.yaml) Copy Section

SpineNet-143 ImageNet classification:

- 79.0 Top-1, 94.4 Top-5 accuracy with init_drop_connect_rate null, activation 'relu', label_smoothing 0.0

- 80.1 Top-1, 95.0 Top-5 accuracy with init_drop_connect_rate 0.2, activation 'swish', label_smoothing 0.1

---

Models/Official/Detection/Configs/Spinenet/Spinenet143 Mrcnn.Yaml (models/official/detection/configs/spinenet/spinenet143_mrcnn.yaml) Copy Section

SpineNet-143 + Mask R-CNN: box mAP: 48.79, mask mAP: 42.71

---

Models/Official/Detection/Configs/Spinenet/Spinenet143 Retinanet.Yaml (models/official/detection/configs/spinenet/spinenet143_retinanet.yaml) Copy Section

SpineNet-143 + RetinaNet:

- 48.0% mAP with init_drop_connect_rate null, activation 'relu', total_steps 162050, learning_rate_steps [148160, 157420], aug_scale_min 0.5, aug_scale_max 2.0

- 50.6% mAP with init_drop_connect_rate: 0.16, activation: 'swish', total_steps 231500, learning_rate_steps [217610, 226870], aug_scale_min 0.1, aug_scale_max 1.9

---

Models/Official/Detection/Configs/Spinenet/Spinenet190 Classification.Yaml (models/official/detection/configs/spinenet/spinenet190_classification.yaml) Copy Section

SpineNet-190 ImageNet classification:

- 80.8 Top-1, 95.3 Top-5 accuracy with init_drop_connect_rate 0.2, activation 'swish', label_smoothing 0.1

---

Models/Official/Detection/Configs/Spinenet/Spinenet190 Mrcnn.Yaml (models/official/detection/configs/spinenet/spinenet190_mrcnn.yaml) Copy Section

SpineNet-190 + Mask R-CNN with stochastic depth and swish activation. 52.0 mAP.

---

Models/Official/Detection/Configs/Spinenet/Spinenet190 Retinanet.Yaml (models/official/detection/configs/spinenet/spinenet190_retinanet.yaml) Copy Section

SpineNet-190 + RetinaNet:

- 52.0% mAP with init_drop_connect_rate: 0.2, activation: 'swish'

---

Models/Official/Detection/Configs/Spinenet/Spinenet49 Classification.Yaml (models/official/detection/configs/spinenet/spinenet49_classification.yaml) Copy Section

SpineNet-49 ImageNet classification:

- 77.0 Top-1, 93.3 Top-5 accuracy with init_drop_connect_rate null, activation 'relu', label_smoothing 0.0

- 78.1 Top-1, 94.0 Top-5 accuracy with init_drop_connect_rate 0.2, activation 'swish', label_smoothing 0.1

---

Models/Official/Detection/Configs/Spinenet/Spinenet49 Mrcnn.Yaml (models/official/detection/configs/spinenet/spinenet49_mrcnn.yaml) Copy Section

SpineNet-49 + Mask R-CNN: box mAP: 42.93, mask mAP: 38.07

---

Models/Official/Detection/Configs/Spinenet/Spinenet49 Retinanet.Yaml (models/official/detection/configs/spinenet/spinenet49_retinanet.yaml) Copy Section

SpineNet-49 + RetinaNet:

- 42.8% mAP with init_drop_connect_rate null, activation 'relu', total_steps 162050, learning_rate_steps [148160, 157420]

- 44.3% mAP with init_drop_connect_rate: 0.2, activation: 'swish', total_steps 231500, learning_rate_steps [217610, 226870]

---

Models/Official/Detection/Configs/Spinenet/Spinenet49s Mrcnn.Yaml (models/official/detection/configs/spinenet/spinenet49s_mrcnn.yaml) Copy Section

SpineNet-49S + Mask R-CNN: box mAP: 39.31, mask mAP: 34.83

---

Models/Official/Detection/Configs/Spinenet/Spinenet49S Retinanet.Yaml (models/official/detection/configs/spinenet/spinenet49S_retinanet.yaml) Copy Section

SpineNet-49S + RetinaNet:

- 39.7% mAP with init_drop_connect_rate null, activation 'relu', total_steps 162050, learning_rate_steps [148160, 157420]

- 41.5% mAP with init_drop_connect_rate: 0.2, activation: 'swish', total_steps 231500, learning_rate_steps [217610, 226870]

---

Models/Official/Detection/Configs/Spinenet/Spinenet96 Classification.Yaml (models/official/detection/configs/spinenet/spinenet96_classification.yaml) Copy Section

SpineNet-96 ImageNet classification:

- 78.2 Top-1, 94.0 Top-5 accuracy with init_drop_connect_rate null, activation 'relu', label_smoothing 0.0

- 79.4 Top-1, 94.6 Top-5 accuracy with init_drop_connect_rate 0.2, activation 'swish', label_smoothing 0.1

---

Models/Official/Detection/Configs/Spinenet/Spinenet96 Mrcnn.Yaml (models/official/detection/configs/spinenet/spinenet96_mrcnn.yaml) Copy Section

SpineNet-96 + Mask R-CNN: box mAP: 47.18, mask mAP: 41.53

---

Models/Official/Detection/Configs/Spinenet/Spinenet96 Retinanet.Yaml (models/official/detection/configs/spinenet/spinenet96_retinanet.yaml) Copy Section

SpineNet-96 + RetinaNet:

- 46.7% mAP with init_drop_connect_rate null, activation 'relu', total_steps 162050, learning_rate_steps [148160, 157420]

- 48.5% mAP with init_drop_connect_rate: 0.2, activation: 'swish', total_steps 231500, learning_rate_steps [217610, 226870]

architecture:
backbone: 'spinenet'
multilevel_features: 'identity'
train:
total_steps: 231500
train_batch_size: 256
learning_rate:
warmup_steps: 2000
init_learning_rate: 0.28
learning_rate_levels: [0.028, 0.0028]
learning_rate_steps: [217610, 226870]
l2_weight_decay: 0.00004
gradient_clip_norm: 10.0
batch_norm_activation:
use_sync_bn: true
activation: 'swish'
spinenet:
model_id: '96'
init_drop_connect_rate: 0.2
anchor:
anchor_size: 3.0
retinanet_parser:
output_size: [1024, 1024]
aug_scale_min: 0.5
aug_scale_max: 2.0

---

Models/Official/Detection/Configs/Yaml/Retinanet Autoaugment.Yaml (models/official/detection/configs/yaml/retinanet_autoaugment.yaml) Copy Section

---------- TRAINING PARAMETERS ----------

AutoAugment achieves best results when trained for long and when not using a pretrained

checkpoint for the backbone.

Right now the code uses a pretrained checkpoint, but a future version will change this.

To train the ResNet 101 or ResNet 200 version, simply change the 'resnet_depth' to 101 or 200.

Expected accuracy on ResNet 50 with using autoaugment: 38.0

Expected accuracy on ResNet 50 without using autoaugment: 36.4

resnet:
resnet_depth: 50

retinanet_parser:
aug_policy: 'v0'
aug_scale_min: 0.8
aug_scale_max: 1.2

---

Models/Official/Detection/Configs/Yaml/Retinanet Nasfpn.Yaml (models/official/detection/configs/yaml/retinanet_nasfpn.yaml) Copy Section

---------- RetianNet + NAS-FPN ----------

Expected accuracy with using NAS-FPN l3-l7 and image size 640x640: 39.5

architecture:
multilevel_features: 'nasfpn'

nasfpn:
fpn_feat_dims: 256
min_level: 3
max_level: 7
num_repeats: 5
use_separable_conv: False

retinanet_parser:
aug_scale_min: 0.8
aug_scale_max: 1.2

---

Models/Official/Detection/K8s/Retinanet K8s.Yaml (models/official/detection/k8s/retinanet_k8s.yaml) Copy Section

Train RetinaNet with COCO dataset using Cloud TPU and Google Kubernetes

Engine.

[Training Data]

Download and preprocess the COCO dataset using https://github.com/tensorflow/tpu/blob/r1.13/tools/datasets/download_and_preprocess_coco_k8s.yaml

if you don't already have the data.

[Instructions]

1. Follow the instructions on https://cloud.google.com/tpu/docs/kubernetes-engine-setup

to create a Kubernetes Engine cluster.

2. Change the environment variable TRAIN_FILE_PATTERN and MODEL_BUCKET below to the

Google Cloud Storage location where you downloaded the COCO dataset and

where you want to store the output model, respectively. For running eval as well

VAL_JSON_FILE and EVAL_FILE_PATTERN.

3. Run kubectl create -f retinanet_k8s.yaml.

apiVersion: batch/v1
kind: Job
metadata:
name: retinanet-tpu
spec:
template:
metadata:
annotations:
# The Cloud TPUs that will be created for this Job must support
# TensorFlow 1.13. This version MUST match the TensorFlow version that
# your model is built on.
tf-version.cloud-tpus.google.com: "1.13"
spec:
restartPolicy: Never
containers:
- name: retinanet-tpu
# The official TensorFlow 1.13 TPU model image built from:
# https://github.com/tensorflow/tpu/blob/r1.13/tools/docker/Dockerfile.
image: gcr.io/tensorflow/tpu-models:r1.13
command:
- /bin/sh
- -c
- >
DEBIAN_FRONTEND=noninteractive apt-get update &&
DEBIAN_FRONTEND=noninteractive apt-get install -y python-dev python-tk &&
pip install Cython matplotlib &&
pip install 'git+https://github.com/cocodataset/cocoapi#egg=pycocotools&subdirectory=PythonAPI' &&
python /tensorflow_tpu_models/models/official/detection/main.py
--use_tpu=True
--model_dir="${MODEL_DIR?}"
--num_cores=8
--mode=train
--eval_after_training=True
--params_override="{ type: retinanet, train: { checkpoint: { path: ${RESNET_CHECKPOINT?},
prefix: resnet50/ }, train_file_pattern: ${TRAIN_FILE_PATTERN?} },
eval: { val_json_file: ${VAL_JSON_FILE?}, eval_file_pattern: ${EVAL_FILE_PATTERN?},
eval_samples: 5000 } }"
env:
# [REQUIRED] Must specify the Google Cloud Storage location where the
# training data is stored.
- name: TRAIN_FILE_PATTERN
value: "gs://<data_bucket>/coco/train*"
# [REQUIRED] Must specify the Google Cloud Storage location where the
# model and the checkpoint will be stored.
- name: MODEL_DIR
value: "gs://<my-model-bucket>/retinanet"
# RetinaNet requires a pre-trained image classification model (like
# ResNet) as a backbone network. This example uses a pretrained
# checkpoint created with the ResNet demonstration model. You can
# instead train your own ResNet model if desired, and specify a
# checkpoint from your ResNet model directory.
- name: RESNET_CHECKPOINT
value: "gs://cloudtpu-coco-data/pretrain/resnet50-checkpoint-2018-02-07"
- name: VAL_JSON_FILE
value: "gs://<data_bucket>/coco/instances_val2017.json"
- name: EVAL_FILE_PATTERN
value: "gs://<data_bucket>/coco/val*"
resources:
limits:
# Request a single v3-8 Cloud TPU device to train the model.
# A single v3-8 Cloud TPU device consists of 4 chips, each of which
# has 2 cores, so there are 8 cores in total.
cloud-tpus.google.com/v3: 8

---

Models/Official/Detection/Projects/Copy Paste/README (models/official/detection/projects/copy_paste/README.md) Copy Section

Simple Copy-Paste Augmentation

Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yi Lin, Ekin D. Cubuk, Quoc V. Le, Barret Zoph
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation

<img src="https://storage.googleapis.com/cloud-tpu-checkpoints/detection/projects/copy-paste/copy_paste.jpg" height="320" />

Training models Copy Chapter

Attributes to update in the config to use Copy-Paste augmentation.

type: 'mask_rcnn' # or 'cascade_mask_rcnn'
train:
  pre_parser_dataset:
    file_pattern: <path to the TFRecord training data>
architecture:
  pre_parser: 'extract_objects_parser'
maskrcnn_parser:
  copy_paste: True

The extract_objects_parser gets an input dataset and parses the objects which will
be pasted in copy-paste augmentation. The path of this dataset can be set via
train.pre_parser_dataset.file_pattern (this path may be set same as the main
training dataset path: train.train_file_pattern).
maskrcnn_parser_with_copy_paste
gets input dataset and also output of
extract_objects_parser and pastes objects on the images to create new images
with Copy-Paste augmentation. Also, it updates the ground-truth data accordingly.

Checkpoints Copy Chapter

Checkpoints of object detection and instance segmentation models trained on COCO:

| model | #FLOPs | #Params | Box AP (val) | Mask AP (val) | download |
| --------------------------------|:---------:| --------:|---------------:|------------------:|-----------------------:|
| Res50-FPN (1024) w/ Copy-Paste | 431 B | 48 M | 48.3 | 42.4 | ckpt \| config |
| Res101-FPN (1024) w/ Copy-Paste | 509 B | 67 M | 49.8 | 43.5 | ckpt \| config |
| Res101-FPN (1280) w/ Copy-Paste | 693 B | 67 M | 50.3 | 44.1 | ckpt \| config |
| Eff-B7 FPN (640) w/ Copy-Paste | 286 B | 86 M | 50.0 | 43.7 | ckpt \| config |
| Eff-B7 FPN (1024) w/ Copy-Paste | 447 B | 86 M | 51.9 | 45.1 | ckpt \| config |
| Eff-B7 FPN (1280) w/ Copy-Paste | 595 B | 86 M | 52.5 | 45.8 | ckpt \| config |
| Cascade Eff-B7 FPN (1280) w/ Copy-Paste | 854 B | 118 M | 54.0 | 46.3 | ckpt \| config |
| Cascade Eff-B7 NAS-FPN (1280) | 1440 B | 185 M | 54.4 | 46.6 | ckpt \| config |
| Cascade Eff-B7 NAS-FPN (1280) w/ Copy-Paste | 1440 B | 185 M | 55.8 | 47.1 | ckpt \| config |
| Cascade Eff-B7 NAS-FPN (1280) w/ self-training Copy-Paste | 1440 B | 185 M | 57.0 | 48.8 | ckpt \| config |

Prepare Data Copy Chapter

The training expects the data in TFExample format stored in TFRecord.
Tools and scripts are provided to download and convert datasets.

| Dataset | Tool |
|:---------:|:-------------:|
| COCO | instructions |

Citation Copy Chapter

@article{ghiasi2020simple,
  title={Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation},
  author={Ghiasi, Golnaz and Cui, Yin and Srinivas, Aravind and Qian, Rui and Lin, Tsung-Yi and Cubuk, Ekin D and Le, Quoc V and Zoph, Barret},
  journal={arXiv preprint arXiv:2012.07177},
  year={2020}
}

---

Models/Official/Detection/Projects/Copy Paste/Configs/Cascade Maskrcnn Effb7 1280.Yaml (models/official/detection/projects/copy_paste/configs/cascade_maskrcnn_effb7_1280.yaml) Copy Section

cascade maskrcnn efficientnet-b7 FPN, with copy-paste, image size 1280

---

Models/Official/Detection/Projects/Copy Paste/Configs/Cascade Maskrcnn Effb7 Nasfpn 1280.Yaml (models/official/detection/projects/copy_paste/configs/cascade_maskrcnn_effb7_nasfpn_1280.yaml) Copy Section

cascade maskrcnn efficientnet-b7 NASFPN, with copy-paste, image size 1280

---

Models/Official/Detection/Projects/Copy Paste/Configs/Cascade Maskrcnn Effb7 Nasfpn Vanilla 1280.Yaml (models/official/detection/projects/copy_paste/configs/cascade_maskrcnn_effb7_nasfpn_vanilla_1280.yaml) Copy Section

cascade maskrcnn efficientnet-b7 NASFPN, image size 1280

---

Models/Official/Detection/Projects/Copy Paste/Configs/Maskrcnn Effb7 1024.Yaml (models/official/detection/projects/copy_paste/configs/maskrcnn_effb7_1024.yaml) Copy Section

maskrcnn efficientnet-b7 FPN, with copy-paste, image size 1024

---

Models/Official/Detection/Projects/Copy Paste/Configs/Maskrcnn Effb7 1280.Yaml (models/official/detection/projects/copy_paste/configs/maskrcnn_effb7_1280.yaml) Copy Section

maskrcnn efficientnet-b7 FPN, with copy-paste, image size 1280

---

Models/Official/Detection/Projects/Copy Paste/Configs/Maskrcnn Effb7 640.Yaml (models/official/detection/projects/copy_paste/configs/maskrcnn_effb7_640.yaml) Copy Section

maskrcnn efficientnet-b7 FPN, with copy-paste, image size 640

---

Models/Official/Detection/Projects/Copy Paste/Configs/Maskrcnn Resnet101 1024.Yaml (models/official/detection/projects/copy_paste/configs/maskrcnn_resnet101_1024.yaml) Copy Section

maskrcnn resnet-101 FPN, with copy-paste, image size 1024

---

Models/Official/Detection/Projects/Copy Paste/Configs/Maskrcnn Resnet101 1280.Yaml (models/official/detection/projects/copy_paste/configs/maskrcnn_resnet101_1280.yaml) Copy Section

maskrcnn resnet-101 FPN, with copy-paste, image size 1280

---

Models/Official/Detection/Projects/Copy Paste/Configs/Maskrcnn Resnet50 1024.Yaml (models/official/detection/projects/copy_paste/configs/maskrcnn_resnet50_1024.yaml) Copy Section

maskrcnn resnet-50 FPN, with copy-paste, image size 1024

---

Models/Official/Detection/Projects/Fashionpedia/README (models/official/detection/projects/fashionpedia/README.md) Copy Section

Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset

Menglin Jia, Mengyun Shi, Mikhail Sirotenko, Yin Cui, Claire Cardie, Bharath Hariharan, Hartwig Adam, Serge Belongie (*equal contribution)
[dataset] [arXiv]

<img src="https://storage.googleapis.com/cloud-tpu-checkpoints/detection/projects/fashionpedia/model.png" height="240"/>

We release the checkpoints of Attribute-Mask R-CNN model with ResNet-FPN and SpineNet backbone.

Other code including data conversion, model training and inference will be released soon.

Checkpoint Copy Chapter

Object detection and instance segmentation on Fashionpedia:

| backbone | input <br/> size | lr <br/> sched | FLOPs | Params | box AP <br/> IoU / IoU+F1 | mask AP <br/> IoU / IoU+F1 | download |
| ---------------|:----------:|:--------------:|:------:|:------:|:----:|:----:|:---------:|
| ResNet-50 FPN | 1024 | 1x | 296.7B | 46.4M | 38.7 / 26.6 | 34.3 / 25.5 | N/A |
| ResNet-50 FPN | 1024 | 2x | 296.7B | 46.4M | 41.6 / 29.3 | 38.1 / 28.5 | N/A |
| ResNet-50 FPN | 1024 | 3x | 296.7B | 46.4M | 43.4 / 30.7 | 39.2 / 29.5 | ckpt \| config |
| ResNet-50 FPN | 1024 | 6x | 296.7B | 46.4M | 42.9 / 31.2 | 38.9 / 30.2 | N/A |
| ResNet-101 FPN | 1024 | 1x | 374.3B | 65.4M | 41.0 / 28.6 | 36.7 / 27.6 | N/A |
| ResNet-101 FPN | 1024 | 2x | 374.3B | 65.4M | 43.5 / 31.0 | 39.2 / 29.8 | N/A |
| ResNet-101 FPN | 1024 | 3x | 374.3B | 65.4M | 44.9 / 32.8 | 40.7 / 31.4 | ckpt \| config |
| ResNet-101 FPN | 1024 | 6x | 374.3B | 65.4M | 44.3 / 32.9 | 39.7 / 31.3 | N/A |
| SpineNet-49 | 1024 | 6x | 267.2B | 40.8M | 43.7 / 32.4 | 39.6 / 31.4 | ckpt \| config |
| SpineNet-96 | 1024 | 6x | 314.0B | 55.2M | 46.4 / 34.0 | 41.2 / 31.8 | ckpt \| config |
| SpineNet-143 | 1280 | 6x | 498.0B | 79.2M | 48.7 / 35.7 | 43.1 / 33.3 | ckpt \| config |

For calculating AP (IoU without attribute prediction or IoU + F1 with attribute prediction), please refer to the [Fahionpedia API].

Citation Copy Chapter

@inproceedings{jia2020fashionpedia,
  title={Fashionpedia: Ontology, Segmentation, and an Attribute Localization Dataset},
  author={Jia, Menglin and Shi, Mengyun and Sirotenko, Mikhail and Cui, Yin and Cardie, Claire and Hariharan, Bharath and Adam, Hartwig and Belongie, Serge},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2020}
}

---

Models/Official/Detection/Projects/Fashionpedia/Configs/Yaml/R101fpn Amrcnn.Yaml (models/official/detection/projects/fashionpedia/configs/yaml/r101fpn_amrcnn.yaml) Copy Section

ResNet-101 FPN + Attribute-Mask R-CNN

3x schedule (~95 Fashionpedia epochs)

Box AP (IoU / IoU+F1): 44.9 / 32.8

Mask AP (IoU / IoU+F1): 40.7 / 31.4

---

Models/Official/Detection/Projects/Fashionpedia/Configs/Yaml/R50fpn Amrcnn.Yaml (models/official/detection/projects/fashionpedia/configs/yaml/r50fpn_amrcnn.yaml) Copy Section

ResNet-50 FPN + Attribute-Mask R-CNN

3x schedule (~95 Fashionpedia epochs)

Box AP (IoU / IoU+F1): 43.4 / 30.7

Mask AP (IoU / IoU+F1): 39.2 / 29.5

---

Models/Official/Detection/Projects/Fashionpedia/Configs/Yaml/Spinenet143 Amrcnn.Yaml (models/official/detection/projects/fashionpedia/configs/yaml/spinenet143_amrcnn.yaml) Copy Section

SpineNet-143 + Attribute-Mask R-CNN

6x schedule (~189 Fashionpedia epochs)

Box AP (IoU / IoU+F1): 48.7 / 35.7

Mask AP (IoU / IoU+F1): 43.1 / 33.3

---

Models/Official/Detection/Projects/Fashionpedia/Configs/Yaml/Spinenet49 Amrcnn.Yaml (models/official/detection/projects/fashionpedia/configs/yaml/spinenet49_amrcnn.yaml) Copy Section

SpineNet-49 + Attribute-Mask R-CNN

6x schedule (~189 Fashionpedia epochs)

Box AP (IoU / IoU+F1): 43.7 / 32.4

Mask AP (IoU / IoU+F1): 39.6 / 31.4

---

Models/Official/Detection/Projects/Fashionpedia/Configs/Yaml/Spinenet96 Amrcnn.Yaml (models/official/detection/projects/fashionpedia/configs/yaml/spinenet96_amrcnn.yaml) Copy Section

SpineNet-96 + Attribute-Mask R-CNN

6x schedule (~189 Fashionpedia epochs)

Box AP (IoU / IoU+F1): 46.4 / 34.0

Mask AP (IoU / IoU+F1): 41.2 / 31.8

---

Models/Official/Detection/Projects/Openseg/README (models/official/detection/projects/openseg/README.md) Copy Section

OpenSeg: Scaling Open-Vocabulary Image Segmentation with Image-Level Labels

Golnaz Ghiasi, Xiuye Gu, Yin Cui, Tsung-Yi Lin
[[arXiv]](https://arxiv.org/abs/2112.12143)
[[demo]](https://colab.sandbox.google.com/github/tensorflow/tpu/blob/master/models/official/detection/projects/openseg/OpenSeg_demo.ipynb)
[[poster]](https://storage.googleapis.com/cloud-tpu-checkpoints/detection/projects/openseg/OpenSeg-ECCV22-poster.pdf)

OpenSeg can organize pixels into meaningful regions indicated by texts.
In contrast to segmentation models trained with close-vocabulary
categories, OpenSeg can handle arbitrary text queries.

<p style="text-align:center;">
<img src="https://storage.googleapis.com/cloud-tpu-checkpoints/detection/projects/openseg/teaser.jpg" height="120" />
</p>

The figure below shows an overview of OpenSeg architecture.

<p style="text-align:center;">
<img src="https://storage.googleapis.com/cloud-tpu-checkpoints/detection/projects/openseg/overview.jpg" height="400" />
</p>

Colab Demo Copy Chapter

colab

jupyter notebook

The image tower of the OpenSeg model used in this colab has a backbone of Efficientnet-b7, initialized with the noisy student checkpoint. The text tower is the frozen text tower of CLIP ViT-L/14@336px.
The model is trained on COCO class-agnostic masks, COCO captions, and localized narrative caption data.

Class names w/wo ensembling and prompt engineering Copy Chapter

We provide class names for ADE20k, COCO Panoptic, PASCAL Context and PASCAL VOC datasets used in OpenSeg. The details are described in Appendix I "Ensembling and prompt engineering" in our paper.

ade20k_150

ade20k_150_with_prompt_eng

ade20k_847

ade20k_847_with_prompt_eng

coco_panoptic

coco_panoptic_with_prompt_eng

pascal_context_459

pascal_context_459_with_prompt_eng

pascal_context_59

pascal_context_59_with_prompt_eng

pascal_voc

pascal_voc_with_prompt_eng

Citation Copy Chapter

@inproceedings{ghiasi2021open,
  title={Scaling Open-Vocabulary Image Segmentation with Image-Level Labels},
  author={Ghiasi, Golnaz and Gu, Xiuye and Cui, Yin and Lin, Tsung-Yi},
  booktitle={ECCV},
  year={2022}
}

---

Models/Official/Detection/Projects/Self Training/README (models/official/detection/projects/self_training/README.md) Copy Section

Rethinking Pre-Training and Self-Training

Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin D. Cubuk, Quoc V. Le
[arXiv]

<img src="https://storage.googleapis.com/cloud-tpu-checkpoints/detection/projects/rethinking-pre-training-and-self-training/logo.png" height="180" />

We release the checkpoints of teacher model and student model in rethinking
pre-training and self-training.

Checkpoint Copy Chapter

Object detection on COCO (results with SoftNMS):

| model | #FLOPs | #Params | AP (val) | AP (test_dev) | download |
| -------------|:---------:| --------:|-----------:|--------------:|-----------:|
| SpineNet-143 | 524B | 67M | 50.9 | 51.0 | ckpt \| config |
| SpineNet-143 w/self-training | 524B | 67M | 52.6 | 52.8 | ckpt \| config |
| SpineNet-190 | 1885B | 164M | 52.6 | 52.8 | ckpt \| config |
| SpineNet-190 w/self-training | 1885B | 164M | 54.2 | 54.3 | ckpt \| config |

Semantic segmentation on PASCAL VOC 2012:

| model | #FLOPs | #Params | mIOU (val) | mIOU (test) | download |
| -----------------------|:---------:| --------:|-----------:|--------------:|-----------:|
| EfficientNet-B7-NASFPN | 60B | 71M | 85.2 | - | ckpt \| config |
| EfficientNet-B7-NASFPN w/ self-training | 60B | 71M | 86.7 | - | ckpt \| config |
| EfficientNet-L2-NASFPN | 229B | 485M | 88.7 | - | ckpt \| config |
| EfficientNet-L2-NASFPN w/ self-training | 229B | 485M | 90.0 | 90.5 | ckpt \| config |

Prepare Data Copy Chapter

The training expects the data in TFExample format stored in TFRecord.
Tools and scripts are provided to download and convert datasets.

| Dataset | Tool |
|:---------:|:-------------:|
| ImageNet | instructions |
| COCO | instructions |
| PASCAL | instructions

Citation Copy Chapter

@article{zoph20selftraining,
  title={Rethinking pre-training and self-training},
  author={Barret Zoph and Golnaz Ghiasi and Tsung-Yi Lin and Yin Cui and Hanxiao Liu and Ekin D. Cubuk and Quoc V. Le},
  journal={CoRR},
  volume={abs/2006.06882},
  year={2020}
}

---

Models/Official/Detection/Projects/Self Training/Configs/Coco Spinenet143 Retinanet.Yaml (models/official/detection/projects/self_training/configs/coco_spinenet143_retinanet.yaml) Copy Section

SpineNet-143 + RetinaNet:

Config is used to run eval on the SpineNet-143 checkpoint trained with semi-supervised learning

Performance: 52.2 AP without SoftNMS, 52.4 with SoftNMS

---

Models/Official/Detection/Projects/Self Training/Configs/Coco Spinenet190 Retinanet.Yaml (models/official/detection/projects/self_training/configs/coco_spinenet190_retinanet.yaml) Copy Section

SpineNet-190 + RetinaNet:

Config is used to run eval on the SpineNet-190 checkpoint trained with semi-supervised learning

Performance: 53.9 AP without SoftNMS, 54.2 with SoftNMS

---

Models/Official/Detection/Projects/Self Training/Configs/Pascal Seg Efficientnet B7 Nasfpn.Yaml (models/official/detection/projects/self_training/configs/pascal_seg_efficientnet-b7-nasfpn.yaml) Copy Section

Template to train NAS-FPN with efficientnet-b7 backbone model on Pascal.

---

Models/Official/Detection/Projects/Self Training/Configs/Pascal Seg Efficientnet L2 Nasfpn.Yaml (models/official/detection/projects/self_training/configs/pascal_seg_efficientnet-l2-nasfpn.yaml) Copy Section

Template to train NAS-FPN with efficientnet-l2 backbone model on Pascal.

---

Models/Official/Detection/Projects/Vild/README (models/official/detection/projects/vild/README.md) Copy Section

Open-Vocabulary Detection via Vision and Language Knowledge Distillation

<p style="text-align:center;"><img src="https://storage.googleapis.com/cloud-tpu-checkpoints/detection/projects/vild/assets/new_teaser.png" alt="teaser" width="500"/></p>

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, Yin Cui,
Open-Vocabulary Detection via Vision and Language Knowledge Distillation.

This repo contains the colab demo, code, and pretrained checkpoints for our open-vocabulary detection method, ViLD (Vision and Language Distillation).

Open-vocabulary object detection detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data. Existing object detection datasets only contain hundreds of categories, and it is costly to scale further. To overcome this challenge, we propose ViLD. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher.

We benchmark on LVIS by holding out all rare categories as novel categories not seen during training. ViLD obtains 16.1 mask APr, even outperforming the supervised counterpart by 3.8 with a ResNet-50 backbone. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP50, 36.6 AP and 11.8 AP on PASCAL VOC, COCO and Objects365, respectively. On COCO, ViLD outperforms previous SOTA by 4.8 on novel AP and 11.4 on overall AP.

The figure below shows an overview of ViLD's architecture.

Colab Demo

Getting Started

Prerequisite Copy Chapter

Data preprocessing Copy Chapter

The DATA_DIR should be organized as below:

DATA_DIR
+-- lvis_v1_val.json
+-- val2017
|   +-- *.jpg
|   +-- ...

2. Create tfrecords for the validation set (adjust max_num_processes if needed; specify DEST_DIR to the tfrecords output directory):

DATA_DIR=[DATA_DIR]
DEST_DIR=[DEST_DIR]
VAL_JSON="${DATA_DIR}/lvis_v1_val.json"
python3 preprocessing/create_lvis_tf_record.py \
  --image_dir="${DATA_DIR}" \
  --json_path="${VAL_JSON}" \
  --dest_dir="${DEST_DIR}" \
  --include_mask=True \
  --split='val' \
  --num_parts=100 \
  --max_num_processes=100

Trained checkpoints Copy Chapter

Inference Copy Chapter

Please refer getting_started.md for more information.

BATCH_SIZE=1
RESNET_DEPTH=[RESNET_DEPTH]
MODEL_DIR=[MODEL_DIR]
EVAL_FILE_PATTERN="[DEST_DIR]/val*"
VAL_JSON_FILE="[DATA_DIR]/lvis_v1_val.json"
RARE_MASK_PATH="[WEIGHTS_DIR]/lvis_rare_masks.npy"
CLASSIFIER_WEIGHT_PATH="[WEIGHTS_DIR]/clip_synonym_prompt.npy"
CONFIG_FILE="tpu/models/official/detection/projects/vild/configs/[CONFIG_FILE]"
python3 tpu/models/official/detection/main.py \
  --model="vild" \
  --model_dir="${MODEL_DIR?}" \
  --mode=eval \
  --use_tpu=False \
  --config_file="${CONFIG_FILE?}" \
  --params_override="{ resnet: {resnet_depth: ${RESNET_DEPTH?}}, predict: {predict_batch_size: ${BATCH_SIZE?}}, eval: {eval_batch_size: ${BATCH_SIZE?}, val_json_file: ${VAL_JSON_FILE?}, eval_file_pattern: ${EVAL_FILE_PATTERN?} }, frcnn_head: {classifier_weight_path: ${CLASSIFIER_WEIGHT_PATH?}}, postprocess: {rare_mask_path: ${RARE_MASK_PATH?}}}"

License

Citation

@article{gu2021open,
  title={Open-Vocabulary Detection via Vision and Language Knowledge Distillation},
  author={Gu, Xiuye and Lin, Tsung-Yi and Kuo, Weicheng and Cui, Yin},
  journal={arXiv preprint arXiv:2104.13921},
  year={2021}
}

Acknowledgements

The code is built upon Cloud TPU detection.

---

Models/Official/Detection/Projects/Vild/Configs/Vild Ensemble Resnet.Yaml (models/official/detection/projects/vild/configs/vild_ensemble_resnet.yaml) Copy Section

anchor:
anchor_size: 8
aspect_ratios: [1.0, 2.0, 0.5]
num_scales: 1
architecture:
backbone: resnet
feat_distill_weight: 0.5
filter_distill_boxes_size: 0
include_mask: true
mask_target_size: 28
max_level: 6
max_num_rois: 300
min_level: 2
multilevel_features: fpn
normalize_feat_during_training: true
num_classes: 1204
parser: vild_parser
pre_parser: null
space_to_depth_block_size: 1
use_bfloat16: false
visual_feature_dim: 512
visual_feature_distill: double_branch
batch_norm_activation:
activation: relu
batch_norm_epsilon: 0.0001
batch_norm_momentum: 0.997
batch_norm_trainable: true
use_sync_bn: true
dropblock:
dropblock_keep_prob: null
dropblock_size: null
enable_summary: false
eval:
eval_batch_size: 8
eval_dataset_type: tfrecord
eval_samples: 19809
eval_timeout: null
min_eval_interval: 5
num_steps_per_eval: 1000
per_category_metrics: false
skip_eval_loss: false
suffix: ''
type: lvis_box_and_mask
use_json_file: true
fpn:
fpn_feat_dims: 256
use_batch_norm: true
use_separable_conv: false
frcnn_box_loss:
huber_loss_delta: 1.0
frcnn_class_loss:
mask_rare: true
frcnn_head:
class_agnostic_bbox_pred: true
clip_dim: 512
fc_dims: 1024
normalize_classifier: true
normalize_visual: true
num_convs: 4
num_fcs: 2
num_filters: 256
temperature: 100.0
use_batch_norm: true
use_separable_conv: false
mask_sampling:
num_mask_samples_per_image: 128
mrcnn_head:
class_agnostic_mask_pred: true
num_convs: 4
num_filters: 256
use_batch_norm: true
use_separable_conv: false
postprocess:
apply_nms: true
apply_sigmoid: false
discard_background: false
max_total_size: 300
nms_iou_threshold: 0.5
nms_version: v1
pre_nms_num_boxes: 1000
score_threshold: 0.0
use_batched_nms: false
predict:
predict_batch_size: 8
resnet:
init_drop_connect_rate: null
resnet_depth: 50
roi_proposal:
rpn_min_size_threshold: 0.0
rpn_nms_threshold: 0.7
rpn_post_nms_top_k: 1000
rpn_pre_nms_top_k: 2000
rpn_score_threshold: 0.0
test_rpn_min_size_threshold: 0.0
test_rpn_nms_threshold: 0.7
test_rpn_post_nms_top_k: 1000
test_rpn_pre_nms_top_k: 1000
test_rpn_score_threshold: 0.0
use_batched_nms: false
roi_sampling:
bg_iou_thresh_hi: 0.5
bg_iou_thresh_lo: 0.0
cascade_iou_thresholds: null
fg_fraction: 0.25
fg_iou_thresh: 0.5
mix_gt_boxes: true
num_samples_per_image: 512
rpn_box_loss:
huber_loss_delta: 0.1111111111111111
rpn_head:
anchors_per_location: null
cast_to_float32: true
num_convs: 2
num_filters: 256
use_batch_norm: true
use_separable_conv: false
rpn_score_loss:
rpn_batch_size_per_im: 256
train:
checkpoint:
path: ''
prefix: ''
skip_variables_regex: ''
frozen_variable_prefix: frcnn_layer_0/fast_rcnn_head/class-predict
gradient_clip_norm: 0.0
input_partition_dims: null
iterations_per_loop: 100
l2_weight_decay: 4.0e-05
learning_rate:
init_learning_rate: 0.32
learning_rate_levels: [0.032, 0.0032]
learning_rate_steps: [162000, 171000, 175500]
type: step
warmup_learning_rate: 0.0032
warmup_steps: 1000
losses: all
num_cores_per_replica: null
optimizer:
momentum: 0.9
type: momentum
pre_parser_dataset:
dataset_type: tfrecord
file_pattern: ''
regularization_variable_regex: .*(kernel|weight):0$
space_to_depth_block_size: 1
total_steps: 180000
train_batch_size: 256
train_dataset_type: tfrecord
transpose_input: true
type: vild
use_tpu: false
vild_parser:
aug_rand_hflip: true
aug_scale_max: 2.0
aug_scale_min: 0.1
copy_paste: false
mask_crop_size: 112
max_num_instances: 300
output_size: [1024, 1024]
regenerate_source_id: false
rpn_batch_size_per_im: 256
rpn_fg_fraction: 0.5
rpn_match_threshold: 0.7
rpn_unmatched_threshold: 0.3
skip_crowd_during_training: true

---

Models/Official/Detection/Projects/Vild/Configs/Vild Resnet.Yaml (models/official/detection/projects/vild/configs/vild_resnet.yaml) Copy Section

anchor:
anchor_size: 8
aspect_ratios: [1.0, 2.0, 0.5]
num_scales: 1
architecture:
backbone: resnet
feat_distill_weight: 0.5
filter_distill_boxes_size: 0
include_mask: true
mask_target_size: 28
max_level: 6
max_num_rois: 300
min_level: 2
multilevel_features: fpn
normalize_feat_during_training: true
num_classes: 1204
parser: vild_parser
pre_parser: null
space_to_depth_block_size: 1
use_bfloat16: false
visual_feature_dim: 512
visual_feature_distill: vanilla
batch_norm_activation:
activation: relu
batch_norm_epsilon: 0.0001
batch_norm_momentum: 0.997
batch_norm_trainable: true
use_sync_bn: true
dropblock:
dropblock_keep_prob: null
dropblock_size: null
enable_summary: false
eval:
eval_batch_size: 8
eval_dataset_type: tfrecord
eval_samples: 19809
eval_timeout: null
min_eval_interval: 5
num_steps_per_eval: 1000
per_category_metrics: false
skip_eval_loss: false
suffix: ''
type: lvis_box_and_mask
use_json_file: true
fpn:
fpn_feat_dims: 256
use_batch_norm: true
use_separable_conv: false
frcnn_box_loss:
huber_loss_delta: 1.0
frcnn_class_loss:
mask_rare: true
frcnn_head:
class_agnostic_bbox_pred: true
clip_dim: 512
fc_dims: 1024
normalize_classifier: true
normalize_visual: true
num_convs: 4
num_fcs: 2
num_filters: 256
temperature: 100.0
use_batch_norm: true
use_separable_conv: false
mask_sampling:
num_mask_samples_per_image: 128
mrcnn_head:
class_agnostic_mask_pred: true
num_convs: 4
num_filters: 256
use_batch_norm: true
use_separable_conv: false
postprocess:
apply_nms: true
apply_sigmoid: false
discard_background: false
max_total_size: 300
nms_iou_threshold: 0.5
nms_version: v1
pre_nms_num_boxes: 1000
score_threshold: 0.0
use_batched_nms: false
predict:
predict_batch_size: 8
resnet:
init_drop_connect_rate: null
resnet_depth: 50
roi_proposal:
rpn_min_size_threshold: 0.0
rpn_nms_threshold: 0.7
rpn_post_nms_top_k: 1000
rpn_pre_nms_top_k: 2000
rpn_score_threshold: 0.0
test_rpn_min_size_threshold: 0.0
test_rpn_nms_threshold: 0.7
test_rpn_post_nms_top_k: 1000
test_rpn_pre_nms_top_k: 1000
test_rpn_score_threshold: 0.0
use_batched_nms: false
roi_sampling:
bg_iou_thresh_hi: 0.5
bg_iou_thresh_lo: 0.0
cascade_iou_thresholds: null
fg_fraction: 0.25
fg_iou_thresh: 0.5
mix_gt_boxes: true
num_samples_per_image: 512
rpn_box_loss:
huber_loss_delta: 0.1111111111111111
rpn_head:
anchors_per_location: null
cast_to_float32: true
num_convs: 2
num_filters: 256
use_batch_norm: true
use_separable_conv: false
rpn_score_loss:
rpn_batch_size_per_im: 256
train:
checkpoint:
path: ''
prefix: ''
skip_variables_regex: ''
frozen_variable_prefix: frcnn_layer_0/fast_rcnn_head/class-predict
gradient_clip_norm: 0.0
input_partition_dims: null
iterations_per_loop: 100
l2_weight_decay: 4.0e-05
learning_rate:
init_learning_rate: 0.32
learning_rate_levels: [0.032, 0.0032]
learning_rate_steps: [162000, 171000, 175500]
type: step
warmup_learning_rate: 0.0032
warmup_steps: 1000
losses: all
num_cores_per_replica: null
optimizer:
momentum: 0.9
type: momentum
pre_parser_dataset:
dataset_type: tfrecord
file_pattern: ''
regularization_variable_regex: .*(kernel|weight):0$
space_to_depth_block_size: 1
total_steps: 180000
train_batch_size: 256
train_dataset_type: tfrecord
transpose_input: true
type: vild
vild_parser:
aug_rand_hflip: true
aug_scale_max: 2.0
aug_scale_min: 0.1
copy_paste: false
mask_crop_size: 112
max_num_instances: 300
output_size: [1024, 1024]
regenerate_source_id: false
rpn_batch_size_per_im: 256
rpn_fg_fraction: 0.5
rpn_match_threshold: 0.7
rpn_unmatched_threshold: 0.3
skip_crowd_during_training: true

---

Models/Official/Efficientnet/README (models/official/efficientnet/README.md) Copy Section

EfficientNets

[1] Mingxing Tan and Quoc V. Le. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML 2019.
Arxiv link: https://arxiv.org/abs/1905.11946.

Updates

- [Mar 2020] Released mobile/IoT device friendly EfficientNet-lite models: README.

- [Feb 2020] Released EfficientNet checkpoints trained with NoisyStudent: paper.

- [Nov 2019] Released EfficientNet checkpoints trained with AdvProp: paper.

- [Oct 2019] Released EfficientNet-CondConv models with conditionally parameterized convolutions: README, paper.

- [Oct 2019] Released EfficientNet models trained with RandAugment: paper.

- [Aug 2019] Released EfficientNet-EdgeTPU models: README and blog post.

- [Jul 2019] Released EfficientNet checkpoints trained with AutoAugment: paper, blog post

- [May 2019] Released EfficientNets code and weights: blog post

1. About EfficientNet Models Copy Chapter

EfficientNets are a family of image classification models, which achieve state-of-the-art accuracy, yet being an order-of-magnitude smaller and faster than previous models.

We develop EfficientNets based on AutoML and Compound Scaling. In particular, we first use AutoML MNAS Mobile framework to develop a mobile-size baseline network, named as EfficientNet-B0; Then, we use the compound scaling method to scale up this baseline to obtain EfficientNet-B1 to B7.

<table border="0">
<tr>
<td>
<img src="./g3doc/params.png" width="100%" />
</td>
<td>
<img src="./g3doc/flops.png", width="100%" />
</td>
</tr>
</table>

EfficientNets achieve state-of-the-art accuracy on ImageNet with an order of magnitude better efficiency:

* In high-accuracy regime, our EfficientNet-B7 achieves state-of-the-art 84.4% top-1 / 97.1% top-5 accuracy on ImageNet with 66M parameters and 37B FLOPS, being 8.4x smaller and 6.1x faster on CPU inference than previous best Gpipe.

* In middle-accuracy regime, our EfficientNet-B1 is 7.6x smaller and 5.7x faster on CPU inference than ResNet-152, with similar ImageNet accuracy.

* Compared with the widely used ResNet-50, our EfficientNet-B4 improves the top-1 accuracy from 76.3% of ResNet-50 to 82.6% (+6.3%), under similar FLOPS constraint.

2. Using Pretrained EfficientNet Checkpoints Copy Chapter

To train EfficientNet on ImageNet, we hold out 25,022 randomly picked images (image filenames, or 20 out of 1024 total shards) as a 'minival' split, and conduct early stopping based on this 'minival' split. The final accuracy is reported on the original ImageNet validation set.

We have provided a list of EfficientNet checkpoints:.

* With baseline ResNet preprocessing, we achieve similar results to the original ICML paper.
* With AutoAugment preprocessing, we achieve higher accuracy than the original ICML paper.
* With RandAugment preprocessing, accuracy is further improved.
* With AdvProp, state-of-the-art results (w/o extra data) are achieved.
* With NoisyStudent, state-of-the-art results (w/ extra JFT-300M unlabeled data) are achieved.

| | B0 | B1 | B2 | B3 | B4 | B5 | B6 | B7 | B8 | L2-475 | L2 |
|---------- |-------- | ------| ------|------ |------ |------ | --- | --- | --- | --- |--- |
| Baseline preprocessing | 76.7% (ckpt) | 78.7% (ckpt) | 79.8% (ckpt) | 81.1% (ckpt) | 82.5% (ckpt) | 83.1% (ckpt) | | || | | |
| AutoAugment (AA) | 77.1% (ckpt) | 79.1% (ckpt) | 80.1% (ckpt) | 81.6% (ckpt) | 82.9% (ckpt) | 83.6% (ckpt) | 84.0% (ckpt) | 84.3% (ckpt) || | |
| RandAugment (RA) | | | | | | 83.7% (ckpt) | | 84.7% (ckpt) | | | |
| AdvProp + AA | 77.6% (ckpt) | 79.6% (ckpt) | 80.5% (ckpt) | 81.9% (ckpt) | 83.3% (ckpt) | 84.3% (ckpt) | 84.8% (ckpt) | 85.2% (ckpt) | 85.5% (ckpt)|| | |
| NoisyStudent + RA | 78.8% (ckpt) | 81.5% (ckpt) | 82.4% (ckpt) | 84.1% (ckpt) | 85.3% (ckpt) | 86.1% (ckpt) | 86.4% (ckpt) | 86.9% (ckpt) | - |88.2%(ckpt)|88.4% (ckpt) |

^{*To train EfficientNets with AutoAugment (code), simply add option "--augment_name=autoaugment". If you use these checkpoints, you can cite this paper.}

^{To train EfficientNets with RandAugment (code), simply add option "--augment_name=randaugment". For EfficientNet-B5 also add "--randaug_num_layers=2 --randaug_magnitude=17". For EfficientNet-B7 or EfficientNet-B8 also add "--randaug_num_layers=2 --randaug_magnitude=28". If you use these checkpoints, you can cite this paper.}

^{* AdvProp training code coming soon. Please set "--advprop_preprocessing=True" for using AdvProp checkpoints. If you use AdvProp checkpoints, you can cite this paper.}

^{* NoisyStudent training code coming soon. L2-475 means the same L2 architecture with input image size 475 (Please set "--input_image_size=475" for using this checkpoint). If you use NoisyStudent checkpoints, you can cite this paper.}

^{*Note that AdvProp and NoisyStudent performance is derived from baselines that don't use holdout eval set. They will be updated in future."}

A quick way to use these checkpoints is to run:

$ export MODEL=efficientnet-b0
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/ckpts/${MODEL}.tar.gz
$ tar xf ${MODEL}.tar.gz
$ wget https://upload.wikimedia.org/wikipedia/commons/f/fe/Giant_Panda_in_Beijing_Zoo_1.JPG -O panda.jpg
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/eval_data/labels_map.json
$ python eval_ckpt_main.py --model_name=$MODEL --ckpt_dir=$MODEL --example_img=panda.jpg --labels_map_file=labels_map.json

Please refer to the following colab for more instructions on how to obtain and use those checkpoints.

* eval_ckpt_example.ipynb: A colab example to load
EfficientNet pretrained checkpoints files and use the restored model to classify images.

3. Using EfficientNet as Feature Extractor Copy Chapter

import efficientnet_builder
    features, endpoints = efficientnet_builder.build_model_base(images, 'efficientnet-b0')

* Use features for classification finetuning.
* Use endpoints['reduction_i'] for detection/segmentation, as the last intermediate feature with reduction level i. For example, if input image has resolution 224x224, then:
* endpoints['reduction_1'] has resolution 112x112
* endpoints['reduction_2'] has resolution 56x56
* endpoints['reduction_3'] has resolution 28x28
* endpoints['reduction_4'] has resolution 14x14
* endpoints['reduction_5'] has resolution 7x7

4. Training EfficientNets on TPUs. Copy Chapter

To train this model on Cloud TPU, you will need:

* A GCE VM instance with an associated Cloud TPU resource
* A GCS bucket to store your training checkpoints (the "model directory")
* Install TensorFlow version >= 1.13 for both GCE VM and Cloud.

Then train the model:

$ export PYTHONPATH="$PYTHONPATH:/path/to/models"
$ python main.py --tpu=TPU_NAME --data_dir=DATA_DIR --model_dir=MODEL_DIR

# TPU_NAME is the name of the TPU node, the same name that appears when you run gcloud compute tpus list, or ctpu ls.
# MODEL_DIR is a GCS location (a URL starting with gs:// where both the GCE VM and the associated Cloud TPU have write access
# DATA_DIR is a GCS location to which both the GCE VM and associated Cloud TPU have read access.

For more instructions, please refer to our tutorial: https://cloud.google.com/tpu/docs/tutorials/efficientnet

---

Models/Official/Efficientnet/Condconv/README (models/official/efficientnet/condconv/README.md) Copy Section

EfficientNet-CondConv

[1] Brandon Yang, Gabriel Bender, Quoc V. Le, Jiquan Ngiam. CondConv: Conditionally Parameterized Convolutions for Efficient Inference. NeurIPS 2019. Arxiv Link: https://arxiv.org/abs/1904.04971.

1. About CondConv Copy Chapter

Conditionally parameterized convolutions (CondConv) are a new building block for convolutional neural networks to increase capacity while maintaining efficient inference. In a traditional convolutional layer, each example is processed with the same kernel. In a CondConv layer, each example is processed with a specialized, example-dependent kernel. As an intuitive motivating example, on the ImageNet classification dataset, we might want to classify dogs and cats with different convolutional kernels.

<table border="0" width="50%">
<tr>
<td>
<img src="../g3doc/condconv-layer.png"/>
</td>
</tr>
</table>

A CondConv layer consists of n experts, each of which are the same size as the convolutional kernel of the original convolutional layer. For each example, the example-dependent convolutional kernel is computed as the weighted sum of experts using an example-dependent routing function. Increasing the number of experts enables us to increase the capacity of a network, while maintaining efficient inference.

Replacing convolutional layers with CondConv layers improves the accuracy versus inference cost trade-off on a wide range of models: MobileNetV1, MobileNetV2, ResNets, and EfficientNets. We measure inference cost in multiply-adds (MADDs). When applied to EfficientNets, we obtain EfficientNet-CondConv models. Our EfficientNet-CondConv-B0 model with 8 experts achieves state-of-the-art accuracy versus inference cost performance.

In this directory, we open-source the code to reproduce the EfficientNet-CondConv results in our paper and enable easy experimentation with EfficientNet-CondConv models. Additionally, we open-source the CondConv2d and DepthwiseCondConv2D Keras layers for easy application in new model architectures.

2. Using pretrained EfficientNet-CondConv checkpoints Copy Chapter

We have provided pre-trained checkpoints for several EfficientNet-CondConv models.

| | CondConv Experts | Params | MADDs | Accuracy |
|--------------------------------|------------------|--------|-------|----------|
| EfficientNet-B0 | - | 5.3M | 391M | 77.3 |
| EfficientNet-CondConv-B0 (ckpt)| 4 | 13.3M | 402M | 77.8 |
| EfficientNet-CondConv-B0 (ckpt)| 8 | 24.0M | 413M | 78.3 |

| | CondConv Experts | Params | MADDs | Accuracy |
|---------------------------------------|------------------|--------|-------|----------|
| EfficientNet-B1 | - | 7.8M | 700M | 79.2 |
| EfficientNet-CondConv-B0-Depth (ckpt) | 8 | 39.7M | 614M | 79.5 |

A quick way to use these checkpoints is to run:

$ export MODEL=efficientnet-condconv-b0-8e
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/${MODEL}.tar.gz
$ tar zxf ${MODEL}.tar.gz
$ wget https://upload.wikimedia.org/wikipedia/commons/f/fe/Giant_Panda_in_Beijing_Zoo_1.JPG -O panda.jpg
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/eval_data/labels_map.txt
$ python eval_ckpt_main.py --model_name=$MODEL --ckpt_dir=$MODEL --example_img=panda.jpg --labels_map_file=labels_map.txt

Please refer to the following colab for more instructions on how to obtain and use those checkpoints.

* eval_ckpt_example.ipynb: A colab example to load
EfficientNet pretrained checkpoints files and use the restored model to classify images.

3. Training EfficientNet-CondConv models on Cloud TPUs Copy Chapter

---

Models/Official/Efficientnet/Edgetpu/README (models/official/efficientnet/edgetpu/README.md) Copy Section

EfficientNet-EdgeTPU

Blog post: https://ai.googleblog.com/2019/08/efficientnet-edgetpu-creating.html

EfficientNet-EdgeTPU are a family of image classification neural network models customized for deployment on Google Edge TPU. These networks are closely related to [EfficientNets] (https://arxiv.org/abs/1905.11946).

EfficientNet-EdgeTPU were developed using the AutoML MNAS framework by augmenting the neural network search space with building blocks tuned to execute efficiently on the EdgeTPU neural network accelerator architecture. The neural architecture search was incentivized to discover models that achieve low parameter footprint and low latency on EdgeTpu, while simultaneously achieving high classification accuracy. This neural architecture search produced a baseline model: edgetpunet-S, which is subsequently scaled up using EfficientNet's compound scaling method to produce the M and L models.

<table border="0" width="70%">
<tr>
<td>
<img src="../g3doc/efficientnet-edgetpu.png"/>
</td>
</tr>
</table>

Using Pretrained EfficientNet-EdgeTPU Checkpoints Copy Section

* EfficientNet-EdgeTPU-S
* EfficientNet-EdgeTPU-M
* EfficientNet-EdgeTPU-L

A quick way to use these checkpoints is to run:

$ export MODEL=efficientnet-edgetpu-S
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/${MODEL}.tar.gz
$ tar zxf ${MODEL}.tar.gz
$ wget https://upload.wikimedia.org/wikipedia/commons/f/fe/Giant_Panda_in_Beijing_Zoo_1.JPG -O panda.jpg
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/eval_data/labels_map.txt
$ python eval_ckpt_main.py --model_name=$MODEL --ckpt_dir=$MODEL --example_img=panda.jpg --labels_map_file=labels_map.txt --include_background_label

Note that these models were trained with label#0 marked as the background label for easier deployment. TFLite models can be evaluated using this tool.

Training EfficientNet-EdgeTPU on Cloud TPUs Copy Section

Post-training quantization Copy Section

EdgeTPUs support inference using integer quantized models only. We found that using the Tensorflow Lite's post-training quantization tool works remarkably well for producing a EdgeTPU-compatible quantized model from a floating-point training checkpoint. For full integer quantization, the post-training quantization tool requires a representative dataset for calibrating the dynamic ranges of the activations.

We provide a tool that invokes the post-training quantization tool to produce quantized tensorflow-lite model:

$ export MODEL=efficientnet-edgetpu-S
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/${MODEL}.tar.gz
$ tar zxf ${MODEL}.tar.gz
$ python export_model.py --model_name=$MODEL --ckpt_dir=$MODEL --data_dir=/path/to/representative_dataset/ --output_tflite=${MODEL}_quant.tflite

To produce a float model that bypasses the post-training quantization:

$ python export_model.py --model_name=$MODEL --ckpt_dir=$MODEL --output_tflite=${MODEL}_float.tflite --quantize=False

|Model | Imagenet top-1 accuracy (float) | Imagenet top-1 accuracy (quantized) |
|------|----------------------|------------------|
|efficientnet-edgetpu-S| 77.23% | 77.0 % |
|efficientnet-edgetpu-M| 78.69 | 78.6 % |
|efficientnet-edgetpu-L| 80.62 | 80.2% |

The export_model.py script can also be used to export a tensorflow saved_model from a training checkpoint:

$ python export_model.py --model_name=$MODEL --ckpt_dir=/path/to/model-ckpt/ --output_saved_model_dir=/path/to/output_saved_model/ --output_tflite=${MODEL}_float.tflite --quantize=False

---

Models/Official/Efficientnet/Lite/README (models/official/efficientnet/lite/README.md) Copy Section

EfficientNet-lite

EfficientNet-lite are a set of mobile/IoT friendly image classification models. Notably, while EfficientNet-EdgeTPU that is specialized for Coral EdgeTPU, these EfficientNet-lite models run well on all mobile CPU/GPU/EdgeTPU.

Due to the requirements from edge devices, we mainly made the following changes based on the original EfficientNets.

* Remove squeeze-and-excite (SE): SE are not well supported for some mobile accelerators.
* Replace all swish with RELU6: for easier post-quantization.
* Fix the stem and head while scaling models up: for keeping models small and fast.

Here are the checkpoints, and their accurracy, params, flops, and Pixel4's CPU/GPU/EdgeTPU latency.

|Model | params | MAdds | FP32 accuracy | FP32 CPU latency | FP32 GPU latency | FP16 GPU latency |INT8 accuracy | INT8 CPU latency | INT8 TPU latency|
|------|-----|-------|-------|-------|-------|-------|-------|-------|-------|
|efficientnet-lite0 ckpt | 4.7M | 407M | 75.1% | 12ms | 9.0ms | 6.0ms | 74.4% | 6.5ms | 3.8ms |
|efficientnet-lite1 ckpt | 5.4M | 631M | 76.7% | 18ms | 12ms | 8.0ms | 75.9% | 9.1ms | 5.4ms |
|efficientnet-lite2 ckpt | 6.1M | 899M | 77.6% | 26ms | 16ms | 10ms | 77.0% | 12ms | 7.9ms |
|efficientnet-lite3 ckpt | 8.2M | 1.44B | 79.8% | 41ms | 23ms | 14ms | 79.0% | 18ms | 9.7ms |
|efficientnet-lite4 ckpt |13.0M | 2.64B | 81.5% | 76ms | 36ms | 21ms | 80.2% | 30ms | - |

* CPU/GPU/TPU latency are measured on Pixel4, with batch size 1 and 4 CPU threads. FP16 GPU latency is measured with default latency, while FP32 GPU latency is measured with additional option --gpu_precision_loss_allowed=false.

* Each checkpoint all contains FP tflite and post-training quantized INT8 tflite files. If you use these models or checkpoints, you can cite this efficientnet paper.

Comparing with MobileNetV2, ResNet-50, and Inception-V4, our models have
better trade-offs between accuracy and size/latency.
The following two figures show the comparison among quantized versions of
these models. The latency numbers are obtained on a Pixel 4 with 4 CPU
threads.

<p align="center">
<img src="../g3doc/lite-quant-size.png" width="60%" />
</p>

<p align="center">
<img src="../g3doc/lite-quant-cpu.png", width="60%" />
</p>

As Tensorflow Lite also provides GPU acceleration for float models, the
following shows the latency comparison among float versions of these
models. Again, the latency numbers are obtained on a Pixel 4.
<p align="center">
<img src="../g3doc/lite-float-gpu.png" width="60%" />
</p>

A quick way to use these checkpoints is to run:

$ export MODEL=efficientnet-lite0
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/${MODEL}.tar.gz
$ tar zxf ${MODEL}.tar.gz
$ wget https://upload.wikimedia.org/wikipedia/commons/f/fe/Giant_Panda_in_Beijing_Zoo_1.JPG -O panda.jpg
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/eval_data/labels_map.txt
$ python eval_ckpt_main.py --model_name=$MODEL --ckpt_dir=$MODEL --example_img=panda.jpg --labels_map_file=labels_map.txt

TFLite models can be evaluated using this tool.

Training EfficientNet-lite on Cloud TPUs Copy Section

Post-training quantization Copy Section

$ export MODEL=efficientnet-lite0
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/${MODEL}.tar.gz
$ tar zxf ${MODEL}.tar.gz
$ python export_model.py --model_name=$MODEL --ckpt_dir=$MODEL --data_dir=/path/to/representative_dataset/ --output_tflite=${MODEL}_quant.tflite

To produce a float model that bypasses the post-training quantization:

$ python export_model.py --model_name=$MODEL --ckpt_dir=$MODEL --output_tflite=${MODEL}_float.tflite --quantize=False

The export_model.py script can also be used to export a tensorflow saved_model from a training checkpoint:

$ python export_model.py --model_name=$MODEL --ckpt_dir=/path/to/model-ckpt/ --output_saved_model_dir=/path/to/output_saved_model/ --output_tflite=${MODEL}_float.tflite --quantize=False

---

Models/Official/Mask Rcnn/README (models/official/mask_rcnn/README.md) Copy Section

Try it

Try to run our pre-trained COCO Mask R-CNN using Colab.

Installing extra packages

Mask R-CNN requires a few extra packages. We can install them now:

sudo apt-get install -y python-tk && \
pip3 install --user Cython matplotlib opencv-python-headless pyyaml Pillow && \
pip3 install --user 'git+https://github.com/cocodataset/cocoapi#egg=pycocotools&subdirectory=PythonAPI'

---

Models/Official/Mask Rcnn/Mask Rcnn K8s.Yaml (models/official/mask_rcnn/mask_rcnn_k8s.yaml) Copy Section

Train Mask-RCNN with Coco dataset using Cloud TPU and Google Kubernetes Engine.

[Training Data]

Download and preprocess the COCO dataset using https://github.com/tensorflow/tpu/blob/r1.13/tools/datasets/download_and_preprocess_coco_k8s.yaml

if you don't already have the data.

[Instructions]

1. Follow the instructions on https://cloud.google.com/tpu/docs/kubernetes-engine-setup

to create a Kubernetes Engine cluster.

2. Change the environment variable MODEL_BUCKET in the Job spec to the

Google Cloud Storage location where you want to store the output model.

3. Run kubectl create -f mask_rcnn_k8s.yaml.

apiVersion: batch/v1
kind: Job
metadata:
name: mask-rcnn-gke-tpu
spec:
template:
metadata:
annotations:
# The Cloud TPUs that will be created for this Job must support
# TensorFlow 1.13. This version MUST match the TensorFlow version that
# your model is built on.
tf-version.cloud-tpus.google.com: "1.13"
spec:
restartPolicy: Never
containers:
- name: mask-rcnn-gke-tpu
# The official TensorFlow 1.13 TPU model image built from https://github.com/tensorflow/tpu/blob/r1.13/tools/docker/Dockerfile.
image: gcr.io/tensorflow/tpu-models:r1.13
command:
- /bin/sh
- -c
- >
DEBIAN_FRONTEND=noninteractive apt-get update &&
DEBIAN_FRONTEND=noninteractive apt-get install -y python-dev python-tk libsm6 libxrender1 libxrender-dev libgtk2.0-dev libxext6 libglib2.0 &&
pip install Cython matplotlib opencv-python-headless &&
pip install 'git+https://github.com/cocodataset/cocoapi#egg=pycocotools&subdirectory=PythonAPI' &&
python /tensorflow_tpu_models/models/official/mask_rcnn/mask_rcnn_main.py
--model_dir=${MODEL_BUCKET}
--params_override=iterations_per_loop=500,resnet_checkpoint=${RESNET_CHECKPOINT},resnet_depth=50,precision=bfloat16,train_batch_size=64,eval_batch_size=8,training_file_pattern=${DATA_BUCKET}/train-,validation_file_pattern=${DATA_BUCKET}/val-,val_json_file=${DATA_BUCKET}/instances_val2017.json,total_steps=22500
env:
# The Google Cloud Storage location to store dataset.
- name: DATA_BUCKET
value: "gs://<my-data-bucket>"
- name: MODEL_BUCKET
value: "gs://<my-model-bucket>/mask_rcnn"
- name: RESNET_CHECKPOINT
value: "gs://cloud-tpu-artifacts/resnet/resnet-nhwc-2018-02-07/model.ckpt-112603"

# Point PYTHONPATH to the top level models folder
- name: PYTHONPATH
value: "/tensorflow_tpu_models/models"
resources:
limits:
# Request a single v3-8 Cloud TPU device to train the model.
# A single v3-8 Cloud TPU device consists of 4 chips, each of which
# has 2 cores, so there are 8 cores in total.
cloud-tpus.google.com/v3: 8

---

Models/Official/Mask Rcnn/Configs/Cloud/V2 128.Yaml (models/official/mask_rcnn/configs/cloud/v2-128.yaml) Copy Section

---------- MODEL PARAMETERS -------------

---------- TRAINING PARAMETERS ----------

---------- EVAL PARAMETERS --------------

---

Models/Official/Mask Rcnn/Configs/Cloud/V2 32.Yaml (models/official/mask_rcnn/configs/cloud/v2-32.yaml) Copy Section

---------- MODEL PARAMETERS -------------

---------- TRAINING PARAMETERS ----------

---------- EVAL PARAMETERS --------------

---

Models/Official/Mask Rcnn/Configs/Cloud/V2 8.Yaml (models/official/mask_rcnn/configs/cloud/v2-8.yaml) Copy Section

---------- MODEL PARAMETERS -------------

---------- TRAINING PARAMETERS ----------

---------- EVAL PARAMETERS --------------

---

Models/Official/Mask Rcnn/Configs/Cloud/V3 128.Yaml (models/official/mask_rcnn/configs/cloud/v3-128.yaml) Copy Section

---------- MODEL PARAMETERS -------------

---------- TRAINING PARAMETERS ----------

---------- EVAL PARAMETERS --------------

---

Models/Official/Mask Rcnn/Configs/Cloud/V3 32.Yaml (models/official/mask_rcnn/configs/cloud/v3-32.yaml) Copy Section

---------- MODEL PARAMETERS -------------

---------- TRAINING PARAMETERS ----------

---------- EVAL PARAMETERS --------------

---

Models/Official/Mask Rcnn/Configs/Cloud/V3 8.Yaml (models/official/mask_rcnn/configs/cloud/v3-8.yaml) Copy Section

---------- MODEL PARAMETERS -------------

---------- TRAINING PARAMETERS ----------

---------- EVAL PARAMETERS --------------

---

Models/Official/Mnasnet/README (models/official/mnasnet/README.md) Copy Section

MnasNet

[1] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, Quoc V. Le. MnasNet: Platform-Aware Neural Architecture Search for Mobile. CVPR 2019.
Arxiv link: https://arxiv.org/abs/1807.11626

About the model and training regime Copy Chapter

MnasNet is a family of mobile hardware friendly neural networks, found by Mobile Neural Architecture Search (MNAS). It improves the accuracy and inference speed than previous state-of-the-art mobile models such as MobileNetV2. Here we provide a few standard-size and small-size AutoML models in mnasnet_models.py including:

* <b>mnasnet-a1 (ckpt)</b> has ~75.2% top-1 ImageNet accuracy with 3.9M parameters and 312M Multiply-Adds.
* <b>mnasnet-small (ckpt)</b> has ~66% top-1 ImageNet accuracy with 2.0M parameters and 68M Multiply-Adds.

The standard size MnasNet-A1 inference has 1.8x faster throughput (55% lower latency) than the
corresponding MobileNetV2 model.

Comparing to MobileNetV2,
MnasNet-A1 model has clear better performance in accuracy when they are at the
same latency level.

Here are the details of Mnasnet-A1 on ImageNet:

ckpt | Input Size | Depth Multiplier | Top-1 Acc | Top-5 Acc | Parameters(M) | Multi-Adds (M) | Pixel1 latency (ms)
------- |------- | ---------| --------- |---------|------|-------- | -------
ckpt | 224 | 1.4 | 77.2 | 93.5 | 6.1 | 591.5 | 135| 77.2
ckpt |224 | 1 | 75.2 | 92.5 | 3.9 | 315.2 | 78 | 75.2
ckpt |224 | 0.75| 73.3 | 91.3 | 2.9 | 226.7 | 61 | 73.3
ckpt |224 | 0.5 | 68.9 | 88.4 | 2.1 | 105.2 | 32 | 68.9
ckpt |224 | 0.35| 64.1 | 85.1 | 1.7 | 63.2 | 22| 64.1
ckpt |192 | 1.4 | 76.1 | 93.0 | 6.1 | 435.1 | 99 | 76.1
ckpt |192 | 1 | 74.0 | 91.6 | 3.9 | 232.0 | 57 | 74
ckpt |192 | 0.75| 72.1 | 90.5 | 2.9 | 166.9 | 45 | 72.1
ckpt |192 | 0.5 | 67.2 | 87.4 | 2.1 | 77.6 | 24| 67.2
ckpt |192 | 0.35| 62.4 | 83.8 | 1.7 | 46.8 | 17| 62.4
ckpt |160 | 1.4 | 74.8 | 92.1 | 6.1 | 302.8 | 72 | 74.8
ckpt |160 | 1 | 72.0 | 90.5 | 3.9 | 161.6 | 41 | 72
ckpt |160 | 0.75| 70.1 | 89.3 | 2.9 | 116.4 | 33 | 70.1
ckpt |160 | 0.5 | 64.9 | 85.8 | 2.1 | 54.4 | 18| 64.9
ckpt |160 | 0.35| 52.3 | 81.5 | 1.7 | 32.9 | 13| 59.3
ckpt |128 | 1.4 | 72.5 | 90.6 | 6.1 | 194.5 | 49 | 72.5
ckpt |128 | 1 | 69.3 | 88.9 | 3.9 | 104.1 | 29 | 69.3
ckpt |128 | 0.75| 67.0 | 87.3 | 2.9 | 75.0 | 23| 67
ckpt |128 | 0.5 | 60.8 | 83.0 | 2.1 | 35.3 | 12| 60.8
ckpt |128 | 0.35| 54.8 | 78.1 | 1.7 | 21.6 | 8.5| 54.8
ckpt |96 |1.4 | 68.6 | 88.1 | 6.1 | 110.3 | 32 | 68.6
ckpt |96 |1 | 64.4 | 85.8 | 3.9 | 59.3 | 18| 64.4
ckpt |96 |0.75| 62.1 | 84.0 | 2.9 | 42.9 | 17| 62.1
ckpt |96 |0.5 | 54.7 | 78.1 | 2.1 | 20.5 | 7.4 | 54.7
ckpt |96 |0.35| 49.3 | 73.4 | 1.7 | 12.7 | 5.4| 49.3

Understanding the code Copy Chapter

For more detailed information, read the documentation within each file.

* imagenet_input.py: Constructs the tf.data.Dataset
input pipeline which handles parsing, preprocessing, shuffling, and batching
the data samples.
* mnasnet_main.py: Main code which constructs the
TPUEstimator and handles training and evaluating the model.
* mnasnet_model.py: Modeling library which constructs the
network via modular MnasBlock.
* mnasnet_models.py: A script that defines benchmark
MnasNet architectures (e.g. MnasNet-A1, MnasNet-small) through decoding string
representation of the network.
* preprocessing.py: Useful utilities for
preprocessing and augmenting ImageNet data for MnasNet training.
* mnasnet_example.ipynb: A colab example to load
MnasNet saved model files and use it to classify images.

Serve the exported model in TFLite Copy Section

The export function of MnasNet trainer will always export TFLite float model
when the '--export_dir' is specified. Furthermore, once '--post_quantize=True'
is set, the trainer will also export a quantized TFLite model using the latest
model checkpoint.

Please see mnasnet_example.ipynb as an example on how to use these exported model.

Using different MnasNet configurations Copy Section

The default MnasNet models have been carefully tested with the default
flags but mnasnet_model.py offers a generic MnasNetBlock
implementation. Thus, user is able to define model configuration through the
string annotation in mnasnet_models.py to define new
models. Meanwhile, '--depth_multiplier', '--depth_divisor', '--min_depth' are
flags offered to adjust MnasNet model to different sizes quickly.

Training MnasNet on TPU Copy Section

please refer to our tutorial: https://cloud.google.com/tpu/docs/tutorials/mnasnet

---

Models/Official/Mnasnet/Configs/Cloud/Gpu.Yaml (models/official/mnasnet/configs/cloud/gpu.yaml) Copy Section

use_tpu: False
train_steps: 3503192 # 1281167 * 350 / train_batch_size
train_batch_size: 128 # 1024 / 8
eval_batch_size: 128 # 1024 / 8
model_name: 'mnasnet-a1'
dropout_rate: null
depth_multiplier: null

---

Models/Official/Mnasnet/Configs/Cloud/V2 32.Yaml (models/official/mnasnet/configs/cloud/v2-32.yaml) Copy Section

train_steps: 109474
train_batch_size: 4096
eval_batch_size: 256
iterations_per_loop: 100
skip_host_call: false
model_name: 'mnasnet-a1'
dropout_rate: null
depth_multiplier: null
use_keras: true
precision: 'float32'

---

Models/Official/Mnasnet/Configs/Cloud/V2 8.Yaml (models/official/mnasnet/configs/cloud/v2-8.yaml) Copy Section

train_steps: 437899
train_batch_size: 1024
eval_batch_size: 1024
iterations_per_loop: 1251
skip_host_call: True
model_name: 'mnasnet-a1'
dropout_rate: null
depth_multiplier: null

---

Models/Official/Mnasnet/Configs/Cloud/V3 32.Yaml (models/official/mnasnet/configs/cloud/v3-32.yaml) Copy Section

train_steps: 109474
train_batch_size: 4096
eval_batch_size: 256
iterations_per_loop: 100
skip_host_call: false
model_name: 'mnasnet-a1'
dropout_rate: null
depth_multiplier: null
use_keras: true
precision: 'float32'

---

Models/Official/Mnasnet/Configs/Cloud/V3 8.Yaml (models/official/mnasnet/configs/cloud/v3-8.yaml) Copy Section

train_steps: 437899
train_batch_size: 1024
eval_batch_size: 1024
iterations_per_loop: 1251
skip_host_call: True
model_name: 'mnasnet-a1'
dropout_rate: null
depth_multiplier: null

---

Models/Official/Mnasnet/Mixnet/README (models/official/mnasnet/mixnet/README.md) Copy Section

MixNet

[1] Mingxing Tan and Quoc V. Le. MixConv: Mixed Depthwise Convolutional Kernels.
BMVC 2019. https://arxiv.org/abs/1907.09595

1. About MixNet Copy Chapter

MixNets are a family of mobile-sizes image classification models equipped with MixConv, a new type of mixed depthwise convolutions. They are developed based on AutoML MNAS Mobile framework, with an extended search space including MixConv. Currently, MixNets achieve better accuracy and efficiency than previous mobile models. In particular, our MixNet-L achieves a new state-of-the-art 78.9% ImageNet top-1 accuracy under typical mobile FLOPS (<600M) constraint:

<table border="0", width="75%", align="center">
<tr>
<td>
<img src="./g3doc/mixnet-flops.png" width="100%" />
</td>
</tr>
</table>

2. Using Pretrained Checkpoints Copy Chapter

We have provided a list of EfficientNet checkpoints for MixNet-S, MixNet-M, and MixNet-L. A quick way to use these checkpoints is to run:

$ export MODEL=mixnet-s
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/mixnet/${MODEL}.tar.gz
$ tar zxf ${MODEL}.tar.gz
$ wget https://upload.wikimedia.org/wikipedia/commons/f/fe/Giant_Panda_in_Beijing_Zoo_1.JPG -O panda.jpg
$ wget https://storage.googleapis.com/cloud-tpu-checkpoints/efficientnet/eval_data/labels_map.txt
$ python eval_ckpt_main.py --model_name=$MODEL --ckpt_dir=$MODEL --example_img=panda.jpg --labels_map_file=labels_map.txt

Please refer to the following colab for more instructions on how to obtain and use those checkpoints.

* mixnet_eval_example.ipynb: A colab example to load
pretrained checkpoints files and use the restored model to classify images.

3. Training and Evaluating MixNets. Copy Chapter

MixNets are trained using the same hyper parameters as MnasNet, except specifying different model_name=mixnet-s/m/l.

For more instructions, please refer to the MnasNet tutorial: https://cloud.google.com/tpu/docs/tutorials/mnasnet

---

Models/Official/Mnist/README (models/official/mnist/README.md) Copy Section

mnist_tpu.py can be used to train a simple model on the MNIST dataset using
a Cloud TPU.

See https://cloud.google.com/tpu/docs/quickstart for more details.

---

Models/Official/Mobilenet/README (models/official/mobilenet/README.md) Copy Section

Cloud TPU Port of the MobileNet v1 model

This is a straightforward port of the MobileNet v1 model. The code was based on the original version from the tensorflow/models repository.

The only adjustments have been to add the required code to enable using the
TPUEstimator interface, along with the data processing pipeline for ImageNet.

Running the model Copy Chapter

Assuming you have a version of ImageNet converted to the tfrecord format located
at gs://my-cloud-bucket/data/imagenet/, you can run this model with the
following command:

python mobilenet.py\ 
  --alsologtostderr\
  --master=$TPU_WORKER\
  --data_dir=gs://my-cloud-bucket/data/imagenet\
  --model_dir=gs://my-cloud-bucket/models/mobilenet/v0\
  --num_shards=8\
  --batch_size=1024\
  --use_tpu=1\

Note that the mobilenet network requires a large number of epochs to converge
completely.

---

Models/Official/Mobilenet/Configs/Cloud/V2 128.Yaml (models/official/mobilenet/configs/cloud/v2-128.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V2 256.Yaml (models/official/mobilenet/configs/cloud/v2-256.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V2 32.Yaml (models/official/mobilenet/configs/cloud/v2-32.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V2 512.Yaml (models/official/mobilenet/configs/cloud/v2-512.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V2 8.Yaml (models/official/mobilenet/configs/cloud/v2-8.yaml) Copy Section

train_batch_size: 1024
train_steps: 8000000
eval_batch_size: 1024
iterations_per_loop: 100
num_cores: 8
train_steps_per_eval: 2000

---

Models/Official/Mobilenet/Configs/Cloud/V3 1024.Yaml (models/official/mobilenet/configs/cloud/v3-1024.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V3 128.Yaml (models/official/mobilenet/configs/cloud/v3-128.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V3 2048.Yaml (models/official/mobilenet/configs/cloud/v3-2048.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V3 256.Yaml (models/official/mobilenet/configs/cloud/v3-256.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V3 32.Yaml (models/official/mobilenet/configs/cloud/v3-32.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V3 512.Yaml (models/official/mobilenet/configs/cloud/v3-512.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V3 64.Yaml (models/official/mobilenet/configs/cloud/v3-64.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Mobilenet/Configs/Cloud/V3 8.Yaml (models/official/mobilenet/configs/cloud/v3-8.yaml) Copy Section

train_batch_size: 1024
train_steps: 8000000
eval_batch_size: 1024
iterations_per_loop: 100
num_cores: 8
train_steps_per_eval: 2000

---

Models/Official/Resnet/README (models/official/resnet/README.md) Copy Section

ResNet and ResNet-RS on TPU

Prerequisites Copy Chapter

If you want to train the model on Cloud TPU through the managed service
Cloud Machine Learning Engine, skip to the Train on Cloud Machine Learning Engine
section.

Setup a Google Cloud project Copy Section

Follow the instructions at the [Quickstart Guide][quickstart-guide] to get a GCE
VM with access to a Cloud TPU. It is also recommended that you try the [Cloud
TPU ResNet tutorial][resnet-tutorial], which covers both the quickstart and
training of the ResNet algorithm.

[quickstart-guide]: https://cloud.google.com/tpu/docs/quickstart
[resnet-tutorial]: https://cloud.google.com/tpu/docs/tutorials/resnet
[cmle]: https://cloud.google.com/ml-engine/

To run this model, you will need:

* A GCE VM instance with an associated Cloud TPU resource
* A GCS bucket to store your training checkpoints (the "model directory")
* (Optional): The ImageNet training and validation data preprocessed into
TFRecord format, and stored in GCS.

Formatting the data Copy Section

The data is expected to be formatted in TFRecord format, as generated by [this
script][imagenet-download-format-as-tfrecord].

If you do not have ImageNet dataset prepared, you can use a randomly generated
fake dataset to test the model. It is located at
gs://cloud-tpu-test-datasets/fake_imagenet.

[imagenet-download-format-as-tfrecord]: https://github.com/tensorflow/tpu/blob/master/tools/datasets/imagenet_to_gcs.py

Training the model Copy Chapter

1. Add the top-level /models folder to the Python path with the command

export PYTHONPATH="$PYTHONPATH:/path/to/models"

1. Train the model by executing the following command (substituting the appropriate
values):

python resnet_main.py \
  --tpu=$TPU_NAME \
  --data_dir=$DATA_DIR \
  --model_dir=$MODEL_DIR

$TPU_NAME is the name of the TPU node, the same name that appears when you
run gcloud compute tpus list, or ctpu ls. (When using the shell
created by ctpu up, this argument may not be necessary.)

$MODEL_DIR is a GCS location (a URL starting with gs:// where both the GCE
VM and the associated Cloud TPU have write access, something like gs://userid-
dev-imagenet-output/model. (TensorFlow can't create the bucket; you have to
create it with gcloud storage buckets create <bucket>.) This bucket is used to save checkpoints
and the training result, so that the training steps are cumulative when you
reuse the model directory. If you do 1000 steps, for example, and you reuse the
model directory, on a subsequent run, it will skip the first 1000 steps, because
it picks up where it left off.

$DATA_DIR is a GCS location to which both the GCE VM and associated Cloud TPU
have read access, something like gs://cloud-tpu-test-datasets/fake_imagenet.
This location is expected to contain files with the prefixes train-* and
validation-*. The former pattern is used to match files used for the training
phase, the latter for the evaluation phase.

Each file is a series of TFExample records. In the case of ResNet-50,
the TFExample records have a specific format, as follows:

keys_to_features = {
    'image/encoded': tf.FixedLenFeature((), tf.string, ''),
    'image/format': tf.FixedLenFeature((), tf.string, 'jpeg'),
    'image/class/label': tf.FixedLenFeature([], tf.int64, -1),
    'image/class/text': tf.FixedLenFeature([], tf.string, ''),
    'image/object/bbox/xmin': tf.VarLenFeature(dtype=tf.float32),
    'image/object/bbox/ymin': tf.VarLenFeature(dtype=tf.float32),
    'image/object/bbox/xmax': tf.VarLenFeature(dtype=tf.float32),
    'image/object/bbox/ymax': tf.VarLenFeature(dtype=tf.float32),
    'image/object/class/label': tf.VarLenFeature(dtype=tf.int64),
}

The training and validation data can also be sourced from Cloud Bigtable:

python resnet_main.py \
  --tpu=$TPU_NAME \
  --model_dir=$MODEL_DIR \
  --bigtable_project=$PROJECT \
  --bigtable_instance=$INSTANCE \
  --bigtable_table=$TABLE

In this case, the TFExample records are stored one per row in a Cloud Bigtable
table. Categories of data are arranged by row prefix, and the rows within that
prefix arranged by zero-filled indexes, e.g. train_0000003892.)

You can also specify the following arguments when sourcing data from Cloud
Bigtable, though they already have the right defaults for ResNet-50:

--bigtable_train_prefix=train_ \        # row prefix for training rows
  --bigtable_eval_prefix=validation_ \    # row prefix for evaluation rows
  --bigtable_column_family=tfexample \
  --bigtable_column_qualifier=example

Note that even when sourcing input data from Cloud Bigtable, $MODEL_DIR must
still be a GCS location.

Project and Zone Copy Section

If you are not running this script on a GCE VM in the same project and zone as
your Cloud TPU, you will need to add the --project and --zone flags
specifying the corresponding values for the Cloud TPU you'd like to use.

This will train a ResNet-50 model on ImageNet with 1024 batch size on a single
Cloud TPU. With the default flags on everything, the model should train to
above 76% accuracy in around 17 hours (including evaluation time every
--steps_per_eval steps).

You can launch TensorBoard (e.g. tensorboard -logdir=$MODEL_DIR) to view loss
curves and other metadata regarding your training run.

Note: if you launch TensorBoard on your GCE VM, be sure to configure either

[SSH port forwarding][ssh-port-fwd] or [SOCKS proxy over SSH][socks-proxy] to

connect to your GCE VM securely (recommended).

> Alternatively, you can modify your GCE firewall rules to open a port, but this

is not recommended as it enables insecure world-wide access for

everyone.

[ssh-port-fwd]: https://cloud.google.com/solutions/connecting-securely#port-forwarding-over-ssh
[socks-proxy]: https://cloud.google.com/solutions/connecting-securely#socks-proxy-over-ssh

Train on Cloud Machine Learning Engine Copy Chapter

To train this model on Machine Learning Engine, you will need:

* A GCP project with Cloud Machine Learning Engine enabled
* A GCS bucket to store your training checkpoints (the "model directory") and for staging the training package
* (Optional): The ImageNet training and validation data preprocessed into
TFRecord format, and stored in GCS.

Run the following command from the top level models folder:

GCS_BUCKET="gs://your-gcs-bucket"
JOB_NAME="tpu_resnet_sample"
REGION=us-central1
DATA_DIR=gs://cloud-tpu-test-datasets/fake_imagenet
BUCKET=$GCS_BUCKET
JOB_DIR=$BUCKET"/"$JOB_NAME
STAGING_BUCKET=$BUCKET
OUTPUT_PATH=$JOB_DIR
gcloud ml-engine jobs submit training $JOB_NAME \
    --staging-bucket $STAGING_BUCKET \
    --runtime-version 1.9 \
    --scale-tier BASIC_TPU \
    --module-name official.resnet.resnet_main \
    --package-path official \
    --region $REGION \
    -- \
    --data_dir=$DATA_DIR \
    --model_dir=$OUTPUT_PATH \
    --resnet_depth=50 \
    --train_steps=1024

Understanding the code Copy Chapter

For more detailed information, read the documentation within each file.

* imagenet_input.py: Constructs the tf.data.Dataset
input pipeline which handles parsing, preprocessing, shuffling, and batching
the data samples.
* resnet_main.py: Main code which constructs the
TPUEstimator and handles training and evaluating the model.
* resnet_model.py: ResNet model code which constructs the
network via modular residual blocks or bottleneck blocks.
* resnet_preprocessing.py: Useful utilities for
preprocessing and augmenting ImageNet data for ResNet training. Significantly
improves final accuracy.

Additional notes Copy Chapter

About the model and training regime Copy Section

The model is based on network architecture presented in Deep Residual Learning
for Image Recognition by Kaiming He, et. al.

Specifically, the model uses post-activation residual units for ResNet-18, and
34 and post-activation bottleneck units for ResNet-50, 101, 152, and 200. There
are a few differences to the model and training compared to the original paper:

* The preprocessing and data augmentation is slightly different. In particular,
we have an additional step during normalization which rescales the inputs
based on the stddev of the RGB values of the dataset. Additionally, we
have also implemented AutoAugment and RandAugment, which are forms of data
augmentation that substantially improve the final performance of the model.
* We use a larger batch size of 1024 (by default) instead of 256 and
linearly scale the learning rate. In addition, we adopt the learning rate
schedule suggested by Accurate, Large Minibatch SGD: Training ImageNet in 1
Hour and train for 90 epochs. Besides
BatchNorm-ReLU (default), we have also implemented EvoNorms that perform better
across a much wider range of batch sizes.
* We use a slightly different weight initialization for batch normalization in
the last batch norm per block, as inspired by the above paper.
* Evaluation is performed on a single center crop of the validation set rather
than a 10-crop from the original paper.

Training/evaluating/predicting on CPU/GPU Copy Section

To run the same code on CPU/GPU, set the flag --use_tpu=False. This will use
the default devices available to TensorFlow on your machine. The checkpoints
created by CPU/GPU and TPU are all identical so it is possible to train on one
type of device and then evaluate/predict using the trained model on a different
device.

Serve the exported model on CPU/GPU Copy Section

To serve the exported model on CPU, set the flag --data_format='channels_last'
as inference on CPU only supports channels_last. Inference on GPU supports
both channels_first and channels_last.

Using different ResNet configurations Copy Section

The default ResNet-50 has been carefully tested with the default flags but
resnet_model.py includes a few other commonly used
configurations including ResNet-18, 34, 101, 152, 200. The 18 and 34 layer
configurations use residual blocks without bottlenecks and the remaining
configurations use bottleneck layers. The configuration can be controlled via
--resnet_size. Bigger models require more training time and more memory, thus
may require lowering the --train_batch_size to avoid running out of memory.

Using your own data Copy Section

To use your own data with this model, you first need to write an input pipeline
similar to jpeg_to_tf_record.py.
It is recommended to use TFRecord format for storing your data on disk and
tf.data.Dataset for the actual pipeline. Then, pass in the dataset
constants as command-line parameters to resnet_main.py.

[jpeg_to_tfrecord]: https://github.com/tensorflow/tpu/blob/master/tools/datasets/jpeg_to_tf_record.py

Benchmarking the training speed Copy Section

Benchmarking code for DAWNBench can be
found under the benchmark/ subdirectory. The benchmarking code
imports the same models, inputs, and training regimes but includes some extra
checkpointing and evaluation.

---

Models/Official/Resnet/Resnet K8s.Yaml (models/official/resnet/resnet_k8s.yaml) Copy Section

Train ResNet-50 with fake ImageNet dataset using Cloud TPU and Google

Kubernetes Engine.

The tutorial is at https://cloud.google.com/tpu/docs/tutorials/kubernetes-engine-resnet.

[Training Data]

In this example, we use randomly generated fake ImageNet dataset at

gs://cloud-tpu-test-datasets/fake_imagenet as the training data.

[Instructions]

1. Follow the instructions on https://cloud.google.com/tpu/docs/kubernetes-engine-setup

to create a Kubernetes Engine cluster.

2. Change the environment variable MODEL_BUCKET in the Job spec to the

Google Cloud Storage location where you want to store the output model.

3. Run kubectl create -f resnet_k8s.yaml.

apiVersion: batch/v1
kind: Job
metadata:
name: resnet-tpu
spec:
template:
metadata:
annotations:
# The Cloud TPUs that will be created for this Job must support
# TensorFlow 1.11. This version MUST match the TensorFlow version that
# your model is built on.
tf-version.cloud-tpus.google.com: "1.11"
spec:
restartPolicy: Never
containers:
- name: resnet-tpu
# The official TensorFlow 1.11 TPU model image built from https://github.com/tensorflow/tpu/blob/r1.11/tools/docker/Dockerfile.
image: gcr.io/tensorflow/tpu-models:r1.11
command:
- python
- /tensorflow_tpu_models/models/official/resnet/resnet_main.py
- --data_dir=$(DATA_BUCKET)
- --model_dir=$(MODEL_BUCKET)
env:
# The Google Cloud Storage location where the fake ImageNet dataset is
# stored.
- name: DATA_BUCKET
value: "gs://cloud-tpu-test-datasets/fake_imagenet"
# [REQUIRED] Must specify the Google Cloud Storage location where your
# output model will be stored.
- name: MODEL_BUCKET
value: "gs://<my-model-bucket>/resnet"
# Point PYTHONPATH to the top level models folder
- name: PYTHONPATH
value: "/tensorflow_tpu_models/models"
resources:
limits:
# Request a single v2-8 Cloud TPU device to train the model.
# A single v2-8 Cloud TPU device consists of 4 chips, each of which
# has 2 cores, so there are 8 cores in total.
cloud-tpus.google.com/v2: 8

---

Models/Official/Resnet/Benchmark/README (models/official/resnet/benchmark/README.md) Copy Section

ResNet-50 Benchmark on Cloud TPU pods

Submission for DAWNBench.

This subdirectory contains the code needed to replicate the DAWNBench results
for ResNet-50 on a Cloud TPU pod. The model used here is identical to the model
in the parent directory. The only difference is that resnet_benchmark.py will
generate checkpoints at every epoch and evaluate in a separate job.

Instructions for training on single Cloud TPU Copy Chapter

1. Add the top-level /models folder to the Python path with the command

export PYTHONPATH="$PYTHONPATH:/path/to/models"

1. Train the model (roughly 90 epochs, 1 checkpoint per epoch):

python resnet_benchmark.py \
  --tpu=[TPU NAME] \
  --mode=train \
  --data_dir=[PATH TO DATA] \
  --model_dir=[PATH TO MODEL] \
  --train_batch_size=1024 \
  --train_steps=112590 \
  --iterations_per_loop=1251

1. Evaluate the model (run after train completes):

python resnet_benchmark.py \
  --tpu=[TPU NAME] \
  --mode=eval \
  --data_dir=[PATH TO DATA] \
  --model_dir=[PATH TO MODEL]

Instructions for training on a half TPU Pod Copy Chapter

Not yet available due to TPU Pod availability in Cloud.

---

Models/Official/Resnet/Configs/Cloud/Randaugment 32.Yaml (models/official/resnet/configs/cloud/randaugment-32.yaml) Copy Section

resnet_depth: 50
train_steps: 56304
train_batch_size: 4096
eval_batch_size: 1024
iterations_per_loop: 1000
num_cores: 32
skip_host_call: True
augment_name: 'randaugment'
randaug_num_layers: 2
randaug_magnitude: 9

---

Models/Official/Resnet/Configs/Cloud/Randaugment 8.Yaml (models/official/resnet/configs/cloud/randaugment-8.yaml) Copy Section

resnet_depth: 50
train_steps: 225216
train_batch_size: 1024
eval_batch_size: 1024
iterations_per_loop: 1000
num_cores: 8
skip_host_call: True
augment_name: 'randaugment'
randaug_num_layers: 2
randaug_magnitude: 9

---

Models/Official/Resnet/Configs/Cloud/V2 128.Yaml (models/official/resnet/configs/cloud/v2-128.yaml) Copy Section

train_steps: 7116
train_batch_size: 16384
eval_batch_size: 1024
iterations_per_loop: 7116
skip_host_call: True
num_cores: 128
enable_lars: True
label_smoothing: 0.1

---

Models/Official/Resnet/Configs/Cloud/V2 256.Yaml (models/official/resnet/configs/cloud/v2-256.yaml) Copy Section

train_steps: 3558
train_batch_size: 32768
eval_batch_size: 1024
iterations_per_loop: 3558
skip_host_call: True
num_cores: 256
enable_lars: True
label_smoothing: 0.1

---

Models/Official/Resnet/Configs/Cloud/V2 32.Yaml (models/official/resnet/configs/cloud/v2-32.yaml) Copy Section

train_steps: 28464
train_batch_size: 4096
eval_batch_size: 1024
iterations_per_loop: 28464
skip_host_call: True
num_cores: 32

---

Models/Official/Resnet/Configs/Cloud/V2 512.Yaml (models/official/resnet/configs/cloud/v2-512.yaml) Copy Section

train_steps: 3558
train_batch_size: 32768
eval_batch_size: 1024
iterations_per_loop: 3558
skip_host_call: True
num_cores: 512
enable_lars: True
label_smoothing: 0.1

---

Models/Official/Resnet/Configs/Cloud/V2 8.Yaml (models/official/resnet/configs/cloud/v2-8.yaml) Copy Section

train_steps: 113854
train_batch_size: 1024
eval_batch_size: 1024
iterations_per_loop: 113854
skip_host_call: True
num_cores: 8

---

Models/Official/Resnet/Configs/Cloud/V3 1024.Yaml (models/official/resnet/configs/cloud/v3-1024.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Resnet/Configs/Cloud/V3 128.Yaml (models/official/resnet/configs/cloud/v3-128.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Resnet/Configs/Cloud/V3 2048.Yaml (models/official/resnet/configs/cloud/v3-2048.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Resnet/Configs/Cloud/V3 256.Yaml (models/official/resnet/configs/cloud/v3-256.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Resnet/Configs/Cloud/V3 32.Yaml (models/official/resnet/configs/cloud/v3-32.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Resnet/Configs/Cloud/V3 512.Yaml (models/official/resnet/configs/cloud/v3-512.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Resnet/Configs/Cloud/V3 64.Yaml (models/official/resnet/configs/cloud/v3-64.yaml) Copy Section

DISCLAIMER: These parameters have not been optimized

---

Models/Official/Resnet/Configs/Cloud/V3 8.Yaml (models/official/resnet/configs/cloud/v3-8.yaml) Copy Section

train_steps: 113854
train_batch_size: 1024
eval_batch_size: 1024
iterations_per_loop: 113854
skip_host_call: True
num_cores: 8

---

Models/Official/Resnet/Resnet Rs/README (models/official/resnet/resnet_rs/README.md) Copy Section

Revisiting ResNets: Improved Training and Scaling Strategies

Revisiting ResNets: Improved Training and Scaling Strategies\
_Irwan Bello, William Fedus, Xianzhi Du, Ekin D. Cubuk, Aravind Srinivas, Tsung-Yi Lin, Jonathon Shlens, Barret Zoph_

<p align="center">
<img src="https://storage.googleapis.com/cloud-tpu-checkpoints/resnet-rs/promo_fig.png" width="70%" />
</p>

ResNet-RS is a family of simple ResNet architectures designed with improved training and scaling strategies that are 1.7x - 2.7x faster than EfficientNets on TPUv3 and 2.1x - 3.3x on V100 GPU.

#### Improved Scaling Strategies
The scaling strategies introduced in the paper are:

- (1) Scale the depth if overfitting can be an issue. If not, scale the width.
- (2) Scale image resolution slowly compared to prior works such as EfficientNet.

The improved scaling strategies also apply to other image classification architectures (e.g. EfficientNet).

#### Improved Training Strategies
The training strategy is a combination of multiple regularization and training techniques (see configs and Table 1 in the paper). These techniques are typically transferable to different architectures and to different tasks/datasets.

ImageNet Checkpoints Copy Chapter

We release configs and checkpoints of the ResNet-RS model family trained on ImageNet in Tensorflow 1.

| Model | Input Size | V100 Lat (s) | TPU Lat (ms) | Top-1 Accuracy | Download |
| ------------ |:-------------:| -----------:|--------:|-----------:|-----------:|
| ResNet-RS-50 | 160x160 | 0.31 | 70 | 78.8 | config \| ckpt |
| ResNet-RS-101 | 160x160 | 0.48 | 120 | 80.3 | config \| ckpt |
| ResNet-RS-101 | 192x192 | 0.70 | 170 | 81.2 | config \| ckpt |
| ResNet-RS-152 | 192x192 | 0.99 | 240 | 82.0 | config \| ckpt |
| ResNet-RS-152 | 224x224 | 1.48 | 320 | 82.2 | config \| ckpt |
| ResNet-RS-152 | 256x256 | 1.76 | 410 | 83.0 | config \| ckpt |
| ResNet-RS-200 | 256x256 | 2.86 | 570 | 83.4 | config \| ckpt |
| ResNet-RS-270 | 256x256 | 3.76 | 780 | 83.8 | config \| ckpt |
| ResNet-RS-350 | 256x256 | 4.72 | 1100| 84.0 | config \| ckpt |
| ResNet-RS-350 | 320x320 | 8.48 | 1630| 84.2 | config \| ckpt |
| ResNet-RS-420 | 320x320 | 10.16 | 2090| 84.4 | config \| ckpt |

#### Benchmarking details:

* Latencies on Tesla V100 GPUs are measured withfull precision (float32).
* Latencies on TPUv3 are measured using bfloat16 precision.
* All latencies are measured with an initial training batch size of 128 images, which is divided by 2 until it fits onto the accelerator.

Code and checkpoints are avaliable in Tensorflow 2 at the official Tensorflow Model Garden.

Citation Copy Chapter

@article{bello2021revisiting,
  title={Revisiting ResNets: Improved Training and Scaling Strategies},
  author={Irwan Bello and William Fedus and Xianzhi Du and Ekin D. Cubuk and Aravind Srinivas and Tsung-Yi Lin and Jonathon Shlens and Barret Zoph},
  journal={arXiv preprint arXiv:2103.07579},
  year={2021}
}

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs101 I160.Yaml (models/official/resnet/resnet_rs/configs/resnetrs101_i160.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: null
dropout_rate: 0.25
eval_batch_size: 200
image_size: 160
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 10
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 101
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs101 I192.Yaml (models/official/resnet/resnet_rs/configs/resnetrs101_i192.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: null
dropout_rate: 0.25
eval_batch_size: 200
image_size: 192
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 15
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 101
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs152 I192.Yaml (models/official/resnet/resnet_rs/configs/resnetrs152_i192.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: null
dropout_rate: 0.25
eval_batch_size: 200
image_size: 192
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 15
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 152
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs152 I224.Yaml (models/official/resnet/resnet_rs/configs/resnetrs152_i224.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: null
dropout_rate: 0.25
eval_batch_size: 200
image_size: 224
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 15
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 152
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs152 I256.Yaml (models/official/resnet/resnet_rs/configs/resnetrs152_i256.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: null
dropout_rate: 0.25
eval_batch_size: 200
image_size: 256
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 15
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 152
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs200 I256.Yaml (models/official/resnet/resnet_rs/configs/resnetrs200_i256.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: 0.1
dropout_rate: 0.25
eval_batch_size: 200
image_size: 256
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 15
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 200
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs270 I256.Yaml (models/official/resnet/resnet_rs/configs/resnetrs270_i256.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: 0.1
dropout_rate: 0.25
eval_batch_size: 200
image_size: 256
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 15
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 270
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs350 I256.Yaml (models/official/resnet/resnet_rs/configs/resnetrs350_i256.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: 0.1
dropout_rate: 0.25
eval_batch_size: 200
image_size: 256
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 15
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 350
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs350 I320.Yaml (models/official/resnet/resnet_rs/configs/resnetrs350_i320.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: 0.1
dropout_rate: 0.4
eval_batch_size: 200
image_size: 320
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 15
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 350
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs420 I320.Yaml (models/official/resnet/resnet_rs/configs/resnetrs420_i320.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: 0.1
dropout_rate: 0.4
eval_batch_size: 200
image_size: 320
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 15
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 420
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Resnet/Resnet Rs/Configs/Resnetrs50 I160.Yaml (models/official/resnet/resnet_rs/configs/resnetrs50_i160.yaml) Copy Section

augment_name: randaugment
base_learning_rate: 0.1
data_format: channels_last
drop_connect_rate: null
dropout_rate: 0.25
eval_batch_size: 200
image_size: 160
label_smoothing: 0.1
momentum: 0.9
num_eval_images: 50000
num_label_classes: 1000
num_train_images: 1281167
precision: bfloat16
randaug_magnitude: 10
randaug_num_layers: 2
replace_stem_max_pool: True
resnet_depth: 50
resnetd_shortcut: True
se_ratio: 0.25
train_batch_size: 4096
train_steps: 109475
transpose_input: True
use_resnetd_stem: True
use_tpu: True
weight_decay: 4.0e-05
moving_average_decay: 0.9999
bn_momentum: 0

---

Models/Official/Retinanet/README (models/official/retinanet/README.md) Copy Section

Training RetinaNet on Cloud TPU

This folder contains an implementation of the
RetinaNet object detection model.

The instructions below assume you are already familiar with running a model on
the TPU. If you haven't already, please review the instructions for running
the ResNet model on the Cloud TPU.

Check for the RetinaNet model Copy Chapter

If you are running on the prepared TPU image, the RetinaNet model files should
be pre-installed:

ls /usr/share/tpu/models/official/retinanet/

If they are not available, you can find the latest version on GitHub:

git clone https://github.com/tensorflow/tpu/
ls tpu/models/official/retinanet

Before we start Copy Chapter

Setting up our TPU VM Copy Section

The commands below assume you have started a TPU VM and set its name in an
environment variable:

gcloud beta compute tpus list
export TPU_NAME=my-tpu-name

GCS bucket for model checkpoints and training data Copy Section

We will also need a bucket to store out data and model files. We'll specify
that with the ${GCS_BUCKET} variable.