models Documentation

File: README.md

Deprecation Notice: We sincerely thank the community for participating in the ONNX Model Zoo effort. As the machine learning ecosystem has evolved, much of the novel model sharing has successfully transitioned to Hugging Face, which maintains a vibrant and healthy state. We are preserving the ONNX Model Zoo repository for historical purposes only. Please note that models will no longer be available for LFS download starting July 1st, 2025. You can still get access to the models that were originally available on this repository by going to https://huggingface.co/onnxmodelzoo.

ONNX Model Zoo

Introduction

Welcome to the ONNX Model Zoo! The Open Neural Network Exchange (ONNX) is an open standard format created to represent machine learning models. Supported by a robust community of partners, ONNX defines a common set of operators and a common file format to enable AI developers to use models with a variety of frameworks, tools, runtimes, and compilers.

This repository is a curated collection of pre-trained, state-of-the-art models in the ONNX format. These models are sourced from prominent open-source repositories and have been contributed by a diverse group of community members. Our aim is to facilitate the spread and usage of machine learning models among a wider audience of developers, researchers, and enthusiasts.

To handle ONNX model files, which can be large, we use Git LFS (Large File Storage).

Models

Currently, we are expanding the ONNX Model Zoo by incorporating additional models from the following categories.
As we are rigorously validating the new models for accuracy, refer to the validated models below that have been successfully validated for accuracy:

Computer Vision
Natural Language Processing (NLP)
Generative AI
Graph Machine Learning

These models are sourced from prominent open-source repositories such as timm, torchvision, torch_hub, and transformers, and exported into the ONNX format using the open-source TurnkeyML toolchain.

Validated Models

Read the Usage section below for more details on the file formats in the ONNX Model Zoo (.onnx, .pb, .npz), downloading multiple ONNX models through Git LFS command line, and starter Python code for validating your ONNX model using test data.

INT8 models are generated by Intel® Neural Compressor. Intel® Neural Compressor is an open-source Python library which supports automatic accuracy-driven tuning strategies to help user quickly find out the best quantized model. It implements dynamic and static quantization for ONNX models and can represent quantized ONNX models with operator oriented as well as tensor oriented (QDQ) ways. Users can use web-based UI service or python code to do quantization. Read the Introduction for more details.

Image Classification

This collection of models take images as input, then classifies the major objects in the images into 1000 object categories such as keyboard, mouse, pencil, and many animals.

Model Class	Reference	Description
MobileNet	Sandler et al.	Light-weight deep neural network best suited for mobile and embedded vision applications. Top-5 error from paper - ~10%
ResNet	He et al.	A CNN model (up to 152 layers). Uses shortcut connections to achieve higher accuracy when classifying images. Top-5 error from paper - ~3.6%
SqueezeNet	Iandola et al.	A light-weight CNN model providing AlexNet level accuracy with 50x fewer parameters. Top-5 error from paper - ~20%
VGG	Simonyan et al.	Deep CNN model(up to 19 layers). Similar to AlexNet but uses multiple smaller kernel-sized filters that provides more accuracy when classifying images. Top-5 error from paper - ~8%
AlexNet	Krizhevsky et al.	A Deep CNN model (up to 8 layers) where the input is an image and the output is a vector of 1000 numbers. Top-5 error from paper - ~15%
GoogleNet	Szegedy et al.	Deep CNN model(up to 22 layers). Comparatively smaller and faster than VGG and more accurate in detailing than AlexNet. Top-5 error from paper - ~6.7%
CaffeNet	Krizhevsky et al.	Deep CNN variation of AlexNet for Image Classification in Caffe where the max pooling precedes the local response normalization (LRN) so that the LRN takes less compute and memory.
RCNN_ILSVRC13	Girshick et al.	Pure Caffe implementation of R-CNN for image classification. This model uses localization of regions to classify and extract features from images.
DenseNet-121	Huang et al.	Model that has every layer connected to every other layer and passes on its own feature providing strong gradient flow and more diversified features.
Inception_V1	Szegedy et al.	This model is same as GoogLeNet, implemented through Caffe2 that has improved utilization of the computing resources inside the network and helps with the vanishing gradient problem. Top-5 error from paper - ~6.7%
Inception_V2	Szegedy et al.	Deep CNN model for Image Classification as an adaptation to Inception v1 with batch normalization. This model has reduced computational cost and improved image resolution compared to Inception v1. Top-5 error from paper ~4.82%
ShuffleNet_V1	Zhang et al.	Extremely computation efficient CNN model that is designed specifically for mobile devices. This model greatly reduces the computational cost and provides a ~13x speedup over AlexNet on ARM-based mobile devices. Compared to MobileNet, ShuffleNet achieves superior performance by a significant margin due to it's efficient structure. Top-1 error from paper - ~32.6%
ShuffleNet_V2	Zhang et al.	Extremely computation efficient CNN model that is designed specifically for mobile devices. This network architecture design considers direct metric such as speed, instead of indirect metric like FLOP. Top-1 error from paper - ~30.6%
ZFNet-512	Zeiler et al.	Deep CNN model (up to 8 layers) that increased the number of features that the network is capable of detecting that helps to pick image features at a finer level of resolution. Top-5 error from paper - ~14.3%
EfficientNet-Lite4	Tan et al.	CNN model with an order of magnitude of few computations and parameters, while still acheiving state-of-the-art accuracy and better efficiency than previous ConvNets. Top-5 error from paper - ~2.9%

Domain-based Image Classification

This subset of models classify images for specific domains and datasets.

Model Class	Reference	Description
MNIST-Handwritten Digit Recognition	Convolutional Neural Network with MNIST	Deep CNN model for handwritten digit identification

Object Detection & Image Segmentation

Object detection models detect the presence of multiple objects in an image and segment out areas of the image where the objects are detected. Semantic segmentation models partition an input image by labeling each pixel into a set of pre-defined categories.

Model Class	Reference	Description
Tiny YOLOv2	Redmon et al.	A real-time CNN for object detection that detects 20 different classes. A smaller version of the more complex full YOLOv2 network.
SSD	Liu et al.	Single Stage Detector: real-time CNN for object detection that detects 80 different classes.
SSD-MobileNetV1	Howard et al.	A variant of MobileNet that uses the Single Shot Detector (SSD) model framework. The model detects 80 different object classes and locates up to 10 objects in an image.
Faster-RCNN	Ren et al.	Increases efficiency from R-CNN by connecting a RPN with a CNN to create a single, unified network for object detection that detects 80 different classes.
Mask-RCNN	He et al.	A real-time neural network for object instance segmentation that detects 80 different classes. Extends Faster R-CNN as each of the 300 elected ROIs go through 3 parallel branches of the network: label prediction, bounding box prediction and mask prediction.
RetinaNet	Lin et al.	A real-time dense detector network for object detection that addresses class imbalance through Focal Loss. RetinaNet is able to match the speed of previous one-stage detectors and defines the state-of-the-art in two-stage detectors (surpassing R-CNN).
YOLO v2-coco	Redmon et al.	A CNN model for real-time object detection system that can detect over 9000 object categories. It uses a single network evaluation, enabling it to be more than 1000x faster than R-CNN and 100x faster than Faster R-CNN. This model is trained with COCO dataset and contains 80 classes.
YOLO v3	Redmon et al.	A deep CNN model for real-time object detection that detects 80 different classes. A little bigger than YOLOv2 but still very fast. As accurate as SSD but 3 times faster.
Tiny YOLOv3	Redmon et al.	A smaller version of YOLOv3 model.
YOLOv4	Bochkovskiy et al.	Optimizes the speed and accuracy of object detection. Two times faster than EfficientDet. It improves YOLOv3's AP and FPS by 10% and 12%, respectively, with mAP50 of 52.32 on the COCO 2017 dataset and FPS of 41.7 on a Tesla V100.
DUC	Wang et al.	Deep CNN based pixel-wise semantic segmentation model with >80% mIOU (mean Intersection Over Union). Trained on cityscapes dataset, which can be effectively implemented in self driving vehicle systems.
FCN	Long et al.	Deep CNN based segmentation model trained end-to-end, pixel-to-pixel that produces efficient inference and learning. Built off of AlexNet, VGG net, GoogLeNet classification methods. contribute

Body, Face & Gesture Analysis

Face detection models identify and/or recognize human faces and emotions in given images. Body and Gesture Analysis models identify gender and age in given image.

Model Class	Reference	Description
ArcFace	Deng et al.	A CNN based model for face recognition which learns discriminative features of faces and produces embeddings for input face images.
UltraFace	Ultra-lightweight face detection model	This model is a lightweight facedetection model designed for edge computing devices.
Emotion FerPlus	Barsoum et al.	Deep CNN for emotion recognition trained on images of faces.
Age and Gender Classification using Convolutional Neural Networks	Rothe et al.	This model accurately classifies gender and age even the amount of learning data is limited.

Image Manipulation

Image manipulation models use neural networks to transform input images to modified output images. Some popular models in this category involve style transfer or enhancing images by increasing resolution.

Model Class	Reference	Description
Unpaired Image to Image Translation using Cycle consistent Adversarial Network	Zhu et al.	The model uses learning to translate an image from a source domain X to a target domain Y in the absence of paired examples. contribute
Super Resolution with sub-pixel CNN	Shi et al.	A deep CNN that uses sub-pixel convolution layers to upscale the input image.
Fast Neural Style Transfer	Johnson et al.	This method uses a loss network pretrained for image classification to define perceptual loss functions that measure perceptual differences in content and style between images. The loss network remains fixed during the training process.

Speech & Audio Processing

This class of models uses audio data to train models that can identify voice, generate music, or even read text out loud.

Model Class	Reference	Description
Speech recognition with deep recurrent neural networks	Graves et al.	A RNN model for sequential data for speech recognition. Labels problems where the input-output alignment is unknowncontribute
Deep voice: Real time neural text to speech	Arik et al.	A DNN model that performs end-to-end neural speech synthesis. Requires fewer parameters and it is faster than other systems. contribute
Sound Generative models	WaveNet: A Generative Model for Raw Audio	A CNN model that generates raw audio waveforms. Has predictive distribution for each audio sample. Generates realistic music fragments. contribute

Machine Comprehension

This subset of natural language processing models that answer questions about a given context paragraph.

Model Class	Reference	Description
Bidirectional Attention Flow	Seo et al.	A model that answers a query about a given context paragraph.
BERT-Squad	Devlin et al.	This model answers questions based on the context of the given input paragraph.
RoBERTa	Liu et al.	A large transformer-based model that predicts sentiment based on given input text.
GPT-2	Radford et al.	A large transformer-based language model that given a sequence of words within some text, predicts the next word.
T5	Raffel et al.	A large transformer-based language model trained on multiple tasks at once to achieve better semantic understanding of the prompt, capable of sentiment-analysis, question-answering, similarity-detection, translation, summarization, etc.

Machine Translation

This class of natural language processing models learns how to translate input text to another language.

Model Class	Reference	Description
Neural Machine Translation by jointly learning to align and translate	Bahdanau et al.	Aims to build a single neural network that can be jointly tuned to maximize the translation performance. contribute
Google's Neural Machine Translation System	Wu et al.	This model helps to improve issues faced by the Neural Machine Translation (NMT) systems like parallelism that helps accelerate the final translation speed.contribute

Language Modelling

This subset of natural language processing models learns representations of language from large corpuses of text.

Model Class	Reference	Description
Deep Neural Network Language Models	Arisoy et al.	A DNN acoustic model. Used in many natural language technologies. Represents a probability distribution over all possible word strings in a language. contribute

Visual Question Answering & Dialog

This subset of natural language processing models uses input images to answer questions about those images.

Model Class	Reference	Description
VQA: Visual Question Answering	Agrawal et al.	A model that takes an image and a free-form, open-ended natural language question about the image and outputs a natural-language answer. contribute
Yin and Yang: Balancing and Answering Binary Visual Questions	Zhang et al.	Addresses VQA by converting the question to a tuple that concisely summarizes the visual concept to be detected in the image. Next, if the concept can be found in the image, it provides a “yes” or “no” answer. Its performance matches the traditional VQA approach on unbalanced dataset, and outperforms it on the balanced dataset. contribute
Making the V in VQA Matter	Goyal et al.	Balances the VQA dataset by collecting complementary images such that every question is associated with a pair of similar images that result in two different answers to the question, providing a unique interpretable model that provides a counter-example based explanation. contribute
Visual Dialog	Das et al.	An AI agent that holds a meaningful dialog with humans in natural, conversational language about visual content. Curates a large-scale Visual Dialog dataset (VisDial). contribute

Other interesting models

There are many interesting deep learning models that do not fit into the categories described above. The ONNX team would like to highly encourage users and researchers to contribute their models to the growing model zoo.

Model Class	Reference	Description
Text to Image	Generative Adversarial Text to image Synthesis	Effectively bridges the advances in text and image modeling, translating visual concepts from characters to pixels. Generates plausible images of birds and flowers from detailed text descriptions. contribute
Time Series Forecasting	Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks	The model extracts short-term local dependency patterns among variables and to discover long-term patterns for time series trends. It helps to predict solar plant energy output, electricity consumption, and traffic jam situations. contribute
Recommender systems	DropoutNet: Addressing Cold Start in Recommender Systems	A collaborative filtering method that makes predictions about an individual’s preference based on preference information from other users.contribute
Collaborative filtering	Neural Collaborative Filtering	A DNN model based on the interaction between user and item features using matrix factorization. contribute
Autoencoders	A Hierarchical Neural Autoencoder for Paragraphs and Documents	An LSTM (long-short term memory) auto-encoder to preserve and reconstruct multi-sentence paragraphs.contribute

Usage

Every ONNX backend should support running the models out of the box. After downloading and extracting the tarball of each model, you will find:

A protobuf file model.onnx that represents the serialized ONNX model.
Test data (in the form of serialized protobuf TensorProto files or serialized NumPy archives).

Usage - Test data starter code

The test data files can be used to validate ONNX models from the Model Zoo. We have provided the following interface examples for you to get started. Please replace onnx_backend in your code with the appropriate framework of your choice that provides ONNX inferencing support, and likewise replace backend.run_model with the framework's model evaluation logic.

There are two different formats for the test data files:

Serialized protobuf TensorProtos (.pb), stored in folders with the naming convention test_data_set_*.

python

import numpy as np
import onnx
import os
import glob
import onnx_backend as backend

from onnx import numpy_helper

model = onnx.load('model.onnx')
test_data_dir = 'test_data_set_0'

# Load inputs
inputs = []
inputs_num = len(glob.glob(os.path.join(test_data_dir, 'input_*.pb')))
for i in range(inputs_num):
    input_file = os.path.join(test_data_dir, 'input_{}.pb'.format(i))
    tensor = onnx.TensorProto()
    with open(input_file, 'rb') as f:
        tensor.ParseFromString(f.read())
    inputs.append(numpy_helper.to_array(tensor))

# Load reference outputs
ref_outputs = []
ref_outputs_num = len(glob.glob(os.path.join(test_data_dir, 'output_*.pb')))
for i in range(ref_outputs_num):
    output_file = os.path.join(test_data_dir, 'output_{}.pb'.format(i))
    tensor = onnx.TensorProto()
    with open(output_file, 'rb') as f:
        tensor.ParseFromString(f.read())
    ref_outputs.append(numpy_helper.to_array(tensor))

# Run the model on the backend
outputs = list(backend.run_model(model, inputs))

# Compare the results with reference outputs.
for ref_o, o in zip(ref_outputs, outputs):
    np.testing.assert_almost_equal(ref_o, o)

Serialized Numpy archives, stored in files with the naming convention test_data_*.npz. Each file contains one set of test inputs and outputs.

python

import numpy as np
import onnx
import onnx_backend as backend

# Load the model and sample inputs and outputs
model = onnx.load(model_pb_path)
sample = np.load(npz_path, encoding='bytes')
inputs = list(sample['inputs'])
outputs = list(sample['outputs'])

# Run the model with an onnx backend and verify the results
np.testing.assert_almost_equal(outputs, backend.run_model(model, inputs))

Usage - Model quantization

You can get quantized ONNX models by using Intel® Neural Compressor. It provides web-based UI service to make quantization easier and supports code-based usage for more abundant quantization settings. Refer to bench document for how to use web-based UI service and example document for a simple code-based demo.

Usage

There are multiple ways to access the ONNX Model Zoo:

Git Clone (Not Recommended)

Cloning the repository using git won't automatically download the ONNX models due to their size. To manage these files, first, install Git LFS by running:

bash

pip install git-lfs

To download a specific model:

bash

git lfs pull --include="[path to model].onnx" --exclude=""

To download all models:

bash

git lfs pull --include="*" --exclude=""

GitHub UI

Alternatively, you can download models directly from GitHub. Navigate to the model's page and click the "Download" button on the top right corner.

Model Visualization

For a graphical representation of each model's architecture, we recommend using Netron.

Contributions

Contributions to the ONNX Model Zoo are welcome! Please check our contribution guidelines for more information on how you can contribute to the growth and improvement of this resource.

Thank you for your interest in the ONNX Model Zoo, and we look forward to your participation in our community!

License

Apache License v2.0

File: validated/vision/super_resolution/sub_pixel_cnn_2016/README.md

Super Resolution

Use cases

The Super Resolution machine learning model sharpens and upscales the input image to refine the details and improve quality.

Description

Super Resolution uses efficient Sub-pixel convolutional layer described for increasing spatial resolution within network tasks. By increasing pixel count, images are then clarified, sharpened, and upscaled without losing the input image’s content and characteristics.

Model

Model	Download	Download (with sample test data)	ONNX version	Opset Version
Super_Resolution	240 KB	7.6 MB	1.5.0	10

Inference

Get started with this model by running through the included inference notebook for Super Resolution or following the steps below.

Input

Image input sizes are dynamic. The inference was done using jpg image.

Preprocessing

Images are resized into (224x224). The image format is changed into YCbCr with color components: greyscale ‘Y’, blue-difference ‘Cb’, and red-difference ‘Cr’. Once the greyscale Y component is extracted, it is then passed through the super resolution model and upscaled.

text

from PIL import Image
from resizeimage import resizeimage
import numpy as np
orig_img = Image.open('IMAGE_FILE_PATH')
img = resizeimage.resize_cover(orig_img, [224,224], validate=False)
img_ycbcr = img.convert('YCbCr')
img_y_0, img_cb, img_cr = img_ycbcr.split()
img_ndarray = np.asarray(img_y_0)
img_4 = np.expand_dims(np.expand_dims(img_ndarray, axis=0), axis=0)
img_5 = img_4.astype(np.float32) / 255.0
img_5

Output

The model outputs a multidimensional array of pixels that are upscaled. Output shape is [batch_size,1,672,672]. The second dimension is one because only the (Y) intensity channel was passed into the super resolution model and upscaled.

Postprocessing

Postprocessing involves converting the array of pixels into an image that is scaled to a higher resolution. The color channels (Cb, Cr) are also scaled to a higher resolution using bicubic interpolation. Then the color channels are combined and converted back to RGB format, producing the final output image.

text

final_img = Image.merge(
"YCbCr", [
    img_out_y,
    img_cb.resize(img_out_y.size, Image.BICUBIC),
    img_cr.resize(img_out_y.size, Image.BICUBIC),
]).convert("RGB")
plt.imshow(final_img)

Dataset

This model is trained on the BSD300 Dataset, using crops from the 200 training images.

Training

View the training notebook to understand details for parameters and network for SuperResolution.

File: validated/vision/style_transfer/fast_neural_style/README.md

Fast Neural Style Transfer

Use-cases

This artistic style transfer model mixes the content of an image with the style of another image. Examples of the styles can be seen in this PyTorch example.

Description

The model uses the method described in Perceptual Losses for Real-Time Style Transfer and Super-Resolution along with Instance Normalization.

Model

Model	Download	Download (with sample test data)	ONNX version	Opset version
Mosaic	6.6 MB	7.2 MB	1.4	9
Candy	6.6 MB	7.2 MB	1.4	9
Rain Princess	6.6 MB	7.2 MB	1.4	9
Udnie	6.6 MB	7.2 MB	1.4	9
Pointilism	6.6 MB	7.2 MB	1.4	9
Mosaic	6.6 MB	7.2 MB	1.4	8
Candy	6.6 MB	7.2 MB	1.4	8
Rain Princess	6.6 MB	7.2 MB	1.4	8
Udnie	6.6 MB	7.2 MB	1.4	8
Pointilism	6.6 MB	7.2 MB	1.4	8

Inference

Refer to style-transfer-ort.ipynb for detailed preprocessing and postprocessing.

Input to model

The input to the model are 3-channel RGB images. The images have to be loaded in a range of [0, 255]. If running into memory issues, try resizing the image by increasing the scale number.

Preprocessing steps

text

from PIL import Image
import numpy as np

# loading input and resize if needed
image = Image.open("PATH TO IMAGE")
size_reduction_factor = 1
image = image.resize((int(image.size[0] / size_reduction_factor), int(image.size[1] / size_reduction_factor)), Image.ANTIALIAS)

# Preprocess image
x = np.array(image).astype('float32')
x = np.transpose(x, [2, 0, 1])
x = np.expand_dims(x, axis=0)

Output of model

The converted ONNX model outputs a NumPy float32 array of shape [1, 3, ‘height’, ‘width’]. The height and width of the output image are the same as the height and width of the input image.

Postprocessing steps

text

result = np.clip(result, 0, 255)
result = result.transpose(1,2,0).astype("uint8")
img = Image.fromarray(result)

Dataset (Train and validation)

The original fast neural style model is from pytorch/examples/fast_neural_style. All models are trained using the COCO 2014 Training images dataset [80K/13GB].

Training

Refer to pytorch/examples/fast_neural_style for training details in PyTorch. Refer to conversion.ipynb to learn how the PyTorch models are converted to ONNX format.

References

Original style transfer model in PyTorch: https://github.com/pytorch/examples/tree/master/fast_neural_style

Contributors

Jennifer Wang

License

BSD-3-Clause

File: validated/vision/object_detection_segmentation/yolov4/README.md

YOLOv4

Description

YOLOv4 optimizes the speed and accuracy of object detection. It is two times faster than EfficientDet. It improves YOLOv3's AP and FPS by 10% and 12%, respectively, with mAP50 of 52.32 on the COCO 2017 dataset and FPS of 41.7 on Tesla 100.

Model

Model	Download	Download (with sample test data)	ONNX version	Opset version	Accuracy
YOLOv4	251 MB	236 MB	1.6	11	mAP of 0.5733

Source

Tensorflow YOLOv4 => ONNX YOLOv4

Inference

Conversion

A tutorial for the conversion process can be found in the conversion notebook.

Validation of the converted model and a graph representation of it can be found in the validation notebook.

Running inference

A tutorial for running inference using onnxruntime can be found in the inference notebook.

Input to model

This model expects input shapes of (1, 416, 416, 3). Each dimension represents the following variables: (batch_size, height, width, channels).

Preprocessing steps

The following code shows how preprocessing is done. For more information and an example on how preprocess is done, please visit the inference notebook.

python

import numpy as np
import cv2

# this function is from tensorflow-yolov4-tflite/core/utils.py
def image_preprocess(image, target_size, gt_boxes=None):

    ih, iw = target_size
    h, w, _ = image.shape

    scale = min(iw/w, ih/h)
    nw, nh = int(scale * w), int(scale * h)
    image_resized = cv2.resize(image, (nw, nh))

    image_padded = np.full(shape=[ih, iw, 3], fill_value=128.0)
    dw, dh = (iw - nw) // 2, (ih-nh) // 2
    image_padded[dh:nh+dh, dw:nw+dw, :] = image_resized
    image_padded = image_padded / 255.

    if gt_boxes is None:
        return image_padded

    else:
        gt_boxes[:, [0, 2]] = gt_boxes[:, [0, 2]] * scale + dw
        gt_boxes[:, [1, 3]] = gt_boxes[:, [1, 3]] * scale + dh
        return image_padded, gt_boxes

# input
input_size = 416

original_image = cv2.imread("input.jpg")
original_image = cv2.cvtColor(original_image, cv2.COLOR_BGR2RGB)
original_image_size = original_image.shape[:2]

image_data = image_preprocess(np.copy(original_image), [input_size, input_size])
image_data = image_data[np.newaxis, ...].astype(np.float32)

Output of model

Output shape: (1, 52, 52, 3, 85)

There are 3 output layers. For each layer, there are 255 outputs: 85 values per anchor, times 3 anchors.

The 85 values of each anchor consists of 4 box coordinates describing the predicted bounding box (x, y, h, w), 1 object confidence, and 80 class confidences. Here is the class list.

Postprocessing steps

The following postprocessing steps are modified from the hunglc007/tensorflow-yolov4-tflite repository.

text

/* Detailed source-code truncated for AI context efficiency. */

Dataset

Pretrained yolov4 weights can be downloaded here.

Validation accuracy

mAP50 on COCO 2017 dataset is 0.5733, based on the original tensorflow model.

Publication/Attribution

YOLOv4: Optimal Speed and Accuracy of Object Detection. Alexey Bochkovskiy, Chien-Yao Wang, Hong-Yuan Mark Liao.
Original models from Darknet Github repository.

References

This model is directly converted from hunglc007/tensorflow-yolov4-tflite.

Contributors

Jennifer Wang

License

MIT License

File: validated/vision/object_detection_segmentation/yolov3/README.md

YOLOv3

Description

This model is a neural network for real-time object detection that detects 80 different classes. It is very fast and accurate.

Model

Model	Download	Download (with sample test data)	ONNX version	Opset version	Accuracy
YOLOv3	237 MB	222 MB	1.5	10	mAP of 0.553
YOLOv3-12	237 MB	222 MB	1.9	12	mAP of 0.2874
YOLOv3-12-int8	60 MB	53 MB	1.9	12	mAP of 0.2693

Compared with the YOLOv3-12, YOLOv3-12-int8's mAP decline is 0.0181 and performance improvement is 2.19x.

Note the performance depends on the test hardware.

Performance data here is collected with Intel® Xeon® Platinum 8280 Processor, 1s 4c per instance, CentOS Linux 8.3, data batch size is 1.

Inference

Input to model

Resized image (1x3x416x416)
Original image size (1x2) which is [image.size[1], image.size[0]]

Preprocessing steps

The images have to be loaded in to a range of [0, 1]. The transformation should preferrably happen at preprocessing.

The following code shows how to preprocess a NCHW tensor:

python

import numpy as np
from PIL import Image

# this function is from yolo3.utils.letterbox_image
def letterbox_image(image, size):
    '''resize image with unchanged aspect ratio using padding'''
    iw, ih = image.size
    w, h = size
    scale = min(w/iw, h/ih)
    nw = int(iw*scale)
    nh = int(ih*scale)

    image = image.resize((nw,nh), Image.BICUBIC)
    new_image = Image.new('RGB', size, (128,128,128))
    new_image.paste(image, ((w-nw)//2, (h-nh)//2))
    return new_image

def preprocess(img):
    model_image_size = (416, 416)
    boxed_image = letterbox_image(img, tuple(reversed(model_image_size)))
    image_data = np.array(boxed_image, dtype='float32')
    image_data /= 255.
    image_data = np.transpose(image_data, [2, 0, 1])
    image_data = np.expand_dims(image_data, 0)
    return image_data

image = Image.open(img_path)
# input
image_data = preprocess(image)
image_size = np.array([image.size[1], image.size[0]], dtype=np.int32).reshape(1, 2)

Output of model

The model has 3 outputs.
boxes: (1x'n_candidates'x4), the coordinates of all anchor boxes,
scores: (1x80x'n_candidates'), the scores of all anchor boxes per class,
indices: ('nbox'x3), selected indices from the boxes tensor. The selected index format is (batch_index, class_index, box_index). The class list is here

Postprocessing steps

Post processing and meaning of output

text

out_boxes, out_scores, out_classes = [], [], []
for idx_ in indices:
    out_classes.append(idx_[1])
    out_scores.append(scores[tuple(idx_)])
    idx_1 = (idx_[0], idx_[2])
    out_boxes.append(boxes[idx_1])

out_boxes, out_scores, out_classes are list of resulting boxes, scores, and classes.

Dataset (Train and validation)

We use pretrained weights from pjreddie.com here.

Validation accuracy

YOLOv3:
Metric is COCO box mAP (averaged over IoU of 0.5:0.95), computed over 2017 COCO val data.
mAP of 0.553 based on original Yolov3 model here

YOLOv3-12 & YOLOv3-12-int8:
Metric is COCO box mAP@[IoU=0.50:0.95 | area=all | maxDets=100], computed over 2017 COCO val data.

Quantization

YOLOv3-12-int8 is obtained by quantizing YOLOv3-12 model. We use Intel® Neural Compressor with onnxruntime backend to perform quantization. View the instructions to understand how to use Intel® Neural Compressor for quantization.

Environment

onnx: 1.9.0
onnxruntime: 1.10.0

Prepare model

shell

wget https://github.com/onnx/models/raw/main/vision/object_detection_segmentation/yolov3/model/yolov3-12.onnx

Model quantize

bash

bash run_tuning.sh --input_model=path/to/model \  # model path as *.onnx
                   --config=yolov3.yaml \
                   --data_path=path/to/COCO2017 \
                   --output_model=path/to/save

Publication/Attribution

Joseph Redmon, Ali Farhadi. YOLOv3: An Incremental Improvement, paper

References

This model is converted from a keras model repository using keras2onnx converter repository.
Intel® Neural Compressor

Contributors

mengniwang95 (Intel)
airMeng (Intel)
ftian1 (Intel)
hshen14 (Intel)

License

MIT License

File: validated/vision/object_detection_segmentation/yolov2-coco/README.md

YOLOv2-COCO

Description

This model aims to detect objects in real time. It detects 80 different classes from the COCO Datasets. For information on network architecture, see the author's page and white paper.

Model

The model was converted to ONNX from PyTorch version of YOLOv2 using PyTorch-Yolo2. The output is fully verified by generating bounding boxes under PyTorch and onnxruntime.

Model	Download	Download (with sample test data)	ONNX version	Opset version
YOLOv2	203.9 MB	182.6 MB	1.5	9

Inference

Input to model

shape (1x3x416x416)

Output of model

shape (1x425x13x13)

Postprocessing steps

The output is a (1x425x13x13) tensor where 13x13 is the number of grid cells that the image gets divided into. Each grid cell corresponds to 5 anchors, made up of the 5 bounding boxes predicted by the grid cell and the 80 classes that describe each bounding box (5 x (80 classes + 5) = 425). For more information on how to derive the final bounding boxes and their corresponding confidence scores, refer to this post and PyTorch source code.

Dataset (Train and validation)

The YOLOv2 model was trained on the COCO datasets and was sourced from the original yolov2-voc .cfg and .weights files from link.

References

"YOLO9000: Better, Faster, Stronger" arXiv:1612.08242

License

MIT License

File: validated/vision/object_detection_segmentation/tiny-yolov3/README.md

Tiny YOLOv3

Description

This model is a neural network for real-time object detection that detects 80 different classes. It is very fast and accurate. It is a smaller version of YOLOv3 model.

Model

Model	Download	Download (with sample test data)	ONNX version	Opset version	Accuracy
Tiny YOLOv3	34 MB	33 MB	1.6	11	mAP of 0.331

Inference

Input to model

Resized image (1x3x416x416)
Original image size (1x2) which is [image.size[1], image.size[0]]

Preprocessing steps

The images have to be loaded in to a range of [0, 1]. The transformation should preferrably happen at preprocessing.

The following code shows how to preprocess a NCHW tensor:

python

import numpy as np
from PIL import Image

# this function is from yolo3.utils.letterbox_image
def letterbox_image(image, size):
    '''resize image with unchanged aspect ratio using padding'''
    iw, ih = image.size
    w, h = size
    scale = min(w/iw, h/ih)
    nw = int(iw*scale)
    nh = int(ih*scale)

    image = image.resize((nw,nh), Image.BICUBIC)
    new_image = Image.new('RGB', size, (128,128,128))
    new_image.paste(image, ((w-nw)//2, (h-nh)//2))
    return new_image

def preprocess(img):
    model_image_size = (416, 416)
    boxed_image = letterbox_image(img, tuple(reversed(model_image_size)))
    image_data = np.array(boxed_image, dtype='float32')
    image_data /= 255.
    image_data = np.transpose(image_data, [2, 0, 1])
    image_data = np.expand_dims(image_data, 0)
    return image_data

image = Image.open(img_path)
# input
image_data = preprocess(image)
image_size = np.array([image.size[1], image.size[0]], dtype=np.float32).reshape(1, 2)

Output of model

Postprocessing steps

Post processing and meaning of output

python

out_boxes, out_scores, out_classes = [], [], []
for idx_ in indices[0]:
    out_classes.append(idx_[1])
    out_scores.append(scores[tuple(idx_)])
    idx_1 = (idx_[0], idx_[2])
    out_boxes.append(boxes[idx_1])

out_boxes, out_scores, out_classes are list of resulting boxes, scores, and classes.

Dataset (Train and validation)

We use pretrained weights from pjreddie.com here.

Validation accuracy

Metric is COCO box mAP (averaged over IoU of 0.5:0.95), computed over 2017 COCO val data.
mAP of 0.331 based on original tiny Yolov3 model here

Publication/Attribution

Joseph Redmon, Ali Farhadi. YOLOv3: An Incremental Improvement, paper

References

This model is converted from a keras model repository using keras2onnx converter repository.

License

MIT License

File: validated/vision/object_detection_segmentation/tiny-yolov2/README.md

Tiny YOLOv2

Description

This model is a real-time neural network for object detection that detects 20 different classes. It is made up of 9 convolutional layers and 6 max-pooling layers and is a smaller version of the more complex full YOLOv2 network.

CoreML TinyYoloV2 ==> ONNX TinyYoloV2

Model

Model	Download	Download (with sample test data)	ONNX version	Opset version
Tiny YOLOv2	62 MB	59 MB	1.2	7
Tiny YOLOv2	62 MB	59 MB	1.3	8

Paper

"YOLO9000: Better, Faster, Stronger" arXiv:1612.08242

Dataset

The Tiny YOLO model was trained on the Pascal VOC dataset.

Source

The model was converted from a Core ML version of Tiny YOLO using ONNXMLTools. The source code can be found here. The Core ML model in turn was converted from the original network implemented in Darknet (via intermediate conversion through Keras).

Inference

Input

shape (1x3x416x416)

Preprocessing

Output

shape (1x125x13x13)

Postprocessing

The output is a (125x13x13) tensor where 13x13 is the number of grid cells that the image gets divided into. Each grid cell corresponds to 125 channels, made up of the 5 bounding boxes predicted by the grid cell and the 25 data elements that describe each bounding box (5x25=125). For more information on how to derive the final bounding boxes and their corresponding confidence scores, refer to this post.

Sample test data

Sets of sample input and output files are provided in

serialized protobuf TensorProtos (.pb), which are stored in the folders test_data_set_*/.

License

MIT

File: validated/vision/object_detection_segmentation/ssd-mobilenetv1/README.md

SSD-MobilenetV1

Description

SSD-MobilenetV1 is an object detection model that uses a Single Shot MultiBox Detector (SSD) approach to predict object classes for boundary boxes.

SSD is a CNN that enables the model to only need to take one single shot to detect multiple objects in an image, and MobileNet is a CNN base network that provides high-level features for object detection. The combination of these two model frameworks produces an efficient, high-accuracy detection model that requires less computational cost.

The SSD-MobilenetV1 is suitable for mobile and embedded vision applications.

Model

Model	Download	Download (with sample test data)	ONNX version	Opset version	mAP
SSD-MobilenetV1	29.3 MB	27.9 MB	1.7.0	10
SSD-MobilenetV1-12	28.1 MB	24.3 MB	1.9.0	12	0.2303
SSD-MobilenetV1-12-int8	9.2 MB	5.6 MB	1.9.0	12	0.2297
SSD-MobilenetV1-12-qdq	9.6 MB	5.7 MB	1.9.0	13	0.2297

Compared with the fp32 SSD-MobilenetV1-12, int8 SSD-MobilenetV1-12's mAP drop ratio is 0.26% and performance improvement is 1.23x.

Note

The performance depends on the test hardware. Performance data here is collected with Intel® Xeon® Platinum 8280 Processor, 1s 4c per instance, CentOS Linux 8.3, data batch size is 1.

Source

Tensorflow SSD-MobileNetV1 ==> ONNX SSD-MobileNetV1
ONNX SSD-MobileNetV1 ==> Quantized ONNX SSD-MobileNetV1

Inference

Running inference

Refer to this conversion and inference notebook for more details on how to inference this model using onnxruntime and define environment variables for the model.

text

import onnxruntime as rt

# Load model and run inference
# Start from ORT 1.10, ORT requires explicitly setting the providers parameter if you want to use execution providers
# other than the default CPU provider (as opposed to the previous behavior of providers getting set/registered by default
# based on the build flags) when instantiating InferenceSession.
# For example, if NVIDIA GPU is available and ORT Python package is built with CUDA, then call API as following:
# rt.InferenceSession(path/to/model, providers=['CUDAExecutionProvider'])
sess = rt.InferenceSession(os.path.join(WORK, MODEL + ".onnx"))
result = sess.run(outputs, {"image_tensor:0": img_data})
num_detections, detection_boxes, detection_scores, detection_classes = result

# print number of detections
print(num_detections)
print(detection_classes)

# produce outputs in this order
outputs = ["num_detections:0", "detection_boxes:0", "detection_scores:0", "detection_classes:0"]

Input

This model does not require fixed image dimensions. Input batch size is 1, with 3 color channels. Image has these variables: (batch_size, height, width, channels).

Preprocessing

The following code shows how preprocessing is done. For more information and an example on how preprocessing is done, please visit the tf2onnx conversion and inference notebook for this model.

text

import numpy as np
from PIL import Image, ImageDraw, ImageColor
import math
import matplotlib.pyplot as plt

# open and display image file
img = Image.open("image file")
plt.axis('off')
plt.imshow(img)
plt.show()

# reshape the flat array returned by img.getdata() to HWC and than add an additial
dimension to make NHWC, aka a batch of images with 1 image in it
img_data = np.array(img.getdata()).reshape(img.size[1], img.size[0], 3)
img_data = np.expand_dims(img_data.astype(np.uint8), axis=0)

Output

It outputs the image with boundary boxes and labels. The full list of classes can be found in the COCO dataset.

Given each batch of images, the model returns 4 tensor arrays:

num_detections: the number of detections.

detection_boxes: a list of bounding boxes. Each list item describes a box with top, left, bottom, right relative to the image size.

detection_scores: the score for each detection with values between 0 and 1 representing probability that a class was detected.

detection_classes: Array of 10 integers (floating point values) indicating the index of a class label from the COCO class.

Postprocessing

text

# draw boundary boxes and label for each detection
def draw_detection(draw, d, c):
    width, height = draw.im.size
    # the box is relative to the image size so we multiply with height and width to get pixels
    top = d[0] * height
    left = d[1] * width
    bottom = d[2] * height
    right = d[3] * width
    top = max(0, np.floor(top + 0.5).astype('int32'))
    left = max(0, np.floor(left + 0.5).astype('int32'))
    bottom = min(height, np.floor(bottom + 0.5).astype('int32'))
    right = min(width, np.floor(right + 0.5).astype('int32'))
    label = coco_classes[c]
    label_size = draw.textsize(label)
    if top - label_size[1] >= 0:
        text_origin = tuple(np.array([left, top - label_size[1]]))
    else:
        text_origin = tuple(np.array([left, top + 1]))
    color = ImageColor.getrgb("red")
    thickness = 0
    draw.rectangle([left + thickness, top + thickness, right - thickness, bottom - thickness],
    outline=color)
    draw.text(text_origin, label, fill=color), font=font)

# loop over the results - each returned tensor is a batch
batch_size = num_detections.shape[0]
draw = ImageDraw.Draw(img)
for batch in range(0, batch_size):
    for detection in range(0, int(num_detections[batch])):
        c = detection_classes[batch][detection]
        d = detection_boxes[batch][detection]
        draw_detection(draw, d, c)

# show image file with object detection boundary boxes and labels
plt.figure(figsize=(80, 40))
plt.axis('off')
plt.imshow(img)
plt.show()

Model Creation

Dataset (Train and validation)

The model was trained using MS COCO 2017 Train Images, Val Images, and Train/Val annotations.

Training

Training details for the SSD-MobileNet model's preprocessing is found in this tutorial notebook.
The notebook also details how the ONNX model was converted.

Quantization

SSD-MobilenetV1-12-int8 and SSD-MobilenetV1-12-qdq are obtained by quantizing fp32 SSD-MobilenetV1-12 model. We use Intel® Neural Compressor with onnxruntime backend to perform quantization. View the instructions to understand how to use Intel® Neural Compressor for quantization.

Environment

onnx: 1.9.0
onnxruntime: 1.8.0

Prepare model

shell

wget https://github.com/onnx/models/raw/main/vision/object_detection_segmentation/ssd-mobilenetv1/model/ssd_mobilenet_v1_12.onnx

Model quantize

Make sure to specify the appropriate dataset path in the configuration file.

bash

bash run_tuning.sh --input_model=path/to/model  \ # model path as *.onnx
                   --config=ssd_mobilenet_v1.yaml \ 
                   --output_model=path/to/save

References

Tensorflow to ONNX conversion tutorial. The notebook references how to run an evaluation on the SSD-MobilenetV1 model and export it as a saved model. It also details how to convert the tensorflow model into onnx, and how to run its preprocessing and postprocessing code for the inputs and outputs.
Intel® Neural Compressor

Contributors

Shirley Su
mengniwang95 (Intel)
yuwenzho (Intel)
airMeng (Intel)
ftian1 (Intel)
hshen14 (Intel)

License

MIT License

File: validated/vision/object_detection_segmentation/ssd/README.md

Single Stage Detector

Description

This model is a real-time neural network for object detection that detects 80 different classes.

Model

Model	Download	Download (with sample test data)	ONNX version	Opset version	Accuracy
SSD	80.4 MB	78.5 MB	1.5	10	mAP of 0.195
SSD	77.6 MB	86.4 MB	1.9	12	mAP of 0.1898
SSD-int8	20 MB	31 MB	1.9	12	mAP of 0.1892
SSD-qdq	20 MB	26 MB	1.9	12	mAP of 0.1863

Compared with the fp32 SSD, SSD-int8's mAP drop ratio is 0.32% and performance improvement is 3.49x.

Note

The performance depends on the test hardware. Performance data here is collected with Intel® Xeon® Platinum 8280 Processor, 1s 4c per instance, CentOS Linux 8.3, data batch size is 1.

Inference

Input to model

Image shape (1x3x1200x1200)

Preprocessing steps

The images have to be loaded in to a range of [0, 1], resized to (1200, 1200) with bilinear interpolation and then normalized using mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. The transformation should preferrably happen at preprocessing.

The following code shows how to preprocess a NCHW tensor:

python

import numpy as np
from PIL import Image

def preprocess(img_path):
    input_shape = (1, 3, 1200, 1200)
    img = Image.open(img_path)
    img = img.resize((1200, 1200), Image.BILINEAR)
    img_data = np.array(img)
    img_data = np.transpose(img_data, [2, 0, 1])
    img_data = np.expand_dims(img_data, 0)
    mean_vec = np.array([0.485, 0.456, 0.406])
    stddev_vec = np.array([0.229, 0.224, 0.225])
    norm_img_data = np.zeros(img_data.shape).astype('float32')
    for i in range(img_data.shape[1]):
        norm_img_data[:,i,:,:] = (img_data[:,i,:,:]/255 - mean_vec[i]) / stddev_vec[i]
    return norm_img_data

Output of model

The model has 3 outputs.
boxes: (1x'nbox'x4)
labels: (1x'nbox')
scores: (1x'nbox')

Dataset (Train and validation)

The SSD model was trained on 2017 COCO train data set - using mlperf/training/single_stage_detector repo , compute mAP on 2017 COCO val data set.

Validation accuracy

Metric is COCO box mAP (averaged over IoU of 0.5:0.95), computed over 2017 COCO val data.
mAP of 0.195

Publication/Attribution

Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C. Berg. SSD: Single Shot MultiBox Detector. In the Proceedings of the European Conference on Computer Vision (ECCV), 2016.

Backbone is ResNet34 pretrained on ILSVRC 2012 (from torchvision). Modifications to the backbone networks: remove conv_5x residual blocks, change the first 3x3 convolution of the conv_4x block from stride 2 to stride1 (this increases the resolution of the feature map to which detector heads are attached), attach all 6 detector heads to the output of the last conv_4x residual block. Thus detections are attached to 38x38, 19x19, 10x10, 5x5, 3x3, and 1x1 feature maps. Convolutions in the detector layers are followed by batch normalization layers.

Quantization

SSD-int8 and SSD-qdq are obtained by quantizing fp32 SSD model. We use Intel® Neural Compressor with onnxruntime backend to perform quantization. View the instructions to understand how to use Intel® Neural Compressor for quantization.

Environment

onnx: 1.9.0
onnxruntime: 1.8.0

Prepare model

shell

wget https://github.com/onnx/models/raw/main/vision/object_detection_segmentation/ssd/model/ssd-12.onnx

Model quantize

Make sure to specify the appropriate dataset path in the configuration file.

bash

bash run_tuning.sh --input_model=path/to/model \  # model path as *.onnx
                   --config=ssd.yaml \
                   --output_model=path/to/save

References

This model is converted from mlperf/inference repository with modifications in repository.
Intel® Neural Compressor

Contributors

mengniwang95 (Intel)
yuwenzho (Intel)
airMeng (Intel)
ftian1 (Intel)
hshen14 (Intel)

License

Apache License 2.0

File: README.md Copy Chapter

ONNX Model Zoo

Introduction Copy Chapter

Models Copy Chapter

Validated Models Copy Chapter

Vision

Language

Other

Image Classification Copy Section

Usage Copy Chapter

Usage - Test data starter code Copy Section

Usage - Model quantization Copy Section

Usage Copy Chapter

Git Clone (Not Recommended) Copy Section

GitHub UI Copy Section

Model Visualization Copy Chapter

Contributions Copy Chapter

License

File: validated/vision/super_resolution/sub_pixel_cnn_2016/README.md Copy Chapter

Super Resolution

Use cases Copy Chapter

Description Copy Chapter

Model Copy Chapter

Inference Copy Chapter

Input Copy Section

Preprocessing Copy Section

Output Copy Section

Postprocessing Copy Section

Dataset Copy Chapter

Training Copy Chapter

File: validated/vision/style_transfer/fast_neural_style/README.md Copy Chapter

Fast Neural Style Transfer

Use-cases Copy Chapter

Description Copy Chapter

Model Copy Chapter

Inference Copy Chapter

Input to model Copy Section

Preprocessing steps Copy Section

Output of model Copy Section

Postprocessing steps Copy Section

Dataset (Train and validation) Copy Chapter

Training Copy Chapter

References Copy Chapter

Contributors Copy Chapter

License Copy Chapter

File: validated/vision/object_detection_segmentation/yolov4/README.md Copy Chapter

YOLOv4

Description Copy Chapter

Model Copy Chapter

Source Copy Section

Inference Copy Chapter

Conversion Copy Section

Running inference Copy Section

Input to model Copy Section

Preprocessing steps Copy Section

Output of model Copy Section

Postprocessing steps Copy Section

Dataset Copy Chapter

Validation accuracy Copy Chapter

Publication/Attribution Copy Chapter

References Copy Chapter

Contributors Copy Chapter

License Copy Chapter

File: validated/vision/object_detection_segmentation/yolov3/README.md Copy Chapter

YOLOv3

Description Copy Chapter

Model Copy Chapter

Inference Copy Chapter

Input to model Copy Section

Preprocessing steps Copy Section

Output of model Copy Section

Postprocessing steps Copy Section

Dataset (Train and validation) Copy Chapter

Validation accuracy Copy Chapter

Quantization Copy Chapter

Environment Copy Section

Prepare model Copy Section

Model quantize Copy Section

Publication/Attribution Copy Chapter

References Copy Chapter

File: README.md

Introduction

Models

Validated Models

Image Classification

Usage

Usage - Test data starter code

Usage - Model quantization

Usage

Git Clone (Not Recommended)

GitHub UI

Model Visualization

Contributions

File: validated/vision/super_resolution/sub_pixel_cnn_2016/README.md

Use cases

Description

Model

Inference

Input

Preprocessing

Output

Postprocessing

Dataset

Training

File: validated/vision/style_transfer/fast_neural_style/README.md

Use-cases

Description

Model

Inference

Input to model

Preprocessing steps

Output of model

Postprocessing steps

Dataset (Train and validation)

Training

References

Contributors

License

File: validated/vision/object_detection_segmentation/yolov4/README.md

Description

Model

Source

Inference

Conversion

Running inference

Input to model

Preprocessing steps

Output of model

Postprocessing steps

Dataset

Validation accuracy

Publication/Attribution

References

Contributors

License

File: validated/vision/object_detection_segmentation/yolov3/README.md

Description

Model

Inference

Input to model

Preprocessing steps

Output of model

Postprocessing steps

Dataset (Train and validation)

Validation accuracy

Quantization

Environment

Prepare model

Model quantize

Publication/Attribution

References

Contributors

License

File: validated/vision/object_detection_segmentation/yolov2-coco/README.md

Description

Model

Inference

Input to model

Output of model

Postprocessing steps