server Documentation

File: README.md Copy Chapter

[!WARNING]
You are currently on the main branch which tracks under-development progress
towards the next release. The current release is version 2.71.0
and corresponds to the 26.07 container release on NVIDIA GPU Cloud (NGC).

Triton Inference Server

Triton Inference Server is an open source inference serving software that
streamlines AI inferencing. Triton enables teams to deploy any AI model from
multiple deep learning and machine learning frameworks, including TensorRT,
PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton
Inference Server supports inference across cloud, data center, edge and embedded
devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference
Server delivers optimized performance for many query types, including real time,
batched, ensembles and audio/video streaming. Triton inference Server is part of
NVIDIA AI Enterprise,
a software platform that accelerates the data science pipeline and streamlines
the development and deployment of production AI.

Major features include:

• Supports multiple deep learning
  frameworks
• Supports multiple machine learning
  frameworks
• Concurrent model
  execution
• Dynamic batching
• Sequence batching and
  implicit state management
  for stateful models
• Provides Backend API that
  allows adding custom backends and pre/post processing operations
• Supports writing custom backends in python, a.k.a.
  Python-based backends.
• Model pipelines using
  Ensembling or Business
  Logic Scripting
  (BLS)
• HTTP/REST and GRPC inference
  protocols based on the community
  developed KServe
  protocol
• A C API and
  Java API
  allow Triton to link directly into your application for edge and other in-process use cases
• Metrics indicating GPU utilization, server
  throughput, server latency, and more

New to Triton Inference Server? Make use of
these tutorials
to begin your Triton journey!

Join the Triton and TensorRT community and
stay current on the latest product updates, bug fixes, content, best practices,
and more. Need enterprise support? NVIDIA global support is available for Triton
Inference Server with the
NVIDIA AI Enterprise software suite.

Serve a Model in 3 Easy Steps Copy Chapter

# Step 1: Create the example model repository
git clone -b r26.07 https://github.com/triton-inference-server/server.git
cd server/docs/examples
./fetch_models.sh

# Step 2: Launch triton from the NGC Triton container
docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:26.07-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx

# Step 3: Sending an Inference Request
# In a separate console, launch the image_client example from the NGC Triton SDK container
docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.07-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg

# Inference should return the following
Image '/workspace/images/mug.jpg':
    15.346230 (504) = COFFEE MUG
    13.224326 (968) = CUP
    10.422965 (505) = COFFEEPOT

Please read the QuickStart guide for additional information
regarding this example. The quickstart guide also contains an example of how to launch Triton on CPU-only systems. New to Triton and wondering where to get started? Watch the Getting Started video.

Examples and Tutorials Copy Chapter

Check out NVIDIA LaunchPad
for free access to a set of hands-on labs with Triton Inference Server hosted on
NVIDIA infrastructure.

Specific end-to-end examples for popular models, such as ResNet, BERT, and DLRM
are located in the
NVIDIA Deep Learning Examples
page on GitHub. The
NVIDIA Developer Zone
contains additional documentation, presentations, and examples.

Documentation Copy Chapter

Build and Deploy Copy Section

The recommended way to build and use Triton Inference Server is with Docker
images.

• Install Triton Inference Server with Docker containers (Recommended)
• Install Triton Inference Server without Docker containers
• Build a custom Triton Inference Server Docker container
• Build Triton Inference Server from source
• Examples for deploying Triton Inference Server with Kubernetes and Helm on GCP,
  AWS, and NVIDIA FleetCommand
• Secure Deployment Considerations

Using Triton Copy Section

Preparing Models for Triton Inference Server

The first step in using Triton to serve your models is to place one or
more models into a model repository. Depending on
the type of the model and on what Triton capabilities you want to enable for
the model, you may need to create a model
configuration for the model.

• Add custom operations to Triton if needed by your model
• Enable model pipelining with Model Ensemble
  and Business Logic Scripting (BLS)
• Optimize your models setting scheduling and batching
  parameters and model instances.
• Use the Model Analyzer tool
  to help optimize your model configuration with profiling
• Learn how to explicitly manage what models are available by loading and
  unloading models

Configure and Use Triton Inference Server

• Read the Quick Start Guide to run Triton Inference
  Server on both GPU and CPU
• Triton supports multiple execution engines, called
  backends, including
  TensorRT,
  PyTorch,
  ONNX,
  OpenVINO,
  Python, and more
• Not all the above backends are supported on every platform supported by Triton.
  Look at the
  Backend-Platform Support Matrix
  to learn which backends are supported on your target platform.
• Learn how to optimize performance using the
  Performance Analyzer
  and
  Model Analyzer
• Learn how to manage loading and unloading models in
  Triton
• Send requests directly to Triton with the HTTP/REST JSON-based
  or gRPC protocols

Client Support and Examples

A Triton client application sends inference and other requests to Triton. The
Python and C++ client libraries
provide APIs to simplify this communication.

• Review client examples for C++,
  Python,
  and Java
• Configure HTTP
  and gRPC
  client options
• Send input data (e.g. a jpeg image) directly to Triton in the body of an HTTP
  request without any additional metadata

Extend Triton Copy Section

Triton Inference Server's architecture is specifically
designed for modularity and flexibility

• Customize Triton Inference Server container for your use case
• Create custom backends
  in either C/C++
  or Python
• Create decoupled backends and models that can send
  multiple responses for a request or not send any responses for a request
• Use a Triton repository agent to add functionality
  that operates when a model is loaded and unloaded, such as authentication,
  decryption, or conversion
• Deploy Triton on Jetson and JetPack
• Use Triton on AWS
  Inferentia

Additional Documentation Copy Section

• FAQ
• User Guide
• Customization Guide
• Release Notes
• GPU, Driver, and CUDA Support
  Matrix

Contributing Copy Chapter

Contributions to Triton Inference Server are more than welcome. To
contribute please review the contribution
guidelines. If you have a backend, client,
example or similar contribution that is not modifying the core of
Triton, then you should file a PR in the contrib
repo.

Reporting problems, asking questions Copy Chapter

We appreciate any feedback, questions or bug reporting regarding this project.
When posting issues in GitHub,
follow the process outlined in the Stack Overflow document.
Ensure posted examples are:

• minimal – use as little code as possible that still produces the
  same problem
• complete – provide all parts needed to reproduce the problem. Check
  if you can strip external dependencies and still show the problem. The
  less time we spend on reproducing problems the more time we have to
  fix it
• verifiable – test the code you're about to provide to make sure it
  reproduces the problem. Remove all other problems that are not
  related to your request/question.

For issues, please use the provided bug report and feature request templates.

For questions, we recommend posting in our community
GitHub Discussions.

For more information Copy Chapter

Please refer to the NVIDIA Developer Triton page
for more information.

File: deploy/oci/README.md Copy Chapter

Kubernetes Deploy: Triton Inference Server Cluster

A helm chart for installing a single cluster of Triton Inference
Server is provided. By default the cluster contains a single instance
of the inference server but the replicaCount configuration parameter
can be set to create a cluster of any size, as described below.

This guide assumes you already have a functional Kubernetes cluster
and helm installed (see below for instructions on installing
helm). Note the following requirements:

• The helm chart deploys Prometheus and Grafana to collect and display Triton metrics. To use this helm chart you must install Prometheus and Grafana in your cluster as described below and your cluster must contain sufficient CPU resources to support these services.

• If you want Triton Server to use GPUs for inferencing, your cluster
  must be configured to contain the desired number of GPU nodes (A10 GPU instances recommended)
  with support for the NVIDIA driver and CUDA version required by the version
  of the inference server you are using.

The steps below describe how to set-up a model repository, use helm to
launch the inference server, and then send inference requests to the
running server. You can access a Grafana endpoint to see real-time
metrics reported by the inference server.

Notes for OKE cluster Copy Chapter

When creating your node pool, the default value for the boot volume is 46.6GB.
Due to the size of the server container, it is recommended to increase this value
to 150GB and set a cloud-init script to increase the partition:

#!/bin/bash
curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-init.sh
bash /var/run/oke-init.sh
sudo /usr/libexec/oci-growfs -y

Installing Helm Copy Chapter

Using Cloud Shell from OCI Web Console Copy Section

It is possible to access your OKE Cluster directly from the OCI Web Console.
Helm v3 is already available from the Cloud Shell.

Helm v3 Copy Section

If you do not already have Helm installed in your Kubernetes cluster,
executing the following steps from the official helm install
guide will
give you a quick setup.

If you're currently using Helm v2 and would like to migrate to Helm v3,
please see the official migration guide.

Helm v2 Copy Section

NOTE: Moving forward this chart will only be tested and maintained for Helm v3.

Below are example instructions for installing Helm v2.

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash
$ kubectl create serviceaccount -n kube-system tiller
serviceaccount/tiller created
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
$ helm init --service-account tiller --wait

If you run into any issues, you can refer to the official installation guide here.

Model Repository Copy Chapter

If you already have a model repository you may use that with this helm
chart. If you do not have a model repository, you can checkout a local
copy of the inference server source repository to create an example
model repository:

$ git clone https://github.com/triton-inference-server/server.git

Triton Server needs a repository of models that it will make available
for inferencing. For this example you will place the model repository
in an S3 compatible OCI Object Storage Bucket.

$ oci os bucket create --compartment-id <COMPARTMENT_OCID> --name triton-inference-server-repository

Following the QuickStart download the
example model repository to your system and copy it into the OCI
Bucket.

$ oci os object bulk-upload -bn triton-inference-server-repository --src-dir docs/examples/model_repository/

OCI Model Repository Copy Section

To load the model from the OCI Object Storage Bucket, you need to convert the following OCI credentials in the base64 format and add it to the values.yaml

echo -n 'REGION' | base64

echo -n 'SECRECT_KEY_ID' | base64

echo -n 'SECRET_ACCESS_KEY' | base64

You also need to adapt modelRepositoryPath in values.yaml to your namespace and OCI region.

s3://https://<OCI_NAMESPACE>.compat.objectstorage.<OCI_REGION>.oraclecloud.com:443/triton-inference-server-repository

Deploy Prometheus and Grafana Copy Chapter

The inference server metrics are collected by Prometheus and viewable
by Grafana. The inference server helm chart assumes that Prometheus
and Grafana are available so this step must be followed even if you
don't want to use Grafana.

Use the kube-prometheus-stack to install these components. The
serviceMonitorSelectorNilUsesHelmValues flag is needed so that
Prometheus can find the inference server metrics in the example
release deployed below.

$ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack

Then port-forward to the Grafana service so you can access it from
your local browser.

$ kubectl port-forward service/example-metrics-grafana 8080:80

Now you should be able to navigate in your browser to localhost:8080
and see the Grafana login page. Use username=admin and
password=prom-operator to login.

Note that it is also possible to set a load balancer service for the grafana dashboard
by running:

$ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false --set grafana.service.type=LoadBalancer prometheus-community/kube-prometheus-stack

You can then see the Public IP of you grafana dashboard by running:

$ kubectl get svc
NAME                                       TYPE           CLUSTER-IP     EXTERNAL-IP       PORT(S)                      AGE
alertmanager-operated                      ClusterIP      None           <none>            9093/TCP,9094/TCP,9094/UDP   2m33s
example-metrics-grafana                    LoadBalancer   10.96.82.33    141.145.220.114   80:31005/TCP                 2m38s

The default load balancer created comes with a fixed shape and a bandwidth of 100Mbps. You can switch to a flexible shape and adapt the bandwidth according to your OCI limits in case the bandwidth is a bottleneck.

An example Grafana dashboard is available in dashboard.json. Use the
import function in Grafana to import and view this dashboard.

Deploy the Inference Server Copy Chapter

Deploy the inference server using the default configuration with the
following commands.

$ cd <directory containing Chart.yaml>
$ helm install example .

Use kubectl to see status and wait until the inference server pods are
running.

$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
example-triton-inference-server-5f74b55885-n6lt7   1/1     Running   0          2m21s

There are several ways of overriding the default configuration as
described in this helm
documentation.

You can edit the values.yaml file directly or you can use the --set
option to override a single parameter with the CLI. For example, to
deploy a cluster of four inference servers use --set to set the
replicaCount parameter.

$ helm install example --set replicaCount=4 .

You can also write your own "config.yaml" file with the values you
want to override and pass it to helm.

$ cat << EOF > config.yaml
namespace: MyCustomNamespace
image:
  imageName: nvcr.io/nvidia/tritonserver:custom-tag
  modelRepositoryPath: s3://https://<OCI_NAMESPACE>.compat.objectstorage.<OCI_REGION>.oraclecloud.com:443/triton-inference-server-repository
EOF
$ helm install example -f config.yaml .

Using Triton Inference Server Copy Chapter

Now that the inference server is running you can send HTTP or GRPC
requests to it to perform inferencing. By default, the inferencing
service is exposed with a LoadBalancer service type. Use the following
to find the external IP for the inference server. In this case it is
34.83.9.133.

$ kubectl get services
NAME                             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                                        AGE
...
example-triton-inference-server  LoadBalancer   10.18.13.28    34.83.9.133   8000:30249/TCP,8001:30068/TCP,8002:32723/TCP   47m

The inference server exposes an HTTP endpoint on port 8000, and GRPC
endpoint on port 8001 and a Prometheus metrics endpoint on
port 8002. You can use curl to get the meta-data of the inference server
from the HTTP endpoint.

$ curl 34.83.9.133:8000/v2

Follow the QuickStart to get the example
image classification client that can be used to perform inferencing
using image classification models being served by the inference
server. For example,

$ image_client -u 34.83.9.133:8000 -m inception_v3_onnx -s INCEPTION -c3 mug.jpg
Request 0, batch size 1
Image 'images/mug.jpg':
    504 (COFFEE MUG) = 0.723992
    968 (CUP) = 0.270953
    967 (ESPRESSO) = 0.00115997

Cleanup Copy Chapter

Once you've finished using the inference server you should use helm to
delete the deployment.

$ helm list
NAME            REVISION  UPDATED                   STATUS    CHART                          APP VERSION   NAMESPACE
example         1         Wed Feb 27 22:16:55 2019  DEPLOYED  triton-inference-server-1.0.0  1.0           default
example-metrics	1       	Tue Jan 21 12:24:07 2020	DEPLOYED	prometheus-operator-6.18.0   	 0.32.0     	 default

$ helm uninstall example
$ helm uninstall example-metrics

For the Prometheus and Grafana services, you should explicitly delete
CRDs:

$ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com alertmanagers.monitoring.coreos.com podmonitors.monitoring.coreos.com probes.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com thanosrulers.monitoring.coreos.com

You may also want to delete the OCI bucket you created to hold the
model repository.

$ oci os bucket delete --bucket-name triton-inference-server-repository --empty

File: deploy/mlflow-triton-plugin/README.md Copy Chapter

MLflow Triton

MLflow plugin for deploying your models from MLflow to Triton Inference Server.
Scripts are included for publishing models, which are in Triton recognized
structure, to your MLflow Model Registry.

Supported flavors Copy Section

MLFlow Triton plugin currently supports the following flavors, you may
substitute the flavor specification in the example below according to the model
to be deployed.

• onnx
• triton

Requirements Copy Chapter

• MLflow
• Triton Python HTTP client
• Triton Inference Server

Installation Copy Chapter

The plugin can be installed from source using the following commands

python setup.py install

Quick Start Copy Chapter

In this documentation, we will use the files in examples to showcase how
the plugin interacts with Triton Inference Server. The onnx_float32_int32_int32
model in examples is a simple model that takes two float32 inputs, INPUT0 and
INPUT1, with shape [-1, 16], and produces two int32 outputs, OUTPUT0 and
OUTPUT1, where OUTPUT0 is the element-wise summation of INPUT0 and INPUT1 and
OUTPUT1 is the element-wise subtraction of INPUT0 and INPUT1.

Start Triton Inference Server in EXPLICIT mode Copy Section

The MLflow Triton plugin must work with a running Triton server, see
documentation
of Triton Inference Server for how to start the server. Note that
the server should be run in EXPLICIT mode (--model-control-mode=explicit)
to exploit the deployment feature of the plugin.

Once the server has started, the following environment must be set so that the plugin
can interact with the server properly:

• TRITON_URL: The address to the Triton HTTP endpoint
• TRITON_MODEL_REPO: The path to the Triton model repository. It can be an s3 URI but keep in
  mind that the env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are needed.

Publish models to MLflow Copy Section

ONNX flavor

The MLFlow ONNX built-in functionalities can be used to publish onnx flavor
models to MLFlow directly, and the MLFlow Triton plugin will prepare the model
to the format expected by Triton. You may also log
config.pbtxt
as additional artifact which Triton will be used to serve the model. Otherwise,
the server should be run with auto-complete feature enabled
(--strict-model-config=false) to generate the model configuration.

import mlflow.onnx
import onnx
model = onnx.load("examples/onnx_float32_int32_int32/1/model.onnx")
mlflow.onnx.log_model(model, "triton", registered_model_name="onnx_float32_int32_int32")

Triton flavor

For other model frameworks that Triton supports but not yet recognized by
the MLFlow Triton plugin, the publish_model_to_mlflow.py script can be used to
publish triton flavor models to MLflow. A triton flavor model is a directory
containing the model files following the
model layout.
Below is an example usage:

cd /scripts

python publish_model_to_mlflow.py --model_name onnx_float32_int32_int32 --model_directory /onnx_float32_int32_int32 --flavor triton

Deploy models tracked in MLflow to Triton Copy Section

Once a model is published and tracked in MLflow, it can be deployed to Triton
via MLflow's deployments command, the following command will download the model
to Triton's model repository and request Triton to load the model.

mlflow deployments create -t triton --flavor triton --name onnx_float32_int32_int32 -m models:/onnx_float32_int32_int32/1

Perform inference Copy Section

After the model is deployed, the following command is the CLI usage to send
inference request to a deployment.

mlflow deployments predict -t triton --name onnx_float32_int32_int32 --input-path /input.json --output-path output.json

The inference result will be written in output.json and you may compare it
with the results in expected_output.json

MLflow Deployments Copy Chapter

"MLflow Deployments" is a set of MLflow APIs for deploying MLflow models to
custom serving tools. The MLflow Triton plugin implements the following
deployment functions to support the interaction with Triton server in MLflow.

Create Deployment Copy Section

MLflow deployments create API deploys a model to the Triton target, which will
download the model to Triton's model repository and request Triton to load the
model.

To create a MLflow deployment using CLI

mlflow deployments create -t triton --flavor triton --name model_name -m models:/model_name/1

To create a MLflow deployment using Python API

from mlflow.deployments import get_deploy_client
client = get_deploy_client('triton')
client.create_deployment("model_name", "models:/model_name/1", flavor="triton")

Delete Deployment Copy Section

MLflow deployments delete API removes an existing deployment from the Triton
target, which will remove the model in Triton's model repository and request
Triton to unload the model.

To delete a MLflow deployment using CLI

mlflow deployments delete -t triton --name model_name

To delete a MLflow deployment using Python API

from mlflow.deployments import get_deploy_client
client = get_deploy_client('triton')
client.delete_deployment("model_name")

Update Deployment Copy Section

MLflow deployments update API updates an existing deployment with another model
(version) tracked in MLflow, which will overwrite the model in Triton's model
repository and request Triton to reload the model.

To update a MLflow deployment using CLI

mlflow deployments update -t triton --flavor triton --name model_name -m models:/model_name/2

To update a MLflow deployment using Python API

from mlflow.deployments import get_deploy_client
client = get_deploy_client('triton')
client.update_deployment("model_name", "models:/model_name/2", flavor="triton")

List Deployments Copy Section

MLflow deployments list API lists all existing deployments in Triton target.

To list all MLflow deployments using CLI

mlflow deployments list -t triton

To list all MLflow deployments using Python API

from mlflow.deployments import get_deploy_client
client = get_deploy_client('triton')
client.list_deployments()

Get Deployment Copy Section

MLflow deployments get API returns information regarding a specific deployments
in Triton target.

To list a specific MLflow deployment using CLI

mlflow deployments get -t triton --name model_name

To list a specific MLflow deployment using Python API

from mlflow.deployments import get_deploy_client
client = get_deploy_client('triton')
client.get_deployment("model_name")

Run Inference on Deployments Copy Section

MLflow deployments predict API runs inference by preparing and sending the
request to Triton and returns the Triton response.

To run inference using CLI

mlflow deployments predict -t triton --name model_name --input-path input_file --output-path output_file

To run inference using Python API

from mlflow.deployments import get_deploy_client
client = get_deploy_client('triton')
client.predict("model_name", inputs)

File: deploy/k8s-onprem/README.md Copy Chapter

Kubernetes Deploy: NVIDIA Triton Inference Server Cluster

This repository includes a Helm chart and instructions for installing NVIDIA Triton
Inference Server in an on-premises or AWS EC2 Kubernetes cluster. You can also use this
repository to enable load balancing and autoscaling for your Triton cluster.

This guide assumes you already have a functional Kubernetes cluster with support for GPUs.
See the NVIDIA GPU Operator documentation
for instructions on how to install Kubernetes and enable GPU access in your Kubernetes cluster.
You must also have Helm installed (see Installing Helm for instructions). Note the following requirements:

• To deploy Prometheus and Grafana to collect and display Triton metrics, your cluster must contain sufficient CPU resources to support these services.

• To use GPUs for inferencing, your cluster must be configured to contain the desired number of GPU nodes, with
  support for the NVIDIA driver and CUDA version required by the version
  of the inference server you are using.

• To enable autoscaling, your cluster's kube-apiserver must have the aggregation layer
  enabled.
  This will allow the horizontal pod autoscaler to read custom metrics from the prometheus adapter.

This Helm chart is available from Triton Inference Server
GitHub.

For more information on Helm and Helm charts, visit the Helm documentation.

Quickstart Copy Chapter

First, clone this repository to a local machine. Then, execute the following commands:

Install helm

$ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
$ chmod 700 get_helm.sh
$ ./get_helm.sh

Deploy Prometheus and Grafana

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack

Deploy Triton with default settings

helm install example ./deploy/k8s-onprem

Installing Helm Copy Chapter

Helm v3 Copy Section

If you do not already have Helm installed in your Kubernetes cluster,
executing the following steps from the official Helm install
guide will
give you a quick setup.

If you are currently using Helm v2 and would like to migrate to Helm v3,
see the official migration guide.

Model Repository Copy Chapter

If you already have a model repository, you may use that with this Helm
chart. If you do not have a model repository, you can check out a local
copy of the server source repository to create an example
model repository:

$ git clone https://github.com/triton-inference-server/server.git

Triton Server needs a repository of models that it will make available
for inferencing. For this example, we are using an existing NFS server and
placing our model files there. See the
Model Repository documentation for other
supported locations.

Following the QuickStart, download the
example model repository to your system and copy it onto your NFS server.
Then, add the url or IP address of your NFS server and the server path of your
model repository to values.yaml.

Deploy Prometheus and Grafana Copy Chapter

The inference server metrics are collected by Prometheus and viewable
through Grafana. The inference server Helm chart assumes that Prometheus
and Grafana are available so this step must be followed even if you
do not want to use Grafana.

Use the kube-prometheus-stack Helm chart to install these components. The
serviceMonitorSelectorNilUsesHelmValues flag is needed so that
Prometheus can find the inference server metrics in the example
release deployed in a later section.

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack

Then port-forward to the Grafana service so you can access it from
your local browser.

$ kubectl port-forward service/example-metrics-grafana 8080:80

Now you should be able to navigate in your browser to localhost:8080
and see the Grafana login page. Use username=admin and
password=prom-operator to log in.

An example Grafana dashboard is available in dashboard.json. Use the
import function in Grafana to import and view this dashboard.

Enable Autoscaling Copy Chapter

To enable autoscaling, ensure that autoscaling tag in values.yamlis set to true.
This will do two things:

1. Deploy a Horizontal Pod Autoscaler that will scale replicas of the triton-inference-server
   based on the information included in values.yaml.

2. Install the prometheus-adapter helm chart, allowing the Horizontal Pod Autoscaler to scale
   based on custom metrics from prometheus.

The included configuration will scale Triton pods based on the average queue time,
as described in this blog post. To customize this,
you may replace or add to the list of custom rules in values.yaml. If you change
the custom metric, be sure to change the values in autoscaling.metrics.

If autoscaling is disabled, the number of Triton server pods is set to the minReplicas
variable in values.yaml.

Enable Load Balancing Copy Chapter

To enable load balancing, ensure that the loadBalancing tag in values.yaml
is set to true. This will do two things:

1. Deploy a Traefik reverse proxy through the Traefik Helm Chart.

2. Configure two Traefik IngressRoutes,
   one for http and one for grpc. This will allow the Traefik service to expose two
   ports that will be forwarded to and balanced across the Triton pods.

To choose the port numbers exposed, or to disable either http or grpc, edit the
configured variables in values.yaml.

Deploy the Inference Server Copy Chapter

Deploy the inference server, autoscaler, and load balancer using the default
configuration with the following commands.

Here, and in the following commands we use the name example for our chart.
This name will be added to the beginning of all resources created during the helm
installation.

$ cd <directory containing Chart.yaml>
$ helm install example .

Use kubectl to see status and wait until the inference server pods are
running.

$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
example-triton-inference-server-5f74b55885-n6lt7   1/1     Running   0          2m21s

There are several ways of overriding the default configuration as
described in this Helm
documentation.

You can edit the values.yaml file directly or you can use the --set
option to override a single parameter with the CLI. For example, to
deploy a cluster with a minimum of two inference servers use --set to
set the autoscaler.minReplicas parameter.

$ helm install example --set autoscaler.minReplicas=2 .

You can also write your own "config.yaml" file with the values you
want to override and pass it to Helm. If you specify a "config.yaml" file, the
values set will override those in values.yaml.

$ cat << EOF > config.yaml
namespace: MyCustomNamespace
image:
  imageName: nvcr.io/nvidia/tritonserver:custom-tag
  modelRepositoryPath: gs://my_model_repository
EOF
$ helm install example -f config.yaml .

Deploying the Inference Server on OpenShift or OKD Copy Chapter

Because of the default security posture of OpenShift and OKD, the configuration
of which uses OpenShift-specific APIs, the chart needs special consideration
when targeting those environments. Any of the above discussed customizations and
prerequisites hold for an OpenShift environment, except that you do not need to
install Prometheus and Grafana and can instead enable monitoring for
user-defined projects by following
the OpenShift documentation on the topic.

To deploy the configurations to enable NFS mounts and the non-root UIDs used in
the Triton deployment, a tag can be enabled alongside any other configurations
discussed above. In the simplest case, to use --set on the command line, you
can simply update the tags.openshift parameter.

$ cd <directory containing Chart.yaml>
$ helm install example --set tags.openshift=true .

Probe Configuration Copy Chapter

In templates/deployment.yaml is configurations for livenessProbe, readinessProbe and startupProbe for the Triton server container.
By default, Triton loads all the models before starting the HTTP server to respond to the probes. The process can take several minutes, depending on the models sizes.
If it is not completed in startupProbe.failureThreshold * startupProbe.periodSeconds seconds then Kubernetes considers this as a pod failure and restarts it,
ending up with an infinite loop of restarting pods, so make sure to sufficiently set these values for your use case.
The liveliness and readiness probes are being sent only after the first success of a startup probe.

For more details, see the Kubernetes probe documentation and the feature page of the startup probe.

Using Triton Inference Server Copy Chapter

Now that the inference server is running you can send HTTP or GRPC
requests to it to perform inferencing. By default, this chart deploys Traefik
and uses IngressRoutes
to balance requests across all available nodes.

To send requests through the Traefik proxy, use the Cluster IP of the
traefik service deployed by the Helm chart. In this case, it is 10.111.128.124.

$ kubectl get services
NAME                              TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                                                    AGE
...
example-traefik                   LoadBalancer   10.111.128.124        8001:31752/TCP,8000:31941/TCP,80:30692/TCP,443:30303/TCP   74m
example-triton-inference-server   ClusterIP      None             <none>        8000/TCP,8001/TCP,8002/TCP                                 74m

Use the following command to refer to the Cluster IP:

cluster_ip=`kubectl get svc -l app.kubernetes.io/name=traefik -o=jsonpath='{.items[0].spec.clusterIP}'`

The Traefik reverse-proxy exposes an HTTP endpoint on port 8000, and GRPC
endpoint on port 8001 and a Prometheus metrics endpoint on
port 8002. You can use curl to get the meta-data of the inference server
from the HTTP endpoint.

$ curl $cluster_ip:8000/v2

Follow the QuickStart to get the example
image classification client that can be used to perform inferencing
using image classification models on the inference
server. For example,

$ image_client -u $cluster_ip:8000 -m inception_v3_onnx -s INCEPTION -c3 mug.jpg
Request 0, batch size 1
Image 'images/mug.jpg':
    504 (COFFEE MUG) = 0.723992
    968 (CUP) = 0.270953
    967 (ESPRESSO) = 0.00115997

Testing Load Balancing and Autoscaling Copy Chapter

After you have confirmed that your Triton cluster is operational and can perform inference,
you can test the load balancing and autoscaling features by sending a heavy load of requests.
One option for doing this is using the
perf_analyzer
application.

You can apply a progressively increasing load with a command like:

perf_analyzer -m simple -u $cluster_ip:8000 --concurrency-range 1:10

From your Grafana dashboard, you should be able to see the number of pods increase
as the load increases, with requests being routed evenly to the new pods.

Cleanup Copy Chapter

After you have finished using the inference server, you should use Helm to
delete the deployment.

$ helm list
NAME            REVISION  UPDATED                   STATUS    CHART                          APP VERSION   NAMESPACE
example         1         Wed Feb 27 22:16:55 2019  DEPLOYED  triton-inference-server-1.0.0  1.0           default
example-metrics	1       	Tue Jan 21 12:24:07 2020	DEPLOYED	prometheus-operator-6.18.0   	 0.32.0     	 default

$ helm uninstall example
$ helm uninstall example-metrics

For the Prometheus and Grafana services, you should explicitly delete
CRDs:

$ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com alertmanagers.monitoring.coreos.com podmonitors.monitoring.coreos.com probes.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com thanosrulers.monitoring.coreos.com

File: deploy/gke-marketplace-app/benchmark/README.md Copy Chapter

Benchmarking with NVIDIA Triton Inference Server GKE Marketplace Application

Table Of Contents

• Models
• Performance

Models Copy Chapter

First, we collect a set of TensorFlow and TensorRT models to compare:

• Get Distill Bert fine-tuned with Squad Q&A task from Huggingface. wget https://huggingface.co/distilbert-base-cased-distilled-squad/blob/main/saved_model.tar.gz
• Get Bert base fine-tuned with Squad Q&A task from Huggingface wget https://huggingface.co/deepset/bert-base-cased-squad2/blob/main/saved_model.tar.gz
• Follow TensorRT Demo Bert to convert BERT base model to TensorRT Engine, choose sequence length of 384 to match previous 2 TensorFlow models. Last step, we choose to create TensorRT engine with 2 optimization profile, profile 0 for batch size 1 and profile 1 for batch size 4 run: python3 builder.py -m models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_384_v19.03.1/model.ckpt -o engines/model.plan -b 8 -s 384 --fp16 --int8 --strict -c models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_384_v19.03.1 --squad-json ./squad/train-v2.0.json -v models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_384_v19.03.1/vocab.txt --calib-num 100 -iln -imh. This needs to be ran on the inference GPU respectively (Engine optimized with A100 cannot be used for inference on T4).

We the place the model into a GCS with following structure, config.pbtxt was provided.

├── bert_base_trt_gpu
    │   ├── 1
    │   │   └── model.plan
    │   └── config.pbtxt
    ├── bert_base_trt_gpu_seqlen128
    │   ├── 1
    │   │   └── model.plan
    │   └── config.pbtxt
    ├── bert_base_tf_gpu
    │   ├── 1
    │   │   └── model.savedmodel
    │   └── config.pbtxt
    ├── bert_base_tf_cpu
    │   ├── 1
    │   │   └── model.savedmodel
    │   └── config.pbtxt
    ├── bert_distill_tf_gpu
    │   ├── 1
    │   │   └── model.savedmodel
    │   └── config.pbtxt
    └── bert_distill_tf_cpu
        ├── 1
        │   └── model.savedmodel
        └── config.pbtxt

When deploy Triton GKE application, point the model repository to directory contains the structure above with actual models.

Performance Copy Chapter

We use perf analyzer of Triton to benchmark the performance of each model, the perf analyzer reside in another pod of the GKE cluster.

export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}')
bash perf_query.sh 35.194.5.119:80 bert_base_trt_gpu 384

We deploy model on n1-standard-96 for CPU BERT BASE and Distill BERT and (n1-standard-4 + T4) for GPU BERT models, the sequence length of the BERT model is 384 token, and measure the latency/throughput with a concurrency sweep with Triton's performance analyzer. The latency includes Istio ingress/load balancing and reflect the true round trip cost in the same GCP zone.

For all the model with sequence length of 384:
CPU BERT BASE: latency: 700ms, throughput: 12 qps
CPU Distill BERT: latency: 369ms, throughput: 24 qps

GPU BERT BASE: latency: 230ms, throughput: 34.7 qps
GPU Distill BERT: latency: 118ms, throughput: 73.3 qps
GPU TensorRT BERT BASE: latency: 50ms, throughput: 465 qps

With n1-standard-96 priced at $4.56/hr and n1-standard-4 at $0.19/hr and T4 at $0.35/hr totaling $0.54/hr. While achieving a much lower latency, the TCO of BERT inference with TensorRT on T4 is over 163 times that of Distill BERT inference on n1-standard-96.

File: deploy/gke-marketplace-app/trt-engine/README.md Copy Chapter

Instruction to create BERT engine for each Triton update

Description Copy Chapter

docker run --gpus all -it --network host \
    --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \
    -v ~:/scripts nvcr.io/nvidia/tensorrt:26.07-py3

pip install onnx six torch tf2onnx tensorflow

git clone -b main https://github.com/NVIDIA/TensorRT.git
cd TensorRT
git submodule update --init --recursive

export TRT_OSSPATH=/workspace/TensorRT
export TRT_LIBPATH=/lib/x86_64-linux-gnu

pushd /usr/local/bin && wget https://ngc.nvidia.com/downloads/ngccli_cat_linux.zip && unzip ngccli_cat_linux.zip && chmod u+x ngc-cli/ngc && rm ngccli_cat_linux.zip ngc-cli.md5 && ln -s ngc-cli/ngc ngc && echo "no-apikey\nascii\n" | ngc config set

popd

cd /workspace/TensorRT/demo/BERT
bash ./scripts/download_squad.sh
bash ./scripts/download_model.sh large 128
# bash ./scripts/download_model.sh large 384

mkdir -p engines

python3 builder.py -m models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/model.ckpt -o engines/bert_large_int8_bs1_s128.engine -b 1 -s 128 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/ -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/vocab.txt --int8 --fp16 --strict --calib-num 1 -iln -imh

gsutil cp bert_large_int8_bs1_s128.engine gs://triton_sample_models/26.07/bert/1/model.plan

For each Triton upgrade, container version used to generate the model, and the model path in GCS gs://triton_sample_models/26.07/ should be updated accordingly with the correct version.

File: deploy/gke-marketplace-app/README.md Copy Chapter

NVIDIA Triton Inference Server GKE Marketplace Application

Table Of Contents

• NVIDIA Triton Inference Server GKE Marketplace Application
  • Description
  • Prerequisites
  • Demo Instruction
  • Additional Resources
  • Known Issues

Description Copy Chapter

This repository contains Google Kubernetes Engine(GKE) Marketplace Application for NVIDIA Triton Inference Server deployer.

• Triton GKE deployer is a helm chart deployer recommended by GKE Marketplace
• Triton GKE deployer deploys a GKE ingress which accepts public inference requests
• Triton GKE deployer includes a horizontal pod autoscaler(HPA) which relies on stack driver custom metrics adaptor to monitor GPU duty cycle, and auto scale GPU nodes.
• This repo also contains a sample to generate BERT model with TensorRT and use Locust to experiment with GPU node autoscaling and monitor client latency/throughput.

Prerequisites Copy Chapter

• Install Google Cloud SDK on your laptop/client workstation, so that gcloud SDK cli interface could be run on the client and sign in with your GCP credentials.
• In addition, user could leverage Google Cloud shell.

Demo Instruction Copy Chapter

First, install this Triton GKE app to an existing GKE cluster with GPU node pool, Google Cloud Marketplace currently doesn't support auto creation of GPU clusters. User has to run following command to create a compatible cluster (gke version >=1.18.7) with GPU node pools, we recommend user to select T4 or A100(MIG) instances type and choose CPU ratio based on profiling of actual inference workflow.

Users need to follow these instructions to create a kubernetes service account. In this example, we use [email protected]. Make sure it has access to artifact registry and monitoring viewer. For example, to grant access to custom metrics which is required for HPA to work:

gcloud iam service-accounts add-iam-policy-binding --role \
  roles/iam.workloadIdentityUser --member \
  "serviceAccount:.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]" \
  <google-service-account>@.iam.gserviceaccount.com

kubectl annotate serviceaccount --namespace custom-metrics \
  custom-metrics-stackdriver-adapter \
  iam.gke.io/gcp-service-account=<google-service-account>@.iam.gserviceaccount.com

Currently, GKE >= 1.18.7 only supported in GKE rapid channel, to find the latest version, please visit GKE release notes.

/* Detailed source-code truncated for AI context efficiency. */

Creating a cluster and adding GPU nodes could take up-to 10 minutes. Please be patient after executing this command. GPU resources in GCP could be fully utilized, so please try a different zone in case compute resource cannot be allocated. After GKE cluster is running, run kubectl get pods --all-namespaces to make sure the client can access the cluster correctly:

If user would like to experiment with A100 MIG partitioned GPU in GKE, please create node pool with following command:

gcloud beta container node-pools create accel \
  --project ${PROJECT_ID} \
  --zone ${ZONE} \
  --cluster ${DEPLOYMENT_NAME} \
  --service-account=${SERVICE_ACCOUNT} \
  --num-nodes 1 \
  --accelerator type=nvidia-tesla-a100,count=1,gpu-partition-size=1g.5gb  \
  --enable-autoscaling --min-nodes 1 --max-nodes 2 \
  --machine-type=a2-highgpu-1g  \
  --disk-size=100 \
  --scopes cloud-platform \
  --verbosity error

Please note that A100 MIG in GKE does not support GPU metrics yet, also Triton GPU Metrics is not compatible with A100 MIG. Hence, please disable GPU metrics by unselect allowGPUMetrics while deploy Triton GKE app. Also for the same reason, this deployer doesn't support inference workfload auto-scaling on A100 MIG as well.

Second, go to this GKE Marketplace link to deploy Triton application.

Users can leave everything as default if their models have already been tested/validated with Triton. They can provide a GCS path pointing to the model repository containing their models. By default, we provide a BERT large model optimized by TensorRT in a public demo GCS bucket that is compatible with the xx.yy release of Triton Server in gs://triton_sample_models/xx_yy. However, please take note of the following about this demo bucket:

• The TensorRT engine provided in the demo bucket is only compatible with Tesla T4 GPUs.
• This bucket is located in us-central1, so loading from this bucket into Triton in other regions may be affected.
• The first deployment of this Triton GKE application will be slower than consecutive runs because the image needs to be pulled into the GKE cluster.
• You can find an example of how this model is generated and uploaded here.

Where <xx.yy> is the version of NGC Triton container needed.

We want to discuss HPA autoscaling metrics users can leverage. GPU Power(Percentage of Power) tends to be a reliable metric, especially for larger GPU like V100 and A100. GKE currently natively support GPU duty cycle which is GPU utilization in nvidia-smi. We ask users always profile their model to determine the autoscaling target and metrics. When attempting to select the right metrics for autoscaling, the goal should be to pick metrics based on the following: 1, meet SLA rrequirement. 2, give consideration to transient request load, 3, keep GPU as fully utilized as possible. Profiling comes in 2 aspects: If user decided to use Duty Cycle or other GPU metric, it is recommend establish baseline to link SLA requirement such as latency with GPU metrics, for example, for model A, latency will be below 10ms 99% of time when Duty Cycle is below 80% utilized. Additionally, profiling also provide insight to model optimization for inference, with tools like Nsight.

Once the application is deployed successfully, get the public ip from ingress:

> kubectl get ingress
NAME              CLASS    HOSTS   ADDRESS          PORTS   AGE
triton-external   <none>   *       35.186.215.182   80      107s

Third, we will try sending request to server with provide client example.

If User selected deploy Triton to accept HTTP request, please launch Locust with Ingress host and port to query Triton Inference Server. In this example script, we send request to Triton server which has loaded a BERT large TensorRT Engine with Sequence length of 128 into GCP bucket. We simulate 1000 concurrent user as target and spawn user at rate of 50 users per second.

locust -f locustfile_bert.py -H http://${INGRESS_HOST}:${INGRESS_PORT}

The client example push about ~650 QPS(Query per second) to Triton Server, and will trigger a auto scale of T4 GPU nodes (We recommend to use T4 and A100[MIG] for inference). From locust UI, we will observer a drop of latency mean and variance for the requests. At the end, after autoscaling, we see the latency stablized at ~200 ms, end to end from US client to europe server, which is excellent for a model that has 345 million parameters. Since for each node, we use 1T4 + n1-standard-4 instance, and it can handle ~450 QPS, with on-demand price, it is ($0.35+$0.19)=$0.54/hr, that translate to 3 million inference per dollar for BERT large model at batch size 1. Further more, with 3 year commitment price, hr rate is ($0.16+$0.08)=$0.24/hr, that translate to 6.75 million inference per dollar.

Alternatively, user can opt to use
Perf Analyzer
to profile and study the performance of Triton Inference Server. Here we also
provide a
client script
to use Perf Analyzer to send gRPC to Triton Server GKE deployment. Perf Analyzer
client requires user to use NGC Triton Client Container.

bash perf_analyzer_grpc.sh ${INGRESS_HOST}:${INGRESS_PORT}

Additional Resources Copy Chapter

See the following resources to learn more about NVIDIA Triton Inference Server and GKE GPU capabilities.

Documentation

• GPU in Google Kubernetes Engine
• Optimize GPU Performance in Google Cloud Platform
• Triton Inference Server
• AI Platform Prediction: Custom container concepts with Triton Server by Kevin Tsai
• AI Platform Prediction: Direct model server setup for NVIDIA Triton Inference Server by Kevin Tsai

Known Issues Copy Chapter

• GKE one click cluster creation doesn't support GPU node pools at the moment, users have to manually create a compatible (>=1.18.7) cluster and attach node pool (T4 and A100 MIG recommended)
• When Horizontal Pod Autoscaler(HPA) expand and all GPU node pool already utilized, GKE will request new GPU node and it can take between 4-7 minutes, it could be a long wait plus GPU driver install and image pulling. We recommend user to leverage multi-tier model serving and Triton's priority feature to create cushion for latency critical models, and allocate active standby GPU node for spike of requests.

File: deploy/gcp/README.md Copy Chapter

Kubernetes Deploy: Triton Inference Server Cluster

A helm chart for installing a single cluster of Triton Inference
Server is provided. By default the cluster contains a single instance
of the inference server but the replicaCount configuration parameter
can be set to create a cluster of any size, as described below.

This guide assumes you already have a functional Kubernetes cluster
and helm installed (see below for instructions on installing
helm). Note the following requirements:

• The helm chart deploys Prometheus and Grafana to collect and display Triton metrics. Your cluster must contain sufficient CPU resources to support these services. At a minimum you will likely require 2 CPU nodes with machine type of n1-standard-2 or greater.

• If you want Triton Server to use GPUs for inferencing, your cluster
  must be configured to contain the desired number of GPU nodes with
  support for the NVIDIA driver and CUDA version required by the version
  of the inference server you are using.

This helm chart is available from Triton Inference Server
GitHub or from the
NVIDIA GPU Cloud (NGC).

The steps below describe how to set-up a model repository, use helm to
launch the inference server, and then send inference requests to the
running server. You can access a Grafana endpoint to see real-time
metrics reported by the inference server.

Installing Helm Copy Chapter

Helm v3 Copy Section

If you do not already have Helm installed in your Kubernetes cluster,
executing the following steps from the official helm install
guide will
give you a quick setup.

If you're currently using Helm v2 and would like to migrate to Helm v3,
please see the official migration guide.

Helm v2 Copy Section

NOTE: Moving forward this chart will only be tested and maintained for Helm v3.

Below are example instructions for installing Helm v2.

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash
$ kubectl create serviceaccount -n kube-system tiller
serviceaccount/tiller created
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
$ helm init --service-account tiller --wait

If you run into any issues, you can refer to the official installation guide here.

Model Repository Copy Chapter

If you already have a model repository you may use that with this helm
chart. If you do not have a model repository, you can checkout a local
copy of the inference server source repository to create an example
model repository::

$ git clone https://github.com/triton-inference-server/server.git

Triton Server needs a repository of models that it will make available
for inferencing. For this example you will place the model repository
in a Google Cloud Storage bucket.

$ gsutil mb gs://triton-inference-server-repository

Following the QuickStart download the
example model repository to your system and copy it into the GCS
bucket.

$ gsutil cp -r docs/examples/model_repository gs://triton-inference-server-repository/model_repository

GCS Permissions Copy Section

Make sure the bucket permissions are set so that the inference server
can access the model repository. If the bucket is public then no
additional changes are needed and you can proceed to "Deploy
Prometheus and Grafana" section.

If bucket premissions need to be set with the
GOOGLE_APPLICATION_CREDENTIALS environment variable then perform the
following steps:

• Generate Google service account JSON with proper permissions called
  gcp-creds.json.

• Create a Kubernetes secret from gcp-creds.json:

$ kubectl create configmap gcpcreds --from-literal "project-id=myproject"
  $ kubectl create secret generic gcpcreds --from-file gcp-creds.json

• Modify templates/deployment.yaml to include the
  GOOGLE_APPLICATION_CREDENTIALS environment variable:

env:
      - name: GOOGLE_APPLICATION_CREDENTIALS
        value: /secret/gcp-creds.json

• Modify templates/deployment.yaml to mount the secret in a volume at
  /secret:

volumeMounts:
      - name: vsecret
        mountPath: "/secret"
        readOnly: true
    ...
    volumes:
    - name: vsecret
      secret:
        secretName: gcpcreds

Deploy Prometheus and Grafana Copy Chapter

The inference server metrics are collected by Prometheus and viewable
by Grafana. The inference server helm chart assumes that Prometheus
and Grafana are available so this step must be followed even if you
don't want to use Grafana.

Use the kube-prometheus-stack to install these components. The
serviceMonitorSelectorNilUsesHelmValues flag is needed so that
Prometheus can find the inference server metrics in the example
release deployed below.

$ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack

Then port-forward to the Grafana service so you can access it from
your local browser.

$ kubectl port-forward service/example-metrics-grafana 8080:80

Now you should be able to navigate in your browser to localhost:8080
and see the Grafana login page. Use username=admin and
password=prom-operator to login.

An example Grafana dashboard is available in dashboard.json. Use the
import function in Grafana to import and view this dashboard.

Deploy the Inference Server Copy Chapter

Deploy the inference server using the default configuration with the
following commands.

$ cd <directory containing Chart.yaml>
$ helm install example .

Use kubectl to see status and wait until the inference server pods are
running.

$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
example-triton-inference-server-5f74b55885-n6lt7   1/1     Running   0          2m21s

There are several ways of overriding the default configuration as
described in this helm
documentation.

You can edit the values.yaml file directly or you can use the --set
option to override a single parameter with the CLI. For example, to
deploy a cluster of four inference servers use --set to set the
replicaCount parameter.

$ helm install example --set replicaCount=4 .

You can also write your own "config.yaml" file with the values you
want to override and pass it to helm.

$ cat << EOF > config.yaml
namespace: MyCustomNamespace
image:
  imageName: nvcr.io/nvidia/tritonserver:custom-tag
  modelRepositoryPath: gs://my_model_repository
EOF
$ helm install example -f config.yaml .

Using Triton Inference Server Copy Chapter

Now that the inference server is running you can send HTTP or GRPC
requests to it to perform inferencing. By default, the inferencing
service is exposed with a LoadBalancer service type. Use the following
to find the external IP for the inference server. In this case it is
34.83.9.133.

$ kubectl get services
NAME                             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                                        AGE
...
example-triton-inference-server  LoadBalancer   10.18.13.28    34.83.9.133   8000:30249/TCP,8001:30068/TCP,8002:32723/TCP   47m

The inference server exposes an HTTP endpoint on port 8000, and GRPC
endpoint on port 8001 and a Prometheus metrics endpoint on
port 8002. You can use curl to get the meta-data of the inference server
from the HTTP endpoint.

$ curl 34.83.9.133:8000/v2

Follow the QuickStart to get the example
image classification client that can be used to perform inferencing
using image classification models being served by the inference
server. For example,

$ image_client -u 34.83.9.133:8000 -m inception_v3_onnx -s INCEPTION -c3 mug.jpg
Request 0, batch size 1
Image 'images/mug.jpg':
    504 (COFFEE MUG) = 0.723992
    968 (CUP) = 0.270953
    967 (ESPRESSO) = 0.00115997

Cleanup Copy Chapter

Once you've finished using the inference server you should use helm to
delete the deployment.

$ helm list
NAME            REVISION  UPDATED                   STATUS    CHART                          APP VERSION   NAMESPACE
example         1         Wed Feb 27 22:16:55 2019  DEPLOYED  triton-inference-server-1.0.0  1.0           default
example-metrics	1       	Tue Jan 21 12:24:07 2020	DEPLOYED	prometheus-operator-6.18.0   	 0.32.0     	 default

$ helm uninstall example
$ helm uninstall example-metrics

For the Prometheus and Grafana services, you should explicitly delete
CRDs:

$ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com alertmanagers.monitoring.coreos.com podmonitors.monitoring.coreos.com probes.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com thanosrulers.monitoring.coreos.com

You may also want to delete the GCS bucket you created to hold the
model repository.

$ gsutil rm -r gs://triton-inference-server-repository

File: deploy/fleetcommand/README.md Copy Chapter

Fleet Command Deploy: NVIDIA Triton Inference Server

A helm chart for installing a single cluster of NVIDIA Triton Inference Server
on Fleet Command is provided. By default the cluster contains a single instance
of the Triton but the replicaCount configuration parameter can be set to
create a cluster of any size, as described below.

This guide assumes you already have a functional Fleet Command location
deployed. Please refer to the Fleet Command
Documentation

The steps below describe how to set-up a model repository, use helm to launch
the Triton, and then send inference requests to the running Triton Inference
Server. You can optionally scrape metrics with Prometheus and access a Grafana
endpoint to see real-time metrics reported by Triton.

Model Repository Copy Chapter

If you already have a model repository you may use that with this helm chart. If
you do not have a model repository, you can checkout a local copy of the Triton
Inference Server source repository to create an example model repository::

$ git clone https://github.com/triton-inference-server/server.git

Triton needs a repository of models that it will make available for inferencing.
For this example you will place the model repository in an S3 Storage bucket
(either in AWS or other S3 API compatible on-premises object storage).

$ aws s3 mb s3://triton-inference-server-repository

Following the QuickStart download the example model
repository to your system and copy it into the AWS S3 bucket.

$ aws s3 cp -r docs/examples/model_repository s3://triton-inference-server-repository/model_repository

AWS Model Repository Copy Section

To load the model from the AWS S3, you need to convert the following AWS
credentials in the base64 format and add it to the Application Configuration
section when creating the Fleet Command Deployment.

echo -n 'REGION' | base64
echo -n 'SECRECT_KEY_ID' | base64
echo -n 'SECRET_ACCESS_KEY' | base64
# Optional for using session token
echo -n 'AWS_SESSION_TOKEN' | base64

Deploy the Triton Inference Server Copy Chapter

Deploy the Triton Inference Server to your Location in Fleet Command by creating
a Deployment. You can specify configuration parameters to override the default
values.yaml in the Application Configuration section.

Note: You must provide a --model-repository parameter with a path to your
prepared model repository in your S3 bucket. Otherwise, the Triton will not
start.

An example Application Configuration for Triton on Fleet Command:

image:
  serverArgs:
    - --model-repository=s3://triton-inference-server-repository

secret:
  region: <region in base 64 >
  id: <access id in base 64 >
  key: <access key in base 64>
  token: <session token in base 64 (optional)>

See Fleet Command documentation
for more info.

Prometheus ServiceMonitor Support Copy Section

If you have prometheus-operator deployed, you can enable the ServiceMonitor
for the Triton Inference Server by setting serviceMonitor.enabled: true in
Application Configuration. This will also deploy a Grafana dashboard for Triton
as a ConfigMap.

Otherwise, metrics can be scraped by pointing an external Prometheus
instance at the metricsNodePort in the values.

Using Triton Inference Server Copy Chapter

Now that the Triton Inference Server is running you can send HTTP or GRPC
requests to it to perform inferencing. By default, the service is exposed with a
NodePort service type, where the same port is opened on all systems in a
Location.

Triton exposes an HTTP endpoint on port 30343, and GRPC endpoint on port 30344
and a Prometheus metrics endpoint on port 30345. These ports can be overridden
in the application configuration when deploying. You can use curl to get the
meta-data of Triton from the HTTP endpoint. For example, if a system in your
location has the IP 34.83.9.133:

$ curl 34.83.9.133:30343/v2

Follow the QuickStart to get the example image
classification client that can be used to perform inferencing using image
classification models being served by the Triton. For example,

$ image_client -u 34.83.9.133:30343 -m densenet_onnx -s INCEPTION -c 3 mug.jpg
Request 0, batch size 1
Image '/workspace/images/mug.jpg':
    15.349568 (504) = COFFEE MUG
    13.227468 (968) = CUP
    10.424893 (505) = COFFEEPOT

File: deploy/aws/README.md Copy Chapter

Kubernetes Deploy: Triton Inference Server Cluster

A helm chart for installing a single cluster of Triton Inference
Server is provided. By default the cluster contains a single instance
of the inference server but the replicaCount configuration parameter
can be set to create a cluster of any size, as described below.

This guide assumes you already have a functional Kubernetes cluster
and helm installed (see below for instructions on installing
helm). Note the following requirements:

• The helm chart deploys Prometheus and Grafana to collect and display Triton metrics. To use this helm chart you must install Prpmetheus and Grafana in your cluster as described below and your cluster must contain sufficient CPU resources to support these services.

• If you want Triton Server to use GPUs for inferencing, your cluster
  must be configured to contain the desired number of GPU nodes (EC2 G4 instances recommended)
  with support for the NVIDIA driver and CUDA version required by the version
  of the inference server you are using.

The steps below describe how to set-up a model repository, use helm to
launch the inference server, and then send inference requests to the
running server. You can access a Grafana endpoint to see real-time
metrics reported by the inference server.

Installing Helm Copy Chapter

Helm v3 Copy Section

If you do not already have Helm installed in your Kubernetes cluster,
executing the following steps from the official helm install
guide will
give you a quick setup.

If you're currently using Helm v2 and would like to migrate to Helm v3,
please see the official migration guide.

Helm v2 Copy Section

NOTE: Moving forward this chart will only be tested and maintained for Helm v3.

Below are example instructions for installing Helm v2.

$ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash
$ kubectl create serviceaccount -n kube-system tiller
serviceaccount/tiller created
$ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
$ helm init --service-account tiller --wait

If you run into any issues, you can refer to the official installation guide here.

Model Repository Copy Chapter

If you already have a model repository you may use that with this helm
chart. If you do not have a model repository, you can checkout a local
copy of the inference server source repository to create an example
model repository::

$ git clone https://github.com/triton-inference-server/server.git

Triton Server needs a repository of models that it will make available
for inferencing. For this example you will place the model repository
in an AWS S3 Storage bucket.

$ aws s3 mb s3://triton-inference-server-repository

Following the QuickStart download the
example model repository to your system and copy it into the AWS S3
bucket.

$ aws s3 cp --recursive docs/examples/model_repository s3://triton-inference-server-repository/model_repository

AWS Model Repository Copy Section

To load the model from the AWS S3, you need to convert the following AWS credentials in the base64 format and add it to the values.yaml

echo -n 'REGION' | base64

echo -n 'SECRECT_KEY_ID' | base64

echo -n 'SECRET_ACCESS_KEY' | base64

Deploy Prometheus and Grafana Copy Chapter

The inference server metrics are collected by Prometheus and viewable
by Grafana. The inference server helm chart assumes that Prometheus
and Grafana are available so this step must be followed even if you
don't want to use Grafana.

Use the kube-prometheus-stack to install these components. The
serviceMonitorSelectorNilUsesHelmValues flag is needed so that
Prometheus can find the inference server metrics in the example
release deployed below.

$ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack

Then port-forward to the Grafana service so you can access it from
your local browser.

$ kubectl port-forward service/example-metrics-grafana 8080:80

Now you should be able to navigate in your browser to localhost:8080
and see the Grafana login page. Use username=admin and
password=prom-operator to login.

An example Grafana dashboard is available in dashboard.json. Use the
import function in Grafana to import and view this dashboard.

Deploy the Inference Server Copy Chapter

Deploy the inference server using the default configuration with the
following commands.

$ cd <directory containing Chart.yaml>
$ helm install example .

Use kubectl to see status and wait until the inference server pods are
running.

$ kubectl get pods
NAME                                               READY   STATUS    RESTARTS   AGE
example-triton-inference-server-5f74b55885-n6lt7   1/1     Running   0          2m21s

There are several ways of overriding the default configuration as
described in this helm
documentation.

You can edit the values.yaml file directly or you can use the --set
option to override a single parameter with the CLI. For example, to
deploy a cluster of four inference servers use --set to set the
replicaCount parameter.

$ helm install example --set replicaCount=4 .

You can also write your own "config.yaml" file with the values you
want to override and pass it to helm.

$ cat << EOF > config.yaml
namespace: MyCustomNamespace
image:
  imageName: nvcr.io/nvidia/tritonserver:custom-tag
  modelRepositoryPath: gs://my_model_repository
EOF
$ helm install example -f config.yaml .

Using Triton Inference Server Copy Chapter

Now that the inference server is running you can send HTTP or GRPC
requests to it to perform inferencing. By default, the inferencing
service is exposed with a LoadBalancer service type. Use the following
to find the external IP for the inference server. In this case it is
34.83.9.133.

$ kubectl get services
NAME                             TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)                                        AGE
...
example-triton-inference-server  LoadBalancer   10.18.13.28    34.83.9.133   8000:30249/TCP,8001:30068/TCP,8002:32723/TCP   47m

The inference server exposes an HTTP endpoint on port 8000, and GRPC
endpoint on port 8001 and a Prometheus metrics endpoint on
port 8002. You can use curl to get the meta-data of the inference server
from the HTTP endpoint.

$ curl 34.83.9.133:8000/v2

Follow the QuickStart to get the example
image classification client that can be used to perform inferencing
using image classification models being served by the inference
server. For example,

$ image_client -u 34.83.9.133:8000 -m inception_v3_onnx -s INCEPTION -c3 mug.jpg
Request 0, batch size 1
Image 'images/mug.jpg':
    504 (COFFEE MUG) = 0.723992
    968 (CUP) = 0.270953
    967 (ESPRESSO) = 0.00115997

Cleanup Copy Chapter

Once you've finished using the inference server you should use helm to
delete the deployment.

$ helm list
NAME            REVISION  UPDATED                   STATUS    CHART                          APP VERSION   NAMESPACE
example         1         Wed Feb 27 22:16:55 2019  DEPLOYED  triton-inference-server-1.0.0  1.0           default
example-metrics	1       	Tue Jan 21 12:24:07 2020	DEPLOYED	prometheus-operator-6.18.0   	 0.32.0     	 default

$ helm uninstall example
$ helm uninstall example-metrics

For the Prometheus and Grafana services, you should explicitly delete
CRDs:

$ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com alertmanagers.monitoring.coreos.com podmonitors.monitoring.coreos.com probes.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com thanosrulers.monitoring.coreos.com

You may also want to delete the AWS bucket you created to hold the
model repository.

$ aws s3 rm -r gs://triton-inference-server-repository