## File: README.md [](https://opensource.org/licenses/BSD-3-Clause) >[!WARNING] >You are currently on the `main` branch which tracks under-development progress >towards the next release. The current release is version [2.71.0](https://github.com/triton-inference-server/server/releases/latest) >and corresponds to the 26.07 container release on NVIDIA GPU Cloud (NGC). # Triton Inference Server Triton Inference Server is an open source inference serving software that streamlines AI inferencing. Triton enables teams to deploy any AI model from multiple deep learning and machine learning frameworks, including TensorRT, PyTorch, ONNX, OpenVINO, Python, RAPIDS FIL, and more. Triton Inference Server supports inference across cloud, data center, edge and embedded devices on NVIDIA GPUs, x86 and ARM CPU, or AWS Inferentia. Triton Inference Server delivers optimized performance for many query types, including real time, batched, ensembles and audio/video streaming. Triton inference Server is part of [NVIDIA AI Enterprise](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/), a software platform that accelerates the data science pipeline and streamlines the development and deployment of production AI. Major features include: - [Supports multiple deep learning frameworks](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton) - [Supports multiple machine learning frameworks](https://github.com/triton-inference-server/fil_backend) - [Concurrent model execution](docs/user_guide/architecture.md#concurrent-model-execution) - [Dynamic batching](docs/user_guide/batcher.md#dynamic-batcher) - [Sequence batching](docs/user_guide/batcher.md#sequence-batcher) and [implicit state management](docs/user_guide/architecture.md#implicit-state-management) for stateful models - Provides [Backend API](https://github.com/triton-inference-server/backend) that allows adding custom backends and pre/post processing operations - Supports writing custom backends in python, a.k.a. [Python-based backends.](https://github.com/triton-inference-server/backend/blob/main/docs/python_based_backends.md#python-based-backends) - Model pipelines using [Ensembling](docs/user_guide/architecture.md#ensemble-models) or [Business Logic Scripting (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) - [HTTP/REST and GRPC inference protocols](docs/customization_guide/inference_protocols.md) based on the community developed [KServe protocol](https://github.com/kserve/kserve/tree/master/docs/predict-api/v2) - A [C API](docs/customization_guide/inprocess_c_api.md) and [Java API](docs/customization_guide/inprocess_java_api.md) allow Triton to link directly into your application for edge and other in-process use cases - [Metrics](docs/user_guide/metrics.md) indicating GPU utilization, server throughput, server latency, and more **New to Triton Inference Server?** Make use of [these tutorials](https://github.com/triton-inference-server/tutorials) to begin your Triton journey! Join the [Triton and TensorRT community](https://www.nvidia.com/en-us/deep-learning-ai/triton-tensorrt-newsletter/) and stay current on the latest product updates, bug fixes, content, best practices, and more. Need enterprise support? NVIDIA global support is available for Triton Inference Server with the [NVIDIA AI Enterprise software suite](https://www.nvidia.com/en-us/data-center/products/ai-enterprise/). ## Serve a Model in 3 Easy Steps ```bash # Step 1: Create the example model repository git clone -b r26.07 https://github.com/triton-inference-server/server.git cd server/docs/examples ./fetch_models.sh # Step 2: Launch triton from the NGC Triton container docker run --gpus=1 --rm --net=host -v ${PWD}/model_repository:/models nvcr.io/nvidia/tritonserver:26.07-py3 tritonserver --model-repository=/models --model-control-mode explicit --load-model densenet_onnx # Step 3: Sending an Inference Request # In a separate console, launch the image_client example from the NGC Triton SDK container docker run -it --rm --net=host nvcr.io/nvidia/tritonserver:26.07-py3-sdk /workspace/install/bin/image_client -m densenet_onnx -c 3 -s INCEPTION /workspace/images/mug.jpg # Inference should return the following Image '/workspace/images/mug.jpg': 15.346230 (504) = COFFEE MUG 13.224326 (968) = CUP 10.422965 (505) = COFFEEPOT ``` Please read the [QuickStart](docs/getting_started/quickstart.md) guide for additional information regarding this example. The quickstart guide also contains an example of how to launch Triton on [CPU-only systems](docs/getting_started/quickstart.md#run-on-cpu-only-system). New to Triton and wondering where to get started? Watch the [Getting Started video](https://youtu.be/NQDtfSi5QF4). ## Examples and Tutorials Check out [NVIDIA LaunchPad](https://www.nvidia.com/en-us/data-center/products/ai-enterprise-suite/trial/) for free access to a set of hands-on labs with Triton Inference Server hosted on NVIDIA infrastructure. Specific end-to-end examples for popular models, such as ResNet, BERT, and DLRM are located in the [NVIDIA Deep Learning Examples](https://github.com/NVIDIA/DeepLearningExamples) page on GitHub. The [NVIDIA Developer Zone](https://developer.nvidia.com/nvidia-triton-inference-server) contains additional documentation, presentations, and examples. ## Documentation ### Build and Deploy The recommended way to build and use Triton Inference Server is with Docker images. - [Install Triton Inference Server with Docker containers](docs/customization_guide/build.md#building-with-docker) (*Recommended*) - [Install Triton Inference Server without Docker containers](docs/customization_guide/build.md#building-without-docker) - [Build a custom Triton Inference Server Docker container](docs/customization_guide/compose.md) - [Build Triton Inference Server from source](docs/customization_guide/build.md#building-on-unsupported-platforms) - Examples for deploying Triton Inference Server with Kubernetes and Helm on [GCP](deploy/gcp/README.md), [AWS](deploy/aws/README.md), and [NVIDIA FleetCommand](deploy/fleetcommand/README.md) - [Secure Deployment Considerations](docs/customization_guide/deploy.md) ### Using Triton #### Preparing Models for Triton Inference Server The first step in using Triton to serve your models is to place one or more models into a [model repository](docs/user_guide/model_repository.md). Depending on the type of the model and on what Triton capabilities you want to enable for the model, you may need to create a [model configuration](docs/user_guide/model_configuration.md) for the model. - [Add custom operations to Triton if needed by your model](docs/user_guide/custom_operations.md) - Enable model pipelining with [Model Ensemble](docs/user_guide/architecture.md#ensemble-models) and [Business Logic Scripting (BLS)](https://github.com/triton-inference-server/python_backend#business-logic-scripting) - Optimize your models setting [scheduling and batching](docs/user_guide/architecture.md#models-and-schedulers) parameters and [model instances](docs/user_guide/model_configuration.md#instance-groups). - Use the [Model Analyzer tool](https://github.com/triton-inference-server/model_analyzer) to help optimize your model configuration with profiling - Learn how to [explicitly manage what models are available by loading and unloading models](docs/user_guide/model_management.md) #### Configure and Use Triton Inference Server - Read the [Quick Start Guide](docs/getting_started/quickstart.md) to run Triton Inference Server on both GPU and CPU - Triton supports multiple execution engines, called [backends](https://github.com/triton-inference-server/backend#where-can-i-find-all-the-backends-that-are-available-for-triton), including [TensorRT](https://github.com/triton-inference-server/tensorrt_backend), [PyTorch](https://github.com/triton-inference-server/pytorch_backend), [ONNX](https://github.com/triton-inference-server/onnxruntime_backend), [OpenVINO](https://github.com/triton-inference-server/openvino_backend), [Python](https://github.com/triton-inference-server/python_backend), and more - Not all the above backends are supported on every platform supported by Triton. Look at the [Backend-Platform Support Matrix](https://github.com/triton-inference-server/backend/blob/main/docs/backend_platform_support_matrix.md) to learn which backends are supported on your target platform. - Learn how to [optimize performance](docs/user_guide/optimization.md) using the [Performance Analyzer](https://github.com/triton-inference-server/perf_analyzer/blob/main/README.md) and [Model Analyzer](https://github.com/triton-inference-server/model_analyzer) - Learn how to [manage loading and unloading models](docs/user_guide/model_management.md) in Triton - Send requests directly to Triton with the [HTTP/REST JSON-based or gRPC protocols](docs/customization_guide/inference_protocols.md#httprest-and-grpc-protocols) #### Client Support and Examples A Triton *client* application sends inference and other requests to Triton. The [Python and C++ client libraries](https://github.com/triton-inference-server/client) provide APIs to simplify this communication. - Review client examples for [C++](https://github.com/triton-inference-server/client/blob/main/src/c%2B%2B/examples), [Python](https://github.com/triton-inference-server/client/blob/main/src/python/examples), and [Java](https://github.com/triton-inference-server/client/blob/main/src/java/src/main/java/triton/client/examples) - Configure [HTTP](https://github.com/triton-inference-server/client#http-options) and [gRPC](https://github.com/triton-inference-server/client#grpc-options) client options - Send input data (e.g. a jpeg image) directly to Triton in the [body of an HTTP request without any additional metadata](https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_binary_data.md#raw-binary-request) ### Extend Triton [Triton Inference Server's architecture](docs/user_guide/architecture.md) is specifically designed for modularity and flexibility - [Customize Triton Inference Server container](docs/customization_guide/compose.md) for your use case - [Create custom backends](https://github.com/triton-inference-server/backend) in either [C/C++](https://github.com/triton-inference-server/backend/blob/main/README.md#triton-backend-api) or [Python](https://github.com/triton-inference-server/python_backend) - Create [decoupled backends and models](docs/user_guide/decoupled_models.md) that can send multiple responses for a request or not send any responses for a request - Use a [Triton repository agent](docs/customization_guide/repository_agents.md) to add functionality that operates when a model is loaded and unloaded, such as authentication, decryption, or conversion - Deploy Triton on [Jetson and JetPack](docs/user_guide/jetson.md) - [Use Triton on AWS Inferentia](https://github.com/triton-inference-server/python_backend/tree/main/inferentia) ### Additional Documentation - [FAQ](docs/user_guide/faq.md) - [User Guide](docs/README.md#user-guide) - [Customization Guide](docs/README.md#customization-guide) - [Release Notes](https://docs.nvidia.com/deeplearning/triton-inference-server/release-notes/index.html) - [GPU, Driver, and CUDA Support Matrix](https://docs.nvidia.com/deeplearning/dgx/support-matrix/index.html) ## Contributing Contributions to Triton Inference Server are more than welcome. To contribute please review the [contribution guidelines](CONTRIBUTING.md). If you have a backend, client, example or similar contribution that is not modifying the core of Triton, then you should file a PR in the [contrib repo](https://github.com/triton-inference-server/contrib). ## Reporting problems, asking questions We appreciate any feedback, questions or bug reporting regarding this project. When posting [issues in GitHub](https://github.com/triton-inference-server/server/issues), follow the process outlined in the [Stack Overflow document](https://stackoverflow.com/help/mcve). Ensure posted examples are: - minimal – use as little code as possible that still produces the same problem - complete – provide all parts needed to reproduce the problem. Check if you can strip external dependencies and still show the problem. The less time we spend on reproducing problems the more time we have to fix it - verifiable – test the code you're about to provide to make sure it reproduces the problem. Remove all other problems that are not related to your request/question. For issues, please use the provided bug report and feature request templates. For questions, we recommend posting in our community [GitHub Discussions.](https://github.com/triton-inference-server/server/discussions) ## For more information Please refer to the [NVIDIA Developer Triton page](https://developer.nvidia.com/nvidia-triton-inference-server) for more information. --- ## File: deploy/oci/README.md [](https://opensource.org/licenses/BSD-3-Clause) # Kubernetes Deploy: Triton Inference Server Cluster A helm chart for installing a single cluster of Triton Inference Server is provided. By default the cluster contains a single instance of the inference server but the *replicaCount* configuration parameter can be set to create a cluster of any size, as described below. This guide assumes you already have a functional Kubernetes cluster and helm installed (see below for instructions on installing helm). Note the following requirements: * The helm chart deploys Prometheus and Grafana to collect and display Triton metrics. To use this helm chart you must install Prometheus and Grafana in your cluster as described below and your cluster must contain sufficient CPU resources to support these services. * If you want Triton Server to use GPUs for inferencing, your cluster must be configured to contain the desired number of GPU nodes (A10 GPU instances recommended) with support for the NVIDIA driver and CUDA version required by the version of the inference server you are using. The steps below describe how to set-up a model repository, use helm to launch the inference server, and then send inference requests to the running server. You can access a Grafana endpoint to see real-time metrics reported by the inference server. ## Notes for OKE cluster When creating your node pool, the default value for the boot volume is 46.6GB. Due to the size of the server container, it is recommended to increase this value to 150GB and set a [cloud-init script to increase the partition](https://blogs.oracle.com/ateam/post/oke-node-sizing-for-very-large-container-images): ``` #!/bin/bash curl --fail -H "Authorization: Bearer Oracle" -L0 http://169.254.169.254/opc/v2/instance/metadata/oke_init_script | base64 --decode >/var/run/oke-init.sh bash /var/run/oke-init.sh sudo /usr/libexec/oci-growfs -y ``` ## Installing Helm ### Using Cloud Shell from OCI Web Console It is possible to access your OKE Cluster [directly from the OCI Web Console](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengaccessingclusterkubectl.htm). Helm v3 is already available from the Cloud Shell. ### Helm v3 If you do not already have Helm installed in your Kubernetes cluster, executing the following steps from the [official helm install guide](https://helm.sh/docs/intro/install/) will give you a quick setup. If you're currently using Helm v2 and would like to migrate to Helm v3, please see the [official migration guide](https://helm.sh/docs/topics/v2_v3_migration/). ### Helm v2 > **NOTE**: Moving forward this chart will only be tested and maintained for Helm v3. Below are example instructions for installing Helm v2. ``` $ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash $ kubectl create serviceaccount -n kube-system tiller serviceaccount/tiller created $ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller $ helm init --service-account tiller --wait ``` If you run into any issues, you can refer to the official installation guide [here](https://v2.helm.sh/docs/install/). ## Model Repository If you already have a model repository you may use that with this helm chart. If you do not have a model repository, you can checkout a local copy of the inference server source repository to create an example model repository: ``` $ git clone https://github.com/triton-inference-server/server.git ``` Triton Server needs a repository of models that it will make available for inferencing. For this example you will place the model repository in an S3 compatible OCI Object Storage Bucket. ``` $ oci os bucket create --compartment-id --name triton-inference-server-repository ``` Following the [QuickStart](../../docs/getting_started/quickstart.md) download the example model repository to your system and copy it into the OCI Bucket. ``` $ oci os object bulk-upload -bn triton-inference-server-repository --src-dir docs/examples/model_repository/ ``` ### OCI Model Repository To load the model from the OCI Object Storage Bucket, you need to convert the following OCI credentials in the base64 format and add it to the values.yaml ``` echo -n 'REGION' | base64 ``` ``` echo -n 'SECRECT_KEY_ID' | base64 ``` ``` echo -n 'SECRET_ACCESS_KEY' | base64 ``` You also need to adapt _modelRepositoryPath_ in values.yaml to your [namespace](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/understandingnamespaces.htm) and [OCI region](https://docs.oracle.com/en-us/iaas/Content/General/Concepts/regions.htm). ``` s3://https://.compat.objectstorage..oraclecloud.com:443/triton-inference-server-repository ``` ## Deploy Prometheus and Grafana The inference server metrics are collected by Prometheus and viewable by Grafana. The inference server helm chart assumes that Prometheus and Grafana are available so this step must be followed even if you don't want to use Grafana. Use the [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) to install these components. The *serviceMonitorSelectorNilUsesHelmValues* flag is needed so that Prometheus can find the inference server metrics in the *example* release deployed below. ``` $ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack ``` Then port-forward to the Grafana service so you can access it from your local browser. ``` $ kubectl port-forward service/example-metrics-grafana 8080:80 ``` Now you should be able to navigate in your browser to localhost:8080 and see the Grafana login page. Use username=admin and password=prom-operator to login. Note that it is also possible to set a load balancer service for the grafana dashboard by running: ``` $ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false --set grafana.service.type=LoadBalancer prometheus-community/kube-prometheus-stack ``` You can then see the Public IP of you grafana dashboard by running: ``` $ kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-operated ClusterIP None 9093/TCP,9094/TCP,9094/UDP 2m33s example-metrics-grafana LoadBalancer 10.96.82.33 141.145.220.114 80:31005/TCP 2m38s ``` The default load balancer created comes with a fixed shape and a bandwidth of 100Mbps. You can switch to a [flexible](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengcreatingloadbalancers-subtopic.htm#contengcreatingloadbalancers_subtopic) shape and adapt the bandwidth according to your OCI limits in case the bandwidth is a bottleneck. An example Grafana dashboard is available in dashboard.json. Use the import function in Grafana to import and view this dashboard. ## Deploy the Inference Server Deploy the inference server using the default configuration with the following commands. ``` $ cd $ helm install example . ``` Use kubectl to see status and wait until the inference server pods are running. ``` $ kubectl get pods NAME READY STATUS RESTARTS AGE example-triton-inference-server-5f74b55885-n6lt7 1/1 Running 0 2m21s ``` There are several ways of overriding the default configuration as described in this [helm documentation](https://helm.sh/docs/using_helm/#customizing-the-chart-before-installing). You can edit the values.yaml file directly or you can use the *--set* option to override a single parameter with the CLI. For example, to deploy a cluster of four inference servers use *--set* to set the replicaCount parameter. ``` $ helm install example --set replicaCount=4 . ``` You can also write your own "config.yaml" file with the values you want to override and pass it to helm. ``` $ cat << EOF > config.yaml namespace: MyCustomNamespace image: imageName: nvcr.io/nvidia/tritonserver:custom-tag modelRepositoryPath: s3://https://.compat.objectstorage..oraclecloud.com:443/triton-inference-server-repository EOF $ helm install example -f config.yaml . ``` ## Using Triton Inference Server Now that the inference server is running you can send HTTP or GRPC requests to it to perform inferencing. By default, the inferencing service is exposed with a LoadBalancer service type. Use the following to find the external IP for the inference server. In this case it is 34.83.9.133. ``` $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ... example-triton-inference-server LoadBalancer 10.18.13.28 34.83.9.133 8000:30249/TCP,8001:30068/TCP,8002:32723/TCP 47m ``` The inference server exposes an HTTP endpoint on port 8000, and GRPC endpoint on port 8001 and a Prometheus metrics endpoint on port 8002. You can use curl to get the meta-data of the inference server from the HTTP endpoint. ``` $ curl 34.83.9.133:8000/v2 ``` Follow the [QuickStart](../../docs/getting_started/quickstart.md) to get the example image classification client that can be used to perform inferencing using image classification models being served by the inference server. For example, ``` $ image_client -u 34.83.9.133:8000 -m inception_v3_onnx -s INCEPTION -c3 mug.jpg Request 0, batch size 1 Image 'images/mug.jpg': 504 (COFFEE MUG) = 0.723992 968 (CUP) = 0.270953 967 (ESPRESSO) = 0.00115997 ``` ## Cleanup Once you've finished using the inference server you should use helm to delete the deployment. ``` $ helm list NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE example 1 Wed Feb 27 22:16:55 2019 DEPLOYED triton-inference-server-1.0.0 1.0 default example-metrics 1 Tue Jan 21 12:24:07 2020 DEPLOYED prometheus-operator-6.18.0 0.32.0 default $ helm uninstall example $ helm uninstall example-metrics ``` For the Prometheus and Grafana services, you should [explicitly delete CRDs](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#uninstall-helm-chart): ``` $ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com alertmanagers.monitoring.coreos.com podmonitors.monitoring.coreos.com probes.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com thanosrulers.monitoring.coreos.com ``` You may also want to delete the OCI bucket you created to hold the model repository. ``` $ oci os bucket delete --bucket-name triton-inference-server-repository --empty ``` --- ## File: deploy/mlflow-triton-plugin/README.md # MLflow Triton MLflow plugin for deploying your models from MLflow to Triton Inference Server. Scripts are included for publishing models, which are in Triton recognized structure, to your MLflow Model Registry. ### Supported flavors MLFlow Triton plugin currently supports the following flavors, you may substitute the flavor specification in the example below according to the model to be deployed. * onnx * triton ## Requirements * MLflow * Triton Python HTTP client * Triton Inference Server ## Installation The plugin can be installed from source using the following commands ```bash python setup.py install ``` ## Quick Start In this documentation, we will use the files in `examples` to showcase how the plugin interacts with Triton Inference Server. The `onnx_float32_int32_int32` model in `examples` is a simple model that takes two float32 inputs, INPUT0 and INPUT1, with shape [-1, 16], and produces two int32 outputs, OUTPUT0 and OUTPUT1, where OUTPUT0 is the element-wise summation of INPUT0 and INPUT1 and OUTPUT1 is the element-wise subtraction of INPUT0 and INPUT1. ### Start Triton Inference Server in EXPLICIT mode The MLflow Triton plugin must work with a running Triton server, see [documentation](https://github.com/triton-inference-server/server/blob/main/docs/getting_started/quickstart.md) of Triton Inference Server for how to start the server. Note that the server should be run in EXPLICIT mode (`--model-control-mode=explicit`) to exploit the deployment feature of the plugin. Once the server has started, the following environment must be set so that the plugin can interact with the server properly: * `TRITON_URL`: The address to the Triton HTTP endpoint * `TRITON_MODEL_REPO`: The path to the Triton model repository. It can be an s3 URI but keep in \ mind that the env vars AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are needed. ### Publish models to MLflow #### ONNX flavor The MLFlow ONNX built-in functionalities can be used to publish `onnx` flavor models to MLFlow directly, and the MLFlow Triton plugin will prepare the model to the format expected by Triton. You may also log [`config.pbtxt`](https://github.com/triton-inference-server/server/blob/main/docs/protocol/extension_model_configuration.md) as additional artifact which Triton will be used to serve the model. Otherwise, the server should be run with auto-complete feature enabled (`--strict-model-config=false`) to generate the model configuration. ```bash import mlflow.onnx import onnx model = onnx.load("examples/onnx_float32_int32_int32/1/model.onnx") mlflow.onnx.log_model(model, "triton", registered_model_name="onnx_float32_int32_int32") ``` #### Triton flavor For other model frameworks that Triton supports but not yet recognized by the MLFlow Triton plugin, the `publish_model_to_mlflow.py` script can be used to publish `triton` flavor models to MLflow. A `triton` flavor model is a directory containing the model files following the [model layout](https://github.com/triton-inference-server/server/blob/main/docs/user_guide/model_repository.md#repository-layout). Below is an example usage: ```bash cd /scripts python publish_model_to_mlflow.py --model_name onnx_float32_int32_int32 --model_directory /onnx_float32_int32_int32 --flavor triton ``` ### Deploy models tracked in MLflow to Triton Once a model is published and tracked in MLflow, it can be deployed to Triton via MLflow's deployments command, the following command will download the model to Triton's model repository and request Triton to load the model. ```bash mlflow deployments create -t triton --flavor triton --name onnx_float32_int32_int32 -m models:/onnx_float32_int32_int32/1 ``` ### Perform inference After the model is deployed, the following command is the CLI usage to send inference request to a deployment. ```bash mlflow deployments predict -t triton --name onnx_float32_int32_int32 --input-path /input.json --output-path output.json ``` The inference result will be written in `output.json` and you may compare it with the results in `expected_output.json` ## MLflow Deployments "MLflow Deployments" is a set of MLflow APIs for deploying MLflow models to custom serving tools. The MLflow Triton plugin implements the following deployment functions to support the interaction with Triton server in MLflow. ### Create Deployment MLflow deployments create API deploys a model to the Triton target, which will download the model to Triton's model repository and request Triton to load the model. To create a MLflow deployment using CLI ```bash mlflow deployments create -t triton --flavor triton --name model_name -m models:/model_name/1 ``` To create a MLflow deployment using Python API ```bash from mlflow.deployments import get_deploy_client client = get_deploy_client('triton') client.create_deployment("model_name", "models:/model_name/1", flavor="triton") ``` ### Delete Deployment MLflow deployments delete API removes an existing deployment from the Triton target, which will remove the model in Triton's model repository and request Triton to unload the model. To delete a MLflow deployment using CLI ```bash mlflow deployments delete -t triton --name model_name ``` To delete a MLflow deployment using Python API ```bash from mlflow.deployments import get_deploy_client client = get_deploy_client('triton') client.delete_deployment("model_name") ``` ### Update Deployment MLflow deployments update API updates an existing deployment with another model (version) tracked in MLflow, which will overwrite the model in Triton's model repository and request Triton to reload the model. To update a MLflow deployment using CLI ```bash mlflow deployments update -t triton --flavor triton --name model_name -m models:/model_name/2 ``` To update a MLflow deployment using Python API ```bash from mlflow.deployments import get_deploy_client client = get_deploy_client('triton') client.update_deployment("model_name", "models:/model_name/2", flavor="triton") ``` ### List Deployments MLflow deployments list API lists all existing deployments in Triton target. To list all MLflow deployments using CLI ```bash mlflow deployments list -t triton ``` To list all MLflow deployments using Python API ```bash from mlflow.deployments import get_deploy_client client = get_deploy_client('triton') client.list_deployments() ``` ### Get Deployment MLflow deployments get API returns information regarding a specific deployments in Triton target. To list a specific MLflow deployment using CLI ```bash mlflow deployments get -t triton --name model_name ``` To list a specific MLflow deployment using Python API ```bash from mlflow.deployments import get_deploy_client client = get_deploy_client('triton') client.get_deployment("model_name") ``` ### Run Inference on Deployments MLflow deployments predict API runs inference by preparing and sending the request to Triton and returns the Triton response. To run inference using CLI ```bash mlflow deployments predict -t triton --name model_name --input-path input_file --output-path output_file ``` To run inference using Python API ```bash from mlflow.deployments import get_deploy_client client = get_deploy_client('triton') client.predict("model_name", inputs) ``` --- ## File: deploy/k8s-onprem/README.md [](https://opensource.org/licenses/BSD-3-Clause) # Kubernetes Deploy: NVIDIA Triton Inference Server Cluster This repository includes a Helm chart and instructions for installing NVIDIA Triton Inference Server in an on-premises or AWS EC2 Kubernetes cluster. You can also use this repository to enable load balancing and autoscaling for your Triton cluster. This guide assumes you already have a functional Kubernetes cluster with support for GPUs. See the [NVIDIA GPU Operator documentation](https://docs.nvidia.com/datacenter/cloud-native/kubernetes/install-k8s.html) for instructions on how to install Kubernetes and enable GPU access in your Kubernetes cluster. You must also have Helm installed (see [Installing Helm](#installing-helm) for instructions). Note the following requirements: * To deploy Prometheus and Grafana to collect and display Triton metrics, your cluster must contain sufficient CPU resources to support these services. * To use GPUs for inferencing, your cluster must be configured to contain the desired number of GPU nodes, with support for the NVIDIA driver and CUDA version required by the version of the inference server you are using. * To enable autoscaling, your cluster's kube-apiserver must have the [aggregation layer enabled](https://kubernetes.io/docs/tasks/extend-kubernetes/configure-aggregation-layer/). This will allow the horizontal pod autoscaler to read custom metrics from the prometheus adapter. This Helm chart is available from [Triton Inference Server GitHub.](https://github.com/triton-inference-server/server) For more information on Helm and Helm charts, visit the [Helm documentation](https://helm.sh/docs/). ## Quickstart First, clone this repository to a local machine. Then, execute the following commands: Install helm ``` $ curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 $ chmod 700 get_helm.sh $ ./get_helm.sh ``` Deploy Prometheus and Grafana ``` $ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts $ helm repo update $ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack ``` Deploy Triton with default settings ``` helm install example ./deploy/k8s-onprem ``` ## Installing Helm ### Helm v3 If you do not already have Helm installed in your Kubernetes cluster, executing the following steps from the [official Helm install guide](https://helm.sh/docs/intro/install/) will give you a quick setup. If you are currently using Helm v2 and would like to migrate to Helm v3, see the [official migration guide](https://helm.sh/docs/topics/v2_v3_migration/). ## Model Repository If you already have a model repository, you may use that with this Helm chart. If you do not have a model repository, you can check out a local copy of the server source repository to create an example model repository: ``` $ git clone https://github.com/triton-inference-server/server.git ``` Triton Server needs a repository of models that it will make available for inferencing. For this example, we are using an existing NFS server and placing our model files there. See the [Model Repository documentation](../../docs/user_guide/model_repository.md) for other supported locations. Following the [QuickStart](../../docs/getting_started/quickstart.md), download the example model repository to your system and copy it onto your NFS server. Then, add the url or IP address of your NFS server and the server path of your model repository to `values.yaml`. ## Deploy Prometheus and Grafana The inference server metrics are collected by Prometheus and viewable through Grafana. The inference server Helm chart assumes that Prometheus and Grafana are available so this step must be followed even if you do not want to use Grafana. Use the [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) Helm chart to install these components. The *serviceMonitorSelectorNilUsesHelmValues* flag is needed so that Prometheus can find the inference server metrics in the *example* release deployed in a later section. ``` $ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts $ helm repo update $ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack ``` Then port-forward to the Grafana service so you can access it from your local browser. ``` $ kubectl port-forward service/example-metrics-grafana 8080:80 ``` Now you should be able to navigate in your browser to localhost:8080 and see the Grafana login page. Use username=admin and password=prom-operator to log in. An example Grafana dashboard is available in dashboard.json. Use the import function in Grafana to import and view this dashboard. ## Enable Autoscaling To enable autoscaling, ensure that autoscaling tag in `values.yaml`is set to `true`. This will do two things: 1. Deploy a Horizontal Pod Autoscaler that will scale replicas of the triton-inference-server based on the information included in `values.yaml`. 2. Install the [prometheus-adapter](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-adapter) helm chart, allowing the Horizontal Pod Autoscaler to scale based on custom metrics from prometheus. The included configuration will scale Triton pods based on the average queue time, as described in [this blog post](https://developer.nvidia.com/blog/deploying-nvidia-triton-at-scale-with-mig-and-kubernetes/#:~:text=Query%20NVIDIA%20Triton%20metrics%20using%20Prometheus). To customize this, you may replace or add to the list of custom rules in `values.yaml`. If you change the custom metric, be sure to change the values in autoscaling.metrics. If autoscaling is disabled, the number of Triton server pods is set to the minReplicas variable in `values.yaml`. ## Enable Load Balancing To enable load balancing, ensure that the loadBalancing tag in `values.yaml` is set to `true`. This will do two things: 1. Deploy a Traefik reverse proxy through the [Traefik Helm Chart](https://github.com/traefik/traefik-helm-chart). 2. Configure two Traefik [IngressRoutes](https://doc.traefik.io/traefik/providers/kubernetes-crd/), one for http and one for grpc. This will allow the Traefik service to expose two ports that will be forwarded to and balanced across the Triton pods. To choose the port numbers exposed, or to disable either http or grpc, edit the configured variables in `values.yaml`. ## Deploy the Inference Server Deploy the inference server, autoscaler, and load balancer using the default configuration with the following commands. Here, and in the following commands we use the name `example` for our chart. This name will be added to the beginning of all resources created during the helm installation. ``` $ cd $ helm install example . ``` Use kubectl to see status and wait until the inference server pods are running. ``` $ kubectl get pods NAME READY STATUS RESTARTS AGE example-triton-inference-server-5f74b55885-n6lt7 1/1 Running 0 2m21s ``` There are several ways of overriding the default configuration as described in this [Helm documentation](https://helm.sh/docs/using_helm/#customizing-the-chart-before-installing). You can edit the values.yaml file directly or you can use the *--set* option to override a single parameter with the CLI. For example, to deploy a cluster with a minimum of two inference servers use *--set* to set the autoscaler.minReplicas parameter. ``` $ helm install example --set autoscaler.minReplicas=2 . ``` You can also write your own "config.yaml" file with the values you want to override and pass it to Helm. If you specify a "config.yaml" file, the values set will override those in values.yaml. ``` $ cat << EOF > config.yaml namespace: MyCustomNamespace image: imageName: nvcr.io/nvidia/tritonserver:custom-tag modelRepositoryPath: gs://my_model_repository EOF $ helm install example -f config.yaml . ``` ## Deploying the Inference Server on OpenShift or OKD Because of the default security posture of OpenShift and OKD, the configuration of which uses OpenShift-specific APIs, the chart needs special consideration when targeting those environments. Any of the above discussed customizations and prerequisites hold for an OpenShift environment, except that you do not need to install Prometheus and Grafana and can instead enable monitoring for user-defined projects by following [the OpenShift documentation on the topic](https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/monitoring/enabling-monitoring-for-user-defined-projects). To deploy the configurations to enable NFS mounts and the non-root UIDs used in the Triton deployment, a tag can be enabled alongside any other configurations discussed above. In the simplest case, to use `--set` on the command line, you can simply update the tags.openshift parameter. ``` $ cd $ helm install example --set tags.openshift=true . ``` ## Probe Configuration In `templates/deployment.yaml` is configurations for `livenessProbe`, `readinessProbe` and `startupProbe` for the Triton server container. By default, Triton loads all the models before starting the HTTP server to respond to the probes. The process can take several minutes, depending on the models sizes. If it is not completed in `startupProbe.failureThreshold * startupProbe.periodSeconds` seconds then Kubernetes considers this as a pod failure and restarts it, ending up with an infinite loop of restarting pods, so make sure to sufficiently set these values for your use case. The liveliness and readiness probes are being sent only after the first success of a startup probe. For more details, see the [Kubernetes probe documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) and the [feature page of the startup probe](https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/950-liveness-probe-holdoff/README.md). ## Using Triton Inference Server Now that the inference server is running you can send HTTP or GRPC requests to it to perform inferencing. By default, this chart deploys [Traefik](https://traefik.io/) and uses [IngressRoutes](https://doc.traefik.io/traefik/providers/kubernetes-crd/) to balance requests across all available nodes. To send requests through the Traefik proxy, use the Cluster IP of the traefik service deployed by the Helm chart. In this case, it is 10.111.128.124. ``` $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ... example-traefik LoadBalancer 10.111.128.124 8001:31752/TCP,8000:31941/TCP,80:30692/TCP,443:30303/TCP 74m example-triton-inference-server ClusterIP None 8000/TCP,8001/TCP,8002/TCP 74m ``` Use the following command to refer to the Cluster IP: ``` cluster_ip=`kubectl get svc -l app.kubernetes.io/name=traefik -o=jsonpath='{.items[0].spec.clusterIP}'` ``` The Traefik reverse-proxy exposes an HTTP endpoint on port 8000, and GRPC endpoint on port 8001 and a Prometheus metrics endpoint on port 8002. You can use curl to get the meta-data of the inference server from the HTTP endpoint. ``` $ curl $cluster_ip:8000/v2 ``` Follow the [QuickStart](../../docs/getting_started/quickstart.md) to get the example image classification client that can be used to perform inferencing using image classification models on the inference server. For example, ``` $ image_client -u $cluster_ip:8000 -m inception_v3_onnx -s INCEPTION -c3 mug.jpg Request 0, batch size 1 Image 'images/mug.jpg': 504 (COFFEE MUG) = 0.723992 968 (CUP) = 0.270953 967 (ESPRESSO) = 0.00115997 ``` ## Testing Load Balancing and Autoscaling After you have confirmed that your Triton cluster is operational and can perform inference, you can test the load balancing and autoscaling features by sending a heavy load of requests. One option for doing this is using the [perf_analyzer](https://github.com/triton-inference-server/perf_analyzer/blob/main/README.md) application. You can apply a progressively increasing load with a command like: ``` perf_analyzer -m simple -u $cluster_ip:8000 --concurrency-range 1:10 ``` From your Grafana dashboard, you should be able to see the number of pods increase as the load increases, with requests being routed evenly to the new pods. ## Cleanup After you have finished using the inference server, you should use Helm to delete the deployment. ``` $ helm list NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE example 1 Wed Feb 27 22:16:55 2019 DEPLOYED triton-inference-server-1.0.0 1.0 default example-metrics 1 Tue Jan 21 12:24:07 2020 DEPLOYED prometheus-operator-6.18.0 0.32.0 default $ helm uninstall example $ helm uninstall example-metrics ``` For the Prometheus and Grafana services, you should [explicitly delete CRDs](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#uninstall-helm-chart): ``` $ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com alertmanagers.monitoring.coreos.com podmonitors.monitoring.coreos.com probes.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com thanosrulers.monitoring.coreos.com ``` --- ## File: deploy/gke-marketplace-app/benchmark/README.md # Benchmarking with NVIDIA Triton Inference Server GKE Marketplace Application **Table Of Contents** - [Models](#models) - [Performance](#performance) ## Models First, we collect a set of TensorFlow and TensorRT models to compare: - Get [Distill Bert fine-tuned with Squad Q&A task](https://huggingface.co/distilbert-base-cased-distilled-squad/tree/main) from Huggingface. `wget https://huggingface.co/distilbert-base-cased-distilled-squad/blob/main/saved_model.tar.gz` - Get [Bert base fine-tuned with Squad Q&A task](https://huggingface.co/deepset/bert-base-cased-squad2/tree/main) from Huggingface `wget https://huggingface.co/deepset/bert-base-cased-squad2/blob/main/saved_model.tar.gz` - Follow [TensorRT Demo Bert](https://github.com/NVIDIA/TensorRT/tree/master/demo/BERT) to convert BERT base model to TensorRT Engine, choose sequence length of 384 to match previous 2 TensorFlow models. Last step, we choose to create TensorRT engine with 2 optimization profile, profile 0 for batch size 1 and profile 1 for batch size 4 run: `python3 builder.py -m models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_384_v19.03.1/model.ckpt -o engines/model.plan -b 8 -s 384 --fp16 --int8 --strict -c models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_384_v19.03.1 --squad-json ./squad/train-v2.0.json -v models/fine-tuned/bert_tf_ckpt_base_qa_squad2_amp_384_v19.03.1/vocab.txt --calib-num 100 -iln -imh`. This needs to be ran on the inference GPU respectively (Engine optimized with A100 cannot be used for inference on T4). We the place the model into a GCS with following structure, `config.pbtxt` was provided. ``` ├── bert_base_trt_gpu │ ├── 1 │ │ └── model.plan │ └── config.pbtxt ├── bert_base_trt_gpu_seqlen128 │ ├── 1 │ │ └── model.plan │ └── config.pbtxt ├── bert_base_tf_gpu │ ├── 1 │ │ └── model.savedmodel │ └── config.pbtxt ├── bert_base_tf_cpu │ ├── 1 │ │ └── model.savedmodel │ └── config.pbtxt ├── bert_distill_tf_gpu │ ├── 1 │ │ └── model.savedmodel │ └── config.pbtxt └── bert_distill_tf_cpu ├── 1 │ └── model.savedmodel └── config.pbtxt ``` When deploy Triton GKE application, point the model repository to directory contains the structure above with actual models. ## Performance We use perf analyzer of Triton to benchmark the performance of each model, the perf analyzer reside in another pod of the GKE cluster. ```bash export INGRESS_HOST=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}') export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].port}') bash perf_query.sh 35.194.5.119:80 bert_base_trt_gpu 384 ``` We deploy model on n1-standard-96 for CPU BERT BASE and Distill BERT and (n1-standard-4 + T4) for GPU BERT models, the sequence length of the BERT model is 384 token, and measure the latency/throughput with a concurrency sweep with Triton's performance analyzer. The latency includes Istio ingress/load balancing and reflect the true round trip cost in the same GCP zone. For all the model with sequence length of 384: CPU BERT BASE: latency: 700ms, throughput: 12 qps CPU Distill BERT: latency: 369ms, throughput: 24 qps GPU BERT BASE: latency: 230ms, throughput: 34.7 qps GPU Distill BERT: latency: 118ms, throughput: 73.3 qps GPU TensorRT BERT BASE: latency: 50ms, throughput: 465 qps With n1-standard-96 priced at $4.56/hr and n1-standard-4 at $0.19/hr and T4 at $0.35/hr totaling $0.54/hr. While achieving a much lower latency, the TCO of BERT inference with TensorRT on T4 is over 163 times that of Distill BERT inference on n1-standard-96. --- ## File: deploy/gke-marketplace-app/trt-engine/README.md # Instruction to create BERT engine for each Triton update ## Description ``` docker run --gpus all -it --network host \ --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \ -v ~:/scripts nvcr.io/nvidia/tensorrt:26.07-py3 pip install onnx six torch tf2onnx tensorflow git clone -b main https://github.com/NVIDIA/TensorRT.git cd TensorRT git submodule update --init --recursive export TRT_OSSPATH=/workspace/TensorRT export TRT_LIBPATH=/lib/x86_64-linux-gnu pushd /usr/local/bin && wget https://ngc.nvidia.com/downloads/ngccli_cat_linux.zip && unzip ngccli_cat_linux.zip && chmod u+x ngc-cli/ngc && rm ngccli_cat_linux.zip ngc-cli.md5 && ln -s ngc-cli/ngc ngc && echo "no-apikey\nascii\n" | ngc config set popd cd /workspace/TensorRT/demo/BERT bash ./scripts/download_squad.sh bash ./scripts/download_model.sh large 128 # bash ./scripts/download_model.sh large 384 mkdir -p engines python3 builder.py -m models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/model.ckpt -o engines/bert_large_int8_bs1_s128.engine -b 1 -s 128 -c models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/ -v models/fine-tuned/bert_tf_ckpt_large_qa_squad2_amp_128_v19.03.1/vocab.txt --int8 --fp16 --strict --calib-num 1 -iln -imh gsutil cp bert_large_int8_bs1_s128.engine gs://triton_sample_models/26.07/bert/1/model.plan ``` For each Triton upgrade, container version used to generate the model, and the model path in GCS `gs://triton_sample_models/26.07/` should be updated accordingly with the correct version. --- ## File: deploy/gke-marketplace-app/README.md # NVIDIA Triton Inference Server GKE Marketplace Application **Table Of Contents** - [NVIDIA Triton Inference Server GKE Marketplace Application](#nvidia-triton-inference-server-gke-marketplace-application) - [Description](#description) - [Prerequisites](#prerequisites) - [Demo Instruction](#demo-instruction) - [Additional Resources](#additional-resources) - [Known Issues](#known-issues) ## Description This repository contains Google Kubernetes Engine(GKE) Marketplace Application for NVIDIA Triton Inference Server deployer. - Triton GKE deployer is a helm chart deployer recommended by GKE Marketplace - Triton GKE deployer deploys a GKE ingress which accepts public inference requests - Triton GKE deployer includes a horizontal pod autoscaler(HPA) which relies on [stack driver custom metrics adaptor](https://github.com/GoogleCloudPlatform/k8s-stackdriver/tree/master/custom-metrics-stackdriver-adapter) to monitor GPU duty cycle, and auto scale GPU nodes. - This repo also contains a sample to generate BERT model with TensorRT and use Locust to experiment with GPU node autoscaling and monitor client latency/throughput. ## Prerequisites - [Install Google Cloud SDK on your laptop/client workstation](https://cloud.google.com/sdk/docs/install), so that `gcloud` SDK cli interface could be run on the client and sign in with your GCP credentials. - In addition, user could leverage [Google Cloud shell](https://cloud.google.com/shell/docs/launching-cloud-shell). ## Demo Instruction First, install this Triton GKE app to an existing GKE cluster with GPU node pool, Google Cloud Marketplace currently doesn't support auto creation of GPU clusters. User has to run following command to create a compatible cluster (gke version >=1.18.7) with GPU node pools, we recommend user to select T4 or A100(MIG) instances type and choose CPU ratio based on profiling of actual inference workflow. Users need to follow these [instructions](https://cloud.google.com/kubernetes-engine/docs/how-to/kubernetes-service-accounts#creating_a_kubernetes_service_account) to create a kubernetes service account. In this example, we use `gke-test@k80-exploration.iam.gserviceaccount.com`. Make sure it has access to artifact registry and monitoring viewer. For example, to grant access to custom metrics which is required for HPA to work: ``` gcloud iam service-accounts add-iam-policy-binding --role \ roles/iam.workloadIdentityUser --member \ "serviceAccount:.svc.id.goog[custom-metrics/custom-metrics-stackdriver-adapter]" \ @.iam.gserviceaccount.com kubectl annotate serviceaccount --namespace custom-metrics \ custom-metrics-stackdriver-adapter \ iam.gke.io/gcp-service-account=@.iam.gserviceaccount.com ``` Currently, GKE >= 1.18.7 only supported in GKE rapid channel, to find the latest version, please visit [GKE release notes](https://cloud.google.com/kubernetes-engine/docs/release-notes). ``` /* Detailed source-code truncated for AI context efficiency. */ ``` Creating a cluster and adding GPU nodes could take up-to 10 minutes. Please be patient after executing this command. GPU resources in GCP could be fully utilized, so please try a different zone in case compute resource cannot be allocated. After GKE cluster is running, run `kubectl get pods --all-namespaces` to make sure the client can access the cluster correctly: If user would like to experiment with A100 MIG partitioned GPU in GKE, please create node pool with following command: ``` gcloud beta container node-pools create accel \ --project ${PROJECT_ID} \ --zone ${ZONE} \ --cluster ${DEPLOYMENT_NAME} \ --service-account=${SERVICE_ACCOUNT} \ --num-nodes 1 \ --accelerator type=nvidia-tesla-a100,count=1,gpu-partition-size=1g.5gb \ --enable-autoscaling --min-nodes 1 --max-nodes 2 \ --machine-type=a2-highgpu-1g \ --disk-size=100 \ --scopes cloud-platform \ --verbosity error ``` Please note that A100 MIG in GKE does not support GPU metrics yet, also Triton GPU Metrics is not compatible with A100 MIG. Hence, please disable GPU metrics by unselect allowGPUMetrics while deploy Triton GKE app. Also for the same reason, this deployer doesn't support inference workfload auto-scaling on A100 MIG as well. Second, go to this [GKE Marketplace link](https://console.cloud.google.com/marketplace/details/nvidia-ngc-public/triton-inference-server) to deploy Triton application. Users can leave everything as default if their models have already been tested/validated with Triton. They can provide a GCS path pointing to the model repository containing their models. By default, we provide a BERT large model optimized by TensorRT in a public demo GCS bucket that is compatible with the `xx.yy` release of Triton Server in `gs://triton_sample_models/xx_yy`. However, please take note of the following about this demo bucket: - The TensorRT engine provided in the demo bucket is only compatible with Tesla T4 GPUs. - This bucket is located in `us-central1`, so loading from this bucket into Triton in other regions may be affected. - The first deployment of this Triton GKE application will be slower than consecutive runs because the image needs to be pulled into the GKE cluster. - You can find an example of how this model is generated and uploaded [here](trt-engine/README.md). Where is the version of NGC Triton container needed. We want to discuss HPA autoscaling metrics users can leverage. GPU Power(Percentage of Power) tends to be a reliable metric, especially for larger GPU like V100 and A100. GKE currently natively support GPU duty cycle which is GPU utilization in `nvidia-smi`. We ask users always profile their model to determine the autoscaling target and metrics. When attempting to select the right metrics for autoscaling, the goal should be to pick metrics based on the following: 1, meet SLA rrequirement. 2, give consideration to transient request load, 3, keep GPU as fully utilized as possible. Profiling comes in 2 aspects: If user decided to use Duty Cycle or other GPU metric, it is recommend establish baseline to link SLA requirement such as latency with GPU metrics, for example, for model A, latency will be below 10ms 99% of time when Duty Cycle is below 80% utilized. Additionally, profiling also provide insight to model optimization for inference, with tools like [Nsight](https://developer.nvidia.com/nsight-systems). Once the application is deployed successfully, get the public ip from ingress: ``` > kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE triton-external * 35.186.215.182 80 107s ``` Third, we will try sending request to server with provide client example. If User selected deploy Triton to accept HTTP request, please launch [Locust](https://docs.locust.io/en/stable/installation.html) with Ingress host and port to query Triton Inference Server. In this [example script](https://github.com/triton-inference-server/server/tree/master/deploy/gke-marketplace-app/client-sample/locustfile_bert.py), we send request to Triton server which has loaded a BERT large TensorRT Engine with Sequence length of 128 into GCP bucket. We simulate 1000 concurrent user as target and spawn user at rate of 50 users per second. ``` locust -f locustfile_bert.py -H http://${INGRESS_HOST}:${INGRESS_PORT} ``` The client example push about ~650 QPS(Query per second) to Triton Server, and will trigger a auto scale of T4 GPU nodes (We recommend to use T4 and A100[MIG] for inference). From locust UI, we will observer a drop of latency mean and variance for the requests. At the end, after autoscaling, we see the latency stablized at ~200 ms, end to end from US client to europe server, which is excellent for a model that has 345 million parameters. Since for each node, we use 1T4 + n1-standard-4 instance, and it can handle ~450 QPS, with on-demand price, it is ($0.35+$0.19)=$0.54/hr, that translate to 3 million inference per dollar for BERT large model at batch size 1. Further more, with 3 year commitment price, hr rate is ($0.16+$0.08)=$0.24/hr, that translate to 6.75 million inference per dollar. Alternatively, user can opt to use [Perf Analyzer](https://github.com/triton-inference-server/perf_analyzer/blob/main/README.md) to profile and study the performance of Triton Inference Server. Here we also provide a [client script](https://github.com/triton-inference-server/server/tree/master/deploy/gke-marketplace-app/client-sample/perf_analyzer_grpc.sh) to use Perf Analyzer to send gRPC to Triton Server GKE deployment. Perf Analyzer client requires user to use NGC Triton Client Container. ``` bash perf_analyzer_grpc.sh ${INGRESS_HOST}:${INGRESS_PORT} ``` ## Additional Resources See the following resources to learn more about NVIDIA Triton Inference Server and GKE GPU capabilities. **Documentation** - [GPU in Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/docs/how-to/gpus) - [Optimize GPU Performance in Google Cloud Platform](https://cloud.google.com/compute/docs/gpus/optimize-gpus) - [Triton Inference Server](https://github.com/triton-inference-server/server) - [AI Platform Prediction: Custom container concepts with Triton Server](https://cloud.google.com/solutions/ai-platform-prediction-custom-container-concepts) by [Kevin Tsai](https://github.com/merlin1649) - [AI Platform Prediction: Direct model server setup for NVIDIA Triton Inference Server](https://cloud.google.com/solutions/ai-platform-prediction-direct-model-server-nvidia) by [Kevin Tsai](https://github.com/merlin1649) ## Known Issues - GKE one click cluster creation doesn't support GPU node pools at the moment, users have to manually create a compatible (>=1.18.7) cluster and attach node pool (T4 and A100 MIG recommended) - When Horizontal Pod Autoscaler(HPA) expand and all GPU node pool already utilized, GKE will request new GPU node and it can take between 4-7 minutes, it could be a long wait plus GPU driver install and image pulling. We recommend user to leverage multi-tier model serving and Triton's priority feature to create cushion for latency critical models, and allocate active standby GPU node for spike of requests. --- ## File: deploy/gcp/README.md [](https://opensource.org/licenses/BSD-3-Clause) # Kubernetes Deploy: Triton Inference Server Cluster A helm chart for installing a single cluster of Triton Inference Server is provided. By default the cluster contains a single instance of the inference server but the *replicaCount* configuration parameter can be set to create a cluster of any size, as described below. This guide assumes you already have a functional Kubernetes cluster and helm installed (see below for instructions on installing helm). Note the following requirements: * The helm chart deploys Prometheus and Grafana to collect and display Triton metrics. Your cluster must contain sufficient CPU resources to support these services. At a minimum you will likely require 2 CPU nodes with machine type of n1-standard-2 or greater. * If you want Triton Server to use GPUs for inferencing, your cluster must be configured to contain the desired number of GPU nodes with support for the NVIDIA driver and CUDA version required by the version of the inference server you are using. This helm chart is available from [Triton Inference Server GitHub](https://github.com/triton-inference-server/server) or from the [NVIDIA GPU Cloud (NGC)](https://ngc.nvidia.com). The steps below describe how to set-up a model repository, use helm to launch the inference server, and then send inference requests to the running server. You can access a Grafana endpoint to see real-time metrics reported by the inference server. ## Installing Helm ### Helm v3 If you do not already have Helm installed in your Kubernetes cluster, executing the following steps from the [official helm install guide](https://helm.sh/docs/intro/install/) will give you a quick setup. If you're currently using Helm v2 and would like to migrate to Helm v3, please see the [official migration guide](https://helm.sh/docs/topics/v2_v3_migration/). ### Helm v2 > **NOTE**: Moving forward this chart will only be tested and maintained for Helm v3. Below are example instructions for installing Helm v2. ``` $ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash $ kubectl create serviceaccount -n kube-system tiller serviceaccount/tiller created $ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller $ helm init --service-account tiller --wait ``` If you run into any issues, you can refer to the official installation guide [here](https://v2.helm.sh/docs/install/). ## Model Repository If you already have a model repository you may use that with this helm chart. If you do not have a model repository, you can checkout a local copy of the inference server source repository to create an example model repository:: ``` $ git clone https://github.com/triton-inference-server/server.git ``` Triton Server needs a repository of models that it will make available for inferencing. For this example you will place the model repository in a Google Cloud Storage bucket. ``` $ gsutil mb gs://triton-inference-server-repository ``` Following the [QuickStart](../../docs/getting_started/quickstart.md) download the example model repository to your system and copy it into the GCS bucket. ``` $ gsutil cp -r docs/examples/model_repository gs://triton-inference-server-repository/model_repository ``` ### GCS Permissions Make sure the bucket permissions are set so that the inference server can access the model repository. If the bucket is public then no additional changes are needed and you can proceed to "Deploy Prometheus and Grafana" section. If bucket premissions need to be set with the GOOGLE_APPLICATION_CREDENTIALS environment variable then perform the following steps: * Generate Google service account JSON with proper permissions called *gcp-creds.json*. * Create a Kubernetes secret from *gcp-creds.json*: ``` $ kubectl create configmap gcpcreds --from-literal "project-id=myproject" $ kubectl create secret generic gcpcreds --from-file gcp-creds.json ``` * Modify templates/deployment.yaml to include the GOOGLE_APPLICATION_CREDENTIALS environment variable: ``` env: - name: GOOGLE_APPLICATION_CREDENTIALS value: /secret/gcp-creds.json ``` * Modify templates/deployment.yaml to mount the secret in a volume at /secret: ``` volumeMounts: - name: vsecret mountPath: "/secret" readOnly: true ... volumes: - name: vsecret secret: secretName: gcpcreds ``` ## Deploy Prometheus and Grafana The inference server metrics are collected by Prometheus and viewable by Grafana. The inference server helm chart assumes that Prometheus and Grafana are available so this step must be followed even if you don't want to use Grafana. Use the [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) to install these components. The *serviceMonitorSelectorNilUsesHelmValues* flag is needed so that Prometheus can find the inference server metrics in the *example* release deployed below. ``` $ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack ``` Then port-forward to the Grafana service so you can access it from your local browser. ``` $ kubectl port-forward service/example-metrics-grafana 8080:80 ``` Now you should be able to navigate in your browser to localhost:8080 and see the Grafana login page. Use username=admin and password=prom-operator to login. An example Grafana dashboard is available in dashboard.json. Use the import function in Grafana to import and view this dashboard. ## Deploy the Inference Server Deploy the inference server using the default configuration with the following commands. ``` $ cd $ helm install example . ``` Use kubectl to see status and wait until the inference server pods are running. ``` $ kubectl get pods NAME READY STATUS RESTARTS AGE example-triton-inference-server-5f74b55885-n6lt7 1/1 Running 0 2m21s ``` There are several ways of overriding the default configuration as described in this [helm documentation](https://helm.sh/docs/using_helm/#customizing-the-chart-before-installing). You can edit the values.yaml file directly or you can use the *--set* option to override a single parameter with the CLI. For example, to deploy a cluster of four inference servers use *--set* to set the replicaCount parameter. ``` $ helm install example --set replicaCount=4 . ``` You can also write your own "config.yaml" file with the values you want to override and pass it to helm. ``` $ cat << EOF > config.yaml namespace: MyCustomNamespace image: imageName: nvcr.io/nvidia/tritonserver:custom-tag modelRepositoryPath: gs://my_model_repository EOF $ helm install example -f config.yaml . ``` ## Using Triton Inference Server Now that the inference server is running you can send HTTP or GRPC requests to it to perform inferencing. By default, the inferencing service is exposed with a LoadBalancer service type. Use the following to find the external IP for the inference server. In this case it is 34.83.9.133. ``` $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ... example-triton-inference-server LoadBalancer 10.18.13.28 34.83.9.133 8000:30249/TCP,8001:30068/TCP,8002:32723/TCP 47m ``` The inference server exposes an HTTP endpoint on port 8000, and GRPC endpoint on port 8001 and a Prometheus metrics endpoint on port 8002. You can use curl to get the meta-data of the inference server from the HTTP endpoint. ``` $ curl 34.83.9.133:8000/v2 ``` Follow the [QuickStart](../../docs/getting_started/quickstart.md) to get the example image classification client that can be used to perform inferencing using image classification models being served by the inference server. For example, ``` $ image_client -u 34.83.9.133:8000 -m inception_v3_onnx -s INCEPTION -c3 mug.jpg Request 0, batch size 1 Image 'images/mug.jpg': 504 (COFFEE MUG) = 0.723992 968 (CUP) = 0.270953 967 (ESPRESSO) = 0.00115997 ``` ## Cleanup Once you've finished using the inference server you should use helm to delete the deployment. ``` $ helm list NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE example 1 Wed Feb 27 22:16:55 2019 DEPLOYED triton-inference-server-1.0.0 1.0 default example-metrics 1 Tue Jan 21 12:24:07 2020 DEPLOYED prometheus-operator-6.18.0 0.32.0 default $ helm uninstall example $ helm uninstall example-metrics ``` For the Prometheus and Grafana services, you should [explicitly delete CRDs](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#uninstall-helm-chart): ``` $ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com alertmanagers.monitoring.coreos.com podmonitors.monitoring.coreos.com probes.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com thanosrulers.monitoring.coreos.com ``` You may also want to delete the GCS bucket you created to hold the model repository. ``` $ gsutil rm -r gs://triton-inference-server-repository ``` --- ## File: deploy/fleetcommand/README.md [](https://opensource.org/licenses/BSD-3-Clause) # Fleet Command Deploy: NVIDIA Triton Inference Server A helm chart for installing a single cluster of NVIDIA Triton Inference Server on Fleet Command is provided. By default the cluster contains a single instance of the Triton but the *replicaCount* configuration parameter can be set to create a cluster of any size, as described below. This guide assumes you already have a functional Fleet Command location deployed. Please refer to the [Fleet Command Documentation](https://docs.nvidia.com/fleet-command/prod_fleet-command/prod_fleet-command/overview.html) The steps below describe how to set-up a model repository, use helm to launch the Triton, and then send inference requests to the running Triton Inference Server. You can optionally scrape metrics with Prometheus and access a Grafana endpoint to see real-time metrics reported by Triton. ## Model Repository If you already have a model repository you may use that with this helm chart. If you do not have a model repository, you can checkout a local copy of the Triton Inference Server source repository to create an example model repository:: ``` $ git clone https://github.com/triton-inference-server/server.git ``` Triton needs a repository of models that it will make available for inferencing. For this example you will place the model repository in an S3 Storage bucket (either in AWS or other S3 API compatible on-premises object storage). ``` $ aws s3 mb s3://triton-inference-server-repository ``` Following the [QuickStart](../../docs/getting_started/quickstart.md) download the example model repository to your system and copy it into the AWS S3 bucket. ``` $ aws s3 cp -r docs/examples/model_repository s3://triton-inference-server-repository/model_repository ``` ### AWS Model Repository To load the model from the AWS S3, you need to convert the following AWS credentials in the base64 format and add it to the Application Configuration section when creating the Fleet Command Deployment. ``` echo -n 'REGION' | base64 echo -n 'SECRECT_KEY_ID' | base64 echo -n 'SECRET_ACCESS_KEY' | base64 # Optional for using session token echo -n 'AWS_SESSION_TOKEN' | base64 ``` ## Deploy the Triton Inference Server Deploy the Triton Inference Server to your Location in Fleet Command by creating a Deployment. You can specify configuration parameters to override the default [values.yaml](values.yaml) in the Application Configuration section. *Note:* You _must_ provide a `--model-repository` parameter with a path to your prepared model repository in your S3 bucket. Otherwise, the Triton will not start. An example Application Configuration for Triton on Fleet Command: ```yaml image: serverArgs: - --model-repository=s3://triton-inference-server-repository secret: region: id: key: token: ``` See [Fleet Command documentation](https://docs.nvidia.com/fleet-command/prod_fleet-command/prod_fleet-command/ug-deploying-to-the-edge.html) for more info. ### Prometheus ServiceMonitor Support If you have `prometheus-operator` deployed, you can enable the ServiceMonitor for the Triton Inference Server by setting `serviceMonitor.enabled: true` in Application Configuration. This will also deploy a Grafana dashboard for Triton as a ConfigMap. Otherwise, metrics can be scraped by pointing an external Prometheus instance at the `metricsNodePort` in the values. ## Using Triton Inference Server Now that the Triton Inference Server is running you can send HTTP or GRPC requests to it to perform inferencing. By default, the service is exposed with a NodePort service type, where the same port is opened on all systems in a Location. Triton exposes an HTTP endpoint on port 30343, and GRPC endpoint on port 30344 and a Prometheus metrics endpoint on port 30345. These ports can be overridden in the application configuration when deploying. You can use curl to get the meta-data of Triton from the HTTP endpoint. For example, if a system in your location has the IP `34.83.9.133`: ``` $ curl 34.83.9.133:30343/v2 ``` Follow the [QuickStart](../../docs/getting_started/quickstart.md) to get the example image classification client that can be used to perform inferencing using image classification models being served by the Triton. For example, ``` $ image_client -u 34.83.9.133:30343 -m densenet_onnx -s INCEPTION -c 3 mug.jpg Request 0, batch size 1 Image '/workspace/images/mug.jpg': 15.349568 (504) = COFFEE MUG 13.227468 (968) = CUP 10.424893 (505) = COFFEEPOT ``` --- ## File: deploy/aws/README.md [](https://opensource.org/licenses/BSD-3-Clause) # Kubernetes Deploy: Triton Inference Server Cluster A helm chart for installing a single cluster of Triton Inference Server is provided. By default the cluster contains a single instance of the inference server but the *replicaCount* configuration parameter can be set to create a cluster of any size, as described below. This guide assumes you already have a functional Kubernetes cluster and helm installed (see below for instructions on installing helm). Note the following requirements: * The helm chart deploys Prometheus and Grafana to collect and display Triton metrics. To use this helm chart you must install Prpmetheus and Grafana in your cluster as described below and your cluster must contain sufficient CPU resources to support these services. * If you want Triton Server to use GPUs for inferencing, your cluster must be configured to contain the desired number of GPU nodes (EC2 G4 instances recommended) with support for the NVIDIA driver and CUDA version required by the version of the inference server you are using. The steps below describe how to set-up a model repository, use helm to launch the inference server, and then send inference requests to the running server. You can access a Grafana endpoint to see real-time metrics reported by the inference server. ## Installing Helm ### Helm v3 If you do not already have Helm installed in your Kubernetes cluster, executing the following steps from the [official helm install guide](https://helm.sh/docs/intro/install/) will give you a quick setup. If you're currently using Helm v2 and would like to migrate to Helm v3, please see the [official migration guide](https://helm.sh/docs/topics/v2_v3_migration/). ### Helm v2 > **NOTE**: Moving forward this chart will only be tested and maintained for Helm v3. Below are example instructions for installing Helm v2. ``` $ curl https://raw.githubusercontent.com/helm/helm/master/scripts/get | bash $ kubectl create serviceaccount -n kube-system tiller serviceaccount/tiller created $ kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller $ helm init --service-account tiller --wait ``` If you run into any issues, you can refer to the official installation guide [here](https://v2.helm.sh/docs/install/). ## Model Repository If you already have a model repository you may use that with this helm chart. If you do not have a model repository, you can checkout a local copy of the inference server source repository to create an example model repository:: ``` $ git clone https://github.com/triton-inference-server/server.git ``` Triton Server needs a repository of models that it will make available for inferencing. For this example you will place the model repository in an AWS S3 Storage bucket. ``` $ aws s3 mb s3://triton-inference-server-repository ``` Following the [QuickStart](../../docs/getting_started/quickstart.md) download the example model repository to your system and copy it into the AWS S3 bucket. ``` $ aws s3 cp --recursive docs/examples/model_repository s3://triton-inference-server-repository/model_repository ``` ### AWS Model Repository To load the model from the AWS S3, you need to convert the following AWS credentials in the base64 format and add it to the values.yaml ``` echo -n 'REGION' | base64 ``` ``` echo -n 'SECRECT_KEY_ID' | base64 ``` ``` echo -n 'SECRET_ACCESS_KEY' | base64 ``` ## Deploy Prometheus and Grafana The inference server metrics are collected by Prometheus and viewable by Grafana. The inference server helm chart assumes that Prometheus and Grafana are available so this step must be followed even if you don't want to use Grafana. Use the [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) to install these components. The *serviceMonitorSelectorNilUsesHelmValues* flag is needed so that Prometheus can find the inference server metrics in the *example* release deployed below. ``` $ helm install example-metrics --set prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false prometheus-community/kube-prometheus-stack ``` Then port-forward to the Grafana service so you can access it from your local browser. ``` $ kubectl port-forward service/example-metrics-grafana 8080:80 ``` Now you should be able to navigate in your browser to localhost:8080 and see the Grafana login page. Use username=admin and password=prom-operator to login. An example Grafana dashboard is available in dashboard.json. Use the import function in Grafana to import and view this dashboard. ## Deploy the Inference Server Deploy the inference server using the default configuration with the following commands. ``` $ cd $ helm install example . ``` Use kubectl to see status and wait until the inference server pods are running. ``` $ kubectl get pods NAME READY STATUS RESTARTS AGE example-triton-inference-server-5f74b55885-n6lt7 1/1 Running 0 2m21s ``` There are several ways of overriding the default configuration as described in this [helm documentation](https://helm.sh/docs/using_helm/#customizing-the-chart-before-installing). You can edit the values.yaml file directly or you can use the *--set* option to override a single parameter with the CLI. For example, to deploy a cluster of four inference servers use *--set* to set the replicaCount parameter. ``` $ helm install example --set replicaCount=4 . ``` You can also write your own "config.yaml" file with the values you want to override and pass it to helm. ``` $ cat << EOF > config.yaml namespace: MyCustomNamespace image: imageName: nvcr.io/nvidia/tritonserver:custom-tag modelRepositoryPath: gs://my_model_repository EOF $ helm install example -f config.yaml . ``` ## Using Triton Inference Server Now that the inference server is running you can send HTTP or GRPC requests to it to perform inferencing. By default, the inferencing service is exposed with a LoadBalancer service type. Use the following to find the external IP for the inference server. In this case it is 34.83.9.133. ``` $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ... example-triton-inference-server LoadBalancer 10.18.13.28 34.83.9.133 8000:30249/TCP,8001:30068/TCP,8002:32723/TCP 47m ``` The inference server exposes an HTTP endpoint on port 8000, and GRPC endpoint on port 8001 and a Prometheus metrics endpoint on port 8002. You can use curl to get the meta-data of the inference server from the HTTP endpoint. ``` $ curl 34.83.9.133:8000/v2 ``` Follow the [QuickStart](../../docs/getting_started/quickstart.md) to get the example image classification client that can be used to perform inferencing using image classification models being served by the inference server. For example, ``` $ image_client -u 34.83.9.133:8000 -m inception_v3_onnx -s INCEPTION -c3 mug.jpg Request 0, batch size 1 Image 'images/mug.jpg': 504 (COFFEE MUG) = 0.723992 968 (CUP) = 0.270953 967 (ESPRESSO) = 0.00115997 ``` ## Cleanup Once you've finished using the inference server you should use helm to delete the deployment. ``` $ helm list NAME REVISION UPDATED STATUS CHART APP VERSION NAMESPACE example 1 Wed Feb 27 22:16:55 2019 DEPLOYED triton-inference-server-1.0.0 1.0 default example-metrics 1 Tue Jan 21 12:24:07 2020 DEPLOYED prometheus-operator-6.18.0 0.32.0 default $ helm uninstall example $ helm uninstall example-metrics ``` For the Prometheus and Grafana services, you should [explicitly delete CRDs](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#uninstall-helm-chart): ``` $ kubectl delete crd alertmanagerconfigs.monitoring.coreos.com alertmanagers.monitoring.coreos.com podmonitors.monitoring.coreos.com probes.monitoring.coreos.com prometheuses.monitoring.coreos.com prometheusrules.monitoring.coreos.com servicemonitors.monitoring.coreos.com thanosrulers.monitoring.coreos.com ``` You may also want to delete the AWS bucket you created to hold the model repository. ``` $ aws s3 rm -r gs://triton-inference-server-repository ```