seldon-core Documentation

Technical Documentation: SeldonIO/seldon-core

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

Deploy Modular, Data-centric AI applications at scale

💡 About Copy Chapter

Seldon Core 2 is an MLOps and LLMOps framework for deploying, managing and scaling AI systems in Kubernetes - from singular models, to modular and data-centric applications. With Core 2 you can deploy in a standardized way across a wide range of model types, on-prem or in any cloud, and production-ready out of the box.

To reach out to Seldon regarding commercial use, visit our website.

📚 Documentation Copy Chapter

The Seldon Core 2 Docs can be found here. For most specific sections, see here:

🧩 Features Copy Chapter

• Pipelines: Deploy composable AI applications, leveraging Kafka for realtime data streaming between components
• Autoscaling for models and application components based on native or custom logic
• Multi-Model Serving: Save infrastructure costs by consolidating multiple models on shared inference servers
• Overcommit: Deploy more models than available memory allows, saving infrastructure costs for unused models
• Experiments: Route data between candidate models or pipelines, with support for A/B tests and shadow deployments
• Custom Components: Implement custom logic, drift & outlier detection, LLMs and more through plug-and-play integrate with the rest of Seldon's ecosytem of ML/AI products!

🔬 Research Copy Chapter

These features are influenced by our position paper on the next generation of ML model serving frameworks:

👉 Desiderata for next generation of ML model serving

📜 License Copy Chapter

Seldon is distributed under the terms of the The Business Source License. A complete version of the license is available in the LICENSE file in this repository. Any contribution made to this project will be licensed under the Business Source License.

Overview (`ansible/README.md`) Copy Section

Ansible Installation

The Ansible installation of a Seldon Core and associated ecosystem is meant for **dev/testing** purposes.
For production use cases follow [Helm installation](https://docs.seldon.io/projects/seldon-core/en/v2/contents/getting-started/kubernetes-installation/helm.html).

Installing Ansible Copy Chapter

Provided Ansible playbooks and roles depends
on kubernetes.core Ansible Collection for performing kubectl
and helm operations.
Check Ansible [documentation] for further information.

We provide a make target to install Ansible together with dependencies and any required collections.
Run the following from the ansible directory:

make install_deps_stable

We have tested provided instructions on Python 3.8 - 3.11 with following version of Python libraries

and kubernetes.core collection in version 2.4.0.

Once installed you can use the following Playbooks that you will find
in Ansible folder of Seldon Core 2 repository.

You also need to have installed kubectl CLI.

Installing Seldon Core 2 using Ansible Copy Chapter

One-liner local kind install (from scratch) Copy Section

If you simply want to install into a fresh local kind
k8s cluster, the seldon-all playbook allows you to do so with a single command:

ansible-playbook playbooks/seldon-all.yaml

This will create a Kind cluster and install ecosystem dependencies (kafka,
prometheus, opentelemetry, jager) as well as all the seldon-specific components.
The seldon components are installed using helm-charts from the current git
checkout (../k8s/helm-charts/).

Internally this runs, in order, the following playbooks (described in more detail
in the sections below):

• kind-cluster.yaml
• setup-ecosystem.yaml
• setup-seldon.yaml

You may pass any of the additonal variables which are configurable for those playbooks
to seldon-all. See the Customizing Ansible Instalation section for details.

For example:

ansible-playbook playbooks/seldon-all.yaml -e seldon_mesh_namespace=my-seldon-mesh -e install_prometheus=no -e @playbooks/vars/set-custom-images.yaml

Running the playbooks individually, as described in the sections below, will give you
more control over what gets run and when (for example, if you want to install into an
existing k8s cluster).

Create Kind Cluster Copy Section

It is recommended to first install Seldon inside Kind cluster.
This allow to test and trial the installation in isolated environment that is easy to remove.

ansible-playbook playbooks/kind-cluster.yaml

Setup Ecosystem Copy Section

Seldon runs by default in seldon-mesh namespace and a Jaeger pod and and OpenTelemetry collector are installed in the
chosen namespace. Run the following:

ansible-playbook playbooks/setup-ecosystem.yaml

The most common change will be to install in another namespace with:

ansible-playbook playbooks/setup-ecosystem.yaml -e seldon_mesh_namespace=<mynamespace>

Install Seldon Core 2 Copy Section

Run the following from the ansible/ folder:

ansible-playbook playbooks/setup-seldon.yaml

If you have changed the namespace you wish to use you will need to run with:

ansible-playbook playbooks/setup-seldon.yaml -e seldon_mesh_namespace=<mynamespace>

Customizing Ansible Installation Copy Chapter

Ecosystem configuration options Copy Section

The ecosystem setup can be parametrized by providing extra Ansible variables, e.g. using -e flag to ansible-playbook
command.

For example run the following from the ansible/ folder:

ansible-playbook playbooks/setup-ecosystem.yaml -e full_install=no -e install_kafka=yes

will only install Kafka when setting up the ecosystem.

Seldon Core 2 configuration options Copy Section

Custom Seldon images and private registries

By default, the container images used in the install are the ones defined by the helm
charts (referring to images publicly available on dockerhub).

If you need to customize the images (i.e pull from private registry, pull given
tag), create a custom images config file following the example in
playbooks/vars/set-custom-images.yaml and run with:

ansible-playbook playbooks/setup-seldon.yaml -e @<path-to-custom-images-config.yaml>

If, instead of pulling images from an external repository you want to build certain components
locally, please read README.dev.md

Private registries

When using private registries, access needs to be authenticated (typically, via a
service account key), and the k8s cluster will need to have access to a secret holding
this key to be able to pull images.

The setup-seldon.yaml playbook will create the required k8s secrets inside the
cluster if it is provided with an auth file in dockerconfigjson format. You provide
the path to this file by cusomizing the custom_image_pull_secrets.dockerconfigjson
variable and define the secret name via custom_image_pull_secrets.name in the custom
images config file (the one passed to the playbook via -e @file).

By default, docker creates the dockerconfigjson auth file in ~/.docker/config.json
after passing the service-account key to docker login.

The docker login command would look like this (key in json format):

cat registry-sa-key.json | docker login -u _json_key --password-stdin <registry-url>

or, for keys in base64 format:

cat registry-sa-key | docker login -u _json_key_base64 --password-stdin <registry-url>

Saving helm-chart customisations

Because the additional custom images config file (starting from the
playbooks/vars/set-custom-images.yaml example) overrides values in the helm-charts
available in the repo, there's also a playbook option of saving those overrides as a
separate values file, which could be used if deploying manually via helm.

This is controlled via two variables:

You can either pass those within the custom images config file or directly when running the
playbook. For example, for just saving the helm-chart overrides (without installing seldon
components), you would run:

ansible-playbook playbooks/setup-seldon.yaml -e full_install=no -e save_helm_components_overrides=yes -e @<path-to-custom-images-config.yaml>

Please note that when deploying outside ansible via helm using this saved overrides file,
and using private registries, you will have to manually create the service-account key
secret with the same name as the one defined in your custom image config file under
custom_image_pull_secrets.name.

Uninstall Copy Chapter

To fully remove the Ansible installation delete the created Kind cluster

kind delete cluster --name seldon

This will stop and delete the Kind cluster freeing all of the resources taken by the dev/trial installation.
You may want to also remove cache resources used for the installation with

rm -rf ~/.cache/seldon/

If you used Ansible to install Seldon Core 2 and its ecosystem into K8s cluster other than Kind you need to manually
remove all the components.

Overview (`docs-gb/README.md`) Copy Section

Discover Seldon Core 2, a Kubernetes-native MLOps framework for deploying ML and LLM systems at scale. Features flexible architecture, standardized workflows, and enhanced observability. Copy Chapter

Production-ready ML Serving Framework

Seldon Core 2 is a Kubernetes-native framework for deploying and managing machine learning (ML) and Large Language Model (LLM) systems at scale. Its data-centric approach and modular architecture enable seamless handling of simple models to complex ML applications across on-premise, hybrid, and multi-cloud environments while ensuring flexibility, standardization, observability, and cost efficiency.

**Flexibility: Real-time, your way** Copy Chapter

Seldon Core 2 offers a platform-agnostic, flexible framework for seamless deployment of different types of ML models across on-premise, cloud, and hybrid environments. Its adaptive architecture enables customizable applications, future-proofing MLOps or LLMOps by scaling deployments as data and applications evolve. The modular design enhances resource-efficiency, allowing dynamic scaling, component reuse, and optimized resource allocation. This ensures long-term scalability, operational efficiency, and adaptability to changing demands.

**Standardization: Consistency across workflows** Copy Chapter

Seldon Core 2 enforces best practices for ML deployment, ensuring consistency, reliability, and efficiency across the entire lifecycle. By automating key deployment steps, it removes operational bottlenecks, enabling faster rollouts and allowing teams to focus on high-value tasks.

With a "learn once, deploy anywhere" approach, Seldon Core 2 standardizes model deployment across on-premise, cloud, and hybrid environments, reducing risk and improving productivity. Its unified execution framework supports conventional, foundational, and LLM models, streamlining deployment and enabling seamless scalability. It also enhances collaboration between MLOps Engineers, Data Scientists, and Software Engineers by providing a customizable framework that fosters knowledge sharing, innovation, and the adoption of new data science capabilities.

**Enhanced Observability** Copy Chapter

Observability in Seldon Core 2 enables real-time monitoring, analysis, and performance tracking of ML systems, covering data pipelines, models, and deployment environments. Its customizable framework combines operational and data science monitoring, ensuring teams have the key metrics needed for maintenance and decision-making.

Seldon simplifies operational monitoring, allowing real-time ML or LLM deployments to expand across organizations while supporting complex, mission-critical use cases. A data-centric approach ensures all prediction data is auditable, maintaining explainability, compliance, and trust in AI-driven decisions.

**Optimization: Modularity to maximize resource efficiency** Copy Chapter

Seldon Core 2 is built for scalability, efficiency, and cost-effective ML operations, enabling you to deploy only the necessary components while maintaining agility and high performance. Its modular architecture ensures that resources are optimized, infrastructure is consolidated, and deployments remain adaptable to evolving business needs.

Scaling for Efficiency: scaling infrastructure dynamically based on demand, auto-scaling for real-time use cases while scaling to zero for on-demand workloads, preserving deployment state for seamless reactivation. By eliminating redundancy and optimizing deployments, it balances cost efficiency and performance, ensuring reliable inference at any scale.

Consolidated Serving Infrastructure: maximizes resource utilization with multi-model serving (MMS) and overcommit, reducing compute overhead while ensuring efficient, reliable inference.

Extendability & Modular Adaptation: integrates with LLMs, Alibi, and other modules, enabling on-demand ML expansion. Its modular design ensures scalable AI, maximizing value extraction, agility, and cost efficiency.

Reusability for Cost Optimization: provides predictable, fixed pricing, enabling cost-effective scaling and innovation while ensuring financial transparency and flexibility.

Next Steps Copy Chapter

• Explore our other Solutions
• Learn about the feature of Seldon Core 2
• Join our Slack Community for updates or for answers to any questions

Overview (`hodometer/README.md`) Copy Section

Hodometer

Hodometer collects and publishes anonymous usage metrics for Seldon Core 2.

Usage Copy Chapter

The metrics collected by Hodometer are used to understand how people interact with Seldon Core.
This includes how many Seldon Core clusters there are and how features are used.

See below for the metrics we collect.

Enabling full metrics collection allows Seldon to best understand what users want.

However, you can reduce the level of information provided by changing the collection level.
Which metrics are covered by which level is documented below.

Alternatively, you can opt out entirely by simply not installing Hodometer, or by removing your existing installation of it.
This does not affect your usage of the software in any way.

Installation Copy Chapter

TODO

Metrics Copy Chapter

There are three levels of metrics that can be enabled:

• Cluster-level -- basic information about the installation
• Resource-level -- high-level details about which Seldon Core 2 resources are used
• Feature-level -- more detailed information about how resources are used

The set of metrics available at each level is:

Privacy Copy Chapter

We aim to not collect any sensitive or identifying information.
For example, we do not collect IP addresses, machine names, or company information.

If you are concerned about any of the information being collected, please raise an issue or PR, or contact Seldon directly.

Overview (`k8s/README.md`) Copy Section

• The generation of helm charts and yaml manifests relies on kubebuilder-generated manifests in
  ../operator/config/[files].

• If you have modified CRD definitions in the operator (../operator/apis/mlops/v1alpha1/* and
  dependencies), you must first run make manifests in the operator directory to update the manifests.
  Only then you should run make create in this directory.

• Never modify files in ../operator/config/[files] manually. These files are generated by
  kubebuilder. Instead, modify the CRD definitions in the operator code

Overview (`operator/README.md`) Copy Section

Creation

This operator was built with Kubebuilder

Version: main.version{KubeBuilderVersion:"3.2.0", KubernetesVendor:"1.22.1", GitCommit:"b7a730c84495122a14a0faff95e9e9615fffbfc5", BuildDate:"2021-10-29T18:32:16Z", GoOs:"linux", GoArch:"amd64"}

Steps to recreate scaffolding

go mod init github.com/seldonio/seldon-core/operator/v2
kubebuilder init --domain seldon.io
kubebuilder edit --multigroup=true
kubebuilder create api --group mlops --version v1alpha1 --kind Model --resource --controller
kubebuilder create api --group mlops --version v1alpha1 --kind Server --resource --controller
kubebuilder create api --group mlops --version v1alpha1 --kind Pipeline --resource --controller
kubebuilder create api --group mlops --version v1alpha1 --kind Explainer --resource --controller

Overview (`prometheus/README.md`) Copy Section

Prometheus

Install Copy Chapter

We recommend install kube-prometheus that provides an all-in-one package with the Prometheus operator.

kubectl apply --server-side -f manifests/setup
until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
kubectl apply -f manifests/

RBAC Copy Section

You will need to modify the default RBAC installed by kube-prometheus as described here.

kubectl apply -f rbac/cr.yaml

Monitors Copy Chapter

We use a PodMonitor for scrapping agent metrics. The envoy and server monitors are there for completeness but not present needed.

kubectl apply -f monitors

Includes:

• Agent pod monitor. Monitors the metrics port of server inference pods.
• Server pod monitor. Monitors the server-metrics port of server inference pods.
• Envoy service monitor. Monitors the Envoy proxies.
• Pipeline gateway pod monitor. Monitors the metrics port of pipeline gateway pods.

Pod monitors were chosen as ports for metrics are not exposed at service level as we do not have a top level service for server replicas but 1 headless service per replica. Future discussions could reference this.

Reference

• Prometheus CRDs

Access the dashboards

• Access to dashboards is explained here

TODO

• Provide Ansible installation

Overview (`samples/README.md`) Copy Section

Samples

Various example notebooks.

Requirements Copy Chapter

• a seldon cli
• jupyter notebook
• jq

Artefacts Copy Chapter

All (Python) artefacts are versioned by their respective version of Triton and
MLServer used at the time of training (e.g.
gs://seldon-models/scv2/samples/mlserver_1.3.5/iris-sklearn).
To re-generate them, you can call the following Makefile targets:

$ make train-all upload-all

The new version of the artefacts will get uploaded to GCS, using the version of
Triton and MLServer specified on scheduler/Makefile.
The upload-all target will also replace all previous references to versioned
artefacts in the notebooks.

Overview (`scheduler/README.md`) Copy Section

Seldon Envoy XDS Server

Development Setup Copy Chapter

The scheduler and its associated control-plane components in Seldon Core 2 are intended to run both in and out of Kubernetes.
While it is possible to run all the individual components locally as plain, OS-level processes (see below), the recommended approach is to use Docker Compose to ensure isolation and reproducibility.

Docker Compose Copy Section

Installation

Ensure that you are using a version of Docker Compose ("Compose") compatible with the 3.9+ file format.

You can install directly by following the official instructions.
Alternatively, there are asdf plugins for Compose which you can choose from:

asdf plugin-list-all | grep docker

You may also want to install shell completion, if it has not been installed for you.
See the official documentation.

Full control plane

There are Makefile targets to start and stop a minimal Seldon Core 2 setup.
You need to choose whether to run with MLServer or Triton as the inference server, but otherwise the setups are almost identical.

To get started:

make start-all-mlserver
# OR
make start-all-triton

and once finished, you can spin down the services with:

make stop-all-mlserver
# OR
make stop-all-triton

Examples

It is possible to interact with and even start or stop individual services when using Compose.
A collection of examples are given below.

NOTE You need to specify the project (-p) and the appropriate manifest and environment files when interacting directly with services start via the Makefile targets.

Compose allows multiple manifests to be specified, with the latter ones able to override configuration from the earlier ones.
This greatly reduces duplication between the MLServer and Triton setups at the cost of an extra command-line argument.

Check running services

$ docker-compose -f all-base.yaml -f all-mlserver.yaml --env-file env.all -p scv2_mlserver ps
          Name                         Command               State                                            Ports
-------------------------------------------------------------------------------------------------------------------------------------------------------------
scv2_mlserver_agent_1       /bin/agent --log-level deb ...   Up      0.0.0.0:8090->8090/tcp,:::8090->8090/tcp, 0.0.0.0:8091->8091/tcp,:::8091->8091/tcp
scv2_mlserver_envoy_1       /docker-entrypoint.sh /bin ...   Up      10000/tcp, 0.0.0.0:9000->9000/tcp,:::9000->9000/tcp,
                                                                     0.0.0.0:9003->9003/tcp,:::9003->9003/tcp
scv2_mlserver_rclone_1      rclone rcd --rc-no-auth -- ...   Up      0.0.0.0:5572->5572/tcp,:::5572->5572/tcp
scv2_mlserver_scheduler_1   /bin/scheduler                   Up      0.0.0.0:9002->9002/tcp,:::9002->9002/tcp, 0.0.0.0:9004->9004/tcp,:::9004->9004/tcp,
                                                                     0.0.0.0:9005->9005/tcp,:::9005->9005/tcp
scv2_mlserver_server_1      mlserver start /mnt/models       Up      0.0.0.0:8080->8080/tcp,:::8080->8080/tcp, 0.0.0.0:8081->8081/tcp,:::8081->8081/tcp

Check logs

$ docker-compose -f all-base.yaml -f all-mlserver.yaml --env-file env.all -p scv2_mlserver logs agent | tail
agent_1      | time="2022-02-04T12:14:03Z" level=info msg="Calling Rclone server: /rc/noop with {\"foo\":\"bar\"}" Source=RCloneClient
agent_1      | time="2022-02-04T12:14:03Z" level=error msg="Rclone not ready" Name=Client error="Post \"http://0.0.0.0:5572/rc/noop\": dial tcp 0.0.0.0:5572: connect: connection refused" func=waitReady
...

Restart a service

The argument to the restart command is the name of the service in the manifest, e.g. server.

$ docker-compose -f all-base.yaml -f all-mlserver.yaml --env-file env.all -p scv2_mlserver restart server
Restarting scv2_mlserver_server_1 ... done

Start a single service

Note that this example does not specify a project, so Compose defaults to the parent directory's name.
This example also does not specify -f all-mlserver.yaml as the Rclone configuration is defined in all-base.yaml and is completely independent of the inference server in use.

We can override values from env.all to specify the repository and tag for the Rclone image.

$ RCLONE_IMAGE_AND_TAG=seldonio/seldon-reclone:latest docker-compose -f all-base.yaml --env-file env.all run -d rclone
Building rclone
Step 1/3 : FROM rclone/rclone:1.56.2
1.56.2: Pulling from rclone/rclone
a0d0a0d46f8b: Already exists
...
Creating scheduler_rclone_run ... done
scheduler_rclone_run_4379d7918894

Inspect environment variables in a container

It can be useful to see what environment variables have been set in a container, as these do not show up in the command column of ps commands.
The below shows a way of inspecting these values for a container called scv2_mlserver_agent_1:

$ docker inspect -f '{{ range $i, $v := .Config.Env }}{{ $v }}{{ println }}{{ end }}' scv2_mlserver_agent_1
SELDON_OVERCOMMIT_PERCENTAGE=0
SELDON_SERVER_HTTP_PORT=8090
SELDON_SERVER_GRPC_PORT=8091
SELDON_DEBUG_GRPC_PORT=7777
SELDON_SCHEDULER_HOST=0.0.0.0
SELDON_SCHEDULER_PORT=9005
MEMORY_REQUEST=1000000
SELDON_SERVER_TYPE=mlserver
SELDON_SERVER_CAPABILITIES=sklearn,xgboost
PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GOLANG_VERSION=1.17.6
GOPATH=/go

Check specific details for running containers

The default view of docker-compose ps is limited, in that it does not have all the same configuration values as the equivalent docker command.
It can also be very dense for a split-screen or smaller screen view.
The below provides an example on customising the display:

$ docker ps --format 'table {{ .Image }}\t{{ .Names }}\t{{ .Status }}\t{{ .Command }}' --no-trunc
IMAGE                                NAMES                    STATUS          COMMAND
seldonio/seldon-envoy-local:latest   scv2_mlserver_envoy_1    Up 2 hours      "/docker-entrypoint.sh /bin/sh -c '/usr/local/bin/envoy -c /etc/envoy.yaml'"
seldonio/seldon-rclone:latest        scv2_mlserver_rclone_1   Up 2 hours      "rclone rcd --rc-no-auth --config=/rclone/rclone.conf --rc-addr=0.0.0.0:5572 --verbose"
seldonio/mlserver:1.0.0.rc1          scv2_mlserver_server_1   Up 10 minutes   "mlserver start /mnt/models"
registry:2                           kind-registry            Up 7 hours      "/entrypoint.sh /etc/docker/registry/config.yml"

gRPC compile Copy Section

Install protoc.

protoc --version
libprotoc 3.18.0

Install go grpc plugins

$ go install google.golang.org/protobuf/cmd/[email protected]
$ go install google.golang.org/grpc/cmd/[email protected]

Local Envoy Install Copy Section

Install or download envoy.

Tested with:

envoy --version

envoy  version: a2a1e3eed4214a38608ec223859fcfa8fb679b14/1.19.1/Clean/RELEASE/BoringSSL

MlServer Copy Section

pip install mlserver

Local Test Copy Chapter

make build

Follow steps in local test notebook

K8S Test Copy Chapter

make kind-image-install-all

Follow steps in k8s test notebook

Docs Copy Chapter

development docs

Overview (`docs-gb/apis/README.md`) Copy Section

APIs

Seldon provides APIs for management and inference.

• API for inference
• Scheduler API for management (Advanced)
• Internal APIs (Reference)

Overview (`docs-gb/architecture/README.md`) Copy Section

Learn about the Seldon Core 2 microservice architecture, control plane and data plane components, and how these services work to provide a scalable, fault-tolerant ML model serving and management. Copy Chapter

Architecture

Seldon Core 2 uses a microservice architecture where each service has limited and well-defined responsibilities working together to orchestrate scalable and fault-tolerant ML serving and management. These components communicate internally using gRPC and they can be scaled independently. Seldon Core 2 services can be split into two categories:

• Control Plane services are responsible for managing the operations and configurations of your ML models and workflows. This includes functionality to instantiate new inference servers, load models, update new versions of models, configure model experiments and pipelines, and expose endpoints that may receive inference requests. The main control plane component is the Scheduler that is responsible for managing the loading and unloading of resources (models, pipelines, experiments) onto the respective components.
• Data Plane services are responsible for managing the flow of data between components or models. Core 2 supports REST and gRPC payloads that follow the Open Inference Protocol (OIP). The main data plane service is Envoy, which acts as a single ingress for all data plane load and routes data to the relevant servers internally (e.g. Seldon MLServer or NVidia Triton pods).

{% hint style="info" %}
Note: Because Core 2 architecture separates control plane and data plane responsibilities, when control plane services are down (e.g. the Scheduler), data plane inference can still be served. In this manner the system is more resilient to failures. For example, an outage of control plane services does not impact the ability of the system to respond to end user traffic. Core 2 can be provisioned to be highly available on the data plane path.
{% endhint %}

The current set of services used in Seldon Core 2 is shown below. Following the diagram, we will describe each control plane and data plan service.

.png>)

Control Plane Copy Chapter

Scheduler Copy Section

This service manages the loading and unloading of Models, Pipelines and Experiments on the relevant micro services. It is also responsible for matching Models with available Servers in a way that optimises infrastructure use. In the current design we can only have one instance of the Scheduler as its internal state is persisted on disk.

When the Scheduler (re)starts there is a synchronisation flow to coordinate the startup process and to attempt to wait for expected Model Servers to connect before proceeding with control plane operations. This is important so that ongoing data plane operations are not interrupted. This then introduces a delay on any control plane operations until the process has finished (including control plan resources status updates). This synchronisation process has a timeout, which has a default of 10 minutes. It can be changed by setting helm seldon-core-v2-components value scheduler.schedulerReadyTimeoutSeconds.

Agent Copy Section

This service manages the loading and unloading of models on a server and access to the server over REST/gRPC. It acts as a reverse proxy to connect end users with the actual Model Servers. In this way the system collects stats and metrics about data plane inferences that helps with observability and scaling.

Controller Copy Section

We also provide a Kubernetes Operator to allow Kubernetes usage. This is implemented in the Controller Manager microservice, which manages CRD reconciliation with Scheduler. Currently Core 2 supports one instance of the Controller.

{% hint style="info" %}
Note: All services besides the Controller are Kubernetes agnostic and can run locally, e.g. on Docker Compose.
{% endhint %}

Data Plane Copy Chapter

Pipeline Gateway Copy Section

This service handles REST/gRPC calls to Pipelines. It translates between synchronous requests to Kafka operations, producing a message on the relevant input topic for a Pipeline and consuming from the output topic to return inference results back to the users.

Model Gateway Copy Section

This service handles the flow of data from models to inference requests on servers and passes on the responses via Kafka.

Dataflow Engine Copy Section

This service handles the flow of data between components in a pipeline, using Kafka Streams. It enables Core 2 to chain and join Models together to provide complex Pipelines.

Envoy Copy Section

Envoy acts as the request proxy for Seldon Core 2, routing traffic to the appropriate inference servers and handling load balancing across available replicas. Envoy configuration of Seldon Core 2 uses weighted least-request load balancing, which dynamically distributes traffic based on both replica weights and current load, helping ensure efficient and stable request routing.

Dataflow Architecture and Pipelines Copy Chapter

To support the movement towards data centric machine learning Seldon Core 2 follows a dataflow paradigm. By taking a decentralized route that focuses on the flow of data, users can have more flexibility and insight as they build and manage complex AI applications in production. This contrasts with more centralized orchestration approaches where data is secondary.

.png>)

Kafka Copy Section

Kafka is used as the backbone for Pipelines allowing decentralized, synchronous and asynchronous usage. This enables Models to be connected together into arbitrary directed acyclic graphs. Models can be reused in different Pipelines. The flow of data between models is handled by the dataflow engine using Kafka Streams.

.png>)

By focusing on the data we allow users to join various flows together using stream joining concepts as shown below.

.png>)

We support several types of joins:

• inner joins, where all inputs need to be present for a transaction to join the tensors passed through the Pipeline;
• outer joins, where only a subset needs to be available during the join window
• triggers, in which data flows need to wait until records on one or more trigger data flows appear. The data in these triggers is not passed onwards from the join.

These techniques allow users to create complex pipeline flows of data between machine learning components.

More discussion on the data flow view of machine learning and its effect on v2 design can be found here.

Overview (`docs-gb/cli/README.md`) Copy Section

CLI

Seldon provides a CLI to allow easy management and testing of Model, Experiment, and Pipeline resources.

At present this needs to be built by hand from the operator folder.

make build-seldon     # for linux/macOS amd64
make build-seldon-arm # for macOS ARM

Then place the bin/seldon executable in your path.

• cli docs

Environment Variables and Services Copy Chapter

The CLI talks to 3 backend services on default endpoints:

1. The Seldon Core 2 Scheduler: default 0.0.0.0:9004
2. The Seldon Core inference endpoint: default 0.0.0.0:9000
3. The Seldon Kafka broker: default: 0.0.0.0:9092

These defaults will be correct when Seldon Core 2 is installed locally as per the docs. For Kubernetes, you will need to change these by defining environment variables.

:language: golang
:start-after: // Defaults
:end-before: // Help statements

Kubernetes Usage Copy Chapter

Inference Service Copy Section

For a default install into the seldon-mesh namespace if you have exposed the inference svc as a loadbalancer you will find it at:

kubectl get svc seldon-mesh -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Use above IP at port 80:

export SELDON_INFER_HOST=<ip>:80

Scheduler Service Copy Section

For a default install into the seldon-mesh namespace if you have exposed the scheduler svc as a loadbalancer you will find it at:

kubectl get svc seldon-scheduler -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Use above IP at port 9004:

export SELDON_SCHEDULE_HOST=<ip>:9004

Kafka Broker Copy Section

The Kafka broker will depend on how you have installed Kafka into your Kubernetes cluster. Find the broker IP and use:

export SELDON_KAFKA_BROKER=<ip>:<port>

Config file Copy Chapter

You can create a config file to manage connections to running Seldon Core 2 installs. The settings will override any environment variable settings.

The definition is shown below:

:language: golang
 :start-after: // start config struct
 :end-before: // end config struct

An example below shows an example where we connect via TLS to the Seldon scheduler using our scheduler client certificate:

{
    "controlplane":{
    "schedulerHost": "seldon-scheduler.svc:9044",
    "tls"; true,
    "keyPath": "/home/certs/seldon-scheduler-client/tls.key",
    "crtPath": "/home/certs/seldon-scheduler-client/tls.crt",
    "caPath": "/home/certs/seldon-scheduler-client/ca.crt"
    }
}

To manage config files and activate them you can use the CLI command seldon config which has subcommands to list, add, remove, activate and decative configs.

For example:

$ seldon config list
config		path						active
------		----						------
kind-sasl	/home/work/seldon/cli/config-sasl.json		*

$ seldon config deactivate kind-sasl

$ seldon config list
config		path						active
------		----						------
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config add gcp-scv2 ~/seldon/cli/gcp.json

$ seldon config list
config		path						active
------		----						------
gcp-scv2	/home/work/seldon/cli/gcp.json
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config activate gcp-scv2

$ seldon config list
config		path						active
------		----						------
gcp-scv2	/home/work/seldon/cli/gcp.json	    		*
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config list kind-sasl
{
  "controlplane": {
    "schedulerHost": "172.19.255.2:9004"
  },
  "kafka": {
    "bootstrap": "172.19.255.3:9093",
    "caPath": "/home/work/gcp/scv2/certs/seldon-cluster-ca-cert/ca.crt"
  }
}

TLS Certificates for Local Use Copy Chapter

For running with Kubernetes TLS connections on the control and/or data plane, certificates will need to be downloaded locally. We provide an example script which will download certificates from a Kubernetes secret and store them in a folder. It can be found in hack/download-k8s-certs.sh and takes 2 or 3 arguments:

./download-k8s-certs.sh <namespace> <secret> [<folder>]

e.g.:

./download-k8s-certs.sh seldon-mesh seldon-scheduler-client

Overview (`docs-gb/concepts/README.md`) Copy Section

Learn how Seldon Core is different from a centralized orchestrator to a data flow architecture, enabling more efficient ML model deployment through stream processing and improved data handling. Copy Chapter

Concepts

In the context of machine learning and Seldon Core 2, concepts provide a framework for understanding key functionalities, architectures, and workflows within the system. Some of the key concepts in Seldon Core 2 are:

• Data-Centric MLOps
• Open Inference Protocol
• Components
• Pipelines
• Servers
• Experiments

Data-Centric MLOps Copy Section

Data-centricity is an approach that puts the management, integrity, and flow of data at the core of machine learning deployment. Rather than focusing solely on models, this approach ensures that data quality, consistency, and adaptability drive successful ML operations. In Seldon Core 2, data-centricity is embedded in every stage of the inference workflow.

Why Data-Centricity Matters?

By adopting a data-centric approach, Seldon Core 2 enables:

• More reliable and high-quality predictions by ensuring clean, well-structured data.
• Scalable and future-proof ML deployments through standardized data management.
• Efficient monitoring and maintenance, reducing risks related to model drift and inconsistencies.

With data-centricity as a core principle, Seldon Core 2 ensures end-to-end control over ML workflows, enabling you to maximize model performance and reliability in production environments.

Open Inference Protocol in Seldon Core 2 Copy Section

The Open Inference Protocol (OIP) defines a standard way for inference servers and clients to communicate in Seldon Core 2. Its goal is to enable interoperability, flexibility, and consistency across different model-serving runtimes. It exposes the Health API, Metadata API, and the Inference API.

Some of the features of Open Inference Protocol includes:

• Transport Agnostic: Servers can implement HTTP/REST or gRPC protocols.
• Runtime Awareness: Use the protocolVersion field in your runtime YAML or consult the supported runtimes table to verify compatibility.

By adopting OIP, Seldon Core 2 promotes a consistent experience across a diverse set of model deployments.

Components in Seldon Core 2 Copy Section

Components are the building blocks of an inference graph, processing data at various stages of the ML inference pipeline. They provide reusable, standardized interfaces, making it easier to maintain and update workflows without disrupting the entire system. Components include ML models, data processors, routers, and supplementary services.

Types of Components

By modularizing inference workflows, components allow you to scale, experiment, and optimize ML deployments efficiently while ensuring data consistency and reliability.

Pipelines Copy Section

In a model serving platform like Seldon Core 2, a pipeline refers to an orchestrated sequence of models and components that work together to serve more complex AI applications. These pipelines allow you to connect models, routers, transformers, and other inference components, with Kafka used to stream data between them. Each component in the pipeline is modular and independently managed—meaning it can be scaled, configured, and updated separately based on its specific input and output requirements.

This use of "pipeline" is distinct from how the term is used in MLOps for CI/CD pipelines, which automate workflows for building, testing, and deploying models. In contrast, Core 2 pipelines operate at runtime and focus on the live composition and orchestration of inference systems in production.

Servers Copy Section

In Core 2, servers are responsible for hosting and serving machine learning models, handling inference requests, and ensuring scalability, efficiency, and observability in production. Core 2 supports multiple inference servers, including MLServer, and NVIDIA Triton Inference Server, enabling flexible and optimized model deployments.

MLServer: A lightweight, extensible inference server designed to work with multiple ML frameworks, including scikit-learn, XGBoost, TensorFlow, and PyTorch. It supports custom Python models and integrates well with MLOps workflows. It is built for *General-purpose model serving, custom model wrappers, multi-framework support.

Experiments Copy Section

In Seldon Core 2, experiments enable controlled A/B testing, model comparisons, and performance evaluations by defining an HTTP traffic split between different models or inference pipelines. This allows organizations to test multiple versions of a model in production while managing risk and ensuring continuous improvements.

Some of the advantages of using Experiments:

• A/B Testing & Model comparison: Compare different models under real-world conditions without full deployment.
• Risk-Free model validation: Test a new model or pipeline in parallel without affecting live predictions.
• Performance monitoring: Assess latency, accuracy, and reliability before a full rollout.
• Continuous improvement: Make data-driven deployment decisions based on real-time model performance.

Overview (`docs-gb/development/README.md`) Copy Section

Development

Development Requirements Copy Chapter

• Go 1.17+ with glibc development libraries installed.
  • e.g. for Ubuntu systems: sudo apt-get install libc6-dev
• Java JDK 17+ and Kotlin 1.6.10+
• Kubebuilder V2
• Docker and docker-compose
• Helm
• Kustomize
• Ansible

Testing resources Copy Chapter

• Kind
• k6
• unix utils: jq, curl, grpcurl

Release Process Copy Chapter

Releases are carried out via Github.

Overview (`docs-gb/examples/README.md`) Copy Section

Examples

Workflows Copy Chapter

Seldon inference is built from atomic Model components. Models as shown here cover a wide range of artifacts including:

• Core machine learning models, e.g. a PyTorch model.
• Feature transformations that might be built with custom python code.
• Drift detectors.
• Outlier detectors.
• Explainers
• Adversarial detectors.

A typical workflow for a production machine learning setup might be as follows:

1. You create a Tensorflow model for your core application use case and test this model in isolation to validate.
2. You create SKLearn feature transformation component before your model to convert the input into the correct form for your model. You also create Drift and Outlier detectors using Seldon's open source Alibi-detect library and test these in isolation.
3. You join these components together into a Pipeline for the final production setup.

These steps are shown in the diagram below:

.png>)

Examples & Tutorials Copy Chapter

This section will provide some examples to allow operations with Seldon to be tested so you can run your own models, experiments, pipelines and explainers.

Getting Started Examples Copy Chapter

• Local examples
• Kubernetes examples

Models Copy Chapter

• Huggingface models
• Model zoo
• Artifact versions

Pipelines Copy Chapter

• Pipeline examples
• Pipeline to pipeline examples
• Cyclic Pipeline

Explainers Copy Chapter

• Explainer examples

Servers Copy Chapter

• Custom Servers

Experiments Copy Chapter

• Local experiments
• Experiment version examples

Making Inference Requests Copy Chapter

• Inference examples
• Tritonclient examples
• Batch Inference examples (kubernetes)
• Batch Inference examples (local)

Misc Copy Chapter

• Checking Pipeline readiness
• Local Overcommit

Further Kubernetes Examples Copy Chapter

• Kubernetes custerwide example

Advanced Examples Copy Chapter

• Huggingface speech to sentiment with explanations pipeline
• Production image classifier with drift and outlier monitoring
• Production income classifier with drift, outlier and explanations
• Conditional pipeline with pandas query model
• Kubernetes Server with PVC

Overview (`docs-gb/getting-started/README.md`) Copy Section

Getting Started

{% hint style="info" %}
Note: Some dependencies may require that the (virtual) machines on which you deploy, support the SSE4.2
instruction set or x86-64-v2 microarchitecture. If lscpu | grep sse4_2 does not return anything on
your machine, your CPU is not compatible, and you may need to update the (virtual) host's CPU.
{% endhint %}

Seldon Core can be installed either with Docker Compose or with Kubernetes:

• Install locally with Docker Compose
• Install onto a Kubernetes cluster

Once installed:

• Try the existing examples.
• Train and deploy your own model artifact.

Core Concepts Copy Chapter

There are three core resources you will use:

• Models - for deploying single machine learning models, custom transformation
  logic, drift detectors and outliers detectors.
• Pipelines - for connecting together flows of data transformations between Models
  with a synchronous path and multiple asynchronous paths.
• Experiments - for testing new versions of models

By default the standard installation will deploy MLServer and Triton inference servers which provide
support for a wide range of machine learning model artifacts including Tensorflow models, PyTorch models,
SKlearn models, XGBoost models, ONNX models, TensorRT models, custom python models and many more. For
advanced use, the creation of new inference servers is manged by two resources:

• Servers - for deploying sets of replicas of core inference servers (MLServer or Triton by default).
• ServerConfigs - for defining server configurations including custom servers.

API for Inference Copy Chapter

Once deployed models can be called using the Seldon V2 inference protocol. This
protocol created by Seldon, NVIDIA and the KServe projects is supported by MLServer and Triton inference
servers amingst others and allows REST and gRPC calls to your model.

Your model is exposed via our internal Envoy gateway. If you wish to expose your models in Kubernetes
outside the cluster you are free to use any Service Mesh or Ingress technology. Various examples are provided
for service mesh integration.

Inference Metrics Copy Chapter

Metrics are exposed for scraping by Prometheus. For Kubernetes we provide example instructions for using kube-prometheus.

Pipeline requirements Copy Chapter

Pipelines are built upon Kafka streaming technology.

Overview (`docs-gb/installation/README.md`) Copy Section

keywords:

• Seldon Core
• MLOps
• Kubernetes
• Docker Compose
• installation guide
• Helm charts
• machine learning deployment
  Learn how to install Seldon Core 2 in various environments - from local
  development with Docker Compose to production-grade Kubernetes clusters.

Installation Overview

Seldon Core 2 can be installed in various setups to suit different stages of the development lifecycle. The most common modes include:

Local Environment Copy Chapter

Ideal for development and testing purposes, a local setup allows for quick iteration and experimentation with minimal overhead. Common tools include:

• Docker Compose:
  Simplifies deployment by orchestrating Seldon Core components and dependencies in Docker containers. Suitable for environments without Kubernetes, providing a lightweight alternative.
• Kind (Kubernetes IN Docker):
  Runs a Kubernetes cluster inside Docker, offering a realistic testing environment.
  Ideal for experimenting with Kubernetes-native features.

Production Environment Copy Chapter

Designed for high-availability and scalable deployments, a production setup ensures security, reliability, and resource efficiency. Typical tools and setups include:

• Managed Kubernetes Clusters:
  Platforms like GKE (Google Kubernetes Engine), EKS (Amazon Elastic Kubernetes Service), and AKS (Azure Kubernetes Service) provide managed Kubernetes solutions.
  Suitable for enterprises requiring scalability and cloud integration.
• On-Premises Kubernetes Clusters:
  For organizations with strict compliance or data sovereignty requirements.
  Can be deployed on platforms like OpenShift or custom Kubernetes setups.

By selecting the appropriate installation mode—whether it's Docker Compose for simplicity, Kind for local Kubernetes experimentation, or production-grade Kubernetes for scalability—you can effectively leverage Seldon Core 2 to meet your specific needs.

Helm Charts Copy Chapter

For more information, see the published Helm charts.

For the description of (some) values that can be configured for these charts, see this Helm parameters section.

Seldon Core 2 Dependencies Copy Chapter

Here is a list of components that Seldon Core 2 requires, along with the minimum and maximum supported versions:

Notes:

• Envoy and Rclone: These components are included as part of the Seldon Core 2 Docker images. You are not required to install them separately but must be aware of the configuration options supported by these versions.
• Kafka: Only required for operating Seldon Core 2 dataflow Pipelines. If not needed, you should avoid installing seldon-modelgateway, seldon-pipelinegateway, and seldon-dataflow-engine.
• Maximum Versions marked with *** indicates no hard limit on the version that can be used.

Get started Copy Section

Overview (`docs-gb/kubernetes/README.md`) Copy Section

Kubernetes

Kubernetes provides the core platform for production use. However, the architecture in
agnostic to the underlying cluster technology as shown by the
local Docker installation.

Overview (`docs-gb/metrics/README.md`) Copy Section

Metrics

There are two kinds of metrics present in Seldon Core 2:

• operational metrics
• usage metrics

Operational metrics describe the performance of components in the system. Some examples of common operational
considerations are memory consumption and CPU usage, request latency and throughput, and cache utilisation rates.
Generally speaking, these are the metrics system administrators, operations teams, and engineers will be interested in.

Usage metrics describe the system at a higher and less dynamic level. Some examples include the number of deployed
servers and models, and component versions. These are not typically metrics that engineers need insight into, but
may be relevant to platform providers and operations teams.

Overview (`docs-gb/models/README.md`) Copy Section

Models

Models provide the atomic building blocks of Seldon. They represents machine learning models,
drift detectors, outlier detectors, explainers, feature transformations, and more complex routing
models such as multi-armed bandits.

• Seldon can handle a wide range of inference artifacts
• Artifacts can be stored on any of the 40 or more cloud storage technologies as well as from
  local (mounted) folder as discussed here.

Kubernetes Example Copy Chapter

A Kubernetes yaml example is shown below for a SKLearn model for iris classification:

# samples/models/sklearn-iris-gs.yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
spec:
  storageUri: "gs://seldon-models/scv2/samples/mlserver_1.5.0/iris-sklearn"
  requirements:
  - sklearn
  memory: 100Ki

Its Kubernetes spec has two core requirements

• A storageUri specifying the location of the artifact. This can be any rclone URI specification.
• A requirements list which provides tags that need to be matched by the Server that can run
  this artifact type. By default when you install Seldon we provide a set of Servers that cover a
  range of artifact types.

GRPC Example Copy Chapter

You can also load models directly over the scheduler grpc service. An example is shown below use grpcurl tool:

!grpcurl -d '{"model":{ \
              "meta":{"name":"iris"},\
              "modelSpec":{"uri":"gs://seldon-models/mlserver/iris",\
                           "requirements":["sklearn"],\
                           "memoryBytes":500},\
              "deploymentSpec":{"replicas":1}}}' \
         -plaintext \
         -import-path ../../apis \
         -proto apis/mlops/scheduler/scheduler.proto  0.0.0.0:9004 seldon.mlops.scheduler.Scheduler/LoadModel

The proto buffer definitions for the scheduler are outlined here.

Multi-model Serving with Overcommit Copy Chapter

Multi-model serving is an architecture pattern where one ML inference server hosts multiple models
at the same time. It is a feature provided out of the box by Nvidia Triton and Seldon MLServer.
Multi-model serving reduces infrastructure hardware requirements (e.g. expensive GPUs) which enables
the deployment of a large number of models while making it efficient to operate the system at scale.

Seldon Core 2 leverages multi-model serving by design and it is the default option for deploying
models. The system will find an appropriate server to load the model onto based on requirements that
the user defines in the Model deployment definition.

Moreover, in many cases demand patterns allow for further Overcommit of resources. Seldon Core 2
is able to register more models than what can be served by the provisioned (memory) infrastructure
and will swap models dynamically according to least used without adding significant latency overheads
to inference workload.

See Multi-model serving for more information.

Autoscaling of Models Copy Chapter

See here for discussion of autoscaling of models.

Scheduling of Models onto Servers Copy Chapter

See here for details on how Core 2 schedules Models onto Servers.

Overview (`docs-gb/operational-monitoring/README.md`) Copy Section

Operational Monitoring

Seldon Core 2 provides robust tools for tracking the performance and health of machine learning models in production.

Monitoring Copy Section

• Real-Time metrics: collects and displays real-time metrics from deployed models, such as response times, error rates, and resource usage.
• Model performance tracking: monitors key performance indicators (KPIs) like accuracy, drift detection, and model degradation over time.
• Custom metrics: allows you to define and track custom metrics specific to their models and use cases.
• Visualization: Provides dashboards and visualizations to easily observe the status and performance of models.

There are two kinds of metrics present in Seldon Core 2 that you can monitor:

• operational metrics
• usage metrics

Operational metrics describe the performance of components in the system. Some examples of common operational considerations are memory consumption and CPU usage, request latency and throughput, and cache utilisation rates. Generally speaking, these are the metrics system administrators, operations teams, and engineers will be interested in.

Usage metrics describe the system at a higher and less dynamic level. Some examples include the number of deployed servers and models, and component versions. These are not typically metrics that engineers need insight into, but may be relevant to platform providers and operations teams.

Overview (`docs-gb/performance-tuning/readme.md`) Copy Section

Introduction

In MLOps, system performance can be defined as how efficiently and effectively an ML model or application operates in a production environment. It is typically measured across several key dimensions, such as latency, throughput, scalability, and resource-efficiency. These factors are deeply connected: changes in configuration often result in tradeoffs, and should be carefully considered. Specifically, latency, throughput and resource usage can all impact each other, and the approach to optimizing system performance depends on the desired balance of these outcomes in order to ensure a positive end-user experience, while also minimising infrastructure costs.

High Level Approach Copy Chapter

There are many different levers that can be considered to tune performance for ML systems deployed with Core 2, across infrastructure, models, inference execution, and the related configurations exposed by Core 2. When reasoning about the performance of an ML-based system deployed using Seldon, we recommend breaking down the problem by first understanding and tuning the performance of deployed Models, and then subsequently considering more complex Pipelines composed of those models (if applicable). For both models and pipelines, it is important to run tests to understand baseline performance characteristics, before making efforts to tune these through changes to models, infrastructure, or inference configurations. The recommended approach can be broken down as follows:

Models Copy Section

1. Load testing to understand latency and throughput behaviour for one model replica
2. Tuning performance for models. This can be done via changes to:
   1. Infrastructure - choosing the right hardware, and configurations in Core related to CPUs, GPUs and memory.
   2. Models - optimizing model artefacts in how they are structured, configured, stored. This can include model pruning, quantization, consideration of different model frameworks, and making sure that the model can achieve a high utilisation of the allocated resources.
   3. Inference - the way in which inference is executed. This can include the choice of communication protocols (REST, gRPC), payload configuration, batching, and efficient execution of concurrent requests.

**Pipelines** Copy Section

1. Testing Pipelines to identify the critical path based on performance of underlying models
2. Core 2 Configuration to optimize data-processing through pipelines
3. Scalability of Pipelines to understand how Core 2 components scale with the number of deployed pipelines and models

Overview (`docs-gb/scaling/README.md`) Copy Section

This page provides an overview of the appraoches to achieving autoscaling with Core 2. Copy Chapter

Autoscaling

Autoscaling in Seldon Core 2 Copy Chapter

Seldon Core 2 provides multiple approaches to scaling your machine learning deployments, allowing you to optimize resource utilization and handle varying workloads efficiently. In Core 2, we separate out Models and Servers, and Servers can have multiple Models loaded on them (Multi-Model Serving). Given this, setting up autoscaling requires defining the logic by which you want to scale your Models and then configuring the autoscaling of Servers such that they autoscale in a coordinated way. The following steps can be followed to set up autoscaling based on specific requirements:

1. Identify metrics that you want to scale Models on. There are a couple of different options here:
   1. Core 2 natively supports scaling based on Inference Lag, meaning the difference between incoming and outgoing requests for a model in a given period of time. This is done by configuring minReplicas or maxReplicas in the Model CRDs and making sure you configure the Core 2 install with the autoscaling.autoscalingModelEnabled helm value set to true (default is false).
   2. Users can expose custom or Kubernetes-native metrics, and then target the scaling of models based on those metrics by using HorizontalPodAutoscaler. This requires exposing the right metrics, using the monitoring tool of your choice (e.g. Prometheus).

Once the approach for Model scaling is implemented, Server scaling needs to be configured.

2. Implement Server Scaling by either:
   1. Enabling Autoscaling of Servers based on Model needs. This is managed by Seldon's scheduler, and is enabled by setting minReplicas and maxReplicas in the Server Custom Resource and making sure you configure the Core 2 install with the autoscaling.autoscalingServerEnabled helm value set to true (the default)
   2. If Models and Servers are to have a one-to-one mapping (no Multi-Model Serving) then users can also define scaling of Servers using an HPA manifest that matches the HPA applied to the associated Models. This approach is outlined here. This approach will only work with custom metrics, as Kubernetes does not allow mutliple HPAs to target the same metrics from Kubernetes directly.

Based on the requirements above, one of the following three options for coordinated autoscaling of Models and Servers can be chosen:

Alternatively, the following decision-tree showcases the approaches we recommend based on users' requirements:

Scaling Seldon Services Copy Chapter

When running Core 2 at scale, it is important to understand the scaling behaviour of Seldon's services as well as the scaling of the Models and Servers themselves. This is outlined in the Scaling Core Services page.

Overview (`kafka/strimzi/README.md`) Copy Section

Strimzi Kafka

Seldon Core 2 requires Kafka to implement data-centric inference Pipelines.
To install Kafka for testing purposed in your k8s cluster, we recommend to use Strimzi Operator.

This page discuss how to install Strimzi Operator and create Kafka cluster for trial, dev, or testing purposes.
For production grade installation consult [Strimzi documentation](https://strimzi.io/documentation/) or use one of managed solutions mentioned [here](./index.md).

You can install and configure Strimzi using either Helm charts or our Ansible playbooks, both documented below.

Helm Copy Chapter

The installation of a Kafka cluster requires the Strimzi Kafka operator installed in the same namespace.
This allows to directly use the mTLS certificates created by Strimzi Operator.
One option to install the Strimzi operator is via Helm.

Note that we are using here KRaft instead of Zookeeper for Kafka.
You can enable featureGates during Helm installation via:

helm upgrade --install strimzi-kafka-operator \
  strimzi/strimzi-kafka-operator \
  --namespace seldon-mesh --create-namespace \

Use with caution!
Currently Kraft installation of Strimzi is not production ready.
See Strimzi [documentation](https://strimzi.io/docs/operators/0.35.0/deploying.html#ref-operator-use-kraft-feature-gate-str) and related GitHub [issue](https://github.com/strimzi/strimzi-kafka-operator/issues/5615) for further details.

Create Kafka cluster in seldon-mesh namespace

helm upgrade seldon-core-v2-kafka kafka/strimzi -n seldon-mesh --install

Note that a specific strimzi operator version is associated with a subset of supported Kafka versions.

Ansible Copy Chapter

We provide automation around the installation of a Kafka cluster for Seldon Core 2 to help with development and testing use cases.
You can follow the steps defined here to install Kafka via ansible.

You can use our Ansible playbooks to install only Strimzi Operator and Kafka cluster by setting extra Ansible vars:

ansible-playbook playbooks/setup-ecosystem.yaml -e full_install=no -e install_kafka=yes

Notes Copy Chapter

• You can check kafka-examples for more details.
• As we are using KRaft, use Kafka version 3.3 or above.
• For security settings check here.

Overview (`scheduler/data-flow/README.md`) Copy Section

Inference Graph Data Flow - Kafka Streams

About Copy Chapter

The data-flow engine is built on top of Kafka Streams.
It implements the operations required to propagate information throughout an inference pipeline in Seldon Core 2.
These operations include multi-way joins, or fan-ins, and conversion of inference v2 response types to request types.

The core idea of asynchronous data flows is that they offer:

• buffering during periods of bursty traffic, e.g. for black-box explainers
• dynamic reconfiguration of pipelines to incorporate advanced monitoring components, e.g. drift detectors
• replay of messages
• auditability

Developer setup Copy Chapter

You will need:

• JDK 17+
• Kotlin 1.6.10+

JDK Copy Section

You can install JDK versions using:

• sdkman
• asdf

To install with asdf, run:

asdf plugin-add java
asdf list-all java | less                   # See available options
ASDF_JAVA_VERSION=graalvm-22.0.0.2+java17   # Or your chosen JDK, `zulu-17.32.13` should also work
asdf install java $ASDF_JAVA_VERSION
asdf global java $ASDF_JAVA_VERSION

GraalVM offers strong performance through intelligent inlining and optimisations.
Zulu is stable, well supported too, and has performant GC behaviour.
Other options, such as Coretto, AdoptOpenJDK, and plain old Oracle OpenJDK also exist.

Kotlin Copy Section

You can install a Kotlin SDK using asdf or sdkman.
Alternatively, you can simply let Gradle do the work... Read on.

Gradle Copy Section

There is a Gradle wrapper included in this repository: ./gradlew.
You can install the necessary build and compilation dependencies by running the build target:

$ ./gradlew build

BUILD SUCCESSFUL in 536ms
6 actionable tasks: 6 up-to-date

You can also run lint check and formatting using ktlint, which are also available as make targets i.e. make lint and make format.

$ ./gradlew ktlintCheck 
$ ./gradlew ktlintFormat

After a successful run, they generate reports in build/reports/ktlint.

Unsupported class file major version

If you see an error like the below, your Gradle version is not high enough:

_BuildScript_' Unsupported class file major version 61

The use of JDK 17 requires Gradle 7.3+.
The default in this project is Gradle 7.4, specified in ./gradle/wrapper/gradle-wrapper.properties.

Check the compatibility matrix for a new enough version
for your Java and Kotlin SDK versions.
You can then update the version in distributionUrl under ./gradle/wrapper/gradle-wrapper.properties.

If you run ./gradlew build again, it should download new dependencies and progress.

If you are in JetBrains IDEA, you will need to close and re-open the project, as it gets confused.

Invalid target release

If you see an error like the below, then there are a few things to check:

> Task :compileJava FAILED
...
Execution failed for task ':compileJava'.
> invalid target release: 17

It could be that your Gradle version is too old to recognise JDK 17+.
In this case, refer to the above section for Unsupported class file major version.

Alternatively, it could be that your JDK version is too low or that Gradle is failing to detect it.
Ensure you have installed a suitable JDK version, then run the below to check which toolchains Gradle is aware of:

$ ./gradlew -q javaToolchains

The output should look similar to the below:

+ Options
     | Auto-detection:     Enabled
     | Auto-download:      Enabled

 + GraalVM Community JDK 17.0.2+8-jvmci-22.0-b05
     | Location:           <home>/.asdf/installs/java/graalvm-22.0.0.2+java17
     | Language Version:   17
     | Vendor:             GraalVM Community
     | Architecture:       amd64
     | Is JDK:             true
     | Detected by:        Current JVM

 + Zulu JDK 17.0.2+8-LTS
     | Location:           <home>/.asdf/installs/java/zulu-17.32.13
     | Language Version:   17
     | Vendor:             Zulu
     | Architecture:       amd64
     | Is JDK:             true
     | Detected by:        asdf-vm

If the expected JDK is not present, check that it is available in your PATH.
If it is present, check the allowed versions and vendors for java.toolchain or kotlin.jvmToolchain in build.gradle.kts.
Ensure your chosen JDK is compatible with these settings.

Unknown JVM target version

If you see an error like the below, your Kotlin version is not high enough.

Task :compileKotlin FAILED
e: Unknown JVM target version: 17

Check for a newer version and update the version in build.gradle.kts:

plugins {
  ...
  kotlin("jvm") version "X.X.X"
}

gRPC protocol buffers Copy Section

Rather than the effort of setting up a Gradle plugin, it is straightforward to use protoc directly.
The steps for Kotlin are documented here.

You will need both the Java and Kotlin extensions for protoc, available at:

• https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-kotlin/
• https://repo1.maven.org/maven2/io/grpc/protoc-gen-grpc-java/

If running manually, the command will look like:

cd $(git rev-parse --show-toplevel)/apis/mlops/chainer
protoc \
  --proto_path=. \
  --plugin=protoc-gen-grpc-java=<path to protoc-gen-grpc-java> \
  --java_out=./kotlin \
  --grpc-java_out=./kotlin \
  --plugin=protoc-gen-grpckt=<path to protoc-gen-grpc-kotlin.sh> \
  --kotlin_out=./kotlin \
  --grpckt_out=./kotlin \
  chainer.proto

Overview (`service-meshes/ambassador/README.md`) Copy Section

Ambassador Notebook Example Copy Chapter

Assumes

• You have installed emissary as per their docs

Tested with

emissary-ingress-7.3.2 insatlled via helm

INGRESS_IP=!kubectl get svc emissary-ingress -n emissary -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
INGRESS_IP=INGRESS_IP[0]
import os
os.environ['INGRESS_IP'] = INGRESS_IP
INGRESS_IP

'172.21.255.1'

Ambassador Single Model Example Copy Section

!kustomize build config/single-model

apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: wildcard
  namespace: seldon-mesh
spec:
  hostname: '*'
  requestPolicy:
    insecure:
      action: Route
---
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: emissary-ingress-listener-8080
  namespace: seldon-mesh
spec:
  hostBinding:
    namespace:
      from: ALL
  port: 8080
  protocol: HTTP
  securityModel: INSECURE
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: iris-grpc
  namespace: seldon-mesh
spec:
  add_request_headers:
    seldon-model:
      value: iris
  grpc: true
  hostname: '*'
  prefix: /inference.GRPCInferenceService
  rewrite: ""
  service: seldon-mesh:80
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: iris-http
  namespace: seldon-mesh
spec:
  add_request_headers:
    seldon-model:
      value: iris
  hostname: '*'
  prefix: /v2/
  rewrite: ""
  service: seldon-mesh:80
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
  namespace: seldon-mesh
spec:
  requirements:
  - sklearn
  storageUri: gs://seldon-models/mlserver/iris

!kustomize build config/single-model | kubectl apply --validate=false -f -

host.getambassador.io/wildcard created
listener.getambassador.io/emissary-ingress-listener-8080 created
mapping.getambassador.io/iris-grpc created
mapping.getambassador.io/iris-http created
model.mlops.seldon.io/iris created

!kubectl wait --for condition=ready --timeout=300s model --all -n seldon-mesh

model.mlops.seldon.io/iris condition met

!curl -v http://${INGRESS_IP}/v2/models/iris/infer -H "Content-Type: application/json"\
        -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

*   Trying 172.21.255.1...
* Connected to 172.21.255.1 (172.21.255.1) port 80 (#0)
> POST /v2/models/iris/infer HTTP/1.1
> Host: 172.21.255.1
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 94
> 
* upload completely sent off: 94 out of 94 bytes
< HTTP/1.1 200 OK
< content-length: 196
< content-type: application/json
< date: Sat, 16 Apr 2022 15:45:43 GMT
< server: envoy
< x-envoy-upstream-service-time: 792
< seldon-route: iris_1
< 
* Connection #0 to host 172.21.255.1 left intact
{"model_name":"iris_1","model_version":"1","id":"72ac79f5-b355-4be3-b8c5-2ebedaa39f60","parameters":null,"outputs":[{"name":"predict","shape":[1],"datatype":"INT64","parameters":null,"data":[2]}]}

!grpcurl -d '{"model_name":"iris","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}' \
    -plaintext \
    -import-path ../../apis \
    -proto ../../apis/mlops/v2_dataplane/v2_dataplane.proto \
    ${INGRESS_IP}:80 inference.GRPCInferenceService/ModelInfer

{
  "modelName": "iris_1",
  "modelVersion": "1",
  "outputs": [
    {
      "name": "predict",
      "datatype": "INT64",
      "shape": [
        "1"
      ],
      "contents": {
        "int64Contents": [
          "2"
        ]
      }
    }
  ]
}

!kustomize build config/single-model | kubectl delete -f -

host.getambassador.io "wildcard" deleted
listener.getambassador.io "emissary-ingress-listener-8080" deleted
mapping.getambassador.io "iris-grpc" deleted
mapping.getambassador.io "iris-http" deleted
model.mlops.seldon.io "iris" deleted

Traffic Split Two Models Copy Section

Currently not working due to this issue

!kustomize build config/traffic-split

apiVersion: getambassador.io/v3alpha1
kind: Host
metadata:
  name: wildcard
  namespace: seldon-mesh
spec:
  hostname: '*'
  requestPolicy:
    insecure:
      action: Route
---
apiVersion: getambassador.io/v3alpha1
kind: Listener
metadata:
  name: emissary-ingress-listener-8080
  namespace: seldon-mesh
spec:
  hostBinding:
    namespace:
      from: ALL
  port: 8080
  protocol: HTTP
  securityModel: INSECURE
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: iris1-grpc
  namespace: seldon-mesh
spec:
  add_request_headers:
    seldon-model:
      value: iris1
  grpc: true
  hostname: '*'
  prefix: /inference.GRPCInferenceService
  rewrite: ""
  service: seldon-mesh:80
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: iris1-http
  namespace: seldon-mesh
spec:
  add_request_headers:
    seldon-model:
      value: iris1
  add_response_headers:
    seldon_model:
      value: iris1
  hostname: '*'
  prefix: /v2
  rewrite: ""
  service: seldon-mesh:80
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: iris2-grpc
  namespace: seldon-mesh
spec:
  add_request_headers:
    seldon-model:
      value: iris2
  grpc: true
  hostname: '*'
  prefix: /inference.GRPCInferenceService
  rewrite: ""
  service: seldon-mesh:80
  weight: 50
---
apiVersion: getambassador.io/v3alpha1
kind: Mapping
metadata:
  name: iris2-http
  namespace: seldon-mesh
spec:
  add_request_headers:
    seldon-model:
      value: iris2
  add_response_headers:
    seldon_model:
      value: iris2
  hostname: '*'
  prefix: /v2
  rewrite: ""
  service: seldon-mesh:80
  weight: 50
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris1
  namespace: seldon-mesh
spec:
  requirements:
  - sklearn
  storageUri: gs://seldon-models/mlserver/iris
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris2
  namespace: seldon-mesh
spec:
  requirements:
  - sklearn
  storageUri: gs://seldon-models/mlserver/iris

!kustomize build config/traffic-split | kubectl apply -f -

host.getambassador.io/wildcard created
listener.getambassador.io/emissary-ingress-listener-8080 created
mapping.getambassador.io/iris1-grpc created
mapping.getambassador.io/iris1-http created
mapping.getambassador.io/iris2-grpc created
mapping.getambassador.io/iris2-http created
model.mlops.seldon.io/iris1 created
model.mlops.seldon.io/iris2 created

!kubectl wait --for condition=ready --timeout=300s model --all -n seldon-mesh

model.mlops.seldon.io/iris1 condition met
model.mlops.seldon.io/iris2 condition met

!curl -v http://${INGRESS_IP}/v2/models/iris/infer -H "Content-Type: application/json" \
        -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

*   Trying 172.21.255.1...
* Connected to 172.21.255.1 (172.21.255.1) port 80 (#0)
> POST /v2/models/iris/infer HTTP/1.1
> Host: 172.21.255.1
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 94
> 
* upload completely sent off: 94 out of 94 bytes
< HTTP/1.1 200 OK
< content-length: 197
< content-type: application/json
< date: Sat, 16 Apr 2022 15:46:17 GMT
< server: envoy
< x-envoy-upstream-service-time: 920
< seldon-route: iris2_1
< seldon_model: iris2
< 
* Connection #0 to host 172.21.255.1 left intact
{"model_name":"iris2_1","model_version":"1","id":"ed521c32-cd85-4cb8-90eb-7c896803f271","parameters":null,"outputs":[{"name":"predict","shape":[1],"datatype":"INT64","parameters":null,"data":[2]}]}

!grpcurl -d '{"model_name":"iris1","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}' \
    -plaintext \
    -import-path ../../apis \
    -proto ../../apis/mlops/v2_dataplane/v2_dataplane.proto \
    ${INGRESS_IP}:80 inference.GRPCInferenceService/ModelInfer

{
  "modelName": "iris2_1",
  "modelVersion": "1",
  "outputs": [
    {
      "name": "predict",
      "datatype": "INT64",
      "shape": [
        "1"
      ],
      "contents": {
        "int64Contents": [
          "2"
        ]
      }
    }
  ]
}

!kustomize build config/traffic-split | kubectl delete -f -

host.getambassador.io "wildcard" deleted
listener.getambassador.io "emissary-ingress-listener-8080" deleted
mapping.getambassador.io "iris1-grpc" deleted
mapping.getambassador.io "iris1-http" deleted
mapping.getambassador.io "iris2-grpc" deleted
mapping.getambassador.io "iris2-http" deleted
model.mlops.seldon.io "iris1" deleted
model.mlops.seldon.io "iris2" deleted

Overview (`service-meshes/istio/README.md`) Copy Section

Istio Notebook Examples Copy Chapter

Assumes

• You have installed istio as per their docs
• You have exposed the ingressgateway as an external loadbalancer

tested with:

istioctl version
1.13.2

istioctl install --set profile=demo -y

INGRESS_IP=!kubectl get svc istio-ingressgateway -n istio-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
INGRESS_IP=INGRESS_IP[0]
import os
os.environ['INGRESS_IP'] = INGRESS_IP
INGRESS_IP

'172.21.255.1'

Istio Single Model Example Copy Section

!kustomize build config/single-model

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
  namespace: seldon-mesh
spec:
  requirements:
  - sklearn
  storageUri: gs://seldon-models/mlserver/iris
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: seldon-gateway
  namespace: seldon-mesh
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      privateKey: /etc/istio/ingressgateway-certs/tls.key
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: iris-route
  namespace: seldon-mesh
spec:
  gateways:
  - istio-system/seldon-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /v2
    name: iris-http
    route:
    - destination:
        host: seldon-mesh.seldon-mesh.svc.cluster.local
      headers:
        request:
          set:
            seldon-model: iris
  - match:
    - uri:
        prefix: /inference.GRPCInferenceService
    name: iris-grpc
    route:
    - destination:
        host: seldon-mesh.seldon-mesh.svc.cluster.local
      headers:
        request:
          set:
            seldon-model: iris

!kustomize build config/single-model | kubectl apply -f -

model.mlops.seldon.io/iris unchanged
gateway.networking.istio.io/seldon-gateway unchanged
virtualservice.networking.istio.io/iris-route configured

!kubectl wait --for condition=ready --timeout=300s model --all -n seldon-mesh

model.mlops.seldon.io/iris condition met

!curl -v http://${INGRESS_IP}/v2/models/iris/infer -H "Content-Type: application/json" \
        -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

*   Trying 172.21.255.1...
* Connected to 172.21.255.1 (172.21.255.1) port 80 (#0)
> POST /v2/models/iris/infer HTTP/1.1
> Host: 172.21.255.1
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 94
> 
* upload completely sent off: 94 out of 94 bytes
< HTTP/1.1 200 OK
< content-length: 196
< content-type: application/json
< date: Sat, 16 Apr 2022 15:34:11 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 802
< seldon-route: iris_1
< 
* Connection #0 to host 172.21.255.1 left intact
{"model_name":"iris_1","model_version":"1","id":"83520c4a-c7f1-4363-9bfd-60c5d8ee2dc5","parameters":null,"outputs":[{"name":"predict","shape":[1],"datatype":"INT64","parameters":null,"data":[2]}]}

!grpcurl -d '{"model_name":"iris","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}' \
    -plaintext \
    -import-path ../../apis \
    -proto ../../apis/mlops/v2_dataplane/v2_dataplane.proto \
    ${INGRESS_IP}:80 inference.GRPCInferenceService/ModelInfer

{
  "modelName": "iris_1",
  "modelVersion": "1",
  "outputs": [
    {
      "name": "predict",
      "datatype": "INT64",
      "shape": [
        "1"
      ],
      "contents": {
        "int64Contents": [
          "2"
        ]
      }
    }
  ]
}

!kustomize build config/single-model | kubectl delete -f -

model.mlops.seldon.io "iris" deleted
gateway.networking.istio.io "seldon-gateway" deleted
virtualservice.networking.istio.io "iris-route" deleted

Traffic Split Two Models Copy Section

!kustomize build config/traffic-split

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris1
  namespace: seldon-mesh
spec:
  requirements:
  - sklearn
  storageUri: gs://seldon-models/mlserver/iris
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris2
  namespace: seldon-mesh
spec:
  requirements:
  - sklearn
  storageUri: gs://seldon-models/mlserver/iris
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: seldon-gateway
  namespace: seldon-mesh
spec:
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: iris-route
  namespace: seldon-mesh
spec:
  gateways:
  - seldon-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /v2/models/iris
    name: iris-http
    route:
    - destination:
        host: seldon-mesh.seldon-mesh.svc.cluster.local
      headers:
        request:
          set:
            seldon-model: iris1
      weight: 50
    - destination:
        host: seldon-mesh.seldon-mesh.svc.cluster.local
      headers:
        request:
          set:
            seldon-model: iris2
      weight: 50
  - match:
    - uri:
        prefix: /inference.GRPCInferenceService
    name: iris-grpc
    route:
    - destination:
        host: seldon-mesh.seldon-mesh.svc.cluster.local
      headers:
        request:
          set:
            seldon-model: iris1
      weight: 50
    - destination:
        host: seldon-mesh.seldon-mesh.svc.cluster.local
      headers:
        request:
          set:
            seldon-model: iris2
      weight: 50

!kustomize build config/traffic-split | kubectl apply -f -

model.mlops.seldon.io/iris1 created
model.mlops.seldon.io/iris2 created
gateway.networking.istio.io/seldon-gateway created
virtualservice.networking.istio.io/iris-route created

!kubectl wait --for condition=ready --timeout=300s model --all -n seldon-mesh

model.mlops.seldon.io/iris1 condition met
model.mlops.seldon.io/iris2 condition met

!curl -v http://${INGRESS_IP}/v2/models/iris/infer -H "Content-Type: application/json" \
        -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

*   Trying 172.21.255.1...
* Connected to 172.21.255.1 (172.21.255.1) port 80 (#0)
> POST /v2/models/iris/infer HTTP/1.1
> Host: 172.21.255.1
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 94
> 
* upload completely sent off: 94 out of 94 bytes
< HTTP/1.1 200 OK
< content-length: 197
< content-type: application/json
< date: Sat, 16 Apr 2022 15:35:01 GMT
< server: istio-envoy
< x-envoy-upstream-service-time: 801
< seldon-route: iris1_1
< 
* Connection #0 to host 172.21.255.1 left intact
{"model_name":"iris1_1","model_version":"1","id":"b54e6d8c-d253-4bb9-bb64-02c2ee49e89f","parameters":null,"outputs":[{"name":"predict","shape":[1],"datatype":"INT64","parameters":null,"data":[2]}]}

!grpcurl -d '{"model_name":"iris1","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}' \
    -plaintext \
    -import-path ../../apis \
    -proto ../../apis/mlops/v2_dataplane/v2_dataplane.proto \
    ${INGRESS_IP}:80 inference.GRPCInferenceService/ModelInfer

{
  "modelName": "iris1_1",
  "modelVersion": "1",
  "outputs": [
    {
      "name": "predict",
      "datatype": "INT64",
      "shape": [
        "1"
      ],
      "contents": {
        "int64Contents": [
          "2"
        ]
      }
    }
  ]
}

!kustomize build config/traffic-split | kubectl delete -f -

model.mlops.seldon.io "iris1" deleted
model.mlops.seldon.io "iris2" deleted
gateway.networking.istio.io "seldon-gateway" deleted
virtualservice.networking.istio.io "iris-route" deleted

Overview (`service-meshes/traefik/README.md`) Copy Section

Traefik Examples Copy Chapter

Assumes

• You have installed Traefik as per their docs into namespace traefik-v2

Tested with traefik-10.19.4

INGRESS_IP=!kubectl get svc traefik -n traefik-v2 -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
INGRESS_IP=INGRESS_IP[0]
import os
os.environ['INGRESS_IP'] = INGRESS_IP
INGRESS_IP

'172.21.255.1'

Traefik Single Model Example Copy Section

!kustomize build config/single-model

apiVersion: v1
kind: Service
metadata:
  name: myapps
  namespace: seldon-mesh
spec:
  ports:
  - name: web
    port: 80
    protocol: TCP
  selector:
    app: traefik-ingress-lb
  type: LoadBalancer
---
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
  namespace: seldon-mesh
spec:
  requirements:
  - sklearn
  storageUri: gs://seldon-models/mlserver/iris
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: iris
  namespace: seldon-mesh
spec:
  entryPoints:
  - web
  routes:
  - kind: Rule
    match: PathPrefix(`/`)
    middlewares:
    - name: iris-header
    services:
    - name: seldon-mesh
      port: 80
      scheme: h2c
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: iris-header
  namespace: seldon-mesh
spec:
  headers:
    customRequestHeaders:
      seldon-model: iris

!kustomize build config/single-model | kubectl apply -f -

service/myapps created
model.mlops.seldon.io/iris created
ingressroute.traefik.containo.us/iris created
middleware.traefik.containo.us/iris-header created

!kubectl wait --for condition=ready --timeout=300s model --all -n seldon-mesh

model.mlops.seldon.io/iris condition met

!curl -v http://${INGRESS_IP}/v2/models/iris/infer -H "Content-Type: application/json" \
        -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

*   Trying 172.21.255.1...
* Connected to 172.21.255.1 (172.21.255.1) port 80 (#0)
> POST /v2/models/iris/infer HTTP/1.1
> Host: 172.21.255.1
> User-Agent: curl/7.47.0
> Accept: */*
> Content-Type: application/json
> Content-Length: 94
> 
* upload completely sent off: 94 out of 94 bytes
< HTTP/1.1 200 OK
< Content-Length: 196
< Content-Type: application/json
< Date: Sat, 16 Apr 2022 15:53:27 GMT
< Seldon-Route: iris_1
< Server: envoy
< X-Envoy-Upstream-Service-Time: 895
< 
* Connection #0 to host 172.21.255.1 left intact
{"model_name":"iris_1","model_version":"1","id":"0dccf477-78fa-4a11-92ff-4d7e4f1cdda8","parameters":null,"outputs":[{"name":"predict","shape":[1],"datatype":"INT64","parameters":null,"data":[2]}]}

!grpcurl -d '{"model_name":"iris","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}' \
    -plaintext \
    -import-path ../../apis \
    -proto ../../apis/mlops/v2_dataplane/v2_dataplane.proto \
    ${INGRESS_IP}:80 inference.GRPCInferenceService/ModelInfer

{
  "modelName": "iris_1",
  "modelVersion": "1",
  "outputs": [
    {
      "name": "predict",
      "datatype": "INT64",
      "shape": [
        "1"
      ],
      "contents": {
        "int64Contents": [
          "2"
        ]
      }
    }
  ]
}

!kustomize build config/single-model | kubectl delete -f -

service "myapps" deleted
model.mlops.seldon.io "iris" deleted
ingressroute.traefik.containo.us "iris" deleted
middleware.traefik.containo.us "iris-header" deleted

Overview (`tests/k6/README.md`) Copy Section

K6 Load Testing

Available environment variables can be found in components/settings.js

Setup Copy Chapter

• Install k6 (load driver)
  • link
  • Alternatively you can build a docker image that includes k6 via: make docker-build
• (Optional) Install kube api make xk6-install, remember to use the local folder k6 binary if kube apis are required
• Local testing
  • Deploy the system using docker compose via: make deploy-local

Examples Copy Chapter

• run tfsimple model
• output http debug to file log.txt
• Only load models and infer (no unload)
• run 500 model loads with 10 concurrent VUs

INFER_HTTP_ITERATIONS=10 SKIP_UNLOAD_MODEL=1 MODEL_TYPE="tfsimple" k6 run -u 10 -i 500 --http-debug --log-output=stdout scenarios/load_predict_unload.js > log.txt

For isolated agent test: Copy Chapter

• start seldon-core with host networking via: make -C scheduler start-all-host
• hit mlserver directly

SCHEDULER_ENDPOINT=0.0.0.0:9004 ENVOY="false" INFER_GRPC_ENDPOINT=0.0.0.0:8081 INFER_HTTP_ENDPOINT=http://0.0.0.0:8080 INFER_HTTP_ITERATIONS=1 INFER_GRPC_ITERATIONS=1 MODEL_TYPE="iris" MAX_NUM_MODELS=10 k6 run -u 1 -i 10 --http-debug scenarios/infer_constant_rate.js

• hit agent reverse proxy, just change the ports for the inference endpoints

SCHEDULER_ENDPOINT=0.0.0.0:9004 ENVOY="false" INFER_GRPC_ENDPOINT=0.0.0.0:9998 INFER_HTTP_ENDPOINT=http://0.0.0.0:9999 INFER_HTTP_ITERATIONS=1 INFER_GRPC_ITERATIONS=1 MODEL_TYPE="iris" MAX_NUM_MODELS=10 k6 run -u 1 -i 10 --http-debug scenarios/infer_constant_rate.js

Note: check ports in scheduler/env.all

For k8s you will need to update the default endpoints to the services exposed, e.g.

MODEL_TYPE="tfsimple" SCHEDULER_ENDPOINT=172.18.255.4:9004 INFER_GRPC_ENDPOINT=172.18.255.3:80 INFER_HTTP_ENDPOINT=http://172.18.255.3 k6 run -u 5 -i 50 scenarios/load_predict_unload.js

Constant Throughput Test Copy Chapter

Run against model name iris which is of type iris against a envoy http ip as given.

MODEL_NAME="iris" MODEL_TYPE="iris" INFER_HTTP_ENDPOINT="http://172.31.255.9" k6 run scenarios/model_constant_rate.js

Run locally but with grpc

INFER_TYPE="grpc" MODEL_TYPE="iris" k6 run scenarios/model_constant_rate.js

Note: there are more scenarios in scenarios and models in components/model.js

Overview (`apis/mlops/agent/README.md`) Copy Section

Agent Service

• Allow scheduler to ask agent to load or unload a model.
• Allow agent to tell scheduler a model has been loaded or unloaded.

Overview (`apis/mlops/scheduler/README.md`) Copy Section

Scheduler Service

• Allow a model server to be added with given replica endpoints
• Allow a model server to be removed
• Allow a model to be started on a server
• Allow a model to be removed

Overview (`docs-gb/apis/inference/README.md`) Copy Section

A comprehensive guide to Seldon's Open Inference Protocol API specification for standardized inference server implementations. Copy Chapter

Inference

Learn about the API specification implemented in all Seldon inference servers. The specification ensures consistency and interoperability across different inference server implementations.

Seldon inference servers must respect the following API specification.

• Seldon, KServe, NVIDIA V2 Inference API Spec

In future, Seldon may provide extensions for use with Pipelines, Experiments and Explainers.

Overview (`docs-gb/apis/internal/README.md`) Copy Section

Internal APIs

Seldon provides some internal gRPC services it uses to manage components.

• Agent API
• Chainer API

Overview (`docs-gb/getting-started/kubernetes-installation/README.md`) Copy Section

Kubernetes Installation

Prerequisites Copy Chapter

• Ensure that the version of the Kubernetes cluster meets the requirement listed below. You can create a KinD cluster on your local computer for testing with Ansible.
• Install the ecosystem components using Ansible.

Core 2 Dependencies Copy Chapter

Here is a list of components that Seldon Core 2 depends on, with minimum and maximum supported versions.

* These components are shipped as part of Seldon Core 2 docker images set, users should not install them separately but they need to be aware of the configuration options that are supported by these versions.
** Kafka is only required to operate Seldon Core 2 dataflow Pipelines. If not required then users should not install seldon-modelgateway, seldon-pipelinegateway, and seldon-dataflow-engine.
*** Not hard limit on the maximum version to be used.

Install Ecosystem Components Copy Chapter

You also need to install our ecosystem components. For this we provide directions for Ansible to install these.

Install Copy Section

To install Seldon Core 2 from the source repository, you can choose one of the following methods:

• Helm (recommended for production systems)
• Ansible (recommended for testing, development, or trial)

The Kubernetes operator that is installed runs in namespaced mode so any resources you create
need to be in the same namespace as you installed into.

Kustomize Copy Section

Seldon recommends installing Seldon Core 2 using Helm or Ansible. If you prefer to use Kustomize, you can base your configuration on the raw YAML files you generate in the k8s/yaml folder. Alternatively, you can follow the steps in the k8s/Makefile, which demonstrates how to build the YAML files from the Kustomize bases.

Operations Copy Chapter

• Security

Overview (`docs-gb/installation/advanced-configurations/README.md`) Copy Section

Advanced Configurations

Overview (`docs-gb/installation/helm/README.md`) Copy Section

Helm Configuration Options

Seldon Core 2 provides a highly configurable deployment framework that allows you to fine-tune various components using Helm configuration options. These options offer control over deployment behavior, resource management, logging, autoscaling, and model lifecycle policies to optimize the performance and scalability of machine learning deployments.

This section details the key Helm configuration parameters for Envoy, Autoscaling, Server Prestop, and Model Control Plane, ensuring that you can customize deployment workflows and enhance operational efficiency.

• Envoy: Manage pre-stop behaviors and configure access logging to track request-level interactions.
• Autoscaling (Experimental): Fine-tune dynamic scaling policies for efficient resource allocation based on real-time inference workloads.
• Servers: Define grace periods for controlled shutdowns and optimize model control plane parameters for efficient model loading, unloading, and error handling.
• Logging: Define log levels for the different components of the system.

Envoy Copy Chapter

Prestop Copy Section

Access Log Copy Section

Autoscaling Copy Chapter

Native autoscaling (experimental) Copy Section

Server Copy Chapter

Prestop Copy Section

Model Control Plane Copy Section

Logging Copy Chapter

Component Log Level Copy Section

Notes:

• We set kafka client library log level from the log level that is passed to the component, which could be different to the level expected by librdkafka (syslog level). In this case we attempt to map the log level value to the best match.

Overview (`docs-gb/installation/learning-environment/README.md`) Copy Section

Learning Environment

You can install Seldon Core 2 on your local computer that is running a Kubernetes cluster using kind.

Seldon publishes the Helm charts that are required to install Seldon Core 2. For more information about the Helm charts and the related dependencies,see Helm charts and Dependencies.

{% hint style="info" %}
Note: These instructions guide you through installing Seldon Core 2 on a local Kubernetes cluster, focusing on ease of learning. Ensure your kind cluster is running on hardware with at least 32GB of RAM and a load balancer such as MetalLB is configured.
{% endhint %}

Prerequisites Copy Chapter

• Install a Kubernetes cluster that is running version 1.27 or later.
• Install kubectl, the Kubernetes command-line tool.
• Install Helm, the package manager for Kubernetes or Ansible, the automation tool used for provisioning, configuration management, and application deployment.

{% hint style="info" %}
Note: Ansible automates provisioning, configuration management, and handles all dependencies required for Seldon Core 2.
With Helm, you need to configure and manage the dependencies yourself.
{% endhint %}

Installing Seldon Core 2 Copy Chapter

{% tabs %}
{% tab title="Helm" %}

1. Create a namespace to contain the main components of Seldon Core 2. For example, create the seldon-mesh namespace.

   bash Copy

   kubectl create ns seldon-mesh || echo "Namespace seldon-mesh already exists"

2. Add and update the Helm charts, seldon-charts, to the repository.

   bash Copy

   helm repo add seldon-charts https://seldonio.github.io/helm-charts/
   helm repo update seldon-charts

3. Install Custom resource definitions for Seldon Core 2.

   bash Copy

   helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
   --namespace default \
   --install

4. Install Seldon Core 2 operator in the seldon-mesh namespace.

   bash Copy

   helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
    --namespace seldon-mesh --set controller.clusterwide=true \
    --install

   This configuration installs the Seldon Core 2 operator across an entire Kubernetes cluster. To perform cluster-wide operations, create ClusterRoles and ensure your user has the necessary permissions during deployment. With cluster-wide operations, you can create SeldonRuntimes in any namespace.

   You can configure the installation to deploy the Seldon Core 2 operator in a specific namespace so that it control resources in the provided namespace. To do this, set controller.clusterwide to false.

5. Install Seldon Core 2 runtimes in the seldon-mesh namespace.

   bash Copy

   helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
   --namespace seldon-mesh \
   --install

6. Install Seldon Core 2 servers in the seldon-mesh namespace. Two example servers named mlserver-0, and triton-0 are installed so that you can load the models to these servers after installation.

   bash Copy

   helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
    --namespace seldon-mesh \
    --install

7. Check Seldon Core 2 operator, runtimes, servers, and CRDS are installed in the seldon-mesh namespace. It might take a couple of minutes for all the Pods to be ready. To check the status of the Pods in real time use this command: kubectl get pods -w -n seldon-mesh.

   bash Copy

   kubectl get pods -n seldon-mesh

   The output should be similar to this:

   text Copy

   NAME                                            READY   STATUS             RESTARTS      AGE
   hodometer-749d7c6875-4d4vw                      1/1     Running            0             4m33s
   mlserver-0                                      3/3     Running            0             4m10s
   seldon-dataflow-engine-7b98c76d67-v2ztq         0/1     CrashLoopBackOff   5 (49s ago)   4m33s
   seldon-envoy-bb99f6c6b-4mpjd                    1/1     Running            0             4m33s
   seldon-modelgateway-5c76c7695b-bhfj5            1/1     Running            0             4m34s
   seldon-pipelinegateway-584c7d95c-bs8c9          1/1     Running            0             4m34s
   seldon-scheduler-0                              1/1     Running            0             4m34s
   seldon-v2-controller-manager-5dd676c7b7-xq5sm   1/1     Running            0             4m52s
   triton-0                                        2/3     Running            0             4m10s

{% hint style="info" %}
Note: Pods with names starting with seldon-dataflow-engine, seldon-pipelinegateway, and seldon-modelgateway may generate log errors until they successfully connect to Kafka. This occurs because Kafka is not yet fully integrated with Seldon Core 2.
{% endhint %}
{% endtab %}

{% tab title="Ansible" %}
{% hint style="info" %}
Note: For more information about configurations, see the supported versions of Python libraries, and customization options
{% endhint %}

You can install Seldon Core 2 and its components using Ansible in one of the following methods:

• Single command
• Multiple commands

Single command

To install Seldon Core 2 into a new local kind Kubernetes cluster, you can use the seldon-all playbook with a single command:

ansible-playbook playbooks/seldon-all.yaml

This creates a kind cluster and installs ecosystem dependencies such kafka,
Prometheus, OpenTelemetry, and Jaeger as well as all the seldon-specific components.
The seldon components are installed using helm-charts from the current git
checkout (../k8s/helm-charts/).

Internally this runs, in order, the following playbooks:

• kind-cluster.yaml
• setup-ecosystem.yaml
• setup-seldon.yaml

You may pass any of the additonal variables which are configurable for those playbooks to seldon-all.

For example:

ansible-playbook playbooks/seldon-all.yaml -e seldon_mesh_namespace=my-seldon-mesh -e install_prometheus=no -e @playbooks/vars/set-custom-images.yaml

Running the playbooks individually gives you more control over what and when it runs. For example, if you want to install into an existing k8s cluster.

Multiple commands

1. Create a kind cluster.

   bash Copy

   ansible-playbook playbooks/kind-cluster.yaml

2. Setup ecosystem.

   bash Copy

   ansible-playbook playbooks/setup-ecosystem.yaml

   Seldon runs by default in the seldon-mesh namespace and a Jaeger pod and OpenTelemetry collector are installed in the namespace.
   To install in a different <mynamespace> namespace:

   bash Copy

   ansible-playbook playbooks/setup-ecosystem.yaml -e seldon_mesh_namespace=<mynamespace>

3. Install Seldon Core 2 in the ansible/ folder.

   bash Copy

   ansible-playbook playbooks/setup-seldon.yaml

   To install in a different namespace, <mynamespace>.

   bash Copy

   ansible-playbook playbooks/setup-seldon.yaml -e seldon_mesh_namespace=<mynamespace>

{% endtab %}
{% endtabs %}

Next Steps Copy Chapter

If you installed Seldon Core 2 using Helm, you need to complete the installation of other components in the following order:

1. Integrating with Kafka
2. Installing a Service mesh

Overview (`docs-gb/installation/production-environment/README.md`) Copy Section

Prerequisites Copy Chapter

• Set up and connect to a Kubernetes cluster running version 1.27 or later. For instructions on connecting to your Kubernetes cluster, refer to the documentation provided by your cloud provider.
• Install kubectl, the Kubernetes command-line tool.
• Install Helm, the package manager for Kubernetes.

To use Seldon Core 2 in a production environment:

1. Create namespaces
2. Install Seldon Core 2

Seldon publishes the Helm charts that are required to install Seldon Core 2. For more information about the Helm charts and the related dependencies, see Helm charts and Dependencies.

Creating Namespaces Copy Chapter

• Create a namespace to contain the main components of Seldon Core 2. For example, create the namespace seldon-mesh:

  bash Copy

  kubectl create ns seldon-mesh || echo "Namespace seldon-mesh already exists"

• Create a namespace to contain the components related to monitoring. For example, create the namespace seldon-monitoring:

  bash Copy

  kubectl create ns seldon-monitoring || echo "Namespace seldon-monitoring already exists"

Installing Seldon Core 2 Copy Chapter

1. Add and update the Helm charts seldon-charts to the repository.

   bash Copy

   helm repo add seldon-charts https://seldonio.github.io/helm-charts/
   helm repo update seldon-charts

2. Install custom resource definitions for Seldon Core 2.

   bash Copy

   helm upgrade seldon-core-v2-crds seldon-charts/seldon-core-v2-crds \
   --namespace default \
   --install

3. Install Seldon Core 2 operator in the seldon-mesh namespace.

   bash Copy

   helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
    --namespace seldon-mesh --set controller.clusterwide=true \
    --install

   This configuration installs the Seldon Core 2 operator across an entire Kubernetes cluster. To perform cluster-wide operations, create ClusterRoles and ensure your user has the necessary permissions during deployment. With cluster-wide operations, you can create SeldonRuntimes in any namespace.

   With cluster-wide installation, you can specify the namespaces to watch by setting controller.watchNamespaces to a comma-separated list of namespaces (e.g., {ns1, ns2}). This allows the Seldon Core 2 operator to monitor and manage resources in those namespaces.

   You can also install multiple operators in different namespaces, and configure them to watch a disjoint set of namespaces. For example, you can install two operators in op-ns1 and op-ns2, and configure them to watch ns1, ns2 and ns3, ns4, respectively, using the following commands:

   bash Copy

   for ns in op-ns1 op-ns2 ns1 ns2 ns3 ns4; do kubectl create ns "$ns"; done

   We now install the first operator in op-ns1 and configure it to watch ns1 and ns2:

   bash Copy

   helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
   --namespace op-ns1 \
   --set controller.clusterwide=true \
   --set "controller.watchNamespaces={ns1,ns2}" \
   --install

   Next, we install the second operator in op-ns2 and configure it to watch ns3 and ns4:

   bash Copy

   helm upgrade seldon-core-v2-setup seldon-charts/seldon-core-v2-setup \
   --namespace op-ns2 \
   --set controller.clusterwide=true \
   --set "controller.watchNamespaces={ns3,ns4}" \
   --set controller.skipClusterRoleCreation=true \
   --install

   Note that the second operator is installed with skipClusterRoleCreation=true to avoid re-creating the ClusterRole and ClusterRoleBinding that were created by the first operator.

   Finally, you can configure the installation to deploy the Seldon Core 2 operator in a specific namespace so that it control resources in the provided namespace. To do this, set controller.clusterwide to false.

4. Install Seldon Core 2 runtimes in the namespace seldon-mesh.

   bash Copy

   helm upgrade seldon-core-v2-runtime seldon-charts/seldon-core-v2-runtime \
   --namespace seldon-mesh \
   --install

5. Install Seldon Core 2 servers in the namespace seldon-mesh. Two example servers named mlserver-0, and triton-0 are installed so that you can load the models to these servers after installation.

   bash Copy

   helm upgrade seldon-core-v2-servers seldon-charts/seldon-core-v2-servers \
    --namespace seldon-mesh \
    --install

6. Check Seldon Core 2 operator, runtimes, servers, and CRDS are installed in the namespace seldon-mesh:

   bash Copy

   kubectl get pods -n seldon-mesh

   The output should be similar to this:

   text Copy

   NAME                                            READY   STATUS             RESTARTS      AGE
   hodometer-749d7c6875-4d4vw                      1/1     Running            0             4m33s
   mlserver-0                                      3/3     Running            0             4m10s
   seldon-dataflow-engine-7b98c76d67-v2ztq         0/1     CrashLoopBackOff   5 (49s ago)   4m33s
   seldon-envoy-bb99f6c6b-4mpjd                    1/1     Running            0             4m33s
   seldon-modelgateway-5c76c7695b-bhfj5            1/1     Running            0             4m34s
   seldon-pipelinegateway-584c7d95c-bs8c9          1/1     Running            0             4m34s
   seldon-scheduler-0                              1/1     Running            0             4m34s
   seldon-v2-controller-manager-5dd676c7b7-xq5sm   1/1     Running            0             4m52s
   triton-0                                        2/3     Running            0             4m10s

{% hint style="info" %}
Note: Pods with names starting with seldon-dataflow-engine, seldon-pipelinegateway, and seldon-modelgateway may generate log errors until they successfully connect to Kafka. This occurs because Kafka is not yet fully integrated with Seldon Core 2.
{% endhint %}

Next steps Copy Chapter

You can integrate Seldon Core 2 with Kafka that is self-hosted or a managed Kafka.

Additional Resources Copy Chapter

• Seldon Enterprise Documentation
• GKE Documentation
• AWS Documentation
• Azure Documentation

Overview (`docs-gb/kubernetes/confluent/README.md`) Copy Section

Confluent

Seldon Core 2 leverages Kafka for implementing data-centric inference Pipelines. This also means that Core 2 users have the ability of integrating with other products from the Kafka ecosystem, such as Schema Registry, Connect or ksqlDB in order to build custom data streaming, processing and logging solutions for machine learning workflows.

• Schema Registry

Overview (`docs-gb/kubernetes/resources/README.md`) Copy Section

Resources

For Kubernetes usage we provide a set of custom resources for interacting with Seldon.

• SeldonRuntime - for installing Seldon in a particular namespace.
• Servers - for deploying sets of replicas of core inference servers (MLServer or Triton).
• Models - for deploying single machine learning models, custom transformation logic, drift detectors, outliers detectors and explainers.
• Experiments - for testing new versions of models.
• Pipelines - for connecting together flows of data between models.

Advanced Customization Copy Chapter

SeldonConfig and ServerConfig define the core installation configuration and machine learning inference server
configuration for Seldon. Normally, you would not need to customize these but this may be required for
your particular custom installation within your organisation.

• ServerConfigs - for defining new types of inference server that can
  be reference by a Server resource.
• SeldonConfig - for defining how seldon is installed

Overview (`docs-gb/kubernetes/service-meshes/README.md`) Copy Section

Service Meshes

The Seldon models and pipelines are exposed via a single service endpoint in the install
namespace called seldon-mesh. All models, pipelines and experiments can be reached via
this single Service endpoint by setting appropriate headers on the inference REST/gRPC
request. By this means Seldon is agnostic to any service mesh you may wish to use in your
organisation. We provide some example integrations for some example service meshes below
(alphabetical order):

• Ambassador
• Istio
• Traefik

We welcome help to extend these to other service meshes.

Overview (`docs-gb/models/parameterized-models/README.md`) Copy Section

Parameterized Models

The Model specification allows parameters to be passed to the loaded model to allow customization. For example:

# samples/models/choice1.yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: choice-is-one
spec:
  storageUri: "gs://seldon-models/scv2/examples/pandasquery"
  requirements:
  - mlserver
  - python
  parameters:
  - name: query
    value: "choice == 1"

This capability is only available for MLServer custom model runtimes. The named keys and
values will be added to the model-settings.json file for the provided model in theparameters.extra Dict. MLServer models are able to read these values in their load method.

Example Parameterized Models Copy Chapter

• Pandas Query

Overview (`docs-gb/performance-tuning/models/README.md`) Copy Section

Model Performance Tuning

This section covers various aspects of optimizing model performance in Seldon Core 2, from initial load testing to infrastructure setup and inference optimization. Each subsection provides detailed guidance on different aspects of model performance tuning:

[Load Testing](load-testing.md) Copy Chapter

Learn how to conduct effective load testing to understand your model's performance characteristics:

• Determining load saturation points
• Understanding closed-loop vs. open-loop testing
• Determining the right number of replicas based on your configuration (model, infrastructure, etc.)
• Setting up reproducible test environments
• Interpreting test results for autoscaling configuration

[Inference](inference.md) Copy Chapter

Explore different approaches to optimize inference performance:

• Choosing between gRPC and REST protocols
• Implementing adaptive batching
• Optimizing input dimensions
• Configuring parallel processing with workers
• Understanding CPU vs. GPU utilization
• Optimizing your model artefact

[Infrastructure Setup](infrastructure-setup.md) Copy Chapter

Understand how to configure the underlying infrastructure for optimal model performance:

• Choosing between CPU and GPU deployments
• Setting appropriate CPU specifications
• Configuring thread affinity
• Managing memory allocation
• Optimizing resource utilization

Each of these aspects plays a crucial role in achieving optimal model performance. We recommend starting with load testing to establish a baseline, then using the insights gained to inform your infrastructure setup and inference optimization strategies.

Overview (`docs-gb/performance-tuning/pipelines/README.md`) Copy Section

Pipeline Performance Tuning

This section covers various aspects of optimizing pipeline performance in Seldon Core 2, from testing methodologies to Core 2 configuration. Each subsection provides detailed guidance on different aspects of pipeline performance tuning:

[Testing Pipelines](testing-pipelines.md) Copy Chapter

Learn how to effectively test and optimize pipeline performance:

• Understanding pipeline latency components
• Identifying and reducing bottlenecks
• Balancing model replicas and resources

[Core 2 Configuration](core-2-configuration.md) Copy Chapter

Explore how to configure Core 2 components for optimal pipeline performance:

• Understanding Core 2 data processing components
• Optimizing Kafka integration
• Configuring Core 2 services

[Scalability of Pipelines](scalability-pipelines.md) Copy Chapter

Understand how Core 2 components scale with the number of deployed pipelines and models:

• Dynamic scaling of dataflow engine, model gateway, and pipeline gateway
• Loading and unloading of models and pipelines
• Assignment of pipelines and models to replicas

Each of these aspects plays a crucial role in achieving optimal pipeline performance. We recommend starting with testing individual models in your pipeline, then using those insights to inform your Core 2 configuration and overall pipeline optimization strategies.

Overview (`docs-gb/user-guide/data-science-monitoring/README.md`) Copy Section

Data Science Monitoring

Overview (`docs-gb/user-guide/inference/README.md`) Copy Section

Inference

Overview (`docs-gb/user-guide/v2/README.md`) Copy Section

Open Inference Protocol

This page describes a predict/inference API independent of any
specific ML/DL framework and model server. These APIs are
able to support both easy-to-use and high-performance use cases.
By implementing this protocol both
inference clients and servers will increase their utility and
portability by being able to operate seamlessly on platforms that have
standardized around this API. This protocol is endorsed by NVIDIA
Triton Inference Server, TensorFlow Serving, and ONNX Runtime
Server. It is sometimes referred to by its old name "V2 Inference Protocol".

For an inference server to be compliant with this protocol the server
must implement all APIs described below, except where an optional
feature is explicitly noted. A compliant inference server may choose
to implement either or both of the HTTP/REST API and the GRPC API.

The protocol supports an extension mechanism as a required part of the
API, but this document does not propose any specific extensions. Any
specific extensions will be proposed separately.

HTTP/REST Copy Chapter

A compliant server must implement the health, metadata, and inference
APIs described in this section.

The HTTP/REST API uses JSON because it is widely supported and
language independent. In all JSON schemas shown in this document
$number, $string, $boolean, $object and $array refer to the
fundamental JSON types. #optional indicates an optional JSON field.

All strings in all contexts are case-sensitive.

For Seldon a server must recognize the following URLs. The
versions portion of the URL is shown as optional to allow
implementations that don’t support versioning or for cases when the
user does not want to specify a specific model version (in which case
the server will choose a version based on its own policies).

Health:

GET v2/health/live
GET v2/health/ready
GET v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]/ready

Server Metadata:

GET v2

Model Metadata:

GET v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]

Inference:

POST v2/models/${MODEL_NAME}[/versions/${MODEL_VERSION}]/infer

Health Copy Section

A health request is made with an HTTP GET to a health endpoint. The
HTTP response status code indicates a boolean result for the health
request. A 200 status code indicates true and a 4xx status code
indicates false. The HTTP response body should be empty. There are
three health APIs.

Server Live

The “server live” API indicates if the inference server is able to
receive and respond to metadata and inference requests. The “server
live” API can be used directly to implement the Kubernetes
livenessProbe.

Server Ready

The “server ready” health API indicates if all the models are ready
for inferencing. The “server ready” health API can be used directly to
implement the Kubernetes readinessProbe.

Model Ready

The “model ready” health API indicates if a specific model is ready
for inferencing. The model name and (optionally) version must be
available in the URL. If a version is not provided the server may
choose a version based on its own policies.

Server Metadata Copy Section

The server metadata endpoint provides information about the server. A
server metadata request is made with an HTTP GET to a server metadata
endpoint. In the corresponding response the HTTP body contains theServer Metadata Response JSON Object
or theServer Metadata Response JSON Error Object.

Server Metadata Response JSON Object

A successful server metadata request is indicated by a 200 HTTP status
code. The server metadata response object, identified as$metadata_server_response, is returned in the HTTP body.

$metadata_server_response =
{
  "name" : $string,
  "version" : $string,
  "extensions" : [ $string, ... ]
}

• “name” : A descriptive name for the server.
• "version" : The server version.
• “extensions” : The extensions supported by the server. Currently no
  standard extensions are defined. Individual inference servers may
  define and document their own extensions.

Server Metadata Response JSON Error Object

A failed server metadata request must be indicated by an HTTP error
status (typically 400). The HTTP body must contain the$metadata_server_error_response object.

$metadata_server_error_response =
{
  "error": $string
}

• “error” : The descriptive message for the error.

Model Metadata Copy Section

The per-model metadata endpoint provides information about a model. A
model metadata request is made with an HTTP GET to a model metadata
endpoint. In the corresponding response the HTTP body contains theModel Metadata Response JSON Object
or theModel Metadata Response JSON Error Object.
The model name and (optionally) version must be available in the
URL. If a version is not provided the server may choose a version
based on its own policies or return an error.

Model Metadata Response JSON Object

A successful model metadata request is indicated by a 200 HTTP status
code. The metadata response object, identified as$metadata_model_response, is returned in the HTTP body for every
successful model metadata request.

$metadata_model_response =
{
  "name" : $string,
  "versions" : [ $string, ... ] #optional,
  "platform" : $string,
  "inputs" : [ $metadata_tensor, ... ],
  "outputs" : [ $metadata_tensor, ... ]
}

• “name” : The name of the model.
• "versions" : The model versions that may be explicitly requested via
  the appropriate endpoint. Optional for servers that don’t support
  versions. Optional for models that don’t allow a version to be
  explicitly requested.
• “platform” : The framework/backend for the model. SeePlatforms.
• “inputs” : The inputs required by the model.
• “outputs” : The outputs produced by the model.

Each model input and output tensors’ metadata is described with a$metadata_tensor object.

$metadata_tensor =
{
  "name" : $string,
  "datatype" : $string,
  "shape" : [ $number, ... ]
}

• “name” : The name of the tensor.
• "datatype" : The data-type of the tensor elements as defined inTensor Data Types.
• "shape" : The shape of the tensor. Variable-size dimensions are
  specified as -1.

Model Metadata Response JSON Error Object

A failed model metadata request must be indicated by an HTTP error
status (typically 400). The HTTP body must contain the$metadata_model_error_response object.

$metadata_model_error_response =
{
  "error": $string
}

• “error” : The descriptive message for the error.

Inference Copy Section

An inference request is made with an HTTP POST to an inference
endpoint. In the request the HTTP body contains theInference Request JSON Object. In
the corresponding response the HTTP body contains theInference Response JSON Object orInference Response JSON Error Object. SeeInference Request Examples for some
example HTTP/REST requests and responses.

Inference Request JSON Object

The inference request object, identified as $inference_request, is
required in the HTTP body of the POST request. The model name and
(optionally) version must be available in the URL. If a version is not
provided the server may choose a version based on its own policies or
return an error.

$inference_request =
{
  "id" : $string #optional,
  "parameters" : $parameters #optional,
  "inputs" : [ $request_input, ... ],
  "outputs" : [ $request_output, ... ] #optional
}

• id : An identifier for this request. Optional, but if specified
  this identifier must be returned in the response.
• parameters : An object containing zero or more parameters for this
  inference request expressed as key/value pairs. SeeParameters for more information.
• inputs : The input tensors. Each input is described using the$request_input schema defined in Request Input.
• outputs : The output tensors requested for this inference. Each
  requested output is described using the $request_output schema
  defined in Request Output. Optional, if not
  specified all outputs produced by the model will be returned using
  default $request_output settings.

Request Input

The $request_input JSON describes an input to the model. If the
input is batched, the shape and data must represent the full shape and
contents of the entire batch.

$request_input =
{
  "name" : $string,
  "shape" : [ $number, ... ],
  "datatype"  : $string,
  "parameters" : $parameters #optional,
  "data" : $tensor_data
}

• "name": The name of the input tensor.
• "shape": The shape of the input tensor. Each dimension must be an
  integer representable as an unsigned 64-bit integer value.
• "datatype": The data-type of the input tensor elements as defined
  in Tensor Data Types.
• "parameters": An object containing zero or more parameters for this
  input expressed as key/value pairs. See Parameters
  for more information.
• “data”: The contents of the tensor. See Tensor Data
  for more information.

Request Output

The $request_output JSON is used to request which output tensors
should be returned from the model.

$request_output =
{
  "name" : $string,
  "parameters" : $parameters #optional,
}

• "name": The name of the output tensor.
• "parameters": An object containing zero or more parameters for this
  output expressed as key/value pairs. See Parameters
  for more information.

Inference Response JSON Object

A successful inference request is indicated by a 200 HTTP status
code. The inference response object, identified as$inference_response, is returned in the HTTP body.

$inference_response =
{
  "model_name" : $string,
  "model_version" : $string #optional,
  "id" : $string,
  "parameters" : $parameters #optional,
  "outputs" : [ $response_output, ... ]
}

• "model_name": The name of the model used for inference.
• "model_version": The specific model version used for
  inference. Inference servers that do not implement versioning should
  not provide this field in the response.
• "id": The "id" identifier given in the request, if any.
• "parameters": An object containing zero or more parameters for this
  response expressed as key/value pairs. See Parameters
  for more information.
• "outputs": The output tensors. Each output is described using the$response_output schema defined inResponse Output.

Response Output

The $response_output JSON describes an output from the model. If the
output is batched, the shape and data represents the full shape of the
entire batch.

$response_output =
{
  "name" : $string,
  "shape" : [ $number, ... ],
  "datatype"  : $string,
  "parameters" : $parameters #optional,
  "data" : $tensor_data
}

• "name": The name of the output tensor.
• "shape": The shape of the output tensor. Each dimension must be an
  integer representable as an unsigned 64-bit integer value.
• "datatype": The data-type of the output tensor elements as defined
  in Tensor Data Types.
• "parameters": An object containing zero or more parameters for this
  input expressed as key/value pairs. See Parameters
  for more information.
• “data”: The contents of the tensor. See Tensor Data
  for more information.

Inference Response JSON Error Object

A failed inference request must be indicated by an HTTP error status
(typically 400). The HTTP body must contain the$inference_error_response object.

$inference_error_response =
{
  "error": <error message string>
}

• “error”: The descriptive message for the error.

Inference Request Examples

The following example shows an inference request to a model with two
inputs and one output. The HTTP Content-Length header gives the size
of the JSON object.

POST /v2/models/mymodel/infer HTTP/1.1
Host: localhost:8000
Content-Type: application/json
Content-Length: <xx>
{
  "id" : "42",
  "inputs" : [
    {
      "name" : "input0",
      "shape" : [ 2, 2 ],
      "datatype" : "UINT32",
      "data" : [ 1, 2, 3, 4 ]
    },
    {
      "name" : "input1",
      "shape" : [ 3 ],
      "datatype" : "BOOL",
      "data" : [ true ]
    }
  ],
  "outputs" : [
    {
      "name" : "output0"
    }
  ]
}

For the above request the inference server must return the “output0”
output tensor. Assuming the model returns a [ 3, 2 ] tensor of data
type FP32 the following response would be returned.

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: <yy>
{
  "id" : "42"
  "outputs" : [
    {
      "name" : "output0",
      "shape" : [ 3, 2 ],
      "datatype"  : "FP32",
      "data" : [ 1.0, 1.1, 2.0, 2.1, 3.0, 3.1 ]
    }
  ]
}

Parameters Copy Section

The $parameters JSON describes zero or more “name”/”value” pairs,
where the “name” is the name of the parameter and the “value” is a$string, $number, or $boolean.

$parameters =
{
  $parameter, ...
}

$parameter = $string : $string | $number | $boolean

Currently no parameters are defined. As required a future proposal may
define one or more standard parameters to allow portable functionality
across different inference servers. A server can implement
server-specific parameters to provide non-standard capabilities.

Tensor Data Copy Section

Tensor data must be presented in row-major order of the tensor
elements. Element values must be given in "linear" order without any
stride or padding between elements. Tensor elements may be presented
in their nature multi-dimensional representation, or as a flattened
one-dimensional representation.

Tensor data given explicitly is provided in a JSON array. Each element
of the array may be an integer, floating-point number, string or
boolean value. The server can decide to coerce each element to the
required type or return an error if an unexpected value is
received. Note that fp16 is problematic to communicate explicitly
since there is not a standard fp16 representation across backends nor
typically the programmatic support to create the fp16 representation
for a JSON number.

For example, the 2-dimensional matrix:

[ 1 2
  4 5 ]

Can be represented in its natural format as:

"data" : [ [ 1, 2 ], [ 4, 5 ] ]

Or in a flattened one-dimensional representation:

"data" : [ 1, 2, 4, 5 ]

GRPC Copy Chapter

The GRPC API closely follows the concepts defined in theHTTP/REST API. A compliant server must implement the
health, metadata, and inference APIs described in this section.

All strings in all contexts are case-sensitive.

The GRPC definition of the service is:

//
// Inference Server GRPC endpoints.
//
service GRPCInferenceService
{
  // Check liveness of the inference server.
  rpc ServerLive(ServerLiveRequest) returns (ServerLiveResponse) {}

  // Check readiness of the inference server.
  rpc ServerReady(ServerReadyRequest) returns (ServerReadyResponse) {}

  // Check readiness of a model in the inference server.
  rpc ModelReady(ModelReadyRequest) returns (ModelReadyResponse) {}

  // Get server metadata.
  rpc ServerMetadata(ServerMetadataRequest) returns (ServerMetadataResponse) {}

  // Get model metadata.
  rpc ModelMetadata(ModelMetadataRequest) returns (ModelMetadataResponse) {}

  // Perform inference using a specific model.
  rpc ModelInfer(ModelInferRequest) returns (ModelInferResponse) {}
}

Health Copy Section

A health request is made using the ServerLive, ServerReady, or
ModelReady endpoint. For each of these endpoints errors are indicated
by the google.rpc.Status returned for the request. The OK code
indicates success and other codes indicate failure.

Server Live

The ServerLive API indicates if the inference server is able to
receive and respond to metadata and inference requests. The request
and response messages for ServerLive are:

message ServerLiveRequest {}

message ServerLiveResponse
{
  // True if the inference server is live, false if not live.
  bool live = 1;
}

Server Ready

The ServerReady API indicates if the server is ready for
inferencing. The request and response messages for ServerReady are:

message ServerReadyRequest {}

message ServerReadyResponse
{
  // True if the inference server is ready, false if not ready.
  bool ready = 1;
}

Model Ready

The ModelReady API indicates if a specific model is ready for
inferencing. The request and response messages for ModelReady are:

message ModelReadyRequest
{
  // The name of the model to check for readiness.
  string name = 1;

  // The version of the model to check for readiness. If not given the
  // server will choose a version based on the model and internal policy.
  string version = 2;
}

message ModelReadyResponse
{
  // True if the model is ready, false if not ready.
  bool ready = 1;
}

Server Metadata Copy Section

The ServerMetadata API provides information about the server. Errors
are indicated by the google.rpc.Status returned for the request. The
OK code indicates success and other codes indicate failure. The
request and response messages for ServerMetadata are:

message ServerMetadataRequest {}

message ServerMetadataResponse
{
  // The server name.
  string name = 1;

  // The server version.
  string version = 2;

  // The extensions supported by the server.
  repeated string extensions = 3;
}

Model Metadata Copy Section

The per-model metadata API provides information about a model. Errors
are indicated by the google.rpc.Status returned for the request. The
OK code indicates success and other codes indicate failure. The
request and response messages for ModelMetadata are:

message ModelMetadataRequest
{
  // The name of the model.
  string name = 1;

  // The version of the model to check for readiness. If not given the
  // server will choose a version based on the model and internal policy.
  string version = 2;
}

message ModelMetadataResponse
{
  // Metadata for a tensor.
  message TensorMetadata
  {
    // The tensor name.
    string name = 1;

    // The tensor data type.
    string datatype = 2;

    // The tensor shape. A variable-size dimension is represented
    // by a -1 value.
    repeated int64 shape = 3;
  }

  // The model name.
  string name = 1;

  // The versions of the model available on the server.
  repeated string versions = 2;

  // The model's platform. See Platforms.
  string platform = 3;

  // The model's inputs.
  repeated TensorMetadata inputs = 4;

  // The model's outputs.
  repeated TensorMetadata outputs = 5;
}

Inference Copy Section

The ModelInfer API performs inference using the specified
model. Errors are indicated by the google.rpc.Status returned for the
request. The OK code indicates success and other codes indicate
failure. The request and response messages for ModelInfer are:

/* Detailed source-code truncated for AI context efficiency. */

Parameters Copy Section

The Parameters message describes a “name”/”value” pair, where the“name” is the name of the parameter and the “value” is a boolean,
integer, or string corresponding to the parameter.

Currently no parameters are defined. As required a future proposal may
define one or more standard parameters to allow portable functionality
across different inference servers. A server can implement
server-specific parameters to provide non-standard capabilities.

//
// An inference parameter value.
//
message InferParameter
{
  // The parameter value can be a string, an int64, a boolean
  // or a message specific to a predefined parameter.
  oneof parameter_choice
  {
    // A boolean parameter value.
    bool bool_param = 1;

    // An int64 parameter value.
    int64 int64_param = 2;

    // A string parameter value.
    string string_param = 3;
  }
}

Tensor Data Copy Section

In all representations tensor data must be flattened to a
one-dimensional, row-major order of the tensor elements. Element
values must be given in "linear" order without any stride or padding
between elements.

Using a "raw" representation of tensors withModelInferRequest::raw_input_contents andModelInferResponse::raw_output_contents will typically allow higher
performance due to the way protobuf allocation and reuse interacts
with GRPC. For example, see issue here.

An alternative to the "raw" representation is to use
InferTensorContents to represent the tensor data in a format that
matches the tensor's data type.

/* Detailed source-code truncated for AI context efficiency. */

Platforms Copy Chapter

A platform is a string indicating a DL/ML framework or
backend. Platform is returned as part of the response to aModel Metadata request but is information only. The
proposed inference APIs are generic relative to the DL/ML framework
used by a model and so a client does not need to know the platform of
a given model to use the API. Platform names use the format“_<format>”. The following platform names are allowed:

• tensorrt_plan: A TensorRT model encoded as a serialized engine or “plan”.
• tensorflow_graphdef: A TensorFlow model encoded as a GraphDef.
• tensorflow_savedmodel: A TensorFlow model encoded as a SavedModel.
• onnx_onnxv1: A ONNX model encoded for ONNX Runtime.
• pytorch_torchscript: A PyTorch model encoded as TorchScript.
• mxnet_mxnet An MXNet model
• caffe2_netdef: A Caffe2 model encoded as a NetDef.

Tensor Data Types Copy Chapter

Tensor data types are shown in the following table along with the size
of each type, in bytes.

References Copy Chapter

This document is based on the KServe original created during the lifetime of the KFServing project in Kubeflow by its various contributors including Seldon, NVIDIA, IBM, Bloomberg and others.

Overview (`k8s/kustomize/helm-components-sc/README.md`) Copy Section

Custom Resource Patching

To allow kustomize to do a strategic patch operation on our Custom Resources we need to give it the open api spec otherwise it will default to a normal patch and overwrite.

• Dowload openapi spec by installing the CRDs on a cluster and running kustomize openapi fetch > crds-schema.json
• As we use a custom podSpec the extensions needed in the OpenApi spec are not added. Also maybe dependong on this . Edit the areas you want strategic merge by adding extension settings, e.g. for containers keyd on name:
  text Copy

  "x-kubernetes-patch-merge-key": "name",
   "x-kubernetes-patch-strategy": "merge",

Overview (`docs-gb/getting-started/kubernetes-installation/security/README.md`) Copy Section

Security

Seldon can be run with secure control plane and data plane operations. There are three areas of concern:

• Control Plane
• Kafka
• Data Plane

The various communication points between services are shown in the diagram below:

.jpg>)

Control Plane Copy Chapter

TLS control plane activation is switched on and off via the environment variable: CONTROL_PLANE_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

Certificates will be loaded and used for the control plane gRPC services. The secrets or folders will be watched for updates (on certificate renewal) and automatically loaded again.

Helm Control Plane Install Copy Section

When installing seldon-core-v2-setup you can set the secret names for your certificates. If using cert-manager example discussed below this would be as follows:

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh \
      --set security.controlplane.protocol=SSL

Kafka Copy Chapter

Kafka secure activation is switched on and off via the environment variable: KAFKA_SECURITY_PROTOCOL whose values can be PLAINTEXT, SSL or SASL_SSL.

Examples are shown below:

• mTLS Strimzi example
• mTLS AWS MSK example
• SASL PLAIN with Confluent Cloud example
• SASL PLAIN with Azure Event Hub example
• SASL SCRAM with Strimzi example
• SASL SCRAM with AWS MSK example
• SASL OAUTH with Confluent Cloud example

Data Plane Copy Chapter

TLS Data plane activation is switched on and off via the environment variable: ENVOY_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

When activated this ensures TLS is used to communicate to Envoy via the xDS server as well as using the SDS service to send cretificates to envoy to use for upstream and downstream networking. Downstream is the external access to Seldon and upstream is the path from Envoy to the model servers or pipeline gateway.

Helm Data Plane Install Copy Section

When installing seldon-core-v2-setup you can set data plane operations to TLS as below. This assumes the secrets installed by the helm chart at the end of this section.

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh --set security.envoy.protocol=SSL

The above uses default secret names defined for the certificates installed. You can change the names of the required certificate secrets as shown in a longer configuration below (again using the default names for illustration).

For this we use the following updated Helm values (k8s/samples/values-tls-dataplane-example.yaml):

security:
  controlplane:
    protocol: PLAINTEXT
    ssl:
      server:
        secret: seldon-controlplane-server
        clientValidationSecret: seldon-controlplane-client
      client:
        secret: seldon-controlplane-client
        serverValidationSecret: seldon-controlplane-server
  envoy:
    protocol: SSL
    ssl:
      upstream:
        server:
          secret: seldon-upstream-server
          clientValidationSecret: seldon-upstream-client
        client:
          secret: seldon-upstream-client
          serverValidationSecret: seldon-upstream-server
      downstream:
        client:
          serverValidationSecret: seldon-downstream-server
        server:
          secret: seldon-downstream-server

We use Envoy internally to direct traffic and in Envoy's terminology upstream is for internal model servers called from Envoy while the downstream server is the entrypoint server running in Envoy to receive grpc and REST calls. The above settings ensure mTLS for internal "upstream" traffic while provides a standard SSL non-mTLS entrpoint.

To use the above with the seldon CLI you would need a custom config file as follow:

{
    "dataplane": {
	"tls": true,
	"skipSSLVerify": true
    }
}

We skip SSL Verify as these are internal self-signed certificates. For production use you would change this to the correct DNS name you are exposing the Seldon entrypoint.

Cetificate Providers Copy Chapter

The installer/cluster controller for Seldon needs to provide the certificates. As part of Seldon we provide an example set of certificate issuers and certificates using cert-manager.

Helm Copy Section

You can install Certificates into the desired namespace, here we use seldon-mesh as an example.

helm install seldon-v2-certs k8s/helm-charts/seldon-core-v2-certs/ -n seldon-mesh

Overview (`docs-gb/installation/production-environment/kafka/README.md`) Copy Section

Kafka Setup

Kafka is a component in the Seldon Core 2 ecosystem, that provides scalable, reliable, and flexible communication for machine learning deployments. It serves as a strong backbone for building complex inference pipelines, managing high-throughput asynchronous predictions, and seamlessly integrating with event-driven systems—key features needed for contemporary enterprise-grade ML platforms.

An inference request is a request sent to a machine learning model to make a prediction or inference based on input data. It is a core concept in deploying machine learning models in production, where models serve predictions to users or systems in real-time or batch mode.

To explore this feature of Seldon Core 2, you need to integrate with Kafka. Integrate Kafka through managed cloud services or by deploying it directly within a Kubernetes cluster.

{% hint style="info" %}
Note: Kafka is an external component outside of the main Seldon stack. Therefore, it is the cluster administrator's responsibility to administrate and manage the Kafka instance used by Seldon. For production installation it is highly recommended to use managed Kafka instance.
{% endhint %}

• Securing Kafka provides more information about the encrytion and authentication.
• Configuration examples provides the steps to configure some of the managed Kafka services.

Self-hosted Kafka Copy Section

Seldon Core 2 requires Kafka to implement data-centric inference Pipelines. To install Kafka for testing purposed in your Kubernetes cluster, use Strimzi Operator. For more information, see Self Hosted Kafka

Overview (`docs-gb/user-guide/v2/rest/README.md`) Copy Section

REST

Source Index (`docs/source/index.md`) Copy Section

Home

:maxdepth: 4
:titlesonly:
:includehidden:

contents/about/index
contents/getting-started/index
contents/tutorials/index
contents/servers/index
contents/models/index
contents/explainers/index
contents/drift/index
contents/outlier/index
contents/experiments/index
contents/pipelines/index
contents/inference/index
contents/kubernetes/index
contents/examples/index
contents/cli/index
contents/apis/index
contents/architecture/index
contents/faqs/index
contents/metrics/index
contents/performance-tests/index
contents/upgrading/index
contents/development/index

Source Contents About Index (`docs/source/contents/about/index.md`) Copy Section

About

Seldon V2 APIs provide a state of the art solution for machine learning inference which can be run locally on a laptop as well as on Kubernetes for production.

:width: 100%

Features Copy Chapter

• A single platform for inference of wide range of standard and custom artifacts.
• Deploy locally in Docker during development and testing of models.
• Deploy at scale on Kubernetes for production.
• Deploy single models to multi-step pipelines.
• Save infrastructure costs by deploying multiple models transparently in inference servers.
• Overcommit on resources to deploy more models than available memory.
• Dynamically extended models with pipelines with a data-centric perspective backed by Kafka.
• Explain individual models and pipelines with state of the art explanation techniques.
• Deploy drift and outlier detectors alongside models.
• Kubernetes Service mesh agnostic - use the service mesh of your choice.

Core features and comparison to Seldon Core V1 APIs Copy Chapter

Our V2 APIs separate out core tasks into separate resources allowing users to get started fast with deploying a Model and the progressing to more complex Pipelines, Explanations and Experiments.

Multi-model serving Copy Chapter

Seldon transparently will provision your model onto the correct inference server.

By packing multiple models onto a smaller set of servers users can save infrastructure costs and efficiently utilize their models.

By allowing over-commit users can provision model models that available memory resources by allowing Seldon to transparently unload models that are not in use.

Inference Servers Copy Chapter

Seldon V2 supports any V2 protocol inference server. At present we include Seldon's MLServer and NVIDIA's Triton inference server automatically on install. These servers cover a wide range of artifacts including custom python models.

Service Mesh Agnostic Copy Chapter

Seldon Core 2 can be integrated with any Kubernetes service mesh. There are current examples with istio, Ambassador and Traefic.

Publication Copy Chapter

These features are influenced by our position paper on the next generation of ML model serving frameworks:

Title: Desiderata for next generation of ML model serving

Workshop: Challenges in deploying and monitoring ML systems workshop - NeurIPS 2022

Source Contents Apis Index (`docs/source/contents/apis/index.md`) Copy Section

APIs

Seldon provides APIs for management and inference.

• API for inference
• Scheduler API for management (Advanced)
• Internal APIs (Reference)

:maxdepth: 1
:hidden:

inference/index.md
scheduler/index.md
internal/index.md

Source Contents Architecture Index (`docs/source/contents/architecture/index.md`) Copy Section

Architecture

The current set of components used in Seldon Core 2 is shown below:

The core components are:

• Scheduler : manages the load and unload of models, pipelines, explainers and experiments.
• Pipeline gateway : handles REST/gRPC calls to pipelines.
• Dataflow engine : handles the flow of data between components in a pipeline.
• Model gateway : handles the flow of data from models to inference requests on servers and passes on the responses.
• Agent : manages the loading and unloading of models on a server and access to the server over REST/gRPC.
• Envoy : manages the proxying of requests to the correct servers including load balancing.

All the above are Kubernetes agnostic and can run locally, e.g. on Docker Compose.

We also provide a Kubernetes Operator to allow Kubernetes usage.

Kafka is used as the backbone for Pipelines allowing a decentralized, synchronous and asynchronous usage.

Kafka Copy Chapter

Kafka is used as the backbone for allowing Pipelines of Models to be connected together into arbitrary directed acyclic graphs. Models can be reused in different Pipelines. The flow of data between models is handled by the dataflow engine using KStreams.

Dataflow Architecture Copy Chapter

Seldon V2 follows a dataflow design paradigm and it's part of the current movement for data centric machine learning. By taking a decentralized route that focuses on the flow of data users can have more flexibility and insight in building complex applications containing machine learning and traditional components. This contrasts with a more centralized orchestration more traditional in service orientated architectures.

By focusing on the data we allow users to join various flows together using stream joining concepts as shown below.

We support several types of joins:

• inner joins, where all inputs need to be present for a transaction to join the tensors passed through the Pipeline;
• outer joins, where only a subset needs to be available during the join window
• triggers, in which data flows need to wait until records on one or more trigger data flows appear. The data in these triggers is not passed onwards from the join.

These techniques allow users to create complex pipeline flows of data between machine learning components.

More discussion on the data flow view of machine learning and its effect on v2 design can be found here.

:maxdepth: 1
:hidden:

dataflow.md

Source Contents Cli Index (`docs/source/contents/cli/index.md`) Copy Section

CLI

Seldon provides a CLI to allow easy management and testing of Model, Experiment, and Pipeline resources.

At present this needs to be built by hand from the operator folder.

make build-seldon     # for linux/macOS amd64
make build-seldon-arm # for macOS ARM

Then place the bin/seldon executable in your path.

• cli docs

Environment Variables and Services Copy Chapter

The CLI talks to 3 backend services on default endpoints:

1. The Seldon Core 2 Scheduler: default 0.0.0.0:9004
2. The Seldon Core inference endpoint: default 0.0.0.0:9000
3. The Seldon Kafka broker: default: 0.0.0.0:9092

These defaults will be correct when Seldon Core 2 is installed locally as per the docs. For Kubernetes, you will need to change these by defining environment variables.

:language: golang
:start-after: // Defaults
:end-before: // Help statements

Kubernetes Usage Copy Chapter

Inference Service Copy Section

For a default install into the seldon-mesh namespace if you have exposed the inference svc as a loadbalancer you will find it at:

kubectl get svc seldon-mesh -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Use above IP at port 80:

export SELDON_INFER_HOST=<ip>:80

Scheduler Service Copy Section

For a default install into the seldon-mesh namespace if you have exposed the scheduler svc as a loadbalancer you will find it at:

kubectl get svc seldon-scheduler -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'

Use above IP at port 9004:

export SELDON_SCHEDULE_HOST=<ip>:9004

Kafka Broker Copy Section

The Kafka broker will depend on how you have installed Kafka into your Kubernetes cluster. Find the broker IP and use:

export SELDON_KAFKA_BROKER=<ip>:<port>

Config file Copy Chapter

You can create a config file to manage connections to running Seldon Core 2 installs. The settings will override any environment variable settings.

The definition is shown below:

:language: golang
 :start-after: // start config struct
 :end-before: // end config struct

An example below shows an example where we connect via TLS to the Seldon scheduler using our scheduler client certificate:

{
    "controlplane":{
	"schedulerHost": "seldon-scheduler.svc:9044",
	"tls"; true,
	"keyPath": "/home/certs/seldon-scheduler-client/tls.key",
	"crtPath": "/home/certs/seldon-scheduler-client/tls.crt",
	"caPath": "/home/certs/seldon-scheduler-client/ca.crt"
    }
}

To manage config files and activate them you can use the CLI command seldon config which has subcommands to list, add, remove, activate and decative configs.

For example:

$ seldon config list
config		path						active
------		----						------
kind-sasl	/home/work/seldon/cli/config-sasl.json		*

$ seldon config deactivate kind-sasl

$ seldon config list
config		path						active
------		----						------
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config add gcp-scv2 ~/seldon/cli/gcp.json

$ seldon config list
config		path						active
------		----						------
gcp-scv2	/home/work/seldon/cli/gcp.json
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config activate gcp-scv2

$ seldon config list
config		path						active
------		----						------
gcp-scv2	/home/work/seldon/cli/gcp.json	    		*
kind-sasl	/home/work/seldon/cli/config-sasl.json

$ seldon config list kind-sasl
{
  "controlplane": {
    "schedulerHost": "172.19.255.2:9004"
  },
  "kafka": {
    "bootstrap": "172.19.255.3:9093",
    "caPath": "/home/work/gcp/scv2/certs/seldon-cluster-ca-cert/ca.crt"
  }
}

TLS Certificates for Local Use Copy Chapter

For running with Kubernetes TLS connections on the control and/or data plane, certificates will need to be downloaded locally. We provide an example script which will download certificates from a Kubernetes secret and store them in a folder. It can be found in hack/download-k8s-certs.sh and takes 2 or 3 arguments:

./download-k8s-certs.sh <namespace> <secret> [<folder>]

e.g.:

./download-k8s-certs.sh seldon-mesh seldon-scheduler-client

:maxdepth: 1
:hidden:

docs/seldon.md
docs/seldon_config.md
docs/seldon_model.md
docs/seldon_experiment.md
docs/seldon_pipeline.md
docs/seldon_server.md
docs/seldon_config_activate.md
docs/seldon_config_add.md
docs/seldon_config_deactivate.md
docs/seldon_config_list.md
docs/seldon_config_remove.md
docs/seldon_model_infer.md
docs/seldon_model_load.md
docs/seldon_model_status.md
docs/seldon_model_unload.md
docs/seldon_experiment_start.md
docs/seldon_experiment_status.md
docs/seldon_experiment_stop.md
docs/seldon_pipeline_infer.md
docs/seldon_pipeline_load.md
docs/seldon_pipeline_status.md
docs/seldon_pipeline_unload.md
docs/seldon_pipeline_inspect.md
docs/seldon_server_status.md

Source Contents Development Index (`docs/source/contents/development/index.md`) Copy Section

Development

Development Requirements Copy Chapter

• Go 1.17+ with glibc development libraries installed.
  • e.g. for Ubuntu systems: sudo apt-get install libc6-dev
• Java JDK 17+ and Kotlin 1.6.10+
• Kubebuilder V2
• Docker and docker-compose
• Helm
• Kustomize
• Ansible

Testing resources Copy Chapter

• Kind
• k6
• unix utils: jq, curl, grpcurl

Release Process Copy Chapter

Releases are carried out via Github.

:maxdepth: 1
:hidden:

release/index.md
licenses.md

Source Contents Drift Index (`docs/source/contents/drift/index.md`) Copy Section

Drift Detection

Drift detection models are treated as any other Model. You can run any saved Alibi-Detect drift detection model by adding the requirement alibi-detect.

An example drift detection model from the CIFAR10 image classification example is shown below:

:language: yaml

Usually you would run these models in an asynchronous part of a Pipeline, i.e. they are not connected to the output of the Pipeline which defines the synchronous path. For example, the CIFAR-10 image detection example uses a pipeline as shown below:

:language: yaml

Note how the cifar10-drift model is not part of the path to the outputs. Drift alerts can be read from the Kafka topic of the model.

Examples Copy Chapter

• CIFAR10 image classification with drift detector
• Tabular income classification model with drift detector

Source Contents Examples Index (`docs/source/contents/examples/index.md`) Copy Section

Examples

This section will provide some examples to allow operations with Seldon to be tested so you can run your own models, experiments, pipelines and explainers.

Getting Started Examples Copy Chapter

• Local examples
• Kubernetes examples

Models Copy Chapter

• Huggingface models
• Model zoo
• Artifact versions

Pipelines Copy Chapter

• Pipeline examples
• Pipeline to pipeline examples

Explainers Copy Chapter

• Explainer examples

Servers Copy Chapter

• Custom Servers

Experiments Copy Chapter

• Local experiments
• Experiment version examples

Making Inference Requests Copy Chapter

• Inference examples
• Tritonclient examples
• Batch Inference examples (kubernetes)
• Batch Inference examples (local)

Misc Copy Chapter

• Checking Pipeline readiness

Further Kubernetes Examples Copy Chapter

• Kubernetes custerwide example

Advanced Examples Copy Chapter

• Huggingface speech to sentiment with explanations pipeline
• Production image classifier with drift and outlier monitoring
• Production income classifier with drift, outlier and explanations
• Conditional pipeline with pandas query model
• Kubernetes Server with PVC

:maxdepth: 1
:hidden:

local-examples.md
k8s-examples.md
huggingface.md
model-zoo.md
multi-version.md
pipeline-examples.md
pipeline-to-pipeline.md
explainer-examples.md
local-experiments.md
experiment-versions.md
inference.md
tritonclient-examples.md
pipeline-ready-and-metadata.md
speech-to-sentiment.md
cifar10.md
income.md
pandasquery.md
k8s-pvc.md

Source Contents Experiments Index (`docs/source/contents/experiments/index.md`) Copy Section

Experiments

An Experiment defines an http traffic split between Models or Pipelines.

Experiments also allow a mirror model or pipeline to be tested where some percentage of the traffic to the main model is sent to the mirror but the result is not returned.

Further details are given here.

Source Contents Explainers Index (`docs/source/contents/explainers/index.md`) Copy Section

Explainers

Explainers are Model resources with some extra settings. They allow a range of explainers from the Alibi-Explain library to be run on MLServer.

An example Anchors explainer definitions is shown below.

:language: yaml

The key additions are:

• type : This must be one of the supported Alibi Explainer types supported by the Alibi Explain runtime in MLServer.
• modelRef : The model name for black box explainers.
• pipelineRef : The pipeline name for black box explainers.

Only one of modelRef and pipelineRef is allowed.

Pipeline Explanations Copy Chapter

Blackbox explainers can explain a Pipeline as well as a model. An example from the Huggingface sentiment demo is show below.

:language: yaml

Examples Copy Chapter

• Tabular income classification model with Anchor Tabular black box model explainer
• Huggingface Sentiment model with Anchor Text black box pipeline explainer
• Anchor Text movies sentiment explainer

Source Contents Faqs Index (`docs/source/contents/faqs/index.md`) Copy Section

FAQs

Should I use Core 1 or Core 2? Copy Chapter

We strongly recommend using Seldon Core 2 for all new and production deployments. Core 1 is now in maintenance mode — it is no longer actively developed, and future improvements will be focused exclusively on Core 2. Core 2 is already being supported for production use by our enterprise-scale customers.

Seldon Core 2 introduces a more modern, scalable, and flexible architecture designed for production-grade machine learning systems. Key advantages include:

• Data-centric ML pipelines with native support for streaming data via Kafka, enabling real-time inference and observability.
• Multi-Model Serving and resource overcommit strategies that improve infrastructure efficiency and model utilization.

Can Seldon Core 2 be used with Seldon Core 1 Copy Chapter

The two projects are able to be run side by side. Existing users of Seldon Core can update to deploy their models using Seldon Core 2 as needed to take advantage of the new functionality. Both will be supported.

Can I do payload logging in Seldon Core 2? Copy Chapter

By default, the input and output of every step in a pipeline (as well as the pipeline itself) is logged in Kafka.
From there it's up to you what to do with the data.
You could use something like Kafka Connect to stream the logs to a datastore of your choice.

Note that there is no automatic request logging for models being accessed directly over REST or gRPC.
Requests need to be sent via pipelines to be recorded in Kafka.

Source Contents Getting Started Index (`docs/source/contents/getting-started/index.md`) Copy Section

Getting Started

Some dependencies may require that the (virtual) machines on which you deploy, support the SSE4.2 instruction set or x86-64-v2 microarchitecture. If `lscpu | grep sse4_2` does not return anything on your machine, your CPU is not compatible, and you may need to update the (virtual) host's CPU.

Seldon Core can be installed either with Docker Compose or with Kubernetes:

• Install locally with Docker Compose
• Install onto a Kubernetes cluster

Once installed:

• Try the existing examples.
• Train and deploy your own model artifact.

Core Concepts Copy Chapter

There are three core resources you will use:

• Models - for deploying single machine learning models, custom transformation logic, drift detectors and outliers detectors.
• Pipelines - for connecting together flows of data transformations between Models with a synchronous path and multiple asynchronous paths.
• Experiments - for testing new versions of models

By default the standard installation will deploy MLServer and Triton inference servers which provide support for a wide range of machine learning model artifacts including Tensorflow models, PyTorch models, SKlearn models, XGBoost models, ONNX models, TensorRT models, custom python models and many more. For advanced use, the creation of new inference servers is manged by two resources:

• Servers - for deploying sets of replicas of core inference servers (MLServer or Triton by default).
• ServerConfigs - for defining server configurations including custom servers.

API for Inference Copy Chapter

Once deployed models can be called using the Seldon V2 inference protocol. This protocol created by Seldon, NVIDIA and the KServe projects is supported by MLServer and Triton inference servers amingst others and allows REST and gRPC calls to your model.

Your model is exposed via our internal Envoy gateway. If you wish to expose your models in Kubernetes outside the cluster you are free to use any Service Mesh or Ingress technology. Various examples are provided for service mesh integration.

Inference Metrics Copy Chapter

Metrics are exposed for scraping by Prometheus. For Kubernetes we provide example instructions for using kube-prometheus.

Pipeline requirements Copy Chapter

Pipelines are built upon Kafka streaming technology.

:maxdepth: 1
:hidden:

docker-installation/index.md
kubernetes-installation/index.md
configuration/index.md
cli.md

Source Contents Inference Index (`docs/source/contents/inference/index.md`) Copy Section

Inference

This section will discuss how to make inference calls against your Seldon models or pipelines.

You can make synchronous inference requests via REST or gRPC or asynchronous requests via Kafka topics.
The content of your request should be an inference v2 protocol payload:

• REST payloads will generally be in the JSON v2 protocol format.
• gRPC and Kafka payloads must be in the Protobuf v2 protocol format.

Synchronous Requests Copy Chapter

For making synchronous requests, the process will generally be:

1. Find the appropriate service endpoint (IP address and port) for accessing the installation of Seldon Core 2.
2. Determine the appropriate headers/metadata for the request.
3. Make requests via REST or gRPC.

Find the Seldon Service Endpoint Copy Section

````{tab} Docker Compose

In the default Docker Compose setup, container ports are accessible from the host machine.
This means you can use `localhost` or `0.0.0.0` as the hostname.

The default port for sending inference requests to the Seldon system is `9000`.
This is controlled by the `ENVOY_DATA_PORT` environment variable for Compose.

Putting this together, you can send inference requests to `0.0.0.0:9000`.
````

````{tab} Kubernetes

In Kubernetes, Seldon creates a single `Service` called `seldon-mesh` in the namespace it is installed into.
By default, this namespace is also called `seldon-mesh`.

If this `Service` is exposed via a load balancer, the appropriate address and port can be found via:

```bash
kubectl get svc seldon-mesh -n seldon-mesh -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
```

If you are not using a `LoadBalancer` for the `seldon-mesh` `Service`, you can still send inference requests.

For development and testing purposes, you can port-forward the `Service` locally using the below.
Inference requests can then be sent to `localhost:8080`.

```
kubectl port-forward svc/seldon-mesh -n seldon-mesh 8080:80
```

If you are using a service mesh like Istio or Ambassador, you will need to use the IP address of the service mesh ingress and determine the appropriate port.
````

Make Inference Requests Copy Section

Let us imagine making inference requests to a model called iris.

This iris model has the following schema, which can be set in a model-settings.json file for MLServer:

{
    "name": "iris",
    "implementation": "mlserver_sklearn.SKLearnModel",
    "inputs": [
        {
            "name": "predict",
            "datatype": "FP32",
            "shape": [-1, 4]
        }
    ],
    "outputs": [
        {
            "name": "predict",
            "datatype": "INT64",
            "shape": [-1, 1]
        }
    ],
    "parameters": {
        "version": "1"
    }
}

Examples are given below for some common tools for making requests.

````{group-tab} Seldon CLI

An example `seldon` request might look like this:

```
seldon model infer iris \
        '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'
```

The default inference mode is REST, but you can also send gRPC requests like this:

```
seldon model infer iris \
        --inference-mode grpc \
        '{"model_name":"iris","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}'
```
````

````{group-tab} cURL

An example `curl` request might look like this:

```
curl -v http://0.0.0.0:9000/v2/models/iris/infer \
        -H "Content-Type: application/json" \
        -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'
```
````

````{group-tab} grpcurl

An example `grpcurl` request might look like this:

```
grpcurl \
	-d '{"model_name":"iris","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}' \
	-plaintext \
	-import-path apis \
	-proto apis/mlops/v2_dataplane/v2_dataplane.proto \
	0.0.0.0:9000 inference.GRPCInferenceService/ModelInfer
```

The above request was run from the project root folder allowing reference to the Protobuf manifests defined in the `apis/` folder.
````

````{group-tab} Python tritonclient

You can use the Python [tritonclient](https://github.com/triton-inference-server/client) package to send inference requests.

A short, self-contained example is:

```python
import tritonclient.http as httpclient
import numpy as np

client = httpclient.InferenceServerClient(
    url="localhost:8080",
    verbose=False,
)

inputs = [httpclient.InferInput("predict", (1, 4), "FP64")]
inputs[0].set_data_from_numpy(
    np.array([[1, 2, 3, 4]]).astype("float64"),
    binary_data=False,
)

result = client.infer("iris", inputs)
print("result is:", result.as_numpy("predict"))
```
````

For pipelines, a synchronous request is possible if the pipeline has an `outputs` section defined in its spec.

Request Routing Copy Section

Seldon Routes

Seldon needs to determine where to route requests to, as models and pipelines might have the same name.
There are two ways of doing this: header-based routing (preferred) and path-based routing.

````{tab} Headers

Seldon can route requests to the correct endpoint via headers in HTTP calls, both for REST (HTTP/1.1) and gRPC (HTTP/2).

Use the `Seldon-Model` header as follows:
* For models, use the model name as the value.
  For example, to send requests to a model named `foo` use the header `Seldon-Model: foo`.
* For pipelines, use the pipeline name followed by `.pipeline` as the value.
  For example, to send requests to a pipeline named `foo` use the header `Seldon-Model: foo.pipeline`.

The `seldon` CLI is aware of these rules and can be used to easily send requests to your deployed resources.
See the [examples](../examples/index) and the [Seldon CLI docs](../cli/index.md) for more information.
````

````{tab} Paths

The inference v2 protocol is only aware of models, thus has no concept of pipelines.
Seldon works around this limitation by introducing _virtual_ endpoints for pipelines.
Virtual means that Seldon understands them, but other v2 protocol-compatible components like inference servers do not.

Use the following rules for paths to route to models and pipelines:
* For models, use the path prefix `/v2/models/{model name}`.
  This is normal usage of the inference v2 protocol.
* For pipelines, you can use the path prefix `/v2/pipelines/{pipeline name}`.
  Otherwise calling pipelines looks just like the inference v2 protocol for models.
  Do **not** use any suffix for the pipeline name as you would for routing headers.
* For pipelines, you can also use the path prefix `/v2/models/{pipeline name}.pipeline`.
  Again, this form looks just like the inference v2 protocol for models.
````

Extending our examples from above, the requests may look like the below when using header-based routing.

````{group-tab} Seldon CLI

No changes are required as the `seldon` CLI already understands how to set the appropriate gRPC and REST headers.

````

````{group-tab} cURL

Note the header in the last line:

```{code-block}
:emphasize-lines: 4

curl -v http://0.0.0.0:9000/v2/models/iris/infer \
        -H "Content-Type: application/json" \
        -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}' \
        -H "Seldon-Model: iris"
```
````

````{group-tab} grpcurl

Note the `rpc-header` flag in the penultimate line:

```{code-block}
:emphasize-lines: 6

grpcurl \
	-d '{"model_name":"iris","inputs":[{"name":"input","contents":{"fp32_contents":[1,2,3,4]},"datatype":"FP32","shape":[1,4]}]}' \
	-plaintext \
	-import-path apis \
	-proto apis/mlops/v2_dataplane/v2_dataplane.proto \
	-rpc-header seldon-model:iris \
	0.0.0.0:9000 inference.GRPCInferenceService/ModelInfer
```
````

````{group-tab} Python tritonclient

Note the `headers` dictionary in the `client.infer()` call:

```{code-block}
:emphasize-lines: 18

import tritonclient.http as httpclient
import numpy as np

client = httpclient.InferenceServerClient(
    url="localhost:8080",
    verbose=False,
)

inputs = [httpclient.InferInput("predict", (1, 4), "FP64")]
inputs[0].set_data_from_numpy(
    np.array([[1, 2, 3, 4]]).astype("float64"),
    binary_data=False,
)

result = client.infer(
    "iris",
    inputs,
    headers={"Seldon-Model": "iris"},
)
print("result is:", result.as_numpy("predict"))
```
````

Ingress Routes

If you are using an ingress controller to make inference requests with Seldon, you will need to configure the routing rules correctly.

There are many ways to do this, but custom path prefixes will not work with gRPC.
This is because gRPC determines the path based on the Protobuf definition.
Some gRPC implementations permit manipulating paths when sending requests, but this is by no means universal.

If you want to expose your inference endpoints via gRPC and REST in a consistent way, you should use virtual hosts, subdomains, or headers.

The downside of using only paths is that you cannot differentiate between different installations of Seldon Core 2 or between traffic to Seldon and any other inference endpoints you may have exposed via the same ingress.

You might want to use a mixture of these methods; the choice is yours.

````{tab} Virtual Hosts

Virtual hosts are a way of differentiating between logical services accessed via the same physical machine(s).

Virtual hosts are defined by the `Host` header for [HTTP/1](https://www.rfc-editor.org/rfc/rfc7230#section-5.4) and the `:authority` pseudo-header for [HTTP/2](https://www.rfc-editor.org/rfc/rfc9113.html#section-8.3.1).
These represent the same thing, and the HTTP/2 specification defines how to translate these when converting between protocol versions.

Many tools and libraries treat these headers as special and have particular ways of handling them.
Some common ones are given below:

* The `seldon` CLI has an `--authority` flag which applies to both REST and gRPC inference calls.
* `curl` accepts `Host` as a normal header.
* `grpcurl` has an `-authority` flag.
* In Go, the standard library's `http.Request` struct has a `Host` field and ignores attempts to set this value via headers.
* In Python, the `requests` library accepts the host as a normal header.

Be sure to check the documentation for how to set this with your preferred tools and languages.
````

````{tab} Subdomains

Subdomain names constitute a part of the overall host name.
As such, specifying a subdomain name for requests will involve setting the appropriate host in the URI.

For example, you may expose inference services in the namespaces `seldon-1` and `seldon-2` as in the following snippets:

```
curl https://seldon-1.example.com/v2/models/iris/infer ...

seldon model infer --inference-host https://seldon-2.example.com/v2/models/iris/infer ...
```

Many popular ingresses support subdomain-based routing, including Istio and Nginx.
Please refer to the documentation for your ingress of choice for further information.
````

````{tab} Headers

Many ingress controllers and service meshes support routing on headers.
You can use whatever headers you prefer, so long as they do not conflict with any Seldon relies upon.

Many tools and libraries support adding custom headers to requests.
Some common ones are given below:
* The `seldon` CLI accepts headers using the `--header` flag, which can be specified multiple times.
* `curl` accepts headers using the `-H` or `--header` flags.
* `grpcurl` accepts headers using the `-H` flag, which can be specified multiple times.
````

````{tab} Paths

It is possible to route on paths by using well-known path prefixes defined by the inference v2 protocol.
For gRPC, the full path (or "method") for an inference call is:
```
/inference.GRPCInferenceService/ModelInfer
```

This corresponds to the package (`inference`), service (`GRPCInferenceService`), and RPC name (`ModelInfer`) in the Protobuf definition of the inference v2 protocol.

You could use an exact match or a regex like `.*inference.*` to match this path, for example.
````

Asynchronous Requests Copy Chapter

The Seldon architecture uses Kafka and therefore asynchronous requests can be sent by pushing inference v2 protocol payloads to the appropriate topic.
Topics have the following form:

seldon.<namespace>.<model|pipeline>.<name>.<inputs|outputs>

If writing to a pipeline topic, you will need to include a Kafka header with the key `pipeline` and the value being the name of the pipeline.

Model Inference Copy Section

For a local install if you have a model iris, you would be able to send a prediction request by pushing to the topic: seldon.default.model.iris.inputs.
The response will appear on seldon.default.model.iris.outputs.

For a Kubernetes install in seldon-mesh if you have a model iris, you would be able to send a prediction request by pushing to the topic: seldon.seldon-mesh.model.iris.inputs.
The response will appear on seldon.seldon-mesh.model.iris.outputs.

Pipeline Inference Copy Section

For a local install if you have a pipeline mypipeline, you would be able to send a prediction request by pushing to the topic: seldon.default.pipeline.mypipeline.inputs. The response will appear on seldon.default.pipeline.mypipeline.outputs.

For a Kubernetes install in seldon-mesh if you have a pipeline mypipeline, you would be able to send a prediction request by pushing to the topic: seldon.seldon-mesh.pipeline.mypipeline.inputs. The response will appear on seldon.seldon-mesh.pipeline.mypipeline.outputs.

Pipeline Metadata Copy Chapter

It may be useful to send metadata alongside your inference.

If using Kafka directly as described above, you can attach Kafka metadata to your request, which will be passed around the graph.
When making synchronous requests to your pipeline with REST or gRPC you can also do this.

• For REST requests add HTTP headers prefixed with X-
• For gRPC requests add metadata with keys starting with X-

You can also do this with the Seldon CLI by setting headers with the --header argument (and also showing response headers with the --show-headers argument)

seldon pipeline infer --show-headers --header X-foo=bar tfsimples \
    '{"inputs":[{"name":"INPUT0","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]},{"name":"INPUT1","data":[1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16],"datatype":"INT32","shape":[1,16]}]}'

For pipeline inference, the response also contains a x-pipeline-version header, indicating which version of pipeline it ran inference with.

Request IDs Copy Chapter

For both model and pipeline requests the response will contain a x-request-id response header. For pipeline requests this can be used to inspect the pipeline steps via the CLI, e.g.:

seldon pipeline inspect tfsimples --request-id carjjolvqj3j2pfbut10 --offset 10

The --offset parameter specifies how many messages (from the latest) you want to search to find your request. If not specified the last request will be shown.

x-request-id will also appear in tracing spans.

If x-request-id is passed in by the caller then this will be used. It is the caller's responsibility to ensure it is unique.

The IDs generated are XIDs.

Source Contents Kubernetes Index (`docs/source/contents/kubernetes/index.md`) Copy Section

Kubernetes

Kubernetes provides the core platform for production use. However, the architecture in agnostic to the underlying cluster technology as shown by the local Docker installation.

:maxdepth: 1
:hidden:

kafka/index.md
resources/index.md
storage-secrets/index.md
metrics/index.md
tracing/index.md
service-meshes/index.md
scaling/index.md
autoscaling/index.md

Source Contents Metrics Index (`docs/source/contents/metrics/index.md`) Copy Section

Metrics

There are two kinds of metrics present in Seldon Core 2:

• operational metrics
• usage metrics

Operational metrics describe the performance of components in the system.
Examples of common operational considerations are memory consumption and CPU usage, request latency and throughput, and cache utilisation rates.
Generally speaking, these are the metrics system administrators, operations teams, and engineers will be interested in.

Usage metrics describe the system at a higher and less dynamic level.
Examples include the number of deployed servers and models, and component versions.
These are not typically metrics that engineers need insight into, but may be relevant to platform providers and operations teams.

:maxdepth: 1
:hidden:

operational.md
usage.md

Source Contents Models Index (`docs/source/contents/models/index.md`) Copy Section

Models

Models provide the atomic building blocks of Seldon. They represents machine learning models, drift detectors, outlier detectors, explainers, feature transformations, and more complex routing models such as multi-armed bandits.

• Seldon can handle a wide range of inference artifacts
• Artifacts can be stored on any of the 40 or more cloud storage technologies as well as from local (mounted) folder as discussed here.

Kubernetes Example Copy Chapter

A Kubernetes yaml example is shown below for a SKLearn model for iris classification:

:language: yaml

Its Kubernetes spec has two core requirements

• A storageUri specifying the location of the artifact. This can be any rclone URI specification.
• A requirements list which provides tags that need to be matched by the Server that can run this artifact type. By default when you install Seldon we provide a set of Servers that cover a range of artifact types.

GRPC Example Copy Chapter

You can also load models directly over the scheduler grpc service. An example is shown below use grpcurl tool:

!grpcurl -d '{"model":{ \
              "meta":{"name":"iris"},\
              "modelSpec":{"uri":"gs://seldon-models/mlserver/iris",\
                           "requirements":["sklearn"],\
                           "memoryBytes":500},\
              "deploymentSpec":{"replicas":1}}}' \
         -plaintext \
         -import-path ../../apis \
         -proto apis/mlops/scheduler/scheduler.proto  0.0.0.0:9004 seldon.mlops.scheduler.Scheduler/LoadModel

The proto buffer definitions for the scheduler are outlined here.

:maxdepth: 1
:hidden:

inference-artifacts/index.md
rclone/index.md
parameterized-models/index.md

Multi-model Serving with Overcommit Copy Chapter

Multi-model serving is an architecture pattern where one ML inference server hosts multiple models at the same time. It is a feature provided out of the box by Nvidia Triton and Seldon MLServer. Multi-model serving reduces infrastructure hardware requirements (e.g. expensive GPUs) which enables the deployment of a large number of models while making it efficient to operate the system at scale.

Seldon Core 2 leverages multi-model serving by design and it is the default option for deploying models. The system will find an appropriate server to load the model onto based on requirements that the user defines in the Model deployment definition.

Moreover, in many cases demand patterns allow for further Overcommit of resources. Seldon Core 2 is able to register more models than what can be served by the provisioned (memory) infrastructure and will swap models dynamically according to least used without adding significant latency overheads to inference workload.

See Multi-model serving for more information.

Autoscaling of Models Copy Chapter

See here for discussion of autoscaling of models.

Source Contents Outlier Index (`docs/source/contents/outlier/index.md`) Copy Section

Outlier Detection

Outlier detection models are treated as any other Model. You can run any saved Alibi-Detect outlier detection model by adding the requirement alibi-detect.

An example outlier detection model from the CIFAR10 image classification example is shown below:

:language: yaml

Examples Copy Chapter

• CIFAR10 image classification with outlier detector
• Tabular income classification model with outlier detector

Source Contents Performance Tests Index (`docs/source/contents/performance-tests/index.md`) Copy Section

Performance tests

This section describes how a user can run performance tests to understand the limits of a particular SCv2 deployment.

The base directly is tests/k6

Driver Copy Chapter

k6 is used to drive requests for load, unload and infer workloads. It is recommended that the load test is run within the same cluster that has SCv2 installed as it requires internal access to some of the services that are not automatically exposed to the outside world. Furthermore having the driver withthin the same cluster minimises link latency to SCv2 entrypoint; therefore infer latencies are more representatives of actual overheads of the system.

Tests Copy Chapter

• Envoy
  Tests synchronous inference requests via envoy

To run: make deploy-envoy-test

• Agent
  Tests inference requests direct to a specific agent, defaults to triton-0 or mlserver-0

To run: make deploy-rproxy-test pr make deploy-rproxy-mlserver-test

• Server
  Tests inference requests direct to a specific server (bypassing agent), defaults to triton-0 or mlserver-0

to run: make deploy-server-test or deploy-server-mlserver-test

• Pipeline gateway (HTTP-Kafka gateway)
  Tests inference requests to one-node pipeline HTTP and GPRC requests

To run: make deploy-kpipeline-test

• Model gateway (Kafka-HTTP gateway)
  Tests inference requests to a model via kafka

To run: deploy-kmodel-test

Results Copy Chapter

One way to look at results is to look at the log of the pod that executed the kubernetes job.

Results can also be persisted to a gs bucket, a service account k6-sa-key in the same namespace is required,

Users can also look at the metrics that are exposed in prometheus while the test is underway

Building k6 image Copy Chapter

In the case a user is modifying the actual scenario of the test:

• export DOCKERHUB_USERNAME=mydockerhubaccount
• build the k6 image via make build-push
• in the same shell environment, deploying jobs will use this custome built docker image

Modifying tests Copy Chapter

Users can modify settings of the tests in tests/k6/configs/k8s/base/k6.yaml. This will apply to all subsequent tests that are deployed using the above process.

Settings Copy Chapter

Some settings that can be changed

• k6 args
  {literalinclude} ../../../../tests/k6/configs/k8s/base/k6.yaml Copy

  :language: yaml
   :start-after: args
   :end-before: env

for a full list, check k6 args

• Environment variables
  {literalinclude} ../../../../tests/k6/configs/k8s/base/k6.yaml Copy

  :language: yaml
   :start-after: SCHEDULER_ENDPOINT}:9004
   :end-before: GOOGLE_APPLICATION_CREDENTIALS

  • for MODEL_TYPE, choose from:
  {literalinclude} ../../../../tests/k6/components/model.js Copy

  :language: javascript
   :end-before: const models

Source Contents Pipelines Index (`docs/source/contents/pipelines/index.md`) Copy Section

Pipelines

Pipelines allow models to be connected into flows of data transformations. This allows more complex machine learning pipelines to be created with multiple models, feature transformations and monitoring components such as drift and outlier detectors.

Creating Pipelines Copy Chapter

The simplest way to create Pipelines is by defining them with the Pipeline resource we provide for Kubernetes. This format is accepted by our Kubernetes implementation but also locally via our seldon CLI.

Internally in both cases Pipelines are created via our Scheduler API. Advanced users could submit Pipelines directly using this gRPC service.

An example that chains two models together is shown below:

apiVersion: mlops.seldon.io/v1alpha1
kind: Pipeline
metadata:
  name: chain
  namespace: seldon-mesh
spec:
  steps:
    - name: model1
    - name: model2
      inputs:
      - model1
  output:
    steps:
    - model2

• steps allow you to specify the models you want to combine into a pipeline. Each step name will correspond to a model of the same name. These models will need to have been deployed and available for the Pipeline to function, however Pipelines can be deployed before or at the same time you deploy the underlying models.
• steps.inputs allow you to specify the inputs to this step.
• outputs.steps allow you to specify the output of the Pipeline. A pipeline can have multiple paths include flows of data that do not reach the output, e.g. Drift detection steps. However, if you wish to call your Pipeline in a synchronous manner via REST/gRPC then an output must be present so the Pipeline can be treated as a function.

Expressing input data sources Copy Chapter

Model step inputs are defined with a dot notation of the form:

<stepName>|<pipelineName>.<inputs|outputs>.<tensorName>

Inputs with just a step name will be assumed to be step.outputs.

The default payloads for Pipelines is the V2 protocol which requires named tensors as inputs and outputs from a model. If you require just certain tensors from a model you can reference those in the inputs, e.g. mymodel.outputs.t1 will reference the tensor t1 from the model mymodel.

For the specification of the V2 protocol.

Chain Copy Chapter

The simplest Pipeline chains models together: the output of one model goes into the input of the next. This will work out of the box if the output tensor names from a model match the input tensor names for the one being chained to. If they do not then the tensorMap construct presently needs to be used to define the mapping explicitly, e.g. see below for a simple chained pipeline of two tfsimple example models:

:language: yaml

:caption: "*A simple chain of two models.*"
  :align: center

  flowchart LR
      classDef pipeIO fill:#F6E083

      subgraph input
          INPUT0:::pipeIO
          INPUT1:::pipeIO
      end

      INPUT0 --> TF1(tfsimple1)
      INPUT1 --> TF1
      TF1 --->|OUTPUT0: INPUT0| TF2(tfsimple2)
      TF1 --->|OUTPUT1: INPUT1| TF2

      subgraph output
          OUTPUT0:::pipeIO
          OUTPUT1:::pipeIO
      end

      TF2 --> OUTPUT0
      TF2 --> OUTPUT1

In the above we rename tensor OUTPUT0 to INPUT0 and OUTPUT1 to INPUT1. This allows these models to be chained together. The shape and data-type of the tensors needs to match as well.

This example can be found in the pipeline-examples examples.

Join Copy Chapter

Joining allows us to combine outputs from multiple steps as input to a new step.

:language: yaml

:caption: "*Joining the outputs of two models into a third model. The dashed lines signify model outputs that are not captured in the output of the pipeline.*"
  :align: center

  flowchart LR
      classDef pipeIO fill:#F6E083
      classDef hidden fill:#ffffff,stroke:#ffffff

      subgraph input
          INPUT0:::pipeIO
          INPUT1:::pipeIO
      end

      INPUT0 --> TF1(tfsimple1)
      INPUT1 --> TF1
      INPUT0 --> TF2(tfsimple2)
      INPUT1 --> TF2
      TF1 -.-> |OUTPUT1| tf1( ):::hidden
      TF1 --> |OUTPUT0: INPUT0| TF3(tfsimple3)
      TF2 -.-> |OUTPUT0| tf2( ):::hidden
      TF2 --> |OUTPUT1: INPUT1| TF3

      subgraph output
          OUTPUT0:::pipeIO
          OUTPUT1:::pipeIO
      end
      TF3 --> OUTPUT0
      TF3 --> OUTPUT1

Here we pass the pipeline inputs to two models and then take one output tensor from each and pass to the final model. We use the same tensorMap technique to rename tensors as disucssed in the previous section.

Joins can have a join type which can be specified with inputsJoinType and can take the values:

• inner : require all inputs to be available to join.
• outer : wait for joinWindowMs to join any inputs. Ignoring any inputs that have not sent any data at that point. This will mean this step of the pipeline is guaranteed to have a latency of at least joinWindowMs.
• any : Wait for any of the specified data sources.

This example can be found in the pipeline-examples examples.

Conditional Logic Copy Chapter

Pipelines can create conditional flows via various methods. We will discuss each in turn.

Model routing via tensors Copy Section

The simplest way is to create a model that outputs different named tensors based on its decision. This way downstream steps can be dependant on different expected tensors. An example is shown below:

:language: yaml

:caption: "*Pipeline with a conditional output model. The model **conditional** only outputs one of the two tensors, so only one path through the graph (red or blue) is taken by a single request*"
  :align: center

  flowchart LR
      classDef pipeIO fill:#F6E083
      classDef join fill:#CEE741;

      subgraph input
      INPUT0:::pipeIO
          CHOICE:::pipeIO
          INPUT0:::pipeIO
          INPUT1:::pipeIO
      end

      CHOICE --> conditional
      INPUT0 --> conditional
      INPUT1 --> conditional

      conditional --> |OUTPUT0: INPUT| add10
      conditional --> |OUTPUT1: INPUT| mul10

      add10 --> |OUTPUT| any(any):::join
      mul10 --> |OUTPUT| any

      subgraph output
          OUTPUT(OUTPUT):::pipeIO
      end

      any --> OUTPUT

      linkStyle 3 stroke:blue,color:blue;
      linkStyle 5 stroke:blue,color:blue;
      linkStyle 4 stroke:red,color:red;
      linkStyle 6 stroke:red,color:red;

In the above we have a step conditional that either outputs a tensor named OUTPUT0 or a tensor named OUTPUT1. The mul10 step depends on an output in OUTPUT0 while the add10 step depends on an output from OUTPUT1.

Note, we also have a final Pipeline output step that does an any join on these two models essentially outputting fron the pipeline whichever data arrives from either model. This type of Pipeline can be used for Multi-Armed bandit solutions where you want to route traffic dynamically.

This example can be found in the pipeline-examples examples.

Errors Copy Section

Its also possible to abort pipelines when an error is produced to in effect create a condition. This is illustrated below:

:language: yaml

This Pipeline runs normally or throws an error based on whether the input tensors have certain values.

This example can be found in the pipeline-examples examples.

Triggers Copy Section

Sometimes you want to run a step if an output is received from a previous step but not to send the data from that step to the model. This is illustrated below:

:language: yaml

:caption: "*A pipeline with a single trigger. The model **tfsimple3** only runs if the model **check** returns a tensor named `OUTPUT`. The green edge signifies that this is a trigger and not an additional input to **tfsimple3**. The dashed lines signify model outputs that are not captured in the output of the pipeline.*"
  :align: center

  flowchart LR
    classDef pipeIO fill:#F6E083
    classDef hidden fill:#ffffff,stroke:#ffffff

    subgraph input
        INPUT0:::pipeIO
        INPUT1:::pipeIO
    end

    INPUT0 --> TF1(tfsimple1)
    INPUT1 --> TF1
    INPUT0 --> TF2(tfsimple2)
    INPUT1 --> TF2
    TF1 -.-> |OUTPUT1| tf1( ):::hidden

    TF1 --> |OUTPUT0: INPUT| check
    TF1 --> |OUTPUT0: INPUT0| TF3(tfsimple3)
    TF2 --> |OUTPUT1: INPUT1| TF3
    TF2 -.-> |OUTPUT0| tf2( ):::hidden

    check --o |OUTPUT| TF3
    linkStyle 9 stroke:#CEE741,color:black;

    subgraph output
      OUTPUT0:::pipeIO
      OUTPUT1:::pipeIO
    end

    TF3 --> OUTPUT0
    TF3 --> OUTPUT1

In this example the last step tfsimple3 runs only if there are outputs from tfsimple1 and tfsimple2 but also data from the check step. However, if the step tfsimple3 is run it only receives the join of data from tfsimple1 and tfsimple2.

This example can be found in the pipeline-examples examples.

Trigger Joins Copy Section

You can also define multiple triggers which need to happen based on a particulr join type. For example:

:language: yaml

:caption: "*A pipeline with multiple triggers and a trigger join of type `any`. The pipeline has four inputs, but three of these are optional (signified by the dashed borders).*"
  :align: center

  flowchart LR
      classDef pipeIO fill:#F6E083
      classDef pipeIOopt fill:#F6E083,stroke-dasharray: 5 5;
      classDef trigger fill:#CEE741;
      classDef hidden fill:#ffffff,stroke:#ffffff
      classDef join fill:#CEE741;

      subgraph input
          ok1:::pipeIOopt
          ok2:::pipeIOopt
          INPUT:::pipeIO
          ok3:::pipeIOopt
      end

      ok1 --o any
      ok2 --o any
      any((any)):::trigger --o mul10
      linkStyle 0 stroke:#CEE741,color:green;
      linkStyle 1 stroke:#CEE741,color:green;
      linkStyle 2 stroke:#CEE741,color:green;

      INPUT --> mul10
      INPUT --> add10

      ok3 --o add10
      linkStyle 5 stroke:#CEE741,color:green;

      subgraph output
        OUTPUT:::pipeIO
      end

      mul10 -->|OUTPUT| anyOut(any):::join
      add10 --> |OUTPUT| anyOut
      anyOut --> OUTPUT

Here the mul10 step is run if data is seen on the pipeline inputs in the ok1 or ok2 tensors based on the any join type. If data is seen on ok3 then the add10 step is run.

If we changed the triggersJoinType for mul10 to inner then both ok1 and ok2 would need to appear before mul10 is run.

Pipeline Inputs Copy Section

Pipelines by default can be accessed synchronously via http/grpc or asynchronously via the Kafka topic created for them. However, it's also possible to create a pipeline to take input from one or more other pipelines by specifying an input section. If for example we already have the tfsimple pipeline shown below:

:language: yaml

We can create another pipeline which takes its input from this pipeline, as shown below:

:language: yaml

:caption: "*A pipeline taking as input the output of another pipeline.*"
  :align: center

  flowchart LR
      classDef pipeIO fill:#F6E083

      subgraph tfsimple.inputs
          INPUT0:::pipeIO
          INPUT1:::pipeIO
      end
      
      INPUT0 --> TF1(tfsimple1)
      INPUT1 --> TF1
      
      subgraph tfsimple.outputs
          OUTPUT0:::pipeIO
          OUTPUT1:::pipeIO
      end
      
      TF1 --> OUTPUT0
      TF1 --> OUTPUT1

      subgraph tfsimple-extended.inputs
        INPUT10(INPUT0):::pipeIO
        INPUT11(INPUT1):::pipeIO
      end

    OUTPUT0 --> INPUT10
    OUTPUT1 --> INPUT11

    INPUT10 --> TF2(tfsimple2)
    INPUT11 --> TF2

      subgraph tfsimple-extended.outputs
          OUTPUT10(OUTPUT0):::pipeIO
          OUTPUT11(OUTPUT1):::pipeIO
      end
      
      TF2 --> OUTPUT10
      TF2 --> OUTPUT11

In this way pipelines can be built to extend existing running pipelines to allow extensibility and sharing of data flows.

The spec follows the same spec for a step except that references to other pipelines are contained in the externalInputs section which takes the form of pipeline or pipeline.step references:

• .(inputs|outputs).<tensorName>
• .(step).<stepName>.<tensorName>

Tensor names are optional and only needed if you want to take just one tensor from an input or output.

There is also an externalTriggers section which allows triggers from other pipelines.

Further examples can be found in the pipeline-to-pipeline examples.

Present caveats:

• Circular dependencies are not presently detected.
• Pipeline status is local to each pipeline.

Data Centric Implementation Copy Chapter

Internally Pipelines are implemented using Kafka. Each input and output to a pipeline step has an associated Kafka topic. This has many advantages and allows auditing, replay and debugging easier as data is preserved from every step in your pipeline.

Tracing allows you to monitor the processing latency of your pipelines.

As each request to a pipelines moves through the steps its data will appear in input and output topics. This allows a full audit of every transformation to be carried out.

Source Contents Servers Index (`docs/source/contents/servers/index.md`) Copy Section

Servers

By default Seldon installs two server farms using MLServer and Triton with 1 replica each. Models are scheduled onto servers based on the server's resources and whether the capabilities of the server matches the requirements specified in the Model request. For example:

:language: yaml

This model specifies the requirement sklearn

There is a default capabilities for each server as follows:

• MLServer
  {literalinclude} ../../../../operator/config/serverconfigs/mlserver.yaml Copy

  :language: yaml
  :start-after: SELDON_SERVER_CAPABILITIES
  :end-before: SELDON_OVERCOMMIT_PERCENTAGE

• Triton
  {literalinclude} ../../../../operator/config/serverconfigs/triton.yaml Copy

  :language: yaml
  :start-after: SELDON_SERVER_CAPABILITIES
  :end-before: SELDON_OVERCOMMIT_PERCENTAGE

Custom Capabilities Copy Chapter

Servers can be defined with a capabilities field to indicate custom configurations (e.g. Python dependencies). For instance:

:language: yaml

These capabilities override the ones from the serverConfig: mlserver. A model that takes advantage of this is shown below:

:language: yaml

This above model will be matched with the previous custom server mlserver-134.

Servers can also be set up with the extraCapabilities that add to existing capabilities from the referenced ServerConfig. For instance:

:language: yaml

This server, mlserver-extra, inherits a default set of capabilities via serverConfig: mlserver.
These defaults are discussed above.
The extraCapabilities are appended to these to create a single list of capabilities for this server.

Models can then specify requirements to select a server that satisfies those requirements as follows.

:language: yaml

The capabilities field takes precedence over the extraCapabilities field.

For some examples see here.

Autoscaling of Servers Copy Chapter

Within docker we don't support this but for Kubernetes see here

Source Contents Tutorials Index (`docs/source/contents/tutorials/index.md`) Copy Section

Workflow and Tutorials

Workflow Copy Chapter

Seldon inference is built from atomic Model components. Models as shown here cover a wide range of artifacts including:

• Core machine learning models, e.g. a Tensorflow model.
• Feature transformations that might be built with custom python code.
• Drift detectors.
• Outlier detectors.
• Explainers
• Adversarial detectors.

A typical workflow for a production machine learning setup might be as follows:

1. You create a Tensorflow model for your core application use case and test this model in isolation to validate.
2. You create SKLearn feature transformation component before your model to convert the input into the correct form for your model. You also create Drift and Outlier detectors using Seldon's open source Alibi-detect library and test these in isolation.
3. You join these components together into a Pipeline for the final production setup.

These steps are shown in the diagram below:

Worked Examples Copy Chapter

See our selection of examples.

Source Contents Upgrading Index (`docs/source/contents/upgrading/index.md`) Copy Section

Upgrading

Upgrading from 2.6 - 2.7 Copy Chapter

All pods provisioned through the operator i.e. SeldonRuntime and Server resources now have the label app.kubernetes.io/name for identifying the pods.

Previously, the labelling has been inconsistent across different versions of Seldon Core 2, with mixture of app and app.kubernetes.io/name used.

If using the Prometheus operator ("Kube Prometheus"), please apply the v2.7.0 manifests for Seldon Core 2 according to the metrics documentation.

Note that these manifests need to be adjusted to discover metrics endpoints based on the existing setup.

If previous pod monitors had namespaceSelector fields set, these should be copied over and applied to the new manifests.

If namespaces do not matter, cluster-wide metrics endpoint discovery can be setup by modifying the namespaceSelector field in the pod monitors:

spec:
  namespaceSelector:
    any: true

Upgrading from 2.5 - 2.6 Copy Chapter

Release 2.6 brings with it new custom resources SeldonConfig and SeldonRuntime, which provide a new way to install Seldon Core 2 in Kubernetes. Upgrading in the same namespace will cause downtime while the pods are being recreated. Alternatively users can have an external service mesh or other means to be used over multiple namespaces to bring up the system in a new namespace and redeploy models before switch traffic between them.

If the new 2.6 charts are used to upgrade in an existing namespace models will eventually be redeloyed but there will be service downtime as the core components are redeployed.

Source Contents Apis Inference Index (`docs/source/contents/apis/inference/index.md`) Copy Section

Inference API

Seldon inference servers must respect the following API specification.

• Seldon, KServe, NVIDIA V2 Inference API Spec

In future, Seldon may provide extensions for use with Pipelines, Experiments and Explainers.

:maxdepth: 1
:hidden:

v2.md

Source Contents Apis Internal Index (`docs/source/contents/apis/internal/index.md`) Copy Section

Internal APIs

Seldon provides some internal gRPC services it uses to manage components.

• Agent API
• Chainer API

:maxdepth: 1
:hidden:

agent.md
chainer.md

Source Contents Apis Scheduler Index (`docs/source/contents/apis/scheduler/index.md`) Copy Section

Scheduler API

The Seldon scheduler API provides a gRPC service to allow Models, Servers, Experiments, and Pipelines to be managed. In Kubernetes the manager deployed by Seldon translates Kubernetes custom resource definitions into calls to the Seldon Scheduler.

In non-Kubernetes environments users of Seldon could create a client to directly control Seldon resources using this API.

Proto Definition Copy Chapter

:language: proto

Source Contents Development Release Index (`docs/source/contents/development/release/index.md`) Copy Section

Release process

Source Contents Getting Started Configuration Index (`docs/source/contents/getting-started/configuration/index.md`) Copy Section

Configuration

Seldon can be configured via various config files.

Kafka Configuration Copy Chapter

We allow configuration of the Kafka integration. In general this configuration looks like:

:language: json

The top level keys are:

• topicPrefix : the prefix to add to kafka topics created by Seldon
• consumerGroupIdPrefix : the prefix to add to Kafka consumer group IDs created by Seldon
• bootstrap.servers : the global bootstrap kafka servers to use
• consumer : consumer settings
• producer : producer settings
• streams : KStreams settings

For topicPrefix you can use any acceptable kafka topic characters which are a-z, A-Z, 0-9, . (dot), _ (underscore), and - (dash). We use . (dot) internally as topic naming separator so we would suggest you don't end your topic prefix with a dot for clarity. For illustration, an example topic could be seldon.default.model.mymodel.inputs where seldon is the topic prefix.

The consumerGroupIdPrefix will ensure that all consumer groups created have a given prefix.

Kubernetes Copy Section

For Kubernetes this is controlled via a ConfigMap called seldon-kafka whose default values are defined in the SeldonConfig custom resource.

:language: yaml
:start-after:     kafkaConfig
:end-before:     serviceConfig

When the SeldonRuntime is installed in a namespace a configMap will be created with these settings for Kafka configuration.

To customize the settings you can add and modify the Kafka configuration via Helm, for example below is a custom Helm values file that add compression for producers:

:language: yaml

To use this with the SeldonRuntime Helm chart:

helm install seldon-v2-runtime k8s/helm-charts/seldon-core-v2-runtime \
    --namespace seldon-mesh \
    --values k8s/samples/values-runtime-kafka-compression.yaml

Topic and consumer isolation Copy Section

If you use a shared Kafka cluster with other applications you may want to isolate the topic names and consumer group IDs from other users of the cluster to ensure there is no name clash. For this we provide two settings:

• topicPrefix: set a prefix for all topics
• consumerGroupIdPrefix: set a prefix for all consumer groups

An example to set this in the configuration when using the helm installation is showm below for creating the default SeldonConfig:

helm upgrade --install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh \
    --set controller.clusterwide=true \
    --set kafka.topicPrefix=myorg \
    --set kafka.consumerGroupIdPrefix=myorg

You can find a worked example here.

You can create alternate SeldonConfigs with different values or override values for particular SeldonRuntime installs.

Tracing Configuration Copy Chapter

We allow configuration of tracing. This file looks like:

:language: json

The top level keys are:

• enable : whether to enable tracing
• otelExporterEndpoint : The host and port for the OTEL exporter
• otelExporterProtocol : The protocol for the OTEL exporter. Currently used for
  jvm-based components only (such as dataflow-engine), because opentelemetry-java-instrumentation
  requires a http(s) URI for the endpoint but defaults to http/protobuf as a protocol.
  Because of this, gRPC connections (over http) can only be set up by setting this option to grpc
• ratio : The ratio of requests to trace. Takes values between 0 and 1 inclusive.

Kubernetes Copy Section

For Kubernetes this is controlled via a ConfigMap call seldon-tracing whose default value is shown below:

:language: yaml

Note, this ConfigMap is created via our Helm charts and there is usually no need to modify it manually.

At present Java instrumentation (for the dataflow engine) is duplicated via separate keys.

Source Contents Getting Started Docker Installation Index (`docs/source/contents/getting-started/docker-installation/index.md`) Copy Section

Docker Installation

Preparation Copy Chapter

1. git clone https://github.com/SeldonIO/seldon-core --branch=v2
2. Build Seldon CLI
3. Install Docker Compose (or directly from GitHub release if not using Docker Desktop).
4. Install make. This will depend on your version of Linux, for example on Ubuntu run sudo apt-get install build-essential.

Deploy Copy Chapter

From the project root run:

make deploy-local

This will run with latest images for the components.

Note: Triton and MLServer are large images at present (11G and 9G respectively) so will take time to download on first usage.

Run a particular version Copy Section

To run a particular release set the environment variable CUSTOM_IMAGE_TAG to the desired version before running the command, e.g.:

export CUSTOM_IMAGE_TAG=0.2.0
make deploy-local

GPU support Copy Section

To enable GPU on servers:

1. Make sure that nvidia-container-runtime is installed, follow link
2. Enable GPU: export GPU_ENABLED=1

Local Models Copy Section

To deploy with a local folder available for loading models set the environment variable LOCAL_MODEL_FOLDER to the folder, e.g.:

export LOCAL_MODEL_FOLDER=/home/seldon/models
make deploy-local

This folder will be mounted at /mnt/models. You can then specify models as shown below:

:language: yaml

If you have set the local model folder as above then this would be looking at /home/seldon/models/iris.

Tracing Copy Chapter

The default local install will provide Jaeger tracing at http://0.0.0.0:16686/search.

Metrics Copy Chapter

The default local install will expose Grafana at http://localhost:3000.

Undeploy Copy Chapter

From the project root run:

make undeploy-local

:maxdepth: 1
:hidden:

Source Contents Getting Started Kubernetes Installation Index (`docs/source/contents/getting-started/kubernetes-installation/index.md`) Copy Section

Kubernetes Installation

Prerequisites Copy Chapter

A running Kubernetes Cluster Copy Section

You will need a running Kubernetes cluster or can create a local KinD one for testing via Ansible.

Install Ecosystem Components Copy Section

You will also need to install our ecosystem components. For this we provide directions for Ansible to install these.

:header-rows: 1

* - Component
  - Summary
* - Kafka
  - Required for inference Pipeline usage.
* - Prometheus
  - (Optional) Exposes metrics.
* - Grafana
  - (Optional) UI for metrics.
* - OpenTelemetry
  - (Optional) Exposes tracing.
* - Jaeger
  - (Optional) UI for traces.

Install Copy Section

To install Seldon Core 2 itself you can choose from the following. At present, all require a clone of the source repository.

• Helm Installation (recommended for production systems)
• Ansible (recommended for test / dev / trial purposes)

The Kubernetes operator that is installed runs in namespaced mode so any resources you create need to be in the same namespace as you installed into.

Kustomize Copy Section

Our recommended and supported way to install is via Helm or Ansible. If you wish to use Kustomize then you can base your configuration on the raw yaml we create in the folder k8s/yaml or follow the steps in k8s/Makefile which illustrate how we build this yaml from our own Kustomize bases.

Operations Copy Chapter

• Security

:maxdepth: 1
:hidden:

helm.md
ansible.md
security/index.md

Source Contents Kubernetes Autoscaling Index (`docs/source/contents/kubernetes/autoscaling/index.md`) Copy Section

Autoscaling

Autoscaling in Seldon applies to various concerns:

• Inference servers autoscaling
• Model autoscaling
• Model memory overcommit

Inference servers autoscaling Copy Chapter

Autoscaling of servers can be done via HorizontalPodAutoscaler (HPA).

HPA can be applied to any deployed Server resource.
In this case HPA will manage the number of server replicas in the corresponding statefulset according to utilisation metrics (e.g. CPU or memory).

For example assuming that a triton server is deployed, then the user can attach an HPA based on cpu utilisation as follows:

kubectl autoscale server triton --cpu-percent=50 --min=1 --max=5

In this case, according to load, the system will add / remove server replicas to / from the triton statefulset.

It is worth considering the following points:

• If HPA adds a new server replica, this new replica will be included in any future scheduling decisions.
  In other words, when deploying a new model or rescheduling failed models this new replica will be considered.

• If HPA deletes an existing server replica, the scheduler will first attempt to drain any loaded model on this server replica before the server replica gets actually deleted. This is achieved by leveraging a PreStop hook on the server replica pod that triggers a process before receiving the termination signal. This draining process is capped by terminationGracePeriodSeconds, which the user can set (default is 2 minutes).

Therefore there should generally be minimal disruption to the inference workload during scaling.

For more details on HPA check this Kubernetes walk-through.

Autoscaling of inference servers via `seldon-scheduler` is under consideration for the roadmap. This allow for more fine grained interactions with model autoscaling.

Autoscaling via HPA for both Models and Servers using custom metrics from Prometheus is possible
for the special case of single model serving (i.e. single model per server). Check the detailed
documentation [here](hpa-rps-autoscaling.md)

Model autoscaling Copy Chapter

As each model server can serve multiple models, models can scale across the available replicas of the server according to load.

Autoscaling of models is enabled if at least MinReplicas or MaxReplicas is set in the model custom resource. Then according to load the system will scale the number of Replicas within this range.

For example the following model will be deployed at first with 1 replica and it can scale up according to load.

:language: yaml

Note that model autoscaling will not attempt to add extra servers if the desired number of replicas cannot be currently fulfilled by the current provisioned number of servers. This is a process left to be done by server autoscaling.

Additionally when the system autoscales, the initial model spec is not changed (e.g. the number of Replicas) and therefore the user cannot reset the number of replicas back to the initial specified value without an explicit change.

If only Replicas is specified by the user, autoscaling of models is disabled and the system will have exactly the number of replicas of this model deployed regardless of inference load.

Architecture Copy Section

The model autoscaling architecture is designed such as each agent decides on which models to scale up / down according to some defined internal metrics and then sends a triggering message to the scheduler. The current metrics are collected from the data plane (inference path), representing a proxy on how loaded is a given model with fulfilling inference requests.

Agent autoscaling stats collection Copy Section

Scale up logic:

The main idea is that we keep the "lag" for each model. We define the "lag" as the difference between incoming and outgoing requests in a given time period. If the lag crosses a threshold, then we trigger a model scale up event. This threshold can be defined via SELDON_MODEL_INFERENCE_LAG_THRESHOLD inference server environment variable.

Scale down logic:

For now we keep things simple and we trigger model scale down events if a model has not been used for a number of seconds. This is defined in SELDON_MODEL_INACTIVE_SECONDS_THRESHOLD inference server environment variable.

Each agent checks the above stats periodically and if any model hits the corresponding threshold, then the agent sends an event to the scheduler to request model scaling.

How often this process executes can be defined via SELDON_SCALING_STATS_PERIOD_SECONDS inference server environment variable.

Scheduler autoscaling Copy Section

The scheduler will perform model autoscale if:

• The model is stable (no state change in the last 5 minutes) and available.
• The desired number of replicas is within range. Note we always have a least 1 replica of any deployed model and we rely on over commit to reduce the resources used further.
• For scaling up, there is enough capacity for the new model replica.

Model memory overcommit Copy Chapter

Servers can hold more models than available memory if overcommit is swictched on (default yes). This allows under utilized models to be moved from inference server memory to allow for other models to take their place. Note that these evicted models are still registered and in the case future inference requests arrive, the system will reload the models back to memory before serving the requests. If traffic patterns for inference of models vary then this can allow more models than available server memory to be run on the system.

Source Contents Kubernetes Kafka Index (`docs/source/contents/kubernetes/kafka/index.md`) Copy Section

Kafka

Seldon Core 2 requires Kafka to implement data-centric inference Pipelines.
See our architecture documentation to learn more on how Seldon Core 2 uses Kafka.

Kafka integration is required to enable data-centric inference pipelines feature.
It is highly advice to configure Kafka integration to take full advantage of Seldon Core 2 features.

We list alternatives below.

Managed Kafka Copy Chapter

We recommend to use managed Kafka solution for production installation.
This allow to take away all the complexity on running secure and scalable Kafka cluster away.

We currently have tested and documented integration with following managed solutions:

• Confluent Cloud (security: SASL/PLAIN)
• Confluent Cloud (security: SASL/OAUTHBEARER)
• Amazon MSK (security: mTLS)
• Amazon MSK (security: SASL/SCRAM)
• Azure Event Hub (security: SASL/PLAIN)

See our Kafka security section for configuration examples.

Self Hosted Kafka Copy Chapter

:heading-offset: 2
:relative-docs: ..

Source Contents Kubernetes Metrics Index (`docs/source/contents/kubernetes/metrics/index.md`) Copy Section

Metrics

Metrics are exposed for scrapping by Prometheus.

Example Installation Copy Chapter

We recommend to install kube-prometheus that provides an all-in-one package with the Prometheus operator.

RBAC Copy Section

You will need to modify the default RBAC installed by kube-prometheus as described here.

From the prometheus folder in the project run:

kubectl apply -f rbac/cr.yaml

Monitors Copy Chapter

We use a PodMonitor for scrapping agent metrics. The envoy and server monitors are there for completeness but not presently needed.

kubectl apply -f monitors

Includes:

• Agent pod monitor. Monitors the metrics port of server inference pods.
• Server pod monitor. Monitors the server-metrics port of inference server pods.
• Envoy service monitor. Monitors the Envoy gateway proxies.
• Pipeline gateway pod monitor. Monitors the metrics port of pipeline gateway pods.

Pod monitors were chosen as ports for metrics are not exposed at service level as we do not have a top level service for server replicas but 1 headless service per replica. Future discussions could reference this.

Example Grafana Dashboard Copy Chapter

Check metrics for more information.

Reference Copy Chapter

• Prometheus CRDs

Source Contents Kubernetes Resources Index (`docs/source/contents/kubernetes/resources/index.md`) Copy Section

Resources

For Kubernetes usage we provide a set of custom resources for interacting with Seldon.

• SeldonRuntime - for installing Seldon in a particular namespace.
• Servers - for deploying sets of replicas of core inference servers (MLServer or Triton).
• Models - for deploying single machine learning models, custom transformation logic, drift detectors, outliers detectors and explainers.
• Experiments - for testing new versions of models.
• Pipelines - for connecting together flows of data between models.

Advanced Customization Copy Chapter

SeldonConfig and ServerConfig define the core installation configuration and machine learning inference server configuration for Seldon. Normally, you would not need to customize these but this may be required for your particular custom installation within your organisation.

• ServerConfigs - for defining new types of inference server that can be reference by a Server resource.
• SeldonConfig - for defining how seldon is installed

:maxdepth: 1
:hidden:

seldonruntime/index.md
model/index.md
experiment/index.md
pipeline/index.md
server/index.md
serverconfig/index.md
seldonconfig/index.md

Source Contents Kubernetes Scaling Index (`docs/source/contents/kubernetes/scaling/index.md`) Copy Section

Scaling

Models Copy Chapter

Models can be scaled by setting their replica count, e.g.

apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: iris
spec:
  storageUri: "gs://seldon-models/scv2/samples/mlserver_1.2.3/iris-sklearn"
  requirements:
  - sklearn
  memory: 100Ki
  replicas: 3

Currently, the number of replicas will need not to exceed the replicas of the Server the model is scheduled to.

Servers Copy Chapter

Servers can be scaled by setting their replica count, e.g.

apiVersion: mlops.seldon.io/v1alpha1
kind: Server
metadata:
  name: mlserver
  namespace: seldon
spec:
  replicas: 4
  serverConfig: mlserver

Currently, models scheduled to a server can only scale up to the server replica count.

Internal Components Copy Chapter

Seldon Core 2 runs with several control and dataplane components. The scaling of these resources is discussed below:

• Pipeline gateway.
  • This pipeline gateway handles REST and gRPC synchronous requests to Pipelines. It is stateless and can be scaled based on traffic demand.
• Model gateway.
  • This component pulls model requests from Kafka and sends them to inference servers. It can be scaled up to the partition factor of your Kafka topics. At present we set a uniform partition factor for all topics in one installation of Seldon Core 2.
• Dataflow engine.
  • The dataflow engine runs KStream topologies to manage Pipelines. It can run as multiple replicas and the scheduler will balance Pipelines to run across it with a consistent hashing load balancer. Each Pipeline is managed up to the partition factor of Kafka (presently hardwired to one).
• Scheduler.
  • This manages the control plane operations. It is presently required to be one replica as it maintains internal state within a BadgerDB held on local persistent storage (stateful set in Kubernetes). Performance tests have shown this not to be a bottleneck at present.
• Kubernetes Controller.
  • The Kubernetes controller manages resources updates on the cluster which it passes on to the Scheduler. It is by default one replica but has the ability to scale.
• Envoy
  • Envoy replicas get their state from the scheduler for routing information and can be scaled as needed.

Future Enhancements Copy Section

• Allow configuration of partition factor for data plane consistent hashing load balancer.
• Allow Model gateway and Pipeline gateway to use consistent hashing load balancer.
• Consider control plane scaling options.

Source Contents Kubernetes Service Meshes Index (`docs/source/contents/kubernetes/service-meshes/index.md`) Copy Section

Service Meshes

The Seldon models and pipelines are exposed via a single service endpoint in the install namespace called seldon-mesh. All models, pipelines and experiments can be reached via this single Service endpoint by setting appropriate headers on the inference REST/gRPC request. By this means Seldon is agnostic to any service mesh you may wish to use in your organisation. We provide some example integrations for some example service meshes below (alphabetical order):

• Ambassador
• Istio
• Traefik

We welcome help to extend these to other service meshes.

:maxdepth: 1
:hidden:

ambassador/index.md
istio/index.md
traefik/index.md

Source Contents Kubernetes Storage Secrets Index (`docs/source/contents/kubernetes/storage-secrets/index.md`) Copy Section

Storage Secrets

Inference artifacts referenced by Models can be stored in any of the storage backends supported by Rclone.
This includes local filesystems, AWS S3, and Google Cloud Storage (GCS), among others.
Configuration is provided out-of-the-box for public GCS buckets, which enables the use of Seldon-provided models like in the below example:

:language: yaml

This configuration is provided by the Kubernetes Secret seldon-rclone-gs-public.
It is made available to Servers as a preloaded secret.
You can define and use your own storage configurations in exactly the same way.

Configuration Format Copy Chapter

To define a new storage configuration, you need the following details:

• Remote name
• Remote type
• Provider parameters

A remote is what Rclone calls a storage location.
The type defines what protocol Rclone should use to talk to this remote.
A provider is a particular implementation for that storage type.
Some storage types have multiple providers, such as s3 having AWS S3 itself, MinIO, Ceph, and so on.

The remote name is your choice.
The prefix you use for models in spec.storageUri must be the same as this remote name.

The remote type is one of the values supported by Rclone.
For example, for AWS S3 it is s3 and for Dropbox it is dropbox.

The provider parameters depend entirely on the remote type and the specific provider you are using.
Please check the Rclone documentation for the appropriate provider.
Note that Rclone docs for storage types call the parameters properties and provide both config and env var formats--you need to use the config format.
For example, the GCS parameter --gcs-client-id described here should be used as client_id.

For reference, this format is described in the Rclone documentation.
Note that we do not support the use of opts discussed in that section.

Kubernetes Secrets Copy Chapter

Kubernetes Secrets are used to store Rclone configurations, or storage secrets, for use by Servers.
Each Secret should contain exactly one Rclone configuration.

A Server can use storage secrets in one of two ways:

• It can dynamically load a secret specified by a Model in its .spec.secretName
• It can use global configurations made available via preloaded secrets

The name of a Secret is entirely your choice, as is the name of the data key in that Secret.
All that matters is that there is a single data key and that its value is in the format described above.

It is possible to use preloaded secrets for some Models and dynamically loaded secrets for others.

Preloaded Secrets Copy Section

Rather than Models always having to specify which secret to use, a Server can load storage secrets ahead of time.
These can then be reused across many Models.

When using a preloaded secret, the Model definition should leave .spec.secretName empty.
The protocol prefix in .spec.storageUri still needs to match the remote name specified by a storage secret.

The secrets to preload are named in a centralised ConfigMap called seldon-agent.
This ConfigMap applies to all Servers managed by the same SeldonRuntime.
By default this ConfigMap only includes seldon-rclone-gs-public, but can be extended with your own secrets as shown below:

:language: yaml

The easiest way to change this is to update your SeldonRuntime.

• If your SeldonRuntime is configured using the seldon-core-v2-runtime Helm chart, the corresponding value is config.agentConfig.rclone.configSecrets.
  This can be used as shown below:
  yaml Copy

  config:
    agentConfig:
      rclone:
        configSecrets:
          - my-s3
          - custom-gcs
          - minio-in-cluster

• Otherwise, if your SeldonRuntime is configured directly, you can add secrets by setting .spec.config.agentConfig.rclone.config_secrets.
  This can be used as follows:
  yaml Copy

  apiVersion: mlops.seldon.io/v1alpha1
  kind: SeldonRuntime
  metadata:
    name: seldon
  spec:
    seldonConfig: default
    config:
      agentConfig:
        rclone:
          config_secrets:
            - my-s3
            - custom-gcs
            - minio-in-cluster
    ...

Examples Copy Chapter

````{group-tab} S3 MinIO

Assuming you have installed MinIO in the `minio-system` namespace, a corresponding secret could be:

```{literalinclude} ../../../../../samples/auth/minio-secret.yaml
:language: yaml
```

You can then reference this in a Model with `.spec.secretName`:

```{literalinclude} ../../../../../samples/models/sklearn-iris-minio.yaml
:language: yaml
```
````

````{group-tab} Google Cloud Storage

[GCS](https://rclone.org/googlecloudstorage/) can use [service accounts](https://cloud.google.com/iam/docs/service-accounts) for access.
You can generate the credentials for a service account using the [gcloud CLI](https://cloud.google.com/sdk/gcloud/reference/iam/service-accounts/keys/create):

```bash
gcloud iam service-accounts keys create \
  gcloud-application-credentials.json \
  --iam-account [SERVICE-ACCOUNT--NAME]@[PROJECT-ID].iam.gserviceaccount.com
```

The contents of `gcloud-application-credentials.json` can be put into a secret:

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: gcs-bucket
type: Opaque
stringData:
  gcs: |
    type: gcs
    name: gcs
    parameters:
      service_account_credentials: '<gcloud-application-credentials.json>'
```

You can then reference this in a Model with `.spec.secretName`:

```yaml
apiVersion: mlops.seldon.io/v1alpha1
kind: Model
metadata:
  name: mymodel
spec:
  storageUri: "gcs://my-bucket/my-path/my-pytorch-model"
  secretName: "gcs-bucket"
  requirements:
  - pytorch
```
````

Source Contents Kubernetes Tracing Index (`docs/source/contents/kubernetes/tracing/index.md`) Copy Section

Tracing

We support Open Telemetry tracing. By default all components will attempt to send OLTP events to seldon-collector.seldon-mesh:4317 which will export to Jaeger at simplest-collector.seldon-mesh:4317.

The components can be installed from the tracing/k8s folder. In future an Ansible playbook will be created. This installs a Open Telemetry collector and a simple Jaeger install with a service that can be port forwarded to at simplest.seldon-mesh:16686.

An example Jaeger trace is show below:

Source Contents Models Inference Artifacts Index (`docs/source/contents/models/inference-artifacts/index.md`) Copy Section

Inference Artifacts

To run your model inside Seldon you must supply an inference artifact that can be downloaded and run on one of MLServer or Triton inference servers. We list artifacts below by alphabetical order below.

:header-rows: 1

* - Type
  - Server
  - Tag
  - Example
* - Alibi-Detect
  - MLServer
  - `alibi-detect`
  - [example](../../examples/cifar10.md)
* - Alibi-Explain
  - MLServer
  - `alibi-explain`
  - [example](../../examples/explainer-examples.md)
* - DALI
  - Triton
  - `dali`
  - TBC
* - Huggingface
  - MLServer
  - `huggingface`
  - [example](../../examples/huggingface.md)
* - LightGBM
  - MLServer
  - `lightgbm`
  - [example](../../examples/model-zoo.md#lightgbm-model)
* - MLFlow
  - MLServer
  - `mlflow`
  - [example](../../examples/model-zoo.md#mlflow-wine-model)
* - ONNX
  - Triton
  - `onnx`
  - [example](../../examples/model-zoo.md#onnx-mnist-model)
* - OpenVino
  - Triton
  - `openvino`
  - TBC
* - Custom Python
  - MLServer
  - `python, mlserver`
  - [example](../../examples/pandasquery.md)
* - Custom Python
  - Triton
  - `python, triton`
  - [example](https://github.com/SeldonIO/triton-python-examples)
* - PyTorch
  - Triton
  - `pytorch`
  - [example](../../examples/model-zoo.md#pytorch-mnist-model)
* - SKLearn
  - MLServer
  - `sklearn`
  - [example](../../examples/income.md)
* - Spark Mlib
  - MLServer
  - `spark-mlib`
  - TBC
* - Tensorflow
  - Triton
  - `tensorflow`
  - [example](../../examples/cifar10.md)
* - TensorRT
  - Triton
  - `tensorrt`
  - TBC
* - Triton FIL
  - Triton
  - `fil`
  - TBC
* - XGBoost
  - MLServer
  - `xgboost`
  - [example](../../examples/model-zoo.md#xgboost-model)

Saving Model artifacts Copy Chapter

For many machine learning artifacts you can simply save them to a folder and load them into Seldon Core 2. Details are given below as well as a link to creating a custom model settings file if needed.

/* Detailed source-code truncated for AI context efficiency. */

Custom MLServer Model Settings Copy Chapter

For MLServer targeted models you can create a model-settings.json file to help MLServer load your model and place this alongside your artifact. See the MLServer project for details.

Custom Triton Configuration Copy Chapter

For Triton inference server models you can create a configuration config.pbtxt file alongside your artifact.

Notes Copy Chapter

• The tag field represents the tag you need to add to the requirements part of the Model spec for your artifact to be loaded on a compatible server. e.g. for an sklearn model:
  {literalinclude} ../../../../../samples/models/sklearn-iris-gs.yaml Copy

  :language: yaml

Source Contents Models Parameterized Models Index (`docs/source/contents/models/parameterized-models/index.md`) Copy Section

Parameterized Models

The Model specification allows parameters to be passed to the loaded model to allow customization. For example:

:language: yaml

This capability is only available for MLServer custom model runtimes. The named keys and values will be added to the model-settings.json file for the provided model in the
parameters.extra Dict. MLServer models are able to read these values in their load method.

Example Parameterized Models Copy Chapter

• Pandas Query

:maxdepth: 1
:hidden:

pandasquery.md

Source Contents Models Rclone Index (`docs/source/contents/models/rclone/index.md`) Copy Section

Rclone

We utilize Rclone to copy model artifacts from a storage location to the model servers. This allows users to take advantage of Rclones support for over 40 cloud storage backends including Amazon S3, Google Storage and many others.

For local storage while developing see here.

For authorization needed for cloud storage when running on Kubernetes see here.

Source Contents Getting Started Kubernetes Installation Security Index (`docs/source/contents/getting-started/kubernetes-installation/security/index.md`) Copy Section

Security

Seldon can be run with secure control plane and data plane operations. There are three areas of concern:

• Control Plane
• Kafka
• Data Plane

The various communication points between services are shown in the diagram below:

Control Plane Copy Chapter

TLS control plane activation is switched on and off via the environment variable: CONTROL_PLANE_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

Certificates will be loaded and used for the control plane gRPC services. The secrets or folders will be watched for updates (on certificate renewal) and automatically loaded again.

Helm Control Plane Install Copy Section

When installing seldon-core-v2-setup you can set the secret names for your certificates. If using cert-manager example discussed below this would be as follows:

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh \
      --set security.controlplane.protocol=SSL

Kafka Copy Chapter

Kafka secure activation is switched on and off via the environment variable: KAFKA_SECURITY_PROTOCOL whose values can be PLAINTEXT, SSL or SASL_SSL.

Examples are shown below:

• mTLS Strimzi example
• mTLS AWS MSK example
• SASL PLAIN with Confluent Cloud example
• SASL PLAIN with Azure Event Hub example
• SASL SCRAM with Strimzi example
• SASL SCRAM with AWS MSK example
• SASL OAUTH with Confluent Cloud example

Data Plane Copy Chapter

TLS Data plane activation is switched on and off via the environment variable: ENVOY_SECURITY_PROTOCOL whose values can be PLAINTEXT or SSL.

When activated this ensures TLS is used to communicate to Envoy via the xDS server as well as using the SDS service to send cretificates to envoy to use for upstream and downstream networking. Downstream is the external access to Seldon and upstream is the path from Envoy to the model servers or pipeline gateway.

Helm Data Plane Install Copy Section

When installing seldon-core-v2-setup you can set data plane operations to TLS as below. This assumes the secrets installed by the helm chart at the end of this section.

helm install seldon-v2 k8s/helm-charts/seldon-core-v2-setup/ -n seldon-mesh --set security.envoy.protocol=SSL

The above uses default secret names defined for the certificates installed. You can change the names of the required certificate secrets as shown in a longer configuration below (again using the default names for illustration).

For this we use the following updated Helm values (k8s/samples/values-tls-dataplane-example.yaml):

:language: yaml

We use Envoy internally to direct traffic and in Envoy's terminology upstream is for internal model servers called from Envoy while the downstream server is the entrypoint server running in Envoy to receive grpc and REST calls. The above settings ensure mTLS for internal "upstream" traffic while provides a standard SSL non-mTLS entrpoint.

To use the above with the seldon CLI you would need a custom config file as follow:

{
    "dataplane": {
	"tls": true,
	"skipSSLVerify": true
    }
}

We skip SSL Verify as these are internal self-signed certificates. For production use you would change this to the correct DNS name you are exposing the Seldon entrypoint.

Cetificate Providers Copy Chapter

The installer/cluster controller for Seldon needs to provide the certificates. As part of Seldon we provide an example set of certificate issuers and certificates using cert-manager.

Helm Copy Section

You can install Certificates into the desired namespace, here we use seldon-mesh as an example.

helm install seldon-v2-certs k8s/helm-charts/seldon-core-v2-certs/ -n seldon-mesh

:maxdepth: 1
:hidden:

strimzi-mtls.md
strimzi-sasl.md
confluent-sasl.md
confluent-oauth.md
azure-event-hub-sasl.md
aws-msk-sasl.md
aws-msk-mtls.md
reference.md

Source Contents Kubernetes Resources Experiment Index (`docs/source/contents/kubernetes/resources/experiment/index.md`) Copy Section

Experiment

An Experiment defines a traffic split between Models or Pipelines. This allows new versions of models and pipelines to be tested.

An experiment spec has three sections:

• candidates (required) : a set of candidate models to split traffic.
• default (optional) : an existing candidate who endpoint should be modified to split traffic as defined by the candidates.
  • Each candidate has a traffic weight. The percentage of traffic will be this weight divided by the sum of traffic weights.
• mirror (optional) : a single model to mirror traffic to the candidates. Responses from this model will not be returned to the caller.

An example experiment with a defaultModel is shown below:

:language: yaml

This defines a split of 50% traffic between two models iris and iris2. In this case we want to expose this traffic split on the existing endpoint created for the iris model. This allows us to test new versions of models (in this case iris2) on an existing endpoint (in this case iris). The default key defines the model whose endpoint we want to change. The experiment will become active when both underplying models are in Ready status.

An experiment over two separate models which exposes a new API endpoint is shown below:

:language: yaml

To call the endpoint add the header seldon-model: <experiment-name>.experiment in this case: seldon-model: experiment-iris.experiment. For example with curl:

curl http://${MESH_IP}/v2/models/experiment-iris/infer \
   -H "Content-Type: application/json" \
   -H "seldon-model: experiment-iris.experiment" \
   -d '{"inputs": [{"name": "predict", "shape": [1, 4], "datatype": "FP32", "data": [[1, 2, 3, 4]]}]}'

For examples see the local experiments notebook.

Pipeline Experiments Copy Chapter

Running an experiment between some pipelines is very similar. The difference is resourceType: pipeline needs to be defined and in this case the candidates or mirrors will refer to pipelines. An example is shown below:

:language: yaml

For an example see the local experiments notebook.

Mirror Experiments Copy Chapter

A mirror can be added easily for model or pipeline experiments. An example model mirror experiment is shown below:

:language: yaml

For an example see the local experiments notebook.

An example pipeline mirror experiment is shown below:

:language: yaml

For an example see the local experiments notebook.

Sticky Sessions Copy Chapter

To allow cohorts to get consistent views in an experiment each inference request passes back a response header x-seldon-route which can be passed in future requests to an experiment to bypass the random traffic splits and get a prediction from the sequence of models and pipelines used in the initial request.

Note: you must pass the normal seldon-model header along with the x-seldon-route header.

This is illustrated in the local experiments notebook.

Caveats:

• Note the models used will be the same but not necessarily the same replica instances. This means at present this will not work for stateful models that need to go to the same model replica instance.

Service Meshes Copy Chapter

As an alternative you can choose to run experiments at the service mesh level if you use one of the popular service meshes that allow header based routing in traffic splits. For further discussion see here.

Source Contents Kubernetes Resources Model Index (`docs/source/contents/kubernetes/resources/model/index.md`) Copy Section

Model

A Model is the core atomic building block. It specifies a machine learning artifact that will be loaded onto one of the running Servers. A model could be a standard machine learning inference component such as

• a Tensorflow model, PyTorch model or SKLearn model.
• an inference transformation component such as a SKLearn pipeline or a piece of custom python logic.
  a monitoring component such as an outlier detector or drift detector.
• An alibi-explain model explainer

An example is shown below for a SKLearn model for iris classification:

:language: yaml

Its Kubernetes spec has two core requirements

• A storageUri specifying the location of the artifact. This can be any rclone URI specification.
• A requirements list which provides tags that need to be matched by the Server that can run this artifact type. By default when you install Seldon we provide a set of Servers that cover a range of artifact types.

Source Contents Kubernetes Resources Pipeline Index (`docs/source/contents/kubernetes/resources/pipeline/index.md`) Copy Section

Pipeline

Pipelines allow one to connect flows of inference data transformed by Model components. A directed acyclic graph (DAG) of steps can be defined to join Models together. Each Model will need to be capable of receiving a V2 inference request and respond with a V2 inference response. An example Pipeline is shown below:

:language: yaml

The steps list shows three models: tfsimple1, tfsimple2 and tfsimple3. These three models each take two tensors called INPUT0 and INPUT1 of integers. The models produce two outputs OUTPUT0 (the sum of the inputs) and OUTPUT1 (subtraction of the second input from the first).

tfsimple1 and tfsimple2 take as inputs the input to the Pipeline: the default assumption when no explicit inputs are defined. tfsimple3 takes one V2 tensor input from each of the outputs of tfsimple1 and tfsimple2. As the outputs of tfsimple1 and tfsimple2 have tensors named OUTPUT0 and OUTPUT1 their names need to be changed to respect the expected input tensors and this is done with a tensorMap component providing this tensor renaming. This is only required if your models can not be directly chained together.

The output of the Pipeline is the output from the tfsimple3 model.

Detailed Specification Copy Chapter

The full GoLang specification for a Pipeline is shown below: