zenml Documentation

Technical Documentation: zenml-io/zenml

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (README.md) Copy Section

ZenML is built for ML or AI Engineers working on traditional ML use-cases, LLM workflows, or agents, in a company setting.

At it's core, ZenML allows you to write workflows (pipelines) that run on any infrastructure backend (stacks). You can embed any Pythonic logic within these pipelines, like training a model, or running an agentic loop. ZenML then operationalizes your application by:

1. Automatically containerizing and tracking your code.
2. Tracking individual runs with metrics, logs, and metadata.
3. Abstracting away infrastructure complexity.
4. Integrating your existing tools and infrastructure e.g. MLflow, Langgraph, Langfuse, Sagemaker, GCP Vertex, etc.
5. Allowing you to quickly iterate on experiments with an observable layer, in development and in production.

...amongst many other features.

ZenML is used by thousands of companies to run their AI workflows. Here are some featured ones:

_{(please email [email protected] if you want to be featured)}

🚀 Get Started (5 minutes) Copy Chapter

# Install ZenML with server capabilities
pip install "zenml[server]"  # pip install zenml will install a slimmer client

# Initialize your ZenML repository
zenml init

# Start local server or connect to a remote one
zenml login

You can then explore any of the examples in this repo. We recommend starting with the quickstart, which demonstrates core ZenML concepts: pipelines, steps, artifacts, snapshots, and deployments.

🏗️ Architecture Overview Copy Section

ZenML uses a client-server architecture with an integrated web dashboard (zenml-io/zenml-dashboard):

• Local Development: pip install "zenml[local]" - runs both client and server locally
• Production: Deploy server separately, connect with pip install zenml + zenml login <server-url>

🎮 Demo Copy Chapter

Here is a short demo:

🖼️ Resources Copy Chapter

The best way to learn about ZenML is through our comprehensive documentation and tutorials:

• Documentation - Complete product documentation
• Your First AI Pipeline - Build and evaluate an AI service in minutes
• Starter Guide - From zero to production in 30 minutes
• LLMOps Guide - Specific patterns for LLM applications
• SDK Reference - Complete SDK reference

📚 More examples Copy Chapter

1. Agent Architecture Comparison - Compare AI agents with LangGraph workflows, LiteLLM integration, and automatic visualizations via custom materializers
2. Deploying ML Models - Deploy classical ML models as production endpoints with monitoring and versioning
3. Deploying Agents - Document analysis service with pipelines, evaluation, and embedded web UI
4. E2E Batch Inference - Complete MLOps pipeline with feature engineering
5. LLM RAG Pipeline - Production RAG with evaluation loops
6. Agentic Workflow (Deep Research) - Orchestrate your agents with ZenML
7. Fine-tuning Pipeline - Fine-tune and deploy LLMs

🗣️ Chat With Your Pipelines: ZenML MCP Server Copy Chapter

Stop clicking through dashboards to understand your ML workflows. The ZenML MCP Server lets you query your pipelines, analyze runs, and trigger deployments using natural language through Claude Desktop, Cursor, or any MCP-compatible client.

💬 "Which pipeline runs failed this week and why?"
📊 "Show me accuracy metrics for all my customer churn models"  
🚀 "Trigger the latest fraud detection pipeline with production data"

Quick Setup:

1. Download the .dxt file from zenml-io/mcp-zenml
2. Drag it into Claude Desktop settings
3. Add your ZenML server URL and API key
4. Start chatting with your ML infrastructure

The MCP (Model Context Protocol) integration transforms your ZenML metadata into conversational insights, making pipeline debugging and analysis as easy as asking a question. Perfect for teams who want to democratize access to ML operations without requiring dashboard expertise.

🤖 Kitaru: Replay-Based Evals for AI Agents Copy Chapter

Running AI agents in production and wondering whether the cheaper model would have held, or whether the new prompt fixes Tuesday's failure? Kitaru is our open-source sister project for agents: traces you can run, not just read.

• Every run is a recording — wrap your agent once (PydanticAI today), or import the Langfuse traces you already collect
• Replay is re-execution — the run happens again against your real code, with recorded tool calls answered from the recording, so nothing touches live systems
• Fork with one thing changed — a cheaper model, a new prompt, the fix in your working tree — and trust the diff
• Scale the decision — freeze real traffic into cohorts, run experiments, gate CI on the result
• Self-hosted, by design — one small server on your infrastructure; replays execute on workers in your environment

The split is clean: ZenML is for ML pipelines, Kitaru is for agents.

pip install kitaru

👉 kitaru.ai · GitHub · Docs

🎓 Books & Resources Copy Section

ZenML is featured in these comprehensive guides to production AI systems.

🤝 Join ML Engineers Building the Future of AI Copy Chapter

Contribute:

• 🌟 Star us on GitHub - Help others discover ZenML
• 🤝 Contributing Guide - Start with good-first-issue
• 💻 Write Integrations - Add your favorite tools

Stay Updated:

• 🗺 Public Roadmap - See what's coming next
• 📰 Blog - Best practices and case studies
• 🎙 Slack - Talk with AI practitioners

❓ FAQs from ML Engineers Like You Copy Chapter

Q: "Do I need to rewrite my agents or models to use ZenML?"

A: No. Wrap your existing code in a @step. Keep using scikit-learn, PyTorch, LangGraph, LlamaIndex, or raw API calls. ZenML orchestrates your tools, it doesn't replace them.

Q: "How is this different from LangSmith/Langfuse?"

A: They provide excellent observability for LLM applications. We orchestrate the full MLOps lifecycle for your entire AI stack. With ZenML, you manage both your classical ML models and your AI agents in one unified framework, from development and evaluation all the way to production deployment.

Q: "Can I use my existing MLflow/W&B setup?"

A: Yes! ZenML integrates with both MLflow and Weights & Biases. Your experiments, our pipelines.

Q: "Is this just MLflow with extra steps?"

A: No. MLflow tracks experiments. We orchestrate the entire development process – from training and evaluation to deployment and monitoring – for both models and agents.

Q: "How do I configure ZenML with Kubernetes?"

A: ZenML integrates with Kubernetes through the native Kubernetes orchestrator, Kubeflow, and other K8s-based orchestrators. See our Kubernetes orchestrator guide and Kubeflow guide, plus deployment documentation.

Q: "What about cost? I can't afford another platform."

A: ZenML's open-source version is free forever. You likely already have the required infrastructure (like a Kubernetes cluster and object storage). We just help you make better use of it for MLOps.

🛠 VS Code / Cursor Extension Copy Section

Manage pipelines directly from your editor:

🖥️ VS Code Extension in Action!

Install from VS Code Marketplace.

📜 License Copy Chapter

ZenML is distributed under the terms of the Apache License Version 2.0. See
LICENSE for details.

README (`docs/README.md`) Copy Section

ZenML Docs

General documentation Copy Chapter

We write our documentation in Markdown files and use GitBook to build it.
The documentation source files can be found in this repository at docs/book

• You can edit the docs by simply editing the markdown files. Once the changes
  get merged into main, the docs will be automatically refreshed. (Note: you
  might need to refresh your browser.)
• If you're adding a new page, make sure to add a reference to it inside the tables of contents docs/book/toc.md
• Make sure to be consistent with the existing docs (similar style/messaging, reuse existing images if they fit your use case)

SDK Docs Copy Chapter

The ZenML SDK docs are generated from our Python docstrings using mkdocs.
The SDK docs will be automatically updated each release using a Github workflow and can be found
athttps://sdkdocs.zenml.io.

Building the SDK Docs locally Copy Section

To build them locally follow these steps:

• Clone the repository
• Install ZenML and Jinja2 dependencies

pip3 install -e ".[server,dev]"
pip3 install "Jinja2==3.0.3"

• Modify docs/mkdocs.yml as follows:

watch:
  - ../src/zenml

watch:
  - src/zenml

• Run python3 docs/mkdocstrings_helper.py
• Run:

rm -rf src/zenml/zen_stores/migrations/env.py
rm -rf src/zenml/zen_stores/migrations/versions
rm -rf src/zenml/zen_stores/migrations/script.py.mako

• Run mkdocs serve -f docs/mkdocs.yml -a localhost: from the repository root -
  running it from elsewhere can lead to unexpected errors. This script will compose the docs hierarchy
  and serve it (http://127.0.0.1:/).

Link Checker Tool Copy Chapter

The link_checker.py script is a utility tool for managing and validating links in our documentation.
It helps maintain the quality of our documentation by:

• Finding broken or outdated links
• Converting relative links to absolute URLs
• Validating link accessibility
• Supporting both directory-wide and file-specific scanning

Usage Copy Section

The script supports several modes of operation:

# Find all links containing 'docs.zenml.io' in a directory
python link_checker.py --dir docs/book --substring docs.zenml.io

# Check specific files for links containing 'how-to'
python link_checker.py --files file1.md file2.md --substring how-to

# Preview link replacements without making changes
python link_checker.py --files file1.md --replace-links --dry-run

# Replace and validate links
python link_checker.py --files file1.md --replace-links --validate-links

# Transform paths using custom URL mappings
python link_checker.py --dir docs/book --replace-links --url-mapping user-guide=user-guides

Features Copy Section

1. Link Detection:

   • Scans for various link types (inline, reference-style, HTML, bare URLs)
   • Supports substring-based filtering
   • Works with both directories and individual files

2. Link Transformation:

   • Converts relative documentation links to absolute URLs
   • Handles README.md files and various link formats
   • Preserves fragments and query parameters
   • Supports custom URL path mappings to transform specific path segments

3. Link Validation:

   • Validates links via HTTP requests
   • Parallel validation for better performance
   • Detailed error reporting
   • Uses narrow default CI policies for external domains that block automated checks. These policies are status-specific: approved bot-hostile URLs or domains may accept specific statuses such as 403, 406, 429, or 502, while non-exempt hard failures such as 404 and 410 still fail.

Requirements Copy Section

For link validation functionality, the requests package is required:

pip install requests

Contributors Copy Chapter

We welcome and recognize all contributions. You can see a list of current contributors here.

Book Api Docs README (`docs/book/api-docs/README.md`) Copy Section

The ZenML API provides programmatic access to ZenML services beyond what's available in the Python SDK. Copy Chapter

Overview

Welcome to the ZenML API documentation. This guide provides information about both the open-source (OSS) and Pro API endpoints available in the ZenML platform.

Use Cases Copy Chapter

While the ZenML Python SDK covers most workflow requirements, the Server API offers additional utility for:

• Automated scaling of operations in production environments.
• Integration with external monitoring and logging systems.
• Programmatic management and audit of resources and configurations.
• Enterprise team management and access control (Pro).

By leveraging the ZenML API, users can enhance the robustness and control of their machine learning workflows, ensuring operations are both efficient and scalable.

Book Sdk Docs README (`docs/book/sdk-docs/README.md`) Copy Section

Overview

ZenML and Kitaru each provide SDKs for working with their APIs:

• ZenML provides a Python SDK and CLI for building, managing, and deploying production-ready ML pipelines.
• Kitaru provides Python and TypeScript SDKs for recording, replaying, and improving AI agents. Its CLI ships with the Python package.

ZenML SDK Copy Chapter

The ZenML SDK provides Python tools and interfaces for building, managing, and deploying production-ready machine learning pipelines. This documentation helps you get the most out of ZenML's features.

Kitaru SDK Copy Chapter

Kitaru is ZenML's sibling project for recording, replaying, and improving AI agents: traces you can run, not just read. Its Python and TypeScript SDKs talk to the same Kitaru server over the same REST API. Framework adapters support Python and TypeScript agents. The generated Python SDK and CLI reference is published on a dedicated site.

Book User Guide README (`docs/book/user-guide/README.md`) Copy Section

Learn ZenML

Discover how to build production-ready ML pipelines with ZenML and evaluate agentic systems with Kitaru through our curated learning resources. Whether you're looking for step-by-step instructions, complete project implementations, or specific examples, you'll find resources to accelerate your workflow.

Kitaru is ZenML's sibling project for evals and improvement loops for agentic systems. It works from recorded or imported production traces through investigation, cohorts, evaluators, replay experiments, regression testing, and CI. ZenML orchestrates pipelines; Kitaru runs this workflow on its own server and workers.

Guides Copy Chapter

Step-by-step instructions to help you master ZenML concepts and features.

Projects Copy Chapter

Complete end-to-end implementations that showcase ZenML in real-world scenarios.
See all projects in our website →

Examples Copy Chapter

Focused code snippets and templates that address specific ML workflow challenges.
See all examples in GitHub →

Book Component Guide Alerters README (`docs/book/component-guide/alerters/README.md`) Copy Section

icon: message-exclamation Copy Chapter

Alerters

Alerters allow you to send messages to chat services (like Slack, Discord, Mattermost, etc.) from within your
pipelines. This is useful to immediately get notified when failures happen, for general monitoring/reporting, and also
for building human-in-the-loop ML.

Alerter Flavors Copy Chapter

Currently, the SlackAlerter and DiscordAlerter are the available alerter integrations. However, it is straightforward to
extend ZenML and build an alerter for other chat services.

{% hint style="info" %}
If you would like to see the available flavors of alerters in your terminal, you can use the following command:

zenml alerter flavor list

{% endhint %}

How to use Alerters with ZenML Copy Chapter

Each alerter integration comes with specific standard steps that you can use out of the box.

However, you first need to register an alerter component in your terminal:

zenml alerter register <ALERTER_NAME> ...

Then you can add it to your stack using

zenml stack register ... -al <ALERTER_NAME>

{% hint style="info" %}
Stacks can have multiple alerters attached. ZenML treats the first attached alerter as the default one, and singular accessors such as Client().active_stack.alerter resolve to that default alerter.
{% endhint %}

Afterward, you can import the alerter standard steps provided by the respective integration and directly use them in
your pipelines.

Using the Ask Step for Human-in-the-Loop Workflows Copy Chapter

All alerters provide an ask() method and corresponding ask steps that enable human-in-the-loop workflows. These are essential for:

• Getting approval before deploying models to production
• Confirming critical pipeline decisions
• Manual intervention points in automated workflows

How Ask Steps Work Copy Section

Ask steps (like discord_alerter_ask_step and slack_alerter_ask_step):

1. Post a message to your chat service with your question
2. Wait for user response containing specific approval or disapproval keywords
3. Return a boolean - True if approved, False if disapproved or timeout

from zenml import step, pipeline
from zenml.integrations.slack.steps.slack_alerter_ask_step import slack_alerter_ask_step

@step
def train_model():
    # Training logic here - this is a placeholder function
    return "trained_model_object"

@step
def deploy_model(model, approved: bool) -> None:
    if approved:
        # Deploy the model to production
        print("Deploying model to production...")
        # deployment logic here
    else:
        print("Deployment cancelled by user")

@pipeline
def deployment_pipeline():
    trained_model = train_model()
    # Ask for human approval before deployment
    approved = slack_alerter_ask_step("Deploy model to production?")
    deploy_model(trained_model, approved)

Default Response Keywords Copy Section

By default, alerters recognize these response options:

Approval: approve, LGTM, ok, yes
Disapproval: decline, disapprove, no, reject

Customizing Response Keywords Copy Section

You can customize the approval and disapproval keywords using alerter parameters:

from zenml.integrations.slack.steps.slack_alerter_ask_step import slack_alerter_ask_step
from zenml.integrations.slack.alerters.slack_alerter import SlackAlerterParameters

# Use custom approval/disapproval keywords
params = SlackAlerterParameters(
    approve_msg_options=["deploy", "ship it", "✅"],
    disapprove_msg_options=["stop", "cancel", "❌"]
)

approved = slack_alerter_ask_step(
    "Deploy model to production?", 
    params=params
)

Important Notes Copy Section

• Return Type: Ask steps return a boolean value - ensure your pipeline logic handles this correctly
• Keywords: Response keywords are case-sensitive (except Slack, which converts to lowercase)
• Timeout: If no valid response is received within the timeout period, the step returns False
• Permissions: Ensure your bot has permissions to read messages in the target channel

Book Component Guide Annotators README (`docs/book/component-guide/annotators/README.md`) Copy Section

icon: expand Copy Chapter

Annotators

Annotators are a stack component that enables the use of data annotation as part of your ZenML stack and pipelines. You
can use the associated CLI command to launch annotation, configure your datasets and get stats on how many labeled tasks
you have ready for use.

Data annotation/labeling is a core part of MLOps that is frequently left out of the conversation. ZenML will
incrementally start to build features that support an iterative annotation workflow that sees the people doing
labeling (and their workflows/behaviors) as integrated parts of their ML process(es).

There are a number of different places in the ML lifecycle where this can happen:

• At the start: You might be starting out without any data, or with a ton of data but no clear sense of which parts
  of it are useful to your particular problem. It’s not uncommon to have a lot of data but to be lacking accurate labels
  for that data. So you can start and get great value from bootstrapping your model: label some data, train your model,
  and use your model to suggest labels allowing you to speed up your labeling, iterating on and on in this way. Labeling
  data early on in the process also helps clarify and condense down your specific rules and standards. For example, you
  might realize that you need to have specific definitions for certain concepts so that your labeling efforts are
  consistent across your team.
• As new data comes in: New data will likely continue to come in, and you might want to check in with the labeling
  process at regular intervals to expose yourself to this new data. (You’ll probably also want to have some kind of
  automation around detecting data or concept drift, but for certain kinds of unstructured data you probably can never
  completely abandon the instant feedback of actual contact with the raw data.)
• Samples generated for inference: Your model will be making predictions on real-world data being passed in. If you
  store and label this data, you’ll gain a valuable set of data that you can use to compare your labels with what the
  model was predicting, another possible way to flag drifts of various kinds. This data can then (subject to
  privacy/user consent) be used in retraining or fine-tuning your model.
• Other ad hoc interventions: You will probably have some kind of process to identify bad labels, or to find the
  kinds of examples that your model finds really difficult to make correct predictions. For these, and for areas where
  you have clear class imbalances, you might want to do ad hoc annotation to supplement the raw materials your model has
  to learn from.

ZenML currently offers standard steps that help you tackle the above use cases, but the stack component and abstraction
will continue to be developed to make it easier to use.

When to use it Copy Section

The annotator is an optional stack component in the ZenML Stack. We designed our abstraction to fit into the larger ML
use cases, particularly the training and deployment parts of the lifecycle.

The core parts of the annotation workflow include:

• using labels or annotations in your training steps in a seamless way
• handling the versioning of annotation data
• allow for the conversion of annotation data to and from custom formats
• handle annotator-specific tasks, for example, the generation of UI config files that Label Studio requires for the web
  annotation interface

List of available annotators Copy Section

For production use cases, some more flavors can be found in specific integrations modules. In terms of annotators,
ZenML features integrations with the following tools.

If you would like to see the available flavors for annotators, you can use the command:

zenml annotator flavor list

How to use it Copy Section

The available implementation of the annotator is built on top of the Label
Studio integration, which means that using an annotator currently is no
different from what's described on the Label Studio page: How to use
it?. (Pigeon is also supported, but has a
very limited functionality and only works within Jupyter notebooks.)

A note on names Copy Section

The various annotation tools have mostly standardized around the naming of key concepts as part of how they build their
tools. Unfortunately, this hasn't been completely unified so ZenML takes an opinion on which names we use for our stack
components and integrations. Key differences to note:

• Label Studio refers to the grouping of a set of annotations/tasks as a 'Project', whereas most other tools use the
  term 'Dataset', so ZenML also calls this grouping a 'Dataset'.
• The individual meta-unit for 'an annotation + the source data' is referred to in different ways, but at ZenML (and
  with Label Studio) we refer to them as 'tasks'.

The remaining core concepts ('annotation' and 'prediction', in particular) are broadly used among annotation tools.

Book Component Guide Artifact Stores README (`docs/book/component-guide/artifact-stores/README.md`) Copy Section

icon: folder-closed Copy Chapter

Artifact Stores

The Artifact Store is a central component in any MLOps stack. As the name suggests, it acts as a data persistence layer where artifacts (e.g. datasets, models) ingested or generated by the machine learning pipelines are stored.

ZenML automatically serializes and saves the data circulated through your pipelines in the Artifact Store: datasets, models, data profiles, data and model validation reports, and generally any object that is returned by a pipeline step. This is coupled with tracking in ZenML to provide extremely useful features such as caching and provenance/lineage tracking and pipeline reproducibility.

{% hint style="info" %}
Not all objects returned by pipeline steps are physically stored in the Artifact Store, nor do they have to be. How artifacts are serialized and deserialized and where their contents are stored are determined by the particular implementation of the Materializer associated with the artifact data type. The majority of Materializers shipped with ZenML use the Artifact Store which is part of the active Stack as the location where artifacts are kept.

If you need to store a particular type of pipeline artifact in a different medium (e.g. use an external model registry to store model artifacts, or an external data lake or data warehouse to store dataset artifacts), you can write your own Materializer to implement the custom logic required for it. In contrast, if you need to use an entirely different storage backend to store artifacts, one that isn't already covered by one of the ZenML integrations, you can extend the Artifact Store abstraction to provide your own Artifact Store implementation.
{% endhint %}

In addition to pipeline artifacts, the Artifact Store may also be used as storage backed by other specialized stack components that need to store their data in the form of persistent object storage. The Great Expectations Data Validator is such an example.

Related concepts:

• the Artifact Store is a type of Stack Component that needs to be registered as part of your ZenML Stack.
• the objects circulated through your pipelines are serialized and stored in the Artifact Store using Materializer. Materializers implement the logic required to serialize and deserialize the artifact contents and to store them and retrieve their contents to/from the Artifact Store.

When to use it Copy Section

The Artifact Store is a mandatory component in the ZenML stack. It is used to store all artifacts produced by pipeline runs, and you are required to configure it in all of your stacks.

Artifact Store Flavors

Out of the box, ZenML comes with a local artifact store already part of the default stack that stores artifacts on your local filesystem. Additional Artifact Stores are provided by integrations:

If you would like to see the available flavors of Artifact Stores, you can use the command:

zenml artifact-store flavor list

{% hint style="info" %}
Every Artifact Store has a path attribute that must be configured when it is registered with ZenML. This is a URI pointing to the root path where all objects are stored in the Artifact Store. It must use a URI schema that is supported by the Artifact Store flavor. For example, the S3 Artifact Store will need a URI that contains the s3:// schema:

zenml artifact-store register s3_store -f s3 --path s3://my_bucket

{% endhint %}

How to use it Copy Section

The Artifact Store provides low-level object storage services for other ZenML mechanisms. When you develop ZenML pipelines, you normally don't even have to be aware of its existence or interact with it directly. ZenML provides higher-level APIs that can be used as an alternative to store and access artifacts:

• return one or more objects from your pipeline steps to have them automatically saved in the active Artifact Store as pipeline artifacts.
• retrieve pipeline artifacts from the active Artifact Store after a pipeline run is complete.

You will probably need to interact with the low-level Artifact Store API directly:

• if you implement custom Materializers for your artifact data types
• if you want to store custom objects in the Artifact Store

The Artifact Store API

All ZenML Artifact Stores implement the same IO API that resembles a standard file system. This allows you to access and manipulate the objects stored in the Artifact Store in the same manner you would normally handle files on your computer and independently of the particular type of Artifact Store that is configured in your ZenML stack.

Accessing the low-level Artifact Store API can be done through the following Python modules:

• zenml.io.fileio provides low-level utilities for manipulating Artifact Store objects (e.g. open, copy, rename , remove, mkdir). These functions work seamlessly across Artifact Stores types. They have the same signature as the Artifact Store abstraction methods ( in fact, they are one and the same under the hood).
• zenml.utils.io_utils includes some higher-level helper utilities that make it easier to find and transfer objects between the Artifact Store and the local filesystem or memory.

{% hint style="info" %}
When calling the Artifact Store API, you should always use URIs that are relative to the Artifact Store root path, otherwise, you risk using an unsupported protocol or storing objects outside the store. You can use the Repository singleton to retrieve the root path of the active Artifact Store and then use it as a base path for artifact URIs, e.g.:

import os
from zenml.client import Client
from zenml.io import fileio

root_path = Client().active_stack.artifact_store.path

artifact_contents = "example artifact"
artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.txt")
fileio.makedirs(artifact_path)
with fileio.open(artifact_uri, "w") as f:
    f.write(artifact_contents)

When using the Artifact Store API to write custom Materializers, the base artifact URI path is already provided. See the documentation on Materializers for an example.
{% endhint %}

The following are some code examples showing how to use the Artifact Store API for various operations:

• creating folders, writing and reading data directly to/from an artifact store object

import os
from zenml.utils import io_utils
from zenml.io import fileio

from zenml.client import Client

root_path = Client().active_stack.artifact_store.path

artifact_contents = "example artifact"
artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.txt")
fileio.makedirs(artifact_path)
io_utils.write_file_contents_as_string(artifact_uri, artifact_contents)

import os
from zenml.utils import io_utils

from zenml.client import Client

root_path = Client().active_stack.artifact_store.path

artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.txt")
artifact_contents = io_utils.read_file_contents_as_string(artifact_uri)

• using a temporary local file/folder to serialize and copy in-memory objects to/from the artifact store (heavily used in Materializers to transfer information between the Artifact Store and external libraries that don't support writing/reading directly to/from the artifact store backend):

import os
import tempfile
import external_lib
from zenml.client import Client
from zenml.io import fileio

root_path = Client().active_stack.artifact_store.path

artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.json")
fileio.makedirs(artifact_path)

with tempfile.NamedTemporaryFile(
        mode="w", suffix=".json", delete=True
) as f:
    external_lib.external_object.save_to_file(f.name)
    # Copy it into artifact store
    fileio.copy(f.name, artifact_uri)

import os
import tempfile
import external_lib
from zenml.client import Client
from zenml.io import fileio

root_path = Client().active_stack.artifact_store.path

artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.json")

with tempfile.NamedTemporaryFile(
        mode="w", suffix=".json", delete=True
) as f:
    # Copy the serialized object from the artifact store
    fileio.copy(artifact_uri, f.name)
    external_lib.external_object.load_from_file(f.name)

Book Component Guide Container Registries README (`docs/book/component-guide/container-registries/README.md`) Copy Section

icon: box Copy Chapter

Container Registries

The container registry is an essential part of most remote MLOps stacks. It is used to store container images that are built to run machine learning pipelines in remote environments. Containerization of the pipeline code creates a portable environment that allows code to run in an isolated manner.

When to use it Copy Section

The container registry is needed whenever other components of your stack need to push or pull container images. Currently, this is the case for most of ZenML's remote orchestrators , step operators, and some model deployers. These containerize your pipeline code and therefore require a container registry to store the resulting Docker images. Take a look at the documentation page of the component you want to use in your stack to see if it requires a container registry or even a specific container registry flavor.

Container Registry Flavors Copy Section

ZenML comes with a few container registry flavors that you can use:

• Default flavor: Allows any URI without validation. Use this if you want to use a local container registry or when using a remote container registry that is not covered by other flavors.
• Specific flavors: Validates your container registry URI and performs additional checks to ensure you're able to push to the registry.

{% hint style="warning" %}
We highly suggest using the specific container registry flavors in favor of the default one to make use of the additional URI validations.
{% endhint %}

If you would like to see the available flavors of container registries, you can use the command:

zenml container-registry flavor list

Book Component Guide Data Validators README (`docs/book/component-guide/data-validators/README.md`) Copy Section

How to enhance and maintain the quality of your data and the performance of your models with data profiling and validation icon: chart-column Copy Chapter

Data Validators

Without good data, even the best machine learning models will yield questionable results. A lot of effort goes into ensuring and maintaining data quality not only in the initial stages of model development, but throughout the entire machine learning project lifecycle. Data Validators are a category of ML libraries, tools and frameworks that grant a wide range of features and best practices that should be employed in the ML pipelines to keep data quality in check and to monitor model performance to keep it from degrading over time.

Data profiling, data integrity testing, data and model drift detection are all ways of employing data validation techniques at different points in your ML pipelines where data is concerned: data ingestion, model training and evaluation and online or batch inference. Data profiles and model performance evaluation results can be visualized and analyzed to detect problems and take preventive or correcting actions.

Related concepts:

• the Data Validator is an optional type of Stack Component that needs to be registered as part of your ZenML Stack.
• Data Validators used in ZenML pipelines usually generate data profiles and data quality check reports that are versioned and stored in the Artifact Store and can be retrieved and visualized later.

When to use it Copy Section

Data-centric AI practices are quickly becoming mainstream and using Data Validators are an easy way to incorporate them into your workflow. These are some common cases where you may consider employing the use of Data Validators in your pipelines:

• early on, even if it's just to keep a log of the quality state of your data and the performance of your models at different stages of development.
• if you have pipelines that regularly ingest new data, you should use data validation to run regular data integrity checks to signal problems before they are propagated downstream.
• in continuous training pipelines, you should use data validation techniques to compare new training data against a data reference and to compare the performance of newly trained models against previous ones.
• when you have pipelines that automate batch inference or if you regularly collect data used as input in online inference, you should use data validation to run data drift analyzes and detect training-serving skew, data drift and model drift.

Data Validator Flavors

Data Validator are optional stack components provided by integrations. The following table lists the currently available Data Validators and summarizes their features and the data types and model types that they can be used with in ZenML pipelines:

If you would like to see the available flavors of Data Validator, you can use the command:

zenml data-validator flavor list

How to use it Copy Section

Every Data Validator has different data profiling and testing capabilities and uses a slightly different way of analyzing your data and your models, but it generally works as follows:

• first, you have to configure and add a Data Validator to your ZenML stack
• every integration includes one or more builtin data validation steps that you can add to your pipelines. Of course, you can also use the libraries directly in your own custom pipeline steps and simply return the results (e.g. data profiles, test reports) as artifacts that are versioned and stored by ZenML in its Artifact Store.
• you can access the data validation artifacts in subsequent pipeline steps, or fetch them afterwards to process them or visualize them as needed.

Consult the documentation for the particular Data Validator flavor that you plan on using or are using in your stack for detailed information about how to use it in your ZenML pipelines.

Book Component Guide Deployers README (`docs/book/component-guide/deployers/README.md`) Copy Section

icon: rocket-launch Copy Chapter

Deployers

Pipeline deployment is the process of making ZenML pipelines available as long-running HTTP services for real-time execution. Unlike traditional batch execution through orchestrators, deployers create persistent web services that can handle on-demand pipeline invocations through HTTP requests.

Deployers are stack components responsible for managing the deployment of pipelines as containerized HTTP services that expose REST APIs for pipeline execution.

A deployed pipeline becomes a web service that can be invoked multiple times in parallel, receiving parameters through HTTP requests and returning pipeline outputs as JSON responses. This enables real-time inference, interactive workflows, and integration with web applications.

When to use it? Copy Section

Deployers are optional components in the ZenML stack. They are useful in the following scenarios:

• Real-time Pipeline Execution: Execute pipelines on-demand through HTTP requests rather than scheduled batch runs
• Interactive Workflows: Build applications that need immediate pipeline responses
• API Integration: Expose ML workflows as REST APIs for web applications or microservices
• Real-time Inference: Serve ML models through pipeline-based inference workflows
• Agent-based Systems: Create AI agents that execute pipelines in response to external events

Use deployers when you need request-response patterns, and orchestrators for scheduled, batch, or long-running workflows.

Deployer Flavors Copy Section

Out of the box, ZenML comes with a local deployer already part of the default stack that deploys pipelines on your local machine in the form of background processes. Additional Deployers are provided by integrations:

If you would like to see the available flavors of deployers, you can use the command:

zenml deployer flavor list

How to use it Copy Section

You don't need to directly interact with the ZenML deployer stack component in your code. As long as the deployer that you want to use is part of your active ZenML stack, you can simply deploy a pipeline or snapshot using the ZenML CLI or the ZenML SDK. The resulting deployment can be managed using the ZenML CLI or the ZenML SDK.

Examples:

• just use the default stack - it has a default local deployer that will deploy the pipeline on your local machine in the form of a background process:

zenml stack set default

• or set up a new stack with a deployer in it:

zenml deployer register docker --flavor=local
zenml stack register docker_deployment -a default -o default -D docker --set

• deploy a pipeline with the ZenML SDK:

from zenml import pipeline

@step
def my_step(name: str) -> str:
    return f"Hello, {name}!"

@pipeline
def my_pipeline(name: str = "John") -> str:
    return my_step(name=name)

if __name__ == "__main__":
    # Deploy the pipeline `my_pipeline` as a deployment named `my_deployment`
    deployment = my_pipeline.deploy(deployment_name="my_deployment")
    print(f"Deployment URL: {deployment.url}")

• deploy the same pipeline with the CLI:

zenml pipeline deploy --name my_deployment my_module.my_pipeline

• send a request to the deployment with the ZenML CLI:

zenml deployment invoke my_deployment --name="Alice"

• or with curl:

curl -X POST http://localhost:8000/invoke \
  -H "Content-Type: application/json" \
  -d '{"parameters": {"name": "Alice"}}'

• alternatively, set up a snapshot and deploy it instead of a pipeline:

zenml pipeline snapshot create --name my_snapshot my_module.my_pipeline
zenml pipeline snapshot deploy my_snapshot --deployment my_deployment

Pipeline Requirements for Deployment

Not all pipelines are suitable for deployment as HTTP services. To be deployable, pipelines should follow these guidelines:

Parameter Requirements:

• Pipelines should accept explicit parameters with default values
• Parameters must be JSON-serializable types (int, float, str, bool, list, dict, Pydantic models)
• Parameter names should match step input names

Output Requirements:

• Pipelines should return meaningful values for HTTP responses
• Return values must be JSON-serializable
• It's recommended to use type annotations to specify output artifact names

Example Deployable Pipeline:

from typing import Annotated
from zenml import pipeline, step

@step
def process_weather(city: str, temperature: float) -> Annotated[str, "weather_analysis"]:
    return f"The weather in {city} is {temperature} degrees Celsius."

@pipeline
def weather_pipeline(city: str = "Paris", temperature: float = 20.0) -> str:
    """A deployable pipeline that processes weather data."""
    analysis = process_weather(city=city, temperature=temperature)
    return analysis

For more information, see the Deployable Pipeline Requirements section of the tutorial.

Deployment Lifecycle Management

The Deployment object represents a pipeline that has been deployed to a serving environment. The Deployment object is saved in the ZenML database and contains information about the deployment configuration, status, and connection details. Deployments are standalone entities that can be managed independently of the active stack through the Deployer stack components that were originally used to provision them.

Some example of how to manage deployments:

• listing deployments with the CLI:

$ zenml deployment list
┏━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃         NAME         │ PIPELINE                             │ URL                            │ STATUS                   ┃
┠──────────────────────┼──────────────────────────────────────┼────────────────────────────────┼──────────────────────────┨
┃  weather_service     │ weather_pipeline                     │ http://localhost:8001          │ ⚙ RUNNING               ┃
┠──────────────────────┼──────────────────────────────────────┼────────────────────────────────┼──────────────────────────┨
┃  ml_inference_api    │ inference_pipeline                   │ http://k8s-cluster/ml-api      │ ⚙ RUNNING               ┃
┗━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

• listing deployments with the SDK:

from zenml.client import Client

client = Client()
deployments = client.list_deployments()
for deployment in deployments:
    print(f"{deployment.name}: {deployment.status}")

• showing detailed information about a deployment with the CLI:

/* Detailed source-code truncated for AI context efficiency. */

• showing detailed information about a deployment with the SDK:

from zenml.client import Client
deployment = client.get_deployment("my_deployment")
print(deployment)

• deprovision and delete a deployment with the CLI:

$ zenml deployment delete my_deployment

• deprovisioning and deleting a deployment with the SDK:

from zenml.client import Client
client = Client()
client.delete_deployment("my_deployment")

• sending a request to a deployment with the CLI:

$ zenml deployment invoke my_deployment --name="John"

Invoked deployment 'my_deployment' with response:
{
  "success": true,
  "outputs": {
    "output": "Hello, John!"
  },
  "execution_time": 3.2781872749328613,
  "metadata": {
    "deployment_id": "95d60dcf-7c37-4e62-a923-a341601903e5",
    "deployment_name": "my_deployment",
    "snapshot_id": "f3122ed4-aa13-4113-9f60-a80545f56244",
    "snapshot_name": "my_snapshot",
    "pipeline_name": "my_pipeline",
    "run_id": "ea448522-d5bf-411e-971e-d4550fdbe713",
    "run_name": "my_pipeline-2025_09_30-12_52_01_012491",
    "parameters_used": {}
  },
  "error": null
}

• sending a request to a deployment with the SDK:

from zenml.deployers.utils import invoke_deployment

response = invoke_deployment(
    deployment_name_or_id="my_deployment",
    name="John",
)
print(response)

By default, invoking a deployment blocks until the pipeline run finishes and returns its outputs. To submit the run for background execution and get the run ID back immediately instead, pass --no-wait on the CLI or submit=True with the SDK:

$ zenml deployment invoke my_deployment --name="John" --no-wait

response = invoke_deployment(
    deployment_name_or_id="my_deployment",
    name="John",
    submit=True,
)

Specifying deployment resources

If your steps require additional hardware resources, you can specify them on your steps as described here.

Book Component Guide Experiment Trackers README (`docs/book/component-guide/experiment-trackers/README.md`) Copy Section

icon: clipboard Copy Chapter

Experiment Trackers

Experiment trackers let you track your ML experiments by logging extended information about your models, datasets, metrics, and other parameters and allowing you to browse them, visualize them and compare them between runs. In the ZenML world, every pipeline run is considered an experiment, and ZenML facilitates the storage of experiment results through Experiment Tracker stack components. This establishes a clear link between pipeline runs and experiments.

Related concepts:

• the Experiment Tracker is an optional type of Stack Component that needs to be registered as part of your ZenML Stack.
• ZenML already provides versioning and tracking for the pipeline artifacts by storing artifacts in the Artifact Store.

When to use it Copy Section

ZenML already records information about the artifacts circulated through your pipelines by means of the mandatory Artifact Store.

However, these ZenML mechanisms are meant to be used programmatically and can be more difficult to work with without a visual interface.

Experiment Trackers on the other hand are tools designed with usability in mind. They include extensive UIs providing users with an interactive and intuitive interface that allows them to browse and visualize the information logged during the ML pipeline runs.

You should add an Experiment Tracker to your ZenML stack and use it when you want to augment ZenML with the visual features provided by experiment tracking tools.

How they experiment trackers slot into the stack Copy Section

Here is an architecture diagram that shows how experiment trackers fit into the overall story of a remote stack.

Experiment Tracker Flavors

Experiment Trackers are optional stack components provided by integrations:

If you would like to see the available flavors of Experiment Tracker, you can use the command:

zenml experiment-tracker flavor list

How to use it Copy Section

Every Experiment Tracker has different capabilities and uses a different way of logging information from your pipeline steps, but it generally works as follows:

• first, you have to configure and add an Experiment Tracker to your ZenML stack
• next, you have to explicitly enable the Experiment Tracker for individual steps in your pipeline by decorating them with the included decorator
• in your steps, you have to explicitly log information (e.g. models, metrics, data) to the Experiment Tracker same as you would if you were using the tool independently of ZenML
• finally, you can access the Experiment Tracker UI to browse and visualize the information logged during your pipeline runs. You can use the following code snippet to get the URL of the experiment tracker UI for the experiment linked to a certain step of your pipeline run:

{% hint style="info" %}
Stacks can have multiple experiment trackers attached. If you enable tracking with @step(experiment_tracker=True), ZenML activates all attached experiment trackers for that step. If you want to use only one tracker, specify it explicitly by name in the @step decorator.
{% endhint %}

from zenml.client import Client

pipeline_run = Client().get_pipeline_run("<PIPELINE_RUN_NAME>")
step = pipeline_run.steps["<STEP_NAME>"]
experiment_tracker_url = step.run_metadata["experiment_tracker_url"].value

{% hint style="info" %}
Experiment trackers will automatically declare runs as failed if the corresponding ZenML pipeline step fails.
{% endhint %}

Consult the documentation for the particular Experiment Tracker flavor that you plan on using or are using in your stack for detailed information about how to use it in your ZenML pipelines.

Book Component Guide Feature Stores README (`docs/book/component-guide/feature-stores/README.md`) Copy Section

icon: database Copy Chapter

Feature Stores

Feature stores allow data teams to serve data via an offline store and an online low-latency store where data is kept in
sync between the two. It also offers a centralized registry where features (and feature schemas) are stored for use
within a team or wider organization.

As a data scientist working on training your model, your requirements for how you access your batch / 'offline' data
will almost certainly be different from how you access that data as part of a real-time or online inference setting.
Feast solves the problem of developing train-serve skew where those two
sources of data diverge from each other.

Feature stores are a relatively recent addition to commonly-used machine learning stacks.

When to use it Copy Section

The feature store is an optional stack component in the ZenML Stack. The feature store as a technology should be used to
store the features and inject them into the process on the server side. This includes

• Productionalize new features
• Reuse existing features across multiple pipelines and models
• Achieve consistency between training and serving data (Training Serving Skew)
• Provide a central registry of features and feature schemas

List of available feature stores Copy Section

For production use cases, some more flavors can be found in specific integrations modules. In terms of features
stores, ZenML features an integration of feast.

If you would like to see the available flavors for feature stores, you can use the command:

zenml feature-store flavor list

How to use it Copy Section

The available implementation of the feature store is built on top of the feast integration, which means that using a
feature store is no different from what's described on the feast page: How to use it?.

Book Component Guide Image Builders README (`docs/book/component-guide/image-builders/README.md`) Copy Section

icon: box-open Copy Chapter

Image Builders

The image builder is an essential part of most remote MLOps stacks. It is used to build container images such that your machine-learning pipelines and steps can be executed in remote environments.

When to use it Copy Section

The image builder is needed whenever other components of your stack need to build container images. Currently, this is the case for most of ZenML's remote orchestrators , step operators, and some model deployers. These containerize your pipeline code and therefore require an image builder to build Docker images.

Image Builder Flavors Copy Section

Out of the box, ZenML comes with a local image builder that builds OCI images on your client machine using Docker or Podman, depending on your global container engine setting. Additional image builders are provided by integrations:

If you would like to see the available flavors of image builders, you can use the command:

zenml image-builder flavor list

How to use it Copy Section

You don't need to directly interact with any image builder in your code. As long as the image builder that you want to use is part of your active ZenML stack, it will be used automatically by any component that needs to build container images.

Book Component Guide Log Stores README (`docs/book/component-guide/log-stores/README.md`) Copy Section

icon: file-lines Copy Chapter

Log Stores

The log store is a stack component responsible for collecting, storing, and retrieving logs generated during pipeline and step execution. It captures everything from standard logging output to print statements and any messages written to stdout/stderr, making it easy to debug and monitor your ML workflows.

How it works Copy Section

ZenML's log capture system is designed to be comprehensive and non-intrusive. Here's what happens under the hood:

1. stdout/stderr wrapping: ZenML wraps the standard output and error streams to capture all printed messages and any output directed to these streams.

2. Root logger handler: A custom handler is added to Python's root logger to capture all log messages with proper metadata from loggers that propagate to the root.

3. Log routing: All captured messages are routed through a LoggingContext to the active log store in your stack.

This approach ensures that you don't miss any output from your pipeline steps, including:

• Standard Python logging messages
• print() statements
• Output from third-party libraries
• Messages from subprocesses that write to stdout/stderr

When to use it Copy Section

The Log Store is automatically used in every ZenML stack. If you don't explicitly configure a log store, ZenML will use an Artifact Log Store by default, which stores logs in your artifact store.

You should consider configuring a dedicated log store when:

• You want to use a centralized logging backend like Datadog, Jaeger, Grafana Tempo, Honeycomb, Lightstep or Dash0 for log aggregation and analysis
• You need advanced log querying capabilities beyond what file-based storage provides
• You're running pipelines at scale and need better log management
• You want to integrate with your organization's existing observability infrastructure

How to use it Copy Section

By default, if no log store is explicitly configured in your stack, ZenML automatically creates an Artifact Log Store that uses your artifact store for log storage. This means logging works out of the box without any additional configuration.

To use a different log store, you need to register it and add it to your stack:

# Register a log store (example with Datadog)
zenml log-store register <LOG_STORE_NAME> \
    --flavor=datadog \
    --api_key=<DATADOG_API_KEY> \
    --application_key=<DATADOG_APPLICATION_KEY>

# Add it to your stack
zenml stack register <STACK_NAME> -a <ARTIFACT_STORE> -o <ORCHESTRATOR> --log_store <LOG_STORE_NAME> --set

Once configured, logs are automatically captured during pipeline execution.

Viewing Logs Copy Section

You can view step logs in the ZenML dashboard or in your logging provider's UI. To retrieve them programmatically, use the Python SDK or REST API.

Python SDK

Use the log store that captured the logs:

from zenml.client import Client
from zenml.utils.logging_utils import search_logs_by_source

client = Client()

run = client.get_pipeline_run("<RUN_NAME_OR_ID>")
step_run = run.steps["<STEP_NAME>"]
logs = search_logs_by_source(step_run.log_collection or [], "step")
if logs is None:
    raise ValueError("This step has no execution log stream.")

log_store = client.active_stack.log_store
page = log_store.fetch(logs_model=logs, limit=1000)

for entry in page.items:
    print(f"[{entry.level}] {entry.message}")

fetch() returns a page with entries in items, ordered from oldest to newest. Set start="oldest" or start="newest" to choose the first page, or omit it to use the backend's default: newest for Datadog, oldest for the artifact log store.

For backends with pagination, pass the returned before cursor to read older entries or after to read newer ones. Only directions supported by the backend's native tokens are available; None means there is no continuation in that direction.

For example, with Datadog, filter the first request and use its cursor to continue:

from zenml.models import LogsEntriesFilter

page = log_store.fetch(
    logs_model=logs,
    start="newest",
    filter_=LogsEntriesFilter(level="ERROR", search="ValueError"),
)
while True:
    for entry in page.items:
        print(entry.message)
    if page.before is None:
        break
    page = log_store.fetch(logs_model=logs, before=page.before)

Datadog cursors retain the filters, page size, and fixed time bounds. Search follows the backend's matching rules, which may differ from a literal substring match.

The artifact log store returns one batch for client-side filtering and pagination. It does not support these filters or start="newest".

Each entry has a UUID id. IDs remain stable for Datadog events and structured artifact logs. Deduplicate entries within a stream by (id, chunk_index) to preserve chunks of the same message. Shared log types, including LogEntry, are available from zenml.models.

REST API

GET /api/v1/logs/{logs_id}/entries accepts start, limit, before, after, search, level, since, and until as query parameters. For example:

GET /api/v1/logs/<LOGS_ID>/entries?start=newest&limit=50
GET /api/v1/logs/<LOGS_ID>/entries?before=<CURSOR>

Invalid query values return 422; unsupported parameters and cursors rejected by ZenML return 400. Backends without log retrieval return 501. Shared log store errors return 429 for throttling, 503 for unavailability, or 502 for other backend errors. When provided, Retry-After specifies the delay in seconds before retrying the original request. Direct SDK calls raise the corresponding LogStoreError subclass.

Runner logs return one batch through the workload manager. Apply filtering and pagination in the client; unsupported runner filters or cursors return 400.

The existing run and step log endpoints continue to return a single list of entries. Use the dedicated entries endpoint for pagination.

Log Store Flavors Copy Section

ZenML provides several log store flavors out of the box:

If you would like to see the available flavors of log stores, you can use the command:

zenml log-store flavor list

{% hint style="info" %}
If you're interested in understanding the base abstraction and how log stores work internally, check out the Develop a Custom Log Store page for a detailed explanation of the architecture.
{% endhint %}

Book Component Guide Model Deployers README (`docs/book/component-guide/model-deployers/README.md`) Copy Section

icon: rocket Copy Chapter

Model Deployers

{% hint style="warning" %}
DEPRECATION NOTICE

The Model Deployer stack component is deprecated in favor of the more flexible Deployer component and Pipeline Deployments.

The Model Deployer abstraction focused exclusively on single-model serving, but modern ML workflows often require multi-step pipelines with preprocessing, tool integration, and custom business logic. The new Pipeline Deployment paradigm provides:

• Unified approach: Deploy any pipeline—classical ML inference, agentic workflows, or hybrid systems—as a long-running HTTP service
• Greater flexibility: Customize your deployment with full FastAPI control, add middleware, custom routes, and even frontend interfaces
• Simpler mental model: One primitive for all deployment scenarios instead of separate abstractions for models vs. pipelines
• Better extensibility: Deploy to Docker, AWS App Runner, GCP Cloud Run, and other platforms with consistent patterns

Migration Path: Instead of using Model Deployer-specific steps, wrap your model inference logic in a regular ZenML pipeline and deploy it using zenml pipeline deploy. See the Pipeline Deployment guide for examples of deploying ML models as HTTP services.

While Model Deployer integrations remain available for backward compatibility, we strongly recommend migrating to Pipeline Deployments for new projects.
{% endhint %}

Model Deployment is the process of making a machine learning model available to make predictions and decisions on
real-world data. Getting predictions from trained models can be done in different ways depending on the use case, a
batch prediction is used to generate predictions for a large amount of data at once, while a real-time prediction is
used to generate predictions for a single data point at a time.

Model deployers are stack components responsible for serving models on a real-time or batch basis.

Online serving is the process of hosting and loading machine-learning models as part of a managed web service and
providing access to the models through an API endpoint like HTTP or GRPC. Once deployed, model inference can be
triggered at any time, and you can send inference requests to the model through the web service's API and receive fast,
low-latency responses.

Batch inference or offline inference is the process of making a machine learning model make predictions on a batch of
observations. This is useful for generating predictions for a large amount of data at once. The predictions are usually
stored as files or in a database for end users or business applications.

When to use it? Copy Section

The model deployers are optional components in the ZenML stack. They are used to deploy machine learning models to a
target environment, either a development (local) or a production (Kubernetes or cloud) environment. The model deployers are mainly used to deploy models for real-time inference use cases. With the model deployers and other stack components, you can build pipelines that are continuously trained and deployed to production.

How model deployers slot into the stack Copy Section

Here is an architecture diagram that shows how model deployers fit into the overall story of a remote stack.

Model Deployers Flavors

ZenML comes with a local MLflow model deployer which is a simple model deployer that deploys models to a local MLflow
server. Additional model deployers that can be used to deploy models on production environments are provided by
integrations:

{% hint style="info" %}
Every model deployer may have different attributes that must be configured in order to interact with the model serving
tool, framework, or platform (e.g. hostnames, URLs, references to credentials, and other client-related configuration
parameters). The following example shows the configuration of the MLflow and Seldon Core model deployers:

# Configure MLflow model deployer
zenml model-deployer register mlflow --flavor=mlflow

# Configure Seldon Core model deployer
zenml model-deployer register seldon --flavor=seldon \
--kubernetes_context=zenml-eks --kubernetes_namespace=zenml-workloads \
--base_url=http://abb84c444c7804aa98fc8c097896479d-377673393.us-east-1.elb.amazonaws.com
...

{% endhint %}

The role that a model deployer plays in a ZenML Stack

• Seamless Model Deployment: Facilitates the deployment of machine learning models to various serving environments, such as local servers, Kubernetes clusters, or cloud platforms, ensuring that models can be deployed and managed efficiently in accordance with the specific requirements of the serving infrastructure by holds all the stack-related configuration attributes required to interact with the remote model serving tool, service, or platform (e.g. hostnames, URLs, references to credentials, and other client-related configuration parameters). The following are examples of configuring the MLflow and Seldon Core Model Deployers and registering them as a Stack component:

  bash Copy

  zenml integration install mlflow
  zenml model-deployer register mlflow --flavor=mlflow
  zenml stack register local_with_mlflow -m default -a default -o default -d mlflow --set

  bash Copy

  zenml integration install seldon
  zenml model-deployer register seldon --flavor=seldon \
  --kubernetes_context=zenml-eks --kubernetes_namespace=zenml-workloads \
  --base_url=http://abb84c444c7804aa98fc8c097896479d-377673393.us-east-1.elb.amazonaws.com
  ...
  zenml stack register seldon_stack -m default -a aws -o default -d seldon

• Lifecycle Management: Provides mechanisms for comprehensive lifecycle management of model servers, including the ability to start, stop, and delete model servers, as well as to update existing servers with new model versions, thereby optimizing resource utilization and facilitating continuous delivery of model updates. Some core methods that can be used to interact with the remote model server include:

  • deploy_model - Deploys a model to the serving environment and returns a Service object that represents the deployed model server.
  • find_model_server - Finds and returns a list of Service objects that
    represent model servers that have been deployed to the serving environment,
    the services are stored in the DB and can be used as a reference to know what and where the model is deployed.
  • stop_model_server - Stops a model server that is currently running in the serving environment.
  • start_model_server - Starts a model server that has been stopped in the serving environment.
  • delete_model_server - Deletes a model server from the serving environment and from the DB.

{% hint style="info" %}
ZenML uses the Service object to represent a model server that has been deployed to a serving environment. The Service object is saved in the DB and can be used as a reference to know what and where the model is deployed. The Service object consists of 2 main attributes, the config and the status. The config attribute holds all the deployment configuration attributes required to create a new deployment, while the status attribute holds the operational status of the deployment, such as the last error message, the prediction URL, and the deployment status.
{% endhint %}

from zenml.integrations.huggingface.model_deployers import HuggingFaceModelDeployer

model_deployer = HuggingFaceModelDeployer.get_active_model_deployer()
services = model_deployer.find_model_server(
    pipeline_name="LLM_pipeline",
    pipeline_step_name="huggingface_model_deployer_step",
    model_name="LLAMA-7B",
)
if services:
    if services[0].is_running:
        print(
            f"Model server {services[0].config['model_name']} is running at {services[0].status['prediction_url']}"
           )
     else:
         print(f"Model server {services[0].config['model_name']} is not running")
         model_deployer.start_model_server(services[0])
 else:
     print("No model server found")
     service = model_deployer.deploy_model(
         pipeline_name="LLM_pipeline",
         pipeline_step_name="huggingface_model_deployer_step",
         model_name="LLAMA-7B",
         model_uri="s3://zenprojects/huggingface_model_deployer_step/output/884/huggingface",
         revision="main",
         task="text-classification",
         region="us-east-1",
         vendor="aws",
         token="huggingface_token",
         namespace="zenml-workloads",
         endpoint_type="public",
     )
     print(f"Model server {service.config['model_name']} is deployed at {service.status['prediction_url']}")

How to Interact with a model deployer after deployment?

When a Model Deployer is part of the active ZenML Stack, it is also possible to interact with it from the CLI to list,
start, stop, or delete the model servers that is managed:

/* Detailed source-code truncated for AI context efficiency. */

In Python, you can alternatively discover the prediction URL of a deployed model by inspecting the metadata of the step
that deployed the model:

from zenml.client import Client

pipeline_run = Client().get_pipeline_run("<PIPELINE_RUN_NAME>")
deployer_step = pipeline_run.steps["<NAME_OF_MODEL_DEPLOYER_STEP>"]
deployed_model_url = deployer_step.run_metadata["deployed_model_url"].value

The ZenML integrations that provide Model Deployer stack components also include standard pipeline steps that can
directly be inserted into any pipeline to achieve a continuous model deployment workflow. These steps take care of all
the aspects of continuously deploying models to an external server and saving the Service configuration into the
Artifact Store, where they can be loaded at a later time and re-create the initial conditions used to serve a particular
model.

Book Component Guide Model Registries README (`docs/book/component-guide/model-registries/README.md`) Copy Section

icon: table-rows Copy Chapter

Model Registries

Model registries are centralized storage solutions for managing and tracking machine learning models across various stages of development and deployment. They help track the different versions and configurations of each model and enable reproducibility. By storing metadata such as version, configuration, and metrics, model registries help streamline the management of trained models. In ZenML, model registries are Stack Components that allow for the easy retrieval, loading, and deployment of trained models. They also provide information on the pipeline in which the model was trained and how to reproduce it.

Model Registry Concepts and Terminology Copy Section

ZenML provides a unified abstraction for model registries through which it is possible to handle and manage the concepts of model groups, versions, and stages in a consistent manner regardless of the underlying registry tool or platform being used. The following concepts are useful to be aware of for this abstraction:

• RegisteredModel: A logical grouping of models that can be used to track different versions of a model. It holds information about the model, such as its name, description, and tags, and can be created by the user or automatically created by the model registry when a new model is logged.
• RegistryModelVersion: A specific version of a model identified by a unique version number or string. It holds information about the model, such as its name, description, tags, and metrics, and a reference to the model artifact logged to the model registry. In ZenML, it also holds a reference to the pipeline name, pipeline run ID, and step name. Each model version is associated with a model registration.
• ModelVersionStage: A model version stage is a state in that a model version can be. It can be one of the following: None, Staging, Production, Archived. The model version stage is used to track the lifecycle of a model version. For example, a model version can be in the Staging stage while it is being tested and then moved to the Production stage once it is ready for deployment.

When to use it Copy Section

ZenML provides a built-in mechanism for storing and versioning pipeline artifacts through its mandatory Artifact Store. While this is a powerful way to manage artifacts programmatically, it can be challenging to use without a visual interface.

Model registries, on the other hand, offer a visual way to manage and track model metadata, particularly when using a remote orchestrator. They make it easy to retrieve and load models from storage, thanks to built-in integrations. A model registry is an excellent choice for interacting with all the models in your pipeline and managing their state in a centralized way.

Using a model registry in your stack is particularly useful if you want to interact with all the logged models in your pipeline, or if you need to manage the state of your models in a centralized way and make it easy to retrieve, load, and deploy these models.

How model registries fit into the ZenML stack Copy Section

Here is an architecture diagram that shows how a model registry fits into the overall story of a remote stack.

Model Registry Flavors

Model Registries are optional stack components provided by integrations:

If you would like to see the available flavors of Model Registry, you can use the command:

zenml model-registry flavor list

How to use it Copy Section

Model registries are an optional component in the ZenML stack that is tied to the experiment tracker. This means that a model registry can only be used if you are also using an experiment tracker. If you're not using an experiment tracker, you can still store your models in ZenML, but you will need to manually retrieve model artifacts from the artifact store. More information on this can be found in the documentation on the fetching runs.

To use model registries, you first need to register a model registry in your stack with the same flavor as your experiment tracker. Then, you can register your trained model in the model registry using one of three methods:

• (1) using the built-in step in the pipeline.
• (2) using the ZenML CLI to register the model from the command line.
• (3) registering the model from the model registry UI. Finally, you can use the model registry to retrieve and load your models for deployment or further experimentation.

Book Component Guide Orchestrators README (`docs/book/component-guide/orchestrators/README.md`) Copy Section

icon: train-track Copy Chapter

Orchestrators

The orchestrator is an essential component in any MLOps stack as it is responsible for running your machine learning pipelines. To do so, the orchestrator provides an environment that is set up to execute the steps of your pipeline. It also makes sure that the steps of your pipeline only get executed once all their inputs (which are outputs of previous steps of your pipeline) are available.

{% hint style="info" %}
Many of ZenML's remote orchestrators build Docker images in order to transport and execute your pipeline code. If you want to learn more about how Docker images are built by ZenML, check out this guide.
{% endhint %}

When to use it Copy Section

The orchestrator is a mandatory component in the ZenML stack. It is used to store all artifacts produced by pipeline runs, and you are required to configure it in all of your stacks.

Orchestrator Flavors Copy Section

Out of the box, ZenML comes with a local orchestrator already part of the default stack that runs pipelines locally. Additional orchestrators are provided by integrations:

If you would like to see the available flavors of orchestrators, you can use the command:

zenml orchestrator flavor list

How to use it Copy Section

You don't need to directly interact with any ZenML orchestrator in your code. As long as the orchestrator that you want to use is part of your active ZenML stack, using the orchestrator is as simple as executing a Python file that runs a ZenML pipeline:

python file_that_runs_a_zenml_pipeline.py

Inspecting Runs in the Orchestrator UI

If your orchestrator comes with a separate user interface (for example Kubeflow, Airflow, Vertex), you can get the URL to the orchestrator UI of a specific pipeline run using the following code snippet:

from zenml.client import Client

pipeline_run = Client().get_pipeline_run("<PIPELINE_RUN_NAME>")
orchestrator_url = pipeline_run.run_metadata["orchestrator_url"].value

Specifying per-step resources

If your steps require the orchestrator to execute them on specific hardware, you can specify them on your steps as described here.

If your orchestrator of choice or the underlying hardware doesn't support this, you can also take a look at step operators.

Book Component Guide Sandboxes README (`docs/book/component-guide/sandboxes/README.md`) Copy Section

icon: box-archive Copy Chapter

Sandboxes

A Sandbox is a stack component that provides an isolated environment (container, microVM, or pod) in which a ZenML step can execute code. It's primarily designed for AI-agent workloads: an agent running inside a step uses the active stack's Sandbox to execute generated code as a tool, possibly across many turns of an agent loop.

A Sandbox is fundamentally different from a Step Operator:

• A Step Operator runs the step itself on a remote backend.
• A Sandbox is a tool the step consumes. The step still runs wherever the orchestrator placed it, and you can run code in the sandbox from within your step code.

The two compose: a step running on, say, the SageMaker step operator can still grab the active stack's Sandbox and use it for code execution.

Concepts Copy Chapter

How to use it from a step Copy Chapter

A Sandbox is reached for from inside a step via the active stack:

from zenml import step
from zenml.client import Client


@step
def agent_step(prompt: str) -> str:
    sandbox = Client().active_stack.sandbox
    if sandbox is None:
        raise RuntimeError("No sandbox configured in the active stack.")

    with sandbox.create_session() as session:
        process = session.exec(["python", "-c", "print(1 + 1)"])
        output = "".join(process.stdout())
        process.wait()
        return output

If your stack contains more than one Sandbox component, address them by name through Client().active_stack.sandboxes (a Dict[str, BaseSandbox]). The .sandbox accessor returns the default (first attached).

Streaming output Copy Section

SandboxProcess.stdout() and stderr() yield strings one line at a time as the underlying command produces them — the standard pattern for streaming an agent's tool output back to a UI or to the step logs.

Snapshots, restore, and attach Copy Section

A Session can optionally be snapshotted (provider-dependent — not all backends support full state capture). The returned SandboxSnapshot is a Pydantic model that's safe to persist as a ZenML artifact:

snap = session.create_snapshot()
zenml.save_artifact(snap, name="agent_checkpoint")
# ... later, possibly in a different pipeline run:
snap = zenml.load_artifact("agent_checkpoint")
session = stack.sandbox.restore(snap)        # new session, state restored

restore always returns a new Session (fresh id); the original Session is unaffected.

For the common subagent / cross-pipeline reuse case where the original Session is still alive, use attach() instead — no snapshot required, the parent just persists the session id as an artifact and the child reconnects to it:

# Parent step
session_id = session.id
zenml.save_artifact(session_id, name="agent_session")

# Child step (possibly in a child pipeline run)
session = stack.sandbox.attach(zenml.load_artifact("agent_session"))

Closing vs destroying Copy Section

• session.close() releases the local handle. The sandbox keeps running on the provider until its TTL expires. Use this when a subagent or follow-up step might still want to attach to the same Session.
• session.destroy() terminates the sandbox on the provider. After this, the id is invalid and attach() will fail.

ZenML does not auto-close Sessions on step exit. Either use a with block (__exit__ calls close()) or call close() / destroy() explicitly.

Configuration Copy Chapter

Per-step settings Copy Section

Step writers can configure sandbox behavior on individual @step invocations via BaseSandboxSettings (or a flavor-specific subclass):

Flavors that boot a container or microVM add their own settings (for example image selection and a provider TTL) on a flavor-specific BaseSandboxSettings subclass.

Sandbox logs Copy Section

Sandbox stdout/stderr automatically lands on the active step as a dedicated sandbox:<session_id> log source. Each session.exec(...) writes a $ <command> marker, then the process output (stdout at INFO, stderr at ERROR), then a trailing OK/FAIL exit code <code> in <seconds>s marker — reads like a shell session. ZenML's own step-level Python logger calls stay in the regular step source: the sandbox source is dedicated to actual sandbox-execution events, no false attribution.

Multi-session steps don't clobber: each session's metadata (flavor, dashboard URL when the flavor exposes one) is keyed by session id (sandbox..flavor, sandbox..dashboard_url).

Security considerations Copy Chapter

Every value you put in a Session's environment (sandbox_environment) is readable by code running inside the Session. If you run LLM-generated code in the Sandbox and care about credential isolation, treat the Session environment as visible to the agent.

Available flavors Copy Chapter

• Local — subprocess-based; no isolation; built-in. Intended for examples, unit tests, and development against the abstraction.
• Kubernetes — pod-backed sessions executed through Kubernetes exec.

Develop a custom Sandbox Copy Chapter

To build a flavor for a new backend, subclass BaseSandbox, SandboxSession, SandboxProcess, and optionally SandboxSnapshot. The minimal required surface is:

• BaseSandbox.create_session(settings=None) -> SandboxSession
• SandboxSession.exec(...) -> SandboxProcess
• SandboxSession.close()
• SandboxProcess.stdout() / stderr() / wait() / kill() / exit_code

Everything else (attach, create_snapshot, restore, aexec, upload_file, download_file, destroy) is opt-in — the base raises NotImplementedError and you override only what your backend supports. Register your flavor via the standard Integration.flavors() hook.

Book Component Guide Step Operators README (`docs/book/component-guide/step-operators/README.md`) Copy Section

icon: arrow-progress Copy Chapter

Step Operators

The step operator enables the execution of individual pipeline steps in specialized runtime environments that are optimized for certain workloads. These specialized environments can give your steps access to resources like GPUs or distributed processing frameworks like Spark.

{% hint style="info" %}
Comparison to orchestrators: The orchestrator is a mandatory stack component that is responsible for executing all steps of a pipeline in the correct order and providing additional features such as scheduling pipeline runs. The step operator on the other hand is used to only execute individual steps of the pipeline in a separate environment in case the environment provided by the orchestrator is not feasible.
{% endhint %}

When to use it Copy Section

A step operator should be used if one or more steps of a pipeline require resources that are not available in the runtime environments provided by the orchestrator. An example would be a step that trains a computer vision model and requires a GPU to run in a reasonable time, combined with a Kubeflow orchestrator running on a Kubernetes cluster that does not contain any GPU nodes. In that case, it makes sense to include a step operator like SageMaker, Vertex, or AzureML to execute the training step with a GPU.

Step Operator Flavors Copy Section

Step operators to execute steps on one of the big cloud providers are provided by the following ZenML integrations:

If you would like to see the available flavors of step operators, you can use the command:

zenml step-operator flavor list

How to use it Copy Section

You don't need to directly interact with any ZenML step operator in your code. As long as the step operator that you want to use is part of your active ZenML stack, you can simply specify it in the @step decorator of your step.

{% hint style="info" %}
Stacks can have multiple step operators attached. If you use @step(step_operator=True), ZenML uses the default attached step operator, which is the first step operator in the stack. If you want to use a non-default step operator, specify it explicitly by name in the @step decorator.
{% endhint %}

from zenml import step


@step(step_operator=True)
def my_step(...) -> ...:
    ...

Specifying per-step resources

If your steps require additional hardware resources, you can specify them on your steps as described here.

Enabling CUDA for GPU-backed hardware

Note that if you wish to use step operators to run steps on a GPU, you will need to follow the instructions on this page to ensure that it works. It requires adding some extra settings customization and is essential to enable CUDA for the GPU to give its full acceleration.

Book Getting Started Deploying Zenml README (`docs/book/getting-started/deploying-zenml/README.md`) Copy Section

icon: rocket-launch Copy Chapter

Deploy

Moving your ZenML Server to a production environment offers several benefits over staying local:

1. Scalability: Production environments are designed to handle large-scale workloads, allowing your models to process more data and deliver faster results.
2. Reliability: Production-grade infrastructure ensures high availability and fault tolerance, minimizing downtime and ensuring consistent performance.
3. Collaboration: A shared production environment enables seamless collaboration between team members, making it easier to iterate on models and share insights.

Despite these advantages, transitioning to production can be challenging due to the complexities involved in setting up the needed infrastructure.

Components Copy Chapter

A ZenML deployment consists of multiple infrastructure components:

• FastAPI server backed with a SQLite or MySQL database
• Python Client
• An open-source companion ReactJS dashboard
• (Optional) ZenML Pro API + Database + ZenML Pro dashboard

You can read more in-depth about the system architecture of ZenML here.
This documentation page will focus on the components required to deploy ZenML OSS.

Details on the ZenML Python Client

The ZenML client is a Python package that you can install on your machine. It is used to interact with the ZenML server. You can install it using the pip command as outlined here.

This Python package gives you the zenml command-line interface which you can use to interact with the ZenML server for common tasks like managing stacks, setting up secrets, and so on. It also gives you the general framework that lets you author and deploy pipelines and so forth.

If you want to have more fine-grained control and access to the metadata that ZenML manages, you can use the Python SDK to access the API. This allows you to create your own custom automations and scripts and is the most common way teams access the metadata stored in the ZenML server. The full documentation for the Python SDK can be found here. The full HTTP API documentation can also be found by adding the/doc suffix to the URL when accessing your deployed ZenML server.

Deployment scenarios Copy Section

When you first get started with ZenML, you have the following architecture on your machine.

The SQLite database that you can see in this diagram is used to store information about pipelines, pipeline runs, stacks, and other configurations. This default setup allows you to get started and try out the core features, but you won't be able to use cloud-based components like serverless orchestrators and so on.

Users can run the zenml login --local command to spin up a local ZenML OSS server to serve the dashboard. For the local OSS server option, the zenml login --local command implicitly connects the client to the server. The diagram for this looks as follows:

In order to move into production, the ZenML server needs to be deployed somewhere centrally so that the different cloud stack components can read from and write to the server. Additionally, this also allows all your team members to connect to it and share stacks and pipelines.

You connect to your deployed ZenML server using the zenml login command, and then you have the full benefits and power of ZenML. You can use all the cloud-based components, your metadata will be stored and synchronized across all the users of the server, and you can leverage features like centralized logs storage and pipeline artifact visualization.

How to deploy ZenML Copy Chapter

Deploying the ZenML Server is a crucial step towards transitioning to a production-grade environment for your machine learning projects. By setting up a deployed ZenML Server instance, you gain access to powerful features, allowing you to use stacks with remote components, centrally track progress, collaborate effectively, and achieve reproducible results.

Currently, there are two main options to access a deployed ZenML server:

1. Managed deployment: With ZenML Pro offering you can utilize a control plane to create ZenML servers, also known as workspaces. These workspaces are managed and maintained by ZenML's dedicated team, alleviating the burden of server management from your end. Importantly, your data remains securely within your stack, and ZenML's role is primarily to handle tracking of metadata and server maintenance.
2. Self-hosted Deployment: Alternatively, you have the ability to deploy ZenML on your own self-hosted environment. This can be achieved through various methods, including using Docker, Helm, or HuggingFace Spaces. We also offer our Pro version for self-hosted deployments, so you can use our full paid feature set while staying fully in control with an air-gapped solution on your infrastructure.

Both options offer distinct advantages, allowing you to choose the deployment approach that best aligns with your organization's needs and infrastructure preferences. Whichever path you select, ZenML facilitates a seamless and efficient way to take advantage of the ZenML Server and enhance your machine learning workflows for production-level success.

Options for deploying ZenML Copy Section

Documentation for the various deployment strategies can be found in the following pages below (in our 'how-to' guides):

Book Getting Started Zenml Pro README (`docs/book/getting-started/zenml-pro/README.md`) Copy Section

cover: .gitbook/assets/procover.png
coverY: 0
layout:
width: default
cover:
visible: true
size: hero
title:
visible: true
visible: true
tableOfContents:
visible: true
outline:
visible: true
pagination:
visible: true
metadata:
visible: true

Introduction

What is ZenML Pro? Copy Chapter

The Pro version of ZenML extends the Open Source product with advanced features for enterprise-grade MLOps. It provides multi-user collaboration, role-based access control, flexible deployment options, and professional support to help teams scale their ML operations.

{% hint style="info" %}
To get access to ZenML Pro, book a call.
{% endhint %}

ZenML OSS vs Pro Feature Comparison Copy Chapter

Security & Compliance Copy Chapter

All ZenML Pro deployments include:

• ✅ SOC 2 Type II certification
• ✅ ISO 27001 certification
• ✅ Vulnerability Assessment Reports available on request
• ✅ Software Bill of Materials (SBOM) available on request

Documentation Guide Copy Chapter

This documentation is organized to help you understand, deploy, and manage ZenML Pro:

Pro Feature Details Copy Chapter

Book User Guide Llmops Guide README (`docs/book/user-guide/llmops-guide/README.md`) Copy Section

icon: robot Copy Chapter

LLMOps guide

Welcome to the ZenML LLMOps Guide, where we dive into the exciting world of Large Language Models (LLMs) and how to integrate them seamlessly into your MLOps pipelines using ZenML. This guide is designed for ML practitioners and MLOps engineers looking to harness the potential of LLMs while maintaining the robustness and scalability of their workflows.

In this guide, we'll explore various aspects of working with LLMs in ZenML, including:

• RAG with ZenML
  • RAG in 85 lines of code
  • Understanding Retrieval-Augmented Generation (RAG)
  • Data ingestion and preprocessing
  • Embeddings generation
  • Storing embeddings in a vector database
  • Basic RAG inference pipeline
• Evaluation and metrics
  • Evaluation in 65 lines of code
  • Retrieval evaluation
  • Generation evaluation
  • Evaluation in practice
• Reranking for better retrieval
  • Understanding reranking
  • Implementing reranking in ZenML
  • Evaluating reranking performance
• Improve retrieval by finetuning embeddings
  • Synthetic data generation
  • Finetuning embeddings with Sentence Transformers
  • Evaluating finetuned embeddings
• Finetuning LLMs with ZenML
  • Finetuning in 100 lines of code
  • Why and when to finetune LLMs
  • Starter choices with finetuning
  • Finetuning with 🤗 Accelerate
  • Evaluation for finetuning
  • Deploying finetuned models
  • Next steps

To follow along with the examples and tutorials in this guide, ensure you have a Python environment set up with ZenML installed. Familiarity with the concepts covered in the Starter Guide and Production Guide is recommended.

We'll showcase a specific application over the course of this LLM guide, showing how you can work from a simple RAG pipeline to a more complex setup that involves finetuning embeddings, reranking retrieved documents, and even finetuning the LLM itself. We'll do this all for a use case relevant to ZenML: a question answering system that can provide answers to common questions about ZenML. This will help you understand how to apply the concepts covered in this guide to your own projects.

By the end of this guide, you'll have a solid understanding of how to leverage LLMs in your MLOps workflows using ZenML, enabling you to build powerful, scalable, and maintainable LLM-powered applications. First up, let's take a look at a super simple implementation of the RAG paradigm to get started.

Book User Guide Production Guide README (`docs/book/user-guide/production-guide/README.md`) Copy Section

icon: tree Copy Chapter

Production guide

The ZenML production guide builds upon the Starter guide and is the next step in the MLOps Engineer journey with ZenML. If you're an ML practitioner hoping to implement a proof of concept within your workplace to showcase the importance of MLOps, this is the place for you.

This guide will focus on shifting gears from running pipelines locally on your machine, to running them in production in the cloud. We'll cover:

• Deploying ZenML
• Understanding stacks
• Connecting remote storage
• Orchestrating on the cloud
• Configuring the pipeline to scale compute
• Configure a code repository

Like in the starter guide, make sure you have a Python environment ready and virtualenv installed to follow along with ease. As now we are dealing with cloud infrastructure, you'll also want to select one of the major cloud providers (AWS, GCP, Azure), and make sure the respective CLIs are installed and authorized.

By the end, you will have completed an end-to-end MLOps project that you can use as inspiration for your own work. Let's get right into it!

{% hint style="info" %}
Throughout this guide, we will be referencing internal ZenML functions and classes, which are more easily discoverable in the SDK Docs. Consult the SDK docs if you're ever stuck!
{% endhint %}

Book User Guide Starter Guide README (`docs/book/user-guide/starter-guide/README.md`) Copy Section

icon: seedling Copy Chapter

Starter guide

Welcome to the ZenML Starter Guide! If you're an MLOps engineer aiming to build robust ML platforms, or a data scientist interested in leveraging the power of MLOps, this is the perfect place to begin. Our guide is designed to provide you with the foundational knowledge of the ZenML framework and equip you with the initial tools to manage the complexity of machine learning operations.

Throughout this guide, we'll cover essential topics including:

• Creating your first ML pipeline
• Understanding caching between pipeline steps
• Managing data and data versioning
• Tracking your machine learning models

Before jumping in, make sure you have a Python environment ready and virtualenv installed to follow along with ease. By the end, you will have completed a starter project, marking the beginning of your journey into MLOps with ZenML.

Let this guide be not only your introduction to ZenML but also a foundational asset in your MLOps toolkit. Prepare your development environment, and let's get started!

{% hint style="info" %}
Throughout this guide, we will be referencing internal ZenML functions and classes, which are more easily discoverable in the SDK Docs. Consult the SDK docs if you're ever stuck!
{% endhint %}

Book Api Docs Oss Api Oss Api README (`docs/book/api-docs/oss-api/oss-api/README.md`) Copy Section

icon: github-alt Copy Chapter

OSS API

Book Api Docs Oss Api Oss Api Getting Started (`docs/book/api-docs/oss-api/oss-api/getting-started.md`) Copy Section

icon: person-from-portal Copy Chapter

Getting Started

The ZenML OSS server is a FastAPI application, therefore the OpenAPI-compliant docs are available at /docs or /redoc of your ZenML server:

{% hint style="info" %}
In the local case (i.e. using zenml login --local, the docs are available on http://127.0.0.1:8237/docs)
{% endhint %}

{% hint style="info" %}
Difference between OpenAPI docs and ReDoc

The OpenAPI docs (/docs) provide an interactive interface where you can try out the API endpoints directly from the browser. It is useful for testing and exploring the API functionalities.

ReDoc (/redoc), on the other hand, offers a more static and visually appealing documentation. It is designed for better readability and is ideal for understanding the API structure and reference.
{% endhint %}

Accessing the ZenML OSS API Copy Chapter

For OSS users: The server_url is the root URL of your ZenML server deployment.

If you are using the ZenML OSS server API using the methods displayed above, it is enough to be logged in to your ZenML account in the same browser session. However, in order to do this programmatically, you can use one of the methods documented in the following sections.

{% hint style="info" %}
Choosing a method:

• Humans at the CLI: use interactive login.
• CI/CD and automation: use service accounts + API keys.
  {% endhint %}

Using a service account and an API key Copy Section

You can use a service account's API key to authenticate to the ZenML server's REST API programmatically. This is particularly useful when you need a long-term, secure way to make authenticated HTTP requests to the ZenML API endpoints.

Start by creating a service account and an API key, e.g.:

```shell
zenml service-account create myserviceaccount
```

Then, there are two methods to authenticate with the API using the API key - one is simpler but less secure, the other is secure and recommended but more complex:

{% tabs %}
{% tab title="Direct API key authentication" %}

{% hint style="warning" %}
This approach, albeit simple, is not recommended because the long-lived API key is exposed with every API request, which makes it easier to be compromised. Use it only in low-risk circumstances.
{% endhint %}

Use the API key directly to authenticate your API requests by including it in the Authorization header. For example, you can use the following command to check your current user:

• using curl:

  bash Copy

  curl -H "Authorization: Bearer YOUR_API_KEY" https://your-zenml-server/api/v1/current-user

• using wget:

  bash Copy

  wget -qO- --header="Authorization: Bearer YOUR_API_KEY" https://your-zenml-server/api/v1/current-user

• using python:

  python Copy

  import requests
  
  response = requests.get(
      "https://your-zenml-server/api/v1/current-user",
      headers={"Authorization": f"Bearer {YOUR_API_KEY}"}
  )
  
  print(response.json())

{% endtab %}

{% tab title="Token exchange authentication" %}

Reduce the risk of API key exposure by periodically exchanging the API key for a short-lived API token.

1. To obtain a short-lived API token using your API key, send a POST request to the /api/v1/login endpoint. Here are examples using common HTTP clients:

   • using curl:

     bash Copy

     curl -X POST -d "password=<YOUR_API_KEY>" https://your-zenml-server/api/v1/login

   • using wget:

     bash Copy

     wget -qO- --post-data="password=<YOUR_API_KEY>" \
         --header="Content-Type: application/x-www-form-urlencoded" \
         https://your-zenml-server/api/v1/login

   • using python:

     python Copy

     import requests
     import json
     
     response = requests.post(
         "https://your-zenml-server/api/v1/login",
         data={"password": "<YOUR_API_KEY>"},
         headers={"Content-Type": "application/x-www-form-urlencoded"}
     )
     
     print(response.json())

This will return a response like this (the short-lived API token is the access_token field):

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI3MGJjZTg5NC1hN2VjLTRkOTYtYjE1Ny1kOTZkYWY5ZWM2M2IiLCJpc3MiOiJmMGQ5NjI1Ni04YmQyLTQxZDctOWVjZi0xMmYwM2JmYTVlMTYiLCJhdWQiOiJmMGQ5NjI1Ni04YmQyLTQxZDctOWVjZi0xMmYwM2JmYTVlMTYiLCJleHAiOjE3MTk0MDk0NjAsImFwaV9rZXlfaWQiOiIzNDkyM2U0NS0zMGFlLTRkMjctODZiZS0wZGRhNTdkMjA5MDcifQ.ByB1ngCPtBenGE6UugsWC6Blga3qPqkAiPJUSFDR-u4",
  "token_type": "bearer",
  "expires_in": 3600,
  "refresh_token": null,
  "scope": null
}

2. Once you have obtained a short-lived API token, you can use it to authenticate your API requests by including it in the Authorization header. When the short-lived API token expires, simply repeat the steps above to obtain a new one. For example, you can use the following command to check your current user:

   • using curl:

     bash Copy

     curl -H "Authorization: Bearer YOUR_API_TOKEN" https://your-zenml-server/api/v1/current-user

   • using wget:

     bash Copy

     wget -qO- --header="Authorization: Bearer YOUR_API_TOKEN" https://your-zenml-server/api/v1/current-user

   • using python:

     python Copy

     import requests
     
     response = requests.get(
         "https://your-zenml-server/api/v1/current-user",
         headers={"Authorization": f"Bearer {YOUR_API_TOKEN}"}
     )
     
     print(response.json())

{% endtab %}
{% endtabs %}

{% hint style="info" %}
Important notes

• Short-lived API tokens are scoped to the service account that created them and inherit their permissions
• Tokens are temporary and will expire after a configured duration (typically 1 hour, but it depends on how the server is configured)
• You can request a new short-lived API token at any time using the same API key
• For security reasons, you should handle short-lived API tokens carefully and never share them
• If your API key is compromised, you can rotate it using the ZenML dashboard or by running the zenml service-account api-key <SERVICE_ACCOUNT_NAME> rotate command
  {% endhint %}

Book Api Docs Pro Api Pro Api README (`docs/book/api-docs/pro-api/pro-api/README.md`) Copy Section

icon: rectangle-pro Copy Chapter

Pro API

Book Api Docs Pro Api Pro Api Getting Started (`docs/book/api-docs/pro-api/pro-api/getting-started.md`) Copy Section

icon: person-from-portal Copy Chapter

Getting Started

The ZenML Pro API extends the open-source API with additional features designed for enterprise users, including:

• Enhanced team collaboration features
• Advanced role-based access control
• Enterprise-grade security features

Whether you're using the SaaS version or a self-hosted ZenML Pro instance, you can leverage the API to manage workspaces, organizations, users, roles, and more.

Understanding the Two APIs in ZenML Pro Copy Chapter

ZenML Pro includes two distinct APIs:

1. Workspace API: This is similar to the OSS API but includes additional endpoints (for example to run snapshots). Each workspace in ZenML Pro has its own API.
2. ZenML Pro API: This is the management API for ZenML Pro and handles organization-level resources like workspaces, users, and roles.

Server URL Copy Section

For Pro users, the server_url for the Workspace API is the URL of your workspace, which can be found in the ZenML Pro dashboard:

Note: The workspace URL can be found in the ZenML Pro dashboard and typically looks like https://1bfe8d94-zenml.cloudinfra.zenml.io.

The SaaS version of ZenML Pro API is hosted at https://cloudapi.zenml.io.

API Overview Copy Chapter

The ZenML Pro API is a RESTful API that follows OpenAPI 3.1.0 specifications. It provides endpoints for various resources and operations, including:

• Workspace management
• Organization management
• Project management
• User management
• Role-based access control (RBAC)
• Authentication and authorization

Authentication Copy Chapter

To use the ZenML Pro API, you need to authenticate your requests. If you are logged in to your ZenML Pro account, you can use the same browser window to authenticate requests to your ZenML Pro API, directly in the OpenAPI docs.

For example, for the SaaS variant, you can access the docs here: https://cloudapi.zenml.io.

Programmatic access with Personal Access Tokens Copy Section

Personal Access Tokens are long-lived credentials scoped to your own user account that can be used to authenticate to the ZenML Pro API and the Workspace API across all the organizations and workspaces that your account has access to. See Personal Access Tokens for setup and examples.

Programmatic access with service accounts and API keys Copy Section

Service accounts and API keys can be used to authenticate to the ZenML Pro API and the Workspace API across all workspaces in your organization. See Service Accounts for setup and examples.

When to Use Which API Copy Chapter

See the high‑level overview: Connect to a server.

Workspace API Authentication Copy Chapter

The Workspace API is different from the ZenML Pro API and supports different authentication methods. This is the API you'll use for running snapshots and other pipeline operations.

Workspace API Authentication Methods Copy Section

Programmatic access to the ZenML Pro workspace API is achieved mostly the same way as the ZenML OSS server API. This is because the Workspace API in ZenML Pro is an extension of the OSS API with some additional endpoints. For automation, use ZenML Pro service accounts and API keys (see Service Accounts).

You can use one of these two methods to authenticate with your workspace API:

• Create and use a Personal Access Token (good for development and short-term tasks)
• Create a ZenML Pro service account and use its API key (recommended for production automation and run template triggering)

When making requests to the Workspace API, make sure to use your workspace URL as the base URL. This is different from the ZenML Pro API URL (cloudapi.zenml.io), which is used for organization-level operations.

Please consult the indicated sections for more information.

Key API Endpoints Copy Chapter

Here are some important endpoints you can use with ZenML Pro:

ZenML Pro API Endpoints Copy Section

These endpoints are available at the ZenML Pro API (e.g., https://cloudapi.zenml.io):

Organization Management

• List organizations: GET /organizations
• Create an organization: POST /organizations
• Get organization details: GET /organizations/{organization_id}
• Update an organization: PATCH /organizations/{organization_id}

Workspace Management

• List workspaces: GET /workspaces
• Create a workspace: POST /workspaces
• Get workspace details: GET /workspaces/{workspace_id}
• Update a workspace: PATCH /workspaces/{workspace_id}

User Management

• List users: GET /users
• Get current user: GET /users/me
• Update user: PATCH /users/{user_id}

Role-Based Access Control

• Create a role: POST /roles
• Assign a role: POST /roles/{role_id}/assignments
• Check permissions: GET /permissions

Remember to refer to the complete API documentation available at https://cloudapi.zenml.io for detailed information about all available endpoints, request/response schemas, and additional features.

Workspace API Endpoints Copy Section

The Workspace API includes all OSS API endpoints plus some additional Pro-specific endpoints. These are available at your workspace URL at the /docs path (e.g., https://1bfe8d94-zenml.cloudinfra.zenml.io/docs):

Running snapshots (Pro-specific)

Snapshots allow you to start pipeline runs from external systems using HTTP requests.

• List snapshots: GET /pipeline_snapshots
• Get snapshot details: GET /pipeline_snapshots/<SNAPSHOT-ID>
• Update a snapshot: PATCH /pipeline_snapshots/<SNAPSHOT-ID>
• Run a snapshot: POST /pipeline_snapshots/<SNAPSHOT-ID>/runs

{% hint style="success" %}
For Snapshot Automation

When running snapshots from external systems or CI/CD pipelines, we strongly recommend using service accounts with API keys. Service accounts provide:

• Long-term credentials that don't expire hourly
• Dedicated authentication for automation (not tied to individual users)
• Consistent access even when team members leave

See the service account documentation for setup instructions.
{% endhint %}

For a complete list of API endpoints available in the Workspace API, refer to the OSS API documentation.

Error Handling Copy Chapter

The API uses standard HTTP status codes to indicate the success or failure of requests. In case of errors, the response body will contain more details about the error, including a message and sometimes additional information.

Rate Limiting Copy Chapter

Be aware that the ZenML Pro API may have rate limiting in place to ensure fair usage. If you exceed the rate limit, you may receive a 429 (Too Many Requests) status code. Implement appropriate backoff and retry logic in your applications to handle this scenario.

Troubleshooting Authentication Copy Chapter

"Which API and credentials should I use?" Copy Section

Use this decision tree to determine the right approach:

Common Issues Copy Section

"My API calls are getting 401 Unauthorized"

• ✅ Check that you're using the correct API URL (workspace URL vs cloudapi.zenml.io)
• ✅ Verify your credentials haven't expired (see next issue)
• ✅ Ensure you're using Bearer <token> format in the Authorization header

"My automation keeps failing after an hour"

• ✅ Check what type of authentication you are using (Personal Access Tokens or service account API keys)
• ✅ Check the expiration date of the credentials you are using and rotate them if they have expired. Optionally adjust the expiration time during rotation.

"I can't find the endpoint to run a snapshot"

• ✅ Running snapshots is only possible using the Workspace API, not the ZenML Pro API
• ✅ Use your workspace URL + /docs to see the full API documentation

Book Component Guide Service Connectors Connector Types README (`docs/book/component-guide/service-connectors/connector-types/README.md`) Copy Section

icon: album-collection Copy Chapter

Connector Types

Book How To Infrastructure Deployment Infrastructure As Code README (`docs/book/how-to/infrastructure-deployment/infrastructure-as-code/README.md`) Copy Section

icon: network-wired Copy Chapter

Infrastructure as code with Terraform

Infrastructure as Code (IaC) is
the practice of managing and provisioning infrastructure through code
instead of through manual processes.

In this section, we will show you how to integrate ZenML with popular
IaC tools such as Terraform.

Terraform is a powerful tool for managing infrastructure as code, and is by far the
most popular IaC tool. Many companies already have existing Terraform setups,
and it is often desirable to integrate ZenML with this setup.

We already got a glimpse on how to deploy a cloud stack with Terraform
using existing Terraform modules that are maintained by the ZenML team. While this
is a great solution for quickly getting started, it might not always be suitable for
your use case.

This guide is for advanced users who want to manage their own custom Terraform code but
want to use ZenML to manage their stacks. For this, the
ZenML provider is a better choice.

Understanding the Two-Phase Approach Copy Chapter

When working with ZenML stacks, there are two distinct phases:

1. Infrastructure Deployment: Creating cloud resources (typically handled by platform teams)
2. ZenML Registration: Registering these resources as ZenML stack components

While our official modules (zenml-stack/aws, zenml-stack/gcp, zenml-stack/azure) handle both phases, you might already have infrastructure deployed. Let's explore how to register existing infrastructure with ZenML.

Phase 1: Infrastructure Deployment Copy Chapter

You likely already have this handled in your existing Terraform configurations:

# Example of existing GCP infrastructure
resource "google_storage_bucket" "ml_artifacts" {
  name     = "company-ml-artifacts"
  location = "US"
}

resource "google_artifact_registry_repository" "ml_containers" {
  repository_id = "ml-containers"
  format        = "DOCKER"
}

Phase 2: ZenML Registration Copy Chapter

Setup the ZenML Provider Copy Section

First, configure the ZenML provider to communicate with your ZenML server:

terraform {
  required_providers {
    zenml = {
      source = "zenml-io/zenml"
    }
  }
}

provider "zenml" {
  # Configuration options will be loaded from environment variables:
  # ZENML_SERVER_URL (for Pro users, this should be your Workspace URL from the dashboard)
  # ZENML_API_KEY
}

{% hint style="info" %}
**For ZenML Pro users:** The `ZENML_SERVER_URL` should be your Workspace URL, which can be found in your dashboard. It typically looks like: `https://1bfe8d94-zenml.cloudinfra.zenml.io`. Make sure you use the complete URL of your workspace, not just the domain. The `ZENML_API_KEY` should be [the ZenML Pro API key](https://docs.zenml.io/pro/access-management/service-accounts) or [Personal Access Token](https://docs.zenml.io/pro/access-management/personal-access-tokens).
{% endhint %}

To generate an API key for an OSS server, use the command:

zenml service-account create <SERVICE_ACCOUNT_NAME>

This will create a service account and generate an API key that you can use to authenticate with the ZenML server.

{% hint style="info" %}
The API key is shown only once during creation. Make sure to save it securely, as you cannot retrieve it later. If you lose it, you'll need to create a new key.
{% endhint %}

You can learn more about how to generate a ZENML_API_KEY via service accounts
here.

If you're using a ZenML Pro server, you will need to create a Personal Access Token or an organization-level service account and an API key for it. You can find more about Personal Access Tokens here and organization-level service accounts and API keys here.

Create the service connectors Copy Section

The key to successful registration is proper authentication between the components.
Service connectors are ZenML's way of managing this:

# First, create a service connector
resource "zenml_service_connector" "gcp_connector" {
  name        = "gcp-${var.environment}-connector"
  type        = "gcp"
  auth_method = "service-account"  
  
  configuration = {
    project_id = var.project_id
    service_account_json = file("service-account.json")
  }
}

# Create a stack component referencing the connector
resource "zenml_stack_component" "artifact_store" {
  name   = "existing-artifact-store"
  type   = "artifact_store"
  flavor = "gcp"
  
  configuration = {
    path = "gs://${google_storage_bucket.ml_artifacts.name}"
  }
  
  connector_id = zenml_service_connector.gcp_connector.id
}

Register the stack components Copy Section

Register different types of components:

# Generic component registration pattern
locals {
  component_configs = {
    artifact_store = {
      type = "artifact_store"
      flavor = "gcp"
      configuration = {
        path = "gs://${google_storage_bucket.ml_artifacts.name}"
      }
    }
    container_registry = {
      type = "container_registry"
      flavor = "gcp"
      configuration = {
        uri = "${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.ml_containers.repository_id}"
      }
    }
    orchestrator = {
      type = "orchestrator"
      flavor = "vertex"
      configuration = {
        project = var.project_id
        region  = var.region
      }
    }
  }
}

# Register multiple components
resource "zenml_stack_component" "components" {
  for_each = local.component_configs
  
  name          = "existing-${each.key}"
  type          = each.value.type
  flavor        = each.value.flavor
  configuration = each.value.configuration
  
  connector_id = zenml_service_connector.env_connector.id
}

Assemble the stack Copy Section

Finally, assemble the components into a stack:

resource "zenml_stack" "ml_stack" {
  name = "${var.environment}-ml-stack"
  
  components = {
    for k, v in zenml_stack_component.components : k => v.id
  }
}

Practical Walkthrough: Registering Existing GCP Infrastructure Copy Chapter

Let's see a complete example of registering an existing GCP infrastructure stack with ZenML.

Prerequisites Copy Section

• A GCS bucket for artifacts
• An Artifact Registry repository
• A service account for ML operations
• Vertex AI enabled for orchestration

Step 1: Variables Configuration Copy Section

# variables.tf
variable "zenml_server_url" {
  description = "URL of the ZenML server (for Pro users, this is your Workspace URL)"
  type        = string
}

variable "zenml_api_key" {
  description = "API key for ZenML server authentication"
  type        = string
  sensitive   = true
}

variable "project_id" {
  description = "GCP project ID"
  type        = string
}

variable "region" {
  description = "GCP region"
  type        = string
  default     = "us-central1"
}

variable "environment" {
  description = "Environment name (e.g., dev, staging, prod)"
  type        = string
}

variable "gcp_service_account_key" {
  description = "GCP service account key in JSON format"
  type        = string
  sensitive   = true
}

Step 2: Main Configuration Copy Section

/* Detailed source-code truncated for AI context efficiency. */

Step 3: Outputs Configuration Copy Section

# outputs.tf
output "stack_id" {
  description = "ID of the created ZenML stack"
  value       = zenml_stack.gcp_stack.id
}

output "stack_name" {
  description = "Name of the created ZenML stack"
  value       = zenml_stack.gcp_stack.name
}

output "artifact_store_path" {
  description = "GCS path for artifacts"
  value       = "${google_storage_bucket.artifacts.name}/artifacts"
}

output "container_registry_uri" {
  description = "URI of the container registry"
  value       = "${var.region}-docker.pkg.dev/${var.project_id}/${google_artifact_registry_repository.containers.repository_id}"
}

Step 4: terraform.tfvars Configuration Copy Section

Create a terraform.tfvars file (remember to never commit this to version control):

zenml_server_url = "https://your-zenml-server.com"  # For Pro users: your Workspace URL from dashboard
project_id       = "your-gcp-project-id"
region           = "us-central1"
environment      = "dev"

Store sensitive variables in environment variables:

export TF_VAR_zenml_api_key="your-zenml-api-key"
export TF_VAR_gcp_service_account_key=$(cat path/to/service-account-key.json)

Usage Instructions Copy Section

1. Install required providers and initializing Terraform:

terraform init

2. Install required ZenML integrations:

zenml integration install gcp

3. Review the planned changes:

terraform plan

4. Apply the configuration:

terraform apply

5. Set the newly created stack as active:

zenml stack set $(terraform output -raw stack_name)

6. Verify the configuration:

zenml stack describe

This complete example demonstrates:

• Setting up necessary GCP infrastructure
• Creating a service connector with proper authentication
• Registering stack components with the infrastructure
• Creating a complete ZenML stack
• Proper variable management and output configuration
• Best practices for sensitive information handling

The same pattern can be adapted for AWS and Azure infrastructure by adjusting the provider configurations and resource types accordingly.

Remember to:

• Use appropriate IAM roles and permissions
• Follow your organization's security practices for handling credentials
• Consider using Terraform workspaces for managing multiple environments
• Regular backup of your Terraform state files
• Version control your Terraform configurations (excluding sensitive files)

To learn more about the ZenML terraform provider, visit the
ZenML provider.

Book How To Manage Zenml Server Connecting To Zenml README (`docs/book/how-to/manage-zenml-server/connecting-to-zenml/README.md`) Copy Section

icon: globe-pointer Copy Chapter

Connect to a server

Once ZenML is deployed, there are various ways to connect to it.

Choose how to connect Copy Chapter

Use this quick guide to pick the right method based on your context:

{% hint style="warning" %}
Which base URL should you call?

• Workspace/OSS API: your server or workspace URL (e.g., https://<workspace-id>.zenml.io).
• ZenML Pro management API: https://cloudapi.zenml.io.

In ZenML Pro, use Personal Access Tokens or ZenML Pro service accounts and API keys. These credentials can be used for both the Workspace API and the Pro management API. See ZenML Pro Personal Access Tokens and ZenML Pro Service Accounts.
{% endhint %}

Common pitfalls Copy Chapter

• 401 Unauthorized: verify you’re using the correct base URL, the token hasn’t expired, and the header is Authorization: Bearer <token>.
• Automation fails after 1 hour: check the expiration date of the PAT or API key and rotate it if it has expired.
• Can’t find Run Template endpoints: they exist on the Workspace/OSS API, not on cloudapi.zenml.io.

Book User Guide Llmops Guide Evaluation README (`docs/book/user-guide/llmops-guide/evaluation/README.md`) Copy Section

Evaluation and metrics

In this section, we'll explore how to evaluate the performance of your RAG pipeline using metrics and visualizations. Evaluating your RAG pipeline is crucial to understanding how well it performs and identifying areas for improvement. With language models in particular, it's hard to evaluate their performance using traditional metrics like accuracy, precision, and recall. This is because language models generate text, which is inherently subjective and difficult to evaluate quantitatively.

Our RAG pipeline is a whole system, moreover, not just a model, and evaluating it requires a holistic approach. We'll look at various ways to evaluate the performance of your RAG pipeline but the two main areas we'll focus on are:

• Retrieval evaluation, so checking that the retrieved documents or document chunks are relevant to the query.
• Generation evaluation, so checking that the generated text is coherent and helpful for our specific use case.

In the previous section we built out a basic RAG pipeline for our documentation question-and-answer use case. We'll use this pipeline to demonstrate how to evaluate the performance of your RAG pipeline.

Everything that follows starts from a dataset you assembled and score offline, which is the right shape for a RAG pipeline. An agent already serving traffic gives you the opposite starting point: real runs, some of which went wrong. Kitaru, ZenML's sibling project for agents, replays one of those recorded runs against your real code with a single thing changed — a cheaper model, a different prompt — and diffs the pair, so a failure you saw once becomes a check you can rerun. See building a regression suite from production for that workflow and evaluators for how Kitaru scores sessions.

{% hint style="info" %}
If you were running this in a production setting, you might want to set up evaluation to check the performance of a raw LLM model (i.e. without any retrieval / RAG components) as a baseline, and then compare this to the performance of your RAG pipeline. This will help you understand how much value the retrieval and generation components are adding to your system. We won't cover this here, but it's a good practice to keep in mind.
{% endhint %}

What are we evaluating? Copy Chapter

When evaluating the performance of your RAG pipeline, your specific use case and the extent to which you can tolerate errors or lower performance will determine what you need to evaluate. For instance, if you're building a user-facing chatbot, you might need to evaluate the following:

• Are the retrieved documents relevant to the query?
• Is the generated answer coherent and helpful for your specific use case?
• Does the generated answer contain hate speech or any sort of toxic language?

These are just examples, and the specific metrics and methods you use will depend on your use case. The generation evaluation functions as an end-to-end evaluation of the RAG pipeline, as it checks the final output of the system. It's during these end-to-end evaluations that you'll have most leeway to use subjective metrics, as you're evaluating the system as a whole.

Before we dive into the details, let's take a moment to look at a short high-level code example showcasing the two main areas of evaluation. Afterwards the following sections will cover the two main areas of evaluation in more detail as well as offer practical guidance on when to run these evaluations and what to look for in the results.

Book User Guide Llmops Guide Finetuning Embeddings README (`docs/book/user-guide/llmops-guide/finetuning-embeddings/README.md`) Copy Section

We previously learned how to use RAG with ZenML to build a production-ready RAG pipeline. In this section, we will explore how to optimize and maintain your embedding models through synthetic data generation and human feedback. So far, we've been using off-the-shelf embeddings, which provide a good baseline and decent performance on standard tasks. However, you can often significantly improve performance by finetuning embeddings on your own domain-specific data.

Our RAG pipeline uses a retrieval-based approach, where it first retrieves the most relevant documents from our vector database, and then uses a language model to generate a response based on those documents. By finetuning our embeddings on a dataset of technical documentation similar to our target domain, we can improve the retrieval step and overall performance of the RAG pipeline.

The work of finetuning embeddings based on synthetic data and human feedback is a multi-step process. We'll go through the following steps:

• generating synthetic data with distilabel
• finetuning embeddings with Sentence Transformers
• evaluating finetuned embeddings and using ZenML's model control plane to get a systematic overview

Besides ZenML, we will do this by using two open source libraries: argilla and distilabel. Both of these libraries focus optimizing model outputs through improving data quality, however, each one of them takes a different approach to tackle the same problem. distilabel provides a scalable and reliable approach to distilling knowledge from LLMs by generating synthetic data or providing AI feedback with LLMs as judges. argilla enables AI engineers and domain experts to collaborate on data projects by allowing them to organize and explore data through within an interactive and engaging UI. Both libraries can be used individually but they work better together. We'll showcase their use via ZenML pipelines.

To follow along with the example explained in this guide, please follow the instructions in the llm-complete-guide repository where the full code is also available. This specific section on embeddings finetuning can be run locally or using cloud compute as you prefer.

Book User Guide Llmops Guide Finetuning Llms README (`docs/book/user-guide/llmops-guide/finetuning-llms/README.md`) Copy Section

Finetuning LLMs with ZenML

So far in our LLMOps journey we've learned how to use RAG with ZenML, how to evaluate our RAG systems, how to use reranking to improve retrieval, and how tofinetune embeddings to support and improve our RAG systems. In this section we will explore LLM finetuning itself. So far we've been using APIs like OpenAI and Anthropic, but there are some scenarios where it makes sense to finetune an LLM on your own data. We'll get into those scenarios and how to finetune an LLM in the pages that follow.

While RAG systems are excellent at retrieving and leveraging external knowledge, there are scenarios where finetuning an LLM can provide additional benefits even with a RAG system in place. For example, you might want to finetune an LLM to improve its ability to generate responses in a specific format, to better understand domain-specific terminology and concepts that appear in your retrieved content, or to reduce the length of prompts needed for consistent outputs. Finetuning can also help when you need the model to follow very specific patterns or protocols that would be cumbersome to encode in prompts, or when you want to optimize for latency by reducing the context window needed for good performance.

We'll go through the following steps in this guide:

• Finetuning in 100 lines of code
• Why and when to finetune LLMs
• Starter choices with finetuning
• Finetuning with 🤗 Accelerate
• Evaluation for finetuning
• Deploying finetuned models
• Next steps

This guide is slightly different from the others in that we don't follow a specific use case as the model for finetuning LLMs. The actual steps needed to finetune an LLM are not that complex, but the important part is to understand when you might need to finetune an LLM, how to evaluate the performance of what you do as well as decisions around what data to use and so on.

To follow along with the example explained in this guide, please follow the instructions in the llm-lora-finetuning repository where the full code is also available. This code can be run locally (if you have a GPU attached to your machine) or using cloud compute as you prefer.

Book User Guide Llmops Guide Rag With Zenml README (`docs/book/user-guide/llmops-guide/rag-with-zenml/README.md`) Copy Section

RAG Pipelines with ZenML

Retrieval-Augmented Generation (RAG) is a powerful technique that combines the
strengths of retrieval-based and generation-based models. In this guide, we'll
explore how to set up RAG pipelines with ZenML, including data ingestion, index
store management, and tracking RAG-associated artifacts.

LLMs are a powerful tool, as they can generate human-like responses to a wide
variety of prompts. However, they can also be prone to generating incorrect or
inappropriate responses, especially when the input prompt is ambiguous or
misleading. They are also (currently) limited in the amount of text they can
understand and/or generate. While there are some LLMs like Google's Gemini 1.5
Pro
that can consistently handle 1 million tokens (small units of text), the vast majority (particularly
the open-source ones currently available) handle far less.

The first part of this guide to RAG pipelines with ZenML is about understanding
the basic components and how they work together. We'll cover the following
topics:

• why RAG exists and what problem it solves
• how to ingest and preprocess data that we'll use in our RAG pipeline
• how to leverage embeddings to represent our data; this will be the basis for
  our retrieval mechanism
• how to store these embeddings in a vector database
• how to track RAG-associated artifacts with ZenML

At the end, we'll bring it all together and show all the components working
together to perform basic RAG inference.

Book User Guide Llmops Guide Reranking README (`docs/book/user-guide/llmops-guide/reranking/README.md`) Copy Section

Rerankers are a crucial component of retrieval systems that use LLMs. They help
improve the quality of the retrieved documents by reordering them based on
additional features or scores. In this section, we'll explore how to add a
reranker to your RAG inference pipeline in ZenML.

In previous sections, we set up the overall workflow, from data ingestion and
preprocessing to embeddings generation and retrieval. We then set up some basic
evaluation metrics to assess the performance of our retrieval system. A reranker
is a way to squeeze a bit of extra performance out of the system by reordering
the retrieved documents based on additional features or scores.

As you can see, reranking is an optional addition we make to what we've already
set up. It's not strictly necessary, but it can help improve the relevance and
quality of the retrieved documents, which in turn can lead to better responses
from the LLM. Let's dive in!

Book Api Docs Oss Api Docs V1 Artifact Versions README (`docs/book/api-docs/oss-api-docs/v1/artifact-versions/README.md`) Copy Section

Artifact versions

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions" method="post" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions/{artifact_version_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions/{artifact_version_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions/{artifact_version_id}" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Model Versions README (`docs/book/api-docs/oss-api-docs/v1/model-versions/README.md`) Copy Section

Model versions

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions/{model_version_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions/{model_version_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions/{model_version_id}" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Models README (`docs/book/api-docs/oss-api-docs/v1/models/README.md`) Copy Section

Models

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/models" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/models/{model_name_or_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/models/{model_name_or_id}" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/models/{model_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Pipelines README (`docs/book/api-docs/oss-api-docs/v1/pipelines/README.md`) Copy Section

Pipelines

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/pipelines" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/pipelines/{pipeline_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/pipelines/{pipeline_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/pipelines/{pipeline_id}" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Run Templates README (`docs/book/api-docs/oss-api-docs/v1/run-templates/README.md`) Copy Section

Run templates

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/run_templates" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/run_templates/{template_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/run_templates/{template_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/run_templates/{template_id}" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Runs README (`docs/book/api-docs/oss-api-docs/v1/runs/README.md`) Copy Section

Runs

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Service Accounts README (`docs/book/api-docs/oss-api-docs/v1/service-accounts/README.md`) Copy Section

Service accounts

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts" method="post" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_name_or_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_name_or_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_name_or_id}" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Service Connectors README (`docs/book/api-docs/oss-api-docs/v1/service-connectors/README.md`) Copy Section

Service connectors

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/{connector_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/{connector_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/{connector_id}" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Steps README (`docs/book/api-docs/oss-api-docs/v1/steps/README.md`) Copy Section

Steps

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps" method="post" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps/{step_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps/{step_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Users README (`docs/book/api-docs/oss-api-docs/v1/users/README.md`) Copy Section

Users

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/users" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/users/{user_name_or_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Pro Api Docs Api Reference Auth README (`docs/book/api-docs/pro-api-docs/api-reference/auth/README.md`) Copy Section

Auth

Book Api Docs Pro Api Docs Api Reference Devices README (`docs/book/api-docs/pro-api-docs/api-reference/devices/README.md`) Copy Section

Devices

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/devices" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/devices/{device_id_or_user_code}" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/devices/{device_id}" method="put" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/devices/{device_id}" method="delete" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Pro Api Docs Api Reference Organizations README (`docs/book/api-docs/pro-api-docs/api-reference/organizations/README.md`) Copy Section

Organizations

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations" method="post" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id_or_name}" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}" method="delete" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}" method="patch" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Pro Api Docs Api Reference Rbac README (`docs/book/api-docs/pro-api-docs/api-reference/rbac/README.md`) Copy Section

Rbac

Book Api Docs Pro Api Docs Api Reference Roles README (`docs/book/api-docs/pro-api-docs/api-reference/roles/README.md`) Copy Section

Roles

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles" method="post" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles/{role_id}" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles/{role_id}" method="delete" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles/{role_id}" method="patch" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Pro Api Docs Api Reference Server README (`docs/book/api-docs/pro-api-docs/api-reference/server/README.md`) Copy Section

Server

Book Api Docs Pro Api Docs Api Reference Teams README (`docs/book/api-docs/pro-api-docs/api-reference/teams/README.md`) Copy Section

Teams

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams" method="post" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams/{team_id}" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams/{team_id}" method="delete" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams/{team_id}" method="patch" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Pro Api Docs Api Reference Tenants README (`docs/book/api-docs/pro-api-docs/api-reference/tenants/README.md`) Copy Section

Tenants

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants" method="post" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants" method="delete" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id_or_name}" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id}" method="delete" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id}" method="patch" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Pro Api Docs Api Reference Users README (`docs/book/api-docs/pro-api-docs/api-reference/users/README.md`) Copy Section

Users

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users/{user_id_or_email}" method="get" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users/{user_id}" method="delete" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users/{user_id}" method="patch" %}
https://cloudapi.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Oss Api Docs V1 Service Accounts Api Keys README (`docs/book/api-docs/oss-api-docs/v1/service-accounts/api-keys/README.md`) Copy Section

Api keys

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys" method="post" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys/{api_key_name_or_id}" method="get" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys/{api_key_name_or_id}" method="put" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys/{api_key_name_or_id}" method="delete" %}
https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json
{% endopenapi %}

Book Api Docs Pro Api Docs Api Reference Organizations Validation README (`docs/book/api-docs/pro-api-docs/api-reference/organizations/validation/README.md`) Copy Section

Validation

Book AGENTS (`docs/book/AGENTS.md`) Copy Section

Documentation Agent Guidelines

This file applies to changes in docs/book/ and below. For detailed
GitBook URL examples, link-checking notes, and docs workflow recipes, use
the repo-root .agents/skills/zenml-repo-workflows/SKILL.md.

Source of Truth Copy Chapter

• docs/book/ is the source of truth for documentation content.
• Other docs directories such as docs/mkdocs/ and docs/site/ contain
  generated output; do not edit those directly.
• Docs changes can affect the API docs buildability check in fast CI.

Structure Copy Chapter

• Multiple docs sections have their own toc.md; update the relevant toc.md
  when adding, moving, or removing pages.
• Assets belong in a .gitbook folder at the same level as the relevant
  toc.md.
• GitBook URLs follow TOC hierarchy, not filesystem paths.
• Use absolute https://docs.zenml.io/... links when linking across major docs
  sections such as OSS docs and Pro docs.

Style Copy Chapter

• Include metadata fields at the top of pages when existing nearby pages do so.
• Match the tone and structure of nearby documentation.
• Prioritize readable prose over dense bullet lists.
• Write documentation from the product user's perspective. Describe public
  behavior, user-facing concepts, and the information needed to complete the
  task. Do not expose database relationships, internal resource types,
  implementation mechanics, or lifecycle details unless users must understand
  them to use or troubleshoot the feature. Express observable consequences in
  product terminology instead.
• Include code examples, cross-references, and usage guidance where they help
  users complete the task.

Verification Copy Chapter

• For changed relative markdown links, run the relevant local link checks.
• For absolute docs URLs, check the current URL or inspect the relevant
  toc.md.
• Use Lychee when the change affects many links:
  lychee --offline --no-progress 'docs/book/**/*.md'.

Book Introduction (`docs/book/introduction.md`) Copy Section

icon: torii-gate
layout:
title:
visible: true
visible: true
tableOfContents:
visible: true
outline:
visible: false
pagination:
visible: false

Welcome to ZenML

ZenML is an open-source framework for orchestrating production ML and LLM pipelines, including pipelines that run agentic workloads.

That story has two open-source projects behind it:

• ZenML is the MLOps framework: portable, production-ready pipelines for ML and LLM workloads, with versioned artifacts, caching, and infrastructure abstracted behind stacks.
• Kitaru is for evaluating and improving agentic systems. It turns recorded or imported production traces into sessions for investigation, cohorts, evaluators, replay experiments, regression testing, and CI.

Each works on its own. ZenML orchestrates pipelines; Kitaru provides evals and improvement loops for agentic systems. Kitaru runs its own lightweight server, with workers that execute replays in your environment.

What are you building? Copy Section

Pick the path that matches your work. Neither path requires the other, and adopting the second one later doesn't mean starting over.

How these docs are organized Copy Section

The documentation is split into spaces — the tabs at the top of this page. Knowing what lives where saves you a lot of searching:

First steps Copy Section

Whichever path you picked, the first steps are the same shape: install, run something real, then learn the concepts.

If you use AI coding tools, see LLM tooling for ZenML's MCP server and Agent Skills (including zenml-scoping and zenml-pipeline-authoring).

Guides Copy Section

Book Toc (`docs/book/toc.md`) Copy Section

Table of contents

Getting Started Copy Chapter

• Welcome to ZenML
• Installation
• Hello World
• Your First AI Pipeline
• Core Concepts
• System Architecture

Deploying ZenML Copy Chapter

• Deploy
  • Deploy with Docker
  • Deploy with Helm
    • Migrate to Gateway API
  • Deploy using HuggingFace Spaces
  • Deploy with custom images
  • Secret management
  • Custom secret stores
  • Live event streaming
• Connect
  • with your User (interactive)
  • with your User (programmatic)
  • with a Service Account
• Manage
  • Best practices for upgrading
  • Using ZenML server in production
  • Troubleshoot your ZenML server
  • Migration guide
    • Migration guide 0.13.2 → 0.20.0
    • Migration guide 0.23.0 → 0.30.0
    • Migration guide 0.39.1 → 0.41.0
    • Migration guide 0.58.2 → 0.60.0

Concepts Copy Chapter

• Steps & Pipelines
  • Configuration
  • Scheduling
  • Logging
  • YAML Configuration
  • Source Code and Imports
  • Execution
  • Wait for External Input
  • Advanced Features
  • Hooks
  • Dynamic Pipelines
  • Command Steps
  • Streaming Events
• Artifacts
  • Materializers
  • Visualizations
• Stack & Components
• Service Connectors
• Pipeline Snapshots
• Pipeline Deployments
  • Deployment Settings
• Containerization
• Code Repositories
• Secrets
• Environment Variables
• Tags
• Metadata
• Models
• Dashboard
• Templates

Reference Copy Chapter

• Community & content
• Environment Variables
• LLM Tooling
• FAQ
• Global settings
• Legacy docs

Changelog Items Info (`docs/changelog-items/info.md`) Copy Section

Adding a New Changelog Item

This guide explains how to add a new announcement/changelog item to the ZenML Changelog Widget.

Schema Overview Copy Chapter

Each changelog item is a JSON object with the following fields. The announcements are stored as an array of these objects.

Required Fields Copy Chapter

`id` (number) Copy Section

• Type: Number
• Required: Yes
• Description: Unique identifier for the announcement
• Example: 1, 2, 3

`slug` (string) Copy Section

• Type: String
• Required: Yes
• Description: URL-friendly identifier for the announcement
• Example: "new-pipeline-feature", "bugfix-artifact-loading"

`title` (string) Copy Section

• Type: String
• Required: Yes
• Description: The headline/title of the announcement
• Example: "New Pipeline Visualization Feature"

`description` (string) Copy Section

• Type: String
• Required: Yes
• Description: Detailed description of the change or feature, can use markdown
• Example: "We've added a new interactive pipeline visualization that helps you understand your ML workflows better."

`published_at` (string) Copy Section

• Type: ISO 8601 datetime string
• Required: Yes
• Description: The date and time when the announcement should be published
• Format: YYYY-MM-DDTHH:mm:ss.sssZ or YYYY-MM-DDTHH:mm:ssZ
• Example: "2025-11-10T14:30:00Z", "2025-11-10T14:30:00.000Z"

Optional Fields Copy Chapter

`feature_image_url` (string) Copy Section

• Type: String
• Required: No
• Description: URL to a feature image/screenshot for the announcement
• Example: "<https://zenml.io/assets/feature-image.png>"

`learn_more_url` (string) Copy Section

• Type: Valid URL string
• Required: No
• Validation: Must be a valid URL format
• Description: Link to a blog post, article, or additional information
• Example: "<https://blog.zenml.io/new-feature>"

`docs_url` (string) Copy Section

• Type: Valid URL string
• Required: No
• Validation: Must be a valid URL format
• Description: Link to documentation for the feature
• Example: "<https://docs.zenml.io/user-guide/pipelines>"

`published` (boolean) Copy Section

• Type: Boolean
• Required: No
• Default: true
• Description: Whether the announcement should be visible to users
• Example: true, false

`highlight_until` (string) Copy Section

• Type: ISO 8601 datetime string
• Required: No
• Description: Date until which the announcement should be highlighted
• Format: YYYY-MM-DDTHH:mm:ss.sssZ or YYYY-MM-DDTHH:mm:ssZ
• Example: "2025-11-20T00:00:00Z"

`should_highlight` (boolean) Copy Section

• Type: Boolean
• Required: No
• Default: false
• Description: Whether to prominently highlight this announcement
• Example: true, false

`video_url` (string) Copy Section

• Type: String
• Required: No
• Description: URL to a video demonstration or explanation
• Example: "<https://www.youtube.com/watch?v=example>"

`audience` (enum) Copy Section

• Type: Enum (one of the specified values)
• Required: No
• Default: "all"
• Allowed Values:
  • "pro" - Show only to ZenML Pro users
  • "oss" - Show only to open-source users
  • "all" - Show to all users
• Example: "all"

`labels` (array of strings) Copy Section

• Type: Array of strings
• Required: No
• Default: [] (empty array)
• Allowed Values: Only the following labels are recognized:
  • "feature" - For new features
  • "improvement" - For enhancements to existing functionality
  • "bugfix" - For bug fixes
  • "deprecation" - For deprecated features or warnings
• Note: Any labels not in the allowed list will be filtered out automatically
• Example: ["feature"], ["bugfix", "improvement"]

Complete Example Copy Chapter

{
	"id": 42,
	"slug": "new-artifact-visualization",
	"title": "Enhanced Artifact Visualization",
	"description": "We've completely redesigned how artifacts are displayed in the dashboard, making it easier to explore your ML artifacts and their metadata.",
	"feature_image_url": "<https://zenml.io/assets/artifact-viz.png>",
	"learn_more_url": "<https://blog.zenml.io/artifact-visualization>",
	"docs_url": "<https://docs.zenml.io/user-guide/artifacts>",
	"published": true,
	"published_at": "2025-11-10T10:00:00Z",
	"highlight_until": "2025-11-17T23:59:59Z",
	"should_highlight": true,
	"video_url": "<https://www.youtube.com/watch?v=example>",
	"audience": "all",
	"labels": ["feature", "improvement"]
}

Minimal Example Copy Chapter

{
	"id": 43,
	"slug": "minor-ui-fix",
	"title": "Fixed Button Alignment Issue",
	"description": "Resolved a visual bug where buttons were misaligned in the settings page.",
	"published_at": "2025-11-10T15:30:00Z"
}

Tips Copy Chapter

1. ID Management: Ensure each announcement has a unique ID. Use the next available number in sequence.
2. Datetime Format: Always use ISO 8601 format for dates. It should be in UTC, including the Z suffix
3. URLs: All URL fields (learn_more_url, docs_url) must be valid URLs or the schema validation will fail.
4. Labels: Use appropriate labels to help users filter and understand the type of change. Multiple labels are allowed.
5. Highlighting: Use should_highlight: true and highlight_until for important announcements that need extra visibility.

Mkdocs Cli (`docs/mkdocs/cli.md`) Copy Section

Cli

::: zenml.cli
handler: python
selection:
members:
- "init.py"
rendering:
show_root_heading: true
show_source: true

Book Api Docs Toc (`docs/book/api-docs/toc.md`) Copy Section

Table of contents

• Overview

OSS API Copy Chapter

• Getting Started
• OSS API
  • Artifacts
  • Artifact versions
    • Batch
    • Visualize
  • Login
  • Logout
  • Device authorization
  • Api token
  • Code repositories
  • Logs
  • Models
    • Model versions
  • Model versions
    • Artifacts
    • Runs
  • Pipelines
    • Runs
  • Runs
    • Steps
    • Pipeline configuration
    • Status
    • Refresh
  • Run templates
    • Runs
  • Schedules
  • Secrets
  • Info
  • Service accounts
    • Api keys
    • Rotate
  • Service connectors
    • Verify
    • Client
    • Full stack resources
  • Services
  • Stacks
  • Components
  • Component types
  • Steps
    • Step configuration
    • Status
    • Logs
  • Tags
  • Users
    • Resource membership
  • Current user
• OSS API Specification

Pro API Copy Chapter

• Getting Started
• Pro API
  • Tenants
    • Deploy
    • Deactivate
    • Members
  • Tenant status
  • Users
    • Authorize server
    • Me
  • Invitations
  • Releases
  • Devices
    • Verify
  • Roles
    • Assignments
  • Permissions
  • Teams
    • Members
  • Organizations
    • Trial
    • Invitations
    • Members
    • Roles
    • Teams
    • Tenants
    • Tenant
    • Entitlement
    • Validation
      • Name
      • Tenant name
  • Health
  • Usage event
  • Usage batch
  • Stigg webhook
  • Auth
    • Login
    • Connections
    • Authorize
    • Callback
    • Logout
    • Device authorization
    • Api token
    • Tenant authorization
  • Rbac
    • Check permissions
    • Allowed resource ids
    • Resource members
  • Server
    • Info
• Pro API Specification

Book Component Guide Component Guide (`docs/book/component-guide/component-guide.md`) Copy Section

Overview

If you are new to the world of MLOps, it is often daunting to be immediately faced with a sea of tools that seemingly all promise and do the same things. It is useful in this case to try to categorize tools in various groups in order to understand their value in your toolchain in a more precise manner.

What is a stack? Copy Chapter

The stack is a fundamental component of the ZenML framework. Put simply, a stack represents the configuration of the infrastructure and tooling that defines where and how a pipeline executes.

A stack comprises different stack components, where each component is responsible for a specific task. For example, a stack might have a container registry, a Kubernetes cluster as an orchestrator, an artifact store, an experiment tracker like MLflow and so on.

Each pipeline run that you execute with ZenML will require a stack and each stack will be required to include at least an orchestrator and an artifact store. Apart from these two, the other components are optional and to be added as your pipeline evolves in MLOps maturity.

{% hint style="info" %}
Stacks are a ZenML concept. Kitaru, ZenML's sibling project for replaying and improving AI agents, deliberately does not use them — it runs a lightweight server of its own, with replays executing on workers in your environment.
{% endhint %}

Stacks as a way to organize your execution environment Copy Chapter

With ZenML, you can run your pipelines on more than one stacks with ease. This pattern helps you test your code across different environments effortlessly.

This enables a case like this: a data scientist starts experimentation locally on their system and then once they are satisfied, move to a cloud environment on your staging cloud account to test more advanced features of your pipeline. Finally, when all looks good, they can mark the pipeline ready for production and have it run on a production-grade stack in your production cloud account.

Having separate stacks for these environments helps:

• avoid wrongfully deploying your staging pipeline to production
• curb costs by running less powerful resources in staging and testing locally first
• control access to environments by granting permissions for only certain stacks to certain users

How to manage credentials for your stacks Copy Chapter

Most stack components require some form of credentials to interact with the underlying infrastructure. For example, a container registry needs to be authenticated to push and pull images, a Kubernetes cluster needs to be authenticated to deploy models as a web service, and so on.

The preferred way to handle credentials in ZenML is to use Service Connectors. Service connectors are a powerful feature of ZenML that allow you to abstract away credentials and sensitive information from your team.

Recommended roles Copy Section

Ideally, you would want that only the people who deal with and have direct access to your cloud resources are the ones that are able to create Service Connectors. This is useful for a few reasons:

• Less chance of credentials leaking: the more people that have access to your cloud resources, the higher the chance that some of them will be leaked.
• Instant revocation of compromised credentials: folks who have direct access to your cloud resources can revoke the credentials instantly if they are compromised, making this a much more secure setup.
• Easier auditing: you can have a much easier time auditing and tracking who did what if you have a clear separation between the people who can create Service Connectors (who have direct access to your cloud resources) and those who can only use them.

Recommended workflow Copy Section

Here's an approach you can take that is a good balance between convenience and security:

• Have a limited set of people that have permissions to create Service Connectors. These are ideally people that have access to your cloud accounts and know what credentials to use.
• You can create one connector for your development or staging environment and let your data scientists use that to register their stack components.
• When you are ready to go to production, you can create another connector with permissions for your production environment and create stacks that use it. This way you can ensure that your production resources are not accidentally used for development or staging.

If you follow this approach, you can keep your data scientists free from the hassle of figuring out the best authentication mechanisms for the different cloud services, having to manage credentials locally, and keep your cloud accounts safe, while still giving them the freedom to run their experiments in the cloud.

{% hint style="info" %}
Please note that restricting permissions for users through roles is a ZenML Pro feature. You can read more about it here. Sign up for a free trial here: https://zenml.io/pro.
{% endhint %}

How to deploy and manage stacks Copy Chapter

Deploying and managing a MLOps stack is tricky.

• Each tool comes with a certain set of requirements. For example, a Kubeflow installation will require you to have a Kubernetes cluster, and so would a Seldon Core deployment.
• Figuring out the defaults for infra parameters is not easy. Even if you have identified the backing infra that you need for a stack component, setting up reasonable defaults for parameters like instance size, CPU, memory, etc., needs a lot of experimentation to figure out.
• Many times, standard tool installations don't work out of the box. For example, to run a custom pipeline in Vertex AI, it is not enough to just run an imported pipeline. You might also need a custom service account that is configured to perform tasks like reading secrets from your secret store or talking to other GCP services that your pipeline might need.
• Some tools need an additional layer of installations to enable a more secure, production-grade setup. For example, a standard MLflow tracking server deployment comes without an authentication frontend which might expose all of your tracking data to the world if deployed as-is.
• All the components that you deploy must have the right permissions to be able to talk to each other. For example, your workloads running in a Kubernetes cluster might require access to the container registry or the code repository, and so on.
• Cleaning up your resources after you're done with your experiments is super important yet very challenging. For example, if your Kubernetes cluster has made use of Load Balancers, you might still have one lying around in your account even after deleting the cluster, costing you money and frustration.

All of these points make taking your pipelines to production a more difficult task than it should be. We believe that the expertise in setting up these often-complex stacks shouldn't be a prerequisite to running your ML pipelines.

This docs section consists of information that makes it easier to provision, configure, and extend stacks and components in ZenML.

Stack Components Guide Copy Chapter

Here is a full list of all stack components currently supported in ZenML, with a description of the role of that component in the MLOps process:

Custom Implementations Copy Chapter

You can take control of how ZenML behaves by creating your own components. This is done by writing custom component flavors.

Book Component Guide Integrations (`docs/book/component-guide/integrations.md`) Copy Section

icon: box-archive Copy Chapter

Integrations

Categorizing the MLOps stack is a good way to write abstractions for an MLOps pipeline and standardize your processes. But ZenML goes further and also provides concrete implementations of these categories by integrating with various tools for each category. Once code is organized into a ZenML pipeline, you can supercharge your ML workflows with the best-in-class solutions from various MLOps areas.

For example, you can orchestrate your ML pipeline workflows using Airflow or Kubeflow, track experiments using MLflow Tracking or Weights & Biases, and transition seamlessly from a local MLflow deployment to a deployed model on Kubernetes using Seldon Core.

There are lots of moving parts for all the MLOps tooling and infrastructure you require for ML in production and ZenML brings them all together and enables you to manage them in one place. This also allows you to delay the decision of which MLOps tool to use in your stack as you have no vendor lock-in with ZenML and can easily switch out tools as soon as your requirements change.

Available integrations Copy Chapter

We have a dedicated webpage that indexes all supported ZenML integrations and their categories.

Another easy way of seeing a list of integrations is to see the list of directories in the integrations directory on our GitHub.

Installing dependencies for integrations and stacks Copy Chapter

ZenML provides a way to export the package requirements for both individual integrations and entire stacks, enabling you to install the necessary dependencies manually. This approach gives you full control over the versions and the installation process.

Exporting integration requirements Copy Section

You can export the requirements for a specific integration using the zenml integration export-requirements command. To write the requirements to a file and install them via pip, run:

zenml integration export-requirements <INTEGRATION_NAME> --output-file integration_requirements.txt
pip install -r integration_requirements.txt

If you prefer to see the requirements without writing them to a file, omit the --output-file flag:

zenml integration export-requirements <INTEGRATION_NAME>

This will print the list of dependencies to the console, which you can then pipe to pip:

zenml integration export-requirements <INTEGRATION_NAME> | xargs pip install

Exporting stack requirements Copy Section

To install all dependencies for a specific ZenML stack at once, you can export your stack's requirements:

zenml stack export-requirements <STACK_NAME> --output-file stack_requirements.txt
pip install -r stack_requirements.txt

Omitting --output-file will print the requirements to the console:

zenml stack export-requirements <STACK_NAME>

You can also pipe the output directly to pip:

zenml stack export-requirements <STACK_NAME> | xargs pip install

{% hint style="info" %}
If you use a different package manager such as uv, you can install the exported requirements by replacing pip install -r … with your package manager's equivalent command.
{% endhint %}

Help us with integrations! Copy Chapter

There are countless tools in the ML / MLOps field. We have made an initial prioritization of which tools to support with integrations that are visible on our public roadmap.

We also welcome community contributions. Check our Contribution Guide and External Integration Guide for more details on how to best contribute to new integrations.

Book Component Guide Toc (`docs/book/component-guide/toc.md`) Copy Section

Table of contents

• Overview
• Integrations

Stack Components Copy Chapter

• Orchestrators
  • Local Orchestrator
  • Local Docker Orchestrator
  • Kubeflow Orchestrator
  • Kubernetes Orchestrator
  • Modal Orchestrator
  • Google Cloud VertexAI Orchestrator
  • AWS Sagemaker Orchestrator
  • AzureML Orchestrator
  • Databricks Orchestrator
  • Tekton Orchestrator
  • Airflow Orchestrator
  • Skypilot VM Orchestrator
  • SSH Orchestrator
  • HyperAI Orchestrator
  • Lightning AI Orchestrator
  • Develop a custom orchestrator
• Deployers
  • Local Deployer
  • Docker Deployer
  • Kubernetes Deployer
  • AWS App Runner Deployer
  • GCP Cloud Run Deployer
  • Hugging Face Deployer
• Artifact Stores
  • Local Artifact Store
  • Amazon Simple Cloud Storage (S3)
  • Google Cloud Storage (GCS)
  • Azure Blob Storage
  • Alibaba Cloud OSS
  • MinIO
  • Backblaze B2
  • DigitalOcean Spaces
  • Develop a custom artifact store
• Container Registries
  • Default Container Registry
  • DockerHub
  • Amazon Elastic Container Registry (ECR)
  • Google Cloud Container Registry
  • Azure Container Registry
  • GitHub Container Registry
  • DigitalOcean Container Registry
  • Develop a custom container registry
• Log Stores
  • Artifact Log Store
  • OpenTelemetry Log Store
  • Datadog Log Store
  • Develop a Custom Log Store
• Sandboxes
  • Local
  • Kubernetes
  • Modal
• Step Operators
  • Amazon SageMaker
  • AzureML
  • Baseten
  • Databricks
  • Google Cloud VertexAI
  • Kubernetes
  • Run:AI
  • Modal
  • SSH
  • Spark
  • Develop a Custom Step Operator
• Experiment Trackers
  • Comet
  • MLflow
  • Neptune
  • Weights & Biases
  • Trackio
  • Google Cloud VertexAI Experiment Tracker
  • Develop a custom experiment tracker
• Image Builders
  • Local Image Builder
  • Kaniko Image Builder
  • AWS Image Builder
  • Google Cloud Image Builder
  • Develop a Custom Image Builder
• Alerters
  • Discord Alerter
  • Slack Alerter
  • Develop a Custom Alerter
• Annotators
  • Argilla
  • Label Studio
  • Pigeon
  • Prodigy
  • Develop a Custom Annotator
• Data Validators
  • Great Expectations
  • Deepchecks
  • Evidently
  • Whylogs
  • Develop a custom data validator
• Feature Stores
  • Feast
  • Develop a Custom Feature Store
• Model Deployers
  • MLflow
  • Seldon
  • BentoML
  • Hugging Face
  • Databricks
  • vLLM
  • Develop a Custom Model Deployer
• Model Registries
  • MLflow Model Registry
  • Develop a Custom Model Registry

Service Connectors Copy Chapter

• Introduction
• Complete guide
• Best practices
• Connector Types
  • Docker Service Connector
  • Kubernetes Service Connector
  • AWS Service Connector
  • GCP Service Connector
  • Azure Service Connector
  • OAuth2 Service Connector
  • HyperAI Service Connector

Popular Stacks Copy Chapter

• AWS
• Azure
• GCP
• Kubernetes

Deployment Copy Chapter

• 1-click Deployment
• Terraform Modules
• Register a cloud stack
• Infrastructure as code

Contribute Copy Chapter

• Custom Stack Component
• Custom Integration

Book Getting Started Core Concepts (`docs/book/getting-started/core-concepts.md`) Copy Section

icon: lightbulb Copy Chapter

Core concepts

ZenML is a unified, extensible, open-source MLOps framework for creating portable, production-ready MLOps pipelines. It's built for data scientists, ML Engineers, and MLOps Developers to collaborate as they develop to production. By extending the battle-tested principles you rely on for classical ML to the new world of AI agents, ZenML serves as one platform to develop, evaluate, and deploy your entire AI portfolio - from decision trees to complex multi-agent systems. In order to achieve this goal, ZenML introduces various concepts for different aspects of ML workflows and AI agent development, and we can categorize these concepts under three different threads:

{% embed url="https://www.youtube.com/embed/iCB4KNjl5vs" %}
If you prefer visual learning, this short video demonstrates the key concepts covered below.
{% endembed %}

1. Development Copy Chapter

First, let's look at the main concepts that play a role during the development stage of ML workflows and AI agent pipelines with ZenML.

Step

Steps are functions annotated with the @step decorator. The easiest one could look like this.

from zenml import step

@step
def step_1() -> str:
    """Returns a string."""
    return "world"

These functions can also have inputs and outputs. For ZenML to work properly, these should preferably be typed.

from zenml import step

@step(enable_cache=False)
def step_2(input_one: str, input_two: str) -> str:
    """Combines the two strings passed in."""
    combined_str = f"{input_one} {input_two}"
    return combined_str

@step
def evaluate_agent_response(prompt: str, test_query: str) -> dict:
    """Evaluates an AI agent's response to a test query."""
    response = call_llm_agent(prompt, test_query)
    return {"query": test_query, "response": response, "quality_score": 0.95}

Pipelines

At its core, ZenML follows a pipeline-based workflow for your projects. A pipeline consists of a series of steps, organized in any order that makes sense for your use case.

As seen in the image, a step might use the outputs from a previous step and thus must wait until the previous step is completed before starting. This is something you can keep in mind when organizing your steps.

Pipelines and steps are defined in code using Python decorators or classes. This is where the core business logic and value of your work live, and you will spend most of your time defining these two things.

Even though pipelines are simple Python functions, you are only allowed to call steps within this function. The inputs for steps called within a pipeline can either be the outputs of previous steps or alternatively, you can pass in values directly or map them onto pipeline parameters (as long as they're JSON-serializable). Similarly, you can return values from a pipeline that are step outputs as long as they are JSON-serializable.

from zenml import pipeline

@pipeline
def my_pipeline():
    output_step_one = step_1()
    step_2(input_one="hello", input_two=output_step_one)

@pipeline
def agent_evaluation_pipeline(query: str = "What is machine learning?") -> str:
    """An AI agent evaluation pipeline."""
    prompt = "You are a helpful assistant. Please answer: {query}"
    evaluation_result = evaluate_agent_response(prompt, query)
    return evaluation_result

{% hint style="info" %}
If you use an AI coding agent, ZenML Agent Skills can help you scope a pipeline architecture (zenml-scoping) and then implement steps and pipelines correctly (zenml-pipeline-authoring). See LLM tooling.
{% endhint %}

Executing the Pipeline is as easy as calling the function that you decorated with the @pipeline decorator.

if __name__ == "__main__":
    my_pipeline()
    agent_evaluation_pipeline(query="What is an LLM?")

Artifacts

Artifacts represent the data that goes through your steps as inputs and outputs, and they are automatically tracked and stored by ZenML in the artifact store. They are produced by and circulated among steps whenever your step returns an object or a value. This means the data is not passed between steps in memory. Rather, when the execution of a step is completed, they are written to storage, and when a new step gets executed, they are loaded from storage.

Artifacts can be traditional ML data (datasets, models, metrics) or AI agent components (prompt templates, agent configurations, evaluation results). The same artifact system seamlessly handles both use cases.

The serialization and deserialization logic of artifacts is defined by Materializers.

Models

Models are used to represent the outputs of a training process along with all metadata associated with that output. In other words: models in ZenML are more broadly defined as the weights as well as any associated information. This includes traditional ML models (scikit-learn, PyTorch, etc.) and AI agent configurations (prompt templates, tool definitions, multi-agent system architectures). Models are first-class citizens in ZenML and as such viewing and using them is unified and centralized in the ZenML API, client, as well as on the ZenML Pro dashboard.

Materializers

Materializers define how artifacts live in between steps. More precisely, they define how data of a particular type can be serialized/deserialized, so that the steps are able to load the input data and store the output data.

All materializers use the base abstraction called the BaseMaterializer class. While ZenML comes built-in with various implementations of materializers for different datatypes, if you are using a library or a tool that doesn't work with our built-in options, you can write your own custom materializer to ensure that your data can be passed from step to step.

Parameters & Settings

When we think about steps as functions, we know they receive input in the form of artifacts. We also know that they produce output (in the form of artifacts, stored in the artifact store). But steps also take parameters. The parameters that you pass into the steps are also (helpfully!) stored by ZenML. This helps freeze the iterations of your experimentation workflow in time, so you can return to them exactly as you run them. On top of the parameters that you provide for your steps, you can also use different Settings to configure runtime configurations for your infrastructure and pipelines.

Model and model versions

ZenML exposes the concept of a Model, which consists of multiple different model versions. A model version represents a unified view of the ML models that are created, tracked, and managed as part of a ZenML project. Model versions link all other entities to a centralized view.

2. Execution Copy Chapter

Once you have implemented your workflow by using the concepts described above, you can focus your attention on the execution of the pipeline run.

Stacks & Components

When you want to execute a pipeline run with ZenML, Stacks come into play. A Stack is a collection of stack components, where each component represents the respective configuration regarding a particular function in your MLOps pipeline, such as pipeline orchestration or deployment systems, artifact repositories and container registries.

Pipelines can be executed in two ways: in batch mode (traditional execution through an orchestrator) or in online mode (long-running HTTP servers that can be invoked via REST API calls). Deploying pipelines for online mode execution allows you to serve your ML workflows as real-time endpoints, making them accessible for live inference and interactive use cases.

For instance, if you take a close look at the default local stack of ZenML, you will see two components that are required in every stack in ZenML, namely an orchestrator and an artifact store. Additional components like deployers can be added to enable specific functionality such as deploying pipelines as HTTP endpoints.

{% hint style="info" %}
Keep in mind that each one of these components is built on top of base abstractions and is completely extensible.
{% endhint %}

Orchestrator

An Orchestrator is a workhorse that coordinates all the steps to run in a pipeline in batch mode. Since pipelines can be set up with complex combinations of steps with various asynchronous dependencies between them, the orchestrator acts as the component that decides what steps to run and when to run them.

ZenML comes with a default local orchestrator designed to run on your local machine. This is useful, especially during the exploration phase of your project. You don't have to rent a cloud instance just to try out basic things.

Artifact Store

An Artifact Store is a component that houses all data that passes through the pipeline as inputs and outputs. Each artifact that gets stored in the artifact store is tracked and versioned and this allows for extremely useful features like data caching, which speeds up your workflows.

Similar to the orchestrator, ZenML comes with a default local artifact store designed to run on your local machine. This is useful, especially during the exploration phase of your project. You don't have to set up a cloud storage system to try out basic things.

Deployer

A Deployer is a stack component that manages the deployment of pipelines as long-running HTTP servers useful for online mode execution. Unlike orchestrators that execute pipelines in batch mode, deployers can create and manage persistent services that wrap your pipeline in a web application, usually containerized, allowing it to be invoked through HTTP requests.

ZenML comes with a Docker deployer that can run deployments on your local machine as Docker containers, making it easy to test and develop real-time pipeline endpoints before moving to production infrastructure.

Flavor

ZenML provides a dedicated base abstraction for each stack component type. These abstractions are used to develop solutions, called Flavors, tailored to specific use cases/tools. With ZenML installed, you get access to a variety of built-in and integrated Flavors for each component type, but users can also leverage the base abstractions to create their own custom flavors.

Stack Switching

When it comes to production-grade solutions, it is rarely enough to just run your workflow locally without including any cloud infrastructure.

Thanks to the separation between the pipeline code and the stack in ZenML, you can easily switch your stack independently from your code. For instance, all it would take you to switch from an experimental local stack running on your machine to a remote stack that employs a full-fledged cloud infrastructure is a single CLI command.

Pipeline Snapshot

A Pipeline Snapshot is an immutable snapshot of your pipeline that includes the pipeline DAG, code, configuration, and container images. Snapshots can be run from the server or dashboard, and can also be deployed.

Pipeline Run

A Pipeline Run is a record of a pipeline execution. When you run a pipeline using an orchestrator, a pipeline run is created tracking information about the execution such as the status, the artifacts and metadata produced by the pipeline and all its steps. When a pipeline is deployed for online mode execution, a pipeline run is similarly created for every HTTP request made to it.

Deployment

A Deployment is a running instance of a pipeline deployed as an HTTP endpoint. When you deploy a pipeline using a deployer, it becomes a long-running service that can be invoked through REST API calls. Each HTTP request to a deployment triggers a new pipeline run, creating the same artifacts and metadata tracking as traditional batch pipeline executions. This enables real-time inference, interactive ML workflows, and seamless integration with web applications and external services.

3. Management Copy Chapter

In order to benefit from the aforementioned core concepts to their fullest extent, it is essential to deploy and manage a production-grade environment that interacts with your ZenML installation.

ZenML Server

To use stack components that are running remotely on a cloud infrastructure, you need to deploy a ZenML Server so it can communicate with these stack components and run your pipelines. The server is also responsible for managing ZenML business entities like pipelines, steps, models, etc.

Server Deployment

In order to benefit from the advantages of using a deployed ZenML server, you can either choose to use the ZenML Pro SaaS offering, which provides a control plane for you to create managed instances of ZenML servers, or deploy it in your self-hosted environment.

Metadata Tracking

On top of the communication with the stack components, the ZenML Server also keeps track of all the bits of metadata around a pipeline run. With a ZenML server, you are able to access all of your previous experiments with the associated details. This is extremely helpful in troubleshooting.

Secrets

The ZenML Server also acts as a centralized secrets store that safely and securely stores sensitive data, such as credentials used to access the services that are part of your stack. It can be configured to use a variety of different backends for this purpose, such as the AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, and Hashicorp Vault.

Secrets are sensitive data that you don't want to store in your code or configure alongside your stacks and pipelines. ZenML includes a centralized secrets store that you can use to store and access your secrets securely.

Collaboration

Collaboration is a crucial aspect of any MLOps team as they often need to bring together individuals with diverse skills and expertise to create a cohesive and effective workflow for machine learning projects and AI agent development. A successful MLOps team requires seamless collaboration between data scientists, engineers, and DevOps professionals to develop, train, deploy, and maintain both traditional ML models and AI agent systems.

With a deployed ZenML Server, users have the ability to create their own teams and project structures. They can easily share pipelines, runs, stacks, and other resources, streamlining the workflow and promoting teamwork across the entire AI development lifecycle.

Dashboard

The ZenML Dashboard also communicates with the ZenML Server to visualize your pipelines, stacks, and stack components. The dashboard serves as a visual interface to showcase collaboration with ZenML. You can invite users and share your stacks with them.

When you start working with ZenML, you'll start with a local ZenML setup, and when you want to transition, you will need to deploy ZenML. Don't worry though, there is a one-click way to do it, which we'll learn about later.

VS Code Extension

ZenML also provides a VS Code extension that allows you to interact with your ZenML stacks, runs, and server directly from your VS Code editor. If you're working on code in your editor, you can easily switch and inspect the stacks you're using, delete and inspect pipelines as well as even switch stacks.

Book Getting Started Hello World (`docs/book/getting-started/hello-world.md`) Copy Section

Your first ML pipeline with ZenML - from local development to cloud deployment in minutes. icon: hand-wave Copy Chapter

Hello World

This guide will help you build and deploy your first ZenML pipeline, starting locally and then transitioning to the cloud without changing your code. The same principles you'll learn here apply whether you're building classical ML models or AI agents.

{% stepper %}
{% step %}

Install ZenML

Start by installing ZenML in a fresh Python environment:

pip install 'zenml[server]'
zenml login

This gives you access to both the ZenML Python SDK and CLI tools. It also surfaces the
ZenML dashboard + connects it to your local client.
{% endstep %}

{% step %}

Write your first pipeline

Create a simple run.py file with a basic workflow:

from zenml import step, pipeline

if name == "main":
basic_pipeline()


Run this pipeline in batch mode locally:

python run.py

You will see ZenML automatically tracks the execution and stores artifacts. View these on the CLI or on the dashboard.

{% endstep %}

{% step %}

Create a Pipeline Snapshot (Optional but Recommended)

Before deploying, you can create a snapshot - an immutable, reproducible version of your pipeline including code, configuration, and container images:

# Create a snapshot of your pipeline
zenml pipeline snapshot create run.basic_pipeline --name my_snapshot

Snapshots are powerful because they:

• Freeze your pipeline state - Ensure the exact same pipeline always runs
• Enable parameterization - Run the same snapshot with different inputs
• Support team collaboration - Share ready-to-use pipeline configurations
• Integrate with automation - Trigger from dashboards, APIs, or CI/CD systems

Learn more about Snapshots
{% endstep %}

{% step %}

Deploy your pipeline as a real-time service

ZenML can deploy your pipeline (or snapshot) as a persistent HTTP service for real-time inference:

# Deploy your pipeline directly
zenml pipeline deploy run.basic_pipeline --name my_deployment

# OR deploy a snapshot (if you created one above)
zenml pipeline snapshot deploy my_snapshot --deployment my_deployment

Your pipeline now runs as a production-ready service! This is perfect for serving predictions to web apps, powering AI agents, or handling real-time requests.

Key insight: When you deploy a pipeline directly with zenml pipeline deploy, ZenML automatically creates an implicit snapshot behind the scenes, ensuring reproducibility.

Learn more about Pipeline Deployments
{% endstep %}

{% step %}

Set up a ZenML Server (For Remote Infrastructure)

To use remote infrastructure (cloud deployers, orchestrators, artifact stores), you need to deploy a ZenML server to manage your pipelines centrally. You can use ZenML Pro (managed, 14-day free trial) or deploy it yourself (self-hosted, open-source).

Connect your local environment:

zenml login
zenml project set <PROJECT_NAME>

Once connected, you'll have a centralized dashboard to manage infrastructure, collaborate with team members, and schedule pipeline runs.
{% endstep %}

{% step %}

Create your first remote stack (Optional)

A "stack" in ZenML represents the infrastructure where your pipelines run. You can now scale from local development to cloud infrastructure without changing any code.

Remote stacks can include:

• Remote Deployers (AWS App Runner, GCP Cloud Run, Azure Container Instances) - for deploying your pipelines as scalable HTTP services on the cloud
• Remote Orchestrators (Kubernetes, GCP Vertex AI, AWS SageMaker) - for running batch pipelines at scale
• Remote Artifact Stores (S3, GCS, Azure Blob) - for storing and versioning pipeline artifacts

The fastest way to create a cloud stack is through the Infrastructure-as-Code option, which uses Terraform to deploy cloud resources and register them as a ZenML stack.

You'll need:

• Terraform version 1.9+ installed locally
• Authentication configured for your preferred cloud provider (AWS, GCP, or Azure)
• Appropriate permissions to create resources in your cloud account

# Create a remote stack using the deployment wizard
zenml stack register <STACK_NAME> \
  --deployer <DEPLOYER_NAME> \
  --orchestrator <ORCHESTRATOR_NAME> \
  --artifact-store <ARTIFACT_STORE_NAME>

The wizard will guide you through each step.
{% endstep %}

{% step %}

Deploy and run on remote infrastructure

Once you have a remote stack, you can:

1. Deploy your service to the cloud - Your deployment runs on managed cloud infrastructure:

zenml stack set <REMOTE_STACK_NAME>
zenml pipeline deploy run.basic_pipeline --name my_production_deployment

2. Run batch pipelines at scale - Use the same code with a cloud orchestrator:

zenml stack set <REMOTE_STACK_NAME>
python run.py  # Automatically runs on cloud infrastructure

ZenML handles packaging code, building containers, orchestrating execution, and tracking artifacts automatically across all cloud providers.

{% step %}

What's next?

Congratulations! You've just experienced the core value proposition of ZenML:

• Write Once, Run Anywhere: The same code runs locally during development and in the cloud for production
• Unified Framework: Use the same MLOps principles for both classical ML models and AI agents
• Separation of Concerns: Infrastructure configuration and ML code are completely decoupled, enabling independent
  evolution of each
• Full Tracking: Every run, artifact, and model is automatically versioned and tracked - whether it's a scikit-learn model or a multi-agent system

To continue your ZenML journey, explore these key topics:

For All AI Workloads:

• AI-assisted pipeline building: Use ZenML Agent Skills like zenml-scoping (architecture planning) and zenml-pipeline-authoring (implementation guidance) — see LLM tooling
• Pipeline Development: Discover advanced features like scheduling and caching
• Artifact Management: Learn how ZenML stores, versions, and tracks your data automatically
• Organization: Use tags and metadata to keep your AI projects structured

For LLMs and AI Agents:

• LLMOps Guide: Write your first AI pipeline for agent development patterns
• Deploying Agents: To see an example of a deployed document extraction agent, see the deploying agents example
• Agent Outer Loop: See the Agent Outer Loop example to learn about training classifiers and improving agents through feedback loops
• Agent Evaluation: Learn to systematically evaluate and compare different agent architectures
• Prompt Management: Version and track prompts, tools, and agent configurations as artifacts

Infrastructure & Deployment:

• Containerization: Understand how ZenML handles containerization for reproducible execution
• Stacks & Infrastructure: Explore the concepts behind stacks and service connectors for authentication
• Secrets Management: Learn how to handle sensitive information securely
• Snapshots: Create reusable snapshots for standardized workflows
  {% endstep %}
  {% endstepper %}

Book Getting Started Installation (`docs/book/getting-started/installation.md`) Copy Section

icon: cauldron Copy Chapter

Installation

{% stepper %}
{% step %}

Install ZenML

ZenML currently supports Python 3.10, 3.11, 3.12, 3.13, and 3.14. Please make sure that you are using a supported Python version.

{% tabs %}
{% tab title="Base package" %}
ZenML is a Python package that can be installed using pip or other Python package managers:

pip install zenml

{% hint style="warning" %}
Installing the base package only allows you to connect to a deployed ZenML server. If you want to use ZenML purely locally, install it with the local extra:

pip install 'zenml[local]'

{% endhint %}
{% endtab %}

{% tab title="Local Dashboard" %}
If you want to use the ZenML dashboard locally, install ZenML with the server extra:

pip install 'zenml[server]'

{% hint style="warning" %}
If you want to run a local server while running on a Mac with Apple Silicon (M1, M2, M3, M4), you should set the following environment variable:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

You can read more about this here.
{% endhint %}

{% endtab %}

{% tab title="Jupyter Notebooks" %}
If you write your ZenML pipelines ins Jupyter notebooks, we recommend installing ZenML with the jupyter extra which includes improved CLI output and logs:

pip install 'zenml[jupyter]'

{% endtab %}

{% endtabs %}
{% endstep %}

{% step %}

Verifying Installations

Once the installation is completed, you can check whether the installation was successful either through Bash or Python:

{% tabs %}
{% tab title="Bash" %}

zenml version

{% endtab %}

{% tab title="Python" %}

import zenml

print(zenml.__version__)

{% endtab %}
{% endtabs %}

If you would like to learn more about the current release, please visit our PyPi package page.
{% endstep %}
{% endstepper %}

Running with Docker Copy Chapter

zenml is also available as a Docker image hosted publicly on DockerHub. Use the following command to get started in a bash environment with zenml available:

docker run -it zenmldocker/zenml /bin/bash

If you would like to run the ZenML server with Docker:

docker run -it -d -p 8080:8080 zenmldocker/zenml-server

Starting the local server Copy Chapter

By default, ZenML runs without a server connected to a local database on your machine. If you want to access the dashboard locally, you need to start a local server:

# Make sure to have the `server` extra installed
pip install "zenml[server]"
zenml login --local  # opens the dashboard locally

However, advanced ZenML features are dependent on a centrally deployed ZenML server accessible to other MLOps stack components. You can read more about it here. For the deployment of ZenML, you have the option to either self-host it or register for a free ZenML Pro account.

Book Getting Started System Architectures (`docs/book/getting-started/system-architectures.md`) Copy Section

icon: building-columns Copy Chapter

System Architecture

This guide walks through the various ways that ZenML can be deployed, from self-hosted OSS to
SaaS to self-hosted ZenML Pro!

ZenML OSS (Self-hosted) Copy Chapter

{% hint style="info" %}
This page is intended as a high-level overview. To learn more about how to deploy ZenML OSS, read this guide.
{% endhint %}

A ZenML OSS deployment consists of the following moving pieces:

• ZenML OSS Server: This is a FastAPI app that manages metadata of pipelines, artifacts, stacks, etc. Note: In ZenML Pro, the notion of a ZenML server is replaced with what is known as a "Workspace". For all intents and purposes, consider a ZenML Workspace to be a ZenML OSS server that comes with more functionality.
• OSS Metadata Store: This is where all ZenML workspace metadata is stored, including ML metadata such as tracking and versioning information about pipelines and models.
• OSS Dashboard: This is a ReactJS app that shows pipelines, runs, etc.
• Secrets Store: All secrets and credentials required to access customer infrastructure services are stored in a secure secrets store. The ZenML Pro API has access to these secrets and uses them to access customer infrastructure services on behalf of the ZenML Pro. The secrets store can be hosted either by the ZenML Pro or by the customer.

ZenML OSS is free with Apache 2.0 license. Learn how to deploy it here.

{% hint style="info" %}
To learn more about the core concepts for ZenML OSS, go here.
{% endhint %}

ZenML Pro (SaaS or Self-hosted) Copy Chapter

{% hint style="info" %}
If you're interested in assessing ZenML Pro SaaS, you can create a free account.

If you would like to self-host ZenML Pro, please book a demo.
{% endhint %}

The above deployment can be augmented with the ZenML Pro components:

• ZenML Pro Control Plane: This is the central controlling entity of all workspaces.
• Pro Dashboard: This is a dashboard that builds on top of the OSS dashboard and adds further functionality.
• Pro Metadata Store: This is a PostgreSQL database where all ZenML Pro-related metadata is stored, such as roles, permissions, teams, and workspace management-related data.
• Pro Add-ons: These are Python modules injected into the OSS Server for enhanced functionality.
• Identity Provider: ZenML Pro offers flexible authentication options. In cloud-hosted deployments, it integrates with Auth0, allowing users to log in via social media or corporate credentials. For self-hosted deployments, customers can configure their own identity management solution, with ZenML Pro supporting custom OIDC provider integration. This allows organizations to leverage their existing identity infrastructure for authentication and authorization, whether using the cloud service or deploying on-premises.

ZenML Pro offers many additional features to increase your team's productivity. No matter your specific needs, the hosting options for ZenML Pro range from easy SaaS integration to completely air-gapped deployments on your own infrastructure.

You might have noticed that this architecture builds on top of the ZenML OSS system architecture. Therefore, if you already have ZenML OSS deployed, it is easy to enroll it as part of a ZenML Pro deployment!

The above components interact with other MLOps stack components, secrets, and data in the following scenarios described below.

{% hint style="info" %}
To learn more about the core concepts for ZenML Pro, go here
{% endhint %}

ZenML Pro SaaS Architecture Copy Section

For the ZenML Pro SaaS deployment case, all ZenML services are hosted on infrastructure hosted by the ZenML Team. Customer secrets and credentials required to access customer infrastructure are stored and managed by the ZenML Pro Control Plane.

On the ZenML Pro infrastructure, only ML metadata (e.g. pipeline and model tracking and versioning information) is stored. All the actual ML data artifacts (e.g. data produced or consumed by pipeline steps, logs and visualizations, models) are stored on the customer cloud. This can be set up quite easily by configuring an artifact store with your MLOps stack.

Your workspace only needs permissions to read from this data to display artifacts on the ZenML dashboard. The workspace also needs direct access to parts of the customer infrastructure services to support dashboard control plane features such as CI/CD, triggering and running pipelines, triggering model deployments and so on.

The advantage of this setup is that it is a fully-managed service, and is very easy to get started with. However, for some clients, even some metadata can be sensitive; these clients should refer to the other architecture diagram.

ZenML Pro Hybrid SaaS Copy Section

The partially self-hosted architecture offers a balanced approach that combines the benefits of cloud-hosted control with on-premises data sovereignty. In this configuration, while the ZenML Pro control plane remains hosted by ZenML (handling user management, authentication, RBAC and global workspace coordination), all other components - including services, data, and secrets - are deployed within your own cloud infrastructure.

This hybrid model is particularly well-suited for organizations with:

• A centralized MLOps or Platform team responsible for standardizing ML practices
• Multiple business units or teams that require autonomy over their data and infrastructure
• Strict security requirements where workspaces must operate behind VPN/corporate firewalls
• Compliance requirements that mandate keeping sensitive data and ML artifact metadata within company infrastructure
• Need for customization of workspace configurations while maintaining centralized governance

The key advantages of this setup include:

• Simplified user management through the ZenML-hosted control plane
• Complete data sovereignty - sensitive data and ML artifacts remain within your infrastructure
• Secure networking - workspaces communicate through outbound-only connections via VPN/private networks
• Ability to customize and configure workspaces according to specific team needs
• Reduced operational overhead compared to fully self-hosted deployments
• Reduced maintenance burden - all control plane updates and maintenance are handled by ZenML
  This architecture strikes a balance between convenience and control, making it a popular choice for enterprises looking to standardize their MLOps practices while maintaining sovereignty.

ZenML Pro Self-Hosted Architecture Copy Section

In the case of self-hosting ZenML Pro, all services, data, and secrets are deployed on the customer
cloud. This is meant for customers who require completely air-gapped deployments, for the tightest security standards. Reach out to us if you want to set this up.

Detailed Architecture Diagram for self-hosted ZenML Pro deployment

Are you interested in ZenML Pro? Sign up and get access with a free trial now!

Data Implications Across Deployment Scenarios Copy Chapter

Book Getting Started Your First Ai Pipeline (`docs/book/getting-started/your-first-ai-pipeline.md`) Copy Section

icon: rocket Copy Chapter

Your First AI Pipeline Copy Chapter

ZenML pipelines work the same for classical ML, AI agents, and hybrid approaches. Choose your path below to get started:

{% hint style="info" %}
Why ZenML pipelines?

• Reproducible & portable: Run the same code locally or on the cloud by switching stacks.
• One approach for models and agents: Steps, pipelines, and artifacts work for sklearn, classical ML, and LLMs alike.
• Observe by default: Lineage and step metadata (e.g., latency, tokens, metrics) are tracked and visible in the dashboard.
  {% endhint %}

{% hint style="info" %}
Plan and build pipelines with AI coding agents: If you use Claude Code, Codex, Copilot, Cursor, or similar tools, ZenML provides Agent Skills that can scope your idea into a realistic multi-pipeline plan (zenml-scoping) and help you implement steps and pipelines (zenml-pipeline-authoring). See LLM tooling for installation instructions.
{% endhint %}

What do you want to build? Copy Chapter

Choose one of the paths below. The same ZenML pipeline pattern works for all of them—the difference is in your steps and how you orchestrate them.

• Build AI Agents - Use LLMs and tools to create autonomous agents
• Build Classical ML Pipelines - Train and serve ML models with scikit-learn, TensorFlow, or PyTorch
• Build Hybrid Systems - Combine ML classifiers with agents

Path 1: Build AI Agents Copy Chapter

Use large language models, prompts, and tools to build intelligent autonomous agents that can reason, take action, and interact with your systems.

{% hint style="info" %}
Evaluating production agents. This path runs your agent inside a ZenML pipeline, which is useful for orchestrated batch workloads and evaluations. If your agent runs elsewhere in production, Kitaru can record or import its traces as sessions. You can investigate those sessions, define cohorts and evaluators, and use replay experiments to test changes and build regression checks. Kitaru does not use ZenML pipelines or stacks for this workflow.
{% endhint %}

Architecture example Copy Section

---
config:
  layout: elk
  theme: mc
---
flowchart TB
  U["CLI / curl / web UI"] --> D["ZenML Deployment<br/>(doc_analyzer)"]

  subgraph PIPE["Pipeline: doc_analyzer"]
    I["ingest_document_step"]
    A["analyze_document_step"]
    R["render_analysis_report_step"]
    I --> A --> R
  end

  D --> PIPE

  subgraph STACK["Stack"]
    OR[("Deployer")]
    AR[("Artifact Store")]
  end

  PIPE --> AR
  D --> OR

View Quick Start & Examples

Quick start Copy Section

git clone --depth 1 https://github.com/zenml-io/zenml.git
cd zenml/examples/deploying_agent
uv pip install -r requirements.txt

Then follow the guide in examples/deploying_agent:

1. Define your steps: Use LLM APIs (OpenAI, Claude, etc.) to build reasoning steps
2. Deploy as HTTP service: Turn your agent into a managed endpoint
3. Invoke and monitor: Use the CLI, curl, or the embedded web UI to interact with your agent
4. Inspect traces: View agent reasoning, tool calls, and metadata in the ZenML dashboard

Example output Copy Section

• Automated document analysis (see deploying_agent)
• Multi-turn chatbots with context
• Autonomous workflows with tool integrations
• Agentic RAG systems with retrieval steps

Related examples Copy Section

• agent_outer_loop: Combine ML classifiers with agents for hybrid intelligent systems
• agentic_hitl_pipeline: Add dynamic fan-out and human approval to an agent workflow
• agent_comparison: Compare different agent architectures and LLM providers
• agent_framework_integrations: Integrate with popular agent frameworks
• llm_finetuning: Fine-tune LLMs for specialized tasks

Path 2: Build Classical ML Pipelines Copy Chapter

Use scikit-learn, TensorFlow, PyTorch, or other ML frameworks to build data processing, feature engineering, training, and inference pipelines.

Architecture example Copy Section

---
config:
  layout: elk
  theme: mc
---
flowchart TB
  subgraph TRAIN["Training"]
    D["generate_churn_data"]
    T["train_churn_model"]
    D --> T
  end

  subgraph INFER["Inference"]
    P["predict_churn"]
  end

  U["Customer Features<br/>(curl / SDK)"] --> INFER

  subgraph STACK["Stack"]
    OR[("Orchestrator")]
    AR[("Artifact Store")]
    DE[("Deployer")]
  end

  TRAIN --> AR
  TRAIN --> OR
  INFER --> DE
  INFER --> AR

View Quick Start & Examples

Quick start Copy Section

git clone --depth 1 https://github.com/zenml-io/zenml.git
cd zenml/examples/deploying_ml_model
uv pip install -r requirements.txt

Then follow the guide in examples/deploying_ml_model:

1. Build your pipeline: Data loading → preprocessing → training → evaluation
2. Deploy the model: Serve your trained model as a real-time HTTP endpoint
3. Monitor performance: Track predictions, latency, and data drift in the dashboard
4. Iterate: Retrain and redeploy without code changes—just switch your orchestrator

Example output Copy Section

• Predictive models (regression, classification)
• Time series forecasting
• NLP pipelines (sentiment analysis, text classification)
• Computer vision workflows
• Model scoring and ranking systems

Related examples Copy Section

• e2e: End-to-end ML pipeline with data validation and model deployment
• e2e_nlp: Domain-specific NLP pipeline example
• mlops_starter: Production-ready MLOps setup with monitoring and governance

Path 3: Build Hybrid Systems Copy Chapter

Combine classical ML models and AI agents in a single pipeline. For example, use a classifier to route requests to specialized agents, or use agents to augment ML predictions.

Architecture example Copy Section

---
config:
  layout: elk
  theme: mc
---
flowchart TB
  U["Customer Input<br/>(curl / SDK)"] --> SA["Agent Service"]

  subgraph TRAIN["Training"]
    D["load_data"]
    T["train_classifier"]
    D --> T
  end

  subgraph SERVE["Serving"]
    C["classify_intent"]
    R["generate_response"]
    C --> R
  end

  SA --> SERVE

  subgraph STACK["Stack"]
    OR[("Orchestrator")]
    AR[("Artifact Store")]
    DE[("Deployer")]
  end

  TRAIN --> AR
  TRAIN --> OR
  SERVE --> AR
  SERVE --> DE

View Quick Start & Examples

Quick start Copy Section

git clone --depth 1 https://github.com/zenml-io/zenml.git
cd zenml/examples/agent_outer_loop
uv pip install -r requirements.txt

Then follow the guide in examples/agent_outer_loop:

1. Define both components: Classical ML classifier + AI agent steps
2. Wire them together: Use the classifier output to influence agent behavior
3. Deploy as one service: The entire hybrid system becomes a single endpoint
4. Monitor both: Track ML metrics and agent traces in the same dashboard