---
ZenML is built for ML or AI Engineers working on traditional ML use-cases, LLM workflows, or agents, in a company setting.
At it's core, ZenML allows you to write **workflows (pipelines)** that run on any **infrastructure backend (stacks)**. You can embed any Pythonic logic within these pipelines, like training a model, or running an agentic loop. ZenML then operationalizes your application by:
1. Automatically containerizing and tracking your code.
2. Tracking individual runs with metrics, logs, and metadata.
3. Abstracting away infrastructure complexity.
4. Integrating your existing tools and infrastructure e.g. MLflow, Langgraph, Langfuse, Sagemaker, GCP Vertex, etc.
5. Allowing you to quickly iterate on experiments with an observable layer, in development and in production.
...amongst many other features.
ZenML is used by thousands of companies to run their AI workflows. Here are some featured ones:
(please email support@zenml.io if you want to be featured)
## 🚀 Get Started (5 minutes)
```bash
# Install ZenML with server capabilities
pip install "zenml[server]" # pip install zenml will install a slimmer client
# Initialize your ZenML repository
zenml init
# Start local server or connect to a remote one
zenml login
```
You can then explore any of the [examples](examples/) in this repo. We recommend starting with the [quickstart](examples/quickstart/), which demonstrates core ZenML concepts: pipelines, steps, artifacts, snapshots, and deployments.
### 🏗️ Architecture Overview
ZenML uses a [**client-server architecture**](https://docs.zenml.io/getting-started/system-architectures) with an integrated web dashboard ([zenml-io/zenml-dashboard](https://github.com/zenml-io/zenml-dashboard)):
- **Local Development**: `pip install "zenml[local]"` - runs both client and server locally
- **Production**: Deploy server separately, connect with `pip install zenml` + `zenml login `
## 🎮 Demo
Here is a short demo:
[](https://zenml.io/demo-video)
## 🖼️ Resources
The best way to learn about ZenML is through our comprehensive documentation and tutorials:
- **[Documentation](https://docs.zenml.io/)** - Complete product documentation
- **[Your First AI Pipeline](https://docs.zenml.io/getting-started/your-first-ai-pipeline)** - Build and evaluate an AI service in minutes
- **[Starter Guide](https://docs.zenml.io/user-guides/starter-guide)** - From zero to production in 30 minutes
- **[LLMOps Guide](https://docs.zenml.io/user-guides/llmops-guide)** - Specific patterns for LLM applications
- **[SDK Reference](https://sdkdocs.zenml.io/)** - Complete SDK reference
## 📚 More examples
1. **[Agent Architecture Comparison](examples/agent_comparison/)** - Compare AI agents with LangGraph workflows, LiteLLM integration, and automatic visualizations via custom materializers
2. **[Deploying ML Models](examples/deploying_ml_model/)** - Deploy classical ML models as production endpoints with monitoring and versioning
3. **[Deploying Agents](examples/deploying_agent/)** - Document analysis service with pipelines, evaluation, and embedded web UI
4. **[E2E Batch Inference](examples/e2e/)** - Complete MLOps pipeline with feature engineering
5. **[LLM RAG Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/llm-complete-guide)** - Production RAG with evaluation loops
6. **[Agentic Workflow (Deep Research)](https://github.com/zenml-io/zenml-projects/tree/main/deep_research)** - Orchestrate your agents with ZenML
7. **[Fine-tuning Pipeline](https://github.com/zenml-io/zenml-projects/tree/main/gamesense)** - Fine-tune and deploy LLMs
## 🗣️ Chat With Your Pipelines: ZenML MCP Server
Stop clicking through dashboards to understand your ML workflows. The **[ZenML MCP Server](https://github.com/zenml-io/mcp-zenml)** lets you query your pipelines, analyze runs, and trigger deployments using natural language through Claude Desktop, Cursor, or any MCP-compatible client.
```
💬 "Which pipeline runs failed this week and why?"
📊 "Show me accuracy metrics for all my customer churn models"
🚀 "Trigger the latest fraud detection pipeline with production data"
```
**Quick Setup:**
1. Download the `.dxt` file from [zenml-io/mcp-zenml](https://github.com/zenml-io/mcp-zenml)
2. Drag it into Claude Desktop settings
3. Add your ZenML server URL and API key
4. Start chatting with your ML infrastructure
The MCP (Model Context Protocol) integration transforms your ZenML metadata into conversational insights, making pipeline debugging and analysis as easy as asking a question. Perfect for teams who want to democratize access to ML operations without requiring dashboard expertise.
## 🤖 Kitaru: Durable AI Agents
Building AI agents that need to survive crashes, pause for human approval, or run on cloud infrastructure? **[Kitaru](https://kitaru.ai)** is our open-source sister project for making Python agents durable.
- **Crash recovery** — checkpoint and replay from failure, not from scratch
- **Human-in-the-loop** — built-in approval gates and wait points
- **Persistent memory** — versioned, durable state across agent runs with full audit trail
- **Framework agnostic** — works with PydanticAI, CrewAI, or raw Python
- **Runs anywhere** — local, Kubernetes, Vertex AI, SageMaker, AzureML
Built on the same infrastructure that powers ZenML. Two decorators (`@flow` + `@checkpoint`) and you're done.
```bash
pip install kitaru
```
👉 **[kitaru.ai](https://kitaru.ai)** · **[GitHub](https://github.com/zenml-io/kitaru)** · **[Docs](https://docs.zenml.io/kitaru)**
### 🎓 Books & Resources
[ZenML](https://zenml.io) is featured in these comprehensive guides to production AI systems.
## 🤝 Join ML Engineers Building the Future of AI
**Contribute:**
- 🌟 [Star us on GitHub](https://github.com/zenml-io/zenml/stargazers) - Help others discover ZenML
- 🤝 [Contributing Guide](CONTRIBUTING.md) - Start with [`good-first-issue`](https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Azenml-io+label%3A%22good+first+issue%22)
- 💻 [Write Integrations](https://github.com/zenml-io/zenml/blob/main/src/zenml/integrations/README.md) - Add your favorite tools
**Stay Updated:**
- 🗺 [Public Roadmap](https://zenml.io/roadmap) - See what's coming next
- 📰 [Blog](https://zenml.io/blog) - Best practices and case studies
- 🎙 [Slack](https://zenml.io/slack) - Talk with AI practitioners
## ❓ FAQs from ML Engineers Like You
**Q: "Do I need to rewrite my agents or models to use ZenML?"**
A: No. Wrap your existing code in a `@step`. Keep using `scikit-learn`, PyTorch, LangGraph, LlamaIndex, or raw API calls. ZenML orchestrates your tools, it doesn't replace them.
**Q: "How is this different from LangSmith/Langfuse?"**
A: They provide excellent observability for LLM applications. We orchestrate the **full MLOps lifecycle for your entire AI stack**. With ZenML, you manage both your classical ML models and your AI agents in one unified framework, from development and evaluation all the way to production deployment.
**Q: "Can I use my existing MLflow/W&B setup?"**
A: Yes! ZenML integrates with both [MLflow](https://docs.zenml.io/stacks/experiment-trackers/mlflow) and [Weights & Biases](https://docs.zenml.io/stacks/experiment-trackers/wandb). Your experiments, our pipelines.
**Q: "Is this just MLflow with extra steps?"**
A: No. MLflow tracks experiments. We orchestrate the entire development process – from training and evaluation to deployment and monitoring – for both models and agents.
**Q: "How do I configure ZenML with Kubernetes?"**
A: ZenML integrates with Kubernetes through the native Kubernetes orchestrator, Kubeflow, and other K8s-based orchestrators. See our [Kubernetes orchestrator guide](https://docs.zenml.io/stacks/orchestrators/kubernetes) and [Kubeflow guide](https://docs.zenml.io/stacks/orchestrators/kubeflow), plus [deployment documentation](https://docs.zenml.io/getting-started/deploying-zenml/deploy-with-helm).
**Q: "What about cost? I can't afford another platform."**
A: ZenML's open-source version is free forever. You likely already have the required infrastructure (like a Kubernetes cluster and object storage). We just help you make better use of it for MLOps.
### 🛠 VS Code / Cursor Extension
Manage pipelines directly from your editor:
🖥️ VS Code Extension in Action!
Install from [VS Code Marketplace](https://marketplace.visualstudio.com/items?itemName=ZenML.zenml-vscode).
## 📜 License
ZenML is distributed under the terms of the Apache License Version 2.0. See
[LICENSE](LICENSE) for details.
---
---
### AGENTS (AGENTS.md)
# ZenML Codex Agent Guidelines
ZenML is an extensible open-source MLOps framework for creating production-ready
ML pipelines. This root guide contains rules that must be loaded for every
Codex session in this repository. For detailed workflows, examples, and
subsystem recipes, use `.agents/skills/zenml-repo-workflows/SKILL.md`.
## Project Structure
- `src/zenml/` - Core source code.
- `tests/` - Unit and integration tests.
- `docs/book/` - Source documentation.
- `examples/` - Example projects.
- `scripts/` - Development utilities.
## Always-Loaded Rules
- Use US English spelling in code, comments, docstrings, and documentation.
- Use Python 3.10+ compatible code.
- Type hint function parameters and return values.
- Follow Google Python style for docstrings. Include `Args`, `Returns`,
`Yields`, and `Raises` sections whenever the function contract requires them;
do not use a summary-only docstring to omit applicable sections.
- Prefer clear names and small functions over explanatory comments.
- Comments should explain intent, trade-offs, constraints, invariants, and
tricky edge cases. Avoid comments that restate obvious code.
- Do not use multi-line banner comments to group classes or functions.
- Prefer typed contracts over `getattr`/`hasattr` capability checks when static
typing can express the requirement.
- Private methods and functions with a leading underscore should not be called
outside their class or module.
- Integrations should avoid using ZenML private methods because externalized
integrations will not be protected by in-repo type checks.
## Common Commands
- Format before committing: `bash scripts/format.sh`.
- Check quality: `bash scripts/lint.sh`.
- Run targeted tests only: `pytest tests/unit/path/to/test_file.py` or
`pytest tests/unit/path/to/test_file.py::test_specific_function`.
- Do NOT run the entire local test suite by default; many tests need special
environments.
- If you make changes after running tests, rerun the relevant tests.
## Branches, Git, and PRs
- `develop` is the primary working branch, not `main`.
- Always branch off `develop` for new work.
- PRs should target `develop`.
- `main` is only updated during releases.
- Use targeted `git add`; do not stage unrelated files.
- Never add anything from `design/` to git history.
- PR titles should be concise and human-readable, without prefixes like
`feat:`.
- Every PR must have exactly one release-notes label:
`release-notes` or `no-release-notes`.
- When pushing new commits to an open PR, check whether the PR description needs
updating.
## Security
- NEVER commit secrets, API keys, tokens, passwords, or credentials.
- Use environment variables or ZenML secret management for sensitive data.
- Validate and sanitize user inputs.
- If secrets are accidentally committed, notify the team immediately.
## Architecture and API Safety
- Check whether a class or function is exported in `zenml.__init__` before
changing a public-looking interface.
- Public APIs need backward-compatible evolution or deprecation.
- For internal non-underscore methods, search for usages and update all
internal callers.
- The term "model" can mean Pydantic models, machine learning models, or ZenML
model namespaces. Be explicit when writing or reviewing.
- When changing cross-cutting features, trace the full path through CLI,
client, server, models, schemas, migrations, tests, and docs.
## FastAPI and Runtime Rules
- ZenML OSS FastAPI work expects FastAPI, SQLModel, SQLAlchemy 2.0, and
Pydantic v2 patterns.
- Implement synchronous `def` route handlers for OSS Codex contributions.
- Keep shared state inside FastAPI dependency injection or the application
factory; never introduce fresh global variables outside initialization.
- Start routes, dependencies, and services with guard clauses.
- Raise `HTTPException` with precise status codes for expected errors.
- Use Pydantic models for route inputs and outputs.
- When changing server framework (e.g., `fastapi`) or database library versions,
check whether related OpenTelemetry SDK, exporter, and instrumentation
dependencies also need updates. Breaking changes in instrumented libraries can
require coordinated OTel updates. Keep OTel SDK/exporter versions aligned with
the matching OpenTelemetry instrumentation beta line.
- Code outside `src/zenml/zen_server/` should NEVER import from `zen_server/`.
## Database and Storage Rules
- ZenML uses SQLModel and SQLAlchemy; no raw SQL unless absolutely necessary.
- Database schema changes require Alembic migrations.
- Never modify existing migrations that are already on `main` or `develop`.
- Always consider backward compatibility for rolling deployments.
- Test migration upgrade paths with `alembic upgrade head` for meaningful
schema changes.
- Code outside `zen_stores/` should not import SQL-related code directly from
`zen_stores`; use `Client` or, rarely, `client.zen_store`.
## Subsystem Pointers
Load `.agents/skills/zenml-repo-workflows/SKILL.md` for detailed guidance on
tests, PRs, migrations, docs, integrations, models, orchestrators, FastAPI,
server code, and storage.
Directory-specific reminders:
- `docs/book/AGENTS.md` - documentation source, GitBook links, and docs checks.
- `src/zenml/cli/AGENTS.md` - CLI import rules and filter/client coupling.
- `src/zenml/integrations/AGENTS.md` - integration flavor import rules.
- `src/zenml/models/AGENTS.md` - domain model compatibility and filter fields.
- `src/zenml/orchestrators/AGENTS.md` - orchestrator IDs and dynamic pipelines.
- `src/zenml/zen_server/AGENTS.md` - server import boundary and endpoint shape.
- `src/zenml/zen_stores/migrations/AGENTS.md` - Alembic migration guidance.
- `src/zenml/zen_stores/schemas/AGENTS.md` - ORM schema and SQL import rules.
## Reviewer Checklist
- Integration PRs: no top-level integration-library imports in flavor files.
- Orchestrator PRs: `get_orchestrator_run_id` is unique per run and stable for
all steps in that run.
- Filter model changes: matching client method signature and body are updated.
- Private method changes: all internal usages are updated.
- Import checks: no `zen_server` imports outside `zen_server`.
- Import checks: no direct SQL imports outside `zen_stores`.
- Model changes: adding properties is usually OK; deleting, renaming, or
incompatible type changes are risky.
- Dependency bumps: dropping old version support is breaking.
- Scheduling changes: check both legacy schedule and trigger stacks.
- Step operator changes: check `BaseStepOperator`, `StepLauncher`, and at least
one concrete integration.
## Documentation Rules
- Documentation source files live in `docs/book/`.
- Do not edit generated docs directories such as `docs/mkdocs/` or
`docs/site/`.
- When adding or removing docs pages, update the relevant `toc.md`.
- Assets belong in a `.gitbook` folder beside the relevant `toc.md`.
For human contributors, see `CONTRIBUTING.md`.
---
### Changelog.Json (changelog.json)
[
{
"id": 4,
"slug": "dynamic-pipelines",
"title": "Dynamic pipelines are now available",
"description": "Introduced Dynamic Pipelines as an experimental feature, allowing you to generate DAG structures at runtime using native Python control flow (loops, conditionals). Key capabilities include dynamic parallelization, Map/Reduce patterns over collections, and granular runtime configuration (inline vs. isolated). Supported on local, Kubernetes, AWS Sagemaker, and Google Cloud Vertex orchestrators. Dynamic pipelines can be run from snapshots with configurable parameters, and include improvements to Kubernetes orchestrator handling and step mapping operations that return future objects with an `unpack()` method for better control flow.",
"published_at": "2025-12-05T06:42:19Z",
"published": true,
"audience": "oss",
"labels": [
"feature"
],
"docs_url": "https://docs.zenml.io/concepts/steps_and_pipelines/dynamic_pipelines",
"should_highlight": true
},
{
"id": 3,
"slug": "panels-are-now-resizable",
"title": "Panels are now resizable",
"feature_image_url": "https://public-flavor-logos.s3.eu-central-1.amazonaws.com/whats_new/resizable-panels.gif",
"description": "You can now resize the panels in the dashboard to see more or less content at once. This is useful for when you want to see more or less information about a run or a pipeline.",
"published": true,
"published_at": "2025-10-23T00:00:00Z",
"should_highlight": false,
"audience": "all",
"labels": [
"feature"
]
},
{
"id": 2,
"slug": "introducing-pipeline-deployments",
"title": "Introducing Pipeline Deployments",
"description": "Pipeline Deployments turn pipelines into persistent HTTP services with warm state, reducing cold start latency by 10-100x while maintaining full traceability. Learn more in our blog post.",
"video_url": "https://www.youtube-nocookie.com/embed/whQytRE7kC8",
"learn_more_url": "https://www.zenml.io/blog/why-pipelines-are-the-right-abstraction-for-real-time-ai-agents-included",
"published": true,
"published_at": "2025-10-02T00:00:00Z",
"should_highlight": false,
"audience": "all",
"labels": [
"feature"
]
},
{
"id": 1,
"slug": "new-timeline-view-for-runs",
"title": "New Timeline View for Runs",
"description": "We've added a new timeline view for runs to help you visualize the execution of your pipelines.",
"feature_image_url": "https://public-flavor-logos.s3.eu-central-1.amazonaws.com/whats_new/new_feature_timeline.png",
"docs_url": "https://docs.zenml.io/concepts/dashboard-features#timeline-view",
"published": true,
"published_at": "2025-09-12T00:00:00Z",
"should_highlight": false,
"audience": "all",
"labels": [
"feature"
]
}
]
---
### CLA (CLA.md)
# Fiduciary License Agreement 2.0
based on the
## Individual Contributor Exclusive License Agreement
(including the Traditional Patent License OPTION)
Thank you for your interest in contributing to ZenML by ZenML GmbH ("We" or "Us").
The purpose of this contributor agreement ("Agreement") is to clarify and document the rights granted by contributors to Us. To make this document effective, please follow the instructions at https://zenml.io/cla/.
### 0. Preamble
Software is deeply embedded in all aspects of our lives and it is important that it empower, rather than restrict us. Free Software gives everybody the rights to use, understand, adapt and share software. These rights help support other fundamental freedoms like freedom of speech, press and privacy.
Development of Free Software can follow many patterns. In some cases whole development is handled by a sole programmer or a small group of people. But usually, the creation and maintenance of software is a complex process that requires the contribution of many individuals. This also affects who owns the rights to the software. In the latter case, rights in software are owned jointly by a great number of individuals.
To tackle this issue some projects require a full copyright assignment to be signed by all contributors. The problem with such assignments is that they often lack checks and balances that would protect the contributors from potential abuse of power from the new copyright holder.
FSFE’s Fiduciary License Agreement (FLA) was created by the Free Software Foundation Europe e.V. with just that in mind – to concentrate all deciding power within one entity and prevent fragmentation of rights on one hand, while on the other preventing that single entity from abusing its power. The main aim is to ensure that the software covered under the FLA will forever remain Free Software.
This process only serves for the transfer of economic rights. So-called moral rights (e.g. authors right to be identified as author) remain with the original author(s) and are inalienable.
How to use this FLA
If You are an employee and have created the Contribution as part of your employment, You need to have Your employer approve this Agreement or sign the Entity version of this document. If You do not own the Copyright in the entire work of authorship, any other author of the Contribution should also sign this – in any event, please contact Us at support@zenml.io
### 1. Definitions
"You" means the individual Copyright owner who Submits a Contribution to Us.
"Contribution" means any original work of authorship, including any original modifications or additions to an existing work of authorship, Submitted by You to Us, in which You own the Copyright.
"Copyright" means all rights protecting works of authorship, including copyright, moral and neighboring rights, as appropriate, for the full term of their existence.
"Material" means the software or documentation made available by Us to third parties. When this Agreement covers more than one software project, the Material means the software or documentation to which the Contribution was Submitted. After You Submit the Contribution, it may be included in the Material.
"Submit" means any act by which a Contribution is transferred to Us by You by means of tangible or intangible media, including but not limited to electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, Us, but excluding any transfer that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
"Documentation" means any non-software portion of a Contribution.
### 2. License grant
#### 2.1 Copyright license to Us
Subject to the terms and conditions of this Agreement, You hereby grant to Us a worldwide, royalty-free, exclusive, perpetual and irrevocable (except as stated in Section 8.2) license, with the right to transfer an unlimited number of non-exclusive licenses or to grant sublicenses to third parties, under the Copyright covering the Contribution to use the Contribution by all means, including, but not limited to:
publish the Contribution,
modify the Contribution,
prepare derivative works based upon or containing the Contribution and/or to combine the Contribution with other Materials,
reproduce the Contribution in original or modified form,
distribute, to make the Contribution available to the public, display and publicly perform the Contribution in original or modified form.
#### 2.2 Moral rights
Moral Rights remain unaffected to the extent they are recognized and not waivable by applicable law. Notwithstanding, You may add your name to the attribution mechanism customary used in the Materials you Contribute to, such as the header of the source code files of Your Contribution, and We will respect this attribution when using Your Contribution.
#### 2.3 Copyright license back to You
Upon such grant of rights to Us, We immediately grant to You a worldwide, royalty-free, non-exclusive, perpetual and irrevocable license, with the right to transfer an unlimited number of non-exclusive licenses or to grant sublicenses to third parties, under the Copyright covering the Contribution to use the Contribution by all means, including, but not limited to:
publish the Contribution,
modify the Contribution,
prepare derivative works based upon or containing the Contribution and/or to combine the Contribution with other Materials,
reproduce the Contribution in original or modified form,
distribute, to make the Contribution available to the public, display and publicly perform the Contribution in original or modified form.
This license back is limited to the Contribution and does not provide any rights to the Material.
### 3. Patents
#### 3.1 Patent license
Subject to the terms and conditions of this Agreement You hereby grant to Us and to recipients of Materials distributed by Us a worldwide, royalty-free, non-exclusive, perpetual and irrevocable (except as stated in Section 3.2) patent license, with the right to transfer an unlimited number of non-exclusive licenses or to grant sublicenses to third parties, to make, have made, use, sell, offer for sale, import and otherwise transfer the Contribution and the Contribution in combination with any Material (and portions of such combination). This license applies to all patents owned or controlled by You, whether already acquired or hereafter acquired, that would be infringed by making, having made, using, selling, offering for sale, importing or otherwise transferring of Your Contribution(s) alone or by combination of Your Contribution(s) with any Material.
#### 3.2 Revocation of patent license
You reserve the right to revoke the patent license stated in section 3.1 if We make any infringement claim that is targeted at your Contribution and not asserted for a Defensive Purpose. An assertion of claims of the Patents shall be considered for a "Defensive Purpose" if the claims are asserted against an entity that has filed, maintained, threatened, or voluntarily participated in a patent infringement lawsuit against Us or any of Our licensees.
### 4. License obligations by Us
We agree to (sub)license the Contribution or any Materials containing, based on or derived from your Contribution under the terms of any licenses the Free Software Foundation classifies as Free Software License and which are approved by the Open Source Initiative as Open Source licenses.
More specifically and in strict accordance with the above paragraph, we agree to (sub)license the Contribution or any Materials containing, based on or derived from the Contribution only under the terms of the following license(s) Apache-2.0 (including any right to adopt any future version of a license if permitted).
We agree to license patents owned or controlled by You only to the extent necessary to (sub)license Your Contribution(s) and the combination of Your Contribution(s) with the Material under the terms of any licenses the Free Software Foundation classifies as Free Software licenses and which are approved by the Open Source Initiative as Open Source licenses..
### 5. Disclaimer
THE CONTRIBUTION IS PROVIDED "AS IS". MORE PARTICULARLY, ALL EXPRESS OR IMPLIED WARRANTIES INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTY OF SATISFACTORY QUALITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT ARE EXPRESSLY DISCLAIMED BY YOU TO US AND BY US TO YOU. TO THE EXTENT THAT ANY SUCH WARRANTIES CANNOT BE DISCLAIMED, SUCH WARRANTY IS LIMITED IN DURATION AND EXTENT TO THE MINIMUM PERIOD AND EXTENT PERMITTED BY LAW.
### 6. Consequential damage waiver
TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL YOU OR WE BE LIABLE FOR ANY LOSS OF PROFITS, LOSS OF ANTICIPATED SAVINGS, LOSS OF DATA, INDIRECT, SPECIAL, INCIDENTAL, CONSEQUENTIAL AND EXEMPLARY DAMAGES ARISING OUT OF THIS AGREEMENT REGARDLESS OF THE LEGAL OR EQUITABLE THEORY (CONTRACT, TORT OR OTHERWISE) UPON WHICH THE CLAIM IS BASED.
### 7. Approximation of disclaimer and damage waiver
IF THE DISCLAIMER AND DAMAGE WAIVER MENTIONED IN SECTION 5. AND SECTION 6. CANNOT BE GIVEN LEGAL EFFECT UNDER APPLICABLE LOCAL LAW, REVIEWING COURTS SHALL APPLY LOCAL LAW THAT MOST CLOSELY APPROXIMATES AN ABSOLUTE WAIVER OF ALL CIVIL OR CONTRACTUAL LIABILITY IN CONNECTION WITH THE CONTRIBUTION.
### 8. Term
#### 8.1 This Agreement shall come into effect upon Your acceptance of the terms and conditions.
#### 8.2 This Agreement shall apply for the term of the copyright and patents licensed here. However, You shall have the right to terminate the Agreement if We do not fulfill the obligations as set forth in Section 4. Such termination must be made in writing.
#### 8.3 In the event of a termination of this Agreement Sections 5., 6., 7., 8., and 9. shall survive such termination and shall remain in full force thereafter. For the avoidance of doubt, Free and Open Source Software (sub)licenses that have already been granted for Contributions at the date of the termination shall remain in full force after the termination of this Agreement.
### 9. Miscellaneous
#### 9.1 This Agreement and all disputes, claims, actions, suits or other proceedings arising out of this agreement or relating in any way to it shall be governed by the laws of Germany excluding its private international law provisions.
#### 9.2 This Agreement sets out the entire agreement between You and Us for Your Contributions to Us and overrides all other agreements or understandings.
#### 9.3 In case of Your death, this agreement shall continue with Your heirs. In case of more than one heir, all heirs must exercise their rights through a commonly authorized person.
#### 9.4 If any provision of this Agreement is found void and unenforceable, such provision will be replaced to the extent possible with a provision that comes closest to the meaning of the original provision and that is enforceable. The terms and conditions set forth in this Agreement shall apply notwithstanding any failure of essential purpose of this Agreement or any limited remedy to the maximum extent possible under law.
#### 9.5 You agree to notify Us of any facts or circumstances of which you become aware that would make this Agreement inaccurate in any respect.
**You**
Date:_______________________________
Name:_______________________________
Title:______________________________
Address:____________________________
**Us**
Date:_______________________________
Name:_______________________________
Title:_______________________________
Address:_______________________________
---
### CLAUDE (CLAUDE.md)
# ZenML Claude Code Guidelines
This document provides guidance for Claude Code when working with the ZenML codebase. ZenML is an extensible, open-source MLOps framework for creating production-ready ML pipelines.
## Project Structure
- `/src/zenml/` - Core source code
- `/tests/` - Test suite (unit, integration)
- `/docs/` - Documentation
- `/examples/` - Example projects
- `/scripts/` - Development utilities
Use filesystem navigation tools to explore the codebase structure as needed.
## Use ZenML Docs via MCP
Claude Code can query ZenML documentation via the built-in GitBook MCP server: https://docs.zenml.io/~gitbook/mcp. This enables real-time, source-of-truth lookups from the docs while you code, reducing hallucinations and speeding up feature discovery.
Quick setup (CLI):
```bash
claude mcp add zenmldocs --transport http https://docs.zenml.io/~gitbook/mcp
```
Note: The MCP server indexes the latest released docs, not the develop branch. For full setup details and editor alternatives, see docs/book/reference/llms-txt.md.
## Code Style & Quality Standards
- **Use US English spelling** in all code, comments, docstrings, and documentation (e.g., "initialize", "stabilize", "color"). The CI enforces this via `typos` (configured in `.typos.toml`).
### Commenting policy — explain why, not what
- Use comments to document intent, trade‑offs, constraints, invariants, and tricky edge cases—i.e., why the code is this way—rather than narrating changes. Prefer self‑explanatory code; add comments only where extra context is needed. Write for a reader 6+ months later.
- Use for: complex logic/algorithms, non‑obvious design decisions, business rules/constraints, API purpose/contracts, edge cases.
- Avoid: change‑tracking comments ("Updated from previous version", "New implementation", "Changed to use X instead of Y", "Refactored this section").
- Avoid simple explanatory comments, where it is already clear from the code itself.
- Avoid useless one-line comments interleaved with code that merely narrate the implementation. Favor expressive names and small, focused functions.
```python
# Bad
x = x + 1 # increment x
# Good
count += 1
```
- Do not use multi-line banner comments to group classes/functions. Use a concise module-level docstring or split code into dedicated modules.
```python
# Bad
"""
===== Dataset Loaders =====
"""
class CSVLoader: ...
class ParquetLoader: ...
# Good (module-level docstring at top)
"""
Dataset loaders used by data ingestion (CSV, Parquet).
"""
class CSVLoader: ...
class ParquetLoader: ...
```
### Formatting and Linting
- Format code with: `bash scripts/format.sh` (requires Python environment with dev dependencies)
- Run this before every commit to ensure proper formatting
- Automatically fixes and formats code using ruff and yamlfix
- Check code quality with: `bash scripts/lint.sh`
- Unlike format.sh, this doesn't auto-fix issues
- Runs Ruff, pydoclint (on `src/zenml tests/harness`), yamlfix, zizmor, and mypy
- Note: Full mypy check is slow on the entire codebase
- For faster checks, run mypy directly on specific files: `mypy src/zenml/path/to/file.py`
- The primary code style is enforced by ruff, configured in `pyproject.toml`
- YAML formatting uses yamlfix: `yamlfix .github -v`
### Python Standards
- Use Python 3.10+ compatible code
- Follow Google Python style for docstrings. Include `Args`, `Returns`,
`Yields`, and `Raises` sections whenever the function contract requires them;
do not use a summary-only docstring to omit applicable sections.
- Type hint all function parameters and return values
- Use descriptive variable names and documentation
- Keep function size manageable (aim for < 50 lines) though there are exceptions
#### Prefer typing over dynamic attribute checks
- Don't use getattr/hasattr for capability checks when static typing can express the contract
- Prefer Protocols/ABCs, Unions with isinstance narrowing, or typed adapters around untyped third-party objects
- If getattr/hasattr is unavoidable, isolate it in a small helper and expose a typed interface
Example:
```python
# Bad
if hasattr(handler, "close"):
handler.close()
# Good
from typing import Protocol
class Closable(Protocol):
def close(self) -> None: ...
def shutdown(h: Closable) -> None:
h.close()
```
### Util Function Placement
When deciding whether to place a helper function in a utils file or on a class, follow these guidelines:
1. **If a method only makes sense within the context of a class** → Put it on the class
2. **If a static/util method is heavily used by subclasses** → Put it on the parent class
**Rationale for placing methods on classes:**
- Saves imports for users and subclasses
- Subclasses can simply call `self.something()` instead of finding and importing from a util file
- Keeps related functionality co-located
**Example:** `requires_resources_in_orchestration_environment` in `base_orchestrator.py:495-514`
```python
# This is a @staticmethod on BaseOrchestrator, not a standalone util
@staticmethod
def requires_resources_in_orchestration_environment(step: "Step") -> bool:
"""Check whether a step needs special orchestration resources.
Args:
step: The step to check.
Returns:
Whether the step needs special resources in the orchestration
environment.
"""
if step.config.step_operator:
return False
return not step.config.resource_settings.empty
```
This method could be a global util, but it's placed on the class because:
- All orchestrator subclasses frequently need it
- Subclasses can call `self.requires_resources_in_orchestration_environment(step)` without imports
- It's conceptually tied to orchestrator behavior
**When to use utils files:**
- Truly generic functions used across unrelated modules
- Functions that don't logically belong to any class
- Pure utility functions (string manipulation, date formatting, etc.)
**Key utils locations:**
- `src/zenml/utils/` — General utilities
- `src/zenml/orchestrators/utils.py` — Orchestrator-specific utilities
- `src/zenml/orchestrators/step_run_utils.py` — Step execution utilities
- `src/zenml/orchestrators/publish_utils.py` — Status/metadata publishing
### Private Methods and API Stability
Methods and functions starting with `_` (underscore) are **private** and should NOT be called from outside their class or module.
**The rule:**
- `_method()` on a class → only call from within that class
- `_function()` in a utils module → only call from within that module
- This isn't always consistently applied in the codebase, but it's the intended convention
**Backwards compatibility — case-by-case judgment:**
There are no strict written rules; evaluate each change individually:
| Symbol type | Part of public API? | Breaking change if modified? |
|-------------|---------------------|------------------------------|
| Classes/functions exported in `zenml.__init__` | ✅ Definitely public | ⚠️ Yes — requires deprecation |
| Public methods on those classes | ✅ Public | ⚠️ Yes — requires deprecation |
| Internal methods deep in the codebase (no underscore) | ❌ Not intended for users | ✅ No — update all internal usages |
| `_private_method()` | ❌ No | ✅ No — can change freely |
**When changing any non-underscore method:**
1. Check if the class/function is exported in `zenml.__init__` — if so, it's public API
2. Search for usages **within the ZenML codebase** (grep/find references)
3. Update all internal usages
4. For truly internal code not exported at the root, no deprecation needed
**Best practice for integrations (future-proofing):**
> ⚠️ **Integrations should avoid using ZenML private methods**
This is primarily a future concern: when integrations eventually move out of the main ZenML repo (external packages), mypy won't detect if a private method they depend on was changed, leading to silent breakage. Even while integrations live in-repo, using only public APIs is good practice and prepares for this transition.
```python
# Bad - integration code using private method
from zenml.orchestrators.base_orchestrator import BaseOrchestrator
class MyOrchestrator(BaseOrchestrator):
def submit_pipeline(self, ...):
self._some_private_helper() # ❌ Don't do this
# Good - use only public methods or reimplement logic
class MyOrchestrator(BaseOrchestrator):
def submit_pipeline(self, ...):
self.public_method() # ✅ Safe
```
### FastAPI Conventions
Router, service, error-handling, and validation conventions for the server live in `src/zenml/zen_server/AGENTS.md` and load automatically when you work in that directory.
### Testing Requirements
- Most new code requires test coverage
- Key exceptions are when the code involves integrations with external
services. (in those cases we generally test things extensively locally and
in the CI. So the developer might have to run things or set things up
locally first.)
- Tests live in the `/tests/` folder with structure loosely mirroring the main codebase
- Unit tests go in `/tests/unit/`
- Integration tests go in `/tests/integration/`
#### Running Tests
- Do NOT try to run the entire test suite locally - many tests require special environments
- Run targeted tests for the specific components you've changed:
- `pytest tests/unit/path/to/test_file.py`
- `pytest tests/unit/path/to/test_file.py::test_specific_function`
- For full coverage, use CI (see CI section below)
- Some tests use: `bash scripts/test-coverage-xml.sh` (but this won't run all tests)
## Dependencies & Runtime Constraints
- Align contributions with the FastAPI + Pydantic v2 + SQLAlchemy 2.0 + SQLModel stack defined for ZenML OSS; confirm any new dependency in `pyproject.toml` before adoption.
- When changing server framework (e.g., `fastapi`) or database library versions, check whether related OpenTelemetry SDK, exporter, and instrumentation dependencies also need updates. Breaking changes in instrumented libraries can require coordinated OTel updates. Keep OTel SDK/exporter versions aligned with the matching OpenTelemetry instrumentation beta line.
- The OSS runtime forbids async I/O in Claude-authored code even though FastAPI supports it—implement synchronous `def` handlers and delegate background/long-running work to workers or dependency-injected services; this supersedes generic async advice found elsewhere.
- Prefer dependency injection over module-level singletons for clients, caches, and repositories so state management stays testable.
- Cache static or frequently accessed data (e.g., dependency-scoped in-memory caches) and lazy-load heavyweight resources to control cold-start latency.
- Document minimum supported versions when modifying dependency-heavy paths and explain performance trade-offs in PRs when serialization or caching strategies change.
## Development Workflow
### Prerequisites
- Set up a Python environment with ZenML dev dependencies
- Install ZenML in development mode: `pip install -e ".[dev]"`
- Most scripts require these dependencies to be available
- ZenML recommends using `uv` for Python package installation in local environments
- `uv` is also used in CI workflows
- It resolves dependencies more quickly and reliably than pip
- It can resolve dependency conflicts that pip sometimes struggles with or takes a long time to resolve
### Environment Variables
- Several environment variables are useful during ZenML development:
- `ZENML_LOGGING_VERBOSITY=DEBUG`: Controls logging verbosity
- `ZENML_ANALYTICS_OPT_IN=false`: Disables analytics during development
- `MLSTACKS_ANALYTICS_OPT_OUT=true`: Disables MLStacks analytics
- `AUTO_OPEN_DASHBOARD=false`: Prevents automatic dashboard opening
- `ZENML_ENABLE_RICH_TRACEBACK=false`: Disables rich traceback formatting
- `TOKENIZERS_PARALLELISM=false`: Avoids tokenizers parallelism warnings
- Always set the following environment variables:
- `ZENML_ANALYTICS_OPT_IN=false`: Disables analytics during development
- `ZENML_DEBUG=true`: Uses the development ZenML analytics server to avoid
sending analytics to the official ZenML analytics server (IMPORTANT!). This
must be set even if `ZENML_ANALYTICS_OPT_IN=true` because in a client-server
setup, the server controls the client-side analytics opt-in status.
### Branch Management
- **IMPORTANT**: `develop` is our primary working branch, NOT `main`
- Always branch off `develop` for all changes
- All PRs should target the `develop` branch
- The `main` branch is only updated during the release process
- If working on a feature branch that's already based on `develop`, you may need to branch off that feature branch for related changes
### Making Changes
1. Run `bash scripts/format.sh` before every commit
2. Run targeted tests to verify changes (see above)
3. Update documentation for user-facing changes (or ensure that nothing was broken)
4. IMPORTANT: **Before opening a PR or making a large commit**, always run `/simplify` to review changed code for reuse opportunities, quality issues, and efficiency improvements. Fix any issues it finds before committing.
### Security Guidelines
- **NEVER** commit secrets, API keys, tokens, or passwords
- Use environment variables or ZenML's secret management for sensitive data
- Review changes for accidental credential exposure before committing
- If you accidentally commit secrets, notify the team immediately
- Follow the principle of least privilege when implementing access controls
- Validate and sanitize all user inputs
### Database and Migration Guidelines
- Database schema changes require Alembic migrations
- Create migrations with descriptive names: `alembic revision -m "Add X to Y table"`
- Test upgrade path: `alembic upgrade head` (downgrade testing is optional—ZenML doesn't support downgrades in most cases)
- Never modify existing migrations that are already on main/develop branches
- Always consider backward compatibility for rolling deployments
- Include both schema changes and data migrations when needed
- Run `scripts/check-alembic-branches.sh` to verify migration consistency
### Commit Message Guidelines
- Write clear, descriptive commit messages explaining the "why" not just the "what"
- First line should be a concise summary (50 chars or less)
- Use imperative mood: "Add feature" not "Added feature"
- Reference issue numbers when applicable: "Fix user auth bug (#1234)"
- For multi-line messages, add a blank line after the summary
- Example:
```
Add retry logic to artifact upload
Previously, artifact uploads would fail immediately on network errors.
This adds exponential backoff retry logic to handle transient failures.
Fixes #1234
```
### When Implementing Features
- Study existing similar implementations first
- Follow the established patterns in the codebase
- Keep backward compatibility in mind
- Add appropriate error handling
- Document public APIs thoroughly
### Field Description Standards
Templates, the quality bar, and worked examples for Pydantic `Field(description=...)` text in stack component configs live in the `field-descriptions` skill. All descriptions must pass `python scripts/validate_descriptions.py` before merging.
### When Fixing Bugs
- Add regression tests that would have caught the bug
- Understand root cause before implementing fix
- Document the fix in commit messages
### Pull Request Guidelines
- Use human-readable names for PRs (no prefixes like "feat:" or "doc:")
- Keep PR titles concise but descriptive
- Write comprehensive PR descriptions:
- Clearly explain what the changes do
- Mention why the changes are needed
- Detail any important implementation decisions
- Note any areas that need special reviewer attention
- Detailed PR descriptions help both reviewers and release note creation
- Use appropriate PR tags where applicable:
- `internal`: For changes relevant only to ZenML team members
- `documentation`: For changes related to documentation
- `bug`: For bug fixes
- `dependencies`: For dependency updates
- `enhancement`: For new features or improvements
- **REQUIRED: Release Notes Labels** - Every PR must have exactly one of these labels:
- `release-notes`: For user-facing features, significant updates, or changes that should appear in the changelog. Use this for new features, important bug fixes affecting users, API changes, or anything users should know about.
- `no-release-notes`: For internal changes, CI fixes, refactoring, minor bug fixes, documentation-only changes, or anything that doesn't need to be surfaced to users.
- The CI will block merging if neither label is present. When in doubt, use `no-release-notes` for internal/maintenance work.
### Continuous Integration
- ZenML uses a two-tier CI approach:
- **Fast CI**: Runs automatically on all PRs (basic tests, linting, type checking)
- **Full CI**: Includes integration tests, tutorial pipeline regression tests, and more extensive test coverage
- The `run-slow-ci` label triggers full CI testing
- Full CI is required before merging - maintainers will add the label if needed
- Tutorial pipeline testing runs all VSCode tutorial examples against the current branch to catch breaking changes
- If your changes touch integrations or core functionality, mention in the PR that full CI should be run
- CI failures will show in the PR checks - review logs to understand any issues
## Core Concepts
### Important Terminology
- The term "model" has multiple distinct meanings in the codebase:
1. **Pydantic models**: Data structures used throughout the codebase (like `PipelineModel`)
2. **ML models**: Actual machine learning models (PyTorch, sklearn, etc.)
3. **ZenML models**: Namespaces that group artifacts, metadata, and other resources related to an ML model
- Be careful with these terms when reading/writing code to avoid confusion
### Pipeline Architecture
- Pipelines are collections of steps
- Steps produce and consume artifacts
- Artifacts are serialized/deserialized by materializers
- Pipelines are executed by orchestrators
- Stack components provide functionality like storage, orchestration, etc.
### Key Abstractions
- `StackComponent` - Base for stack components
- `Pipeline` - Pipeline definition
- `BaseStep` - Step implementation
- `BaseMaterializer` - Artifact serialization
- `BaseOrchestrator` - Pipeline execution
- `BaseStepOperator` - Remote step execution (submit/status/wait/cancel lifecycle)
## Common Tasks
### Adding New Integrations
1. Create integration package in `/src/zenml/integrations/`
2. Implement required abstractions and register flavors
3. Add tests in `/tests/integrations/`
4. Add documentation in `/docs/book/component-guide/`
### Modifying Core Functionality
1. Understand the impact on existing components
2. Maintain backward compatibility where possible
3. Add comprehensive test coverage
4. Update type hints and documentation
## Task Planning Approach
When tackling complex tasks:
1. Break down the task into smaller sub-tasks
2. Research existing implementations in the codebase
3. Plan approach before implementation
4. Test incrementally as you implement
5. Document design decisions in code comments
## Expert Tips
- ZenML follows a plugin architecture - study how components are registered
- API stability is important - don't break public interfaces
- Review similar PRs for implementation patterns
- Pipeline execution is complex - test thoroughly when modifying
- Centralize FastAPI logging, tracing, and unexpected error handling inside middleware; measure latency/throughput for new endpoints, cache static payloads, lazy-load heavyweight resources, and articulate serialization trade-offs in PR notes.
### Summary Checklist for PR Reviewers
Quick reference for common review concerns. Detailed explanations live in the nested AGENTS.md files.
- [ ] **Integration PRs:** No library imports in flavor files (`src/zenml/integrations/AGENTS.md`)
- [ ] **Orchestrator PRs:** Verify `get_orchestrator_run_id` is unique per run but same for all steps (`src/zenml/orchestrators/AGENTS.md`)
- [ ] **Filter model changes:** Check corresponding client method is updated (`src/zenml/models/AGENTS.md`)
- [ ] **Private method changes:** Check all internal usages (see "Private Methods" above)
- [ ] **Import checking:** No `zen_server` imports from outside `zen_server` (`src/zenml/zen_server/AGENTS.md`)
- [ ] **Import checking:** No direct SQL imports from outside `zen_stores` (`src/zenml/zen_stores/schemas/AGENTS.md`)
- [ ] **Model changes:** Adding properties OK, deleting/making optional is breaking (`src/zenml/models/AGENTS.md`)
- [ ] **Dependency bumps:** If dropping old version support, it's a breaking change (`src/zenml/integrations/AGENTS.md`)
- [ ] **Scheduling changes:** Must span both legacy schedule and trigger stacks (CLI + client + server + models + schemas)
- [ ] **Step operator changes:** Check `BaseStepOperator`, `StepLauncher`, and at least one concrete integration
## Documentation Guidelines
Structure, GitBook conventions, `toc.md` handling, and link checking live in `docs/book/AGENTS.md` and load automatically when you work with files under `docs/book/`.
---
*This document is maintained to help Claude Code work effectively with the
ZenML codebase. For human contributors, see CONTRIBUTING.md.*
---
### CODE OF CONDUCT (CODE-OF-CONDUCT.md)
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
[support@zenml.io](mailto:support@zenml.io).
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available
at [https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/inclusion
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
---
### CONTRIBUTING (CONTRIBUTING.md)
# 🧑💻 Contributing to ZenML
A big welcome and thank you for considering contributing to ZenML! It’s people
like you that make it a reality for users
in our community.
Reading and following these guidelines will help us make the contribution
process easy and effective for everyone
involved. It also communicates that you agree to respect the developers' time
management and develop these open-source projects. In return, we will reciprocate that respect by reading your
issue, assessing changes, and helping
you finalize your pull requests.
## ⚡️ Quicklinks
- [🧑💻 Contributing to ZenML](#-contributing-to-zenml)
- [⚡️ Quicklinks](#-quicklinks)
- [🧑⚖️ Code of Conduct](#-code-of-conduct)
- [🛫 Getting Started](#-getting-started)
- [⁉️ Issues](#-issues)
- [🏷 Pull Requests: When to make one](#-pull-requests-when-to-make-one)
- [💯 Pull Requests: Workflow to Contribute](#-pull-requests-workflow-to-contribute)
- [🧱 Pull Requests: Rebase on develop](#-pull-requests-rebase-your-branch-on-develop)
- [🧐 Linting, formatting, and tests](#-linting-formatting-and-tests)
- [🚨 Reporting a Vulnerability](#-reporting-a-vulnerability)
- [Coding Conventions](#coding-conventions)
- [👷 Creating a new Integration](#-creating-a-new-integration)
- [🆘 Getting Help](#-getting-help)
## 🧑⚖️ Code of Conduct
We take our open-source community seriously and hold ourselves and other
contributors to high standards of communication.
By participating and contributing to this project, you agree to uphold
our [Code of Conduct](https://github.com/zenml-io/zenml/blob/master/CODE-OF-CONDUCT.md)
.
## 🛫 Getting Started
Contributions are made to this repo via Issues and Pull Requests (PRs). A few
general guidelines that cover both:
- To report security vulnerabilities, please get in touch
at [support@zenml.io](mailto:support@zenml.io), monitored by
our security team.
- Search for existing Issues and PRs before creating your own.
- We work hard to make sure issues are handled on time, but it could take a
while to investigate the root cause depending on the impact.
A friendly ping in the comment thread to the submitter or a contributor can help
draw attention if your issue is blocking.
### Good First Issues for New Contributors
The best way to start is to check the
[`good-first-issue`](https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Azenml-io+label%3A%22good+first+issue%22)
label on the issue board. The core team creates these issues as necessary
smaller tasks that you can work on to get deeper into ZenML internals. These
should generally require relatively simple changes, probably affecting just one
or two files which we think are ideal for people new to ZenML.
The next step after that would be to look at the
[`good-second-issue`](https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Azenml-io+label%3A%22good+second+issue%22)
label on the issue board. These are a bit more complex, might involve more
files, but should still be well-defined and achievable to people relatively new
to ZenML.
### ⁉️ Issues
Issues should be used to report problems with the library, request a new
feature, or to discuss potential changes before
a PR is created. When you create a new Issue, a template will be loaded that
will guide you through collecting and
providing the information we need to investigate.
If you find an Issue that addresses your problem, please add your own
reproduction information to the
existing issue rather than creating a new one. Adding
a [reaction](https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/)
can also help by
indicating to our maintainers that a particular issue is affecting more than
just the reporter.
### 🏷 Pull Requests: When to make one
Pull Requests (PRs) to ZenML are always welcome and can be a quick way to get your fix or
improvement slated for the next release. In
general, PRs should:
- Only fix/add the functionality in question **OR** address widespread
whitespace/style issues, not both.
- Add unit or integration tests for fixed or changed functionality (if a test
suite already exists).
- Address a single concern in the least number of changed lines as possible.
- Include documentation in the repo or in your Pull Request.
- Be accompanied by a filled-out Pull Request template (loaded automatically when
a PR is created).
For changes that address core functionality or would require breaking changes (e.g. a major release), it's best to open
an Issue to discuss your proposal first. This is not required but can save time
creating and reviewing changes.
### 💯 Pull Requests: Workflow to Contribute
Please note that development in ZenML happens off of the develop branch, not main,
which is the default branch on GitHub. Therefore, please pay particular attention to step 5 and step 9 below.
Run agents durably, replay any run with one change, and keep the version that wins on cost, latency, and quality. Start with the Agents guide, or jump to the Kitaru quickstart.
### How these docs are organized
The documentation is split into spaces — the tabs at the top of this page. Knowing what lives where saves you a lot of searching:
| Space | What you'll find there |
| --- | --- |
| **ZenML** (you are here) | The pipelines framework: installation, core concepts, deployment, and how-to guides |
| **[Kitaru](https://docs.zenml.io/kitaru)** | The agents project: quickstart, flows and checkpoints, replay and execution management, framework adapters |
| **[Learn](https://docs.zenml.io/user-guides)** | Narrative guides for both projects: Starter, Production, LLMOps, and Agents tracks, plus tutorials and best practices |
| **[Stacks](https://docs.zenml.io/stacks)** | The infrastructure components — orchestrators, artifact stores, and more — that both pipelines and agents run on |
| **[SDK reference](https://docs.zenml.io/sdk-reference)** / **[API reference](https://docs.zenml.io/api-reference)** | Client and REST API references, organized per project |
| **[Changelog](https://docs.zenml.io/changelog)** | Release notes, version by version |
### First steps
Whichever path you picked, the first steps are the same shape: install, run something real, then learn the concepts.
| | ML Pipelines (ZenML) | AI Agents (Kitaru) |
| --- | --- | --- |
| **Install** | [Installation](getting-started/installation.md) | [Installation](https://docs.zenml.io/kitaru/getting-started/installation) |
| **First run** | [Hello World](getting-started/hello-world.md) | [Quickstart](https://docs.zenml.io/kitaru/getting-started/quickstart) |
| **Concepts** | [Core Concepts](getting-started/core-concepts.md) | [Core Concepts](https://docs.zenml.io/kitaru/concepts) |
If you use AI coding tools, see [LLM tooling](reference/llms-txt.md) for ZenML's MCP server and Agent Skills (including `zenml-scoping` and `zenml-pipeline-authoring`).
### Guides
Starter Guide
Get started with ZenML fundamentals and set up your first pipeline
## Use Cases
While the ZenML Python SDK covers most workflow requirements, the Server API offers additional utility for:
* Automated scaling of operations in production environments.
* Integration with external monitoring and logging systems.
* Programmatic management and audit of resources and configurations.
* Enterprise team management and access control (Pro).
By leveraging the ZenML API, users can enhance the robustness and control of their machine learning workflows, ensuring operations are both efficient and scalable.
---
### Book/Api Docs/Toc (docs/book/api-docs/toc.md)
# Table of contents
* [Overview](README.md)
## OSS API
* [Getting Started](oss-api/oss-api/getting-started.md)
* [OSS API](oss-api/oss-api/README.md)
* [Artifacts](oss-api-docs/v1/artifacts.md)
* [Artifact versions](oss-api-docs/v1/artifact-versions/README.md)
* [Batch](oss-api-docs/v1/artifact-versions/batch.md)
* [Visualize](oss-api-docs/v1/artifact-versions/visualize.md)
* [Login](oss-api-docs/v1/login.md)
* [Logout](oss-api-docs/v1/logout.md)
* [Device authorization](oss-api-docs/v1/device-authorization.md)
* [Api token](oss-api-docs/v1/api-token.md)
* [Code repositories](oss-api-docs/v1/code-repositories.md)
* [Logs](oss-api-docs/v1/logs.md)
* [Models](oss-api-docs/v1/models/README.md)
* [Model versions](oss-api-docs/v1/models/model-versions.md)
* [Model versions](oss-api-docs/v1/model-versions/README.md)
* [Artifacts](oss-api-docs/v1/model-versions/artifacts.md)
* [Runs](oss-api-docs/v1/model-versions/runs.md)
* [Pipelines](oss-api-docs/v1/pipelines/README.md)
* [Runs](oss-api-docs/v1/pipelines/runs.md)
* [Runs](oss-api-docs/v1/runs/README.md)
* [Steps](oss-api-docs/v1/runs/steps.md)
* [Pipeline configuration](oss-api-docs/v1/runs/pipeline-configuration.md)
* [Status](oss-api-docs/v1/runs/status.md)
* [Refresh](oss-api-docs/v1/runs/refresh.md)
* [Run templates](oss-api-docs/v1/run-templates/README.md)
* [Runs](oss-api-docs/v1/run-templates/runs.md)
* [Schedules](oss-api-docs/v1/schedules.md)
* [Secrets](oss-api-docs/v1/secrets.md)
* [Info](oss-api-docs/v1/info.md)
* [Service accounts](oss-api-docs/v1/service-accounts/README.md)
* [Api keys](oss-api-docs/v1/service-accounts/api-keys/README.md)
* [Rotate](oss-api-docs/v1/service-accounts/api-keys/rotate.md)
* [Service connectors](oss-api-docs/v1/service-connectors/README.md)
* [Verify](oss-api-docs/v1/service-connectors/verify.md)
* [Client](oss-api-docs/v1/service-connectors/client.md)
* [Full stack resources](oss-api-docs/v1/service-connectors/full-stack-resources.md)
* [Services](oss-api-docs/v1/services.md)
* [Stacks](oss-api-docs/v1/stacks.md)
* [Components](oss-api-docs/v1/components.md)
* [Component types](oss-api-docs/v1/component-types.md)
* [Steps](oss-api-docs/v1/steps/README.md)
* [Step configuration](oss-api-docs/v1/steps/step-configuration.md)
* [Status](oss-api-docs/v1/steps/status.md)
* [Logs](oss-api-docs/v1/steps/logs.md)
* [Tags](oss-api-docs/v1/tags.md)
* [Users](oss-api-docs/v1/users/README.md)
* [Resource membership](oss-api-docs/v1/users/resource-membership.md)
* [Current user](oss-api-docs/v1/current-user.md)
* [OSS API Specification](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
## Pro API
* [Getting Started](pro-api/pro-api/getting-started.md)
* [Pro API](pro-api/pro-api/README.md)
* [Tenants](pro-api-docs/api-reference/tenants/README.md)
* [Deploy](pro-api-docs/api-reference/tenants/deploy.md)
* [Deactivate](pro-api-docs/api-reference/tenants/deactivate.md)
* [Members](pro-api-docs/api-reference/tenants/members.md)
* [Tenant status](pro-api-docs/api-reference/tenant-status.md)
* [Users](pro-api-docs/api-reference/users/README.md)
* [Authorize server](pro-api-docs/api-reference/users/authorize-server.md)
* [Me](pro-api-docs/api-reference/users/me.md)
* [Invitations](pro-api-docs/api-reference/invitations.md)
* [Releases](pro-api-docs/api-reference/releases.md)
* [Devices](pro-api-docs/api-reference/devices/README.md)
* [Verify](pro-api-docs/api-reference/devices/verify.md)
* [Roles](pro-api-docs/api-reference/roles/README.md)
* [Assignments](pro-api-docs/api-reference/roles/assignments.md)
* [Permissions](pro-api-docs/api-reference/permissions.md)
* [Teams](pro-api-docs/api-reference/teams/README.md)
* [Members](pro-api-docs/api-reference/teams/members.md)
* [Organizations](pro-api-docs/api-reference/organizations/README.md)
* [Trial](pro-api-docs/api-reference/organizations/trial.md)
* [Invitations](pro-api-docs/api-reference/organizations/invitations.md)
* [Members](pro-api-docs/api-reference/organizations/members.md)
* [Roles](pro-api-docs/api-reference/organizations/roles.md)
* [Teams](pro-api-docs/api-reference/organizations/teams.md)
* [Tenants](pro-api-docs/api-reference/organizations/tenants.md)
* [Tenant](pro-api-docs/api-reference/organizations/tenant.md)
* [Entitlement](pro-api-docs/api-reference/organizations/entitlement.md)
* [Validation](pro-api-docs/api-reference/organizations/validation/README.md)
* [Name](pro-api-docs/api-reference/organizations/validation/name.md)
* [Tenant name](pro-api-docs/api-reference/organizations/validation/tenant-name.md)
* [Health](pro-api-docs/api-reference/health.md)
* [Usage event](pro-api-docs/api-reference/usage-event.md)
* [Usage batch](pro-api-docs/api-reference/usage-batch.md)
* [Stigg webhook](pro-api-docs/api-reference/stigg-webhook.md)
* [Auth](pro-api-docs/api-reference/auth/README.md)
* [Login](pro-api-docs/api-reference/auth/login.md)
* [Connections](pro-api-docs/api-reference/auth/connections.md)
* [Authorize](pro-api-docs/api-reference/auth/authorize.md)
* [Callback](pro-api-docs/api-reference/auth/callback.md)
* [Logout](pro-api-docs/api-reference/auth/logout.md)
* [Device authorization](pro-api-docs/api-reference/auth/device-authorization.md)
* [Api token](pro-api-docs/api-reference/auth/api-token.md)
* [Tenant authorization](pro-api-docs/api-reference/auth/tenant-authorization.md)
* [Rbac](pro-api-docs/api-reference/rbac/README.md)
* [Check permissions](pro-api-docs/api-reference/rbac/check-permissions.md)
* [Allowed resource ids](pro-api-docs/api-reference/rbac/allowed-resource-ids.md)
* [Resource members](pro-api-docs/api-reference/rbac/resource-members.md)
* [Server](pro-api-docs/api-reference/server/README.md)
* [Info](pro-api-docs/api-reference/server/info.md)
* [Pro API Specification](https://cloudapi.zenml.io/openapi.json)
---
### Book/Api Docs/Oss Api/Oss Api/README (docs/book/api-docs/oss-api/oss-api/README.md)
---
icon: github-alt
---
# OSS API
---
### Book/Api Docs/Oss Api/Oss Api/Getting Started (docs/book/api-docs/oss-api/oss-api/getting-started.md)
---
icon: person-from-portal
---
# Getting Started
The ZenML OSS server is a FastAPI application, therefore the OpenAPI-compliant docs are available at `/docs` or `/redoc` of your ZenML server:
{% hint style="info" %}
In the local case (i.e. using `zenml login --local`, the docs are available on `http://127.0.0.1:8237/docs`)
{% endhint %}
{% hint style="info" %}
**Difference between OpenAPI docs and ReDoc**
The OpenAPI docs (`/docs`) provide an interactive interface where you can try out the API endpoints directly from the browser. It is useful for testing and exploring the API functionalities.
ReDoc (`/redoc`), on the other hand, offers a more static and visually appealing documentation. It is designed for better readability and is ideal for understanding the API structure and reference.
{% endhint %}
## Accessing the ZenML OSS API
**For OSS users**: The `server_url` is the root URL of your ZenML server deployment.
If you are using the ZenML OSS server API using the methods displayed above, it is enough to be logged in to your ZenML account in the same browser session. However, in order to do this programmatically, you can use one of the methods documented in the following sections.
{% hint style="info" %}
Choosing a method:
- Humans at the CLI: use [interactive login](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-in-with-your-user-interactive).
- CI/CD and automation: use [service accounts + API keys](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account).
{% endhint %}
### Using a service account and an API key
You can use a service account's API key to authenticate to the ZenML server's REST API programmatically. This is particularly useful when you need a long-term, secure way to make authenticated HTTP requests to the ZenML API endpoints.
Start by [creating a service account and an API key](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml/connect-with-a-service-account), e.g.:
```shell
zenml service-account create myserviceaccount
```
Then, there are two methods to authenticate with the API using the API key - one is simpler but less secure, the other is secure and recommended but more complex:
{% tabs %}
{% tab title="Direct API key authentication" %}
{% hint style="warning" %}
This approach, albeit simple, is not recommended because the long-lived API key is exposed with every API request, which makes it easier to be compromised. Use it only in low-risk circumstances.
{% endhint %}
Use the API key directly to authenticate your API requests by including it in the `Authorization` header. For example, you can use the following command to check your current user:
* using curl:
```bash
curl -H "Authorization: Bearer YOUR_API_KEY" https://your-zenml-server/api/v1/current-user
```
* using wget:
```bash
wget -qO- --header="Authorization: Bearer YOUR_API_KEY" https://your-zenml-server/api/v1/current-user
```
* using python:
```python
import requests
response = requests.get(
"https://your-zenml-server/api/v1/current-user",
headers={"Authorization": f"Bearer {YOUR_API_KEY}"}
)
print(response.json())
```
{% endtab %}
{% tab title="Token exchange authentication" %}
Reduce the risk of API key exposure by periodically exchanging the API key for a short-lived API token.
1. To obtain a short-lived API token using your API key, send a POST request to the `/api/v1/login` endpoint. Here are examples using common HTTP clients:
* using curl:
```bash
curl -X POST -d "password=" https://your-zenml-server/api/v1/login
```
* using wget:
```bash
wget -qO- --post-data="password=" \
--header="Content-Type: application/x-www-form-urlencoded" \
https://your-zenml-server/api/v1/login
```
* using python:
```python
import requests
import json
response = requests.post(
"https://your-zenml-server/api/v1/login",
data={"password": ""},
headers={"Content-Type": "application/x-www-form-urlencoded"}
)
print(response.json())
```
This will return a response like this (the short-lived API token is the `access_token` field):
```json
{
"access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiI3MGJjZTg5NC1hN2VjLTRkOTYtYjE1Ny1kOTZkYWY5ZWM2M2IiLCJpc3MiOiJmMGQ5NjI1Ni04YmQyLTQxZDctOWVjZi0xMmYwM2JmYTVlMTYiLCJhdWQiOiJmMGQ5NjI1Ni04YmQyLTQxZDctOWVjZi0xMmYwM2JmYTVlMTYiLCJleHAiOjE3MTk0MDk0NjAsImFwaV9rZXlfaWQiOiIzNDkyM2U0NS0zMGFlLTRkMjctODZiZS0wZGRhNTdkMjA5MDcifQ.ByB1ngCPtBenGE6UugsWC6Blga3qPqkAiPJUSFDR-u4",
"token_type": "bearer",
"expires_in": 3600,
"refresh_token": null,
"scope": null
}
```
2. Once you have obtained a short-lived API token, you can use it to authenticate your API requests by including it in the `Authorization` header. When the short-lived API token expires, simply repeat the steps above to obtain a new one. For example, you can use the following command to check your current user:
* using curl:
```bash
curl -H "Authorization: Bearer YOUR_API_TOKEN" https://your-zenml-server/api/v1/current-user
```
* using wget:
```bash
wget -qO- --header="Authorization: Bearer YOUR_API_TOKEN" https://your-zenml-server/api/v1/current-user
```
* using python:
```python
import requests
response = requests.get(
"https://your-zenml-server/api/v1/current-user",
headers={"Authorization": f"Bearer {YOUR_API_TOKEN}"}
)
print(response.json())
```
{% endtab %}
{% endtabs %}
{% hint style="info" %}
**Important notes**
* Short-lived API tokens are scoped to the service account that created them and inherit their permissions
* Tokens are temporary and will expire after a configured duration (typically 1 hour, but it depends on how the server is configured)
* You can request a new short-lived API token at any time using the same API key
* For security reasons, you should handle short-lived API tokens carefully and never share them
* If your API key is compromised, you can rotate it using the ZenML dashboard or by running the `zenml service-account api-key rotate` command
{% endhint %}
---
### Book/Api Docs/Oss Api Docs/V1/Api Token (docs/book/api-docs/oss-api-docs/v1/api-token.md)
# Api token
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/api_token" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Artifacts (docs/book/api-docs/oss-api-docs/v1/artifacts.md)
# Artifacts
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifacts" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifacts" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifacts/{artifact_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifacts/{artifact_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifacts/{artifact_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Code Repositories (docs/book/api-docs/oss-api-docs/v1/code-repositories.md)
# Code repositories
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/code_repositories" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/code_repositories/{code_repository_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/code_repositories/{code_repository_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/code_repositories/{code_repository_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Component Types (docs/book/api-docs/oss-api-docs/v1/component-types.md)
# Component types
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/component-types" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Components (docs/book/api-docs/oss-api-docs/v1/components.md)
# Components
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/components" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/components/{component_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/components/{component_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/components/{component_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Current User (docs/book/api-docs/oss-api-docs/v1/current-user.md)
# Current user
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/current-user" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Device Authorization (docs/book/api-docs/oss-api-docs/v1/device-authorization.md)
# Device authorization
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/device_authorization" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Info (docs/book/api-docs/oss-api-docs/v1/info.md)
# Info
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/info" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Login (docs/book/api-docs/oss-api-docs/v1/login.md)
# Login
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/login" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Logout (docs/book/api-docs/oss-api-docs/v1/logout.md)
# Logout
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/logout" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Logs (docs/book/api-docs/oss-api-docs/v1/logs.md)
# Logs
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/logs/{logs_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Schedules (docs/book/api-docs/oss-api-docs/v1/schedules.md)
# Schedules
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/schedules" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/schedules/{schedule_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/schedules/{schedule_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/schedules/{schedule_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Secrets (docs/book/api-docs/oss-api-docs/v1/secrets.md)
# Secrets
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/secrets" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/secrets/{secret_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/secrets/{secret_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/secrets/{secret_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Services (docs/book/api-docs/oss-api-docs/v1/services.md)
# Services
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/services" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/services" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/services/{service_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/services/{service_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/services/{service_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Stacks (docs/book/api-docs/oss-api-docs/v1/stacks.md)
# Stacks
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/stacks" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/stacks/{stack_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/stacks/{stack_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/stacks/{stack_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Tags (docs/book/api-docs/oss-api-docs/v1/tags.md)
# Tags
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/tags" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/tags" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/tags/{tag_name_or_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/tags/{tag_name_or_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/tags/{tag_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Artifact Versions/README (docs/book/api-docs/oss-api-docs/v1/artifact-versions/README.md)
# Artifact versions
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions/{artifact_version_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions/{artifact_version_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions/{artifact_version_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Artifact Versions/Batch (docs/book/api-docs/oss-api-docs/v1/artifact-versions/batch.md)
# Batch
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions/batch" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Artifact Versions/Visualize (docs/book/api-docs/oss-api-docs/v1/artifact-versions/visualize.md)
# Visualize
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/artifact_versions/{artifact_version_id}/visualize" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Model Versions/README (docs/book/api-docs/oss-api-docs/v1/model-versions/README.md)
# Model versions
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions/{model_version_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions/{model_version_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions/{model_version_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Model Versions/Artifacts (docs/book/api-docs/oss-api-docs/v1/model-versions/artifacts.md)
# Artifacts
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions/{model_version_id}/artifacts/{model_version_artifact_link_name_or_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions/{model_version_id}/artifacts" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Model Versions/Runs (docs/book/api-docs/oss-api-docs/v1/model-versions/runs.md)
# Runs
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/model_versions/{model_version_id}/runs/{model_version_pipeline_run_link_name_or_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Models/README (docs/book/api-docs/oss-api-docs/v1/models/README.md)
# Models
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/models" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/models/{model_name_or_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/models/{model_name_or_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/models/{model_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Models/Model Versions (docs/book/api-docs/oss-api-docs/v1/models/model-versions.md)
# Model versions
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/models/{model_name_or_id}/model_versions" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Pipelines/README (docs/book/api-docs/oss-api-docs/v1/pipelines/README.md)
# Pipelines
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/pipelines" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/pipelines/{pipeline_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/pipelines/{pipeline_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/pipelines/{pipeline_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Pipelines/Runs (docs/book/api-docs/oss-api-docs/v1/pipelines/runs.md)
# Runs
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/pipelines/{pipeline_id}/runs" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Run Templates/README (docs/book/api-docs/oss-api-docs/v1/run-templates/README.md)
# Run templates
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/run_templates" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/run_templates/{template_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/run_templates/{template_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/run_templates/{template_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Run Templates/Runs (docs/book/api-docs/oss-api-docs/v1/run-templates/runs.md)
# Runs
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/run_templates/{template_id}/runs" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Runs/README (docs/book/api-docs/oss-api-docs/v1/runs/README.md)
# Runs
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Runs/Pipeline Configuration (docs/book/api-docs/oss-api-docs/v1/runs/pipeline-configuration.md)
# Pipeline configuration
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}/pipeline-configuration" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Runs/Refresh (docs/book/api-docs/oss-api-docs/v1/runs/refresh.md)
# Refresh
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}/refresh" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Runs/Status (docs/book/api-docs/oss-api-docs/v1/runs/status.md)
# Status
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}/status" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Runs/Steps (docs/book/api-docs/oss-api-docs/v1/runs/steps.md)
# Steps
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/runs/{run_id}/steps" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Service Accounts/README (docs/book/api-docs/oss-api-docs/v1/service-accounts/README.md)
# Service accounts
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_name_or_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_name_or_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_name_or_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Service Accounts/Api Keys/README (docs/book/api-docs/oss-api-docs/v1/service-accounts/api-keys/README.md)
# Api keys
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys/{api_key_name_or_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys/{api_key_name_or_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys/{api_key_name_or_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Service Accounts/Api Keys/Rotate (docs/book/api-docs/oss-api-docs/v1/service-accounts/api-keys/rotate.md)
# Rotate
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_accounts/{service_account_id}/api_keys/{api_key_name_or_id}/rotate" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Service Connectors/README (docs/book/api-docs/oss-api-docs/v1/service-connectors/README.md)
# Service connectors
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/{connector_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/{connector_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/{connector_id}" method="delete" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Service Connectors/Client (docs/book/api-docs/oss-api-docs/v1/service-connectors/client.md)
# Client
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/{connector_id}/client" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Service Connectors/Full Stack Resources (docs/book/api-docs/oss-api-docs/v1/service-connectors/full-stack-resources.md)
# Full stack resources
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/full_stack_resources" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Service Connectors/Verify (docs/book/api-docs/oss-api-docs/v1/service-connectors/verify.md)
# Verify
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/verify" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/service_connectors/{connector_id}/verify" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Steps/README (docs/book/api-docs/oss-api-docs/v1/steps/README.md)
# Steps
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps/{step_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps/{step_id}" method="put" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Steps/Logs (docs/book/api-docs/oss-api-docs/v1/steps/logs.md)
# Logs
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps/{step_id}/logs" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Steps/Status (docs/book/api-docs/oss-api-docs/v1/steps/status.md)
# Status
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps/{step_id}/status" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Steps/Step Configuration (docs/book/api-docs/oss-api-docs/v1/steps/step-configuration.md)
# Step configuration
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/steps/{step_id}/step-configuration" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Users/README (docs/book/api-docs/oss-api-docs/v1/users/README.md)
# Users
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/users" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/users/{user_name_or_id}" method="get" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Oss Api Docs/V1/Users/Resource Membership (docs/book/api-docs/oss-api-docs/v1/users/resource-membership.md)
# Resource membership
{% openapi src="https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json" path="/api/v1/users/{user_name_or_id}/resource_membership" method="post" %}
[https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json](https://1cf18d95-zenml.cloudinfra.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api/Pro Api/README (docs/book/api-docs/pro-api/pro-api/README.md)
---
icon: rectangle-pro
---
# Pro API
---
### Book/Api Docs/Pro Api/Pro Api/Getting Started (docs/book/api-docs/pro-api/pro-api/getting-started.md)
---
icon: person-from-portal
---
# Getting Started
The ZenML Pro API extends the open-source API with additional features designed for enterprise users, including:
* Enhanced team collaboration features
* Advanced role-based access control
* Enterprise-grade security features
Whether you're using the [SaaS version](https://zenml.io/pro) or a self-hosted ZenML Pro instance, you can leverage the API to manage workspaces, organizations, users, roles, and more.
## Understanding the Two APIs in ZenML Pro
ZenML Pro includes two distinct APIs:
1. **Workspace API**: This is similar to the [OSS API](../../oss-api/oss-api/) but includes additional endpoints (for example to run snapshots). Each workspace in ZenML Pro has its own API.
2. **ZenML Pro API**: This is the management API for ZenML Pro and handles organization-level resources like workspaces, users, and roles.
### Server URL
For Pro users, the `server_url` for the Workspace API is the URL of your workspace, which can be found in the ZenML Pro dashboard:
**Note:** The workspace URL can be found in the ZenML Pro dashboard and typically looks like `https://1bfe8d94-zenml.cloudinfra.zenml.io`.
The SaaS version of ZenML Pro API is hosted at [https://cloudapi.zenml.io](https://cloudapi.zenml.io).
## API Overview
The ZenML Pro API is a RESTful API that follows OpenAPI 3.1.0 specifications. It provides endpoints for various resources and operations, including:
* Workspace management
* Organization management
* Project management
* User management
* Role-based access control (RBAC)
* Authentication and authorization
## Authentication
To use the ZenML Pro API, you need to authenticate your requests. If you are logged in to your ZenML Pro account, you can use the same browser window to authenticate requests to your ZenML Pro API, directly in the OpenAPI docs.
For example, for the SaaS variant, you can access the docs here: [https://cloudapi.zenml.io](https://cloudapi.zenml.io).
### Programmatic access with Personal Access Tokens
Personal Access Tokens are long-lived credentials scoped to your own user account that can be used to authenticate to the ZenML Pro API and the Workspace API across all the organizations and workspaces that your account has access to. See [Personal Access Tokens](https://docs.zenml.io/pro/access-management/personal-access-tokens) for setup and examples.
### Programmatic access with service accounts and API keys
Service accounts and API keys can be used to authenticate to the ZenML Pro API and the Workspace API across all workspaces in your organization. See [Service Accounts](https://docs.zenml.io/pro/access-management/service-accounts) for setup and examples.
## When to Use Which API
| Task | Use This API | Authentication Method |
|------|-------------|----------------------|
| Managing organizations, workspaces, users | **ZenML Pro API** (cloudapi.zenml.io) | Personal Access Tokens |
| Accessing the ZenML Pro API programmatically from external systems | **ZenML Pro API** (cloudapi.zenml.io) | Service Accounts + API keys (recommended for automation) |
| Running snapshots, pipeline operations | **Workspace API** (your workspace URL) | Service Accounts + API keys (recommended for automation) |
| Pipeline development, artifact management | **Workspace API** (your workspace URL) | Personal Access Tokens or Service Accounts + API Keys |
See the high‑level overview: [Connect to a server](https://docs.zenml.io/how-to/manage-zenml-server/connecting-to-zenml#choose-how-to-connect).
## Workspace API Authentication
The **Workspace API** is different from the ZenML Pro API and supports different authentication methods. This is the API you'll use for running snapshots and other pipeline operations.
### Workspace API Authentication Methods
Programmatic access to the ZenML Pro workspace API is achieved mostly the same way as the ZenML OSS server API. This is because the Workspace API in ZenML Pro is an extension of the OSS API with some additional endpoints. For automation, use ZenML Pro service accounts and API keys (see [Service Accounts](https://docs.zenml.io/pro/access-management/service-accounts)).
You can use one of these two methods to authenticate with your workspace API:
* [Create and use a Personal Access Token](https://docs.zenml.io/pro/access-management/personal-access-tokens) (good for development and short-term tasks)
* [Create a ZenML Pro service account and use its API key](https://docs.zenml.io/pro/access-management/service-accounts) (recommended for production automation and run template triggering)
When making requests to the Workspace API, make sure to use your workspace URL as the base URL. This is different from the ZenML Pro API URL (cloudapi.zenml.io), which is used for organization-level operations.
Please consult the indicated sections for more information.
## Key API Endpoints
Here are some important endpoints you can use with ZenML Pro:
### ZenML Pro API Endpoints
These endpoints are available at the ZenML Pro API (e.g., https://cloudapi.zenml.io):
#### Organization Management
* List organizations: `GET /organizations`
* Create an organization: `POST /organizations`
* Get organization details: `GET /organizations/{organization_id}`
* Update an organization: `PATCH /organizations/{organization_id}`
#### Workspace Management
* List workspaces: `GET /workspaces`
* Create a workspace: `POST /workspaces`
* Get workspace details: `GET /workspaces/{workspace_id}`
* Update a workspace: `PATCH /workspaces/{workspace_id}`
#### User Management
* List users: `GET /users`
* Get current user: `GET /users/me`
* Update user: `PATCH /users/{user_id}`
#### Role-Based Access Control
* Create a role: `POST /roles`
* Assign a role: `POST /roles/{role_id}/assignments`
* Check permissions: `GET /permissions`
Remember to refer to the complete API documentation available at [https://cloudapi.zenml.io](https://cloudapi.zenml.io) for detailed information about all available endpoints, request/response schemas, and additional features.
### Workspace API Endpoints
The Workspace API includes all OSS API endpoints plus some additional Pro-specific endpoints. These are available at your workspace URL at the `/docs` path (e.g., https://1bfe8d94-zenml.cloudinfra.zenml.io/docs):
#### Running snapshots (Pro-specific)
Snapshots allow you to start pipeline runs from external systems using HTTP requests.
* List snapshots: `GET /pipeline_snapshots`
* Get snapshot details: `GET /pipeline_snapshots/`
* Update a snapshot: `PATCH /pipeline_snapshots/`
* **Run a snapshot: `POST /pipeline_snapshots//runs`**
{% hint style="success" %}
**For Snapshot Automation**
When running snapshots from external systems or CI/CD pipelines, we strongly recommend using **service accounts with API keys**. Service accounts provide:
* Long-term credentials that don't expire hourly
* Dedicated authentication for automation (not tied to individual users)
* Consistent access even when team members leave
See the [service account documentation](../../oss-api/oss-api/#using-a-service-account-and-an-api-key) for setup instructions.
{% endhint %}
For a complete list of API endpoints available in the Workspace API, refer to the [OSS API documentation](../../oss-api/oss-api/).
## Error Handling
The API uses standard HTTP status codes to indicate the success or failure of requests. In case of errors, the response body will contain more details about the error, including a message and sometimes additional information.
## Rate Limiting
Be aware that the ZenML Pro API may have rate limiting in place to ensure fair usage. If you exceed the rate limit, you may receive a 429 (Too Many Requests) status code. Implement appropriate backoff and retry logic in your applications to handle this scenario.
## Troubleshooting Authentication
### "Which API and credentials should I use?"
Use this decision tree to determine the right approach:
| **What do you want to do?** | **Which API?** | **Authentication Method** | **Why?** |
|------------------------------|----------------|---------------------------|----------|
| Run snapshots from CI/CD or external systems | **Workspace API** (your workspace URL) | Service account + API key | Long-lived, automation-friendly, not tied to users |
| Manage organizations, create workspaces, invite users | **ZenML Pro API** (cloudapi.zenml.io) | Personal Access Tokens | Administrative operations need user-level permissions |
| Access the ZenML Pro API programmatically from external systems | **ZenML Pro API** (cloudapi.zenml.io) | Service accounts + API keys (recommended for automation) | Long-lived, automation-friendly, not tied to users |
| Develop pipelines, explore artifacts | **Workspace API** (your workspace URL) | Personal Access Tokens or Service Accounts + API keys | Flexible for development needs |
| One-off pipeline operations | **Workspace API** (your workspace URL) | Personal Access Tokens | Quick and easy for manual tasks |
### Common Issues
**"My API calls are getting 401 Unauthorized"**
- ✅ Check that you're using the correct API URL (workspace URL vs cloudapi.zenml.io)
- ✅ Verify your credentials haven't expired (see next issue)
- ✅ Ensure you're using `Bearer ` format in the Authorization header
**"My automation keeps failing after an hour"**
- ✅ Check what type of authentication you are using (Personal Access Tokens or service account API keys)
- ✅ Check the expiration date of the credentials you are using and rotate them if they have expired. Optionally adjust the expiration time during rotation.
**"I can't find the endpoint to run a snapshot"**
- ✅ Running snapshots is only possible using the **Workspace API**, not the ZenML Pro API
- ✅ Use your workspace URL + `/docs` to see the full API documentation
---
### Book/Api Docs/Pro Api Docs/Api Reference/Health (docs/book/api-docs/pro-api-docs/api-reference/health.md)
# Health
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/health" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Invitations (docs/book/api-docs/pro-api-docs/api-reference/invitations.md)
# Invitations
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/invitations/{invitation_id}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/invitations/{invitation_id}" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/invitations" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Permissions (docs/book/api-docs/pro-api-docs/api-reference/permissions.md)
# Permissions
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/permissions" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Releases (docs/book/api-docs/pro-api-docs/api-reference/releases.md)
# Releases
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/releases" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/releases/{release_service}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Stigg Webhook (docs/book/api-docs/pro-api-docs/api-reference/stigg-webhook.md)
# Stigg webhook
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/stigg-webhook" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Tenant Status (docs/book/api-docs/pro-api-docs/api-reference/tenant-status.md)
# Tenant status
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenant_status" method="patch" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Usage Batch (docs/book/api-docs/pro-api-docs/api-reference/usage-batch.md)
# Usage batch
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/usage-batch" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Usage Event (docs/book/api-docs/pro-api-docs/api-reference/usage-event.md)
# Usage event
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/usage-event" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Auth/README (docs/book/api-docs/pro-api-docs/api-reference/auth/README.md)
# Auth
---
### Book/Api Docs/Pro Api Docs/Api Reference/Auth/Api Token (docs/book/api-docs/pro-api-docs/api-reference/auth/api-token.md)
# Api token
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/auth/api_token" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Auth/Authorize (docs/book/api-docs/pro-api-docs/api-reference/auth/authorize.md)
# Authorize
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/auth/authorize" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Auth/Callback (docs/book/api-docs/pro-api-docs/api-reference/auth/callback.md)
# Callback
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/auth/callback" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Auth/Connections (docs/book/api-docs/pro-api-docs/api-reference/auth/connections.md)
# Connections
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/auth/connections" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Auth/Device Authorization (docs/book/api-docs/pro-api-docs/api-reference/auth/device-authorization.md)
# Device authorization
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/auth/device_authorization" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Auth/Login (docs/book/api-docs/pro-api-docs/api-reference/auth/login.md)
# Login
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/auth/login" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Auth/Logout (docs/book/api-docs/pro-api-docs/api-reference/auth/logout.md)
# Logout
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/auth/logout" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Auth/Tenant Authorization (docs/book/api-docs/pro-api-docs/api-reference/auth/tenant-authorization.md)
# Tenant authorization
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/auth/tenant_authorization/{tenant_id}" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Devices/README (docs/book/api-docs/pro-api-docs/api-reference/devices/README.md)
# Devices
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/devices" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/devices/{device_id_or_user_code}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/devices/{device_id}" method="put" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/devices/{device_id}" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Devices/Verify (docs/book/api-docs/pro-api-docs/api-reference/devices/verify.md)
# Verify
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/devices/verify" method="put" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/README (docs/book/api-docs/pro-api-docs/api-reference/organizations/README.md)
# Organizations
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id_or_name}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}" method="patch" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Entitlement (docs/book/api-docs/pro-api-docs/api-reference/organizations/entitlement.md)
# Entitlement
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/entitlement/{feature}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Invitations (docs/book/api-docs/pro-api-docs/api-reference/organizations/invitations.md)
# Invitations
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/invitations" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/invitations" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/invitations/{invitation_id}" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Members (docs/book/api-docs/pro-api-docs/api-reference/organizations/members.md)
# Members
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/members" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/members" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Roles (docs/book/api-docs/pro-api-docs/api-reference/organizations/roles.md)
# Roles
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/roles" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Teams (docs/book/api-docs/pro-api-docs/api-reference/organizations/teams.md)
# Teams
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/teams" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Tenant (docs/book/api-docs/pro-api-docs/api-reference/organizations/tenant.md)
# Tenant
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/tenant/{tenant_name}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Tenants (docs/book/api-docs/pro-api-docs/api-reference/organizations/tenants.md)
# Tenants
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/tenants" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Trial (docs/book/api-docs/pro-api-docs/api-reference/organizations/trial.md)
# Trial
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/trial" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Validation/README (docs/book/api-docs/pro-api-docs/api-reference/organizations/validation/README.md)
# Validation
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Validation/Name (docs/book/api-docs/pro-api-docs/api-reference/organizations/validation/name.md)
# Name
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/validation/name/{organization_name}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Organizations/Validation/Tenant Name (docs/book/api-docs/pro-api-docs/api-reference/organizations/validation/tenant-name.md)
# Tenant name
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/organizations/{organization_id}/validation/tenant_name/{tenant_name}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Rbac/README (docs/book/api-docs/pro-api-docs/api-reference/rbac/README.md)
# Rbac
---
### Book/Api Docs/Pro Api Docs/Api Reference/Rbac/Allowed Resource Ids (docs/book/api-docs/pro-api-docs/api-reference/rbac/allowed-resource-ids.md)
# Allowed resource ids
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/rbac/allowed_resource_ids" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Rbac/Check Permissions (docs/book/api-docs/pro-api-docs/api-reference/rbac/check-permissions.md)
# Check permissions
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/rbac/check_permissions" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Rbac/Resource Members (docs/book/api-docs/pro-api-docs/api-reference/rbac/resource-members.md)
# Resource members
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/rbac/resource_members" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/rbac/resource_members" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Roles/README (docs/book/api-docs/pro-api-docs/api-reference/roles/README.md)
# Roles
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles/{role_id}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles/{role_id}" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles/{role_id}" method="patch" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Roles/Assignments (docs/book/api-docs/pro-api-docs/api-reference/roles/assignments.md)
# Assignments
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles/{role_id}/assignments" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles/{role_id}/assignments" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/roles/{role_id}/assignments" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Server/README (docs/book/api-docs/pro-api-docs/api-reference/server/README.md)
# Server
---
### Book/Api Docs/Pro Api Docs/Api Reference/Server/Info (docs/book/api-docs/pro-api-docs/api-reference/server/info.md)
# Info
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/server/info" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Teams/README (docs/book/api-docs/pro-api-docs/api-reference/teams/README.md)
# Teams
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams/{team_id}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams/{team_id}" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams/{team_id}" method="patch" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Teams/Members (docs/book/api-docs/pro-api-docs/api-reference/teams/members.md)
# Members
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams/{team_id}/members" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams/{team_id}/members" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/teams/{team_id}/members" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Tenants/README (docs/book/api-docs/pro-api-docs/api-reference/tenants/README.md)
# Tenants
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id_or_name}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id}" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id}" method="patch" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Tenants/Deactivate (docs/book/api-docs/pro-api-docs/api-reference/tenants/deactivate.md)
# Deactivate
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id}/deactivate" method="patch" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Tenants/Deploy (docs/book/api-docs/pro-api-docs/api-reference/tenants/deploy.md)
# Deploy
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id}/deploy" method="patch" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Tenants/Members (docs/book/api-docs/pro-api-docs/api-reference/tenants/members.md)
# Members
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id}/members" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id}/members" method="post" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/tenants/{tenant_id}/members" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Users/README (docs/book/api-docs/pro-api-docs/api-reference/users/README.md)
# Users
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users/{user_id_or_email}" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users/{user_id}" method="delete" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users/{user_id}" method="patch" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Users/Authorize Server (docs/book/api-docs/pro-api-docs/api-reference/users/authorize-server.md)
# Authorize server
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users/authorize_server" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Api Docs/Pro Api Docs/Api Reference/Users/Me (docs/book/api-docs/pro-api-docs/api-reference/users/me.md)
# Me
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users/me" method="get" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
{% openapi src="https://cloudapi.zenml.io/openapi.json" path="/users/me" method="patch" %}
[https://cloudapi.zenml.io/openapi.json](https://cloudapi.zenml.io/openapi.json)
{% endopenapi %}
---
### Book/Component Guide/Component Guide (docs/book/component-guide/component-guide.md)
---
description: Overview of categories of MLOps components and third-party integrations.
---
# Overview
If you are new to the world of MLOps, it is often daunting to be immediately faced with a sea of tools that seemingly all promise and do the same things. It is useful in this case to try to categorize tools in various groups in order to understand their value in your toolchain in a more precise manner.
## What is a stack?
The [stack](https://docs.zenml.io/user-guides/production-guide/understand-stacks) is a fundamental component of the ZenML framework. Put simply, a stack represents the configuration of the infrastructure and tooling that defines where and how a pipeline executes.
A stack comprises different stack components, where each component is responsible for a specific task. For example, a stack might have a [container registry](https://docs.zenml.io/stacks/container-registries), a [Kubernetes cluster](https://docs.zenml.io/stacks/orchestrators/kubernetes) as an [orchestrator](https://docs.zenml.io/stacks/orchestrators), an [artifact store](https://docs.zenml.io/stacks/artifact-stores), an [experiment tracker](https://docs.zenml.io/stacks/experiment-trackers) like MLflow and so on.
Each pipeline run that you execute with ZenML will require a **stack** and each **stack** will be required to include at least an **orchestrator** and an **artifact store**. Apart from these two, the other components are optional and to be added as your pipeline evolves in MLOps maturity.
{% hint style="info" %}
Stacks are not just for ML pipelines. [Kitaru](https://docs.zenml.io/kitaru), ZenML's sibling project for running and improving AI agents, runs agent workflows on these same stacks: a [Kitaru stack](https://docs.zenml.io/kitaru/stacks) is a focused view of a ZenML stack (execution placement, artifact persistence, and container registry) built from the components documented in this section.
{% endhint %}
## Stacks as a way to organize your execution environment
With ZenML, you can run your pipelines on more than one stacks with ease. This pattern helps you test your code across different environments effortlessly.
This enables a case like this: a data scientist starts experimentation locally on their system and then once they are satisfied, move to a cloud environment on your staging cloud account to test more advanced features of your pipeline. Finally, when all looks good, they can mark the pipeline ready for production and have it run on a production-grade stack in your production cloud account.
Having separate stacks for these environments helps:
* avoid wrongfully deploying your staging pipeline to production
* curb costs by running less powerful resources in staging and testing locally first
* control access to environments by granting permissions for only certain stacks to certain users
## How to manage credentials for your stacks
Most stack components require some form of credentials to interact with the underlying infrastructure. For example, a container registry needs to be authenticated to push and pull images, a Kubernetes cluster needs to be authenticated to deploy models as a web service, and so on.
The preferred way to handle credentials in ZenML is to use [Service Connectors](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/service-connectors-guide). Service connectors are a powerful feature of ZenML that allow you to abstract away credentials and sensitive information from your team.
### Recommended roles
Ideally, you would want that only the people who deal with and have direct access to your cloud resources are the ones that are able to create Service Connectors. This is useful for a few reasons:
* **Less chance of credentials leaking**: the more people that have access to your cloud resources, the higher the chance that some of them will be leaked.
* **Instant revocation of compromised credentials**: folks who have direct access to your cloud resources can revoke the credentials instantly if they are compromised, making this a much more secure setup.
* **Easier auditing**: you can have a much easier time auditing and tracking who did what if you have a clear separation between the people who can create Service Connectors (who have direct access to your cloud resources) and those who can only use them.
### Recommended workflow
Here's an approach you can take that is a good balance between convenience and security:
* Have a limited set of people that have permissions to create Service Connectors. These are ideally people that have access to your cloud accounts and know what credentials to use.
* You can create one connector for your development or staging environment and let your data scientists use that to register their stack components.
* When you are ready to go to production, you can create another connector with permissions for your production environment and create stacks that use it. This way you can ensure that your production resources are not accidentally used for development or staging.
If you follow this approach, you can keep your data scientists free from the hassle of figuring out the best authentication mechanisms for the different cloud services, having to manage credentials locally, and keep your cloud accounts safe, while still giving them the freedom to run their experiments in the cloud.
{% hint style="info" %}
Please note that restricting permissions for users through roles is a ZenML Pro feature. You can read more about it [here](https://docs.zenml.io/pro/access-management/roles). Sign up for a free trial here: https://zenml.io/pro.
{% endhint %}
## How to deploy and manage stacks
Deploying and managing a MLOps stack is tricky.
* Each tool comes with a certain set of requirements. For example, a [Kubeflow installation](https://www.kubeflow.org/docs/started/installing-kubeflow/) will require you to have a Kubernetes cluster, and so would a **Seldon Core deployment**.
* Figuring out the defaults for infra parameters is not easy. Even if you have identified the backing infra that you need for a stack component, setting up reasonable defaults for parameters like instance size, CPU, memory, etc., needs a lot of experimentation to figure out.
* Many times, standard tool installations don't work out of the box. For example, to run a custom pipeline in [Vertex AI](https://cloud.google.com/vertex-ai), it is not enough to just run an imported pipeline. You might also need a custom service account that is configured to perform tasks like reading secrets from your secret store or talking to other GCP services that your pipeline might need.
* Some tools need an additional layer of installations to enable a more secure, production-grade setup. For example, a standard **MLflow tracking server** deployment comes without an authentication frontend which might expose all of your tracking data to the world if deployed as-is.
* All the components that you deploy must have the right permissions to be able to talk to each other. For example, your workloads running in a Kubernetes cluster might require access to the container registry or the code repository, and so on.
* Cleaning up your resources after you're done with your experiments is super important yet very challenging. For example, if your Kubernetes cluster has made use of [Load Balancers](https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer), you might still have one lying around in your account even after deleting the cluster, costing you money and frustration.
All of these points make taking your pipelines to production a more difficult task than it should be. We believe that the expertise in setting up these often-complex stacks shouldn't be a prerequisite to running your ML pipelines.
This docs section consists of information that makes it easier to provision, configure, and extend stacks and components in ZenML.
## Stack Components Guide
Here is a full list of all stack components currently supported in ZenML, with a description of the role of that component in the MLOps process:
## Custom Implementations
You can take control of how ZenML behaves by creating your own components. This is done by writing custom component `flavors`.
---
### Book/Component Guide/Integrations (docs/book/component-guide/integrations.md)
---
icon: box-archive
---
# Integrations
Categorizing the MLOps stack is a good way to write abstractions for an MLOps pipeline and standardize your processes. But ZenML goes further and also provides concrete implementations of these categories by **integrating** with various tools for each category. Once code is organized into a ZenML pipeline, you can supercharge your ML workflows with the best-in-class solutions from various MLOps areas.
For example, you can orchestrate your ML pipeline workflows using [Airflow](orchestrators/airflow.md) or [Kubeflow](orchestrators/kubeflow.md), track experiments using [MLflow Tracking](experiment-trackers/mlflow.md) or [Weights & Biases](experiment-trackers/wandb.md), and transition seamlessly from a local [MLflow deployment](model-deployers/mlflow.md) to a deployed model on Kubernetes using [Seldon Core](model-deployers/seldon.md).
There are lots of moving parts for all the MLOps tooling and infrastructure you require for ML in production and ZenML brings them all together and enables you to manage them in one place. This also allows you to delay the decision of which MLOps tool to use in your stack as you have no vendor lock-in with ZenML and can easily switch out tools as soon as your requirements change.
## Available integrations
We have a [dedicated webpage](https://zenml.io/integrations) that indexes all supported ZenML integrations and their categories.
Another easy way of seeing a list of integrations is to see the list of directories in the [integrations directory](https://github.com/zenml-io/zenml/tree/main/src/zenml/integrations) on our GitHub.
## Installing dependencies for integrations and stacks
ZenML provides a way to export the package requirements for both individual integrations and entire stacks, enabling you to install the necessary dependencies manually. This approach gives you full control over the versions and the installation process.
### Exporting integration requirements
You can export the requirements for a specific integration using the `zenml integration export-requirements` command. To write the requirements to a file and install them via pip, run:
```bash
zenml integration export-requirements --output-file integration_requirements.txt
pip install -r integration_requirements.txt
```
If you prefer to see the requirements without writing them to a file, omit the `--output-file` flag:
```bash
zenml integration export-requirements
```
This will print the list of dependencies to the console, which you can then pipe to pip:
```bash
zenml integration export-requirements | xargs pip install
```
### Exporting stack requirements
To install all dependencies for a specific ZenML stack at once, you can export your stack's requirements:
```bash
zenml stack export-requirements --output-file stack_requirements.txt
pip install -r stack_requirements.txt
```
Omitting `--output-file` will print the requirements to the console:
```bash
zenml stack export-requirements
```
You can also pipe the output directly to pip:
```bash
zenml stack export-requirements | xargs pip install
```
{% hint style="info" %}
If you use a different package manager such as [`uv`](https://github.com/astral-sh/uv), you can install the exported requirements by replacing `pip install -r …` with your package manager's equivalent command.
{% endhint %}
## Help us with integrations!
There are countless tools in the ML / MLOps field. We have made an initial prioritization of which tools to support with integrations that are visible on our public [roadmap](https://zenml.io/roadmap).
We also welcome community contributions. Check our [Contribution Guide](https://github.com/zenml-io/zenml/blob/main/CONTRIBUTING.md) and [External Integration Guide](https://github.com/zenml-io/zenml/blob/main/src/zenml/integrations/README.md) for more details on how to best contribute to new integrations.
---
### Book/Component Guide/Toc (docs/book/component-guide/toc.md)
# Table of contents
* [Overview](component-guide.md)
* [Integrations](integrations.md)
## Stack Components
* [Orchestrators](orchestrators/README.md)
* [Local Orchestrator](orchestrators/local.md)
* [Local Docker Orchestrator](orchestrators/local-docker.md)
* [Kubeflow Orchestrator](orchestrators/kubeflow.md)
* [Kubernetes Orchestrator](orchestrators/kubernetes.md)
* [Modal Orchestrator](orchestrators/modal.md)
* [Google Cloud VertexAI Orchestrator](orchestrators/vertex.md)
* [AWS Sagemaker Orchestrator](orchestrators/sagemaker.md)
* [AzureML Orchestrator](orchestrators/azureml.md)
* [Databricks Orchestrator](orchestrators/databricks.md)
* [Tekton Orchestrator](orchestrators/tekton.md)
* [Airflow Orchestrator](orchestrators/airflow.md)
* [Skypilot VM Orchestrator](orchestrators/skypilot-vm.md)
* [SSH Orchestrator](orchestrators/ssh.md)
* [HyperAI Orchestrator](orchestrators/hyperai.md)
* [Lightning AI Orchestrator](orchestrators/lightning.md)
* [Develop a custom orchestrator](orchestrators/custom.md)
* [Deployers](deployers/README.md)
* [Local Deployer](deployers/local.md)
* [Docker Deployer](deployers/docker.md)
* [Kubernetes Deployer](deployers/kubernetes.md)
* [AWS App Runner Deployer](deployers/aws-app-runner.md)
* [GCP Cloud Run Deployer](deployers/gcp-cloud-run.md)
* [Hugging Face Deployer](deployers/huggingface.md)
* [Artifact Stores](artifact-stores/README.md)
* [Local Artifact Store](artifact-stores/local.md)
* [Amazon Simple Cloud Storage (S3)](artifact-stores/s3.md)
* [Google Cloud Storage (GCS)](artifact-stores/gcp.md)
* [Azure Blob Storage](artifact-stores/azure.md)
* [Alibaba Cloud OSS](artifact-stores/alibaba-oss.md)
* [MinIO](artifact-stores/minio.md)
* [Backblaze B2](artifact-stores/b2.md)
* [DigitalOcean Spaces](artifact-stores/digitalocean.md)
* [Develop a custom artifact store](artifact-stores/custom.md)
* [Container Registries](container-registries/README.md)
* [Default Container Registry](container-registries/default.md)
* [DockerHub](container-registries/dockerhub.md)
* [Amazon Elastic Container Registry (ECR)](container-registries/aws.md)
* [Google Cloud Container Registry](container-registries/gcp.md)
* [Azure Container Registry](container-registries/azure.md)
* [GitHub Container Registry](container-registries/github.md)
* [DigitalOcean Container Registry](container-registries/digitalocean.md)
* [Develop a custom container registry](container-registries/custom.md)
* [Log Stores](log-stores/README.md)
* [Artifact Log Store](log-stores/artifact.md)
* [OpenTelemetry Log Store](log-stores/otel.md)
* [Datadog Log Store](log-stores/datadog.md)
* [Develop a Custom Log Store](log-stores/custom.md)
* [Sandboxes](sandboxes/README.md)
* [Local](sandboxes/local.md)
* [Kubernetes](sandboxes/kubernetes.md)
* [Modal](sandboxes/modal.md)
* [Step Operators](step-operators/README.md)
* [Amazon SageMaker](step-operators/sagemaker.md)
* [AzureML](step-operators/azureml.md)
* [Baseten](step-operators/baseten.md)
* [Databricks](step-operators/databricks.md)
* [Google Cloud VertexAI](step-operators/vertex.md)
* [Kubernetes](step-operators/kubernetes.md)
* [Run:AI](step-operators/runai.md)
* [Modal](step-operators/modal.md)
* [SSH](step-operators/ssh.md)
* [Spark](step-operators/spark-kubernetes.md)
* [Develop a Custom Step Operator](step-operators/custom.md)
* [Experiment Trackers](experiment-trackers/README.md)
* [Comet](experiment-trackers/comet.md)
* [MLflow](experiment-trackers/mlflow.md)
* [Neptune](experiment-trackers/neptune.md)
* [Weights & Biases](experiment-trackers/wandb.md)
* [Trackio](experiment-trackers/trackio.md)
* [Google Cloud VertexAI Experiment Tracker](experiment-trackers/vertexai.md)
* [Develop a custom experiment tracker](experiment-trackers/custom.md)
* [Image Builders](image-builders/README.md)
* [Local Image Builder](image-builders/local.md)
* [Kaniko Image Builder](image-builders/kaniko.md)
* [AWS Image Builder](image-builders/aws.md)
* [Google Cloud Image Builder](image-builders/gcp.md)
* [Develop a Custom Image Builder](image-builders/custom.md)
* [Alerters](alerters/README.md)
* [Discord Alerter](alerters/discord.md)
* [Slack Alerter](alerters/slack.md)
* [Develop a Custom Alerter](alerters/custom.md)
* [Annotators](annotators/README.md)
* [Argilla](annotators/argilla.md)
* [Label Studio](annotators/label-studio.md)
* [Pigeon](annotators/pigeon.md)
* [Prodigy](annotators/prodigy.md)
* [Develop a Custom Annotator](annotators/custom.md)
* [Data Validators](data-validators/README.md)
* [Great Expectations](data-validators/great-expectations.md)
* [Deepchecks](data-validators/deepchecks.md)
* [Evidently](data-validators/evidently.md)
* [Whylogs](data-validators/whylogs.md)
* [Develop a custom data validator](data-validators/custom.md)
* [Feature Stores](feature-stores/README.md)
* [Feast](feature-stores/feast.md)
* [Develop a Custom Feature Store](feature-stores/custom.md)
* [Model Deployers](model-deployers/README.md)
* [MLflow](model-deployers/mlflow.md)
* [Seldon](model-deployers/seldon.md)
* [BentoML](model-deployers/bentoml.md)
* [Hugging Face](model-deployers/huggingface.md)
* [Databricks](model-deployers/databricks.md)
* [vLLM](model-deployers/vllm.md)
* [Develop a Custom Model Deployer](model-deployers/custom.md)
* [Model Registries](model-registries/README.md)
* [MLflow Model Registry](model-registries/mlflow.md)
* [Develop a Custom Model Registry](model-registries/custom.md)
## Service Connectors
* [Introduction](service-connectors/auth-management.md)
* [Complete guide](service-connectors/service-connectors-guide.md)
* [Best practices](service-connectors/best-security-practices.md)
* [Connector Types](service-connectors/connector-types/README.md)
* [Docker Service Connector](service-connectors/connector-types/docker-service-connector.md)
* [Kubernetes Service Connector](service-connectors/connector-types/kubernetes-service-connector.md)
* [AWS Service Connector](service-connectors/connector-types/aws-service-connector.md)
* [GCP Service Connector](service-connectors/connector-types/gcp-service-connector.md)
* [Azure Service Connector](service-connectors/connector-types/azure-service-connector.md)
* [OAuth2 Service Connector](service-connectors/connector-types/oauth2-service-connector.md)
* [HyperAI Service Connector](service-connectors/connector-types/hyperai-service-connector.md)
## Popular Stacks
* [AWS](../how-to/popular-integrations/aws-guide.md)
* [Azure](../how-to/popular-integrations/azure-guide.md)
* [GCP](../how-to/popular-integrations/gcp-guide.md)
* [Kubernetes](../how-to/popular-integrations/kubernetes.md)
## Deployment
* [1-click Deployment](../how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack.md)
* [Terraform Modules](../how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform.md)
* [Register a cloud stack](../how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack.md)
* [Infrastructure as code](../how-to/infrastructure-deployment/infrastructure-as-code/README.md)
## Contribute
* [Custom Stack Component](contribute/custom-stack-component.md)
* [Custom Integration](../how-to/contribute-to-zenml/implement-a-custom-integration.md)
---
### Book/Component Guide/.Gitbook.Yaml (docs/book/component-guide/.gitbook.yaml)
root: .
structure:
readme: component-guide.md
summary: toc.md
redirects:
alerters: alerters/README.md
alerters/custom: alerters/custom.md
alerters/discord: alerters/discord.md
alerters/slack: alerters/slack.md
annotators: annotators/README.md
annotators/argilla: annotators/argilla.md
annotators/custom: annotators/custom.md
annotators/label-studio: annotators/label-studio.md
annotators/pigeon: annotators/pigeon.md
annotators/prodigy: annotators/prodigy.md
artifact-stores: artifact-stores/README.md
artifact-stores/azure: artifact-stores/azure.md
artifact-stores/custom: artifact-stores/custom.md
artifact-stores/gcp: artifact-stores/gcp.md
artifact-stores/local: artifact-stores/local.md
artifact-stores/s3: artifact-stores/s3.md
container-registries: container-registries/README.md
container-registries/aws: container-registries/aws.md
container-registries/azure: container-registries/azure.md
container-registries/custom: container-registries/custom.md
container-registries/default: container-registries/default.md
container-registries/dockerhub: container-registries/dockerhub.md
container-registries/gcp: container-registries/gcp.md
container-registries/github: container-registries/github.md
data-validators: data-validators/README.md
data-validators/custom: data-validators/custom.md
data-validators/deepchecks: data-validators/deepchecks.md
data-validators/evidently: data-validators/evidently.md
data-validators/great-expectations: data-validators/great-expectations.md
data-validators/whylogs: data-validators/whylogs.md
experiment-trackers: experiment-trackers/README.md
experiment-trackers/comet: experiment-trackers/comet.md
experiment-trackers/custom: experiment-trackers/custom.md
experiment-trackers/mlflow: experiment-trackers/mlflow.md
experiment-trackers/neptune: experiment-trackers/neptune.md
experiment-trackers/vertexai: experiment-trackers/vertexai.md
experiment-trackers/wandb: experiment-trackers/wandb.md
experiment-trackers/trackio: experiment-trackers/trackio.md
feature-stores: feature-stores/README.md
feature-stores/custom: feature-stores/custom.md
feature-stores/feast: feature-stores/feast.md
image-builders: image-builders/README.md
image-builders/aws: image-builders/aws.md
image-builders/custom: image-builders/custom.md
image-builders/gcp: image-builders/gcp.md
image-builders/kaniko: image-builders/kaniko.md
image-builders/local: image-builders/local.md
model-deployers: model-deployers/README.md
model-deployers/bentoml: model-deployers/bentoml.md
model-deployers/custom: model-deployers/custom.md
model-deployers/databricks: model-deployers/databricks.md
model-deployers/huggingface: model-deployers/huggingface.md
model-deployers/mlflow: model-deployers/mlflow.md
model-deployers/seldon: model-deployers/seldon.md
model-deployers/vllm: model-deployers/vllm.md
model-registries: model-registries/README.md
model-registries/custom: model-registries/custom.md
model-registries/mlflow: model-registries/mlflow.md
orchestrators: orchestrators/README.md
orchestrators/airflow: orchestrators/airflow.md
orchestrators/azureml: orchestrators/azureml.md
orchestrators/custom: orchestrators/custom.md
orchestrators/databricks: orchestrators/databricks.md
orchestrators/hyperai: orchestrators/hyperai.md
orchestrators/kubeflow: orchestrators/kubeflow.md
orchestrators/kubernetes: orchestrators/kubernetes.md
orchestrators/lightning: orchestrators/lightning.md
orchestrators/local: orchestrators/local.md
orchestrators/local-docker: orchestrators/local-docker.md
orchestrators/modal: orchestrators/modal.md
orchestrators/sagemaker: orchestrators/sagemaker.md
orchestrators/skypilot-vm: orchestrators/skypilot-vm.md
orchestrators/tekton: orchestrators/tekton.md
orchestrators/vertex: orchestrators/vertex.md
step-operators: step-operators/README.md
step-operators/azureml: step-operators/azureml.md
step-operators/custom: step-operators/custom.md
step-operators/kubernetes: step-operators/kubernetes.md
step-operators/modal: step-operators/modal.md
step-operators/sagemaker: step-operators/sagemaker.md
step-operators/spark-kubernetes: step-operators/spark-kubernetes.md
step-operators/vertex: step-operators/vertex.md
---
### Book/Component Guide/Alerters/README (docs/book/component-guide/alerters/README.md)
---
description: Sending automated alerts to chat services.
icon: message-exclamation
---
# Alerters
**Alerters** allow you to send messages to chat services (like Slack, Discord, Mattermost, etc.) from within your
pipelines. This is useful to immediately get notified when failures happen, for general monitoring/reporting, and also
for building human-in-the-loop ML.
## Alerter Flavors
Currently, the [SlackAlerter](slack.md) and [DiscordAlerter](discord.md) are the available alerter integrations. However, it is straightforward to
extend ZenML and [build an alerter for other chat services](custom.md).
| Alerter | Flavor | Integration | Notes |
|------------------------------------|-----------|-------------|--------------------------------------------------------------------|
| [Slack](slack.md) | `slack` | `slack` | Interacts with a Slack channel |
| [Discord](discord.md) | `discord` | `discord` | Interacts with a Discord channel |
| [Custom Implementation](custom.md) | _custom_ | | Extend the alerter abstraction and provide your own implementation |
{% hint style="info" %}
If you would like to see the available flavors of alerters in your terminal, you can use the following command:
```shell
zenml alerter flavor list
```
{% endhint %}
## How to use Alerters with ZenML
Each alerter integration comes with specific standard steps that you can use out of the box.
However, you first need to register an alerter component in your terminal:
```shell
zenml alerter register ...
```
Then you can add it to your stack using
```shell
zenml stack register ... -al
```
{% hint style="info" %}
Stacks can have multiple alerters attached. ZenML treats the first attached alerter as the default one, and singular accessors such as `Client().active_stack.alerter` resolve to that default alerter.
{% endhint %}
Afterward, you can import the alerter standard steps provided by the respective integration and directly use them in
your pipelines.
## Using the Ask Step for Human-in-the-Loop Workflows
All alerters provide an `ask()` method and corresponding ask steps that enable human-in-the-loop workflows. These are essential for:
- Getting approval before deploying models to production
- Confirming critical pipeline decisions
- Manual intervention points in automated workflows
### How Ask Steps Work
Ask steps (like `discord_alerter_ask_step` and `slack_alerter_ask_step`):
1. **Post a message** to your chat service with your question
2. **Wait for user response** containing specific approval or disapproval keywords
3. **Return a boolean** - `True` if approved, `False` if disapproved or timeout
```python
from zenml import step, pipeline
from zenml.integrations.slack.steps.slack_alerter_ask_step import slack_alerter_ask_step
@step
def train_model():
# Training logic here - this is a placeholder function
return "trained_model_object"
@step
def deploy_model(model, approved: bool) -> None:
if approved:
# Deploy the model to production
print("Deploying model to production...")
# deployment logic here
else:
print("Deployment cancelled by user")
@pipeline
def deployment_pipeline():
trained_model = train_model()
# Ask for human approval before deployment
approved = slack_alerter_ask_step("Deploy model to production?")
deploy_model(trained_model, approved)
```
### Default Response Keywords
By default, alerters recognize these response options:
**Approval:** `approve`, `LGTM`, `ok`, `yes`
**Disapproval:** `decline`, `disapprove`, `no`, `reject`
### Customizing Response Keywords
You can customize the approval and disapproval keywords using alerter parameters:
```python
from zenml.integrations.slack.steps.slack_alerter_ask_step import slack_alerter_ask_step
from zenml.integrations.slack.alerters.slack_alerter import SlackAlerterParameters
# Use custom approval/disapproval keywords
params = SlackAlerterParameters(
approve_msg_options=["deploy", "ship it", "✅"],
disapprove_msg_options=["stop", "cancel", "❌"]
)
approved = slack_alerter_ask_step(
"Deploy model to production?",
params=params
)
```
### Important Notes
- **Return Type**: Ask steps return a boolean value - ensure your pipeline logic handles this correctly
- **Keywords**: Response keywords are case-sensitive (except Slack, which converts to lowercase)
- **Timeout**: If no valid response is received within the timeout period, the step returns `False`
- **Permissions**: Ensure your bot has permissions to read messages in the target channel
---
### Book/Component Guide/Alerters/Custom (docs/book/component-guide/alerters/custom.md)
---
description: Learning how to develop a custom alerter.
---
# Develop a Custom Alerter
{% hint style="info" %}
Before diving into the specifics of this component type, it is beneficial to familiarize yourself with our [general guide to writing custom component flavors in ZenML](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/implement-a-custom-stack-component). This guide provides an essential understanding of ZenML's component flavor concepts.
{% endhint %}
### Base Abstraction
The base abstraction for alerters is very basic, as it only defines two abstract methods that subclasses should implement:
* `post()` takes a string, posts it to the desired chat service, and returns `True` if the operation succeeded, else `False`.
* `ask()` does the same as `post()`, but after sending the message, it waits until someone approves or rejects the operation from within the chat service (e.g., by sending "approve" / "reject" to the bot as a response). `ask()` then only returns `True` if the operation succeeded and was approved, else `False`.
The `ask()` method is particularly useful for implementing human-in-the-loop workflows. When implementing this method, you should:
- Wait for user responses containing approval keywords (like `"approve"`, `"yes"`, `"ok"`, `"LGTM"`)
- Wait for user responses containing disapproval keywords (like `"reject"`, `"no"`, `"cancel"`, `"stop"`)
- Return `True` only when explicit approval is received
- Return `False` for disapproval, timeout, or any errors
- Consider implementing configurable approval/disapproval keywords via parameters
Then base abstraction looks something like this:
```python
from abc import ABC
from typing import Optional
from zenml.stack import StackComponent
from zenml.alerter import BaseAlerterStepParameters
class BaseAlerter(StackComponent, ABC):
"""Base class for all ZenML alerters."""
def post(
self, message: str, params: Optional[BaseAlerterStepParameters]
) -> bool:
"""Post a message to a chat service."""
return True
def ask(
self, question: str, params: Optional[BaseAlerterStepParameters]
) -> bool:
"""Post a message to a chat service and wait for approval."""
return True
```
{% hint style="info" %}
This is a slimmed-down version of the base implementation. To see the full docstrings and imports, please check [the source code on GitHub](https://github.com/zenml-io/zenml/blob/main/src/zenml/alerter/base\_alerter.py).
{% endhint %}
### Building your own custom alerter
Creating your own custom alerter can be done in four steps:
1. Create a class that inherits from the `BaseAlerter` and implement the `post()` and `ask()` methods.
```
/* Detailed source-code truncated for AI context efficiency. */
```
2. If you need to configure your custom alerter, you can also implement a config object.
```python
from zenml.alerter.base_alerter import BaseAlerterConfig
class MyAlerterConfig(BaseAlerterConfig):
my_param: str
```
3. Optionally, you can create custom parameter classes to support configurable approval/disapproval keywords:
```python
from typing import List, Optional
from zenml.alerter.base_alerter import BaseAlerterStepParameters
class MyAlerterParameters(BaseAlerterStepParameters):
"""Custom parameters for MyAlerter."""
# Custom approval/disapproval message options
approve_msg_options: Optional[List[str]] = None
disapprove_msg_options: Optional[List[str]] = None
# Any other custom parameters for your alerter
custom_channel: Optional[str] = None
```
4. Finally, you can bring the implementation and the configuration together in a new flavor object.
```python
from typing import Type, TYPE_CHECKING
from zenml.alerter import BaseAlerterFlavor
if TYPE_CHECKING:
from zenml.stack import StackComponent, StackComponentConfig
class MyAlerterFlavor(BaseAlerterFlavor):
@property
def name(self) -> str:
return "my_alerter"
@property
def config_class(self) -> Type[StackComponentConfig]:
from my_alerter_config import MyAlerterConfig
return MyAlerterConfig
@property
def implementation_class(self) -> Type[StackComponent]:
from my_alerter import MyAlerter
return MyAlerter
```
Once you are done with the implementation, you can register your new flavor through the CLI. Please ensure you **point to the flavor class via dot notation**:
```shell
zenml alerter flavor register
```
For example, if your flavor class `MyAlerterFlavor` is defined in `flavors/my_flavor.py`, you'd register it by doing:
```shell
zenml alerter flavor register flavors.my_flavor.MyAlerterFlavor
```
{% hint style="warning" %}
ZenML resolves the flavor class by taking the path where you initialized zenml (via `zenml init`) as the starting point of resolution. Therefore, please ensure you follow [the best practice](https://docs.zenml.io/how-to/project-setup-and-management/setting-up-a-project-repository/set-up-repository) of initializing zenml at the root of your repository.
If ZenML does not find an initialized ZenML repository in any parent directory, it will default to the current working directory, but usually, it's better to not have to rely on this mechanism and initialize zenml at the root.
{% endhint %}
Afterward, you should see the new custom alerter flavor in the list of available alerter flavors:
```shell
zenml alerter flavor list
```
{% hint style="warning" %}
It is important to draw attention to when and how these abstractions are coming into play in a ZenML workflow.
* The **MyAlerterFlavor** class is imported and utilized upon the creation of the custom flavor through the CLI.
* The **MyAlerterConfig** class is imported when someone tries to register/update a stack component with the `my_alerter` flavor. Especially, during the registration process of the stack component, the config will be used to validate the values given by the user. As `Config` objects are inherently `pydantic` objects, you can also add your own custom validators here.
* The **MyAlerter** only comes into play when the component is ultimately in use.
The design behind this interaction lets us separate the configuration of the flavor from its implementation. This way we can register flavors and components even when the major dependencies behind their implementation are not installed in our local setting (assuming the `MyAlerterFlavor` and the `MyAlerterConfig` are implemented in a different module/path than the actual `MyAlerter`).
{% endhint %}
---
### Book/Component Guide/Alerters/Discord (docs/book/component-guide/alerters/discord.md)
---
description: Sending automated alerts to a Discord channel.
---
# Discord Alerter
The `DiscordAlerter` enables you to send messages to a dedicated Discord channel directly from within your ZenML pipelines.
The `discord` integration contains the following two standard steps:
* [discord\_alerter\_post\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord) takes a string message, posts it to a Discord channel, and returns whether the operation was successful.
* [discord\_alerter\_ask\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord) also posts a message to a Discord channel, but waits for user feedback, and only returns `True` if a user explicitly approved the operation from within Discord (e.g., by sending "approve" / "reject" to the bot in response).
Interacting with Discord from within your pipelines can be very useful in practice:
* The `discord_alerter_post_step` allows you to get notified immediately when failures happen (e.g., model performance degradation, data drift, ...),
* The `discord_alerter_ask_step` allows you to integrate a human-in-the-loop into your pipelines before executing critical steps, such as deploying new models.
## How to use it
### Requirements
Before you can use the `DiscordAlerter`, you first need to install ZenML's `discord` integration:
```shell
zenml integration install discord -y
```
{% hint style="info" %}
See the [Integrations](https://docs.zenml.io/component-guide) page for more details on ZenML integrations and how to install and use them.
{% endhint %}
### Setting Up a Discord Bot
In order to use the `DiscordAlerter`, you first need to have a Discord workspace set up with a channel that you want your pipelines to post to. This is the `` you will need when registering the discord alerter component.
Then, you need to [create a Discord App with a bot in your server](https://discordpy.readthedocs.io/en/latest/discord.html) .
{% hint style="info" %}
Note in the bot token copy step, if you don't find the copy button then click on reset token to reset the bot and you will get a new token which you can use. Also, make sure you give necessary permissions to the bot required for sending and receiving messages.
{% endhint %}
### Registering a Discord Alerter in ZenML
Next, you need to register a `discord` alerter in ZenML and link it to the bot you just created. You can do this with the following command:
```shell
zenml alerter register discord_alerter \
--flavor=discord \
--discord_token= \
--default_discord_channel_id=
```
{% hint style="info" %}
**Using Secrets for Token Management**: Instead of passing your Discord token directly, it's recommended to store it as a ZenML secret and reference it in your alerter configuration. This approach keeps sensitive information secure:
```shell
# Create a secret for your Discord token
zenml secret create discord_secret --discord_token=
# Register the alerter referencing the secret
zenml alerter register discord_alerter \
--flavor=discord \
--discord_token={{discord_secret.discord_token}} \
--default_discord_channel_id=
```
Learn more about [referencing secrets in stack component attributes and settings](https://docs.zenml.io/concepts/secrets#reference-secrets-in-stack-component-attributes-and-settings).
{% endhint %}
After you have registered the `discord_alerter`, you can add it to your stack like this:
```shell
zenml stack register ... -al discord_alerter
```
Here is where you can find the required parameters:
#### DISCORD\_CHANNEL\_ID
Open the discord server, then right-click on the text channel and click on the 'Copy Channel ID' option.
{% hint style="info" %}
If you don't see any 'Copy Channel ID' option for your channel, go to "User Settings" > "Advanced" and make sure "Developer Mode" is active.
{% endhint %}
#### DISCORD\_TOKEN
This is the Discord token of your bot. You can find the instructions on how to set up a bot, invite it to your channel, and find its token [here](https://discordpy.readthedocs.io/en/latest/discord.html).
{% hint style="warning" %}
When inviting the bot to your channel, make sure it has at least the following permissions:
* Read Messages/View Channels
* Send Messages
* Send Messages in Threads
{% endhint %}
### How to Use the Discord Alerter
After you have a `DiscordAlerter` configured in your stack, you can directly import the [discord\_alerter\_post\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord) and [discord\_alerter\_ask\_step](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord) steps and use them in your pipelines.
Since these steps expect a string message as input (which needs to be the output of another step), you typically also need to define a dedicated formatter step that takes whatever data you want to communicate and generates the string message that the alerter should post.
As an example, adding `discord_alerter_ask_step()` to your pipeline could look like this:
```python
from zenml.integrations.discord.steps.discord_alerter_ask_step import discord_alerter_ask_step
from zenml import step, pipeline
@step
def my_formatter_step(artifact_to_be_communicated) -> str:
return f"Here is my artifact {artifact_to_be_communicated}!"
@step
def process_approval_response(artifact, approved: bool) -> None:
if approved:
# Proceed with the operation
print(f"User approved! Processing {artifact}")
# Your logic here
else:
print("User disapproved. Skipping operation.")
@pipeline
def my_pipeline(...):
...
artifact_to_be_communicated = ...
message = my_formatter_step(artifact_to_be_communicated)
approved = discord_alerter_ask_step(message)
process_approval_response(artifact_to_be_communicated, approved)
if __name__ == "__main__":
my_pipeline()
```
## Using Custom Approval Keywords
You can customize which words trigger approval or disapproval by using `DiscordAlerterParameters`:
```python
from zenml.integrations.discord.steps.discord_alerter_ask_step import discord_alerter_ask_step
from zenml.integrations.discord.alerters.discord_alerter import DiscordAlerterParameters
# Custom approval/disapproval keywords
params = DiscordAlerterParameters(
approve_msg_options=["deploy", "ship it", "✅"],
disapprove_msg_options=["stop", "cancel", "❌"]
)
approved = discord_alerter_ask_step(
"Deploy model to production?",
params=params
)
```
### Default Response Keywords
By default, the Discord alerter recognizes these keywords:
**Approval:** `approve`, `LGTM`, `ok`, `yes`
**Disapproval:** `decline`, `disapprove`, `no`, `reject`
**Important Notes:**
- The ask step returns a boolean (`True` for approval, `False` for disapproval/timeout)
- **Keywords are case-sensitive** - you must respond with exact case (e.g., `LGTM` not `lgtm`)
- If no valid response is received, the step returns `False`
{% hint style="warning" %}
**Discord Case Sensitivity**: The Discord alerter implementation requires exact case matching for approval keywords. Make sure to respond with the exact case specified (e.g., `LGTM`, not `lgtm`).
{% endhint %}
For more information and a full list of configurable attributes of the Discord alerter, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-discord.html#zenml.integrations.discord) .
---
### Book/Component Guide/Alerters/Slack (docs/book/component-guide/alerters/slack.md)
---
description: Sending automated alerts to a Slack channel.
---
# Slack Alerter
The `SlackAlerter` enables you to send messages or ask questions within a
dedicated Slack channel directly from within your ZenML pipelines and steps.
## How to Create
### Set up a Slack app
In order to use the `SlackAlerter`, you first need to have a Slack workspace
set up with a channel that you want your pipelines to post to.
Then, you need to [create a Slack App](https://api.slack.com/apps?new\_app=1)
with a bot in your workspace. Make sure to give it the following permissions in
the `OAuth & Permissions` tab under `Scopes`:
* `chat:write`,
* `channels:read`
* `channels:history`
In order to be able to use the `ask()` functionality, you need to invite the app
to your channel. You can either use the `/invite` command directly in the
desired channel or add it through the channel settings:
{% hint style="warning" %}
It might take some time for your app to register within your workspace and
show up in the available list of applications.
{% endhint %}
### Registering a Slack Alerter in ZenML
To create a `SlackAlerter`, you first need to install ZenML's `slack`
integration:
```shell
zenml integration install slack -y
```
Once the integration is installed, you can use the ZenML CLI to create a
secret and register an alerter linked to the app you just created:
```shell
zenml secret create slack_token --oauth_token=
zenml alerter register slack_alerter \
--flavor=slack \
--slack_token={{slack_token.oauth_token}} \
--slack_channel_id=
```
{% hint style="info" %}
**Using Secrets for Token Management**: The example above demonstrates the recommended approach of storing your Slack token as a ZenML secret and referencing it using the `{{secret_name.key}}` syntax. This keeps sensitive information secure and follows security best practices.
Learn more about [referencing secrets in stack component attributes and settings](https://docs.zenml.io/concepts/secrets#reference-secrets-in-stack-component-attributes-and-settings).
{% endhint %}
Here is where you can find the required parameters:
* ``: The channel ID can be found in the channel details.
It starts with `C....`.
* ``: This is the Slack token of your bot. You can find it in the
Slack app settings under `OAuth & Permissions`.
After you have registered the `slack_alerter`, you can add it to your stack
like this:
```shell
zenml stack register ... -al slack_alerter --set
```
## How to Use
In ZenML, you can use alerters in various ways.
### Use the `post()` and `ask()` directly
You can use the client to fetch the active alerter within your stack and
use the `post` and `ask` methods directly:
```python
from zenml import pipeline, step
from zenml.client import Client
@step
def post_statement() -> None:
Client().active_stack.alerter.post("Step finished!")
@step
def ask_question() -> bool:
return Client().active_stack.alerter.ask("Should I continue?")
@pipeline(enable_cache=False)
def my_pipeline():
# Step using alerter.post
post_statement()
# Step using alerter.ask
ask_question()
if __name__ == "__main__":
my_pipeline()
```
{% hint style="warning" %}
In case of an error, the output of the `ask()` method default to `False`.
{% endhint %}
### Use it with custom settings
The Slack alerter comes equipped with a set of options that you can set during
runtime:
```python
from zenml import pipeline, step
from zenml.client import Client
# E.g, You can use a different channel ID through the settings. However, if you
# want to use the `ask` functionality, make sure that you app is invited to
# this channel first.
@step(settings={"alerter": {"slack_channel_id": "YOUR_SLACK_CHANNEL_ID"}})
def post_statement() -> None:
alerter = Client().active_stack.alerter
alerter.post("Posting to another channel!")
@pipeline(enable_cache=False)
def my_pipeline():
# Using alerter.post
post_statement()
if __name__ == "__main__":
my_pipeline()
```
## Use it with `SlackAlerterParameters` and `SlackAlerterPayload`
You can use these additional classes to further edit your messages:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Use the predefined steps
If you want to only use it in a simple manner, you can also use the steps
`slack_alerter_post_step` and `slack_alerter_ask_step`, that are built-in to
the Slack integration of ZenML:
```python
from zenml import pipeline, step
from zenml.integrations.slack.steps.slack_alerter_post_step import (
slack_alerter_post_step
)
from zenml.integrations.slack.steps.slack_alerter_ask_step import (
slack_alerter_ask_step,
)
@step
def process_approval_response(approved: bool) -> None:
if approved:
print("Operation approved!")
else:
print("Operation declined.")
@pipeline(enable_cache=False)
def my_pipeline():
slack_alerter_post_step("Posting a statement.")
approved = slack_alerter_ask_step("Asking a question. Should I continue?")
process_approval_response(approved)
if __name__ == "__main__":
my_pipeline()
```
## Default Response Keywords and Ask Step Behavior
The `ask()` method and `slack_alerter_ask_step` recognize these keywords by default:
**Approval:** `approve`, `LGTM`, `ok`, `yes`
**Disapproval:** `decline`, `disapprove`, `no`, `reject`
**Important Notes:**
- The ask step returns a boolean (`True` for approval, `False` for disapproval/timeout)
- **Response keywords are case-insensitive** - keywords are converted to lowercase before matching (e.g., both `LGTM` and `lgtm` work)
- If no valid response is received within the timeout period, the step returns `False`
- The default timeout is 300 seconds (5 minutes) but can be configured
{% hint style="info" %}
**Slack Case Handling**: The Slack alerter implementation automatically converts all response keywords to lowercase before matching, making responses case-insensitive. You can respond with `LGTM`, `lgtm`, or `Lgtm` - they'll all work.
{% endhint %}
For more information and a full list of configurable attributes of the Slack
alerter, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-slack.html#zenml.integrations.slack) .
---
### Book/Component Guide/Annotators/README (docs/book/component-guide/annotators/README.md)
---
icon: expand
description: Annotating the data in your workflow.
---
# Annotators
Annotators are a stack component that enables the use of data annotation as part of your ZenML stack and pipelines. You
can use the associated CLI command to launch annotation, configure your datasets and get stats on how many labeled tasks
you have ready for use.
Data annotation/labeling is a core part of MLOps that is frequently left out of the conversation. ZenML will
incrementally start to build features that support an iterative annotation workflow that sees the people doing
labeling (and their workflows/behaviors) as integrated parts of their ML process(es).
There are a number of different places in the ML lifecycle where this can happen:
* **At the start**: You might be starting out without any data, or with a ton of data but no clear sense of which parts
of it are useful to your particular problem. It’s not uncommon to have a lot of data but to be lacking accurate labels
for that data. So you can start and get great value from bootstrapping your model: label some data, train your model,
and use your model to suggest labels allowing you to speed up your labeling, iterating on and on in this way. Labeling
data early on in the process also helps clarify and condense down your specific rules and standards. For example, you
might realize that you need to have specific definitions for certain concepts so that your labeling efforts are
consistent across your team.
* **As new data comes in**: New data will likely continue to come in, and you might want to check in with the labeling
process at regular intervals to expose yourself to this new data. (You’ll probably also want to have some kind of
automation around detecting data or concept drift, but for certain kinds of unstructured data you probably can never
completely abandon the instant feedback of actual contact with the raw data.)
* **Samples generated for inference**: Your model will be making predictions on real-world data being passed in. If you
store and label this data, you’ll gain a valuable set of data that you can use to compare your labels with what the
model was predicting, another possible way to flag drifts of various kinds. This data can then (subject to
privacy/user consent) be used in retraining or fine-tuning your model.
* **Other ad hoc interventions**: You will probably have some kind of process to identify bad labels, or to find the
kinds of examples that your model finds really difficult to make correct predictions. For these, and for areas where
you have clear class imbalances, you might want to do ad hoc annotation to supplement the raw materials your model has
to learn from.
ZenML currently offers standard steps that help you tackle the above use cases, but the stack component and abstraction
will continue to be developed to make it easier to use.
### When to use it
The annotator is an optional stack component in the ZenML Stack. We designed our abstraction to fit into the larger ML
use cases, particularly the training and deployment parts of the lifecycle.
The core parts of the annotation workflow include:
* using labels or annotations in your training steps in a seamless way
* handling the versioning of annotation data
* allow for the conversion of annotation data to and from custom formats
* handle annotator-specific tasks, for example, the generation of UI config files that Label Studio requires for the web
annotation interface
### List of available annotators
For production use cases, some more flavors can be found in specific `integrations` modules. In terms of annotators,
ZenML features integrations with the following tools.
| Annotator | Flavor | Integration | Notes |
|-----------------------------------------|----------------|----------------|----------------------------------------------------------------------|
| [ArgillaAnnotator](argilla.md) | `argilla` | `argilla` | Connect ZenML with Argilla |
| [LabelStudioAnnotator](label-studio.md) | `label_studio` | `label_studio` | Connect ZenML with Label Studio |
| [PigeonAnnotator](pigeon.md) | `pigeon` | `pigeon` | Connect ZenML with Pigeon. Notebook only & for image and text classification tasks. |
| [ProdigyAnnotator](prodigy.md) | `prodigy` | `prodigy` | Connect ZenML with [Prodigy](https://prodi.gy/) |
| [Custom Implementation](custom.md) | _custom_ | | Extend the annotator abstraction and provide your own implementation |
If you would like to see the available flavors for annotators, you can use the command:
```shell
zenml annotator flavor list
```
### How to use it
The available implementation of the annotator is built on top of the Label
Studio integration, which means that using an annotator currently is no
different from what's described on the [Label Studio page: How to use
it?](label-studio.md#how-do-you-use-it). ([Pigeon](pigeon.md) is also supported, but has a
very limited functionality and only works within Jupyter notebooks.)
### A note on names
The various annotation tools have mostly standardized around the naming of key concepts as part of how they build their
tools. Unfortunately, this hasn't been completely unified so ZenML takes an opinion on which names we use for our stack
components and integrations. Key differences to note:
* Label Studio refers to the grouping of a set of annotations/tasks as a 'Project', whereas most other tools use the
term 'Dataset', so ZenML also calls this grouping a 'Dataset'.
* The individual meta-unit for 'an annotation + the source data' is referred to in different ways, but at ZenML (and
with Label Studio) we refer to them as 'tasks'.
The remaining core concepts ('annotation' and 'prediction', in particular) are broadly used among annotation tools.
---
### Book/Component Guide/Annotators/Argilla (docs/book/component-guide/annotators/argilla.md)
---
description: Annotating data using Argilla.
---
# Argilla
[Argilla](https://github.com/argilla-io/argilla) is a collaboration tool for AI engineers and domain experts who need to build high-quality datasets for their projects. It enables users to build robust language models through faster data curation using both human and machine feedback, providing support for each step in the MLOps cycle, from data labeling to model monitoring.
Argilla distinguishes itself for its focus on specific use cases and human-in-the-loop approaches. While it does offer programmatic features, Argilla's core value lies in actively involving human experts in the tool-building process, setting it apart from other competitors.
### When would you want to use it?
If you need to label textual data as part of your ML workflow, that is the point at which you could consider adding the Argilla annotator stack component as part of your ZenML stack.
We currently support the use of annotation at the various stages described in[the main annotators docs page](./). The Argilla integration currently is built to support annotation using a local (Docker-backed) instance of Argilla as well as a deployed instance of Argilla. There is an easy way to deploy Argilla as a [Hugging Face Space](https://huggingface.co/docs/hub/spaces-sdks-docker-argilla), for instance, which is documented in the [Argilla documentation](https://argilla.io/).
### How to deploy it?
The Argilla Annotator flavor is provided by the Argilla ZenML integration. You need to install it to be able to register it as an Annotator and add it to your stack:
```shell
zenml integration install argilla
```
You can either pass the `api_key` directly into the `zenml annotator register` command or you can register it as a secret and pass the secret name into the command. We recommend the latter approach for security reasons. If you want to take the latter approach, be sure to register a secret for whichever artifact store you choose, and then you should make sure to pass the name of that secret into the annotator as the `--authentication_secret`. For example, you'd run:
```shell
zenml secret create argilla_secrets --api_key=""
```
(Visit the Argilla documentation and interface to obtain your API key.)
Then register your annotator with ZenML:
```shell
zenml annotator register argilla --flavor argilla --authentication_secret=argilla_secrets --port=6900
```
When using a deployed instance of Argilla, the instance URL must be specified without any trailing `/` at the end. If you are using a Hugging Face Spaces instance and its visibility is set to private, you must also set the`headers` parameter which would include a Hugging Face token. For example:
```shell
zenml annotator register argilla --flavor argilla --authentication_secret=argilla_secrets --instance_url="https://[your-owner-name]-[your_space_name].hf.space" --headers='{"Authorization": "Bearer {[your_hugging_face_token]}"}'
```
Finally, add all these components to a stack and set it as your active stack. For example:
```shell
zenml stack copy default annotation
# this must be done separately so that the other required stack components are first registered
zenml stack update annotation -an
zenml stack set annotation
# optionally also
zenml stack describe
```
Now if you run a simple CLI command like `zenml annotator dataset list` this should work without any errors. You're ready to use your annotator in your ML workflow!
### How do you use it?
ZenML supports access to your data and annotations via the `zenml annotator ...` CLI command. We have also implemented an interface to some of the common Argilla functionality via the ZenML SDK.
You can access information about the datasets you're using with the `zenml annotator dataset list`. To work on annotation for a particular dataset, you can run `zenml annotator dataset annotate `. This will open the Argilla web interface for you to start annotating the dataset.
#### Argilla Annotator Stack Component
Our Argilla annotator component inherits from the `BaseAnnotator` class. There are some methods that are core methods that must be defined, like being able to register or get a dataset. Most annotators handle things like the storage of state and have their own custom features, so there are quite a few extra methods specific to Argilla.
The core Argilla functionality that's currently enabled includes a way to register your datasets, export any annotations for use in separate steps as well as start the annotator daemon process. (Argilla requires a server to be running in order to use the web interface, and ZenML handles the connection to this server using the details you passed in when registering the component.)
#### Argilla Annotator SDK
Visit [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-argilla.html) to learn more about the methods that ZenML exposes for the Argilla annotator. To access the SDK through Python, you would first get the client object and then call the methods you need. For example:
```python
from zenml.client import Client
client = Client()
annotator = client.active_stack.annotator
# list dataset names
dataset_names = annotator.get_dataset_names()
# get a specific dataset
dataset = annotator.get_dataset("dataset_name")
# get the annotations for a dataset
annotations = annotator.get_labeled_data(dataset_name="dataset_name")
```
For more detailed information on how to use the Argilla annotator and the functionality it provides, visit the [Argilla documentation](https://argilla.io/).
---
### Book/Component Guide/Annotators/Custom (docs/book/component-guide/annotators/custom.md)
---
description: Learning how to develop a custom annotator.
---
# Develop a Custom Annotator
{% hint style="info" %}
Before diving into the specifics of this component type, it is beneficial to familiarize yourself with our [general guide to writing custom component flavors in ZenML](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/implement-a-custom-stack-component). This guide provides an essential understanding of ZenML's component flavor concepts.
{% endhint %}
Annotators are a stack component that enables the use of data annotation as part of your ZenML stack and pipelines. You can use the associated CLI command to launch annotation, configure your datasets and get stats on how many labeled tasks you have ready for use.
{% hint style="warning" %}
**Base abstraction in progress!**
We are actively working on the base abstraction for the annotators, which will be available soon. As a result, their extension is not possible at the moment. If you would like to use an annotator in your stack, please check the list of already available feature stores down below.
{% endhint %}
---
### Book/Component Guide/Annotators/Label Studio (docs/book/component-guide/annotators/label-studio.md)
---
description: Annotating data using Label Studio.
---
# Label Studio
Label Studio is one of the leading open-source annotation platforms available to data scientists and ML practitioners. It is used to create or edit datasets that you can then use as part of training or validation workflows. It supports a broad range of annotation types, including:
* Computer Vision (image classification, object detection, semantic segmentation)
* Audio & Speech (classification, speaker diarization, emotion recognition, audio transcription)
* Text / NLP (classification, NER, question answering, sentiment analysis)
* Time Series (classification, segmentation, event recognition)
* Multi-Modal / Domain (dialogue processing, OCR, time series with reference)
### When would you want to use it?
If you need to label data as part of your ML workflow, that is the point at which you could consider adding the optional annotator stack component as part of your ZenML stack.
We currently support the use of annotation at the various stages described in [the main annotators docs page](./), and also offer custom utility functions to generate Label Studio label config files for image classification and object detection. (More will follow in due course.)
The Label Studio integration currently is built to support workflows using the following three cloud artifact stores: AWS S3, GCP/GCS, and Azure Blob Storage. Purely local stacks will currently _not_ work if you want to do add the annotation stack component as part of your stack.
### How to deploy it?
The Label Studio Annotator flavor is provided by the Label Studio ZenML integration, you need to install it, to be able to register it as an Annotator and add it to your stack:
```shell
zenml integration install label_studio
```
You will then need to obtain your Label Studio API key. This will give you access to the web annotation interface. (The following steps apply to a local instance of Label Studio, but feel free to obtain your API key directly from your deployed instance if that's what you are using.)
```shell
git clone https://github.com/HumanSignal/label-studio.git
cd label-studio
docker-compose up -d # starts label studio at http://localhost:8080
```
Then visit [http://localhost:8080/](http://localhost:8080/) to log in, and then visit [http://localhost:8080/user/account](http://localhost:8080/user/account) and get your Label Studio API key (from the upper right-hand corner). You will need it for the next step. Keep the Label Studio server running, because the ZenML Label Studio annotator will use it as the backend.
At this point you should register the API key under a custom secret name, making sure to replace the two parts in `<>` with whatever you choose:
```shell
zenml secret create label_studio_secrets --api_key=""
```
Then register your annotator with ZenML:
```shell
zenml annotator register label_studio --flavor label_studio --authentication_secret="label_studio_secrets" --port=8080
# for deployed instances of Label Studio, you can also pass in the URL as follows, for example:
# zenml annotator register label_studio --flavor label_studio --authentication_secret="" --instance_url="" --port=80
```
When using a deployed instance of Label Studio, the instance URL must be specified without any trailing `/` at the end. You should specify the port, for example, port 80 for a standard HTTP connection. For a Hugging Face deployment (the easiest way to get going with Label Studio), please read the [Hugging Face deployment documentation](https://huggingface.co/docs/hub/spaces-sdks-docker-label-studio).
Finally, add all these components to a stack and set it as your active stack. For example:
```shell
zenml stack copy default annotation
zenml stack update annotation -a
# this must be done separately so that the other required stack components are first registered
zenml stack update annotation -an
zenml stack set annotation
# optionally also
zenml stack describe
```
Now if you run a simple CLI command like `zenml annotator dataset list` this should work without any errors. You're ready to use your annotator in your ML workflow!
### How do you use it?
ZenML assumes that users have registered a cloud artifact store and an annotator as described above. ZenML currently only supports this setup, but we will add in the fully local stack option in the future.
ZenML supports access to your data and annotations via the `zenml annotator ...` CLI command.
You can access information about the datasets you're using with the `zenml annotator dataset list`. To work on annotation for a particular dataset, you can run `zenml annotator dataset annotate `.
[Our computer vision end to end example](https://github.com/zenml-io/zenml-projects/tree/main/end-to-end-computer-vision) is the best place to see how all the pieces of making this integration work fit together. What follows is an overview of some key components to the Label Studio integration and how it can be used.
#### Label Studio Annotator Stack Component
Our Label Studio annotator component inherits from the `BaseAnnotator` class. There are some methods that are core methods that must be defined, like being able to register or get a dataset. Most annotators handle things like the storage of state and have their own custom features, so there are quite a few extra methods specific to Label Studio.
The core Label Studio functionality that's currently enabled includes a way to register your datasets, export any annotations for use in separate steps as well as start the annotator daemon process. (Label Studio requires a server to be running in order to use the web interface, and ZenML handles the provisioning of this server locally using the details you passed in when registering the component unless you've specified that you want to use a deployed instance.)
#### Standard Steps
ZenML offers some standard steps (and their associated config objects) which will get you up and running with the Label Studio integration quickly. These include:
* `LabelStudioDatasetRegistrationConfig` - a step config object to be used when registering a dataset with Label studio using the `get_or_create_dataset` step
* `LabelStudioDatasetSyncConfig` - a step config object to be used when registering a dataset with Label studio using the `sync_new_data_to_label_studio` step. Note that this requires a ZenML secret to have been pre-registered with your artifact store as being the one that holds authentication secrets specific to your particular cloud provider. (Label Studio provides some documentation on what permissions these secrets require [here](https://labelstud.io/guide/tasks.html).)
* `get_or_create_dataset` step - This takes a `LabelStudioDatasetRegistrationConfig` config object which includes the name of the dataset. If it exists, this step will return the name, but if it doesn't exist then ZenML will register the dataset along with the appropriate label config with Label Studio.
* `get_labeled_data` step - This step will get all labeled data available for a particular dataset. Note that these are output in a Label Studio annotation format, which will subsequently be converted into a format appropriate for your specific use case.
* `sync_new_data_to_label_studio` step - This step is for ensuring that ZenML is handling the annotations and that the files being used are stored and synced with the ZenML cloud artifact store. This is an important step as part of a continuous annotation workflow since you want all the subsequent steps of your workflow to remain in sync with whatever new annotations are being made or have been created.
#### Helper Functions
Label Studio requires the use of what it calls 'label config' when you are creating/registering your dataset. These are strings containing HTML-like syntax that allow you to define a custom interface for your annotation. ZenML provides three helper functions that will construct these label config strings in the case of object detection, image classification, and OCR. See the[`integrations.label_studio.label_config_generators`](https://github.com/zenml-io/zenml/blob/main/src/zenml/integrations/label_studio/label_config_generators/label_config_generators.py) module for those three functions.
---
### Book/Component Guide/Annotators/Pigeon (docs/book/component-guide/annotators/pigeon.md)
---
description: Annotating data using Pigeon.
---
# Pigeon
Pigeon is a lightweight, open-source annotation tool designed for quick and easy labeling of data directly within Jupyter notebooks. It provides a simple and intuitive interface for annotating various types of data, including:
* Text Classification
* Image Classification
* Text Captioning
### When would you want to use it?
If you need to label a small to medium-sized dataset as part of your ML workflow and prefer the convenience of doing it directly within your Jupyter notebook, Pigeon is a great choice. It is particularly useful for:
* Quick labeling tasks that don't require a full-fledged annotation platform
* Iterative labeling during the exploratory phase of your ML project
* Collaborative labeling within a Jupyter notebook environment
### How to deploy it?
To use the Pigeon annotator, you first need to install the ZenML Pigeon integration:
```shell
zenml integration install pigeon
```
Next, register the Pigeon annotator with ZenML, specifying the output directory where the annotation files will be stored:
```shell
zenml annotator register pigeon --flavor pigeon --output_dir="path/to/dir"
```
Note that the `output_dir` is relative to the repository or notebook root.
Finally, add the Pigeon annotator to your stack and set it as the active stack:
```shell
zenml stack update --annotator pigeon
```
Now you're ready to use the Pigeon annotator in your ML workflow!
### How do you use it?
With the Pigeon annotator registered and added to your active stack, you can easily access it using the ZenML client within your Jupyter notebook.
For text classification tasks, you can launch the Pigeon annotator as follows:
````python
from zenml.client import Client
annotator = Client().active_stack.annotator
annotations = annotator.annotate(
data=[
'I love this movie',
'I was really disappointed by the book'
],
options=[
'positive',
'negative'
]
)
````
For image classification tasks, you can provide a custom display function to render the images:
````python
from zenml.client import Client
from IPython.display import display, Image
annotator = Client().active_stack.annotator
annotations = annotator.annotate(
data=[
'/path/to/image1.png',
'/path/to/image2.png'
],
options=[
'cat',
'dog'
],
display_fn=lambda filename: display(Image(filename))
)
````
The `launch` method returns the annotations as a list of tuples, where each tuple contains the data item and its corresponding label.
You can also use the `zenml annotator dataset` commands to manage your datasets:
* `zenml annotator dataset list` - List all available datasets
* `zenml annotator dataset delete ` - Delete a specific dataset
* `zenml annotator dataset stats ` - Get statistics for a specific dataset
Annotation files are saved as JSON files in the specified output directory. Each
annotation file represents a dataset, with the filename serving as the dataset
name.
## Acknowledgements
Pigeon was created by [Anastasis Germanidis](https://github.com/agermanidis) and
released as a [Python package](https://pypi.org/project/pigeon-jupyter/) and
[Github repository](https://github.com/agermanidis/pigeon). It is licensed under
the Apache License. It has been updated to work with more recent `ipywidgets`
versions and some small UI improvements were added. We are grateful to Anastasis
for creating this tool and making it available to the community.
---
### Book/Component Guide/Annotators/Prodigy (docs/book/component-guide/annotators/prodigy.md)
---
description: Annotating data using Prodigy.
---
# Prodigy
[Prodigy](https://prodi.gy/) is a modern annotation tool for creating training
and evaluation data for machine learning models. You can also use Prodigy to
help you inspect and clean your data, do error analysis and develop rule-based
systems to use in combination with your statistical models.
{% hint style="info" %} Prodigy is a paid annotation tool. You will need a
Prodigy is a paid tool. A license is required to download and use it with ZenML.
{% endhint %}
The Prodigy Python library includes a range of pre-built workflows and
command-line commands for various tasks, and well-documented components for
implementing your own workflow scripts. Your scripts can specify how the data is
loaded and saved, change which questions are asked in the annotation interface,
and can even define custom HTML and JavaScript to change the behavior of the
front-end. The web application is optimized for fast, intuitive and efficient
annotation.
### When would you want to use it?
If you need to label data as part of your ML workflow, that is the point at
which you could consider adding the optional annotator stack component as part
of your ZenML stack.
### How to deploy it?
The Prodigy Annotator flavor is provided by the Prodigy ZenML integration. You
need to install it to be able to register it as an Annotator and add it to your
stack:
```shell
zenml integration export-requirements --output-file prodigy-requirements.txt prodigy
```
Note that you'll need to install Prodigy separately since it requires a license.
Please [visit the Prodigy docs](https://prodi.gy/docs/install) for information
on how to install it. Currently Prodigy also requires the `urllib3<2`
dependency, so make sure to install that.
Then register your annotator with ZenML:
```shell
zenml annotator register prodigy --flavor prodigy
# optionally also pass in --custom_config_path=""
```
See https://prodi.gy/docs/install#config for more on custom Prodigy config
files. Passing a `custom_config_path` allows you to override the default Prodigy
config.
Finally, add all these components to a stack and set it as your active stack.
For example:
```shell
zenml stack copy default annotation
zenml stack update annotation -an prodigy
zenml stack set annotation
# optionally also
zenml stack describe
```
Now if you run a simple CLI command like `zenml annotator dataset list` this
should work without any errors. You're ready to use your annotator in your ML
workflow!
### How do you use it?
With Prodigy, there is no need to specially start the annotator ahead of time
like with [Label Studio](label-studio.md). Instead, just use Prodigy as per the
[Prodigy docs](https://prodi.gy) and then you can use the ZenML wrapper / API to
get your labeled data etc using our Python methods.
ZenML supports access to your data and annotations via the `zenml annotator ...`
CLI command.
You can access information about the datasets you're using with the `zenml
annotator dataset list`. To work on annotation for a particular dataset, you can
run `zenml annotator dataset annotate `. This is
the equivalent of running `prodigy ` in the terminal. For
example, you might run:
```shell
zenml annotator dataset annotate your_dataset --command="textcat.manual news_topics ./news_headlines.jsonl --label Technology,Politics,Economy,Entertainment"
```
This would launch the Prodigy interface for [the `textcat.manual` recipe](https://prodi.gy/docs/recipes#textcat-manual) with the
`news_topics` dataset and the labels `Technology`, `Politics`, `Economy`, and
`Entertainment`. The data would be loaded from the `news_headlines.jsonl` file.
A common workflow for Prodigy is to annotate data as you would usually do, and
then use the connection into ZenML to import those annotations within a step in
your pipeline (if running locally). For example, within a ZenML step:
```python
from typing import List, Dict, Any
from zenml import step
from zenml.client import Client
@step
def import_annotations() -> List[Dict[str, Any]:
zenml_client = Client()
annotations = zenml_client.active_stack.annotator.get_labeled_data(dataset_name="my_dataset")
# Do something with the annotations
return annotations
```
If you're running in a cloud environment, you can manually export the
annotations, store them somewhere in a cloud environment and then reference or
use those within ZenML. The precise way you do this will be very case-dependent,
however, so it's difficult to provide a one-size-fits-all solution.
#### Prodigy Annotator Stack Component
Our Prodigy annotator component inherits from the `BaseAnnotator` class. There
are some methods that are core methods that must be defined, like being able to
register or get a dataset. Most annotators handle things like the storage of
state and have their own custom features, so there are quite a few extra methods
specific to Prodigy.
The core Prodigy functionality that's currently enabled from within the
`annotator` stack component interface includes a way to register your datasets
and export any annotations for use in separate steps.
---
### Book/Component Guide/Artifact Stores/README (docs/book/component-guide/artifact-stores/README.md)
---
description: Setting up a persistent storage for your artifacts.
icon: folder-closed
---
# Artifact Stores
The Artifact Store is a central component in any MLOps stack. As the name suggests, it acts as a data persistence layer where artifacts (e.g. datasets, models) ingested or generated by the machine learning pipelines are stored.
ZenML automatically serializes and saves the data circulated through your pipelines in the Artifact Store: datasets, models, data profiles, data and model validation reports, and generally any object that is returned by a pipeline step. This is coupled with tracking in ZenML to provide extremely useful features such as caching and provenance/lineage tracking and pipeline reproducibility.
{% hint style="info" %}
Not all objects returned by pipeline steps are physically stored in the Artifact Store, nor do they have to be. How artifacts are serialized and deserialized and where their contents are stored are determined by the particular implementation of the [Materializer](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types) associated with the artifact data type. The majority of Materializers shipped with ZenML use the Artifact Store which is part of the active Stack as the location where artifacts are kept.
If you need to store _a particular type of pipeline artifact_ in a different medium (e.g. use an external model registry to store model artifacts, or an external data lake or data warehouse to store dataset artifacts), you can write your own [Materializer](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types) to implement the custom logic required for it. In contrast, if you need to use an entirely different storage backend to store artifacts, one that isn't already covered by one of the ZenML integrations, you can [extend the Artifact Store abstraction](custom.md) to provide your own Artifact Store implementation.
{% endhint %}
In addition to pipeline artifacts, the Artifact Store may also be used as storage backed by other specialized stack components that need to store their data in the form of persistent object storage. The [Great Expectations Data Validator](https://docs.zenml.io/stacks/data-validators/great-expectations) is such an example.
Related concepts:
* the Artifact Store is a type of Stack Component that needs to be registered as part of your ZenML [Stack](https://docs.zenml.io/user-guides/production-guide/understand-stacks).
* the objects circulated through your pipelines are serialized and stored in the Artifact Store using [Materializer](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types). Materializers implement the logic required to serialize and deserialize the artifact contents and to store them and retrieve their contents to/from the Artifact Store.
### When to use it
The Artifact Store is a mandatory component in the ZenML stack. It is used to store all artifacts produced by pipeline runs, and you are required to configure it in all of your stacks.
#### Artifact Store Flavors
Out of the box, ZenML comes with a `local` artifact store already part of the default stack that stores artifacts on your local filesystem. Additional Artifact Stores are provided by integrations:
| Artifact Store | Flavor | Integration | URI Schema(s) | Notes |
| ---------------------------------- | -------- | ----------- | ------------------ | -------------------------------------------------------------------------------------------------------------------------------- |
| [Local](local.md) | `local` | _built-in_ | None | This is the default Artifact Store. It stores artifacts on your local filesystem. Should be used only for running ZenML locally. |
| [Amazon S3](s3.md) | `s3` | `s3` | `s3://` | Uses AWS S3 as an object store backend |
| [Backblaze B2](b2.md) | `b2` | `b2` | `s3://` | Uses Backblaze B2 through its S3-compatible API |
| [Google Cloud Storage](gcp.md) | `gcp` | `gcp` | `gs://` | Uses Google Cloud Storage as an object store backend |
| [Azure](azure.md) | `azure` | `azure` | `abfs://`, `az://` | Uses Azure Blob Storage as an object store backend |
| [Alibaba Cloud OSS](alibaba-oss.md) | `s3` | `s3` | `s3://` | Uses S3 integration to connect to Alibaba Cloud OSS |
| [MinIO](minio.md) | `s3` | `s3` | `s3://` | Uses S3 integration to connect to self-hosted MinIO |
| [Custom Implementation](custom.md) | _custom_ | | _custom_ | Extend the Artifact Store abstraction and provide your own implementation |
If you would like to see the available flavors of Artifact Stores, you can use the command:
```shell
zenml artifact-store flavor list
```
{% hint style="info" %}
Every Artifact Store has a `path` attribute that must be configured when it is registered with ZenML. This is a URI pointing to the root path where all objects are stored in the Artifact Store. It must use a URI schema that is supported by the Artifact Store flavor. For example, the S3 Artifact Store will need a URI that contains the `s3://` schema:
```shell
zenml artifact-store register s3_store -f s3 --path s3://my_bucket
```
{% endhint %}
### How to use it
The Artifact Store provides low-level object storage services for other ZenML mechanisms. When you develop ZenML pipelines, you normally don't even have to be aware of its existence or interact with it directly. ZenML provides higher-level APIs that can be used as an alternative to store and access artifacts:
* return one or more objects from your pipeline steps to have them automatically saved in the active Artifact Store as pipeline artifacts.
* [retrieve pipeline artifacts](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/load-artifacts-into-memory) from the active Artifact Store after a pipeline run is complete.
You will probably need to interact with the [low-level Artifact Store API](./#the-artifact-store-api) directly:
* if you implement custom [Materializers](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types) for your artifact data types
* if you want to store custom objects in the Artifact Store
#### The Artifact Store API
All ZenML Artifact Stores implement [the same IO API](custom.md) that resembles a standard file system. This allows you to access and manipulate the objects stored in the Artifact Store in the same manner you would normally handle files on your computer and independently of the particular type of Artifact Store that is configured in your ZenML stack.
Accessing the low-level Artifact Store API can be done through the following Python modules:
* `zenml.io.fileio` provides low-level utilities for manipulating Artifact Store objects (e.g. `open`, `copy`, `rename` , `remove`, `mkdir`). These functions work seamlessly across Artifact Stores types. They have the same signature as the [Artifact Store abstraction methods](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.base_artifact_store) ( in fact, they are one and the same under the hood).
* [zenml.utils.io\_utils](https://sdkdocs.zenml.io/latest/core_code_docs/core-utils.html#zenml.utils.io_utils) includes some higher-level helper utilities that make it easier to find and transfer objects between the Artifact Store and the local filesystem or memory.
{% hint style="info" %}
When calling the Artifact Store API, you should always use URIs that are relative to the Artifact Store root path, otherwise, you risk using an unsupported protocol or storing objects outside the store. You can use the `Repository` singleton to retrieve the root path of the active Artifact Store and then use it as a base path for artifact URIs, e.g.:
```python
import os
from zenml.client import Client
from zenml.io import fileio
root_path = Client().active_stack.artifact_store.path
artifact_contents = "example artifact"
artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.txt")
fileio.makedirs(artifact_path)
with fileio.open(artifact_uri, "w") as f:
f.write(artifact_contents)
```
When using the Artifact Store API to write custom Materializers, the base artifact URI path is already provided. See the documentation on [Materializers](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/handle-custom-data-types) for an example.
{% endhint %}
The following are some code examples showing how to use the Artifact Store API for various operations:
* creating folders, writing and reading data directly to/from an artifact store object
```python
import os
from zenml.utils import io_utils
from zenml.io import fileio
from zenml.client import Client
root_path = Client().active_stack.artifact_store.path
artifact_contents = "example artifact"
artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.txt")
fileio.makedirs(artifact_path)
io_utils.write_file_contents_as_string(artifact_uri, artifact_contents)
```
```python
import os
from zenml.utils import io_utils
from zenml.client import Client
root_path = Client().active_stack.artifact_store.path
artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.txt")
artifact_contents = io_utils.read_file_contents_as_string(artifact_uri)
```
* using a temporary local file/folder to serialize and copy in-memory objects to/from the artifact store (heavily used in Materializers to transfer information between the Artifact Store and external libraries that don't support writing/reading directly to/from the artifact store backend):
```python
import os
import tempfile
import external_lib
from zenml.client import Client
from zenml.io import fileio
root_path = Client().active_stack.artifact_store.path
artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.json")
fileio.makedirs(artifact_path)
with tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=True
) as f:
external_lib.external_object.save_to_file(f.name)
# Copy it into artifact store
fileio.copy(f.name, artifact_uri)
```
```python
import os
import tempfile
import external_lib
from zenml.client import Client
from zenml.io import fileio
root_path = Client().active_stack.artifact_store.path
artifact_path = os.path.join(root_path, "artifacts", "examples")
artifact_uri = os.path.join(artifact_path, "test.json")
with tempfile.NamedTemporaryFile(
mode="w", suffix=".json", delete=True
) as f:
# Copy the serialized object from the artifact store
fileio.copy(artifact_uri, f.name)
external_lib.external_object.load_from_file(f.name)
```
---
### Book/Component Guide/Artifact Stores/Alibaba Oss (docs/book/component-guide/artifact-stores/alibaba-oss.md)
---
description: Storing artifacts in Alibaba Cloud Object Storage Service (OSS).
---
# Alibaba Cloud OSS
[Alibaba Cloud Object Storage Service (OSS)](https://www.alibabacloud.com/product/object-storage-service) is an S3-compatible object storage service. Since OSS provides an S3-compatible API, you can use ZenML's S3 Artifact Store integration to connect to [Alibaba Cloud](https://www.alibabacloud.com) OSS.
{% hint style="warning" %}
**Important:** When using Alibaba Cloud OSS, you must set the following `config_kwargs`:
```json
{"signature_version": "s3", "s3": {"addressing_style": "virtual"}}
```
This is required for proper compatibility with Alibaba Cloud OSS's S3 API implementation.
{% endhint %}
### When would you want to use it?
You should use the Alibaba Cloud OSS Artifact Store when:
* Your infrastructure is already deployed on Alibaba Cloud and you want to maintain data locality
* You require artifact storage in specific geographic regions served by Alibaba Cloud (China, Asia-Pacific, Europe, Middle East)
* You need S3-compatible object storage with Alibaba Cloud's pricing model and service level agreements
* Compliance requirements mandate data residency in Alibaba Cloud regions
### How do you deploy it?
Since Alibaba Cloud OSS is S3-compatible, you'll use the S3 integration. First, install it:
```shell
zenml integration install s3 -y
```
You'll also need to create an OSS bucket and obtain your access credentials from the Alibaba Cloud console.
### How do you configure it?
To use Alibaba Cloud OSS with ZenML, you need to configure the S3 Artifact Store with specific settings for OSS compatibility:
{% hint style="info" %}
Alibaba Cloud OSS does not support ZenML Service Connectors. Use ZenML Secrets to securely store and reference your Alibaba Cloud credentials.
{% endhint %}
{% tabs %}
{% tab title="Using a ZenML Secret (recommended)" %}
First, create a ZenML secret with your Alibaba Cloud credentials:
```shell
zenml secret create alibaba_secret \
--access_key_id='' \
--secret_access_key=''
```
Then register the artifact store with the required OSS configuration:
```shell
zenml artifact-store register alibaba_store -f s3 \
--path='s3://your-bucket-name' \
--authentication_secret=alibaba_secret \
--client_kwargs='{"endpoint_url": "https://oss-.aliyuncs.com"}' \
--config_kwargs='{"signature_version": "s3", "s3": {"addressing_style": "virtual"}}'
```
{% endtab %}
{% endtabs %}
Replace `` with your OSS region (e.g., `eu-central-1`, `cn-hangzhou`, `ap-southeast-1`). You can find the list of available regions and their endpoints in the [Alibaba Cloud OSS documentation](https://www.alibabacloud.com/help/en/oss/user-guide/regions-and-endpoints).
Finally, add the artifact store to your stack:
```shell
zenml stack register custom_stack -a alibaba_store ... --set
```
### How do you use it?
Using the Alibaba Cloud OSS Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it). ZenML handles the S3-compatible API translation automatically.
For more details on the S3 Artifact Store configuration options, refer to the [S3 Artifact Store documentation](s3.md).
---
### Book/Component Guide/Artifact Stores/Azure (docs/book/component-guide/artifact-stores/azure.md)
---
description: Storing artifacts using Azure Blob Storage
---
# Azure Blob Storage
The Azure Artifact Store is an [Artifact Store](./) flavor provided with the Azure ZenML integration that uses [the Azure Blob Storage managed object storage service](https://azure.microsoft.com/en-us/services/storage/blobs/) to store ZenML artifacts in an Azure Blob Storage container.
### When would you want to use it?
Running ZenML pipelines with [the local Artifact Store](local.md) is usually sufficient if you just want to evaluate ZenML or get started quickly without incurring the trouble and the cost of employing cloud storage services in your stack. However, the local Artifact Store becomes insufficient or unsuitable if you have more elaborate needs for your project:
* if you want to share your pipeline run results with other team members or stakeholders inside or outside your organization
* if you have other components in your stack that are running remotely (e.g. a Kubeflow or Kubernetes Orchestrator running in a public cloud).
* if you outgrow what your local machine can offer in terms of storage space and need to use some form of private or public storage service that is shared with others
* if you are running pipelines at scale and need an Artifact Store that can handle the demands of production-grade MLOps
In all these cases, you need an Artifact Store that is backed by a form of public cloud or self-hosted shared object storage service.
You should use the Azure Artifact Store when you decide to keep your ZenML artifacts in a shared object storage and if you have access to the Azure Blob Storage managed service. You should consider one of the other [Artifact Store flavors](./#artifact-store-flavors) if you don't have access to the Azure Blob Storage service.
### How do you deploy it?
{% hint style="info" %}
Would you like to skip ahead and deploy a full ZenML cloud stack already, including an Azure Artifact Store? Check out the[in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), or [the ZenML Azure Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register this stack component.
{% endhint %}
The Azure Artifact Store flavor is provided by the Azure ZenML integration, you need to install it on your local machine to be able to register an Azure Artifact Store and add it to your stack:
```shell
zenml integration install azure -y
```
The only configuration parameter mandatory for registering an Azure Artifact Store is the root path URI, which needs to point to an Azure Blog Storage container and take the form `az://container-name` or `abfs://container-name`. Please read [the Azure Blob Storage documentation](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal) on how to configure an Azure Blob Storage container.
With the URI to your Azure Blob Storage container known, registering an Azure Artifact Store can be done as follows:
```shell
# Register the Azure artifact store
zenml artifact-store register az_store -f azure --path=az://container-name
# Register and set a stack with the new artifact store
zenml stack register custom_stack -a az_store ... --set
```
Depending on your use case, however, you may also need to provide additional configuration parameters pertaining to [authentication](azure.md#authentication-methods) to match your deployment scenario.
#### Authentication Methods
Integrating and using an Azure Artifact Store in your pipelines is not possible without employing some form of authentication. If you're looking for a quick way to get started locally, you can use the _Implicit Authentication_ method. However, the recommended way to authenticate to the Azure cloud platform is through [an Azure Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector). This is particularly useful if you are configuring ZenML stacks that combine the Azure Artifact Store with other remote stack components also running in Azure.
You will need the following information to configure Azure credentials for ZenML, depending on which type of Azure credentials you want to use:
* an Azure connection string
* an Azure account key
* the client ID, client secret and tenant ID of the Azure service principal
For more information on how to retrieve information about your Azure Storage Account and Access Key or connection string, please refer to this [Azure guide](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-python?tabs=environment-variable-windows#copy-your-credentials-from-the-azure-portal).
For information on how to configure an Azure service principal, please consult the [Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal).
{% tabs %}
{% tab title="Implicit Authentication" %}
This method uses the implicit Azure authentication available _in the environment where the ZenML code is running_. On your local machine, this is the quickest way to configure an Azure Artifact Store. You don't need to supply credentials explicitly when you register the Azure Artifact Store, instead, you have to set one of the following sets of environment variables:
* to use [an Azure storage account key](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage) , set `AZURE_STORAGE_ACCOUNT_NAME` to your account name and one of `AZURE_STORAGE_ACCOUNT_KEY` or `AZURE_STORAGE_SAS_TOKEN` to the Azure key value.
* to use [an Azure storage account key connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage) , set `AZURE_STORAGE_CONNECTION_STRING` to your Azure Storage Key connection string
* to use [Azure Service Principal credentials](https://learn.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals) , [create an Azure Service Principal](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) and then set `AZURE_STORAGE_ACCOUNT_NAME` to your account name and `AZURE_STORAGE_CLIENT_ID` , `AZURE_STORAGE_CLIENT_SECRET` and `AZURE_STORAGE_TENANT_ID` to the client ID, secret and tenant ID of your service principal
{% hint style="warning" %}
Certain dashboard functionality, such as visualizing or deleting artifacts, is not available when using an implicitly authenticated artifact store together with a deployed ZenML server because the ZenML server will not have permission to access the filesystem.
The implicit authentication method also needs to be coordinated with other stack components that are highly dependent on the Artifact Store and need to interact with it directly to the function. If these components are not running on your machine, they do not have access to the local environment variables and will encounter authentication failures while trying to access the Azure Artifact Store:
* [Orchestrators](https://docs.zenml.io/stacks/orchestrators/) need to access the Artifact Store to manage pipeline artifacts
* [Step Operators](https://docs.zenml.io/stacks/step-operators/) need to access the Artifact Store to manage step-level artifacts
* [Model Deployers](https://docs.zenml.io/stacks/model-deployers/) need to access the Artifact Store to load served models
To enable these use cases, it is recommended to use [an Azure Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector) to link your Azure Artifact Store to the remote Azure Blob storage container.
{% endhint %}
{% endtab %}
{% tab title="Azure Service Connector (recommended)" %}
To set up the Azure Artifact Store to authenticate to Azure and access an Azure Blob storage container, it is recommended to leverage the many features provided by [the Azure Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector) such as auto-configuration, best security practices regarding long-lived credentials and reusing the same credentials across multiple stack components.
If you don't already have an Azure Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command. You have the option to configure an Azure Service Connector that can be used to access more than one Azure blob storage container or even more than one type of Azure resource:
```sh
zenml service-connector register --type azure -i
```
A non-interactive CLI example that uses [Azure Service Principal credentials](https://learn.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals) to configure an Azure Service Connector targeting a single Azure Blob storage container is:
```sh
zenml service-connector register --type azure --auth-method service-principal --tenant_id= --client_id= --client_secret= --resource-type blob-container --resource-id
```
{% code title="Example Command Output" %}
```
$ zenml service-connector register azure-blob-demo --type azure --auth-method service-principal --tenant_id=a79f3633-8f45-4a74-a42e-68871c17b7fb --client_id=8926254a-8c3f-430a-a2fd-bdab234d491e --client_secret=AzureSuperSecret --resource-type blob-container --resource-id az://demo-zenmlartifactstore
Successfully registered service connector `azure-blob-demo` with access to the following resources:
┏━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠───────────────────┼──────────────────────────────┨
┃ 📦 blob-container │ az://demo-zenmlartifactstore ┃
┗━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
> **Note**: Please remember to grant the Azure service principal permissions to read and write to your Azure Blob storage container as well as to list accessible storage accounts and Blob containers. For a full list of permissions required to use an AWS Service Connector to access one or more S3 buckets, please refer to the [Azure Service Connector Blob storage container resource type documentation](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector#azure-blob-storage-container) or read the documentation available in the interactive CLI commands and dashboard. The Azure Service Connector supports [many different authentication methods](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector#authentication-methods) with different levels of security and convenience. You should pick the one that best fits your use-case.
If you already have one or more Azure Service Connectors configured in your ZenML deployment, you can check which of them can be used to access the Azure Blob storage container you want to use for your Azure Artifact Store by running e.g.:
```sh
zenml service-connector list-resources --resource-type blob-container
```
{% code title="Example Command Output" %}
```
The following 'blob-container' resources can be accessed by service connectors:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼─────────────────────────┼────────────────┼───────────────────┼──────────────────────────────┨
┃ 273d2812-2643-4446-82e6-6098b8ccdaa4 │ azure-service-principal │ 🇦 azure │ 📦 blob-container │ az://demo-zenmlartifactstore ┃
┠──────────────────────────────────────┼─────────────────────────┼────────────────┼───────────────────┼──────────────────────────────┨
┃ f6b329e1-00f7-4392-94c9-264119e672d0 │ azure-blob-demo │ 🇦 azure │ 📦 blob-container │ az://demo-zenmlartifactstore ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
After having set up or decided on an Azure Service Connector to use to connect to the target Azure Blob storage container, you can register the Azure Artifact Store as follows:
```sh
# Register the Azure artifact-store and reference the target blob storage container
zenml artifact-store register -f azure \
--path='az://your-container'
# Connect the Azure artifact-store to the target container via an Azure Service Connector
zenml artifact-store connect -i
```
A non-interactive version that connects the Azure Artifact Store to a target blob storage container through an Azure Service Connector:
```sh
zenml artifact-store connect --connector
```
{% code title="Example Command Output" %}
```
$ zenml artifact-store connect azure-blob-demo --connector azure-blob-demo
Successfully connected artifact store `azure-blob-demo` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼─────────────────┼────────────────┼───────────────────┼──────────────────────────────┨
┃ f6b329e1-00f7-4392-94c9-264119e672d0 │ azure-blob-demo │ 🇦 azure │ 📦 blob-container │ az://demo-zenmlartifactstore ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
As a final step, you can use the Azure Artifact Store in a ZenML Stack:
```sh
# Register and set a stack with the new artifact store
zenml stack register -a ... --set
```
{% endtab %}
{% tab title="ZenML Secret" %}
When you register the Azure Artifact Store, you can create a [ZenML Secret](https://docs.zenml.io/getting-started/deploying-zenml/secret-management) to store a variety of Azure credentials and then reference it in the Artifact Store configuration:
* to use [an Azure storage account key](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage) , set `account_name` to your account name and one of `account_key` or `sas_token` to the Azure key or SAS token value as attributes in the ZenML secret
* to use [an Azure storage account key connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage) , configure the `connection_string` attribute in the ZenML secret to your Azure Storage Key connection string
* to use [Azure Service Principal credentials](https://learn.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals) , [create an Azure Service Principal](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) and then set `account_name` to your account name and `client_id`, `client_secret` and `tenant_id` to the client ID, secret and tenant ID of your service principal in the ZenML secret
This method has some advantages over the implicit authentication method:
* you don't need to install and configure the Azure CLI on your host
* you don't need to care about enabling your other stack components (orchestrators, step operators and model deployers) to have access to the artifact store through Azure Managed Identities
* you can combine the Azure artifact store with other stack components that are not running in Azure
Configuring Azure credentials in a ZenML secret and then referencing them in the Artifact Store configuration could look like this:
```shell
# Store the Azure storage account key in a ZenML secret
zenml secret create az_secret \
--account_name='' \
--account_key=''
# or if you want to use a connection string
zenml secret create az_secret \
--connection_string=''
# or if you want to use Azure ServicePrincipal credentials
zenml secret create az_secret \
--account_name='' \
--tenant_id='' \
--client_id='' \
--client_secret=''
# Alternatively for providing key-value pairs, you can utilize the '--values' option by specifying a file path containing
# key-value pairs in either JSON or YAML format.
# File content example: {"account_name":"",...}
zenml secret create az_secret \
--values=@path/to/file.txt
# Register the Azure artifact store and reference the ZenML secret
zenml artifact-store register az_store -f azure \
--path='az://your-container' \
--authentication_secret=az_secret
# Register and set a stack with the new artifact store
zenml stack register custom_stack -a az_store ... --set
```
{% endtab %}
{% endtabs %}
For more, up-to-date information on the Azure Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-azure.html#zenml.integrations.azure) .
### How do you use it?
Aside from the fact that the artifacts are stored in Azure Blob Storage, using the Azure Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it).
---
### Book/Component Guide/Artifact Stores/B2 (docs/book/component-guide/artifact-stores/b2.md)
---
description: Storing artifacts in a Backblaze B2 cloud storage bucket.
---
# Backblaze B2
The Backblaze B2 Artifact Store is an [Artifact Store](./) flavor provided with the B2 ZenML integration that uses [Backblaze B2 Cloud Storage](https://www.backblaze.com/cloud-storage) to store ZenML artifacts. Backblaze B2 exposes an [S3-compatible API](https://www.backblaze.com/docs/cloud-storage-s3-compatible-api), so this flavor reuses ZenML's S3 implementation under the hood while adding B2-friendly runtime fallbacks for the endpoint URL and credential environment variables.
### When would you want to use it?
Running ZenML pipelines with [the local Artifact Store](local.md) is usually sufficient if you just want to evaluate ZenML or get started quickly without incurring the trouble and the cost of employing cloud storage services in your stack. However, the local Artifact Store becomes insufficient or unsuitable if you have more elaborate needs for your project:
* if you want to share your pipeline run results with other team members or stakeholders inside or outside your organization
* if you have other components in your stack that are running remotely (e.g. a Kubeflow or Kubernetes Orchestrator running in a public cloud).
* if you outgrow what your local machine can offer in terms of storage space and need to use some form of private or public storage service that is shared with others
* if you are running pipelines at scale and need an Artifact Store that can handle the demands of production-grade MLOps
You should use the Backblaze B2 Artifact Store when you want to keep your ZenML artifacts in shared object storage and you prefer Backblaze B2's pricing, egress terms, or geographic regions over the hyperscalers. Because the integration speaks B2's S3-compatible API, you get the same workflow as the [S3 Artifact Store](s3.md) without having to manage any additional B2-native tooling. You should consider one of the other [Artifact Store flavors](./#artifact-store-flavors) if you don't have a Backblaze B2 account or if you require an S3 service that is co-located with other AWS resources in your stack.
### How do you deploy it?
The Backblaze B2 Artifact Store flavor is provided by the B2 ZenML integration, you need to install it on your local machine to be able to register a B2 Artifact Store and add it to your stack:
```shell
zenml integration install b2 -y
```
You will also need a Backblaze B2 account and a private bucket to hold your ZenML artifacts. The high-level setup steps in the [Backblaze console](https://secure.backblaze.com/) are:
1. **Create a B2 bucket.** From the Backblaze web console, navigate to **B2 Cloud Storage > Buckets** and click **Create a Bucket**. Pick a globally-unique bucket name (this becomes the `s3://` path used by ZenML), choose **Private**, and select the region you want to store data in. The region segment of the bucket's S3 endpoint (e.g. `us-west-004`, `eu-central-003`) is needed in the next step.
2. **Generate a B2 application key.** Navigate to **Account > Application Keys** and click **Add a New Application Key**. Scope the key to the bucket you just created and grant at minimum **Read and Write** access. Backblaze will display a `keyID` and an `applicationKey` exactly once — copy both values immediately.
3. **Export the credentials as environment variables.** ZenML's B2 artifact store reads Backblaze's standard environment variable names at runtime if no key/secret is configured directly on the artifact store:
```shell
export B2_APPLICATION_KEY_ID=
export B2_APPLICATION_KEY=
```
This convention matches the [Backblaze CLI and SDKs](https://www.backblaze.com/docs/cloud-storage-application-keys), so the same environment that authenticates `b2-cli` will authenticate ZenML.
For full Backblaze documentation on bucket creation and application keys, see the [B2 Cloud Storage docs](https://www.backblaze.com/docs/cloud-storage).
### How do you use it?
The only configuration parameter mandatory for registering a B2 Artifact Store is the root path URI, which needs to point to a B2 bucket using the `s3://` scheme — this is intentional, because the B2 flavor talks to Backblaze through its S3-compatible API:
```shell
# Register the B2 artifact store (uses the s3:// URI scheme since B2 speaks the S3 API)
zenml artifact-store register my_b2 --flavor=b2 --path=s3://my-bucket/zenml-artifacts
# Register and set a stack that uses the new artifact store
zenml stack register custom_stack -a my_b2 ... --set
```
If `B2_APPLICATION_KEY_ID` and `B2_APPLICATION_KEY` are exported in the shell that runs ZenML, no further authentication configuration is required. The artifact store client will pick them up when it is created without saving them as component configuration. To keep credentials out of the shell environment, store them in a [ZenML Secret](https://docs.zenml.io/getting-started/deploying-zenml/secret-management) and reference them on registration:
```shell
# Store the B2 application key in a ZenML secret
zenml secret create b2_secret \
--access_key_id='' \
--secret_access_key=''
# Register the B2 artifact store and reference the secret
zenml artifact-store register my_b2 --flavor=b2 \
--path=s3://my-bucket/zenml-artifacts \
--authentication_secret=b2_secret
```
#### Endpoint defaults and regions
Configure the endpoint that matches your B2 bucket region at registration time:
```shell
zenml artifact-store register my_b2 --flavor=b2 \
--path=s3://my-bucket/zenml-artifacts \
--client_kwargs='{"endpoint_url": "https://s3.eu-central-003.backblazeb2.com"}'
```
Use the S3 endpoint shown for your bucket in the Backblaze console. It always has the form `https://s3..backblazeb2.com`. If you omit `client_kwargs.endpoint_url`, the runtime client falls back to `https://s3.us-west-004.backblazeb2.com` without saving that fallback on the component configuration.
#### Advanced configuration
Because the B2 flavor subclasses the S3 flavor, every advanced option supported by the [S3 Artifact Store](s3.md#advanced-configuration) — `client_kwargs`, `config_kwargs`, and `s3_additional_kwargs` — is available here too and is passed transparently to the underlying [s3fs](https://s3fs.readthedocs.io/en/latest/#s3-compatible-storage) / [botocore](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html) stack. Use these to configure things like server-side encryption, custom retry behavior, or alternative B2 endpoints.
Aside from the fact that the artifacts are stored in Backblaze B2, using the B2 Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it).
---
### Book/Component Guide/Artifact Stores/Custom (docs/book/component-guide/artifact-stores/custom.md)
---
description: Learning how to develop a custom artifact store.
---
# Develop a custom artifact store
{% hint style="info" %}
Before diving into the specifics of this component type, it is beneficial to familiarize yourself with our [general guide to writing custom component flavors in ZenML](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/implement-a-custom-stack-component). This guide provides an essential understanding of ZenML's component flavor concepts.
{% endhint %}
ZenML comes equipped with [Artifact Store implementations](./#artifact-store-flavors) that you can use to store artifacts on a local filesystem or in the managed AWS, GCP, or Azure cloud object storage services. However, if you need to use a different type of object storage service as a backend for your ZenML Artifact Store, you can extend ZenML to provide your own custom Artifact Store implementation.
### Base Abstraction
The Artifact Store establishes one of the main components in every ZenML stack. Now, let us take a deeper dive into the fundamentals behind its abstraction, namely [the `BaseArtifactStore` class](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.base_artifact_store):
1. As ZenML only supports filesystem-based artifact stores, it features a configuration parameter called `path`, which will indicate the root path of the artifact store. When registering an artifact store, users will have to define this parameter.
2. Moreover, there is another variable in the config class called `SUPPORTED_SCHEMES`. This is a class variable that needs to be defined in every subclass of the base artifact store configuration. It indicates the supported file path schemes for the corresponding implementation. For instance, for the Azure artifact store, this set will be defined as `{"abfs://", "az://"}`.
3. Lastly, the base class features a set of `abstractmethod`s: `open`, `copyfile`,`exists`,`glob`,`isdir`,`listdir` ,`makedirs`,`mkdir`,`remove`, `rename`,`rmtree`,`stat`,`walk`. In the implementation of every `ArtifactStore` flavor, it is required to define these methods with respect to the flavor at hand.
Putting all these considerations together, we end up with the following implementation:
```
/* Detailed source-code truncated for AI context efficiency. */
```
{% hint style="info" %}
This is a slimmed-down version of the base implementation which aims to highlight the abstraction layer. In order to see the full implementation and get the complete docstrings, please check the [SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.base_artifact_store) .
{% endhint %}
**The effect on the `zenml.io.fileio`**
If you created an instance of an artifact store, added it to your stack, and activated the stack, it will create a filesystem each time you run a ZenML pipeline and make it available to the `zenml.io.fileio` module.
This means that when you utilize a method such as `fileio.open(...)` with a file path that starts with one of the `SUPPORTED_SCHEMES` within your steps or materializers, it will be able to use the `open(...)` method that you defined within your artifact store.
### Build your own custom artifact store
If you want to implement your own custom Artifact Store, you can follow the following steps:
1. Create a class that inherits from [the `BaseArtifactStore` class](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.base_artifact_store) and implements the abstract methods.
2. Create a class that inherits from [the `BaseArtifactStoreConfig` class](custom.md) and fill in the `SUPPORTED_SCHEMES` based on your file system.
3. Bring both of these classes together by inheriting from [the `BaseArtifactStoreFlavor` class](custom.md).
Once you are done with the implementation, you can register it through the CLI. Please ensure you **point to the flavor class via dot notation**:
```shell
zenml artifact-store flavor register
```
For example, if your flavor class `MyArtifactStoreFlavor` is defined in `flavors/my_flavor.py`, you'd register it by doing:
```shell
zenml artifact-store flavor register flavors.my_flavor.MyArtifactStoreFlavor
```
{% hint style="warning" %}
ZenML resolves the flavor class by taking the path where you initialized zenml (via `zenml init`) as the starting point of resolution. Therefore, please ensure you follow [the best practice](https://docs.zenml.io/user-guides/best-practices/iac) of initializing zenml at the root of your repository.
If ZenML does not find an initialized ZenML repository in any parent directory, it will default to the current working directory, but usually, it's better to not have to rely on this mechanism and initialize zenml at the root.
{% endhint %}
Afterward, you should see the new custom artifact store flavor in the list of available artifact store flavors:
```shell
zenml artifact-store flavor list
```
{% hint style="warning" %}
It is important to draw attention to when and how these base abstractions are coming into play in a ZenML workflow.
* The **CustomArtifactStoreFlavor** class is imported and utilized upon the creation of the custom flavor through the CLI.
* The **CustomArtifactStoreConfig** class is imported when someone tries to register/update a stack component with this custom flavor. Especially, during the registration process of the stack component, the config will be used to validate the values given by the user. As `Config` objects are inherently `pydantic` objects, you can also add your own custom validators here.
* The **CustomArtifactStore** only comes into play when the component is ultimately in use.
The design behind this interaction lets us separate the configuration of the flavor from its implementation. This way we can register flavors and components even when the major dependencies behind their implementation are not installed in our local setting (assuming the `CustomArtifactStoreFlavor` and the `CustomArtifactStoreConfig` are implemented in a different module/path than the actual `CustomArtifactStore`).
{% endhint %}
#### Enabling Artifact Visualizations with Custom Artifact Stores
ZenML automatically saves visualizations for many common data types and allows you to view these visualizations in the ZenML dashboard. Under the hood, this works by saving the visualizations together with the artifacts in the artifact store.
In order to load and display these visualizations, ZenML needs to be able to load and access the corresponding artifact store. This means that your custom artifact store needs to be configured in a way that allows authenticating to the back-end without relying on the local environment, e.g., by embedding the authentication credentials in the stack component configuration or by referencing a secret.
Furthermore, for deployed ZenML instances, you need to install the package dependencies of your artifact store implementation in the environment where you have deployed ZenML. See the [Documentation on deploying ZenML with custom Docker images](https://docs.zenml.io/getting-started/deploying-zenml/deploy-with-custom-image) for more information on how to do that.
---
### Book/Component Guide/Artifact Stores/Digitalocean (docs/book/component-guide/artifact-stores/digitalocean.md)
---
description: Storing artifacts in a DigitalOcean Spaces bucket.
---
# DigitalOcean Spaces
The DigitalOcean Spaces Artifact Store is an [Artifact Store](./) flavor provided with the DigitalOcean ZenML integration that uses [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces) to store ZenML artifacts. Spaces exposes an [S3-compatible API](https://docs.digitalocean.com/products/spaces/reference/s3-compatibility/), so this flavor reuses ZenML's S3 implementation under the hood and only adds a `region` setting from which the Spaces endpoint URL is derived at runtime.
### When would you want to use it?
Running ZenML pipelines with [the local Artifact Store](local.md) is usually sufficient if you just want to evaluate ZenML or get started quickly. However, the local Artifact Store becomes insufficient once you want to share pipeline results with teammates, run other stack components remotely (e.g. an orchestrator on DigitalOcean Kubernetes), or run production-grade pipelines at scale.
You should use the DigitalOcean Spaces Artifact Store when your infrastructure runs on DigitalOcean and you want your ZenML artifacts co-located with it — for example next to a DOKS cluster running the [Kubernetes orchestrator](https://docs.zenml.io/stacks/orchestrators/kubernetes). Because the integration speaks the S3-compatible Spaces API, you get the same workflow as the [S3 Artifact Store](s3.md) without managing any additional tooling. Consider one of the other [Artifact Store flavors](./#artifact-store-flavors) if you don't use DigitalOcean.
### How do you deploy it?
The DigitalOcean Spaces Artifact Store flavor is provided by the DigitalOcean ZenML integration. Install it on your local machine to be able to register the artifact store and add it to your stack:
```shell
zenml integration install digitalocean -y
```
You will also need a Space (bucket) and a set of Spaces access keys:
1. **Create a Space.** In the [DigitalOcean control panel](https://cloud.digitalocean.com/), go to **Spaces Object Storage** and create a bucket. Note the bucket name (this becomes the `s3://` path used by ZenML) and the region slug shown in the bucket's endpoint (e.g. `nyc3`, `ams3`, `fra1`).
2. **Generate Spaces access keys.** Go to **Storage > Spaces Object Storage > Access Keys** and generate a new key pair. Spaces keys are separate credentials from your DigitalOcean API token; copy both the access key and the secret when they are displayed.
### How do you use it?
Register the artifact store with the bucket path (using the `s3://` scheme, since Spaces speaks the S3 API) and the region slug:
```shell
zenml artifact-store register do_spaces \
--flavor=digitalocean \
--path=s3://my-space \
--region=fra1
# Register and set a stack that uses the new artifact store
zenml stack register do_stack -a do_spaces ... --set
```
The Spaces endpoint URL (`https://.digitaloceanspaces.com`) is derived from the `region` at runtime and is not persisted in the component configuration. If you need to address a different endpoint (for example a custom domain), pass it explicitly instead — an explicit endpoint always takes precedence over the region-derived one:
```shell
zenml artifact-store register do_spaces \
--flavor=digitalocean \
--path=s3://my-space \
--client_kwargs='{"endpoint_url": "https://fra1.digitaloceanspaces.com"}'
```
To authenticate, store the Spaces access keys in a [ZenML Secret](https://docs.zenml.io/getting-started/deploying-zenml/secret-management) and reference them on registration:
```shell
zenml secret create spaces_secret \
--access_key_id='' \
--secret_access_key=''
zenml artifact-store register do_spaces \
--flavor=digitalocean \
--path=s3://my-space \
--region=fra1 \
--authentication_secret=spaces_secret
```
All other configuration options of the [S3 Artifact Store](s3.md) (`client_kwargs`, `config_kwargs`, `s3_additional_kwargs`) are available on this flavor as well and are passed through to the underlying S3-compatible filesystem.
---
### Book/Component Guide/Artifact Stores/Gcp (docs/book/component-guide/artifact-stores/gcp.md)
---
description: Storing artifacts using GCP Cloud Storage.
---
# Google Cloud Storage (GCS)
The GCS Artifact Store is an [Artifact Store](./) flavor provided with the GCP ZenML integration that uses [the Google Cloud Storage managed object storage service](https://cloud.google.com/storage/docs/introduction) to store ZenML artifacts in a GCP Cloud Storage bucket.
### When would you want to use it?
Running ZenML pipelines with [the local Artifact Store](local.md) is usually sufficient if you just want to evaluate ZenML or get started quickly without incurring the trouble and the cost of employing cloud storage services in your stack. However, the local Artifact Store becomes insufficient or unsuitable if you have more elaborate needs for your project:
* if you want to share your pipeline run results with other team members or stakeholders inside or outside your organization
* if you have other components in your stack that are running remotely (e.g. a Kubeflow or Kubernetes Orchestrator running in a public cloud).
* if you outgrow what your local machine can offer in terms of storage space and need to use some form of private or public storage service that is shared with others
* if you are running pipelines at scale and need an Artifact Store that can handle the demands of production-grade MLOps
In all these cases, you need an Artifact Store that is backed by a form of public cloud or self-hosted shared object storage service.
You should use the GCS Artifact Store when you decide to keep your ZenML artifacts in a shared object storage and if you have access to the Google Cloud Storage managed service. You should consider one of the other [Artifact Store flavors](./#artifact-store-flavors) if you don't have access to the GCP Cloud Storage service.
### How do you deploy it?
{% hint style="info" %}
Would you like to skip ahead and deploy a full ZenML cloud stack already, including a GCS Artifact Store? Check out the[in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), or [the ZenML GCP Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register this stack component.
{% endhint %}
The GCS Artifact Store flavor is provided by the GCP ZenML integration, you need to install it on your local machine to be able to register a GCS Artifact Store and add it to your stack:
```shell
zenml integration install gcp -y
```
The only configuration parameter mandatory for registering a GCS Artifact Store is the root path URI, which needs to point to a GCS bucket and take the form `gs://bucket-name`. Please read [the Google Cloud Storage documentation](https://cloud.google.com/storage/docs/creating-buckets) on how to configure a GCS bucket.
With the URI to your GCS bucket known, registering a GCS Artifact Store can be done as follows:
```shell
# Register the GCS artifact store
zenml artifact-store register gs_store -f gcp --path=gs://bucket-name
# Register and set a stack with the new artifact store
zenml stack register custom_stack -a gs_store ... --set
```
Depending on your use case, however, you may also need to provide additional configuration parameters pertaining to [authentication](gcp.md#authentication-methods) to match your deployment scenario.
#### Authentication Methods
Integrating and using a GCS Artifact Store in your pipelines is not possible without employing some form of authentication. If you're looking for a quick way to get started locally, you can use the _Implicit Authentication_ method. However, the recommended way to authenticate to the GCP cloud platform is through [a GCP Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector). This is particularly useful if you are configuring ZenML stacks that combine the GCS Artifact Store with other remote stack components also running in GCP.
{% tabs %}
{% tab title="Implicit Authentication" %}
This method uses the implicit GCP authentication available _in the environment where the ZenML code is running_. On your local machine, this is the quickest way to configure a GCS Artifact Store. You don't need to supply credentials explicitly when you register the GCS Artifact Store, as it leverages the local credentials and configuration that the Google Cloud CLI stores on your local machine. However, you will need to install and set up the Google Cloud CLI on your machine as a prerequisite, as covered in [the Google Cloud documentation](https://cloud.google.com/sdk/docs/install-sdk) , before you register the GCS Artifact Store.
{% hint style="warning" %}
Certain dashboard functionality, such as visualizing or deleting artifacts, is not available when using an implicitly authenticated artifact store together with a deployed ZenML server because the ZenML server will not have permission to access the filesystem.
The implicit authentication method also needs to be coordinated with other stack components that are highly dependent on the Artifact Store and need to interact with it directly to the function. If these components are not running on your machine, they do not have access to the local Google Cloud CLI configuration and will encounter authentication failures while trying to access the GCS Artifact Store:
* [Orchestrators](https://docs.zenml.io/stacks/orchestrators/) need to access the Artifact Store to manage pipeline artifacts
* [Step Operators](https://docs.zenml.io/stacks/step-operators/) need to access the Artifact Store to manage step-level artifacts
* [Model Deployers](https://docs.zenml.io/stacks/model-deployers/) need to access the Artifact Store to load served models
To enable these use cases, it is recommended to use [a GCP Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector) to link your GCS Artifact Store to the remote GCS bucket.
{% endhint %}
{% endtab %}
{% tab title="GCP Service Connector (recommended)" %}
To set up the GCS Artifact Store to authenticate to GCP and access a GCS bucket, it is recommended to leverage the many features provided by [the GCP Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector) such as auto-configuration, best security practices regarding long-lived credentials and reusing the same credentials across multiple stack components.
If you don't already have a GCP Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command. You have the option to configure a GCP Service Connector that can be used to access more than one GCS bucket or even more than one type of GCP resource:
```sh
zenml service-connector register --type gcp -i
```
A non-interactive CLI example that leverages [the Google Cloud CLI configuration](https://cloud.google.com/sdk/docs/install-sdk) on your local machine to auto-configure a GCP Service Connector targeting a single GCS bucket is:
```sh
zenml service-connector register --type gcp --resource-type gcs-bucket --resource-name --auto-configure
```
{% code title="Example Command Output" %}
```
$ zenml service-connector register gcs-zenml-bucket-sl --type gcp --resource-type gcs-bucket --resource-id gs://zenml-bucket-sl --auto-configure
⠸ Registering service connector 'gcs-zenml-bucket-sl'...
Successfully registered service connector `gcs-zenml-bucket-sl` with access to the following resources:
┏━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠───────────────┼──────────────────────┨
┃ 📦 gcs-bucket │ gs://zenml-bucket-sl ┃
┗━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
> **Note**: Please remember to grant the entity associated with your GCP credentials permissions to read and write to your GCS bucket as well as to list accessible GCS buckets. For a full list of permissions required to use a GCP Service Connector to access one or more GCS buckets, please refer to the [GCP Service Connector GCS bucket resource type documentation](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector#gcs-bucket) or read the documentation available in the interactive CLI commands and dashboard. The GCP Service Connector supports [many different authentication methods](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector#authentication-methods) with different levels of security and convenience. You should pick the one that best fits your use case.
If you already have one or more GCP Service Connectors configured in your ZenML deployment, you can check which of them can be used to access the GCS bucket you want to use for your GCS Artifact Store by running e.g.:
```sh
zenml service-connector list-resources --resource-type gcs-bucket
```
{% code title="Example Command Output" %}
```
/* Detailed source-code truncated for AI context efficiency. */
```
{% endcode %}
After having set up or decided on a GCP Service Connector to use to connect to the target GCS bucket, you can register the GCS Artifact Store as follows:
```sh
# Register the GCS artifact-store and reference the target GCS bucket
zenml artifact-store register -f gcp \
--path='gs://your-bucket'
# Connect the GCS artifact-store to the target bucket via a GCP Service Connector
zenml artifact-store connect -i
```
A non-interactive version that connects the GCS Artifact Store to a target GCP Service Connector:
```sh
zenml artifact-store connect --connector
```
{% code title="Example Command Output" %}
```
$ zenml artifact-store connect gcs-zenml-bucket-sl --connector gcs-zenml-bucket-sl
Successfully connected artifact store `gcs-zenml-bucket-sl` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼─────────────────────┼────────────────┼───────────────┼──────────────────────┨
┃ 2a0bec1b-9787-4bd7-8d4a-9a47b6f61643 │ gcs-zenml-bucket-sl │ 🔵 gcp │ 📦 gcs-bucket │ gs://zenml-bucket-sl ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
As a final step, you can use the GCS Artifact Store in a ZenML Stack:
```sh
# Register and set a stack with the new artifact store
zenml stack register -a ... --set
```
{% endtab %}
{% tab title="GCP Credentials" %}
When you register the GCS Artifact Store, you can [generate a GCP Service Account Key](https://cloud.google.com/docs/authentication/application-default-credentials#attached-sa), store it in a [ZenML Secret](https://docs.zenml.io/getting-started/deploying-zenml/secret-management) and then reference it in the Artifact Store configuration.
This method has some advantages over the implicit authentication method:
* you don't need to install and configure the GCP CLI on your host
* you don't need to care about enabling your other stack components (orchestrators, step operators and model deployers) to have access to the artifact store through GCP Service Accounts and Workload Identity
* you can combine the GCS artifact store with other stack components that are not running in GCP
For this method, you need to [create a user-managed GCP service account](https://cloud.google.com/iam/docs/service-accounts-create), grant it minimal privileges to read and write to your GCS bucket, and then [create a service account key](https://cloud.google.com/iam/docs/keys-create-delete#creating).
{% hint style="info" %}
**Security Best Practice:** Instead of using the broad `Storage Object Admin` role, create a custom role with only the specific permissions needed:
- `storage.buckets.get`
- `storage.buckets.list`
- `storage.objects.create`
- `storage.objects.delete`
- `storage.objects.get`
- `storage.objects.list`
- `storage.objects.update`
Alternatively, you can use the `Storage Object Admin` role scoped to specific buckets rather than project-wide access.
{% endhint %}
With the service account key downloaded to a local file, you can register a ZenML secret and reference it in the GCS Artifact Store configuration as follows:
```shell
# Store the GCP credentials in a ZenML
zenml secret create gcp_secret \
--token=@path/to/service_account_key.json
# Register the GCS artifact store and reference the ZenML secret
zenml artifact-store register gcs_store -f gcp \
--path='gs://your-bucket' \
--authentication_secret=gcp_secret
# Register and set a stack with the new artifact store
zenml stack register custom_stack -a gs_store ... --set
```
{% endtab %}
{% endtabs %}
For more, up-to-date information on the GCS Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-gcp.html#zenml.integrations.gcp) .
### How do you use it?
Aside from the fact that the artifacts are stored in GCP Cloud Storage, using the GCS Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it).
---
### Book/Component Guide/Artifact Stores/Local (docs/book/component-guide/artifact-stores/local.md)
---
description: Storing artifacts on your local filesystem.
---
# Local Artifact Store
The local Artifact Store is a built-in ZenML [Artifact Store](./) flavor that uses a folder on your local filesystem to store artifacts.
### When would you want to use it?
The local Artifact Store is a great way to get started with ZenML, as it doesn't require you to provision additional local resources or to interact with managed object-store services like Amazon S3 and Google Cloud Storage. All you need is the local filesystem. You should use the local Artifact Store if you're just evaluating or getting started with ZenML, or if you are still in the experimental phase and don't need to share your pipeline artifacts (dataset, models, etc.) with others.
{% hint style="warning" %}
The local Artifact Store is not meant to be utilized in production. The local filesystem cannot be shared across your team and the artifacts stored in it cannot be accessed from other machines. This also means that [artifact visualizations](https://docs.zenml.io/how-to/data-artifact-management/visualize-artifacts/) will not be available when using a local Artifact Store through a [ZenML instance deployed in the cloud](https://docs.zenml.io/getting-started/deploying-zenml/).
Furthermore, the local Artifact Store doesn't cover services like high-availability, scalability, backup and restore and other features that are expected from a production grade MLOps system.
The fact that it stores artifacts on your local filesystem also means that not all stack components can be used in the same stack as a local Artifact Store:
* only [Orchestrators](https://docs.zenml.io/stacks/orchestrators/) running on the local machine, such as the [local Orchestrator](https://docs.zenml.io/stacks/orchestrators/local), a [local Kubeflow Orchestrator](https://docs.zenml.io/stacks/orchestrators/kubeflow), or a [local Kubernetes Orchestrator](https://docs.zenml.io/stacks/orchestrators/kubernetes) can be combined with a local Artifact Store
* only [Model Deployers](https://docs.zenml.io/stacks/model-deployers/) that are running locally, such as the [MLflow Model Deployer](https://docs.zenml.io/stacks/model-deployers/mlflow), can be used in combination with a local Artifact Store
* [Step Operators](https://docs.zenml.io/stacks/step-operators/): none of the Step Operators can be used in the same stack as a local Artifact Store, given that their very purpose is to run ZenML steps in remote specialized environments
As you transition to a team setting or a production setting, you can replace the local Artifact Store in your stack with one of the other flavors that are better suited for these purposes, with no changes required in your code.
{% endhint %}
### How do you deploy it?
The `default` stack that comes pre-configured with ZenML already contains a local Artifact Store:
```
/* Detailed source-code truncated for AI context efficiency. */
```
As shown by the `PATH` value in the `zenml artifact-store describe` output, the artifacts are stored inside a folder on your local filesystem.
You can create additional instances of local Artifact Stores and use them in your stacks as you see fit, e.g.:
```shell
# Register the local artifact store
zenml artifact-store register custom_local --flavor local
# Register and set a stack with the new artifact store
zenml stack register custom_stack -o default -a custom_local --set
```
{% hint style="warning" %}
Same as all other Artifact Store flavors, the local Artifact Store does take in a `path` configuration parameter that can be set during registration to point to a custom path on your machine. However, it is highly recommended that you rely on the default `path` value, otherwise, it may lead to unexpected results. Other local stack components depend on the convention used for the default path to be able to access the local Artifact Store.
{% endhint %}
For more, up-to-date information on the local Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-artifact_stores.html#zenml.artifact_stores.local_artifact_store) .
### How do you use it?
Aside from the fact that the artifacts are stored locally, using the local Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it).
---
### Book/Component Guide/Artifact Stores/Minio (docs/book/component-guide/artifact-stores/minio.md)
---
description: Storing artifacts in MinIO object storage.
---
# MinIO
[MinIO](https://min.io/) is a high-performance, S3-compatible object storage system. Since MinIO provides a fully S3-compatible API, you can use ZenML's S3 Artifact Store integration to connect to MinIO.
{% hint style="warning" %}
**Maintenance Mode**: The open-source MinIO project is currently in maintenance mode and is not accepting new changes. Only critical security fixes may be evaluated on a case-by-case basis. For development and testing purposes, MinIO remains a viable option, but for production use cases requiring active support, consider [MinIO AIStor](https://min.io/product/aistor) or alternative S3-compatible storage solutions like [Ceph RGW](https://ceph.io/en/discover/technology/#object).
{% endhint %}
### When would you want to use it?
You should use the MinIO Artifact Store when:
* You require self-hosted object storage for data sovereignty or compliance requirements
* Your MLOps infrastructure runs on-premises or in a private cloud environment
* You need S3-compatible storage co-located with your Kubernetes-based ZenML deployment
* You want to eliminate cloud vendor dependencies while maintaining S3 API compatibility
* You're developing locally and need a lightweight S3-compatible storage backend for testing
### How do you deploy it?
Since MinIO is S3-compatible, you'll use the S3 integration. First, install it:
```shell
zenml integration install s3 -y
```
You'll also need a running MinIO instance. MinIO can be deployed in various ways:
* **Docker**: `docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ":9001"`
* **Kubernetes**: Follow the instructions [here](https://docs.min.io/enterprise/aistor-object-store/installation/kubernetes/install/deploy-aistor-on-kubernetes/)
* **Binary**: Download from [MinIO's website](https://min.io/download)
### How do you configure it?
To use MinIO with ZenML, configure the S3 Artifact Store with your MinIO endpoint:
{% tabs %}
{% tab title="Using a ZenML Secret (recommended)" %}
First, create a ZenML secret with your MinIO credentials:
```shell
zenml secret create minio_secret \
--access_key_id='' \
--secret_access_key=''
```
Then register the artifact store:
```shell
zenml artifact-store register minio_store -f s3 \
--path='s3://your-bucket-name' \
--authentication_secret=minio_secret \
--client_kwargs='{"endpoint_url": "http://minio.example.com:9000"}'
```
{% endtab %}
{% endtabs %}
Replace `http://minio.example.com:9000` with your actual MinIO endpoint. If you're running MinIO locally for development, this might be `http://localhost:9000`.
{% hint style="info" %}
If your MinIO instance uses HTTPS with a self-signed certificate, you may need to configure SSL verification. Consult the [S3 Artifact Store documentation](s3.md#advanced-configuration) for advanced configuration options.
{% endhint %}
Finally, add the artifact store to your stack:
```shell
zenml stack register custom_stack -a minio_store ... --set
```
### How do you use it?
Using the MinIO Artifact Store is no different from [using any other flavor of Artifact Store](./#how-to-use-it). ZenML handles the S3-compatible API translation automatically.
For more details on the S3 Artifact Store configuration options, refer to the [S3 Artifact Store documentation](s3.md).
---
### Book/Component Guide/Artifact Stores/S3 (docs/book/component-guide/artifact-stores/s3.md)
---
description: Storing artifacts in an AWS S3 bucket.
---
# Amazon Simple Cloud Storage (S3)
The S3 Artifact Store is an [Artifact Store](./) flavor provided with the S3 ZenML integration that uses [the AWS S3 managed object storage service](https://aws.amazon.com/s3/) or one of the self-hosted S3 alternatives, such as [MinIO](https://min.io/) or [Ceph RGW](https://ceph.io/en/discover/technology/#object), to store artifacts in an S3 compatible object storage backend.
### When would you want to use it?
Running ZenML pipelines with [the local Artifact Store](local.md) is usually sufficient if you just want to evaluate ZenML or get started quickly without incurring the trouble and the cost of employing cloud storage services in your stack. However, the local Artifact Store becomes insufficient or unsuitable if you have more elaborate needs for your project:
* if you want to share your pipeline run results with other team members or stakeholders inside or outside your organization
* if you have other components in your stack that are running remotely (e.g. a Kubeflow or Kubernetes Orchestrator running in a public cloud).
* if you outgrow what your local machine can offer in terms of storage space and need to use some form of private or public storage service that is shared with others
* if you are running pipelines at scale and need an Artifact Store that can handle the demands of production-grade MLOps
In all these cases, you need an Artifact Store that is backed by a form of public cloud or self-hosted shared object storage service.
You should use the S3 Artifact Store when you decide to keep your ZenML artifacts in a shared object storage and if you have access to the AWS S3 managed service or one of the S3 compatible alternatives (e.g. Minio, Ceph RGW). You should consider one of the other [Artifact Store flavors](./#artifact-store-flavors) if you don't have access to an S3-compatible service.
### How do you deploy it?
{% hint style="info" %}
Would you like to skip ahead and deploy a full ZenML cloud stack already, including an S3 Artifact Store? Check out the[in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), or [the ZenML AWS Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register this stack component.
{% endhint %}
The S3 Artifact Store flavor is provided by the S3 ZenML integration, you need to install it on your local machine to be able to register an S3 Artifact Store and add it to your stack:
```shell
zenml integration install s3 -y
```
The only configuration parameter mandatory for registering an S3 Artifact Store is the root path URI, which needs to point to an S3 bucket and take the form `s3://bucket-name`. Please read the documentation relevant to the S3 service that you are using on how to create an S3 bucket. For example, the AWS S3 documentation is available [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/create-bucket-overview.html).
With the URI to your S3 bucket known, registering an S3 Artifact Store and using it in a stack can be done as follows:
```shell
# Register the S3 artifact-store
zenml artifact-store register s3_store -f s3 --path=s3://bucket-name
# Register and set a stack with the new artifact store
zenml stack register custom_stack -a s3_store ... --set
```
Depending on your use case, however, you may also need to provide additional configuration parameters pertaining to [authentication](s3.md#authentication-methods) or [pass advanced configuration parameters](s3.md#advanced-configuration) to match your S3-compatible service or deployment scenario.
#### Authentication Methods
Integrating and using an S3-compatible Artifact Store in your pipelines is not possible without employing some form of authentication. If you're looking for a quick way to get started locally, you can use the _Implicit Authentication_ method. However, the recommended way to authenticate to the AWS cloud platform is through [an AWS Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector). This is particularly useful if you are configuring ZenML stacks that combine the S3 Artifact Store with other remote stack components also running in AWS.
{% tabs %}
{% tab title="Implicit Authentication" %}
This method uses the implicit AWS authentication available _in the environment where the ZenML code is running_. On your local machine, this is the quickest way to configure an S3 Artifact Store. You don't need to supply credentials explicitly when you register the S3 Artifact Store, as it leverages the local credentials and configuration that the AWS CLI stores on your local machine. However, you will need to install and set up the AWS CLI on your machine as a prerequisite, as covered in [the AWS CLI documentation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), before you register the S3 Artifact Store.
{% hint style="warning" %}
Certain dashboard functionality, such as visualizing or deleting artifacts, is not available when using an implicitly authenticated artifact store together with a deployed ZenML server because the ZenML server will not have permission to access the filesystem.
The implicit authentication method also needs to be coordinated with other stack components that are highly dependent on the Artifact Store and need to interact with it directly to work. If these components are not running on your machine, they do not have access to the local AWS CLI configuration and will encounter authentication failures while trying to access the S3 Artifact Store:
* [Orchestrators](https://docs.zenml.io/stacks/orchestrators/) need to access the Artifact Store to manage pipeline artifacts
* [Step Operators](https://docs.zenml.io/stacks/step-operators/) need to access the Artifact Store to manage step-level artifacts
* [Model Deployers](https://docs.zenml.io/stacks/model-deployers/) need to access the Artifact Store to load served models
To enable these use-cases, it is recommended to use [an AWS Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector) to link your S3 Artifact Store to the remote S3 bucket.
{% endhint %}
{% endtab %}
{% tab title="AWS Service Connector (recommended)" %}
To set up the S3 Artifact Store to authenticate to AWS and access an S3 bucket, it is recommended to leverage the many features provided by [the AWS Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector) such as auto-configuration, best security practices regarding long-lived credentials and fine-grained access control and reusing the same credentials across multiple stack components.
If you don't already have an AWS Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command. You have the option to configure an AWS Service Connector that can be used to access more than one S3 bucket or even more than one type of AWS resource:
```sh
zenml service-connector register --type aws -i
```
A non-interactive CLI example that leverages [the AWS CLI configuration](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) on your local machine to auto-configure an AWS Service Connector targeting a single S3 bucket is:
```sh
zenml service-connector register --type aws --resource-type s3-bucket --resource-name --auto-configure
```
{% code title="Example Command Output" %}
```
$ zenml service-connector register s3-zenfiles --type aws --resource-type s3-bucket --resource-id s3://zenfiles --auto-configure
⠸ Registering service connector 's3-zenfiles'...
Successfully registered service connector `s3-zenfiles` with access to the following resources:
┏━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠───────────────┼────────────────┨
┃ 📦 s3-bucket │ s3://zenfiles ┃
┗━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
```
{% endcode %}
> **Note**: Please remember to grant the entity associated with your AWS credentials permissions to read and write to your S3 bucket as well as to list accessible S3 buckets. For a full list of permissions required to use an AWS Service Connector to access one or more S3 buckets, please refer to the [AWS Service Connector S3 bucket resource type documentation](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector#s3-bucket) or read the documentation available in the interactive CLI commands and dashboard. The AWS Service Connector supports [many different authentication methods](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector#authentication-methods) with different levels of security and convenience. You should pick the one that best fits your use case.
If you already have one or more AWS Service Connectors configured in your ZenML deployment, you can check which of them can be used to access the S3 bucket you want to use for your S3 Artifact Store by running e.g.:
```sh
zenml service-connector list-resources --resource-type s3-bucket
```
{% code title="Example Command Output" %}
```
/* Detailed source-code truncated for AI context efficiency. */
```
{% endcode %}
After having set up or decided on an AWS Service Connector to use to connect to the target S3 bucket, you can register the S3 Artifact Store as follows:
```sh
# Register the S3 artifact-store and reference the target S3 bucket
zenml artifact-store register -f s3 \
--path='s3://your-bucket'
# Connect the S3 artifact-store to the target bucket via an AWS Service Connector
zenml artifact-store connect -i
```
A non-interactive version that connects the S3 Artifact Store to a target S3 bucket through an AWS Service Connector:
```sh
zenml artifact-store connect --connector
```
{% code title="Example Command Output" %}
```
$ zenml artifact-store connect s3-zenfiles --connector s3-zenfiles
Successfully connected artifact store `s3-zenfiles` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼───────────────┼────────────────┨
┃ c4ee3f0a-bc69-4c79-9a74-297b2dd47d50 │ s3-zenfiles │ 🔶 aws │ 📦 s3-bucket │ s3://zenfiles ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
```
{% endcode %}
As a final step, you can use the S3 Artifact Store in a ZenML Stack:
```sh
# Register and set a stack with the new artifact store
zenml stack register -a ... --set
```
{% endtab %}
{% tab title="ZenML Secret" %}
When you register the S3 Artifact Store, you can [generate an AWS access key](https://docs.aws.amazon.com/cli/latest/reference/iam/create-access-key.html), store it in a [ZenML Secret](https://docs.zenml.io/getting-started/deploying-zenml/secret-management) and then reference it in the Artifact Store configuration.
This method has some advantages over the implicit authentication method:
* you don't need to install and configure the AWS CLI on your host
* you don't need to care about enabling your other stack components (orchestrators, step operators, and model deployers) to have access to the artifact store through IAM roles and policies
* you can combine the S3 artifact store with other stack components that are not running in AWS
> **Note**: When you create the IAM user for your AWS access key, please remember to grant the created IAM user permissions to read and write to your S3 bucket (i.e. at a minimum: `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`, `s3:DeleteObject`, `s3:GetBucketVersioning`, `s3:ListBucketVersions`, `s3:DeleteObjectVersion`)
After having set up the IAM user and generated the access key, as described in the [AWS documentation](https://docs.aws.amazon.com/cli/latest/reference/iam/create-access-key.html), you can register the S3 Artifact Store as follows:
```shell
# Store the AWS access key in a ZenML secret
zenml secret create s3_secret \
--access_key_id='' \
--secret_access_key=''
# Register the S3 artifact-store and reference the ZenML secret
zenml artifact-store register s3_store -f s3 \
--path='s3://your-bucket' \
--authentication_secret=s3_secret
# Register and set a stack with the new artifact store
zenml stack register custom_stack -a s3_store ... --set
```
{% endtab %}
{% endtabs %}
#### Advanced Configuration
The S3 Artifact Store accepts a range of advanced configuration options that can be used to further customize how ZenML connects to the S3 storage service that you are using. These are accessible via the `client_kwargs`, `config_kwargs` and `s3_additional_kwargs` configuration attributes and are passed transparently to [the underlying S3Fs library](https://s3fs.readthedocs.io/en/latest/#s3-compatible-storage):
* `client_kwargs`: arguments that will be transparently passed to [the botocore client](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html#boto3.session.Session.client) . You can use it to configure parameters like `endpoint_url` and `region_name` when connecting to an S3-compatible endpoint (e.g. Minio).
* `config_kwargs`: advanced parameters passed to [botocore.client.Config](https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html).
* `s3_additional_kwargs`: advanced parameters that are used when calling S3 API, typically used for things like `ServerSideEncryption` and `ACL`.
To include these advanced parameters in your Artifact Store configuration, pass them using JSON format during registration, e.g.:
```shell
zenml artifact-store register minio_store -f s3 \
--path='s3://minio_bucket' \
--authentication_secret=s3_secret \
--client_kwargs='{"endpoint_url": "http://minio.cluster.local:9000", "region_name": "us-east-1"}'
```
For more, up-to-date information on the S3 Artifact Store implementation and its configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-s3.html#zenml.integrations.s3) .
### How do you use it?
Aside from the fact that the artifacts are stored in an S3 compatible backend, using the S3 Artifact Store is no different than [using any other flavor of Artifact Store](./#how-to-use-it).
---
### Book/Component Guide/Container Registries/README (docs/book/component-guide/container-registries/README.md)
---
description: Setting up a storage for Docker images.
icon: box
---
# Container Registries
The container registry is an essential part of most remote MLOps stacks. It is used to store container images that are built to run machine learning pipelines in remote environments. Containerization of the pipeline code creates a portable environment that allows code to run in an isolated manner.
### When to use it
The container registry is needed whenever other components of your stack need to push or pull container images. Currently, this is the case for most of ZenML's remote [orchestrators](https://docs.zenml.io/stacks/orchestrators/) , [step operators](https://docs.zenml.io/stacks/step-operators/), and some [model deployers](https://docs.zenml.io/stacks/model-deployers/). These containerize your pipeline code and therefore require a container registry to store the resulting [Docker](https://www.docker.com/) images. Take a look at the documentation page of the component you want to use in your stack to see if it requires a container registry or even a specific container registry flavor.
### Container Registry Flavors
ZenML comes with a few container registry flavors that you can use:
* Default flavor: Allows any URI without validation. Use this if you want to use a local container registry or when using a remote container registry that is not covered by other flavors.
* Specific flavors: Validates your container registry URI and performs additional checks to ensure you're able to push to the registry.
{% hint style="warning" %}
We highly suggest using the specific container registry flavors in favor of the `default` one to make use of the additional URI validations.
{% endhint %}
| Container Registry | Flavor | Integration | URI example |
| ------------------------------------------ | ----------- | ----------- | ----------------------------------------- |
| [DefaultContainerRegistry](default.md) | `default` | _built-in_ | - |
| [DockerHubContainerRegistry](dockerhub.md) | `dockerhub` | _built-in_ | docker.io/zenml |
| [GCPContainerRegistry](gcp.md) | `gcp` | _built-in_ | gcr.io/zenml |
| [AzureContainerRegistry](azure.md) | `azure` | _built-in_ | zenml.azurecr.io |
| [GitHubContainerRegistry](github.md) | `github` | _built-in_ | ghcr.io/zenml |
| [AWSContainerRegistry](aws.md) | `aws` | `aws` | 123456789.dkr.ecr.us-east-1.amazonaws.com |
If you would like to see the available flavors of container registries, you can use the command:
```shell
zenml container-registry flavor list
```
---
### Book/Component Guide/Container Registries/Aws (docs/book/component-guide/container-registries/aws.md)
---
description: Storing container images in Amazon ECR.
---
# Amazon Elastic Container Registry (ECR)
The AWS container registry is a [container registry](./) flavor provided with the ZenML `aws` integration and uses [Amazon ECR](https://aws.amazon.com/ecr/) to store container images.
### When to use it
You should use the AWS container registry if:
* one or more components of your stack need to pull or push container images.
* you have access to AWS ECR. If you're not using AWS, take a look at the other [container registry flavors](./#container-registry-flavors).
### How to deploy it
{% hint style="info" %}
Would you like to skip ahead and deploy a full ZenML cloud stack already, including an AWS ECR container registry? Check out the[in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), or [the ZenML AWS Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register this stack component.
{% endhint %}
The ECR registry is automatically activated once you create an AWS account. However, you'll need to create a `Repository` in order to push container images to it:
* Go to the [ECR website](https://console.aws.amazon.com/ecr).
* Make sure the correct region is selected on the top right.
* Click on `Create repository`.
* Create a private repository. The name of the repository depends on the [orchestrator](https://docs.zenml.io/stacks/orchestrators/) or [step operator](https://docs.zenml.io/stacks/step-operators/) you're using in your stack.
### URI format
The AWS container registry URI should have the following format:
```shell
.dkr.ecr..amazonaws.com
# Examples:
123456789.dkr.ecr.eu-west-2.amazonaws.com
987654321.dkr.ecr.ap-south-1.amazonaws.com
135792468.dkr.ecr.af-south-1.amazonaws.com
```
To figure out the URI for your registry:
* Go to the [AWS console](https://console.aws.amazon.com/) and click on your user account in the top right to see the `Account ID`.
* Go [here](https://docs.aws.amazon.com/general/latest/gr/rande.html#regional-endpoints) and choose the region in which you would like to store your container images. Make sure to choose a nearby region for faster access.
* Once you have both these values, fill in the values in this template `.dkr.ecr..amazonaws.com` to get your container registry URI.
### How to use it
To use the AWS container registry, we need:
* The ZenML `aws` integration installed. If you haven't done so, run
```shell
zenml integration install aws
```
* [Docker](https://www.docker.com) installed and running.
* The registry URI. Check out the [previous section](aws.md#how-to-deploy-it) on the URI format and how to get the URI for your registry.
We can then register the container registry and use it in our active stack:
```shell
zenml container-registry register \
--flavor=aws \
--uri=
# Add the container registry to the active stack
zenml stack update -c
```
You also need to set up [authentication](aws.md#authentication-methods) required to log in to the container registry.
#### Authentication Methods
Integrating and using an AWS Container Registry in your pipelines is not possible without employing some form of authentication. If you're looking for a quick way to get started locally, you can use the _Local Authentication_ method. However, the recommended way to authenticate to the AWS cloud platform is through [an AWS Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector). This is particularly useful if you are configuring ZenML stacks that combine the AWS Container Registry with other remote stack components also running in AWS.
{% tabs %}
{% tab title="Local Authentication" %}
This method uses the Docker client authentication available _in the environment where the ZenML code is running_. On your local machine, this is the quickest way to configure an AWS Container Registry. You don't need to supply credentials explicitly when you register the AWS Container Registry, as it leverages the local credentials and configuration that the AWS CLI and Docker client store on your local machine. However, you will need to install and set up the AWS CLI on your machine as a prerequisite, as covered in [the AWS CLI documentation](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), before you register the AWS Container Registry.
With the AWS CLI installed and set up with credentials, we'll need to log in to the container registry so Docker can pull and push images:
```shell
# Fill your REGISTRY_URI and REGION in the placeholders in the following command.
# You can find the REGION as part of your REGISTRY_URI: `.dkr.ecr..amazonaws.com`
aws ecr get-login-password --region | docker login --username AWS --password-stdin
```
{% hint style="warning" %}
Stacks using the AWS Container Registry set up with local authentication are not portable across environments. To make ZenML pipelines fully portable, it is recommended to use [an AWS Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector) to link your AWS Container Registry to the remote ECR registry.
{% endhint %}
{% endtab %}
{% tab title="AWS Service Connector (recommended)" %}
To set up the AWS Container Registry to authenticate to AWS and access an ECR registry, it is recommended to leverage the many features provided by [the AWS Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector) such as auto-configuration, local login, best security practices regarding long-lived credentials and fine-grained access control and reusing the same credentials across multiple stack components.
If you don't already have an AWS Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command. You have the option to configure an AWS Service Connector that can be used to access an ECR registry or even more than one type of AWS resource:
```sh
zenml service-connector register --type aws -i
```
A non-interactive CLI example that leverages [the AWS CLI configuration](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) on your local machine to auto-configure an AWS Service Connector targeting an ECR registry is:
```sh
zenml service-connector register --type aws --resource-type docker-registry --auto-configure
```
{% code title="Example Command Output" %}
```
$ zenml service-connector register aws-us-east-1 --type aws --resource-type docker-registry --auto-configure
⠸ Registering service connector 'aws-us-east-1'...
Successfully registered service connector `aws-us-east-1` with access to the following resources:
┏━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠────────────────────┼──────────────────────────────────────────────┨
┃ 🐳 docker-registry │ 715803424590.dkr.ecr.us-east-1.amazonaws.com ┃
┗━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
> **Note**: Please remember to grant the entity associated with your AWS credentials permissions to read and write to one or more ECR repositories as well as to list accessible ECR repositories. For a full list of permissions required to use an AWS Service Connector to access an ECR registry, please refer to the [AWS Service Connector ECR registry resource type documentation](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector#ecr-container-registry) or read the documentation available in the interactive CLI commands and dashboard. The AWS Service Connector supports [many different authentication methods](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector#authentication-methods) with different levels of security and convenience. You should pick the one that best fits your use case.
If you already have one or more AWS Service Connectors configured in your ZenML deployment, you can check which of them can be used to access the ECR registry you want to use for your AWS Container Registry by running e.g.:
```sh
zenml service-connector list-resources --connector-type aws --resource-type docker-registry
```
{% code title="Example Command Output" %}
```
The following 'docker-registry' resources can be accessed by service connectors that you have configured:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼─────────────────────────┼────────────────┼────────────────────┼──────────────────────────────────────────────┨
┃ 37c97fa0-fa47-4d55-9970-e2aa6e1b50cf │ aws-secret-key │ 🔶 aws │ 🐳 docker-registry │ 715803424590.dkr.ecr.us-east-1.amazonaws.com ┃
┠──────────────────────────────────────┼─────────────────────────┼────────────────┼────────────────────┼──────────────────────────────────────────────┨
┃ d400e0c6-a8e7-4b95-ab34-0359229c5d36 │ aws-us-east-1 │ 🔶 aws │ 🐳 docker-registry │ 715803424590.dkr.ecr.us-east-1.amazonaws.com ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
After having set up or decided on an AWS Service Connector to use to connect to the target ECR registry, you can register the AWS Container Registry as follows:
```sh
# Register the AWS container registry and reference the target ECR registry URI
zenml container-registry register -f aws \
--uri=
# Connect the AWS container registry to the target ECR registry via an AWS Service Connector
zenml container-registry connect -i
```
A non-interactive version that connects the AWS Container Registry to a target ECR registry through an AWS Service Connector:
```sh
zenml container-registry connect --connector
```
{% code title="Example Command Output" %}
```
$ zenml container-registry connect aws-us-east-1 --connector aws-us-east-1
Successfully connected container registry `aws-us-east-1` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────────┼──────────────────────────────────────────────┨
┃ d400e0c6-a8e7-4b95-ab34-0359229c5d36 │ aws-us-east-1 │ 🔶 aws │ 🐳 docker-registry │ 715803424590.dkr.ecr.us-east-1.amazonaws.com ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
As a final step, you can use the AWS Container Registry in a ZenML Stack:
```sh
# Register and set a stack with the new container registry
zenml stack register -c ... --set
```
{% hint style="info" %}
Linking the AWS Container Registry to a Service Connector means that your local Docker client is no longer authenticated to access the remote registry. If you need to manually interact with the remote registry via the Docker CLI, you can use the [local login Service Connector feature](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/service-connectors-guide#configure-local-clients) to temporarily authenticate your local Docker client to the remote registry:
```sh
zenml service-connector login --resource-type docker-registry
```
{% code title="Example Command Output" %}
```
$ zenml service-connector login aws-us-east-1 --resource-type docker-registry
⠼ Attempting to configure local client using service connector 'aws-us-east-1'...
WARNING! Your password will be stored unencrypted in /home/stefan/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
The 'aws-us-east-1' Docker Service Connector was used to successfully configure the local Docker/OCI container registry client/SDK.
```
{% endcode %}
{% endhint %}
{% endtab %}
{% endtabs %}
For more information and a full list of configurable attributes of the AWS container registry, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-aws.html#zenml.integrations.aws).
---
### Book/Component Guide/Container Registries/Azure (docs/book/component-guide/container-registries/azure.md)
---
description: Storing container images in Azure.
---
# Azure Container Registry
The Azure container registry is a [container registry](./) flavor that comes built-in with ZenML and uses the [Azure Container Registry](https://azure.microsoft.com/en-us/services/container-registry/) to store container images.
### When to use it
You should use the Azure container registry if:
* one or more components of your stack need to pull or push container images.
* you have access to Azure. If you're not using Azure, take a look at the other [container registry flavors](./#container-registry-flavors).
### How to deploy it
{% hint style="info" %}
Would you like to skip ahead and deploy a full ZenML cloud stack already, including an Azure container registry? Check out the[in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), or [the ZenML Azure Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register this stack component.
{% endhint %}
Go [here](https://portal.azure.com/#create/Microsoft.ContainerRegistry) and choose a subscription, resource group, location, and registry name. Then click on `Review + Create` and to create your container registry.
### How to find the registry URI
The Azure container registry URI should have the following format:
```shell
.azurecr.io
# Examples:
zenmlregistry.azurecr.io
myregistry.azurecr.io
```
To figure out the URI for your registry:
* Go to the [Azure portal](https://portal.azure.com/#home).
* In the search bar, enter `container registries` and select the container registry you want to use. If you don't have any container registries yet, check out the [deployment section](azure.md#how-to-deploy-it) on how to create one.
* Use the name of your registry to fill the template `.azurecr.io` and get your URI.
### How to use it
To use the Azure container registry, we need:
* [Docker](https://www.docker.com) installed and running.
* The registry URI. Check out the [previous section](azure.md#how-to-find-the-registry-uri) on the URI format and how to get the URI for your registry.
We can then register the container registry and use it in our active stack:
```shell
zenml container-registry register \
--flavor=azure \
--uri=
# Add the container registry to the active stack
zenml stack update -c
```
You also need to set up [authentication](azure.md#authentication-methods) required to log in to the container registry.
#### Authentication Methods
Integrating and using an Azure Container Registry in your pipelines is not possible without employing some form of authentication. If you're looking for a quick way to get started locally, you can use the _Local Authentication_ method. However, the recommended way to authenticate to the Azure cloud platform is through [an Azure Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector). This is particularly useful if you are configuring ZenML stacks that combine the Azure Container Registry with other remote stack components also running in Azure.
{% tabs %}
{% tab title="Local Authentication" %}
This method uses the Docker client authentication available _in the environment where the ZenML code is running_. On your local machine, this is the quickest way to configure an Azure Container Registry. You don't need to supply credentials explicitly when you register the Azure Container Registry, as it leverages the local credentials and configuration that the Azure CLI and Docker client store on your local machine. However, you will need to install and set up the Azure CLI on your machine as a prerequisite, as covered in [the Azure CLI documentation](https://docs.microsoft.com/en-us/cli/azure/install-azure-cli), before you register the Azure Container Registry.
With the Azure CLI installed and set up with credentials, you need to login to the container registry so Docker can pull and push images:
```shell
# Fill your REGISTRY_NAME in the placeholder in the following command.
# You can find the REGISTRY_NAME as part of your registry URI: `.azurecr.io`
az acr login --name=
```
{% hint style="warning" %}
Stacks using the Azure Container Registry set up with local authentication are not portable across environments. To make ZenML pipelines fully portable, it is recommended to use [an Azure Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector) to link your Azure Container Registry to the remote ACR registry.
{% endhint %}
{% endtab %}
{% tab title="Azure Service Connector (recommended)" %}
To set up the Azure Container Registry to authenticate to Azure and access an ACR registry, it is recommended to leverage the many features provided by [the Azure Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector) such as auto-configuration, local login, best security practices regarding long-lived credentials and reusing the same credentials across multiple stack components.
If you don't already have an Azure Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command. You have the option to configure an Azure Service Connector that can be used to access a ACR registry or even more than one type of Azure resource:
```sh
zenml service-connector register --type azure -i
```
A non-interactive CLI example that uses [Azure Service Principal credentials](https://learn.microsoft.com/en-us/azure/active-directory/develop/app-objects-and-service-principals) to configure an Azure Service Connector targeting a single ACR registry is:
```sh
zenml service-connector register --type azure --auth-method service-principal --tenant_id= --client_id= --client_secret= --resource-type docker-registry --resource-id
```
{% code title="Example Command Output" %}
```
$ zenml service-connector register azure-demo --type azure --auth-method service-principal --tenant_id=a79f3633-8f45-4a74-a42e-68871c17b7fb --client_id=8926254a-8c3f-430a-a2fd-bdab234d491e --client_secret=AzureSuperSecret --resource-type docker-registry --resource-id demozenmlcontainerregistry.azurecr.io
⠸ Registering service connector 'azure-demo'...
Successfully registered service connector `azure-demo` with access to the following resources:
┏━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠────────────────────┼───────────────────────────────────────┨
┃ 🐳 docker-registry │ demozenmlcontainerregistry.azurecr.io ┃
┗━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
> **Note**: Please remember to grant the entity associated with your Azure credentials permissions to read and write to your ACR registry as well as to list accessible ACR registries. For a full list of permissions required to use an Azure Service Connector to access a ACR registry, please refer to the [Azure Service Connector ACR registry resource type documentation](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector#acr-container-registry) or read the documentation available in the interactive CLI commands and dashboard. The Azure Service Connector supports [many different authentication methods](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/azure-service-connector#authentication-methods) with different levels of security and convenience. You should pick the one that best fits your use case.
If you already have one or more Azure Service Connectors configured in your ZenML deployment, you can check which of them can be used to access the ACR registry you want to use for your Azure Container Registry by running e.g.:
```sh
zenml service-connector list-resources --connector-type azure --resource-type docker-registry
```
{% code title="Example Command Output" %}
```
The following 'docker-registry' resources can be accessed by 'azure' service connectors that you have configured:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────────┼───────────────────────────────────────┨
┃ db5821d0-a658-4504-ae96-04c3302d8f85 │ azure-demo │ 🇦 azure │ 🐳 docker-registry │ demozenmlcontainerregistry.azurecr.io ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
After having set up or decided on an Azure Service Connector to use to connect to the target ACR registry, you can register the Azure Container Registry as follows:
```sh
# Register the Azure container registry and reference the target ACR registry URI
zenml container-registry register -f azure \
--uri=
# Connect the Azure container registry to the target ACR registry via an Azure Service Connector
zenml container-registry connect -i
```
A non-interactive version that connects the Azure Container Registry to a target ACR registry through an Azure Service Connector:
```sh
zenml container-registry connect --connector
```
{% code title="Example Command Output" %}
```
$ zenml container-registry connect azure-demo --connector azure-demo
Successfully connected container registry `azure-demo` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────────┼───────────────────────────────────────┨
┃ db5821d0-a658-4504-ae96-04c3302d8f85 │ azure-demo │ 🇦 azure │ 🐳 docker-registry │ demozenmlcontainerregistry.azurecr.io ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
As a final step, you can use the Azure Container Registry in a ZenML Stack:
```sh
# Register and set a stack with the new container registry
zenml stack register -c ... --set
```
{% hint style="info" %}
Linking the Azure Container Registry to a Service Connector means that your local Docker client is no longer authenticated to access the remote registry. If you need to manually interact with the remote registry via the Docker CLI, you can use the [local login Service Connector feature](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/service-connectors-guide#configure-local-clients) to temporarily authenticate your local Docker client to the remote registry:
```sh
zenml service-connector login --resource-type docker-registry --resource-id
```
{% code title="Example Command Output" %}
```
$ zenml service-connector login azure-demo --resource-type docker-registry --resource-id demozenmlcontainerregistry.azurecr.io
⠹ Attempting to configure local client using service connector 'azure-demo'...
WARNING! Your password will be stored unencrypted in /home/stefan/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
The 'azure-demo' Docker Service Connector was used to successfully configure the local Docker/OCI container registry client/SDK.
```
{% endcode %}
{% endhint %}
{% endtab %}
{% endtabs %}
For more information and a full list of configurable attributes of the Azure container registry, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-container_registries.html#zenml.container_registries.azure_container_registry) .
---
### Book/Component Guide/Container Registries/Custom (docs/book/component-guide/container-registries/custom.md)
---
description: Learning how to develop a custom container registry.
---
# Develop a custom container registry
{% hint style="info" %}
Before diving into the specifics of this component type, it is beneficial to familiarize yourself with our [general guide to writing custom component flavors in ZenML](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/implement-a-custom-stack-component). This guide provides an essential understanding of ZenML's component flavor concepts.
{% endhint %}
### Base Abstraction
In the current version of ZenML, container registries have a rather basic base abstraction. In essence, their base configuration only features a `uri` and their implementation features a non-abstract `prepare_image_push` method for validation.
```python
from abc import abstractmethod
from typing import Type
from zenml.enums import StackComponentType
from zenml.stack import Flavor
from zenml.stack.authentication_mixin import (
AuthenticationConfigMixin,
AuthenticationMixin,
)
class BaseContainerRegistryConfig(AuthenticationConfigMixin):
"""Base config for a container registry."""
uri: str
class BaseContainerRegistry(AuthenticationMixin):
"""Base class for all ZenML container registries."""
def prepare_image_push(self, image_name: str) -> None:
"""Conduct necessary checks/preparations before an image gets pushed."""
class BaseContainerRegistryFlavor(Flavor):
"""Base flavor for container registries."""
@property
@abstractmethod
def name(self) -> str:
"""Returns the name of the flavor."""
@property
def type(self) -> StackComponentType:
"""Returns the flavor type."""
return StackComponentType.CONTAINER_REGISTRY
@property
def config_class(self) -> Type[BaseContainerRegistryConfig]:
"""Config class for this flavor."""
return BaseContainerRegistryConfig
@property
def implementation_class(self) -> Type[BaseContainerRegistry]:
"""Implementation class."""
return BaseContainerRegistry
```
{% hint style="info" %}
This is a slimmed-down version of the base implementation which aims to highlight the abstraction layer. In order to see the full implementation and get the complete docstrings, please check the [SDK docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-container_registries.html#zenml.container_registries.base_container_registry) .
{% endhint %}
### Building your own container registry
If you want to create your own custom flavor for a container registry, you can follow the following steps:
1. Create a class that inherits from the `BaseContainerRegistry` class and if you need to execute any checks/validation before the image gets pushed, you can define these operations in the `prepare_image_push` method. As an example, you can check the `AWSContainerRegistry`.
2. If you need further configuration, you can create a class which inherits from the `BaseContainerRegistryConfig` class.
3. Bring both the implementation and the configuration together by inheriting from the `BaseContainerRegistryFlavor` class.
Once you are done with the implementation, you can register it through the CLI. Please ensure you **point to the flavor class via dot notation**:
```shell
zenml container-registry flavor register
```
For example, your flavor class `MyContainerRegistryFlavor` is defined in `flavors/my_flavor.py`, you'd register it by doing:
```shell
zenml container-registry flavor register flavors.my_flavor.MyContainerRegistryFlavor
```
{% hint style="warning" %}
ZenML resolves the flavor class by taking the path where you initialized zenml (via `zenml init`) as the starting point of resolution. Therefore, please ensure you follow [the best practice](https://docs.zenml.io/user-guides/best-practices/iac) of initializing zenml at the root of your repository.
If ZenML does not find an initialized ZenML repository in any parent directory, it will default to the current working directory, but usually it's better to not have to rely on this mechanism, and initialize zenml at the root.
{% endhint %}
Afterward, you should see the new flavor in the list of available flavors:
```shell
zenml container-registry flavor list
```
{% hint style="warning" %}
It is important to draw attention to when and how these base abstractions are coming into play in a ZenML workflow.
* The **CustomContainerRegistryFlavor** class is imported and utilized upon the creation of the custom flavor through the CLI.
* The **CustomContainerRegistryConfig** class is imported when someone tries to register/update a stack component with this custom flavor. Especially, during the registration process of the stack component, the config will be used to validate the values given by the user. As `Config` object are inherently `pydantic` objects, you can also add your own custom validators here.
* The **CustomContainerRegistry** only comes into play when the component is ultimately in use.
The design behind this interaction lets us separate the configuration of the flavor from its implementation. This way we can register flavors and components even when the major dependencies behind their implementation are not installed in our local setting (assuming the `CustomContainerRegistryFlavor` and the `CustomContainerRegistryConfig` are implemented in a different module/path than the actual `CustomContainerRegistry`).
{% endhint %}
---
### Book/Component Guide/Container Registries/Default (docs/book/component-guide/container-registries/default.md)
---
description: Storing container images locally.
---
# Default Container Registry
The Default container registry is a [container registry](./) flavor that comes built-in with ZenML and allows container registry URIs of any format.
### When to use it
You should use the Default container registry if you want to use a **local** container registry or when using a remote container registry that is not covered by other [container registry flavors](./#container-registry-flavors).
### Local registry URI format
To specify a URI for a local container registry, use the following format:
```shell
localhost:
# Examples:
localhost:5000
localhost:8000
localhost:9999
```
### How to use it
To use the Default container registry, we need:
* [Docker](https://www.docker.com) installed and running.
* The registry URI. If you're using a local container registry, check out
* the [previous section](default.md#local-registry-uri-format) on the URI format.
We can then register the container registry and use it in our active stack:
```shell
zenml container-registry register \
--flavor=default \
--uri=
# Add the container registry to the active stack
zenml stack update -c
```
You may also need to set up [authentication](default.md#authentication-methods) required to log in to the container registry.
#### Authentication Methods
If you are using a private container registry, you will need to configure some form of authentication to login to the registry. If you're looking for a quick way to get started locally, you can use the _Local Authentication_ method. However, the recommended way to authenticate to a remote private container registry is through [a Docker Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/docker-service-connector).
If your target private container registry comes from a cloud provider like AWS, GCP or Azure, you should use the [container registry flavor](./#container-registry-flavors) targeted at that cloud provider. For example, if you're using AWS, you should use the [AWS Container Registry](aws.md) flavor. These cloud provider flavors also use specialized cloud provider Service Connectors to authenticate to the container registry.
{% tabs %}
{% tab title="Local Authentication" %}
This method uses the Docker client authentication available _in the environment where the ZenML code is running_. On your local machine, this is the quickest way to configure a Default Container Registry. You don't need to supply credentials explicitly when you register the Default Container Registry, as it leverages the local credentials and configuration that the Docker client stores on your local machine.
To log in to the container registry so Docker can pull and push images, you'll need to run the `docker login` command and supply your credentials, e.g.:
```shell
docker login --username --password-stdin
```
{% hint style="warning" %}
Stacks using the Default Container Registry set up with local authentication are not portable across environments. To make ZenML pipelines fully portable, it is recommended to use [a Docker Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/docker-service-connector) to link your Default Container Registry to the remote private container registry.
{% endhint %}
{% endtab %}
{% tab title="Docker Service Connector (recommended)" %}
To set up the Default Container Registry to authenticate to and access a private container registry, it is recommended to leverage the features provided by [the Docker Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/docker-service-connector) such as local login and reusing the same credentials across multiple stack components.
If you don't already have a Docker Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command:
```sh
zenml service-connector register --type docker -i
```
A non-interactive CLI example is:
```sh
zenml service-connector register --type docker --username= --password=
```
{% code title="Example Command Output" %}
```
$ zenml service-connector register dockerhub --type docker --username=username --password=password
Successfully registered service connector `dockerhub` with access to the following resources:
┏━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠────────────────────┼────────────────┨
┃ 🐳 docker-registry │ docker.io ┃
┗━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
```
{% endcode %}
If you already have one or more Docker Service Connectors configured in your ZenML deployment, you can check which of them can be used to access the container registry you want to use for your Default Container Registry by running e.g.:
```sh
zenml service-connector list-resources --connector-type docker --resource-id
```
{% code title="Example Command Output" %}
```
$ zenml service-connector list-resources --connector-type docker --resource-id docker.io
The resource with name 'docker.io' can be accessed by 'docker' service connectors that you have configured:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────────┼────────────────┨
┃ cf55339f-dbc8-4ee6-862e-c25aff411292 │ dockerhub │ 🐳 docker │ 🐳 docker-registry │ docker.io ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
```
{% endcode %}
After having set up or decided on a Docker Service Connector to use to connect to the target container registry, you can register the Docker Container Registry as follows:
```sh
# Register the container registry and reference the target registry URI
zenml container-registry register -f default \
--uri=
# Connect the container registry to the target registry via a Docker Service Connector
zenml container-registry connect -i
```
A non-interactive version that connects the Default Container Registry to a target registry through a Docker Service Connector:
```sh
zenml container-registry connect --connector
```
{% code title="Example Command Output" %}
```
$ zenml container-registry connect dockerhub --connector dockerhub
Successfully connected container registry `dockerhub` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────────┼────────────────┨
┃ cf55339f-dbc8-4ee6-862e-c25aff411292 │ dockerhub │ 🐳 docker │ 🐳 docker-registry │ docker.io ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┛
```
{% endcode %}
As a final step, you can use the Default Container Registry in a ZenML Stack:
```sh
# Register and set a stack with the new container registry
zenml stack register -c ... --set
```
{% hint style="info" %}
Linking the Default Container Registry to a Service Connector means that your local Docker client is no longer authenticated to access the remote registry. If you need to manually interact with the remote registry via the Docker CLI, you can use the [local login Service Connector feature](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/service-connectors-guide#configure-local-clients) to temporarily authenticate your local Docker client to the remote registry:
```sh
zenml service-connector login
```
{% code title="Example Command Output" %}
```
$ zenml service-connector login dockerhub
⠹ Attempting to configure local client using service connector 'dockerhub'...
WARNING! Your password will be stored unencrypted in /home/stefan/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
The 'dockerhub' Docker Service Connector was used to successfully configure the local Docker/OCI container registry client/SDK.
```
{% endcode %}
{% endhint %}
{% endtab %}
{% endtabs %}
For more information and a full list of configurable attributes of the Default container registry, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-aws.html#zenml.integrations.aws) .
---
### Book/Component Guide/Container Registries/Digitalocean (docs/book/component-guide/container-registries/digitalocean.md)
---
description: Storing container images in the DigitalOcean Container Registry.
---
# DigitalOcean Container Registry
The DigitalOcean container registry is a [container registry](./) flavor that comes built-in with the DigitalOcean ZenML integration and uses the [DigitalOcean Container Registry (DOCR)](https://www.digitalocean.com/products/container-registry) to store container images.
### When to use it
You should use the DigitalOcean container registry if:
* one or more components of your stack need to pull or push container images.
* your infrastructure runs on DigitalOcean — for example a DOKS cluster running the [Kubernetes orchestrator](https://docs.zenml.io/stacks/orchestrators/kubernetes) that pulls pipeline images from DOCR.
### How to deploy it
The flavor is provided by the DigitalOcean ZenML integration:
```shell
zenml integration install digitalocean -y
```
You also need a DOCR registry. In the [DigitalOcean control panel](https://cloud.digitalocean.com/), go to **Container Registry** and create one (each account has a single registry with a unique name). The registry URI has the form:
```
registry.digitalocean.com/
```
### How to use it
To use the DigitalOcean container registry, you need:
* [Docker](https://www.docker.com) installed and running.
* The registry URI (see above).
You can then register the container registry and use it in your active stack:
```shell
zenml container-registry register do_registry \
--flavor=digitalocean \
--uri=registry.digitalocean.com/
# Add the container registry to the active stack
zenml stack update -c do_registry
```
You also need to authenticate your local Docker client with DOCR. The simplest way is [`doctl`](https://docs.digitalocean.com/reference/doctl/):
```shell
doctl registry login
```
{% hint style="info" %}
DOCR login credentials issued by `doctl registry login` are short-lived and expire after a while, so you may need to re-run the command. When pipeline images are pulled by a DOKS cluster, you can instead [integrate the registry with the cluster](https://docs.digitalocean.com/products/container-registry/how-to/use-registry-docker-kubernetes/) so nodes authenticate automatically.
{% endhint %}
For more information and a full list of configurable attributes of the DigitalOcean container registry, check out the [source code on GitHub](https://github.com/zenml-io/zenml/tree/main/src/zenml/integrations/digitalocean).
---
### Book/Component Guide/Container Registries/Dockerhub (docs/book/component-guide/container-registries/dockerhub.md)
---
description: Storing container images in DockerHub.
---
# DockerHub
The DockerHub container registry is a [container registry](./) flavor that comes built-in with ZenML and uses [DockerHub](https://hub.docker.com/) to store container images.
### When to use it
You should use the DockerHub container registry if:
* one or more components of your stack need to pull or push container images.
* you have a DockerHub account. If you're not using DockerHub, take a look at the other [container registry flavors](./#container-registry-flavors).
### How to deploy it
To use the DockerHub container registry, all you need to do is create a [DockerHub](https://hub.docker.com/) account.
When this container registry is used in a ZenML stack, the Docker images that are built will be published in a \*\* public\*\* repository and everyone will be able to pull your images. If you want to use a **private** repository instead, you'll have to [create a private repository](https://docs.docker.com/docker-hub/repos/#creating-repositories) on the website before running the pipeline. The repository name depends on the remote [orchestrator](https://docs.zenml.io/stacks/orchestrators/) or [step operator](https://docs.zenml.io/stacks/step-operators/) that you're using in your stack.
### How to find the registry URI
The DockerHub container registry URI should have one of the two following formats:
```shell
# or
docker.io/
# Examples:
zenml
my-username
docker.io/zenml
docker.io/my-username
```
To figure out the URI for your registry:
* Find out the account name of your [DockerHub](https://hub.docker.com/) account.
* Use the account name to fill the template `docker.io/` and get your URI.
### How to use it
To use the DockerHub container registry, we need:
* [Docker](https://www.docker.com) installed and running.
* The registry URI. Check out the [previous section](dockerhub.md#how-to-find-the-registry-uri) on the URI format and how to get the URI for your registry.
We can then register the container registry and use it in our active stack:
```shell
zenml container-registry register \
--flavor=dockerhub \
--uri=
# Add the container registry to the active stack
zenml stack update -c
```
Additionally, we'll need to log in to the container registry so Docker can pull and push images. This will require your DockerHub account name and either your password or preferably a [personal access token](https://docs.docker.com/docker-hub/access-tokens/).
```shell
docker login
```
---
### Book/Component Guide/Container Registries/Gcp (docs/book/component-guide/container-registries/gcp.md)
---
description: Storing container images in GCP.
---
# Google Cloud Container Registry
The GCP container registry is a [container registry](./) flavor that comes built-in with ZenML and uses the [Google Artifact Registry](https://cloud.google.com/artifact-registry).
{% hint style="warning" %}
**Important Notice: Google Container Registry** [**is being replaced by Artifact Registry**](https://cloud.google.com/artifact-registry/docs/transition/transition-from-gcr)**. Please start using Artifact Registry for your containers. As per Google's documentation, "after May 15, 2024, Artifact Registry will host images for the gcr.io domain in Google Cloud projects without previous Container Registry usage. After March 18, 2025, Container Registry will be shut down."** The terms `container registry` and `artifact registry` will be used interchangeably throughout this document.
{% endhint %}
### When to use it
You should use the GCP container registry if:
* one or more components of your stack need to pull or push container images.
* you have access to GCP. If you're not using GCP, take a look at the other [container registry flavors](./#container-registry-flavors).
### How to deploy it
{% hint style="info" %}
Would you like to skip ahead and deploy a full ZenML cloud stack already, including a Google Artifact Registry? Check out the[in-browser stack deployment wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack), the [stack registration wizard](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/register-a-cloud-stack), or [the ZenML GCP Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register this stack component.
{% endhint %}
When using the Google Artifact Registry, you need to:
* enable it [here](https://console.cloud.google.com/marketplace/product/google/artifactregistry.googleapis.com)
* go [here](https://console.cloud.google.com/artifacts) and create a `Docker` repository.
## How to find the registry URI
When using the Google Artifact Registry, the GCP container registry URI should have the following format:
```shell
-docker.pkg.dev//
# Examples:
europe-west1-docker.pkg.dev/zenml/my-repo
southamerica-east1-docker.pkg.dev/zenml/zenml-test
asia-docker.pkg.dev/my-project/another-repo
```
To figure out the URI for your registry:
* Go [here](https://console.cloud.google.com/artifacts) and select the repository that you want to use to store Docker images. If you don't have a repository yet, take a look at the [deployment section](gcp.md#how-to-deploy-it).
* On the top, click the copy button to copy the full repository URL.
### How to use it
To use the GCP container registry, we need:
* [Docker](https://www.docker.com) installed and running.
* The registry URI. Check out the [previous section](gcp.md#how-to-find-the-registry-uri) on the URI format and how to get the URI for your registry.
We can then register the container registry and use it in our active stack:
```shell
zenml container-registry register \
--flavor=gcp \
--uri=
# Add the container registry to the active stack
zenml stack update -c
```
You also need to set up [authentication](gcp.md#authentication-methods) required to log in to the container registry.
#### Authentication Methods
Integrating and using a GCP Container Registry in your pipelines is not possible without employing some form of authentication. If you're looking for a quick way to get started locally, you can use the _Local Authentication_ method. However, the recommended way to authenticate to the GCP cloud platform is through [a GCP Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector). This is particularly useful if you are configuring ZenML stacks that combine the GCP Container Registry with other remote stack components also running in GCP.
{% tabs %}
{% tab title="Local Authentication" %}
This method uses the Docker client authentication available _in the environment where the ZenML code is running_. On your local machine, this is the quickest way to configure a GCP Container Registry. You don't need to supply credentials explicitly when you register the GCP Container Registry, as it leverages the local credentials and configuration that the GCP CLI and Docker client store on your local machine. However, you will need to install and set up the GCP CLI on your machine as a prerequisite, as covered in [the GCP CLI documentation](https://cloud.google.com/sdk/docs/install-sdk), before you register the GCP Container Registry.
With the GCP CLI installed and set up with credentials, we'll need to configure Docker, so it can pull and push images:
* for a Google Container Registry:
```shell
gcloud auth configure-docker
```
* for a Google Artifact Registry:
```shell
gcloud auth configure-docker -docker.pkg.dev
```
{% hint style="warning" %}
Stacks using the GCP Container Registry set up with local authentication are not portable across environments. To make ZenML pipelines fully portable, it is recommended to use [a GCP Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector) to link your GCP Container Registry to the remote GCR registry.
{% endhint %}
{% endtab %}
{% tab title="GCP Service Connector (recommended)" %}
To set up the GCP Container Registry to authenticate to GCP and access a Google Artifact Registry, it is recommended to leverage the many features provided by [the GCP Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector) such as auto-configuration, local login, best security practices regarding long-lived credentials and reusing the same credentials across multiple stack components.
If you don't already have a GCP Service Connector configured in your ZenML deployment, you can register one using the interactive CLI command. You have the option to configure a GCP Service Connector that can be used to access a Google Artifact Registry or even more than one type of GCP resource:
```sh
zenml service-connector register --type gcp -i
```
A non-interactive CLI example that leverages [the GCP CLI configuration](https://cloud.google.com/sdk/docs/install-sdk) on your local machine to auto-configure a GCP Service Connector targeting a GCR registry is:
```sh
zenml service-connector register --type gcp --resource-type docker-registry --auto-configure
```
{% code title="Example Command Output" %}
```
$ zenml service-connector register gcp-zenml-core --type gcp --resource-type docker-registry --auto-configure
⠸ Registering service connector 'gcp-zenml-core'...
Successfully registered service connector `gcp-zenml-core` with access to the following resources:
┏━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ RESOURCE TYPE │ RESOURCE NAMES ┃
┠────────────────────┼─────────────────────────────────────────────────┨
┃ 🐳 docker-registry │ gcr.io/zenml-core ┃
┃ │ us.gcr.io/zenml-core ┃
┃ │ eu.gcr.io/zenml-core ┃
┃ │ asia.gcr.io/zenml-core ┃
┃ │ asia-docker.pkg.dev/zenml-core/asia.gcr.io ┃
┃ │ europe-docker.pkg.dev/zenml-core/eu.gcr.io ┃
┃ │ europe-west1-docker.pkg.dev/zenml-core/test ┃
┃ │ us-docker.pkg.dev/zenml-core/gcr.io ┃
┃ │ us-docker.pkg.dev/zenml-core/us.gcr.io ┃
┗━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
> **Note**: Please remember to grant the entity associated with your GCP credentials permissions to read and write to your GCR registry. For a full list of permissions required to use a GCP Service Connector to access a GCR registry, please refer to the [GCP Service Connector GCR registry resource type documentation](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector#gcr-container-registry) or read the documentation available in the interactive CLI commands and dashboard. The GCP Service Connector supports [many different authentication methods](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector#authentication-methods) with different levels of security and convenience. You should pick the one that best fits your use-case.
If you already have one or more GCP Service Connectors configured in your ZenML deployment, you can check which of them can be used to access the GCR registry you want to use for your GCP Container Registry by running e.g.:
```sh
zenml service-connector list-resources --connector-type gcp --resource-type docker-registry
```
{% code title="Example Command Output" %}
```
/* Detailed source-code truncated for AI context efficiency. */
```
{% endcode %}
After having set up or decided on a GCP Service Connector to use to connect to the target GCR registry, you can register the GCP Container Registry as follows:
```sh
# Register the GCP container registry and reference the target GCR registry URI
zenml container-registry register -f gcp \
--uri=
# Connect the GCP container registry to the target GCR registry via a GCP Service Connector
zenml container-registry connect -i
```
A non-interactive version that connects the GCP Container Registry to a target GCR registry through a GCP Service Connector:
```sh
zenml container-registry connect --connector
```
{% hint style="info" %}
Linking the GCP Container Registry to a Service Connector means that your local Docker client is no longer authenticated to access the remote registry. If you need to manually interact with the remote registry via the Docker CLI, you can use the [local login Service Connector feature](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/service-connectors-guide#configure-local-clients) to temporarily authenticate your local Docker client to the remote registry:
```sh
zenml service-connector login --resource-type docker-registry
```
{% code title="Example Command Output" %}
```
$ zenml service-connector login gcp-zenml-core --resource-type docker-registry
⠋ Attempting to configure local client using service connector 'gcp-zenml-core'...
WARNING! Your password will be stored unencrypted in /home/stefan/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
The 'gcp-zenml-core' Docker Service Connector was used to successfully configure the local Docker/OCI container registry client/SDK.
```
{% endcode %}
{% endhint %}
{% code title="Example Command Output" %}
```
$ zenml container-registry connect gcp-zenml-core --connector gcp-zenml-core
Successfully connected container registry `gcp-zenml-core` to the following resources:
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE │ RESOURCE TYPE │ RESOURCE NAMES ┃
┠──────────────────────────────────────┼────────────────┼────────────────┼────────────────────┼─────────────────────────────────────────────┨
┃ 561b776a-af8b-491c-a4ed-14349b440f30 │ gcp-zenml-core │ 🔵 gcp │ 🐳 docker-registry │ europe-west1-docker.pkg.dev/zenml-core/test ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
{% endcode %}
As a final step, you can use the GCP Container Registry in a ZenML Stack:
```sh
# Register and set a stack with the new container registry
zenml stack register -c ... --set
```
{% endtab %}
{% endtabs %}
For more information and a full list of configurable attributes of the GCP container registry, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-container_registries.html#zenml.container_registries.gcp_container_registry) .
---
### Book/Component Guide/Container Registries/Github (docs/book/component-guide/container-registries/github.md)
---
description: Storing container images in GitHub.
---
# GitHub Container Registry
The GitHub container registry is a [container registry](./) flavor that comes built-in with ZenML and uses the [GitHub Container Registry](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) to store container images.
### When to use it
You should use the GitHub container registry if:
* one or more components of your stack need to pull or push container images.
* you're using GitHub for your projects. If you're not using GitHub, take a look at the other [container registry flavors](./#container-registry-flavors).
### How to deploy it
The GitHub container registry is enabled by default when you create a GitHub account.
### How to find the registry URI
The GitHub container registry URI should have the following format:
```shell
ghcr.io/
# Examples:
ghcr.io/zenml
ghcr.io/my-username
ghcr.io/my-organization
```
To figure our the URI for your registry:
* Use the GitHub user or organization name to fill the template `ghcr.io/` and get your URI.
### How to use it
To use the GitHub container registry, we need:
* [Docker](https://www.docker.com) installed and running.
* The registry URI. Check out the [previous section](github.md#how-to-find-the-registry-uri) on the URI format and how to get the URI for your registry.
* Our Docker client configured, so it can pull and push images. Follow [this guide](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#authenticating-to-the-container-registry) to create a personal access token and login to the container registry.
We can then register the container registry and use it in our active stack:
```shell
zenml container-registry register \
--flavor=github \
--uri=
# Add the container registry to the active stack
zenml stack update -c
```
For more information and a full list of configurable attributes of the GitHub container registry, check out the [SDK Docs](https://sdkdocs.zenml.io/latest/core_code_docs/core-container_registries.html#zenml.container_registries.github_container_registry) .
---
### Book/Component Guide/Contribute/Custom Stack Component (docs/book/component-guide/contribute/custom-stack-component.md)
---
description: How to write a custom stack component flavor
icon: cube
---
# Custom Stack Component
When building a sophisticated MLOps Platform, you will often need to come up with custom-tailored solutions for your infrastructure or tooling. ZenML is built around the values of composability and reusability which is why the stack component flavors in ZenML are designed to be modular and straightforward to extend.
This guide will help you understand what a flavor is, and how you can develop and use your own custom flavors in ZenML.
## Understanding component flavors
In ZenML, a component type is a broad category that defines the functionality of a stack component. Each type can have multiple flavors, which are specific implementations of the component type. For instance, the type `artifact_store` can have flavors like `local`, `s3`, etc. Each flavor defines a unique implementation of functionality that an artifact store brings to a stack.
## Base Abstractions
Before we get into the topic of creating custom stack component flavors, let us briefly discuss the three core abstractions related to stack components: the `StackComponent`, the `StackComponentConfig`, and the `Flavor`.
### Base Abstraction 1: `StackComponent`
The `StackComponent` is the abstraction that defines the core functionality. As an example, check out the `BaseArtifactStore` definition below: The `BaseArtifactStore` inherits from `StackComponent` and establishes the public interface of all artifact stores. Any artifact store flavor needs to follow the standards set by this base class.
```python
from zenml.stack import StackComponent
class BaseArtifactStore(StackComponent):
"""Base class for all ZenML artifact stores."""
# --- public interface ---
@abstractmethod
def open(self, path, mode = "r"):
"""Open a file at the given path."""
@abstractmethod
def exists(self, path):
"""Checks if a path exists."""
...
```
As each component defines a different interface, make sure to check out the base class definition of the component type that you want to implement and also check out the [documentation on how to extend specific stack components](../contribute/custom-stack-component.md).
{% hint style="info" %}
If you would like to automatically track some metadata about your custom stack component with each pipeline run, you can do so by defining some additional methods in your stack component implementation class as shown in the [Tracking Custom Stack Component Metadata](https://docs.zenml.io/how-to/model-management-metrics/track-metrics-metadata/fetch-metadata-within-steps) section.
{% endhint %}
See the full code of the base `StackComponent` class [here](https://github.com/zenml-io/zenml/blob/main/src/zenml/stack/stack\_component.py#L301).
### Base Abstraction 2: `StackComponentConfig`
As the name suggests, the `StackComponentConfig` is used to configure a stack component instance. It is separated from the actual implementation on purpose. This way, ZenML can use this class to validate the configuration of a stack component during its registration/update, without having to import heavy (or even non-installed) dependencies.
{% hint style="info" %}
The `config` and `settings` of a stack component are two separate, yet related entities. The `config` is the static part of your flavor's configuration, defined when you register your flavor. The `settings` are the dynamic part of your flavor's configuration that can be overridden at runtime.
You can read more about the differences [here](https://docs.zenml.io/concepts/steps_and_pipelines/configuration).
{% endhint %}
Let us now continue with the base artifact store example from above and take a look at the `BaseArtifactStoreConfig`:
```python
from zenml.stack import StackComponentConfig
class BaseArtifactStoreConfig(StackComponentConfig):
"""Config class for `BaseArtifactStore`."""
path: str
SUPPORTED_SCHEMES: ClassVar[Set[str]]
...
```
Through the `BaseArtifactStoreConfig`, each artifact store will require users to define a `path` variable. Additionally, the base config requires all artifact store flavors to define a `SUPPORTED_SCHEMES` class variable that ZenML will use to check if the user-provided `path` is actually supported by the flavor.
See the full code of the base `StackComponentConfig` class [here](https://github.com/zenml-io/zenml/blob/main/src/zenml/stack/stack\_component.py#L44).
### Base Abstraction 3: `Flavor`
Finally, the `Flavor` abstraction is responsible for bringing the implementation of a `StackComponent` together with the corresponding `StackComponentConfig` definition and also defines the `name` and `type` of the flavor. As an example, check out the definition of the `local` artifact store flavor below:
```python
from zenml.enums import StackComponentType
from zenml.stack import Flavor
class LocalArtifactStore(BaseArtifactStore):
...
class LocalArtifactStoreConfig(BaseArtifactStoreConfig):
...
class LocalArtifactStoreFlavor(Flavor):
@property
def name(self) -> str:
"""Returns the name of the flavor."""
return "local"
@property
def type(self) -> StackComponentType:
"""Returns the flavor type."""
return StackComponentType.ARTIFACT_STORE
@property
def config_class(self) -> Type[LocalArtifactStoreConfig]:
"""Config class of this flavor."""
return LocalArtifactStoreConfig
@property
def implementation_class(self) -> Type[LocalArtifactStore]:
"""Implementation class of this flavor."""
return LocalArtifactStore
```
See the full code of the base `Flavor` class definition [here](https://github.com/zenml-io/zenml/blob/main/src/zenml/stack/flavor.py#L29).
## Implementing a Custom Stack Component Flavor
Let's recap what we just learned by reimplementing the `S3ArtifactStore` from the `aws` integration as a custom flavor.
We can start with the configuration class: here we need to define the `SUPPORTED_SCHEMES` class variable introduced by the `BaseArtifactStore`. We also define several additional configuration values that users can use to configure how the artifact store will authenticate with AWS:
```python
from zenml.artifact_stores import BaseArtifactStoreConfig
from zenml.utils.secret_utils import SecretField
class MyS3ArtifactStoreConfig(BaseArtifactStoreConfig):
"""Configuration for the S3 Artifact Store."""
SUPPORTED_SCHEMES: ClassVar[Set[str]] = {"s3://"}
key: Optional[str] = SecretField(default=None)
secret: Optional[str] = SecretField(default=None)
token: Optional[str] = SecretField(default=None)
client_kwargs: Optional[Dict[str, Any]] = None
config_kwargs: Optional[Dict[str, Any]] = None
s3_additional_kwargs: Optional[Dict[str, Any]] = None
```
{% hint style="info" %}
You can pass sensitive configuration values as [secrets](https://docs.zenml.io/how-to/project-setup-and-management/interact-with-secrets) by defining them as type `SecretField` in the configuration class.
{% endhint %}
With the configuration defined, we can move on to the implementation class, which will use the S3 file system to implement the abstract methods of the `BaseArtifactStore`:
```python
import s3fs
from zenml.artifact_stores import BaseArtifactStore
class MyS3ArtifactStore(BaseArtifactStore):
"""Custom artifact store implementation."""
_filesystem: Optional[s3fs.S3FileSystem] = None
@property
def filesystem(self) -> s3fs.S3FileSystem:
"""Get the underlying S3 file system."""
if self._filesystem:
return self._filesystem
self._filesystem = s3fs.S3FileSystem(
key=self.config.key,
secret=self.config.secret,
token=self.config.token,
client_kwargs=self.config.client_kwargs,
config_kwargs=self.config.config_kwargs,
s3_additional_kwargs=self.config.s3_additional_kwargs,
)
return self._filesystem
def open(self, path, mode="r"):
"""Custom logic goes here."""
return self.filesystem.open(path=path, mode=mode)
def exists(self, path):
"""Custom logic goes here."""
return self.filesystem.exists(path=path)
```
{% hint style="info" %}
The configuration values defined in the corresponding configuration class are always available in the implementation class under `self.config`.
{% endhint %}
Finally, let's define a custom flavor that brings these two classes together. Make sure that you give your flavor a globally unique name here.
```python
from zenml.artifact_stores import BaseArtifactStoreFlavor
class MyS3ArtifactStoreFlavor(BaseArtifactStoreFlavor):
"""Custom artifact store implementation."""
@property
def name(self):
"""The name of the flavor."""
return 'my_s3_artifact_store'
@property
def implementation_class(self):
"""Implementation class for this flavor."""
from ... import MyS3ArtifactStore
return MyS3ArtifactStore
@property
def config_class(self):
"""Configuration class for this flavor."""
from ... import MyS3ArtifactStoreConfig
return MyS3ArtifactStoreConfig
```
{% hint style="info" %}
For flavors that require additional dependencies, you should make sure to define your implementation, config, and flavor classes in separate Python files and to only import the implementation class inside the `implementation_class` property of the flavor class. Otherwise, ZenML will not be able to load and validate your flavor configuration without the dependencies installed.
{% endhint %}
## Managing a Custom Stack Component Flavor
Once you have defined your implementation, config, and flavor classes, you can register your new flavor through the ZenML CLI:
```shell
zenml artifact-store flavor register
```
{% hint style="info" %}
Make sure to point to the flavor class via dot notation!
{% endhint %}
For example, if your flavor class `MyS3ArtifactStoreFlavor` is defined in `flavors/my_flavor.py`, you'd register it by doing:
```shell
zenml artifact-store flavor register flavors.my_flavor.MyS3ArtifactStoreFlavor
```
Afterwards, you should see the new custom artifact store flavor in the list of available artifact store flavors:
```shell
zenml artifact-store flavor list
```
And that's it! You now have a custom stack component flavor that you can use in your stacks just like any other flavor you used before, e.g.:
```shell
zenml artifact-store register \
--flavor=my_s3_artifact_store \
--path='some-path' \
...
zenml stack register \
--artifact-store \
...
```
## Tips and best practices
* ZenML resolves the flavor classes by taking the path where you initialized ZenML (via `zenml init`) as the starting point of resolution. Therefore, you and your team should remember to execute `zenml init` in a consistent manner (usually at the root of the repository where the `.git` folder lives). If the `zenml init` command was not executed, the current working directory is used to find implementation classes, which could lead to unexpected behavior.
* You can use the ZenML CLI to find which exact configuration values a specific flavor requires. Check out [this 3-minute video](https://www.youtube.com/watch?v=CQRVSKbBjtQ) for more information.
* You can keep changing the `Config` and `Settings` of your flavor after registration. ZenML will pick up these "live" changes when running pipelines.
* Note that changing the config in a breaking way requires an update of the component (not a flavor). E.g., adding a mandatory name to flavor X field will break a registered component of that flavor. This may lead to a completely broken state where one should delete the component and re-register it.
* Always test your flavor thoroughly before using it in production. Make sure it works as expected and handles errors gracefully.
* Keep your flavor code clean and well-documented. This will make it easier for others to use and contribute to your flavor.
* Follow best practices for the language and libraries you're using. This will help ensure your flavor is efficient, reliable, and easy to maintain.
* We recommend you develop new flavors by using existing flavors as a reference. A good starting point is the flavors defined in the [official ZenML integrations](https://github.com/zenml-io/zenml/tree/main/src/zenml/integrations).
## Extending Specific Stack Components
If you would like to learn more about how to build a custom stack component flavor for a specific stack component type, check out the links below:
| **Type of Stack Component** | **Description** |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------- |
| [Orchestrator](https://docs.zenml.io/stacks/orchestrators/custom) | Orchestrating the runs of your pipeline |
| [Artifact Store](https://docs.zenml.io/stacks/artifact-stores/custom) | Storage for the artifacts created by your pipelines |
| [Container Registry](https://docs.zenml.io/stacks/container-registries/custom) | Store for your containers |
| [Step Operator](https://docs.zenml.io/stacks/step-operators/custom) | Execution of individual steps in specialized runtime environments |
| [Model Deployer](https://docs.zenml.io/stacks/model-deployers/custom) | Services/platforms responsible for online model serving |
| [Feature Store](https://docs.zenml.io/stacks/feature-stores/custom) | Management of your data/features |
| [Experiment Tracker](https://docs.zenml.io/stacks/experiment-trackers/custom) | Tracking your ML experiments |
| [Alerter](https://docs.zenml.io/stacks/alerters/custom) | Sending alerts through specified channels |
| [Annotator](https://docs.zenml.io/stacks/annotators/custom) | Annotating and labeling data |
| [Data Validator](https://docs.zenml.io/stacks/data-validators/custom) | Validating and monitoring your data |
---
### Book/Component Guide/Data Validators/README (docs/book/component-guide/data-validators/README.md)
---
description: >-
How to enhance and maintain the quality of your data and the performance of
your models with data profiling and validation
icon: chart-column
---
# Data Validators
Without good data, even the best machine learning models will yield questionable results. A lot of effort goes into ensuring and maintaining data quality not only in the initial stages of model development, but throughout the entire machine learning project lifecycle. Data Validators are a category of ML libraries, tools and frameworks that grant a wide range of features and best practices that should be employed in the ML pipelines to keep data quality in check and to monitor model performance to keep it from degrading over time.
Data profiling, data integrity testing, data and model drift detection are all ways of employing data validation techniques at different points in your ML pipelines where data is concerned: data ingestion, model training and evaluation and online or batch inference. Data profiles and model performance evaluation results can be visualized and analyzed to detect problems and take preventive or correcting actions.
Related concepts:
* the Data Validator is an optional type of Stack Component that needs to be registered as part of your ZenML [Stack](https://docs.zenml.io/user-guides/production-guide/understand-stacks).
* Data Validators used in ZenML pipelines usually generate data profiles and data quality check reports that are versioned and stored in the [Artifact Store](https://docs.zenml.io/stacks/artifact-stores/) and can be [retrieved and visualized](https://docs.zenml.io/how-to/data-artifact-management/visualize-artifacts/) later.
### When to use it
[Data-centric AI practices](https://blog.zenml.io/data-centric-mlops/) are quickly becoming mainstream and using Data Validators are an easy way to incorporate them into your workflow. These are some common cases where you may consider employing the use of Data Validators in your pipelines:
* early on, even if it's just to keep a log of the quality state of your data and the performance of your models at different stages of development.
* if you have pipelines that regularly ingest new data, you should use data validation to run regular data integrity checks to signal problems before they are propagated downstream.
* in continuous training pipelines, you should use data validation techniques to compare new training data against a data reference and to compare the performance of newly trained models against previous ones.
* when you have pipelines that automate batch inference or if you regularly collect data used as input in online inference, you should use data validation to run data drift analyzes and detect training-serving skew, data drift and model drift.
#### Data Validator Flavors
Data Validator are optional stack components provided by integrations. The following table lists the currently available Data Validators and summarizes their features and the data types and model types that they can be used with in ZenML pipelines:
| Data Validator | Validation Features | Data Types | Model Types | Notes | Flavor/Integration |
| ------------------------------------------- | --------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | -------------------- |
| [Deepchecks](deepchecks.md) |
data quality data drift model drift model performance
| Add Deepchecks data and model validation tests to your pipelines | `deepchecks` |
| [Evidently](evidently.md) |
data quality data drift model drift model performance
| tabular: `pandas.DataFrame` | N/A | Use Evidently to generate a variety of data quality and data/model drift reports and visualizations | `evidently` |
| [Great Expectations](great-expectations.md) |
data profiling data quality
| tabular: `pandas.DataFrame` | N/A | Perform data testing, documentation and profiling with Great Expectations | `great_expectations` |
| [Whylogs/WhyLabs](whylogs.md) | data drift | tabular: `pandas.DataFrame` | N/A | Generate data profiles with whylogs. Hosted WhyLabs platform is being discontinued after Apple's acquisition—see the integration page for OSS deployment options. | `whylogs` |
If you would like to see the available flavors of Data Validator, you can use the command:
```shell
zenml data-validator flavor list
```
### How to use it
Every Data Validator has different data profiling and testing capabilities and uses a slightly different way of analyzing your data and your models, but it generally works as follows:
* first, you have to configure and add a Data Validator to your ZenML stack
* every integration includes one or more builtin data validation steps that you can add to your pipelines. Of course, you can also use the libraries directly in your own custom pipeline steps and simply return the results (e.g. data profiles, test reports) as artifacts that are versioned and stored by ZenML in its Artifact Store.
* you can access the data validation artifacts in subsequent pipeline steps, or [fetch them afterwards](https://docs.zenml.io/how-to/data-artifact-management/handle-data-artifacts/load-artifacts-into-memory) to process them or visualize them as needed.
Consult the documentation for the particular [Data Validator flavor](./#data-validator-flavors) that you plan on using or are using in your stack for detailed information about how to use it in your ZenML pipelines.
---
### Book/Component Guide/Data Validators/Custom (docs/book/component-guide/data-validators/custom.md)
---
description: How to develop a custom data validator
---
# Develop a custom data validator
{% hint style="info" %}
Before diving into the specifics of this component type, it is beneficial to familiarize yourself with our [general guide to writing custom component flavors in ZenML](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/implement-a-custom-stack-component). This guide provides an essential understanding of ZenML's component flavor concepts.
{% endhint %}
{% hint style="warning" %}
**Base abstraction in progress!**
We are actively working on the base abstraction for the Data Validators, which will be available soon. As a result, their extension is not recommended at the moment. When you are selecting a data validator for your stack, you can use one of [the existing flavors](./#data-validator-flavors).
If you need to implement your own Data Validator flavor, you can still do so, but keep in mind that you may have to refactor it when the base abstraction is updated.
{% endhint %}
ZenML comes equipped with [Data Validator implementations](./#data-validator-flavors) that integrate a variety of data logging and validation libraries, frameworks and platforms. However, if you need to use a different library or service as a backend for your ZenML Data Validator, you can extend ZenML to provide your own custom Data Validator implementation.
### Build your own custom data validator
If you want to implement your own custom Data Validator, you can follow the following steps:
1. Create a class which inherits from [the `BaseDataValidator` class](https://sdkdocs.zenml.io/latest/core_code_docs/core-data_validators.html#zenml.data_validators.base_data_validator) and override one or more of the abstract methods, depending on the capabilities of the underlying library/service that you want to integrate.
2. If you need any configuration, you can create a class which inherits from the `BaseDataValidatorConfig` class.
3. Bring both of these classes together by inheriting from the `BaseDataValidatorFlavor`.
4. (Optional) You should also provide some standard steps that others can easily insert into their pipelines for instant access to data validation features.
Once you are done with the implementation, you can register it through the CLI. Please ensure you **point to the flavor class via dot notation**:
```shell
zenml data-validator flavor register
```
For example, if your flavor class `MyDataValidatorFlavor` is defined in `flavors/my_flavor.py`, you'd register it by doing:
```shell
zenml data-validator flavor register flavors.my_flavor.MyDataValidatorFlavor
```
{% hint style="warning" %}
ZenML resolves the flavor class by taking the path where you initialized zenml (via `zenml init`) as the starting point of resolution. Therefore, please ensure you follow [the best practice](https://docs.zenml.io/user-guides/best-practices/iac) of initializing zenml at the root of your repository.
If ZenML does not find an initialized ZenML repository in any parent directory, it will default to the current working directory, but usually it's better to not have to rely on this mechanism, and initialize zenml at the root.
{% endhint %}
Afterwards, you should see the new flavor in the list of available flavors:
```shell
zenml data-validator flavor list
```
{% hint style="warning" %}
It is important to draw attention to when and how these base abstractions are coming into play in a ZenML workflow.
* The **CustomDataValidatorFlavor** class is imported and utilized upon the creation of the custom flavor through the CLI.
* The **CustomDataValidatorConfig** class is imported when someone tries to register/update a stack component with this custom flavor. Especially, during the registration process of the stack component, the config will be used to validate the values given by the user. As `Config` object are inherently `pydantic` objects, you can also add your own custom validators here.
* The **CustomDataValidator** only comes into play when the component is ultimately in use.
The design behind this interaction lets us separate the configuration of the flavor from its implementation. This way we can register flavors and components even when the major dependencies behind their implementation are not installed in our local setting (assuming the `CustomDataValidatorFlavor` and the `CustomDataValidatorConfig` are implemented in a different module/path than the actual `CustomDataValidator`).
{% endhint %}
---
### Book/Component Guide/Data Validators/Deepchecks (docs/book/component-guide/data-validators/deepchecks.md)
---
description: >-
How to test the data and models used in your pipelines with Deepchecks test
suites
---
# Deepchecks
The Deepchecks [Data Validator](./) flavor provided with the ZenML integration uses [Deepchecks](https://github.com/deepchecks/deepchecks) to run data integrity, data drift, model drift and model performance tests on the datasets and models circulated in your ZenML pipelines. The test results can be used to implement automated corrective actions in your pipelines or to render interactive representations for further visual interpretation, evaluation and documentation.
### When would you want to use it?
Deepchecks is an open-source library that you can use to run a variety of data and model validation tests, from data integrity tests that work with a single dataset to model evaluation tests to data drift analyzes and model performance comparison tests. All this can be done with minimal configuration input from the user, or customized with specialized conditions that the validation tests should perform.
Deepchecks works with both tabular data and computer vision data. For tabular, the supported dataset format is `pandas.DataFrame` and the supported model format is `sklearn.base.ClassifierMixin`. For computer vision, the supported dataset format is `torch.utils.data.dataloader.DataLoader` and supported model format is `torch.nn.Module`.
You should use the Deepchecks Data Validator when you need the following data and/or model validation features that are possible with Deepchecks:
* Data Integrity Checks [for tabular](https://docs.deepchecks.com/stable/tabular/auto_checks/data_integrity/index.html) or [computer vision](https://docs.deepchecks.com/stable/vision/auto_checks/data_integrity/index.html) data: detect data integrity problems within a single dataset (e.g. missing values, conflicting labels, mixed data types etc.).
* Data Drift Checks [for tabular](https://docs.deepchecks.com/stable/tabular/auto_checks/train_test_validation/index.html) or [computer vision](https://docs.deepchecks.com/stable/vision/auto_checks/train_test_validation/index.html) data: detect data skew and data drift problems by comparing a target dataset against a reference dataset (e.g. feature drift, label drift, new labels etc.).
* Model Performance Checks [for tabular](https://docs.deepchecks.com/stable/tabular/auto_checks/model_evaluation/index.html) or [computer vision](https://docs.deepchecks.com/stable/vision/auto_checks/model_evaluation/index.html) data: evaluate a model and detect problems with its performance (e.g. confusion matrix, boosting overfit, model error analysis)
* Multi-Model Performance Reports [for tabular](https://docs.deepchecks.com/stable/tabular/auto_checks/model_evaluation/plot_multi_model_performance_report.html#sphx-glr-tabular-auto-checks-model-evaluation-plot-multi-model-performance-report-py): produce a summary of performance scores for multiple models on test datasets.
You should consider one of the other [Data Validator flavors](./#data-validator-flavors) if you need a different set of data validation features.
### How do you deploy it?
The Deepchecks Data Validator flavor is included in the Deepchecks ZenML integration, you need to install it on your local machine to be able to register a Deepchecks Data Validator and add it to your stack:
```shell
zenml integration install deepchecks -y
```
The Data Validator stack component does not have any configuration parameters. Adding it to a stack is as simple as running e.g.:
```shell
# Register the Deepchecks data validator
zenml data-validator register deepchecks_data_validator --flavor=deepchecks
# Register and set a stack with the new data validator
zenml stack register custom_stack -dv deepchecks_data_validator ... --set
```
### How do you use it?
The ZenML integration restructures the way Deepchecks validation checks are organized in four categories, based on the type and number of input parameters that they expect as input. This makes it easier to reason about them when you decide which tests to use in your pipeline steps:
* **data integrity checks** expect a single dataset as input. These correspond one-to-one to the set of Deepchecks data integrity checks [for tabular](https://docs.deepchecks.com/stable/tabular/auto_checks/data_integrity/index.html) and [computer vision](https://docs.deepchecks.com/stable/vision/auto_checks/data_integrity/index.html) data
* **data drift checks** require two datasets as input: target and reference. These correspond one-to-one to the set of Deepchecks train-test checks [for tabular data](https://docs.deepchecks.com/stable/tabular/auto_checks/train_test_validation/index.html) and [for computer vision](https://docs.deepchecks.com/stable/vision/auto_checks/train_test_validation/index.html).
* **model validation checks** require a single dataset and a mandatory model as input. This list includes a subset of the model evaluation checks provided by Deepchecks [for tabular data](https://docs.deepchecks.com/stable/tabular/auto_checks/model_evaluation/index.html) and [for computer vision](https://docs.deepchecks.com/stable/vision/auto_checks/model_evaluation/index.html) that expect a single dataset as input.
* **model drift checks** require two datasets and a mandatory model as input. This list includes a subset of the model evaluation checks provided by Deepchecks [for tabular data](https://docs.deepchecks.com/stable/tabular/auto_checks/model_evaluation/index.html) and [for computer vision](https://docs.deepchecks.com/stable/vision/auto_checks/model_evaluation/index.html) that expect two datasets as input: target and reference.
This structure is directly reflected in how Deepchecks can be used with ZenML: there are four different Deepchecks standard steps and four different [ZenML enums for Deepchecks checks](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-deepchecks.html) . [The Deepchecks Data Validator API](deepchecks.md#the-deepchecks-data-validator) is also modeled to reflect this same structure.
A notable characteristic of Deepchecks is that you don't need to customize the set of Deepchecks tests that are part of a test suite. Both ZenML and Deepchecks provide sane defaults that will run all available Deepchecks tests in a given category with their default conditions if a custom list of tests and conditions are not provided.
There are three ways you can use Deepchecks in your ZenML pipelines that allow different levels of flexibility:
* instantiate, configure and insert one or more of [the standard Deepchecks steps](deepchecks.md#the-deepchecks-standard-steps) shipped with ZenML into your pipelines. This is the easiest way and the recommended approach, but can only be customized through the supported step configuration parameters.
* call the data validation methods provided by [the Deepchecks Data Validator](deepchecks.md#the-deepchecks-data-validator) in your custom step implementation. This method allows for more flexibility concerning what can happen in the pipeline step, but you are still limited to the functionality implemented in the Data Validator.
* [use the Deepchecks library directly](deepchecks.md#call-deepchecks-directly) in your custom step implementation. This gives you complete freedom in how you are using Deepchecks' features.
You can visualize Deepchecks results in Jupyter notebooks or view them directly in the ZenML dashboard.
### Warning! Usage in remote orchestrators
The current ZenML version has a limitation in its base Docker image that requires a workaround for _all_ pipelines using Deepchecks with a remote orchestrator (e.g. [Kubeflow](https://docs.zenml.io/stacks/orchestrators/kubeflow) , [Vertex](https://docs.zenml.io/stacks/orchestrators/vertex)). The limitation being that the base Docker image needs to be extended to include binaries that are required by `opencv2`, which is a package that Deepchecks requires.
While these binaries might be available on most operating systems out of the box (and therefore not a problem with the default local orchestrator), we need to tell ZenML to add them to the containerization step when running in remote settings. Here is how:
First, create a file called `deepchecks-zenml.Dockerfile` and place it on the same level as your runner script (commonly called `run.py`). The contents of the Dockerfile are as follows:
```shell
ARG ZENML_VERSION=0.20.0
FROM zenmldocker/zenml:${ZENML_VERSION} AS base
RUN apt-get update
RUN apt-get install ffmpeg libsm6 libxext6 -y
```
Then, place the following snippet above your pipeline definition. Note that the path of the `dockerfile` are relative to where the pipeline definition file is. Read [the containerization guide](https://docs.zenml.io/how-to/customize-docker-builds/) for more details:
```python
import zenml
from zenml import pipeline
from zenml.config import DockerSettings
from pathlib import Path
import sys
docker_settings = DockerSettings(
dockerfile="deepchecks-zenml.Dockerfile",
build_options={
"buildargs": {
"ZENML_VERSION": f"{zenml.__version__}"
},
},
)
@pipeline(settings={"docker": docker_settings})
def my_pipeline(...):
# same code as always
...
```
From here on, you can continue to use the deepchecks integration as is explained below.
#### The Deepchecks standard steps
ZenML wraps the Deepchecks functionality for tabular data in the form of four standard steps:
* [`deepchecks_data_integrity_check_step`](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-deepchecks.html#zenml.integrations.deepchecks): use it in your pipelines to run data integrity tests on a single dataset
* [`deepchecks_data_drift_check_step`](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-deepchecks.html#zenml.integrations.deepchecks): use it in your pipelines to run data drift tests on two datasets as input: target and reference.
* [`deepchecks_model_validation_check_step`](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-deepchecks.html#zenml.integrations.deepchecks): use it in your pipelines to run model performance tests using a single dataset and a mandatory model artifact as input
* [`deepchecks_model_drift_check_step`](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-deepchecks.html#zenml.integrations.deepchecks): use it in your pipelines to run model comparison/drift tests using a mandatory model artifact and two datasets as input: target and reference.
The integration doesn't yet include standard steps for computer vision, but you can still write your own custom steps that call [the Deepchecks Data Validator API](deepchecks.md#the-deepchecks-data-validator) or even [call the Deepchecks library directly](deepchecks.md#call-deepchecks-directly).
All four standard steps behave similarly regarding the configuration parameters and returned artifacts, with the following differences:
* the type and number of input artifacts are different, as mentioned above
* each step expects a different enum data type to be used when explicitly listing the checks to be performed via the `check_list` configuration attribute. See the [`zenml.integrations.deepchecks.validation_checks`](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-deepchecks.html) module for more details about these enums (e.g. the data integrity step expects a list of `DeepchecksDataIntegrityCheck` values).
This section will only cover how you can use the data integrity step, with a similar usage to be easily inferred for the other three steps.
To instantiate a data integrity step that will run all available Deepchecks data integrity tests with their default configuration, e.g.:
```python
from zenml.integrations.deepchecks.steps import (
deepchecks_data_integrity_check_step,
)
data_validator = deepchecks_data_integrity_check_step.with_options(
parameters=dict(
dataset_kwargs=dict(label="target", cat_features=[]),
),
)
```
The step can then be inserted into your pipeline where it can take in a dataset, e.g.:
```python
docker_settings = DockerSettings(required_integrations=[DEEPCHECKS, SKLEARN])
@pipeline(settings={"docker": docker_settings})
def data_validation_pipeline():
df_train, df_test = data_loader()
data_validator(dataset=df_train)
data_validation_pipeline()
```
As can be seen from the step definition, the step takes in a dataset and it returns a Deepchecks `SuiteResult` object that contains the test results:
```python
@step
def deepchecks_data_integrity_check_step(
dataset: pd.DataFrame,
check_list: Optional[Sequence[DeepchecksDataIntegrityCheck]] = None,
dataset_kwargs: Optional[Dict[str, Any]] = None,
check_kwargs: Optional[Dict[str, Any]] = None,
run_kwargs: Optional[Dict[str, Any]] = None,
) -> SuiteResult:
...
```
If needed, you can specify a custom list of data integrity Deepchecks tests to be executed by supplying a `check_list` argument:
```python
from zenml.integrations.deepchecks.validation_checks import DeepchecksDataIntegrityCheck
from zenml.integrations.deepchecks.steps import deepchecks_data_integrity_check_step
@pipeline
def validation_pipeline():
deepchecks_data_integrity_check_step(
check_list=[
DeepchecksDataIntegrityCheck.TABULAR_MIXED_DATA_TYPES,
DeepchecksDataIntegrityCheck.TABULAR_DATA_DUPLICATES,
DeepchecksDataIntegrityCheck.TABULAR_CONFLICTING_LABELS,
],
dataset=...
)
```
You should consult [the official Deepchecks documentation](https://docs.deepchecks.com/stable/tabular/auto_checks/data_integrity/index.html) for more information on what each test is useful for.
For more customization, the data integrity step also allows for additional keyword arguments to be supplied to be passed transparently to the Deepchecks library:
* `dataset_kwargs`: Additional keyword arguments to be passed to the Deepchecks `tabular.Dataset` or `vision.VisionData` constructor. This is used to pass additional information about how the data is structured, e.g.:
```python
deepchecks_data_integrity_check_step(
dataset_kwargs=dict(label='class', cat_features=['country', 'state']),
...
)
```
* `check_kwargs`: Additional keyword arguments to be passed to the Deepchecks check object constructors. Arguments are grouped for each check and indexed using the full check class name or check enum value as dictionary keys, e.g.:
```python
deepchecks_data_integrity_check_step(
check_list=[
DeepchecksDataIntegrityCheck.TABULAR_OUTLIER_SAMPLE_DETECTION,
DeepchecksDataIntegrityCheck.TABULAR_STRING_LENGTH_OUT_OF_BOUNDS,
DeepchecksDataIntegrityCheck.TABULAR_STRING_MISMATCH,
],
check_kwargs={
DeepchecksDataIntegrityCheck.TABULAR_OUTLIER_SAMPLE_DETECTION: dict(
nearest_neighbors_percent=0.01,
extent_parameter=3,
),
DeepchecksDataIntegrityCheck.TABULAR_STRING_LENGTH_OUT_OF_BOUNDS: dict(
num_percentiles=1000,
min_unique_values=3,
),
},
...
)
```
* `run_kwargs`: Additional keyword arguments to be passed to the Deepchecks Suite `run` method.
The `check_kwargs` attribute can also be used to customize [the conditions](https://docs.deepchecks.com/stable/general/usage/customizations/auto_examples/plot_configure_check_conditions.html#configure-check-conditions) configured for each Deepchecks test. ZenML attaches a special meaning to all check arguments that start with `condition_` and have a dictionary as value. This is required because there is no declarative way to specify conditions for Deepchecks checks. For example, the following step configuration:
```python
deepchecks_data_integrity_check_step(
check_list=[
DeepchecksDataIntegrityCheck.TABULAR_OUTLIER_SAMPLE_DETECTION,
DeepchecksDataIntegrityCheck.TABULAR_STRING_LENGTH_OUT_OF_BOUNDS,
],
dataset_kwargs=dict(label='class', cat_features=['country', 'state']),
check_kwargs={
DeepchecksDataIntegrityCheck.TABULAR_OUTLIER_SAMPLE_DETECTION: dict(
nearest_neighbors_percent=0.01,
extent_parameter=3,
condition_outlier_ratio_less_or_equal=dict(
max_outliers_ratio=0.007,
outlier_score_threshold=0.5,
),
condition_no_outliers=dict(
outlier_score_threshold=0.6,
)
),
DeepchecksDataIntegrityCheck.TABULAR_STRING_LENGTH_OUT_OF_BOUNDS: dict(
num_percentiles=1000,
min_unique_values=3,
condition_number_of_outliers_less_or_equal=dict(
max_outliers=3,
)
),
},
...
)
```
is equivalent to running the following Deepchecks tests:
```python
import deepchecks.tabular.checks as tabular_checks
from deepchecks.tabular import Suite
from deepchecks.tabular import Dataset
train_dataset = Dataset(
reference_dataset,
label='class',
cat_features=['country', 'state']
)
suite = Suite(name="custom")
check = tabular_checks.OutlierSampleDetection(
nearest_neighbors_percent=0.01,
extent_parameter=3,
)
check.add_condition_outlier_ratio_less_or_equal(
max_outliers_ratio=0.007,
outlier_score_threshold=0.5,
)
check.add_condition_no_outliers(
outlier_score_threshold=0.6,
)
suite.add(check)
check = tabular_checks.StringLengthOutOfBounds(
num_percentiles=1000,
min_unique_values=3,
)
check.add_condition_number_of_outliers_less_or_equal(
max_outliers=3,
)
suite.run(train_dataset=train_dataset)
```
#### The Deepchecks Data Validator
The Deepchecks Data Validator implements the same interface as do all Data Validators, so this method forces you to maintain some level of compatibility with the overall Data Validator abstraction, which guarantees an easier migration in case you decide to switch to another Data Validator.
All you have to do is call the Deepchecks Data Validator methods when you need to interact with Deepchecks to run tests, e.g.:
```python
import pandas as pd
from deepchecks.core.suite import SuiteResult
from zenml.integrations.deepchecks.data_validators import DeepchecksDataValidator
from zenml.integrations.deepchecks.validation_checks import DeepchecksDataIntegrityCheck
from zenml import step
@step
def data_integrity_check(
dataset: pd.DataFrame,
) -> SuiteResult:
"""Custom data integrity check step with Deepchecks
Args:
dataset: input Pandas DataFrame
Returns:
Deepchecks test suite execution result
"""
# validation pre-processing (e.g. dataset preparation) can take place here
data_validator = DeepchecksDataValidator.get_active_data_validator()
suite = data_validator.data_validation(
dataset=dataset,
check_list=[
DeepchecksDataIntegrityCheck.TABULAR_OUTLIER_SAMPLE_DETECTION,
DeepchecksDataIntegrityCheck.TABULAR_STRING_LENGTH_OUT_OF_BOUNDS,
],
)
# validation post-processing (e.g. interpret results, take actions) can happen here
return suite
```
The arguments that the Deepchecks Data Validator methods can take in are the same as those used for [the Deepchecks standard steps](deepchecks.md#the-deepchecks-standard-steps).
Have a look at [the complete list of methods and parameters available in the `DeepchecksDataValidator` API](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-deepchecks.html#zenml.integrations.deepchecks) in the SDK docs.
#### Call Deepchecks directly
You can use the Deepchecks library directly in your custom pipeline steps, and only leverage ZenML's capability of serializing, versioning and storing the `SuiteResult` objects in its Artifact Store, e.g.:
```python
import pandas as pd
import deepchecks.tabular.checks as tabular_checks
from deepchecks.core.suite import SuiteResult
from deepchecks.tabular import Suite
from deepchecks.tabular import Dataset
from zenml import step
@step
def data_integrity_check(
dataset: pd.DataFrame,
) -> SuiteResult:
"""Custom data integrity check step with Deepchecks
Args:
dataset: a Pandas DataFrame
Returns:
Deepchecks test suite execution result
"""
# validation pre-processing (e.g. dataset preparation) can take place here
train_dataset = Dataset(
dataset,
label='class',
cat_features=['country', 'state']
)
suite = Suite(name="custom")
check = tabular_checks.OutlierSampleDetection(
nearest_neighbors_percent=0.01,
extent_parameter=3,
)
check.add_condition_outlier_ratio_less_or_equal(
max_outliers_ratio=0.007,
outlier_score_threshold=0.5,
)
suite.add(check)
check = tabular_checks.StringLengthOutOfBounds(
num_percentiles=1000,
min_unique_values=3,
)
check.add_condition_number_of_outliers_less_or_equal(
max_outliers=3,
)
results = suite.run(train_dataset=train_dataset)
# validation post-processing (e.g. interpret results, take actions) can happen here
return results
```
#### Visualizing Deepchecks Suite Results
You can view visualizations of the suites and results generated by your pipeline steps directly in the ZenML dashboard by clicking on the respective artifact in the pipeline run DAG.
Alternatively, if you are running inside a Jupyter notebook, you can load and render the suites and results using the [artifact.visualize() method](https://docs.zenml.io/how-to/data-artifact-management/visualize-artifacts/), e.g.:
```python
from zenml.client import Client
def visualize_results(pipeline_name: str, step_name: str) -> None:
pipeline = Client().get_pipeline(pipeline=pipeline_name)
last_run = pipeline.last_run
step = last_run.steps[step_name]
step.visualize()
if __name__ == "__main__":
visualize_results("data_validation_pipeline", "data_integrity_check")
```
---
### Book/Component Guide/Data Validators/Evidently (docs/book/component-guide/data-validators/evidently.md)
---
description: >-
How to keep your data quality in check and guard against data and model drift
with Evidently profiling
---
# Evidently
The Evidently [Data Validator](./) flavor provided with the ZenML integration uses [Evidently](https://evidentlyai.com/) to perform data quality, data drift, model drift and model performance analyzes, to generate reports and run checks. The reports and check results can be used to implement automated corrective actions in your pipelines or to render interactive representations for further visual interpretation, evaluation and documentation.
### When would you want to use it?
[Evidently](https://evidentlyai.com/) is an open-source library that you can use to monitor and debug machine learning models by analyzing the data that they use through a powerful set of data profiling and visualization features, or to run a variety of data and model validation reports and tests, from data integrity tests that work with a single dataset to model evaluation tests to data drift analysis and model performance comparison tests. All this can be done with minimal configuration input from the user, or customized with specialized conditions that the validation tests should perform.
Evidently currently works with tabular data in `pandas.DataFrame` or CSV file formats and can handle both regression and classification tasks.
You should use the Evidently Data Validator when you need the following data and/or model validation features that are possible with Evidently:
* [Data Quality](https://docs.evidentlyai.com/metrics/preset_data_summary) reports and tests: provides detailed feature statistics and a feature behavior overview for a single dataset. It can also compare any two datasets. E.g. you can use it to compare train and test data, reference and current data, or two subgroups of one dataset.
* [Data Drift](https://docs.evidentlyai.com/metrics/preset_data_drift) reports and tests: helps detects and explore feature distribution changes in the input data by comparing two datasets with identical schema.
* [Target Drift](https://docs.evidentlyai.com/metrics/preset_data_drift) reports and tests: helps detect and explore changes in the target function and/or model predictions by comparing two datasets where the target and/or prediction columns are available.
* [Regression Performance](https://docs.evidentlyai.com/metrics/preset_regression) or [Classification Performance](https://docs.evidentlyai.com/metrics/preset_classification) reports and tests: evaluate the performance of a model by analyzing a single dataset where both the target and prediction columns are available. It can also compare it to the past performance of the same model, or the performance of an alternative model by providing a second dataset.
You should consider one of the other [Data Validator flavors](./#data-validator-flavors) if you need a different set of data validation features.
### How do you deploy it?
The Evidently Data Validator flavor is included in the Evidently ZenML integration, you need to install it on your local machine to be able to register an Evidently Data Validator and add it to your stack:
```shell
zenml integration install evidently -y
```
The Data Validator stack component does not have any configuration parameters. Adding it to a stack is as simple as running e.g.:
```shell
# Register the Evidently data validator
zenml data-validator register evidently_data_validator --flavor=evidently
# Register and set a stack with the new data validator
zenml stack register custom_stack -dv evidently_data_validator ... --set
```
### How do you use it?
#### Data Profiling
Evidently's profiling functions take in a `pandas.DataFrame` dataset or a pair of datasets and generate results in the form of a `Report` object.
One of Evidently's notable characteristics is that it only requires datasets as input. Even when running model performance comparison analyzes, no model needs to be present. However, that does mean that the input data needs to include additional `target` and `prediction` columns for some profiling reports and, you have to include additional information about the dataset columns in the form of [column mappings](https://docs.evidentlyai.com/docs/library/data_definition). Depending on how your data is structured, you may also need to include additional steps in your pipeline before the data validation step to insert the additional `target` and `prediction` columns into your data. This may also require interacting with one or more models.
There are three ways you can use Evidently to generate data reports in your ZenML pipelines that allow different levels of flexibility:
* instantiate, configure and insert the standard Evidently report step shipped with ZenML into your pipelines. This is the easiest way and the recommended approach.
* call the data validation methods provided by [the Evidently Data Validator](evidently.md#the-evidently-data-validator) in your custom step implementation. This method allows for more flexibility concerning what can happen in the pipeline step.
* [use the Evidently library directly](evidently.md#call-evidently-directly) in your custom step implementation. This gives you complete freedom in how you are using Evidently's features.
You can [visualize Evidently reports](evidently.md#visualizing-evidently-reports) in Jupyter notebooks or view them directly in the ZenML dashboard by clicking on the respective artifact in the pipeline run DAG.
**The Evidently Report step**
ZenML wraps the Evidently data profiling functionality in the form of a standard Evidently report pipeline step that you can simply instantiate and insert in your pipeline. Here you can see how instantiating and configuring the standard Evidently report step can be done:
```python
from zenml.integrations.evidently.metrics import EvidentlyMetricConfig
from zenml.integrations.evidently.steps import (
EvidentlyColumnMapping,
evidently_report_step,
)
text_data_report = evidently_report_step.with_options(
parameters=dict(
column_mapping=EvidentlyColumnMapping(
target="Rating",
numerical_features=["Age", "Positive_Feedback_Count"],
categorical_features=[
"Division_Name",
"Department_Name",
"Class_Name",
],
text_features=["Review_Text", "Title"],
),
metrics=[
EvidentlyMetricConfig.metric("DataQualityPreset"),
EvidentlyMetricConfig.metric(
"TextOverviewPreset", column_name="Review_Text"
),
EvidentlyMetricConfig.metric_generator(
"ColumnRegExpMetric",
columns=["Review_Text", "Title"],
reg_exp=r"[A-Z][A-Za-z0-9 ]*",
),
],
# We need to download the NLTK data for the TextOverviewPreset
download_nltk_data=True,
),
)
```
The configuration shown in the example is the equivalent of running the following Evidently code inside the step:
```python
from evidently.legacy.metrics import ColumnRegExpMetric
from evidently.legacy.metric_preset import DataQualityPreset, TextOverviewPreset
from evidently.legacy.pipeline.column_mapping import ColumnMapping
from evidently.legacy.report import Report
from evidently.legacy.metrics.base_metric import generate_column_metrics
import nltk
nltk.download("words")
nltk.download("wordnet")
nltk.download("omw-1.4")
column_mapping = ColumnMapping(
target="Rating",
numerical_features=["Age", "Positive_Feedback_Count"],
categorical_features=[
"Division_Name",
"Department_Name",
"Class_Name",
],
text_features=["Review_Text", "Title"],
)
report = Report(
metrics=[
DataQualityPreset(),
TextOverviewPreset(column_name="Review_Text"),
generate_column_metrics(
ColumnRegExpMetric,
columns=["Review_Text", "Title"],
parameters={"reg_exp": r"[A-Z][A-Za-z0-9 ]*"}
)
]
)
# The datasets are those that are passed to the Evidently step
# as input artifacts
report.run(
current_data=current_dataset,
reference_data=reference_dataset,
column_mapping=column_mapping,
)
```
Let's break this down...
We configure the `evidently_report_step` using parameters that you would normally pass to the Evidently `Report` object to [configure and run an Evidently report](https://docs.evidentlyai.com/docs/library/report). It consists of the following fields:
* `column_mapping`: This is an `EvidentlyColumnMapping` object that is the exact equivalent of [the `ColumnMapping` object in Evidently](https://docs.evidentlyai.com/docs/library/data_definition). It is used to describe the columns in the dataset and how they should be treated (e.g. as categorical, numerical, or text features).
* `metrics`: This is a list of `EvidentlyMetricConfig` objects that are used to configure the metrics that should be used to generate the report in a declarative way. This is the same as configuring the `metrics` that go in the Evidently `Report`.
* `download_nltk_data`: This is a boolean that is used to indicate whether the NLTK data should be downloaded. This is only needed if you are using Evidently reports that handle text data, which require the NLTK data to be downloaded ahead of time.
There are several ways you can reference the Evidently metrics when configuring `EvidentlyMetricConfig` items:
* by class name: this is the easiest way to reference an Evidently metric. You can use the name of a metric or metric preset class as it appears in the Evidently documentation (e.g.`"DataQualityPreset"`, `"DatasetDriftMetric"`).
* by full class path: you can also use the full Python class path of the metric or metric preset class ( e.g. `"evidently.legacy.metric_preset.DataQualityPreset"`, `"evidently.legacy.metrics.DatasetDriftMetric"`). This is useful if you want to use metrics or metric presets that are not included in Evidently library.
* by passing in the class itself: you can also import and pass in an Evidently metric or metric preset class itself, e.g.:
```python
from evidently.legacy.metrics import DatasetDriftMetric
...
evidently_report_step.with_options(
parameters=dict(
metrics=[EvidentlyMetricConfig.metric(DatasetDriftMetric)]
),
)
```
As can be seen in the example, there are two basic ways of adding metrics to your Evidently report step configuration:
* to add a single metric or metric preset: call `EvidentlyMetricConfig.metric` with an Evidently metric or metric preset class name (or class path or class). The rest of the parameters are the same ones that you would usually pass to the Evidently metric or metric preset class constructor.
* to generate multiple metrics, similar to calling [the Evidently column metric generator](https://docs.evidentlyai.com/docs/library/metric_generator): call `EvidentlyMetricConfig.metric_generator` with an Evidently metric or metric preset class name (or class path or class) and a list of column names. The rest of the parameters are the same ones that you would usually pass to the Evidently metric or metric preset class constructor.
The ZenML Evidently report step can then be inserted into your pipeline where it can take in two datasets and outputs the Evidently report generated in both JSON and HTML formats, e.g.:
```python
from zenml import pipeline
from zenml.config import DockerSettings
# Note: docker_settings would be defined elsewhere
# Note: data_loader, data_splitter, text_data_report, text_data_test, text_analyzer would be custom step functions
@pipeline(enable_cache=False, settings={"docker": docker_settings})
def text_data_report_test_pipeline():
"""Links all the steps together in a pipeline."""
data = data_loader()
reference_dataset, comparison_dataset = data_splitter(data)
report, _ = text_data_report(
reference_dataset=reference_dataset,
comparison_dataset=comparison_dataset,
)
test_report, _ = text_data_test(
reference_dataset=reference_dataset,
comparison_dataset=comparison_dataset,
)
text_analyzer(report)
text_data_report_test_pipeline()
```
For a version of the same step that works with a single dataset, simply don't pass any comparison dataset:
```python
text_data_report(reference_dataset=reference_dataset)
```
You should consult [the official Evidently documentation](https://docs.evidentlyai.com/metrics/all_metrics) for more information on what each metric is useful for and what data columns it requires as input.
The `evidently_report_step` step also allows for additional Report [options](https://docs.evidentlyai.com/metrics/customize_metric) to be passed to the `Report` constructor e.g.:
```python
from zenml.integrations.evidently.steps import (
EvidentlyColumnMapping,
)
text_data_report = evidently_report_step.with_options(
parameters=dict(
report_options = [
(
"evidently.legacy.options.ColorOptions", {
"primary_color": "#5a86ad",
"fill_color": "#fff4f2",
"zero_line_color": "#016795",
"current_data_color": "#c292a1",
"reference_data_color": "#017b92",
}
),
],
)
)
```
You can view [the complete list of configuration parameters](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-evidently.html#zenml.integrations.evidently) in the SDK docs.
#### Data Validation
Aside from data profiling, Evidently can also be used to configure and run automated data validation tests on your data.
Similar to using Evidently through ZenML to run data profiling, there are three ways you can use Evidently to run data validation tests in your ZenML pipelines that allow different levels of flexibility:
* instantiate, configure and insert [the standard Evidently test step](evidently.md) shipped with ZenML into your pipelines. This is the easiest way and the recommended approach.
* call the data validation methods provided by [the Evidently Data Validator](evidently.md#the-evidently-data-validator) in your custom step implementation. This method allows for more flexibility concerning what can happen in the pipeline step.
* [use the Evidently library directly](evidently.md#call-evidently-directly) in your custom step implementation. This gives you complete freedom in how you are using Evidently's features.
You can visualize Evidently reports in Jupyter notebooks or view them directly in the ZenML dashboard by clicking on the respective artifact in the pipeline run DAG.
You can [visualize Evidently reports](evidently.md#visualizing-evidently-reports) in Jupyter notebooks or view them directly in the ZenML dashboard by clicking on the respective artifact in the pipeline run DAG.
ZenML wraps the Evidently data validation functionality in the form of a standard Evidently test pipeline step that you can simply instantiate and insert in your pipeline. Here you can see how instantiating and configuring the standard Evidently test step can be done using our included `evidently_test_step` utility function:
```python
from zenml.integrations.evidently.steps import (
EvidentlyColumnMapping,
evidently_test_step,
)
from zenml.integrations.evidently.tests import EvidentlyTestConfig
text_data_test = evidently_test_step.with_options(
parameters=dict(
column_mapping=EvidentlyColumnMapping(
target="Rating",
numerical_features=["Age", "Positive_Feedback_Count"],
categorical_features=[
"Division_Name",
"Department_Name",
"Class_Name",
],
text_features=["Review_Text", "Title"],
),
tests=[
EvidentlyTestConfig.test("DataQualityTestPreset"),
EvidentlyTestConfig.test_generator(
"TestColumnRegExp",
columns=["Review_Text", "Title"],
reg_exp=r"[A-Z][A-Za-z0-9 ]*",
),
],
# We need to download the NLTK data for the TestColumnRegExp test
download_nltk_data=True,
),
)
```
The configuration shown in the example is the equivalent of running the following Evidently code inside the step:
```python
from evidently.legacy.tests import TestColumnRegExp
from evidently.legacy.test_preset import DataQualityTestPreset
from evidently.legacy.pipeline.column_mapping import ColumnMapping
from evidently.legacy.test_suite import TestSuite
from evidently.legacy.tests.base_test import generate_column_tests
import nltk
nltk.download("words")
nltk.download("wordnet")
nltk.download("omw-1.4")
column_mapping = ColumnMapping(
target="Rating",
numerical_features=["Age", "Positive_Feedback_Count"],
categorical_features=[
"Division_Name",
"Department_Name",
"Class_Name",
],
text_features=["Review_Text", "Title"],
)
test_suite = TestSuite(
tests=[
DataQualityTestPreset(),
generate_column_tests(
TestColumnRegExp,
columns=["Review_Text", "Title"],
parameters={"reg_exp": r"[A-Z][A-Za-z0-9 ]*"}
)
]
)
# The datasets are those that are passed to the Evidently step
# as input artifacts
test_suite.run(
current_data=current_dataset,
reference_data=reference_dataset,
column_mapping=column_mapping,
)
```
Let's break this down...
We configure the `evidently_test_step` using parameters that you would normally pass to the Evidently `TestSuite` object to [configure and run an Evidently test suite](https://docs.evidentlyai.com/docs/library/tests) . It consists of the following fields:
* `column_mapping`: This is an `EvidentlyColumnMapping` object that is the exact equivalent of [the `ColumnMapping` object in Evidently](https://docs.evidentlyai.com/docs/library/data_definition). It is used to describe the columns in the dataset and how they should be treated (e.g. as categorical, numerical, or text features).
* `tests`: This is a list of `EvidentlyTestConfig` objects that are used to configure the tests that will be run as part of your test suite in a declarative way. This is the same as configuring the `tests` that go in the Evidently `TestSuite`.
* `download_nltk_data`: This is a boolean that is used to indicate whether the NLTK data should be downloaded. This is only needed if you are using Evidently tests or test presets that handle text data, which require the NLTK data to be downloaded ahead of time.
There are several ways you can reference the Evidently tests when configuring `EvidentlyTestConfig` items, similar to how you reference them in an `EvidentlyMetricConfig` object:
* by class name: this is the easiest way to reference an Evidently test. You can use the name of a test or test preset class as it appears in the Evidently documentation (e.g.`"DataQualityTestPreset"`, `"TestColumnRegExp"`).
* by full class path: you can also use the full Python class path of the test or test preset class ( e.g. `"evidently.legacy.test_preset.DataQualityTestPreset"`, `"evidently.legacy.tests.TestColumnRegExp"`). This is useful if you want to use tests or test presets that are not included in Evidently library.
* by passing in the class itself: you can also import and pass in an Evidently test or test preset class itself, e.g.:
```python
from evidently.legacy.tests import TestColumnRegExp
...
evidently_test_step.with_options(
parameters=dict(
tests=[EvidentlyTestConfig.test(TestColumnRegExp)]
),
)
```
As can be seen in the example, there are two basic ways of adding tests to your Evidently test step configuration:
* to add a single test or test preset: call `EvidentlyTestConfig.test` with an Evidently test or test preset class name (or class path or class). The rest of the parameters are the same ones that you would usually pass to the Evidently test or test preset class constructor.
* to generate multiple tests, similar to calling [the Evidently column test generator](https://docs.evidentlyai.com/docs/library/metric_generator): call `EvidentlyTestConfig.test_generator` with an Evidently test or test preset class name (or class path or class) and a list of column names. The rest of the parameters are the same ones that you would usually pass to the Evidently test or test preset class constructor.
The ZenML Evidently test step can then be inserted into your pipeline where it can take in two datasets and outputs the Evidently test suite results generated in both JSON and HTML formats, e.g.:
```python
@pipeline(enable_cache=False, settings={"docker": docker_settings})
def text_data_test_pipeline():
"""Links all the steps together in a pipeline."""
data = data_loader()
reference_dataset, comparison_dataset = data_splitter(data)
json_report, html_report = text_data_test(
reference_dataset=reference_dataset,
comparison_dataset=comparison_dataset,
)
text_data_test_pipeline()
```
For a version of the same step that works with a single dataset, simply don't pass any comparison dataset:
```python
text_data_test(reference_dataset=reference_dataset)
```
You should consult [the official Evidently documentation](https://docs.evidentlyai.com/docs/library/tests) for more information on what each test is useful for and what data columns it requires as input.
The `evidently_test_step` step also allows for additional Test [options](https://docs.evidentlyai.com/metrics/customize_metric) to be passed to the `TestSuite` constructor e.g.:
```python
from zenml.integrations.evidently.steps import (
EvidentlyColumnMapping,
)
text_data_test = evidently_test_step.with_options(
parameters=dict(
test_options = [
(
"evidently.legacy.options.ColorOptions", {
"primary_color": "#5a86ad",
"fill_color": "#fff4f2",
"zero_line_color": "#016795",
"current_data_color": "#c292a1",
"reference_data_color": "#017b92",
}
),
],
),
)
```
You can view [the complete list of configuration parameters](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-evidently.html#zenml.integrations.evidently) in the SDK docs.
#### The Evidently Data Validator
The Evidently Data Validator implements the same interface as do all Data Validators, so this method forces you to maintain some level of compatibility with the overall Data Validator abstraction, which guarantees an easier migration in case you decide to switch to another Data Validator.
All you have to do is call the Evidently Data Validator methods when you need to interact with Evidently to generate data reports or to run test suites, e.g.:
```
/* Detailed source-code truncated for AI context efficiency. */
```
Have a look at [the complete list of methods and parameters available in the `EvidentlyDataValidator` API](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-evidently.html#zenml.integrations.evidently) in the SDK docs.
#### Call Evidently directly
You can use the Evidently library directly in your custom pipeline steps, e.g.:
```
/* Detailed source-code truncated for AI context efficiency. */
```
### Visualizing Evidently Reports
You can view visualizations of the Evidently reports generated by your pipeline steps directly in the ZenML dashboard by clicking on the respective artifact in the pipeline run DAG.
Alternatively, if you are running inside a Jupyter notebook, you can load and render the reports using the [artifact.visualize() method](https://docs.zenml.io/how-to/data-artifact-management/visualize-artifacts/), e.g.:
```python
from zenml.client import Client
def visualize_results(pipeline_name: str, step_name: str) -> None:
pipeline = Client().get_pipeline(pipeline=pipeline_name)
evidently_step = pipeline.last_run.steps[step_name]
evidently_step.visualize()
if __name__ == "__main__":
visualize_results("text_data_report_pipeline", "text_report")
visualize_results("text_data_test_pipeline", "text_test")
```
---
### Book/Component Guide/Data Validators/Great Expectations (docs/book/component-guide/data-validators/great-expectations.md)
---
description: >-
How to use Great Expectations to run data quality checks in your pipelines and
document the results
---
# Great Expectations
The Great Expectations [Data Validator](./) flavor provided with the ZenML integration uses [Great Expectations](https://greatexpectations.io/) to run data profiling and data quality tests on the data circulated through your pipelines. The test results can be used to implement automated corrective actions in your pipelines. They are also automatically rendered into documentation for further visual interpretation and evaluation.
### When would you want to use it?
[Great Expectations](https://greatexpectations.io/) is an open-source library that helps keep the quality of your data in check through data testing, documentation, and profiling, and to improve communication and observability. Great Expectations works with tabular data in a variety of formats and data sources, of which ZenML currently supports only `pandas.DataFrame` as part of its pipelines.
You should use the Great Expectations Data Validator when you need the following data validation features that are possible with Great Expectations:
* [Data Profiling](https://docs.greatexpectations.io/docs/oss/guides/expectations/creating_custom_expectations/how_to_add_support_for_the_auto_initializing_framework_to_a_custom_expectation/#build-a-custom-profiler-for-your-expectation): generates a set of validation rules (Expectations) automatically by inferring them from the properties of an input dataset.
* [Data Quality](https://docs.greatexpectations.io/docs/oss/guides/validation/checkpoints/how_to_pass_an_in_memory_dataframe_to_a_checkpoint/): runs a set of predefined or inferred validation rules (Expectations) against an in-memory dataset.
* [Data Docs](https://docs.greatexpectations.io/docs/reference/learn/terms/data_docs_store/): generate and maintain human-readable documentation of all your data validation rules, data quality checks and their results.
You should consider one of the other [Data Validator flavors](./#data-validator-flavors) if you need a different set of data validation features.
### How do you deploy it?
The Great Expectations Data Validator flavor is included in the Great Expectations ZenML integration, you need to install it on your local machine to be able to register a Great Expectations Data Validator and add it to your stack:
```shell
zenml integration install great_expectations -y
```
Depending on how you configure the Great Expectations Data Validator, it can reduce or even completely eliminate the complexity associated with setting up the store backends for Great Expectations. If you're only looking for a quick and easy way of adding Great Expectations to your stack and are not concerned with the configuration details, you can simply run:
```shell
# Register the Great Expectations data validator
zenml data-validator register ge_data_validator --flavor=great_expectations
# Register and set a stack with the new data validator
zenml stack register custom_stack -dv ge_data_validator ... --set
```
If you already have a Great Expectations deployment, you can configure the Great Expectations Data Validator to reuse or even replace your current configuration. You should consider the pros and cons of every deployment use-case and choose the one that best fits your needs:
1. let ZenML initialize and manage the Great Expectations configuration. The Artifact Store will serve as a storage backend for all the information that Great Expectations needs to persist (e.g. Expectation Suites, Validation Results). However, you will not be able to setup new Data Sources, Metadata Stores or Data Docs sites. Any changes you try and make to the configuration through code will not be persisted and will be lost when your pipeline completes or your local process exits.
2. use ZenML with your existing Great Expectations configuration. You can tell ZenML to replace your existing Metadata Stores with the active ZenML Artifact Store by setting the `configure_zenml_stores` attribute in the Data Validator. The downside is that you will only be able to run pipelines locally with this setup, given that the Great Expectations configuration is a file on your local machine.
3. migrate your existing Great Expectations configuration to ZenML. This is a compromise between 1. and 2. that allows you to continue to use your existing Data Sources, Metadata Stores and Data Docs sites even when running pipelines remotely.
{% hint style="warning" %}
Some Great Expectations CLI commands will not work well with the deployment methods that puts ZenML in charge of your Great Expectations configuration (i.e. 1. and 3.). You will be required to use Python code to manage your Expectations and you will have to edit the Jupyter notebooks generated by the Great Expectations CLI to connect them to your ZenML managed configuration. .
{% endhint %}
{% tabs %}
{% tab title="Let ZenML Manage The Configuration" %}
The default Data Validator setup plugs Great Expectations directly into the [Artifact Store](https://docs.zenml.io/stacks/artifact-stores/) component that is part of the same stack. As a result, the Expectation Suites, Validation Results and Data Docs are stored in the ZenML Artifact Store and you don't have to configure Great Expectations at all, ZenML takes care of that for you:
```shell
# Register the Great Expectations data validator
zenml data-validator register ge_data_validator --flavor=great_expectations
# Register and set a stack with the new data validator
zenml stack register custom_stack -dv ge_data_validator ... --set
```
{% endtab %}
{% tab title="Use Your Own Configuration" %}
If you have an existing Great Expectations configuration that you would like to reuse with your ZenML pipelines, the Data Validator allows you to do so. All you need is to point it to the folder where your local `great_expectations.yaml` configuration file is located:
```shell
# Register the Great Expectations data validator
zenml data-validator register ge_data_validator --flavor=great_expectations \
--context_root_dir=/path/to/my/great_expectations
# Register and set a stack with the new data validator
zenml stack register custom_stack -dv ge_data_validator ... --set
```
You can continue to edit your local Great Expectations configuration (e.g. add new Data Sources, update the Metadata Stores etc.) and these changes will be visible in your ZenML pipelines. You can also use the Great Expectations CLI as usual to manage your configuration and your Expectations.
{% endtab %}
{% tab title="Migrate Your Configuration to ZenML" %}
This deployment method migrates your existing Great Expectations configuration to ZenML and allows you to use it with local as well as remote orchestrators. You have to load the Great Expectations configuration contents in one of the Data Validator configuration parameters using the `@` operator, e.g.:
```shell
# Register the Great Expectations data validator
zenml data-validator register ge_data_validator --flavor=great_expectations \
--context_config=@/path/to/my/great_expectations/great_expectations.yaml
# Register and set a stack with the new data validator
zenml stack register custom_stack -dv ge_data_validator ... --set
```
When you are migrating your existing Great Expectations configuration to ZenML, keep in mind that the Metadata Stores that you configured there will also need to be accessible from the location where pipelines are running. For example, you cannot use a non-local orchestrator with a Great Expectations Metadata Store that is located on your filesystem.
{% endtab %}
{% endtabs %}
#### Advanced Configuration
The Great Expectations Data Validator has a few advanced configuration attributes that might be useful for your particular use-case:
* `configure_zenml_stores`: if set, ZenML will automatically update the Great Expectation configuration to include Metadata Stores that use the Artifact Store as a backend. If neither `context_root_dir` nor `context_config` are set, this is the default behavior. You can set this flag to use the ZenML Artifact Store as a backend for Great Expectations with any of the deployment methods described above. Note that ZenML will not copy the information in your existing Great Expectations stores (e.g. Expectation Suites, Validation Results) in the ZenML Artifact Store. This is something that you will have to do yourself.
* `configure_local_docs`: set this flag to configure a local Data Docs site where Great Expectations docs are generated and can be visualized locally. Use this in case you don't already have a local Data Docs site in your existing Great Expectations configuration.
For more, up-to-date information on the Great Expectations Data Validator configuration, you can have a look at [the SDK docs](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-great_expectations.html#zenml.integrations.great_expectations) .
### How do you use it?
The core Great Expectations concepts that you should be aware of when using it within ZenML pipelines are Expectations / Expectation Suites, Validations and Data Docs.
ZenML wraps the Great Expectations' functionality in the form of two standard steps:
* a Great Expectations data profiler that can be used to automatically generate Expectation Suites from an input `pandas.DataFrame` dataset
* a Great Expectations data validator that uses an existing Expectation Suite to validate an input `pandas.DataFrame` dataset
You can visualize Great Expectations Suites and Results in Jupyter notebooks or view them directly in the ZenML dashboard.
#### The Great Expectation's data profiler step
The standard Great Expectation's data profiler step builds an Expectation Suite automatically by running a [`UserConfigurableProfiler`](https://docs.greatexpectations.io/docs/guides/expectations/how_to_create_and_edit_expectations_with_a_profiler) on an input `pandas.DataFrame` dataset. The generated Expectation Suite is saved in the Great Expectations Expectation Store, but also returned as an `ExpectationSuite` artifact that is versioned and saved in the ZenML Artifact Store. The step automatically rebuilds the Data Docs.
At a minimum, the step configuration expects a name to be used for the Expectation Suite:
```python
from zenml.integrations.great_expectations.steps import (
great_expectations_profiler_step,
)
ge_profiler_step = great_expectations_profiler_step.with_options(
parameters={
"expectation_suite_name": "steel_plates_suite",
"data_asset_name": "steel_plates_train_df",
}
)
```
The step can then be inserted into your pipeline where it can take in a pandas dataframe, e.g.:
```python
from zenml import pipeline
docker_settings = DockerSettings(required_integrations=[SKLEARN, GREAT_EXPECTATIONS])
@pipeline(settings={"docker": docker_settings})
def profiling_pipeline():
"""Data profiling pipeline for Great Expectations.
The pipeline imports a reference dataset from a source then uses the builtin
Great Expectations profiler step to generate an expectation suite (i.e.
validation rules) inferred from the schema and statistical properties of the
reference dataset.
Args:
importer: reference data importer step
profiler: data profiler step
"""
dataset, _ = importer()
ge_profiler_step(dataset)
profiling_pipeline()
```
As can be seen from the step definition, the step takes in a `pandas.DataFrame` dataset, and it returns a Great Expectations `ExpectationSuite` object:
```python
@step
def great_expectations_profiler_step(
dataset: pd.DataFrame,
expectation_suite_name: str,
data_asset_name: Optional[str] = None,
profiler_kwargs: Optional[Dict[str, Any]] = None,
overwrite_existing_suite: bool = True,
) -> ExpectationSuite:
...
```
#### The Great Expectations data validator step
The standard Great Expectations data validator step validates an input `pandas.DataFrame` dataset by running an existing Expectation Suite on it. The validation results are saved in the Great Expectations Validation Store, but also returned as an `CheckpointResult` artifact that is versioned and saved in the ZenML Artifact Store. The step automatically rebuilds the Data Docs.
At a minimum, the step configuration expects the name of the Expectation Suite to be used for the validation:
```python
from zenml.integrations.great_expectations.steps import (
great_expectations_validator_step,
)
ge_validator_step = great_expectations_validator_step.with_options(
parameters={
"expectation_suite_name": "steel_plates_suite",
"data_asset_name": "steel_plates_train_df",
}
)
```
The step can then be inserted into your pipeline where it can take in a pandas dataframe and a bool flag used solely for order reinforcement purposes, e.g.:
```python
docker_settings = DockerSettings(required_integrations=[SKLEARN, GREAT_EXPECTATIONS])
@pipeline(settings={"docker": docker_settings})
def validation_pipeline():
"""Data validation pipeline for Great Expectations.
The pipeline imports a test data from a source, then uses the builtin
Great Expectations data validation step to validate the dataset against
the expectation suite generated in the profiling pipeline.
Args:
importer: test data importer step
validator: dataset validation step
checker: checks the validation results
"""
dataset, condition = importer()
results = ge_validator_step(dataset, condition)
message = checker(results)
validation_pipeline()
```
As can be seen from the step definition, the step takes in a `pandas.DataFrame` dataset and a boolean `condition` and it returns a Great Expectations `CheckpointResult` object. The boolean `condition` is only used as a means of ordering steps in a pipeline (e.g. if you must force it to run only after the data profiling step generates an Expectation Suite):
```python
@step
def great_expectations_validator_step(
dataset: pd.DataFrame,
expectation_suite_name: str,
data_asset_name: Optional[str] = None,
action_list: Optional[List[Dict[str, Any]]] = None,
exit_on_error: bool = False,
) -> CheckpointResult:
```
#### Call Great Expectations directly
You can use the Great Expectations library directly in your custom pipeline steps, while leveraging ZenML's capability of serializing, versioning and storing the `ExpectationSuite` and `CheckpointResult` objects in its Artifact Store. To use the Great Expectations configuration managed by ZenML while interacting with the Great Expectations library directly, you need to use the Data Context managed by ZenML instead of the default one provided by Great Expectations, e.g.:
```python
import great_expectations as ge
from zenml.integrations.great_expectations.data_validators import (
GreatExpectationsDataValidator
)
import pandas as pd
from great_expectations.core import ExpectationSuite
from zenml import step
@step
def create_custom_expectation_suite(
) -> ExpectationSuite:
"""Custom step that creates an Expectation Suite
Returns:
An Expectation Suite
"""
context = GreatExpectationsDataValidator.get_data_context()
# instead of:
# context = ge.get_context()
expectation_suite_name = "custom_suite"
suite = context.create_expectation_suite(
expectation_suite_name=expectation_suite_name
)
expectation_configuration = ExpectationConfiguration(...)
suite.add_expectation(expectation_configuration=expectation_configuration)
...
context.save_expectation_suite(
expectation_suite=suite,
expectation_suite_name=expectation_suite_name,
)
context.build_data_docs()
return suite
```
The same approach must be used if you are using a Great Expectations configuration managed by ZenML and are using the Jupyter notebooks generated by the Great Expectations CLI.
#### Visualizing Great Expectations Suites and Results
You can view visualizations of the suites and results generated by your pipeline steps directly in the ZenML dashboard by clicking on the respective artifact in the pipeline run DAG.
Alternatively, if you are running inside a Jupyter notebook, you can load and render the suites and results using the [`artifact.visualize()` method](https://docs.zenml.io/how-to/data-artifact-management/visualize-artifacts/), e.g.:
```python
from zenml.client import Client
def visualize_results(pipeline_name: str, step_name: str) -> None:
pipeline = Client().get_pipeline(pipeline_name)
last_run = pipeline.last_run
validation_step = last_run.steps[step_name]
validation_step.visualize()
if __name__ == "__main__":
visualize_results("validation_pipeline", "profiler")
visualize_results("validation_pipeline", "train_validator")
visualize_results("validation_pipeline", "test_validator")
```
---
### Book/Component Guide/Data Validators/Whylogs (docs/book/component-guide/data-validators/whylogs.md)
---
description: >-
How to collect and visualize statistics to track changes in your pipelines'
data with whylogs/WhyLabs profiling.
---
# Whylogs
The whylogs/WhyLabs [Data Validator](./) flavor provided with the ZenML integration uses the open-source [whylogs](https://github.com/whylabs/whylogs) library together with the now open-sourced [WhyLabs platform](https://github.com/whylabs/whylabs-oss) to generate and track data profiles, highly accurate descriptive representations of your data. The profiles can be used to implement automated corrective actions in your pipelines, or to render interactive representations for further visual interpretation, evaluation and documentation.
> **Warning:** [WhyLabs was acquired by Apple](https://whylabs.ai/) and the hosted WhyLabs platform is being discontinued. While the whylogs library remains open source and the WhyLabs platform source code is publicly available, hosted deployments may no longer be accessible. Make sure to plan your usage of the integration accordingly and consider self-hosting the OSS platform if you still need WhyLabs features.
### When would you want to use it?
[Whylogs](https://github.com/whylabs/whylogs) is an open-source library that analyzes your data and creates statistical summaries called whylogs profiles. Whylogs profiles can be processed in your pipelines and visualized locally or uploaded to a WhyLabs deployment for more in depth analysis. The official hosted WhyLabs service is being discontinued, but you can continue to operate a WhyLabs instance yourself by using the open-source release at [https://github.com/whylabs/whylabs-oss](https://github.com/whylabs/whylabs-oss). Even though [whylogs also supports other data types](https://github.com/whylabs/whylogs#data-types), the ZenML whylogs integration currently only works with tabular data in `pandas.DataFrame` format.
You should use the whylogs/WhyLabs Data Validator when you need the following data validation features that are possible with whylogs and WhyLabs:
* Data Quality: validate data quality in model inputs or in a data pipeline
* Data Drift: detect data drift in model input features
* Model Drift: Detect training-serving skew, concept drift, and model performance degradation
You should consider one of the other [Data Validator flavors](./#data-validator-flavors) if you need a different set of data validation features.
### How do you deploy it?
The whylogs Data Validator flavor is included in the whylogs ZenML integration, you need to install it on your local machine to be able to register a whylogs Data Validator and add it to your stack:
```shell
zenml integration install whylogs -y
```
If you don't need to connect to a WhyLabs deployment to upload and store the generated whylogs data profiles, the Data Validator stack component does not require any configuration parameters. Adding it to a stack is as simple as running e.g.:
```shell
# Register the whylogs data validator
zenml data-validator register whylogs_data_validator --flavor=whylogs
# Register and set a stack with the new data validator
zenml stack register custom_stack -dv whylogs_data_validator ... --set
```
Adding WhyLabs logging capabilities to your whylogs Data Validator is just slightly more complicated, as you also need to create a [ZenML Secret](https://docs.zenml.io/getting-started/deploying-zenml/secret-management) to store the sensitive WhyLabs authentication information in a secure location and then reference the secret in the Data Validator configuration. To generate a WhyLabs access token for a deployment that you host yourself, refer to the guidance in the [WhyLabs OSS repository](https://github.com/whylabs/whylabs-oss).
Then, you can register the whylogs Data Validator with WhyLabs logging capabilities as follows:
```shell
# Create the secret referenced in the data validator
zenml secret create whylabs_secret \
--whylabs_default_org_id= \
--whylabs_api_key=
# Register the whylogs data validator
zenml data-validator register whylogs_data_validator --flavor=whylogs \
--authentication_secret=whylabs_secret
```
You'll also need to enable whylabs logging for your custom pipeline steps if you want to upload the whylogs data profiles that they return as artifacts to your WhyLabs deployment. This is enabled by default for the standard whylogs step. For custom steps, you can enable WhyLabs logging by setting the `upload_to_whylabs` parameter to `True` in the step configuration, e.g.:
```python
from typing import Annotated
from typing import Tuple
import pandas as pd
import whylogs as why
from sklearn import datasets
from whylogs.core import DatasetProfileView
from zenml.integrations.whylogs.flavors.whylogs_data_validator_flavor import (
WhylogsDataValidatorSettings,
)
from zenml import step
@step(
settings={
"data_validator": WhylogsDataValidatorSettings(
enable_whylabs=True, dataset_id="model-1"
)
}
)
def data_loader() -> Tuple[
Annotated[pd.DataFrame, "data"],
Annotated[DatasetProfileView, "profile"]
]:
"""Load the diabetes dataset."""
X, y = datasets.load_diabetes(return_X_y=True, as_frame=True)
# merge X and y together
df = pd.merge(X, y, left_index=True, right_index=True)
profile = why.log(pandas=df).profile().view()
return df, profile
```
### How do you use it?
Whylogs's profiling functions take in a `pandas.DataFrame` dataset generate a `DatasetProfileView` object containing all the relevant information extracted from the dataset.
There are three ways you can use whylogs in your ZenML pipelines that allow different levels of flexibility:
* instantiate, configure and insert [the standard `WhylogsProfilerStep`](whylogs.md#the-whylogs-standard-step) shipped with ZenML into your pipelines. This is the easiest way and the recommended approach, but can only be customized through the supported step configuration parameters.
* call the data validation methods provided by [the whylogs Data Validator](whylogs.md#the-whylogs-data-validator) in your custom step implementation. This method allows for more flexibility concerning what can happen in the pipeline step, but you are still limited to the functionality implemented in the Data Validator.
* [use the whylogs library directly](whylogs.md#call-whylogs-directly) in your custom step implementation. This gives you complete freedom in how you are using whylogs's features.
You can [visualize whylogs profiles](whylogs.md#visualizing-whylogs-profiles) in Jupyter notebooks or view them directly in the ZenML dashboard.
#### The whylogs standard step
ZenML wraps the whylogs/WhyLabs functionality in the form of a standard `WhylogsProfilerStep` step. The only field in the step config is a `dataset_timestamp` attribute which is only relevant when you upload the profiles to a WhyLabs deployment that uses this field to group and merge together profiles belonging to the same dataset. The helper function `get_whylogs_profiler_step` used to create an instance of this standard step takes in an optional `dataset_id` parameter that is also used only in the context of WhyLabs uploads to identify the model in the context of which the profile is uploaded, e.g.:
```python
from zenml.integrations.whylogs.steps import get_whylogs_profiler_step
train_data_profiler = get_whylogs_profiler_step(dataset_id="model-2")
test_data_profiler = get_whylogs_profiler_step(dataset_id="model-3")
```
The step can then be inserted into your pipeline where it can take in a `pandas.DataFrame` dataset, e.g.:
```python
from zenml import pipeline
@pipeline
def data_profiling_pipeline():
data, _ = data_loader()
train, test = data_splitter(data)
train_data_profiler(train)
test_data_profiler(test)
data_profiling_pipeline()
```
As can be seen from the [step definition](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-whylogs.html#zenml.integrations.whylogs) , the step takes in a dataset and returns a whylogs `DatasetProfileView` object:
```python
@step
def whylogs_profiler_step(
dataset: pd.DataFrame,
dataset_timestamp: Optional[datetime.datetime] = None,
) -> DatasetProfileView:
...
```
You should consult [the official whylogs documentation](https://whylogs.readthedocs.io/en/latest/index.html) for more information on what you can do with the collected profiles.
You can view [the complete list of configuration parameters](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-whylogs.html#zenml.integrations.whylogs) in the SDK docs.
#### The whylogs Data Validator
The whylogs Data Validator implements the same interface as do all Data Validators, so this method forces you to maintain some level of compatibility with the overall Data Validator abstraction, which guarantees an easier migration in case you decide to switch to another Data Validator.
All you have to do is call the whylogs Data Validator methods when you need to interact with whylogs to generate data profiles. You may optionally enable whylabs logging to automatically upload the returned whylogs profile to your WhyLabs deployment, e.g.:
```python
import pandas as pd
from whylogs.core import DatasetProfileView
from zenml.integrations.whylogs.data_validators.whylogs_data_validator import (
WhylogsDataValidator,
)
from zenml.integrations.whylogs.flavors.whylogs_data_validator_flavor import (
WhylogsDataValidatorSettings,
)
from zenml import step
whylogs_settings = WhylogsDataValidatorSettings(
enable_whylabs=True, dataset_id=""
)
@step(
settings={
"data_validator": whylogs_settings
}
)
def data_profiler(
dataset: pd.DataFrame,
) -> DatasetProfileView:
"""Custom data profiler step with whylogs
Args:
dataset: a Pandas DataFrame
Returns:
Whylogs profile generated for the data
"""
# validation pre-processing (e.g. dataset preparation) can take place here
data_validator = WhylogsDataValidator.get_active_data_validator()
profile = data_validator.data_profiling(
dataset,
)
# optionally upload the profile to your WhyLabs deployment, if WhyLabs credentials are configured
data_validator.upload_profile_view(profile)
# validation post-processing (e.g. interpret results, take actions) can happen here
return profile
```
Have a look at [the complete list of methods and parameters available in the `WhylogsDataValidator` API](https://sdkdocs.zenml.io/latest/integration_code_docs/integrations-whylogs.html#zenml.integrations.whylogs) in the SDK docs.
#### Call whylogs directly
You can use the whylogs library directly in your custom pipeline steps, and only leverage ZenML's capability of serializing, versioning and storing the `DatasetProfileView` objects in its Artifact Store. You may optionally enable whylabs logging to automatically upload the returned whylogs profile to your WhyLabs deployment, e.g.:
```python
import pandas as pd
from whylogs.core import DatasetProfileView
import whylogs as why
from zenml import step
from zenml.integrations.whylogs.flavors.whylogs_data_validator_flavor import (
WhylogsDataValidatorSettings,
)
whylogs_settings = WhylogsDataValidatorSettings(
enable_whylabs=True, dataset_id=""
)
@step(
settings={
"data_validator": whylogs_settings
}
)
def data_profiler(
dataset: pd.DataFrame,
) -> DatasetProfileView:
"""Custom data profiler step with whylogs
Args:
dataset: a Pandas DataFrame
Returns:
Whylogs Profile generated for the dataset
"""
# validation pre-processing (e.g. dataset preparation) can take place here
results = why.log(dataset)
profile = results.profile()
# validation post-processing (e.g. interpret results, take actions) can happen here
return profile.view()
```
### Visualizing whylogs Profiles
You can view visualizations of the whylogs profiles generated by your pipeline steps directly in the ZenML dashboard by clicking on the respective artifact in the pipeline run DAG.
Alternatively, if you are running inside a Jupyter notebook, you can load and render the whylogs profiles using the [artifact.visualize() method](https://docs.zenml.io/how-to/data-artifact-management/visualize-artifacts/), e.g.:
```python
from zenml.client import Client
def visualize_statistics(
step_name: str, reference_step_name: Optional[str] = None
) -> None:
"""Helper function to visualize whylogs statistics from step artifacts.
Args:
step_name: step that generated and returned a whylogs profile
reference_step_name: an optional second step that generated a whylogs
profile to use for data drift visualization where two whylogs
profiles are required.
"""
pipe = Client().get_pipeline(pipeline="data_profiling_pipeline")
whylogs_step = pipe.last_run.steps[step_name]
whylogs_step.visualize()
if __name__ == "__main__":
visualize_statistics("data_loader")
visualize_statistics("train_data_profiler", "test_data_profiler")
```
---
### Book/Component Guide/Deployers/README (docs/book/component-guide/deployers/README.md)
---
description: Deploy pipelines as HTTP services for real-time execution
icon: rocket-launch
---
# Deployers
Pipeline deployment is the process of making ZenML pipelines available as long-running HTTP services for real-time execution. Unlike traditional batch execution through orchestrators, deployers create persistent web services that can handle on-demand pipeline invocations through HTTP requests.
Deployers are stack components responsible for managing the deployment of pipelines as containerized HTTP services that expose REST APIs for pipeline execution.
A deployed pipeline becomes a web service that can be invoked multiple times in parallel, receiving parameters through HTTP requests and returning pipeline outputs as JSON responses. This enables real-time inference, interactive workflows, and integration with web applications.
### When to use it?
Deployers are optional components in the ZenML stack. They are useful in the following scenarios:
- **Real-time Pipeline Execution**: Execute pipelines on-demand through HTTP requests rather than scheduled batch runs
- **Interactive Workflows**: Build applications that need immediate pipeline responses
- **API Integration**: Expose ML workflows as REST APIs for web applications or microservices
- **Real-time Inference**: Serve ML models through pipeline-based inference workflows
- **Agent-based Systems**: Create AI agents that execute pipelines in response to external events
Use deployers when you need request-response patterns, and orchestrators for scheduled, batch, or long-running workflows.
### Deployer Flavors
Out of the box, ZenML comes with a `local` deployer already part of the default stack that deploys pipelines on your local machine in the form of background processes. Additional Deployers are provided by integrations:
| Deployer | Flavor | Integration | Notes |
|------------------------------------|-----------|---------------|------------------------------------------------------------------------------|
| [Local](local.md) | `local` | _built-in_ | This is the default Deployer. It deploys pipelines on your local machine in the form of background processes. Should be used only for running ZenML locally. |
| [Docker](docker.md) | `docker` | Built-in | Deploys pipelines as locally running Docker containers |
| [Kubernetes](kubernetes.md) | `kubernetes` | `kubernetes` | Deploys pipelines to any Kubernetes cluster with full control over resources, networking, and scaling |
| [GCP Cloud Run](gcp-cloud-run.md) | `gcp` | `gcp` | Deploys pipelines to Google Cloud Run for serverless execution |
| [AWS App Runner](aws-app-runner.md) | `aws` | `aws` | Deploys pipelines to AWS App Runner for serverless execution |
| [Hugging Face](huggingface.md) | `huggingface` | `huggingface` | Deploys pipelines to Hugging Face Spaces as Docker Spaces |
If you would like to see the available flavors of deployers, you can use the command:
```shell
zenml deployer flavor list
```
### How to use it
You don't need to directly interact with the ZenML deployer stack component in your code. As long as the deployer that you want to use is part of your active [ZenML stack](../../user-guide/production-guide/understand-stacks.md), you can simply deploy a pipeline or snapshot using the ZenML CLI or the ZenML SDK. The resulting deployment can be managed using the ZenML CLI or the ZenML SDK.
Examples:
* just use the default stack - it has a default local deployer that will deploy the pipeline on your local machine in the form of a background process:
```bash
zenml stack set default
```
* or set up a new stack with a deployer in it:
```bash
zenml deployer register docker --flavor=local
zenml stack register docker_deployment -a default -o default -D docker --set
```
* deploy a pipeline with the ZenML SDK:
```python
from zenml import pipeline
@step
def my_step(name: str) -> str:
return f"Hello, {name}!"
@pipeline
def my_pipeline(name: str = "John") -> str:
return my_step(name=name)
if __name__ == "__main__":
# Deploy the pipeline `my_pipeline` as a deployment named `my_deployment`
deployment = my_pipeline.deploy(deployment_name="my_deployment")
print(f"Deployment URL: {deployment.url}")
```
* deploy the same pipeline with the CLI:
```bash
zenml pipeline deploy --name my_deployment my_module.my_pipeline
```
* send a request to the deployment with the ZenML CLI:
```bash
zenml deployment invoke my_deployment --name="Alice"
```
* or with curl:
```bash
curl -X POST http://localhost:8000/invoke \
-H "Content-Type: application/json" \
-d '{"parameters": {"name": "Alice"}}'
```
* alternatively, set up a snapshot and deploy it instead of a pipeline:
```bash
zenml pipeline snapshot create --name my_snapshot my_module.my_pipeline
zenml pipeline snapshot deploy my_snapshot --deployment my_deployment
```
#### Pipeline Requirements for Deployment
Not all pipelines are suitable for deployment as HTTP services. To be deployable, pipelines should follow these guidelines:
**Parameter Requirements:**
- Pipelines should accept explicit parameters with default values
- Parameters must be JSON-serializable types (int, float, str, bool, list, dict, Pydantic models)
- Parameter names should match step input names
**Output Requirements:**
- Pipelines should return meaningful values for HTTP responses
- Return values must be JSON-serializable
- It's recommended to use type annotations to specify output artifact names
Example Deployable Pipeline:
```python
from typing import Annotated
from zenml import pipeline, step
@step
def process_weather(city: str, temperature: float) -> Annotated[str, "weather_analysis"]:
return f"The weather in {city} is {temperature} degrees Celsius."
@pipeline
def weather_pipeline(city: str = "Paris", temperature: float = 20.0) -> str:
"""A deployable pipeline that processes weather data."""
analysis = process_weather(city=city, temperature=temperature)
return analysis
```
For more information, see the [Deployable Pipeline Requirements](https://docs.zenml.io/concepts/deployment#deployable-pipeline-requirements) section of the tutorial.
#### Deployment Lifecycle Management
The Deployment object represents a pipeline that has been deployed to a serving environment. The Deployment object is saved in the ZenML database and contains information about the deployment configuration, status, and connection details. Deployments are standalone entities that can be managed independently of the active stack through the Deployer stack components that were originally used to provision them.
Some example of how to manage deployments:
* listing deployments with the CLI:
```bash
$ zenml deployment list
┏━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ NAME │ PIPELINE │ URL │ STATUS ┃
┠──────────────────────┼──────────────────────────────────────┼────────────────────────────────┼──────────────────────────┨
┃ weather_service │ weather_pipeline │ http://localhost:8001 │ ⚙ RUNNING ┃
┠──────────────────────┼──────────────────────────────────────┼────────────────────────────────┼──────────────────────────┨
┃ ml_inference_api │ inference_pipeline │ http://k8s-cluster/ml-api │ ⚙ RUNNING ┃
┗━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
```
* listing deployments with the SDK:
```python
from zenml.client import Client
client = Client()
deployments = client.list_deployments()
for deployment in deployments:
print(f"{deployment.name}: {deployment.status}")
```
* showing detailed information about a deployment with the CLI:
```
/* Detailed source-code truncated for AI context efficiency. */
```
* showing detailed information about a deployment with the SDK:
```python
from zenml.client import Client
deployment = client.get_deployment("my_deployment")
print(deployment)
```
* deprovision and delete a deployment with the CLI:
```bash
$ zenml deployment delete my_deployment
```
* deprovisioning and deleting a deployment with the SDK:
```python
from zenml.client import Client
client = Client()
client.delete_deployment("my_deployment")
```
* sending a request to a deployment with the CLI:
```bash
$ zenml deployment invoke my_deployment --name="John"
Invoked deployment 'my_deployment' with response:
{
"success": true,
"outputs": {
"output": "Hello, John!"
},
"execution_time": 3.2781872749328613,
"metadata": {
"deployment_id": "95d60dcf-7c37-4e62-a923-a341601903e5",
"deployment_name": "my_deployment",
"snapshot_id": "f3122ed4-aa13-4113-9f60-a80545f56244",
"snapshot_name": "my_snapshot",
"pipeline_name": "my_pipeline",
"run_id": "ea448522-d5bf-411e-971e-d4550fdbe713",
"run_name": "my_pipeline-2025_09_30-12_52_01_012491",
"parameters_used": {}
},
"error": null
}
```
* sending a request to a deployment with the SDK:
```python
from zenml.deployers.utils import invoke_deployment
response = invoke_deployment(
deployment_name_or_id="my_deployment",
name="John",
)
print(response)
```
By default, invoking a deployment blocks until the pipeline run finishes and returns its outputs. To submit the run for background execution and get the run ID back immediately instead, pass `--no-wait` on the CLI or `submit=True` with the SDK:
```bash
$ zenml deployment invoke my_deployment --name="John" --no-wait
```
```python
response = invoke_deployment(
deployment_name_or_id="my_deployment",
name="John",
submit=True,
)
```
#### Specifying deployment resources
If your steps require additional hardware resources, you can specify them on your steps as described [here](https://docs.zenml.io/user-guides/tutorial/distributed-training/).
---
### Book/Component Guide/Deployers/Aws App Runner (docs/book/component-guide/deployers/aws-app-runner.md)
---
description: Deploying your pipelines to AWS App Runner.
---
# AWS App Runner Deployer
[AWS App Runner](https://aws.amazon.com/apprunner/) is a fully managed serverless platform that allows you to deploy and run your code in a production-ready, repeatable cloud environment without the need to manage any infrastructure. The AWS App Runner deployer is a [deployer](./) flavor included in the ZenML AWS integration that deploys your pipelines to AWS App Runner.
{% hint style="warning" %}
This component is only meant to be used within the context of a [remote ZenML installation](https://docs.zenml.io/getting-started/deploying-zenml/). Usage with a local ZenML setup may lead to unexpected behavior!
{% endhint %}
## When to use it
You should use the AWS App Runner deployer if:
* you're already using AWS.
* you're looking for a proven production-grade deployer.
* you're looking for a serverless solution for deploying your pipelines as HTTP micro-services.
* you want automatic scaling with pay-per-use pricing.
* you need to deploy containerized applications with minimal configuration.
## How to deploy it
{% hint style="info" %}
Would you like to skip ahead and deploy a full ZenML cloud stack already, including an AWS App Runner deployer? Check out [the ZenML AWS Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register this stack component and everything else needed by it.
{% endhint %}
{% hint style="warning" %}
App Runner is available only in [specific AWS regions](https://docs.aws.amazon.com/general/latest/gr/apprunner.html#apprunner_region).
{% endhint %}
In order to use an AWS App Runner deployer, you need to first deploy [ZenML to the cloud](https://docs.zenml.io/getting-started/deploying-zenml/). It would be recommended to deploy ZenML in the same AWS account and region as where the AWS App Runner infrastructure is deployed, but it is not necessary to do so. You must ensure that you are connected to the remote ZenML server before using this stack component.
The AWS App Runner deployer requires that you have [the necessary IAM permissions](aws-app-runner.md#aws-credentials-and-permissions) to create and manage App Runner services, and optionally access to AWS Secrets Manager and CloudWatch Logs for enhanced functionality.
## How to use it
To use the AWS App Runner deployer, you need:
* The ZenML `aws` integration installed. If you haven't done so, run
```shell
zenml integration install aws
```
* [Docker](https://www.docker.com) installed and running.
* A [remote artifact store](https://docs.zenml.io/stacks/artifact-stores/) as part of your stack.
* A [remote container registry](https://docs.zenml.io/stacks/container-registries/) as part of your stack (**NOTE**: must be Amazon ECR or ECR Public).
* [AWS credentials with proper permissions](aws-app-runner.md#aws-credentials-and-permissions) to create and manage the App Runner services themselves.
* When using a private ECR container registry, an IAM role with specific ECR permissions should also be created and configured as [the App Runner access role](https://docs.aws.amazon.com/apprunner/latest/dg/security_iam_service-with-iam.html#security_iam_service-with-iam-roles) (see [Required IAM Permissions](aws-app-runner.md#required-iam-permissions) below). If this is not configured, App Runner will attempt to use the default `AWSServiceRoleForAppRunner` service role, which may not have ECR access permissions.
* If opting to store sensitive information in the AWS Secrets Manager (enabled by default), an IAM role with specific Secrets Manager permissions should also be created and configured as [the App Runner instance role](https://docs.aws.amazon.com/apprunner/latest/dg/security_iam_service-with-iam.html#security_iam_service-with-iam-roles) (see [Required IAM Permissions](aws-app-runner.md#required-iam-permissions) below). If this is not configured, App Runner will attempt to use the default `AWSServiceRoleForAppRunner` service role, which may not have Secrets Manager access permissions.
* The AWS region in which you want to deploy your pipelines.
### AWS credentials and permissions
You have two different options to provide credentials to the AWS App Runner deployer:
* use the [AWS CLI](https://aws.amazon.com/cli/) to authenticate locally with AWS
* (recommended) configure [an AWS Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector) with AWS credentials and then link the AWS App Runner deployer stack component to the Service Connector.
#### AWS Permissions
Depending on how you configure the AWS App Runner deployer, there can be at most three different sets of permissions involved:
* the client permissions - these are the permissions needed by the Deployer stack component itself to interact with the App Runner service and optionally to manage AWS Secrets Manager secrets. These permissions need to come from either the local AWS SDK or the AWS Service Connector:
* the permissions in the `AWSAppRunnerFullAccess` policy.
* the following permissions for AWS Secrets Manager are also required if the deployer is configured to use secrets to pass sensitive information to the App Runner services instead of regular environment variables (i.e. if the `use_secrets_manager` setting is set to `True`):
* `secretsmanager:CreateSecret`
* `secretsmanager:UpdateSecret`
* `secretsmanager:DeleteSecret`
* `secretsmanager:DescribeSecret`
* `secretsmanager:GetSecretValue`
* `secretsmanager:PutSecretValue`
* `secretsmanager:TagResource`
These permissions should additionally be restricted to only allow access to secrets with a name starting with `zenml-` in the target region and account. Note that this prefix is also configurable and can be changed by setting the `secret_name_prefix` setting.
* CloudWatch Logs permissions (for log retrieval):
* `logs:DescribeLogGroups`
* `logs:DescribeLogStreams`
* `logs:GetLogEvents`
* `iam:PassRole` permission granted for the App Runner access role and instance role, if they are also configured (see below).
* [the App Runner access role](https://docs.aws.amazon.com/apprunner/latest/dg/security_iam_service-with-iam.html#security_iam_service-with-iam-roles) - this is a role that App Runner uses for accessing images in Amazon ECR in your account. It's only required to access an image in Amazon ECR, and isn't required with Amazon ECR Public. This role should include the `AWSAppRunnerServicePolicyForECRAccess` policy or something similar restricted to the target ECR repository.
* [the App Runner instance role](https://docs.aws.amazon.com/apprunner/latest/dg/security_iam_service-with-iam.html#security_iam_service-with-iam-roles) - this is a role that the App Runner instances themselves use for accessing the AWS Secrets Manager secrets. It's only required if you use the AWS Secrets Manager to store sensitive information (i.e. if you keep the `use_secrets_manager` option set to `True` in the [deployer settings](aws-app-runner.md#additional-configuration)). This role should include the `secretsmanager:GetSecretValue` permission optionally restricted to only allow access to secrets with a name starting with `zenml-` in the target region and account. Note that this prefix is also configurable and can be changed by setting the `secret_name_prefix` setting.
#### Configuration use-case: local AWS CLI with user account
This configuration use-case assumes you have configured the [AWS CLI](https://aws.amazon.com/cli/) to authenticate locally with your AWS account (i.e. by running `aws configure`). It also assumes that your AWS account has [the client permissions required to use the AWS App Runner deployer](aws-app-runner.md#aws-permissions).
This is the easiest way to configure the AWS App Runner deployer, but it has the following drawbacks:
* the setup is not portable on other machines and reproducible by other users (i.e. other users won't be able to use the Deployer to deploy pipelines or manage your Deployments, although they would still be able to access their exposed endpoints and send HTTP requests).
* it uses your personal AWS credentials, which may have broader permissions than necessary for the deployer.
The deployer can be registered as follows:
```shell
zenml deployer register \
--flavor=aws \
--region= \
--instance_role_arn= \
--access_role_arn=
```
#### Configuration use-case: AWS Service Connector
This use-case assumes you have already configured an AWS IAM user or role with the [client permissions required to use the AWS App Runner deployer](aws-app-runner.md#aws-permissions).
It also assumes you have already created access keys for this IAM user and have them available (access key ID and secret access key), although there are [ways to authenticate with AWS through an AWS Service Connector that don't require long-term access keys](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector#aws-iam-role).
With the IAM credentials ready, you can register [the AWS Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/aws-service-connector) and AWS App Runner deployer as follows:
```shell
zenml service-connector register --type aws --auth-method=secret-key --aws_access_key_id= --aws_secret_access_key= --region= --resource-type aws-generic
zenml deployer register \
--flavor=aws \
--instance_role_arn= \
--access_role_arn= \
--connector
```
### Configuring the stack
With the deployer registered, it can be used in the active stack:
```shell
# Register and activate a stack with the new deployer
zenml stack register -D ... --set
```
{% hint style="info" %}
ZenML will build a Docker image called `/zenml:` and use it to deploy your pipeline as an App Runner service. The container registry must be Amazon ECR (private) or ECR Public. Check out [this page](https://docs.zenml.io/how-to/customize-docker-builds/) if you want to learn more about how ZenML builds these images and how you can customize them.
{% endhint %}
You can now [deploy any ZenML pipeline](https://docs.zenml.io/concepts/deployment) using the AWS App Runner deployer:
```shell
zenml pipeline deploy --name my_deployment my_module.my_pipeline
```
### Additional configuration
For additional configuration of the AWS App Runner deployer, you can pass the following `AWSDeployerSettings` attributes defined in the `zenml.integrations.aws.flavors.aws_deployer_flavor` module when configuring the deployer or defining or deploying your pipeline:
* Basic settings common to all Deployers:
* `auth_key`: A user-defined authentication key to use to authenticate with deployment API calls.
* `generate_auth_key`: Whether to generate and use a random authentication key instead of the user-defined one.
* `lcm_timeout`: The maximum time in seconds to wait for the deployment lifecycle management to complete.
* AWS App Runner-specific settings:
* `region` (default: `None`): AWS region where the App Runner service will be deployed. If not specified, the region will be determined from the authenticated session. App Runner is available in specific regions: https://docs.aws.amazon.com/apprunner/latest/dg/regions.html. Setting this has no effect if the deployer is configured with an AWS Service Connector.
* `service_name_prefix` (default: `"zenml-"`): Prefix for service names in App Runner to avoid naming conflicts.
* `port` (default: `8080`): Port on which the container listens for requests.
* `health_check_grace_period_seconds` (default: `20`): Grace period for health checks in seconds. Range: 0-20.
* `health_check_interval_seconds` (default: `10`): Interval between health checks in seconds. Range: 1-20.
* `health_check_path` (default: `"/health"`): Health check path for the App Runner service.
* `health_check_protocol` (default: `"TCP"`): Health check protocol. Options: 'TCP', 'HTTP'.
* `health_check_timeout_seconds` (default: `2`): Timeout for health checks in seconds. Range: 1-20.
* `health_check_healthy_threshold` (default: `1`): Number of consecutive successful health checks required.
* `health_check_unhealthy_threshold` (default: `5`): Number of consecutive failed health checks before unhealthy.
* `is_publicly_accessible` (default: `True`): Whether the App Runner service is publicly accessible.
* `ingress_vpc_configuration` (default: `None`): VPC configuration for private App Runner services. JSON string with VpcId, VpcEndpointId, and VpcIngressConnectionName.
* `environment_variables` (default: `{}`): Dictionary of environment variables to set in the App Runner service.
* `tags` (default: `{}`): Dictionary of tags to apply to the App Runner service.
* `use_secrets_manager` (default: `True`): Whether to store sensitive environment variables in AWS Secrets Manager instead of directly in the App Runner service configuration. When this is set to `True`, the deployer will also require additional permissions to access the AWS Secrets Manager secrets and an [App Runner instance role](https://docs.aws.amazon.com/apprunner/latest/dg/security_iam_service-with-iam.html#security_iam_service-with-iam-roles) to be configured as [the App Runner instance role](aws-app-runner.md#aws-permissions).
* `secret_name_prefix` (default: `"zenml-"`): Prefix for secret names in Secrets Manager to avoid naming conflicts.
* `observability_configuration_arn` (default: `None`): ARN of the observability configuration to associate with the App Runner service.
* `encryption_kms_key` (default: `None`): KMS key ARN for encrypting App Runner service data.
* `instance_role_arn` (default: `None`): ARN of the IAM role to assign to the App Runner service instances. Required if the `use_secrets_manager` setting is set to `True`.
* `access_role_arn` (default: `None`): ARN of the IAM role that App Runner uses to access the image repository (ECR). Required for private ECR repositories.
* `strict_resource_matching` (default: `False`): Whether to enforce strict matching of resource requirements to AWS App Runner supported CPU and memory combinations. When True, raises an error if no exact match is found. When False, automatically selects the closest matching supported combination.
Check out [this docs page](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) for more information on how to specify settings.
For example, if you wanted to disable the use of AWS Secrets Manager for the deployment, you would configure settings as follows:
```python
from zenml import step, pipeline
from zenml.integrations.aws.flavors.aws_deployer_flavor import AWSDeployerSettings
@step
def greet(name: str) -> str:
return f"Hello {name}!"
settings = {
"deployer": AWSDeployerSettings(
use_secrets_manager=False
)
}
@pipeline(settings=settings)
def greet_pipeline(name: str = "John"):
greet(name=name)
```
### Resource and scaling settings
You can specify the resource and scaling requirements for the pipeline deployment using the `ResourceSettings` class at the pipeline level, as described in our documentation on [resource settings](https://docs.zenml.io/concepts/steps_and_pipelines/configuration#resource-settings):
```python
from zenml import step, pipeline
from zenml.config import ResourceSettings
resource_settings = ResourceSettings(
cpu_count=1.0,
memory="2GB",
min_replicas=4,
max_replicas=25,
max_concurrency=100
)
...
@pipeline(settings={"resources": resource_settings})
def greet_pipeline(name: str = "John"):
greet(name=name)
```
{% hint style="warning" %}
AWS App Runner defines specific rules concerning allowed combinations of CPU (vCPU) and memory (GB) values. For more information, see the [AWS App Runner documentation](https://docs.aws.amazon.com/apprunner/latest/dg/architecture.html#architecture.vcpu-memory).
Supported combinations (as of October 2025) include:
- 0.25 vCPU: 0.5 GB, 1 GB
- 0.5 vCPU: 1 GB
- 1 vCPU: 2 GB, 3 GB, 4 GB
- 2 vCPU: 4 GB, 6 GB
- 4 vCPU: 8 GB, 10 GB, 12 GB
By default, specifying `cpu_count` and `memory` values that are not valid according to these rules will **not** result in an error when deploying the pipeline. Instead, the values will be automatically adjusted to the nearest matching valid combination using an algorithm that prioritizes CPU requirements over memory requirements and aims to minimize waste. You can enable `strict_resource_matching=True` in the deployer settings to enforce exact matches and raise an error if no valid combination is found. You can also override and configure your own allowed resource combinations in the deployer's configuration via the `resource_combinations` option.
{% endhint %}
---
### Book/Component Guide/Deployers/Docker (docs/book/component-guide/deployers/docker.md)
---
description: Deploying your pipelines locally with Docker.
---
# Docker Deployer
The Docker deployer is a [deployer](./) flavor that comes built-in with ZenML and deploys your pipelines locally using Docker.
## When to use it
You should use the Docker deployer if:
* you need a quick and easy way to deploy your pipelines locally.
* you want to debug issues that happen when deploying your pipeline in Docker containers without waiting and paying for remote infrastructure.
* you need an easy way to test out how pipeline deployments work
## How to deploy it
To use the Docker deployer, you only need to have [Docker](https://www.docker.com/) installed and running.
## How to use it
To use the Docker deployer, you can register it and use it in your active stack:
```shell
zenml deployer register docker --flavor=docker
# Register and activate a stack with the new deployer
zenml stack register docker-deployer -D docker -o default -a default --set
```
{% hint style="info" %}
ZenML will build a local Docker image called `zenml:` and use it to deploy your pipeline as a Docker container. Check out [this page](https://docs.zenml.io/how-to/customize-docker-builds/) if you want to learn more about how ZenML builds these images and how you can customize them.
{% endhint %}
You can now [deploy any ZenML pipeline](https://docs.zenml.io/concepts/deployment) using the Docker deployer:
```shell
zenml pipeline deploy my_module.my_pipeline
```
### Additional configuration
For additional configuration of the Docker deployer, you can pass the following `DockerDeployerSettings` attributes defined in the `zenml.deployers.docker.docker_deployer` module when configuring the deployer or defining or deploying your pipeline:
* Basic settings common to all Deployers:
* `auth_key`: A user-defined authentication key to use to authenticate with deployment API calls.
* `generate_auth_key`: Whether to generate and use a random authentication key instead of the user-defined one.
* `lcm_timeout`: The maximum time in seconds to wait for the deployment lifecycle management to complete.
* Docker-specific settings:
* `port`: The port to expose the deployment on.
* `allocate_port_if_busy`: If True, allocate a free port if the configured port is busy.
* `port_range`: The range of ports to search for a free port.
* `run_args`: Arguments to pass to the `docker run` call. A full list of what can be passed in via the `run_args` can be found [in the Docker Python SDK documentation](https://docker-py.readthedocs.io/en/stable/containers.html).
Check out [this docs page](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) for more information on how to specify settings.
For example, if you wanted to specify the port to use for the deployment, you would configure settings as follows:
```python
from zenml import step, pipeline
from zenml.deployers.docker.docker_deployer import DockerDeployerSettings
@step
def greet(name: str) -> str:
return f"Hello {name}!"
settings = {
"deployer": DockerDeployerSettings(
port=8000
)
}
@pipeline(settings=settings)
def greet_pipeline(name: str = "John"):
greet(name=name)
```
---
### Book/Component Guide/Deployers/Gcp Cloud Run (docs/book/component-guide/deployers/gcp-cloud-run.md)
---
description: Deploying your pipelines to GCP Cloud Run.
---
# GCP Cloud Run Deployer
[GCP Cloud Run](https://cloud.google.com/run) is a fully managed serverless platform that allows you to deploy and run your code in a production-ready, repeatable cloud environment without the need to manage any infrastructure. The GCP Cloud Run deployer is a [deployer](./) flavor included in the ZenML GCP integration that deploys your pipelines to GCP Cloud Run.
{% hint style="warning" %}
This component is only meant to be used within the context of a [remote ZenML installation](https://docs.zenml.io/getting-started/deploying-zenml/). Usage with a local ZenML setup may lead to unexpected behavior!
{% endhint %}
## When to use it
You should use the GCP Cloud Run deployer if:
* you're already using GCP.
* you're looking for a proven production-grade deployer.
* you're looking for a serverless solution for deploying your pipelines as HTTP micro-services.
* you want automatic scaling with pay-per-use pricing.
* you need to deploy containerized applications with minimal configuration.
## How to deploy it
{% hint style="info" %}
Would you like to skip ahead and deploy a full ZenML cloud stack already, including a GCP Cloud Run deployer? Check out [the ZenML GCP Terraform module](https://docs.zenml.io/how-to/infrastructure-deployment/stack-deployment/deploy-a-cloud-stack-with-terraform) for a shortcut on how to deploy & register this stack component and everything else needed by it.
{% endhint %}
In order to use a GCP Cloud Run deployer, you need to first deploy [ZenML to the cloud](https://docs.zenml.io/getting-started/deploying-zenml/). It would be recommended to deploy ZenML in the same Google Cloud project as where the GCP Cloud Run infrastructure is deployed, but it is not necessary to do so. You must ensure that you are connected to the remote ZenML server before using this stack component.
The only other thing necessary to use the ZenML GCP Cloud Run deployer is enabling GCP Cloud Run-relevant APIs on the Google Cloud project.
## How to use it
To use the GCP Cloud Run deployer, you need:
* The ZenML `gcp` integration installed. If you haven't done so, run
```shell
zenml integration install gcp
```
* [Docker](https://www.docker.com) installed and running.
* A [remote artifact store](https://docs.zenml.io/stacks/artifact-stores/) as part of your stack.
* A [remote container registry](https://docs.zenml.io/stacks/container-registries/) as part of your stack.
* [GCP credentials with proper permissions](gcp-cloud-run.md#gcp-credentials-and-permissions)
* The GCP project ID and location in which you want to deploy your pipelines.
### GCP credentials and permissions
You have two different options to provide credentials to the GCP Cloud Run deployer:
* use the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud) to authenticate locally with GCP
* (recommended) configure [a GCP Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector) with GCP credentials and then link the GCP Cloud Run deployer stack component to the Service Connector.
#### GCP Permissions
Regardless of the authentication method used, the credentials used with the GCP Cloud Run deployer need the following permissions in the target GCP project:
* the `roles/run.admin` role - for managing Cloud Run services
* the following permissions to manage GCP secrets are required only if the Deployer is configured to use secrets to pass sensitive information to the Cloud Run services instead of regular environment variables (i.e. if the `use_secret_manager` setting is set to `True`):
* the unconditional `secretmanager.secrets.create` permission is required to create new secrets in the target GCP project.
* the `roles/secretmanager.admin` role restricted to only manage secrets with a name prefix of `zenml-`. Note that this prefix is also configurable and can be changed by setting the `secret_name_prefix` setting.
As a simpler alternative, the `roles/secretmanager.admin` role can be granted at the project level with no condition applied.
#### Configuration use-case: local `gcloud` CLI with user account
This configuration use-case assumes you have configured the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud) to authenticate locally with your GCP account (i.e. by running `gcloud auth login`). It also assumes that your GCP account has [the permissions required to use the GCP Cloud Run deployer](gcp-cloud-run.md#gcp-permissions).
This is the easiest way to configure the GCP Cloud Run deployer, but it has the following drawbacks:
* the setup is not portable on other machines and reproducible by other users (i.e. other users won't be able to use the Deployer to deploy pipelines or manage your Deployments, although they would still be able to access their exposed endpoints and send HTTP requests).
* it uses the Compute Engine default service account, which is not recommended, given that it has a lot of permissions by default and is used by many other GCP services.
The deployer can be registered as follows:
```shell
zenml deployer register \
--flavor=gcp \
--project= \
--location= \
```
#### Configuration use-case: GCP Service Connector
This use-case assumes you have already configured a GCP service account with the [permissions required to use the GCP Cloud Run deployer](gcp-cloud-run.md#gcp-permissions).
It also assumes you have already created a service account key for this service account and downloaded it to your local machine (e.g. in a `zenml-cloud-run-deployer.json` file), although there are [ways to authenticate with GCP through a GCP Service Connector that don't require a service account key](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector#external-account-gcp-workload-identity).
With the service account and the key ready, you can register [the GCP Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/gcp-service-connector) and GCP Cloud Run deployer as follows:
```shell
zenml service-connector register --type gcp --auth-method=service-account --project_id= --service_account_json=@zenml-cloud-run-deployer.json --resource-type gcp-generic
zenml deployer register \
--flavor=gcp \
--location= \
--connector
```
### Configuring the stack
With the deployer registered, it can be used in the active stack:
```shell
# Register and activate a stack with the new deployer
zenml stack register -D ... --set
```
{% hint style="info" %}
ZenML will build a Docker image called `/zenml:` and use it to deploy your pipeline as a Cloud Run service. Check out [this page](https://docs.zenml.io/how-to/customize-docker-builds/) if you want to learn more about how ZenML builds these images and how you can customize them.
{% endhint %}
You can now [deploy any ZenML pipeline](https://docs.zenml.io/concepts/deployment) using the GCP Cloud Run deployer:
```shell
zenml pipeline deploy --name my_deployment my_module.my_pipeline
```
### Additional configuration
For additional configuration of the GCP Cloud Run deployer, you can pass the following `GCPDeployerSettings` attributes defined in the `zenml.integrations.gcp.flavors.gcp_deployer_flavor` module when configuring the deployer or defining or deploying your pipeline:
* Basic settings common to all Deployers:
* `auth_key`: A user-defined authentication key to use to authenticate with deployment API calls.
* `generate_auth_key`: Whether to generate and use a random authentication key instead of the user-defined one.
* `lcm_timeout`: The maximum time in seconds to wait for the deployment lifecycle management to complete.
* GCP Cloud Run-specific settings:
* `location` (default: `"europe-west3"`): Name of GCP region where the pipeline will be deployed. Cloud Run is available in specific regions: https://cloud.google.com/run/docs/locations
* `service_name_prefix` (default: `"zenml-"`): Prefix for service names in Cloud Run to avoid naming conflicts.
* `timeout_seconds` (default: `300`): Request timeout in seconds. Must be between 1 and 3600 seconds (1 hour maximum).
* `ingress` (default: `"all"`): Ingress settings for the service. Available options: `'all'`, `'internal'`, `'internal-and-cloud-load-balancing'`.
* `vpc_connector` (default: `None`): VPC connector for private networking. Format: `projects/PROJECT_ID/locations/LOCATION/connectors/CONNECTOR_NAME`
* `service_account` (default: `None`): Service account email to run the Cloud Run service. If not specified, uses the default Compute Engine service account.
* `environment_variables` (default: `{}`): Dictionary of environment variables to set in the Cloud Run service.
* `labels` (default: `{}`): Dictionary of labels to apply to the Cloud Run service for organization and billing purposes.
* `annotations` (default: `{}`): Dictionary of annotations to apply to the Cloud Run service for additional metadata.
* `execution_environment` (default: `"gen2"`): Execution environment generation. Available options: `'gen1'`, `'gen2'`.
* `traffic_allocation` (default: `{"LATEST": 100}`): Traffic allocation between revisions. Keys are revision names or `'LATEST'`, values are percentages that must sum to 100.
* `allow_unauthenticated` (default: `True`): Whether to allow unauthenticated requests to the service. Set to `False` for private services requiring GCP specific authentication.
* `use_secret_manager` (default: `True`): Whether to store sensitive environment variables in GCP Secret Manager instead of directly in the Cloud Run service configuration for enhanced security.
* `secret_name_prefix` (default: `"zenml-"`): Prefix for secret names in Secret Manager to avoid naming conflicts when using Secret Manager for sensitive data.
Check out [this docs page](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) for more information on how to specify settings.
For example, if you wanted to disable the use of GCP Secret Manager for the deployment, you would configure settings as follows:
```python
from zenml import step, pipeline
from zenml.integrations.gcp.flavors.gcp_deployer_flavor import GCPDeployerSettings
@step
def greet(name: str) -> str:
return f"Hello {name}!"
settings = {
"deployer": GCPDeployerSettings(
use_secret_manager=False
)
}
@pipeline(settings=settings)
def greet_pipeline(name: str = "John"):
greet(name=name)
```
### Resource and scaling settings
You can specify the resource and scaling requirements for the pipeline deployment using the `ResourceSettings` class at the pipeline level, as described in our documentation on [resource settings](https://docs.zenml.io/concepts/steps_and_pipelines/configuration#resource-settings):
```python
from zenml import step, pipeline
from zenml.config import ResourceSettings
resource_settings = ResourceSettings(
cpu_count=2,
memory="32GB",
min_replicas=0,
max_replicas=10,
max_concurrency=50
)
...
@pipeline(settings={"resources": resource_settings})
def greet_pipeline(name: str = "John"):
greet(name=name)
```
If resource settings are not set, the default values are as follows:
* `cpu_count` is `1`
* `memory` is `2GiB`
* `min_replicas` is `1`
* `max_replicas` is `100`
* `max_concurrency` is `80`
{% hint style="warning" %}
GCP Cloud Run defines specific rules concerning allowed combinations of CPU and memory values. The following rules apply (as of October 2025):
* CPU constraints:
* fractional CPUs: 0.08 to < 1.0 (in increments of 0.01)
* integer CPUs: 1, 2, 4, 6, or 8 (no fractional values allowed >= 1.0)
* minimum memory requirements per CPU configuration:
* <=1 CPU: 128 MiB minimum
* 2 CPU: 128 MiB minimum
* 4 CPU: 2 GiB minimum
* 6 CPU: 4 GiB minimum
* 8 CPU: 4 GiB minimum
For more information, see the [GCP Cloud Run documentation](https://cloud.google.com/run/docs/configuring/services/cpu).
Specifying `cpu_count` and `memory` values that are not valid according to these rules will **not** result in an error when deploying the pipeline. Instead, the values will be automatically adjusted to the nearest matching valid values that satisfy the rules. Some examples:
* `cpu_count=0.25` and `memory="100MiB"` will be adjusted to `cpu_count=0.25` and `memory="128MiB"`
* `cpu_count=1.5` and `memory` not specified will be adjusted to `cpu_count=2` and `memory="128MiB"`
* `cpu_count=6` and `memory="1GB"` will be adjusted to `cpu_count=6` and `memory="4GiB"`
{% endhint %}
---
### Book/Component Guide/Deployers/Huggingface (docs/book/component-guide/deployers/huggingface.md)
---
description: Deploying your pipelines to Hugging Face Spaces.
---
# Hugging Face Deployer
[Hugging Face Spaces](https://huggingface.co/spaces) is a platform for hosting and sharing machine learning applications. The Hugging Face deployer is a [deployer](./) flavor included in the ZenML Hugging Face integration that deploys your pipelines to Hugging Face Spaces as Docker-based applications.
{% hint style="warning" %}
This component is only meant to be used within the context of a [remote ZenML installation](https://docs.zenml.io/getting-started/deploying-zenml). Usage with a local ZenML setup may lead to unexpected behavior!
{% endhint %}
## When to use it
You should use the Hugging Face deployer if:
* you're already using Hugging Face for model hosting or datasets.
* you want to share your AI pipelines as publicly accessible or private Spaces.
* you're looking for a simple, managed platform for deploying Docker-based applications.
* you want to leverage Hugging Face's infrastructure for hosting your pipeline deployments.
* you need an easy way to showcase ML workflows to the community.
## How to deploy it
{% hint style="info" %}
The Hugging Face deployer requires a remote ZenML installation. You must ensure that you are connected to the remote ZenML server before using this stack component.
{% endhint %}
In order to use a Hugging Face deployer, you need to first deploy [ZenML to the cloud](https://docs.zenml.io/getting-started/deploying-zenml/).
The only other requirement is having a Hugging Face account and generating an access token with write permissions.
## How to use it
To use the Hugging Face deployer, you need:
* The ZenML `huggingface` integration installed. If you haven't done so, run
```shell
zenml integration install huggingface
```
* [Docker](https://www.docker.com) installed and running.
* A [remote artifact store](https://docs.zenml.io/stacks/artifact-stores/) as part of your stack.
* A [remote container registry](https://docs.zenml.io/stacks/container-registries/) as part of your stack.
* A [Hugging Face access token with write permissions](https://huggingface.co/settings/tokens)
### Hugging Face credentials
You need a Hugging Face access token with write permissions to deploy pipelines. You can create one at [https://huggingface.co/settings/tokens](https://huggingface.co/settings/tokens).
You have two options to provide credentials to the Hugging Face deployer:
* Pass the token directly when registering the deployer using the `--token` parameter
* (recommended) Store the token in a ZenML secret and reference it using [secret reference syntax](https://docs.zenml.io/how-to/project-setup-and-management/interact-with-secrets)
### Registering the deployer
The deployer can be registered as follows:
```shell
# Option 1: Direct token (not recommended for production)
zenml deployer register \
--flavor=huggingface \
--token=
# Option 2: Using a secret (recommended)
zenml secret create hf_token --token=
zenml deployer register \
--flavor=huggingface \
--token='{{hf_token.token}}'
```
### Configuring the stack
With the deployer registered, it can be used in the active stack:
```shell
# Register and activate a stack with the new deployer
zenml stack register -D ... --set
```
{% hint style="info" %}
ZenML will build a Docker image called `/zenml:` which will be referenced in a Dockerfile deployed to your Hugging Face Space. Check out [this page](https://docs.zenml.io/how-to/customize-docker-builds/) if you want to learn more about how ZenML builds these images and how you can customize them.
{% endhint %}
You can now [deploy any ZenML pipeline](https://docs.zenml.io/concepts/deployment) using the Hugging Face deployer:
```shell
zenml pipeline deploy --name my_deployment my_module.my_pipeline
```
### Additional configuration
For additional configuration of the Hugging Face deployer, you can pass the following `HuggingFaceDeployerSettings` attributes defined in the `zenml.integrations.huggingface.flavors.huggingface_deployer_flavor` module when configuring the deployer or defining or deploying your pipeline:
* Basic settings common to all Deployers:
* `auth_key`: A user-defined authentication key to use to authenticate with deployment API calls.
* `generate_auth_key`: Whether to generate and use a random authentication key instead of the user-defined one.
* `lcm_timeout`: The maximum time in seconds to wait for the deployment lifecycle management to complete.
* Hugging Face Spaces-specific settings:
* `space_hardware` (default: `None`): Hardware tier for the Space (e.g., `'cpu-basic'`, `'cpu-upgrade'`, `'t4-small'`, `'t4-medium'`, `'a10g-small'`, `'a10g-large'`). If not specified, uses free CPU tier. See [Hugging Face Spaces GPU documentation](https://huggingface.co/docs/hub/spaces-gpus) for available options and pricing.
* `space_storage` (default: `None`): Persistent storage tier for the Space (e.g., `'small'`, `'medium'`, `'large'`). If not specified, no persistent storage is allocated.
* `private` (default: `True`): Whether to create the Space as private. Set to `False` to make the Space publicly visible to everyone.
* `app_port` (default: `8000`): Port number where your deployment server listens. Defaults to 8000 (ZenML server default). Hugging Face Spaces will route traffic to this port.
Check out [this docs page](https://docs.zenml.io/concepts/steps_and_pipelines/configuration) for more information on how to specify settings.
For example, if you wanted to deploy on GPU hardware with persistent storage, you would configure settings as follows:
```python
from zenml.integrations.huggingface.deployers import HuggingFaceDeployerSettings
huggingface_settings = HuggingFaceDeployerSettings(
space_hardware="t4-small",
space_storage="small",
# private=True is the default for security
)
@pipeline(
settings={
"deployer": huggingface_settings
}
)
def my_pipeline(...):
...
```
### Managing deployments
Once deployed, you can manage your deployments using the ZenML CLI:
```shell
# List all deployments
zenml deployment list
# Get deployment status
zenml deployment describe
# Get deployment logs
zenml deployment logs
# Delete a deployment
zenml deployment delete
```
The deployed pipeline will be available as a Hugging Face Space at:
```
https://huggingface.co/spaces//-
```
By default, the space prefix is `zenml` but this can be configured using the `space_prefix` parameter when registering the deployer.
## Important Requirements
### Secure Secrets and Environment Variables
{% hint style="success" %}
The Hugging Face deployer handles secrets and environment variables **securely** using Hugging Face's Space Secrets and Variables API. Credentials are **never** written to the Dockerfile.
{% endhint %}
**How it works:**
- Environment variables are set using `HfApi.add_space_variable()` - stored securely by Hugging Face
- Secrets are set using `HfApi.add_space_secret()` - encrypted and never exposed in the Space repository
- **Nothing is baked into the Dockerfile** - no risk of leaked credentials even in public Spaces
**What this means:**
- ✅ Safe to use with both private and public Spaces
- ✅ Secrets remain encrypted and hidden from view
- ✅ Environment variables are managed through HF's secure API
- ✅ No credentials exposed in Dockerfile or repository files
This secure approach ensures that if you choose to make your Space public (`private=False`), credentials remain protected and are never visible to anyone viewing your Space's repository.
### Container Registry Requirement
{% hint style="warning" %}
The Hugging Face deployer **requires** a container registry to be part of your ZenML stack. The Docker image must be pre-built and pushed to a **publicly accessible** container registry.
{% endhint %}
**Why public access is required:**
Hugging Face Spaces cannot authenticate with private Docker registries when building Docker Spaces. The platform pulls your Docker image during the build process, which means it needs public access.
**Recommended registries:**
- [Docker Hub](https://hub.docker.com/) public repositories
- [GitHub Container Registry (GHCR)](https://ghcr.io) with public images
- Any other public container registry
**Example setup with GitHub Container Registry:**
```shell
# Register a public container registry
zenml container-registry register ghcr_public \
--flavor=default \
--uri=ghcr.io/
# Add it to your stack
zenml stack update --container-registry=ghcr_public
```
### Configuring iframe Embedding (X-Frame-Options)
By default, ZenML's deployment server sends an `X-Frame-Options` header that prevents the deployment UI from being embedded in iframes. This causes issues with Hugging Face Spaces, which displays deployments in an iframe.
**To fix this**, you must configure your pipeline's `DeploymentSettings` to disable the `X-Frame-Options` header:
```python
from zenml import pipeline
from zenml.config import DeploymentSettings, SecureHeadersConfig
# Configure deployment settings
deployment_settings = DeploymentSettings(
app_title="My ZenML Pipeline",
app_description="ML pipeline deployed to Hugging Face Spaces",
app_version="1.0.0",
secure_headers=SecureHeadersConfig(
xfo=False, # Disable X-Frame-Options to allow iframe embedding
server=True,
hsts=False,
content=True,
referrer=True,
cache=True,
permissions=True,
),
cors={
"allow_origins": ["*"],
"allow_methods": ["GET", "POST", "OPTIONS"],
"allow_headers": ["*"],
"allow_credentials": False,
},
)
@pipeline(
name="my_hf_pipeline",
settings={"deployment": deployment_settings}
)
def my_pipeline():
# Your pipeline steps here
pass
```
Without this configuration, the Hugging Face Spaces UI will show a blank page or errors when trying to display your deployment.
## Additional Resources
* [Hugging Face Spaces Documentation](https://huggingface.co/docs/hub/spaces)
* [Docker Spaces Guide](https://huggingface.co/docs/hub/spaces-sdks-docker)
* [Hugging Face Hardware Options](https://huggingface.co/docs/hub/spaces-gpus)
* [ZenML Deployment Concepts](https://docs.zenml.io/concepts/deployment)
---
### Book/Component Guide/Deployers/Kubernetes (docs/book/component-guide/deployers/kubernetes.md)
---
description: Deploying your pipelines to Kubernetes clusters.
---
# Kubernetes Deployer
[Kubernetes](https://kubernetes.io/) is the industry-standard container orchestration platform for deploying and managing containerized applications at scale. The Kubernetes deployer is a [deployer](./) flavor included in the ZenML Kubernetes integration that deploys your pipelines to any Kubernetes cluster as production-ready services.
{% hint style="warning" %}
This component is only meant to be used within the context of a [remote ZenML installation](https://docs.zenml.io/getting-started/deploying-zenml/). Usage with a local ZenML setup may lead to unexpected behavior!
{% endhint %}
## When to use it
You should use the Kubernetes deployer if:
* you already have a Kubernetes cluster (EKS, GKE, AKS, or self-managed).
* you need fine-grained control over deployment configuration (resources, networking, security).
* you want production-grade features like autoscaling, health probes, and high availability.
* you need to deploy to on-premises infrastructure or air-gapped environments.
* you want to leverage existing Kubernetes expertise and tooling in your organization.
* you need to integrate with existing Kubernetes resources (Ingress, NetworkPolicies, ServiceMonitors, etc.).
## Prerequisites
To use the Kubernetes deployer, you need:
* The ZenML `kubernetes` integration installed:
```shell
zenml integration install kubernetes
```
* [Docker](https://www.docker.com) installed and running.
* A [remote artifact store](https://docs.zenml.io/stacks/artifact-stores/) as part of your stack.
* A [remote container registry](https://docs.zenml.io/stacks/container-registries/) as part of your stack.
* A running Kubernetes cluster (version 1.21 or higher recommended)
* [Kubernetes cluster access](#kubernetes-access-and-permissions) either via `kubectl` or a service connector.
### Kubernetes access and permissions
You have two different options to provide cluster access to the Kubernetes deployer:
* use `kubectl` to authenticate locally with your Kubernetes cluster
* (recommended) configure [a Kubernetes Service Connector](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/kubernetes-service-connector) and link it to the Kubernetes deployer stack component.
#### Kubernetes Permissions
The Kubernetes deployer needs the following permissions in the target namespace:
* **Deployments**: `create`, `get`, `list`, `watch`, `update`, `delete`
* **Services**: `create`, `get`, `list`, `watch`, `update`, `delete`
* **Secrets**: `create`, `get`, `list`, `watch`, `update`, `delete`
* **Pods**: `get`, `list`, `watch` (for logs and status)
* **Namespaces**: `create`, `get` (if creating namespaces)
If using additional resources (Ingress, HPA, NetworkPolicy, etc.), you'll also need permissions for those resource types.
#### Configuration use-case: local kubectl with context
This configuration assumes you have configured `kubectl` to authenticate with your cluster (i.e. by running `kubectl config use-context `). This is the easiest way to get started:
```shell
zenml deployer register \
--flavor=kubernetes \
--kubernetes_context= \
--kubernetes_namespace=zenml-deployments
```
{% hint style="warning" %}
This setup is not portable to other machines unless they have the same kubectl context configured. For production and team environments, use a service connector instead.
{% endhint %}
#### Configuration use-case: Kubernetes Service Connector
This is the recommended approach for production and team environments. It makes credentials portable and manageable:
```shell
# Register a Kubernetes service connector
zenml service-connector register \
--type kubernetes \
--auth-method=token \
--token= \
--server= \
--certificate_authority= \
--resource-type kubernetes-cluster
# Register the deployer and link it to the connector
zenml deployer register \
--flavor=kubernetes \
--kubernetes_namespace=zenml-deployments \
--connector
```
See the [Kubernetes Service Connector documentation](https://docs.zenml.io/how-to/infrastructure-deployment/auth-management/kubernetes-service-connector) for more authentication methods including:
* Service account tokens
* Kubeconfig files
* Cloud provider authentication (EKS, GKE, AKS)
#### Configuration use-case: In-cluster deployment
If your ZenML server runs inside the same Kubernetes cluster where you want to deploy pipelines, you can use in-cluster authentication:
```shell
zenml deployer register
---
### AGENTS (AGENTS.md)
# ZenML Codex Agent Guidelines
ZenML is an extensible open-source MLOps framework for creating production-ready
ML pipelines. This root guide contains rules that must be loaded for every
Codex session in this repository. For detailed workflows, examples, and
subsystem recipes, use `.agents/skills/zenml-repo-workflows/SKILL.md`.
## Project Structure
- `src/zenml/` - Core source code.
- `tests/` - Unit and integration tests.
- `docs/book/` - Source documentation.
- `examples/` - Example projects.
- `scripts/` - Development utilities.
## Always-Loaded Rules
- Use US English spelling in code, comments, docstrings, and documentation.
- Use Python 3.10+ compatible code.
- Type hint function parameters and return values.
- Follow Google Python style for docstrings. Include `Args`, `Returns`,
`Yields`, and `Raises` sections whenever the function contract requires them;
do not use a summary-only docstring to omit applicable sections.
- Prefer clear names and small functions over explanatory comments.
- Comments should explain intent, trade-offs, constraints, invariants, and
tricky edge cases. Avoid comments that restate obvious code.
- Do not use multi-line banner comments to group classes or functions.
- Prefer typed contracts over `getattr`/`hasattr` capability checks when static
typing can express the requirement.
- Private methods and functions with a leading underscore should not be called
outside their class or module.
- Integrations should avoid using ZenML private methods because externalized
integrations will not be protected by in-repo type checks.
## Common Commands
- Format before committing: `bash scripts/format.sh`.
- Check quality: `bash scripts/lint.sh`.
- Run targeted tests only: `pytest tests/unit/path/to/test_file.py` or
`pytest tests/unit/path/to/test_file.py::test_specific_function`.
- Do NOT run the entire local test suite by default; many tests need special
environments.
- If you make changes after running tests, rerun the relevant tests.
## Branches, Git, and PRs
- `develop` is the primary working branch, not `main`.
- Always branch off `develop` for new work.
- PRs should target `develop`.
- `main` is only updated during releases.
- Use targeted `git add`; do not stage unrelated files.
- Never add anything from `design/` to git history.
- PR titles should be concise and human-readable, without prefixes like
`feat:`.
- Every PR must have exactly one release-notes label:
`release-notes` or `no-release-notes`.
- When pushing new commits to an open PR, check whether the PR description needs
updating.
## Security
- NEVER commit secrets, API keys, tokens, passwords, or credentials.
- Use environment variables or ZenML secret management for sensitive data.
- Validate and sanitize user inputs.
- If secrets are accidentally committed, notify the team immediately.
## Architecture and API Safety
- Check whether a class or function is exported in `zenml.__init__` before
changing a public-looking interface.
- Public APIs need backward-compatible evolution or deprecation.
- For internal non-underscore methods, search for usages and update all
internal callers.
- The term "model" can mean Pydantic models, machine learning models, or ZenML
model namespaces. Be explicit when writing or reviewing.
- When changing cross-cutting features, trace the full path through CLI,
client, server, models, schemas, migrations, tests, and docs.
## FastAPI and Runtime Rules
- ZenML OSS FastAPI work expects FastAPI, SQLModel, SQLAlchemy 2.0, and
Pydantic v2 patterns.
- Implement synchronous `def` route handlers for OSS Codex contributions.
- Keep shared state inside FastAPI dependency injection or the application
factory; never introduce fresh global variables outside initialization.
- Start routes, dependencies, and services with guard clauses.
- Raise `HTTPException` with precise status codes for expected errors.
- Use Pydantic models for route inputs and outputs.
- When changing server framework (e.g., `fastapi`) or database library versions,
check whether related OpenTelemetry SDK, exporter, and instrumentation
dependencies also need updates. Breaking changes in instrumented libraries can
require coordinated OTel updates. Keep OTel SDK/exporter versions aligned with
the matching OpenTelemetry instrumentation beta line.
- Code outside `src/zenml/zen_server/` should NEVER import from `zen_server/`.
## Database and Storage Rules
- ZenML uses SQLModel and SQLAlchemy; no raw SQL unless absolutely necessary.
- Database schema changes require Alembic migrations.
- Never modify existing migrations that are already on `main` or `develop`.
- Always consider backward compatibility for rolling deployments.
- Test migration upgrade paths with `alembic upgrade head` for meaningful
schema changes.
- Code outside `zen_stores/` should not import SQL-related code directly from
`zen_stores`; use `Client` or, rarely, `client.zen_store`.
## Subsystem Pointers
Load `.agents/skills/zenml-repo-workflows/SKILL.md` for detailed guidance on
tests, PRs, migrations, docs, integrations, models, orchestrators, FastAPI,
server code, and storage.
Directory-specific reminders:
- `docs/book/AGENTS.md` - documentation source, GitBook links, and docs checks.
- `src/zenml/cli/AGENTS.md` - CLI import rules and filter/client coupling.
- `src/zenml/integrations/AGENTS.md` - integration flavor import rules.
- `src/zenml/models/AGENTS.md` - domain model compatibility and filter fields.
- `src/zenml/orchestrators/AGENTS.md` - orchestrator IDs and dynamic pipelines.
- `src/zenml/zen_server/AGENTS.md` - server import boundary and endpoint shape.
- `src/zenml/zen_stores/migrations/AGENTS.md` - Alembic migration guidance.
- `src/zenml/zen_stores/schemas/AGENTS.md` - ORM schema and SQL import rules.
## Reviewer Checklist
- Integration PRs: no top-level integration-library imports in flavor files.
- Orchestrator PRs: `get_orchestrator_run_id` is unique per run and stable for
all steps in that run.
- Filter model changes: matching client method signature and body are updated.
- Private method changes: all internal usages are updated.
- Import checks: no `zen_server` imports outside `zen_server`.
- Import checks: no direct SQL imports outside `zen_stores`.
- Model changes: adding properties is usually OK; deleting, renaming, or
incompatible type changes are risky.
- Dependency bumps: dropping old version support is breaking.
- Scheduling changes: check both legacy schedule and trigger stacks.
- Step operator changes: check `BaseStepOperator`, `StepLauncher`, and at least
one concrete integration.
## Documentation Rules
- Documentation source files live in `docs/book/`.
- Do not edit generated docs directories such as `docs/mkdocs/` or
`docs/site/`.
- When adding or removing docs pages, update the relevant `toc.md`.
- Assets belong in a `.gitbook` folder beside the relevant `toc.md`.
For human contributors, see `CONTRIBUTING.md`.
---
### Changelog.Json (changelog.json)
[
{
"id": 4,
"slug": "dynamic-pipelines",
"title": "Dynamic pipelines are now available",
"description": "Introduced Dynamic Pipelines as an experimental feature, allowing you to generate DAG structures at runtime using native Python control flow (loops, conditionals). Key capabilities include dynamic parallelization, Map/Reduce patterns over collections, and granular runtime configuration (inline vs. isolated). Supported on local, Kubernetes, AWS Sagemaker, and Google Cloud Vertex orchestrators. Dynamic pipelines can be run from snapshots with configurable parameters, and include improvements to Kubernetes orchestrator handling and step mapping operations that return future objects with an `unpack()` method for better control flow.",
"published_at": "2025-12-05T06:42:19Z",
"published": true,
"audience": "oss",
"labels": [
"feature"
],
"docs_url": "https://docs.zenml.io/concepts/steps_and_pipelines/dynamic_pipelines",
"should_highlight": true
},
{
"id": 3,
"slug": "panels-are-now-resizable",
"title": "Panels are now resizable",
"feature_image_url": "https://public-flavor-logos.s3.eu-central-1.amazonaws.com/whats_new/resizable-panels.gif",
"description": "You can now resize the panels in the dashboard to see more or less content at once. This is useful for when you want to see more or less information about a run or a pipeline.",
"published": true,
"published_at": "2025-10-23T00:00:00Z",
"should_highlight": false,
"audience": "all",
"labels": [
"feature"
]
},
{
"id": 2,
"slug": "introducing-pipeline-deployments",
"title": "Introducing Pipeline Deployments",
"description": "Pipeline Deployments turn pipelines into persistent HTTP services with warm state, reducing cold start latency by 10-100x while maintaining full traceability. Learn more in our blog post.",
"video_url": "https://www.youtube-nocookie.com/embed/whQytRE7kC8",
"learn_more_url": "https://www.zenml.io/blog/why-pipelines-are-the-right-abstraction-for-real-time-ai-agents-included",
"published": true,
"published_at": "2025-10-02T00:00:00Z",
"should_highlight": false,
"audience": "all",
"labels": [
"feature"
]
},
{
"id": 1,
"slug": "new-timeline-view-for-runs",
"title": "New Timeline View for Runs",
"description": "We've added a new timeline view for runs to help you visualize the execution of your pipelines.",
"feature_image_url": "https://public-flavor-logos.s3.eu-central-1.amazonaws.com/whats_new/new_feature_timeline.png",
"docs_url": "https://docs.zenml.io/concepts/dashboard-features#timeline-view",
"published": true,
"published_at": "2025-09-12T00:00:00Z",
"should_highlight": false,
"audience": "all",
"labels": [
"feature"
]
}
]
---
### CLA (CLA.md)
# Fiduciary License Agreement 2.0
based on the
## Individual Contributor Exclusive License Agreement
(including the Traditional Patent License OPTION)
Thank you for your interest in contributing to ZenML by ZenML GmbH ("We" or "Us").
The purpose of this contributor agreement ("Agreement") is to clarify and document the rights granted by contributors to Us. To make this document effective, please follow the instructions at https://zenml.io/cla/.
### 0. Preamble
Software is deeply embedded in all aspects of our lives and it is important that it empower, rather than restrict us. Free Software gives everybody the rights to use, understand, adapt and share software. These rights help support other fundamental freedoms like freedom of speech, press and privacy.
Development of Free Software can follow many patterns. In some cases whole development is handled by a sole programmer or a small group of people. But usually, the creation and maintenance of software is a complex process that requires the contribution of many individuals. This also affects who owns the rights to the software. In the latter case, rights in software are owned jointly by a great number of individuals.
To tackle this issue some projects require a full copyright assignment to be signed by all contributors. The problem with such assignments is that they often lack checks and balances that would protect the contributors from potential abuse of power from the new copyright holder.
FSFE’s Fiduciary License Agreement (FLA) was created by the Free Software Foundation Europe e.V. with just that in mind – to concentrate all deciding power within one entity and prevent fragmentation of rights on one hand, while on the other preventing that single entity from abusing its power. The main aim is to ensure that the software covered under the FLA will forever remain Free Software.
This process only serves for the transfer of economic rights. So-called moral rights (e.g. authors right to be identified as author) remain with the original author(s) and are inalienable.
How to use this FLA
If You are an employee and have created the Contribution as part of your employment, You need to have Your employer approve this Agreement or sign the Entity version of this document. If You do not own the Copyright in the entire work of authorship, any other author of the Contribution should also sign this – in any event, please contact Us at support@zenml.io
### 1. Definitions
"You" means the individual Copyright owner who Submits a Contribution to Us.
"Contribution" means any original work of authorship, including any original modifications or additions to an existing work of authorship, Submitted by You to Us, in which You own the Copyright.
"Copyright" means all rights protecting works of authorship, including copyright, moral and neighboring rights, as appropriate, for the full term of their existence.
"Material" means the software or documentation made available by Us to third parties. When this Agreement covers more than one software project, the Material means the software or documentation to which the Contribution was Submitted. After You Submit the Contribution, it may be included in the Material.
"Submit" means any act by which a Contribution is transferred to Us by You by means of tangible or intangible media, including but not limited to electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, Us, but excluding any transfer that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution."
"Documentation" means any non-software portion of a Contribution.
### 2. License grant
#### 2.1 Copyright license to Us
Subject to the terms and conditions of this Agreement, You hereby grant to Us a worldwide, royalty-free, exclusive, perpetual and irrevocable (except as stated in Section 8.2) license, with the right to transfer an unlimited number of non-exclusive licenses or to grant sublicenses to third parties, under the Copyright covering the Contribution to use the Contribution by all means, including, but not limited to:
publish the Contribution,
modify the Contribution,
prepare derivative works based upon or containing the Contribution and/or to combine the Contribution with other Materials,
reproduce the Contribution in original or modified form,
distribute, to make the Contribution available to the public, display and publicly perform the Contribution in original or modified form.
#### 2.2 Moral rights
Moral Rights remain unaffected to the extent they are recognized and not waivable by applicable law. Notwithstanding, You may add your name to the attribution mechanism customary used in the Materials you Contribute to, such as the header of the source code files of Your Contribution, and We will respect this attribution when using Your Contribution.
#### 2.3 Copyright license back to You
Upon such grant of rights to Us, We immediately grant to You a worldwide, royalty-free, non-exclusive, perpetual and irrevocable license, with the right to transfer an unlimited number of non-exclusive licenses or to grant sublicenses to third parties, under the Copyright covering the Contribution to use the Contribution by all means, including, but not limited to:
publish the Contribution,
modify the Contribution,
prepare derivative works based upon or containing the Contribution and/or to combine the Contribution with other Materials,
reproduce the Contribution in original or modified form,
distribute, to make the Contribution available to the public, display and publicly perform the Contribution in original or modified form.
This license back is limited to the Contribution and does not provide any rights to the Material.
### 3. Patents
#### 3.1 Patent license
Subject to the terms and conditions of this Agreement You hereby grant to Us and to recipients of Materials distributed by Us a worldwide, royalty-free, non-exclusive, perpetual and irrevocable (except as stated in Section 3.2) patent license, with the right to transfer an unlimited number of non-exclusive licenses or to grant sublicenses to third parties, to make, have made, use, sell, offer for sale, import and otherwise transfer the Contribution and the Contribution in combination with any Material (and portions of such combination). This license applies to all patents owned or controlled by You, whether already acquired or hereafter acquired, that would be infringed by making, having made, using, selling, offering for sale, importing or otherwise transferring of Your Contribution(s) alone or by combination of Your Contribution(s) with any Material.
#### 3.2 Revocation of patent license
You reserve the right to revoke the patent license stated in section 3.1 if We make any infringement claim that is targeted at your Contribution and not asserted for a Defensive Purpose. An assertion of claims of the Patents shall be considered for a "Defensive Purpose" if the claims are asserted against an entity that has filed, maintained, threatened, or voluntarily participated in a patent infringement lawsuit against Us or any of Our licensees.
### 4. License obligations by Us
We agree to (sub)license the Contribution or any Materials containing, based on or derived from your Contribution under the terms of any licenses the Free Software Foundation classifies as Free Software License and which are approved by the Open Source Initiative as Open Source licenses.
More specifically and in strict accordance with the above paragraph, we agree to (sub)license the Contribution or any Materials containing, based on or derived from the Contribution only under the terms of the following license(s) Apache-2.0 (including any right to adopt any future version of a license if permitted).
We agree to license patents owned or controlled by You only to the extent necessary to (sub)license Your Contribution(s) and the combination of Your Contribution(s) with the Material under the terms of any licenses the Free Software Foundation classifies as Free Software licenses and which are approved by the Open Source Initiative as Open Source licenses..
### 5. Disclaimer
THE CONTRIBUTION IS PROVIDED "AS IS". MORE PARTICULARLY, ALL EXPRESS OR IMPLIED WARRANTIES INCLUDING, WITHOUT LIMITATION, ANY IMPLIED WARRANTY OF SATISFACTORY QUALITY, FITNESS FOR A PARTICULAR PURPOSE AND NON-INFRINGEMENT ARE EXPRESSLY DISCLAIMED BY YOU TO US AND BY US TO YOU. TO THE EXTENT THAT ANY SUCH WARRANTIES CANNOT BE DISCLAIMED, SUCH WARRANTY IS LIMITED IN DURATION AND EXTENT TO THE MINIMUM PERIOD AND EXTENT PERMITTED BY LAW.
### 6. Consequential damage waiver
TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, IN NO EVENT WILL YOU OR WE BE LIABLE FOR ANY LOSS OF PROFITS, LOSS OF ANTICIPATED SAVINGS, LOSS OF DATA, INDIRECT, SPECIAL, INCIDENTAL, CONSEQUENTIAL AND EXEMPLARY DAMAGES ARISING OUT OF THIS AGREEMENT REGARDLESS OF THE LEGAL OR EQUITABLE THEORY (CONTRACT, TORT OR OTHERWISE) UPON WHICH THE CLAIM IS BASED.
### 7. Approximation of disclaimer and damage waiver
IF THE DISCLAIMER AND DAMAGE WAIVER MENTIONED IN SECTION 5. AND SECTION 6. CANNOT BE GIVEN LEGAL EFFECT UNDER APPLICABLE LOCAL LAW, REVIEWING COURTS SHALL APPLY LOCAL LAW THAT MOST CLOSELY APPROXIMATES AN ABSOLUTE WAIVER OF ALL CIVIL OR CONTRACTUAL LIABILITY IN CONNECTION WITH THE CONTRIBUTION.
### 8. Term
#### 8.1 This Agreement shall come into effect upon Your acceptance of the terms and conditions.
#### 8.2 This Agreement shall apply for the term of the copyright and patents licensed here. However, You shall have the right to terminate the Agreement if We do not fulfill the obligations as set forth in Section 4. Such termination must be made in writing.
#### 8.3 In the event of a termination of this Agreement Sections 5., 6., 7., 8., and 9. shall survive such termination and shall remain in full force thereafter. For the avoidance of doubt, Free and Open Source Software (sub)licenses that have already been granted for Contributions at the date of the termination shall remain in full force after the termination of this Agreement.
### 9. Miscellaneous
#### 9.1 This Agreement and all disputes, claims, actions, suits or other proceedings arising out of this agreement or relating in any way to it shall be governed by the laws of Germany excluding its private international law provisions.
#### 9.2 This Agreement sets out the entire agreement between You and Us for Your Contributions to Us and overrides all other agreements or understandings.
#### 9.3 In case of Your death, this agreement shall continue with Your heirs. In case of more than one heir, all heirs must exercise their rights through a commonly authorized person.
#### 9.4 If any provision of this Agreement is found void and unenforceable, such provision will be replaced to the extent possible with a provision that comes closest to the meaning of the original provision and that is enforceable. The terms and conditions set forth in this Agreement shall apply notwithstanding any failure of essential purpose of this Agreement or any limited remedy to the maximum extent possible under law.
#### 9.5 You agree to notify Us of any facts or circumstances of which you become aware that would make this Agreement inaccurate in any respect.
**You**
Date:_______________________________
Name:_______________________________
Title:______________________________
Address:____________________________
**Us**
Date:_______________________________
Name:_______________________________
Title:_______________________________
Address:_______________________________
---
### CLAUDE (CLAUDE.md)
# ZenML Claude Code Guidelines
This document provides guidance for Claude Code when working with the ZenML codebase. ZenML is an extensible, open-source MLOps framework for creating production-ready ML pipelines.
## Project Structure
- `/src/zenml/` - Core source code
- `/tests/` - Test suite (unit, integration)
- `/docs/` - Documentation
- `/examples/` - Example projects
- `/scripts/` - Development utilities
Use filesystem navigation tools to explore the codebase structure as needed.
## Use ZenML Docs via MCP
Claude Code can query ZenML documentation via the built-in GitBook MCP server: https://docs.zenml.io/~gitbook/mcp. This enables real-time, source-of-truth lookups from the docs while you code, reducing hallucinations and speeding up feature discovery.
Quick setup (CLI):
```bash
claude mcp add zenmldocs --transport http https://docs.zenml.io/~gitbook/mcp
```
Note: The MCP server indexes the latest released docs, not the develop branch. For full setup details and editor alternatives, see docs/book/reference/llms-txt.md.
## Code Style & Quality Standards
- **Use US English spelling** in all code, comments, docstrings, and documentation (e.g., "initialize", "stabilize", "color"). The CI enforces this via `typos` (configured in `.typos.toml`).
### Commenting policy — explain why, not what
- Use comments to document intent, trade‑offs, constraints, invariants, and tricky edge cases—i.e., why the code is this way—rather than narrating changes. Prefer self‑explanatory code; add comments only where extra context is needed. Write for a reader 6+ months later.
- Use for: complex logic/algorithms, non‑obvious design decisions, business rules/constraints, API purpose/contracts, edge cases.
- Avoid: change‑tracking comments ("Updated from previous version", "New implementation", "Changed to use X instead of Y", "Refactored this section").
- Avoid simple explanatory comments, where it is already clear from the code itself.
- Avoid useless one-line comments interleaved with code that merely narrate the implementation. Favor expressive names and small, focused functions.
```python
# Bad
x = x + 1 # increment x
# Good
count += 1
```
- Do not use multi-line banner comments to group classes/functions. Use a concise module-level docstring or split code into dedicated modules.
```python
# Bad
"""
===== Dataset Loaders =====
"""
class CSVLoader: ...
class ParquetLoader: ...
# Good (module-level docstring at top)
"""
Dataset loaders used by data ingestion (CSV, Parquet).
"""
class CSVLoader: ...
class ParquetLoader: ...
```
### Formatting and Linting
- Format code with: `bash scripts/format.sh` (requires Python environment with dev dependencies)
- Run this before every commit to ensure proper formatting
- Automatically fixes and formats code using ruff and yamlfix
- Check code quality with: `bash scripts/lint.sh`
- Unlike format.sh, this doesn't auto-fix issues
- Runs Ruff, pydoclint (on `src/zenml tests/harness`), yamlfix, zizmor, and mypy
- Note: Full mypy check is slow on the entire codebase
- For faster checks, run mypy directly on specific files: `mypy src/zenml/path/to/file.py`
- The primary code style is enforced by ruff, configured in `pyproject.toml`
- YAML formatting uses yamlfix: `yamlfix .github -v`
### Python Standards
- Use Python 3.10+ compatible code
- Follow Google Python style for docstrings. Include `Args`, `Returns`,
`Yields`, and `Raises` sections whenever the function contract requires them;
do not use a summary-only docstring to omit applicable sections.
- Type hint all function parameters and return values
- Use descriptive variable names and documentation
- Keep function size manageable (aim for < 50 lines) though there are exceptions
#### Prefer typing over dynamic attribute checks
- Don't use getattr/hasattr for capability checks when static typing can express the contract
- Prefer Protocols/ABCs, Unions with isinstance narrowing, or typed adapters around untyped third-party objects
- If getattr/hasattr is unavoidable, isolate it in a small helper and expose a typed interface
Example:
```python
# Bad
if hasattr(handler, "close"):
handler.close()
# Good
from typing import Protocol
class Closable(Protocol):
def close(self) -> None: ...
def shutdown(h: Closable) -> None:
h.close()
```
### Util Function Placement
When deciding whether to place a helper function in a utils file or on a class, follow these guidelines:
1. **If a method only makes sense within the context of a class** → Put it on the class
2. **If a static/util method is heavily used by subclasses** → Put it on the parent class
**Rationale for placing methods on classes:**
- Saves imports for users and subclasses
- Subclasses can simply call `self.something()` instead of finding and importing from a util file
- Keeps related functionality co-located
**Example:** `requires_resources_in_orchestration_environment` in `base_orchestrator.py:495-514`
```python
# This is a @staticmethod on BaseOrchestrator, not a standalone util
@staticmethod
def requires_resources_in_orchestration_environment(step: "Step") -> bool:
"""Check whether a step needs special orchestration resources.
Args:
step: The step to check.
Returns:
Whether the step needs special resources in the orchestration
environment.
"""
if step.config.step_operator:
return False
return not step.config.resource_settings.empty
```
This method could be a global util, but it's placed on the class because:
- All orchestrator subclasses frequently need it
- Subclasses can call `self.requires_resources_in_orchestration_environment(step)` without imports
- It's conceptually tied to orchestrator behavior
**When to use utils files:**
- Truly generic functions used across unrelated modules
- Functions that don't logically belong to any class
- Pure utility functions (string manipulation, date formatting, etc.)
**Key utils locations:**
- `src/zenml/utils/` — General utilities
- `src/zenml/orchestrators/utils.py` — Orchestrator-specific utilities
- `src/zenml/orchestrators/step_run_utils.py` — Step execution utilities
- `src/zenml/orchestrators/publish_utils.py` — Status/metadata publishing
### Private Methods and API Stability
Methods and functions starting with `_` (underscore) are **private** and should NOT be called from outside their class or module.
**The rule:**
- `_method()` on a class → only call from within that class
- `_function()` in a utils module → only call from within that module
- This isn't always consistently applied in the codebase, but it's the intended convention
**Backwards compatibility — case-by-case judgment:**
There are no strict written rules; evaluate each change individually:
| Symbol type | Part of public API? | Breaking change if modified? |
|-------------|---------------------|------------------------------|
| Classes/functions exported in `zenml.__init__` | ✅ Definitely public | ⚠️ Yes — requires deprecation |
| Public methods on those classes | ✅ Public | ⚠️ Yes — requires deprecation |
| Internal methods deep in the codebase (no underscore) | ❌ Not intended for users | ✅ No — update all internal usages |
| `_private_method()` | ❌ No | ✅ No — can change freely |
**When changing any non-underscore method:**
1. Check if the class/function is exported in `zenml.__init__` — if so, it's public API
2. Search for usages **within the ZenML codebase** (grep/find references)
3. Update all internal usages
4. For truly internal code not exported at the root, no deprecation needed
**Best practice for integrations (future-proofing):**
> ⚠️ **Integrations should avoid using ZenML private methods**
This is primarily a future concern: when integrations eventually move out of the main ZenML repo (external packages), mypy won't detect if a private method they depend on was changed, leading to silent breakage. Even while integrations live in-repo, using only public APIs is good practice and prepares for this transition.
```python
# Bad - integration code using private method
from zenml.orchestrators.base_orchestrator import BaseOrchestrator
class MyOrchestrator(BaseOrchestrator):
def submit_pipeline(self, ...):
self._some_private_helper() # ❌ Don't do this
# Good - use only public methods or reimplement logic
class MyOrchestrator(BaseOrchestrator):
def submit_pipeline(self, ...):
self.public_method() # ✅ Safe
```
### FastAPI Conventions
Router, service, error-handling, and validation conventions for the server live in `src/zenml/zen_server/AGENTS.md` and load automatically when you work in that directory.
### Testing Requirements
- Most new code requires test coverage
- Key exceptions are when the code involves integrations with external
services. (in those cases we generally test things extensively locally and
in the CI. So the developer might have to run things or set things up
locally first.)
- Tests live in the `/tests/` folder with structure loosely mirroring the main codebase
- Unit tests go in `/tests/unit/`
- Integration tests go in `/tests/integration/`
#### Running Tests
- Do NOT try to run the entire test suite locally - many tests require special environments
- Run targeted tests for the specific components you've changed:
- `pytest tests/unit/path/to/test_file.py`
- `pytest tests/unit/path/to/test_file.py::test_specific_function`
- For full coverage, use CI (see CI section below)
- Some tests use: `bash scripts/test-coverage-xml.sh` (but this won't run all tests)
## Dependencies & Runtime Constraints
- Align contributions with the FastAPI + Pydantic v2 + SQLAlchemy 2.0 + SQLModel stack defined for ZenML OSS; confirm any new dependency in `pyproject.toml` before adoption.
- When changing server framework (e.g., `fastapi`) or database library versions, check whether related OpenTelemetry SDK, exporter, and instrumentation dependencies also need updates. Breaking changes in instrumented libraries can require coordinated OTel updates. Keep OTel SDK/exporter versions aligned with the matching OpenTelemetry instrumentation beta line.
- The OSS runtime forbids async I/O in Claude-authored code even though FastAPI supports it—implement synchronous `def` handlers and delegate background/long-running work to workers or dependency-injected services; this supersedes generic async advice found elsewhere.
- Prefer dependency injection over module-level singletons for clients, caches, and repositories so state management stays testable.
- Cache static or frequently accessed data (e.g., dependency-scoped in-memory caches) and lazy-load heavyweight resources to control cold-start latency.
- Document minimum supported versions when modifying dependency-heavy paths and explain performance trade-offs in PRs when serialization or caching strategies change.
## Development Workflow
### Prerequisites
- Set up a Python environment with ZenML dev dependencies
- Install ZenML in development mode: `pip install -e ".[dev]"`
- Most scripts require these dependencies to be available
- ZenML recommends using `uv` for Python package installation in local environments
- `uv` is also used in CI workflows
- It resolves dependencies more quickly and reliably than pip
- It can resolve dependency conflicts that pip sometimes struggles with or takes a long time to resolve
### Environment Variables
- Several environment variables are useful during ZenML development:
- `ZENML_LOGGING_VERBOSITY=DEBUG`: Controls logging verbosity
- `ZENML_ANALYTICS_OPT_IN=false`: Disables analytics during development
- `MLSTACKS_ANALYTICS_OPT_OUT=true`: Disables MLStacks analytics
- `AUTO_OPEN_DASHBOARD=false`: Prevents automatic dashboard opening
- `ZENML_ENABLE_RICH_TRACEBACK=false`: Disables rich traceback formatting
- `TOKENIZERS_PARALLELISM=false`: Avoids tokenizers parallelism warnings
- Always set the following environment variables:
- `ZENML_ANALYTICS_OPT_IN=false`: Disables analytics during development
- `ZENML_DEBUG=true`: Uses the development ZenML analytics server to avoid
sending analytics to the official ZenML analytics server (IMPORTANT!). This
must be set even if `ZENML_ANALYTICS_OPT_IN=true` because in a client-server
setup, the server controls the client-side analytics opt-in status.
### Branch Management
- **IMPORTANT**: `develop` is our primary working branch, NOT `main`
- Always branch off `develop` for all changes
- All PRs should target the `develop` branch
- The `main` branch is only updated during the release process
- If working on a feature branch that's already based on `develop`, you may need to branch off that feature branch for related changes
### Making Changes
1. Run `bash scripts/format.sh` before every commit
2. Run targeted tests to verify changes (see above)
3. Update documentation for user-facing changes (or ensure that nothing was broken)
4. IMPORTANT: **Before opening a PR or making a large commit**, always run `/simplify` to review changed code for reuse opportunities, quality issues, and efficiency improvements. Fix any issues it finds before committing.
### Security Guidelines
- **NEVER** commit secrets, API keys, tokens, or passwords
- Use environment variables or ZenML's secret management for sensitive data
- Review changes for accidental credential exposure before committing
- If you accidentally commit secrets, notify the team immediately
- Follow the principle of least privilege when implementing access controls
- Validate and sanitize all user inputs
### Database and Migration Guidelines
- Database schema changes require Alembic migrations
- Create migrations with descriptive names: `alembic revision -m "Add X to Y table"`
- Test upgrade path: `alembic upgrade head` (downgrade testing is optional—ZenML doesn't support downgrades in most cases)
- Never modify existing migrations that are already on main/develop branches
- Always consider backward compatibility for rolling deployments
- Include both schema changes and data migrations when needed
- Run `scripts/check-alembic-branches.sh` to verify migration consistency
### Commit Message Guidelines
- Write clear, descriptive commit messages explaining the "why" not just the "what"
- First line should be a concise summary (50 chars or less)
- Use imperative mood: "Add feature" not "Added feature"
- Reference issue numbers when applicable: "Fix user auth bug (#1234)"
- For multi-line messages, add a blank line after the summary
- Example:
```
Add retry logic to artifact upload
Previously, artifact uploads would fail immediately on network errors.
This adds exponential backoff retry logic to handle transient failures.
Fixes #1234
```
### When Implementing Features
- Study existing similar implementations first
- Follow the established patterns in the codebase
- Keep backward compatibility in mind
- Add appropriate error handling
- Document public APIs thoroughly
### Field Description Standards
Templates, the quality bar, and worked examples for Pydantic `Field(description=...)` text in stack component configs live in the `field-descriptions` skill. All descriptions must pass `python scripts/validate_descriptions.py` before merging.
### When Fixing Bugs
- Add regression tests that would have caught the bug
- Understand root cause before implementing fix
- Document the fix in commit messages
### Pull Request Guidelines
- Use human-readable names for PRs (no prefixes like "feat:" or "doc:")
- Keep PR titles concise but descriptive
- Write comprehensive PR descriptions:
- Clearly explain what the changes do
- Mention why the changes are needed
- Detail any important implementation decisions
- Note any areas that need special reviewer attention
- Detailed PR descriptions help both reviewers and release note creation
- Use appropriate PR tags where applicable:
- `internal`: For changes relevant only to ZenML team members
- `documentation`: For changes related to documentation
- `bug`: For bug fixes
- `dependencies`: For dependency updates
- `enhancement`: For new features or improvements
- **REQUIRED: Release Notes Labels** - Every PR must have exactly one of these labels:
- `release-notes`: For user-facing features, significant updates, or changes that should appear in the changelog. Use this for new features, important bug fixes affecting users, API changes, or anything users should know about.
- `no-release-notes`: For internal changes, CI fixes, refactoring, minor bug fixes, documentation-only changes, or anything that doesn't need to be surfaced to users.
- The CI will block merging if neither label is present. When in doubt, use `no-release-notes` for internal/maintenance work.
### Continuous Integration
- ZenML uses a two-tier CI approach:
- **Fast CI**: Runs automatically on all PRs (basic tests, linting, type checking)
- **Full CI**: Includes integration tests, tutorial pipeline regression tests, and more extensive test coverage
- The `run-slow-ci` label triggers full CI testing
- Full CI is required before merging - maintainers will add the label if needed
- Tutorial pipeline testing runs all VSCode tutorial examples against the current branch to catch breaking changes
- If your changes touch integrations or core functionality, mention in the PR that full CI should be run
- CI failures will show in the PR checks - review logs to understand any issues
## Core Concepts
### Important Terminology
- The term "model" has multiple distinct meanings in the codebase:
1. **Pydantic models**: Data structures used throughout the codebase (like `PipelineModel`)
2. **ML models**: Actual machine learning models (PyTorch, sklearn, etc.)
3. **ZenML models**: Namespaces that group artifacts, metadata, and other resources related to an ML model
- Be careful with these terms when reading/writing code to avoid confusion
### Pipeline Architecture
- Pipelines are collections of steps
- Steps produce and consume artifacts
- Artifacts are serialized/deserialized by materializers
- Pipelines are executed by orchestrators
- Stack components provide functionality like storage, orchestration, etc.
### Key Abstractions
- `StackComponent` - Base for stack components
- `Pipeline` - Pipeline definition
- `BaseStep` - Step implementation
- `BaseMaterializer` - Artifact serialization
- `BaseOrchestrator` - Pipeline execution
- `BaseStepOperator` - Remote step execution (submit/status/wait/cancel lifecycle)
## Common Tasks
### Adding New Integrations
1. Create integration package in `/src/zenml/integrations/`
2. Implement required abstractions and register flavors
3. Add tests in `/tests/integrations/`
4. Add documentation in `/docs/book/component-guide/`
### Modifying Core Functionality
1. Understand the impact on existing components
2. Maintain backward compatibility where possible
3. Add comprehensive test coverage
4. Update type hints and documentation
## Task Planning Approach
When tackling complex tasks:
1. Break down the task into smaller sub-tasks
2. Research existing implementations in the codebase
3. Plan approach before implementation
4. Test incrementally as you implement
5. Document design decisions in code comments
## Expert Tips
- ZenML follows a plugin architecture - study how components are registered
- API stability is important - don't break public interfaces
- Review similar PRs for implementation patterns
- Pipeline execution is complex - test thoroughly when modifying
- Centralize FastAPI logging, tracing, and unexpected error handling inside middleware; measure latency/throughput for new endpoints, cache static payloads, lazy-load heavyweight resources, and articulate serialization trade-offs in PR notes.
### Summary Checklist for PR Reviewers
Quick reference for common review concerns. Detailed explanations live in the nested AGENTS.md files.
- [ ] **Integration PRs:** No library imports in flavor files (`src/zenml/integrations/AGENTS.md`)
- [ ] **Orchestrator PRs:** Verify `get_orchestrator_run_id` is unique per run but same for all steps (`src/zenml/orchestrators/AGENTS.md`)
- [ ] **Filter model changes:** Check corresponding client method is updated (`src/zenml/models/AGENTS.md`)
- [ ] **Private method changes:** Check all internal usages (see "Private Methods" above)
- [ ] **Import checking:** No `zen_server` imports from outside `zen_server` (`src/zenml/zen_server/AGENTS.md`)
- [ ] **Import checking:** No direct SQL imports from outside `zen_stores` (`src/zenml/zen_stores/schemas/AGENTS.md`)
- [ ] **Model changes:** Adding properties OK, deleting/making optional is breaking (`src/zenml/models/AGENTS.md`)
- [ ] **Dependency bumps:** If dropping old version support, it's a breaking change (`src/zenml/integrations/AGENTS.md`)
- [ ] **Scheduling changes:** Must span both legacy schedule and trigger stacks (CLI + client + server + models + schemas)
- [ ] **Step operator changes:** Check `BaseStepOperator`, `StepLauncher`, and at least one concrete integration
## Documentation Guidelines
Structure, GitBook conventions, `toc.md` handling, and link checking live in `docs/book/AGENTS.md` and load automatically when you work with files under `docs/book/`.
---
*This document is maintained to help Claude Code work effectively with the
ZenML codebase. For human contributors, see CONTRIBUTING.md.*
---
### CODE OF CONDUCT (CODE-OF-CONDUCT.md)
# Contributor Covenant Code of Conduct
## Our Pledge
We as members, contributors, and leaders pledge to make participation in our
community a harassment-free experience for everyone, regardless of age, body
size, visible or invisible disability, ethnicity, sex characteristics, gender
identity and expression, level of experience, education, socio-economic status,
nationality, personal appearance, race, religion, or sexual identity
and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming,
diverse, inclusive, and healthy community.
## Our Standards
Examples of behavior that contributes to a positive environment for our
community include:
* Demonstrating empathy and kindness toward other people
* Being respectful of differing opinions, viewpoints, and experiences
* Giving and gracefully accepting constructive feedback
* Accepting responsibility and apologizing to those affected by our mistakes,
and learning from the experience
* Focusing on what is best not just for us as individuals, but for the
overall community
Examples of unacceptable behavior include:
* The use of sexualized language or imagery, and sexual attention or
advances of any kind
* Trolling, insulting or derogatory comments, and personal or political attacks
* Public or private harassment
* Publishing others' private information, such as a physical or email
address, without their explicit permission
* Other conduct which could reasonably be considered inappropriate in a
professional setting
## Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of
acceptable behavior and will take appropriate and fair corrective action in
response to any behavior that they deem inappropriate, threatening, offensive,
or harmful.
Community leaders have the right and responsibility to remove, edit, or reject
comments, commits, code, wiki edits, issues, and other contributions that are
not aligned to this Code of Conduct, and will communicate reasons for moderation
decisions when appropriate.
## Scope
This Code of Conduct applies within all community spaces, and also applies when
an individual is officially representing the community in public spaces.
Examples of representing our community include using an official e-mail address,
posting via an official social media account, or acting as an appointed
representative at an online or offline event.
## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be
reported to the community leaders responsible for enforcement at
[support@zenml.io](mailto:support@zenml.io).
All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the
reporter of any incident.
## Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining
the consequences for any action they deem in violation of this Code of Conduct:
### 1. Correction
**Community Impact**: Use of inappropriate language or other behavior deemed
unprofessional or unwelcome in the community.
**Consequence**: A private, written warning from community leaders, providing
clarity around the nature of the violation and an explanation of why the
behavior was inappropriate. A public apology may be requested.
### 2. Warning
**Community Impact**: A violation through a single incident or series
of actions.
**Consequence**: A warning with consequences for continued behavior. No
interaction with the people involved, including unsolicited interaction with
those enforcing the Code of Conduct, for a specified period of time. This
includes avoiding interactions in community spaces as well as external channels
like social media. Violating these terms may lead to a temporary or
permanent ban.
### 3. Temporary Ban
**Community Impact**: A serious violation of community standards, including
sustained inappropriate behavior.
**Consequence**: A temporary ban from any sort of interaction or public
communication with the community for a specified period of time. No public or
private interaction with the people involved, including unsolicited interaction
with those enforcing the Code of Conduct, is allowed during this period.
Violating these terms may lead to a permanent ban.
### 4. Permanent Ban
**Community Impact**: Demonstrating a pattern of violation of community
standards, including sustained inappropriate behavior, harassment of an
individual, or aggression toward or disparagement of classes of individuals.
**Consequence**: A permanent ban from any sort of public interaction within
the community.
## Attribution
This Code of Conduct is adapted from the [Contributor Covenant][homepage],
version 2.0, available at
[https://www.contributor-covenant.org/version/2/0/code_of_conduct.html][v2.0].
Community Impact Guidelines were inspired by
[Mozilla's code of conduct enforcement ladder][Mozilla CoC].
For answers to common questions about this code of conduct, see the FAQ at
[https://www.contributor-covenant.org/faq][FAQ]. Translations are available
at [https://www.contributor-covenant.org/translations][translations].
[homepage]: https://www.contributor-covenant.org
[v2.0]: https://www.contributor-covenant.org/version/2/0/code_of_conduct.html
[Mozilla CoC]: https://github.com/mozilla/inclusion
[FAQ]: https://www.contributor-covenant.org/faq
[translations]: https://www.contributor-covenant.org/translations
---
### CONTRIBUTING (CONTRIBUTING.md)
# 🧑💻 Contributing to ZenML
A big welcome and thank you for considering contributing to ZenML! It’s people
like you that make it a reality for users
in our community.
Reading and following these guidelines will help us make the contribution
process easy and effective for everyone
involved. It also communicates that you agree to respect the developers' time
management and develop these open-source projects. In return, we will reciprocate that respect by reading your
issue, assessing changes, and helping
you finalize your pull requests.
## ⚡️ Quicklinks
- [🧑💻 Contributing to ZenML](#-contributing-to-zenml)
- [⚡️ Quicklinks](#-quicklinks)
- [🧑⚖️ Code of Conduct](#-code-of-conduct)
- [🛫 Getting Started](#-getting-started)
- [⁉️ Issues](#-issues)
- [🏷 Pull Requests: When to make one](#-pull-requests-when-to-make-one)
- [💯 Pull Requests: Workflow to Contribute](#-pull-requests-workflow-to-contribute)
- [🧱 Pull Requests: Rebase on develop](#-pull-requests-rebase-your-branch-on-develop)
- [🧐 Linting, formatting, and tests](#-linting-formatting-and-tests)
- [🚨 Reporting a Vulnerability](#-reporting-a-vulnerability)
- [Coding Conventions](#coding-conventions)
- [👷 Creating a new Integration](#-creating-a-new-integration)
- [🆘 Getting Help](#-getting-help)
## 🧑⚖️ Code of Conduct
We take our open-source community seriously and hold ourselves and other
contributors to high standards of communication.
By participating and contributing to this project, you agree to uphold
our [Code of Conduct](https://github.com/zenml-io/zenml/blob/master/CODE-OF-CONDUCT.md)
.
## 🛫 Getting Started
Contributions are made to this repo via Issues and Pull Requests (PRs). A few
general guidelines that cover both:
- To report security vulnerabilities, please get in touch
at [support@zenml.io](mailto:support@zenml.io), monitored by
our security team.
- Search for existing Issues and PRs before creating your own.
- We work hard to make sure issues are handled on time, but it could take a
while to investigate the root cause depending on the impact.
A friendly ping in the comment thread to the submitter or a contributor can help
draw attention if your issue is blocking.
### Good First Issues for New Contributors
The best way to start is to check the
[`good-first-issue`](https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Azenml-io+label%3A%22good+first+issue%22)
label on the issue board. The core team creates these issues as necessary
smaller tasks that you can work on to get deeper into ZenML internals. These
should generally require relatively simple changes, probably affecting just one
or two files which we think are ideal for people new to ZenML.
The next step after that would be to look at the
[`good-second-issue`](https://github.com/issues?q=is%3Aopen+is%3Aissue+archived%3Afalse+user%3Azenml-io+label%3A%22good+second+issue%22)
label on the issue board. These are a bit more complex, might involve more
files, but should still be well-defined and achievable to people relatively new
to ZenML.
### ⁉️ Issues
Issues should be used to report problems with the library, request a new
feature, or to discuss potential changes before
a PR is created. When you create a new Issue, a template will be loaded that
will guide you through collecting and
providing the information we need to investigate.
If you find an Issue that addresses your problem, please add your own
reproduction information to the
existing issue rather than creating a new one. Adding
a [reaction](https://github.blog/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/)
can also help by
indicating to our maintainers that a particular issue is affecting more than
just the reporter.
### 🏷 Pull Requests: When to make one
Pull Requests (PRs) to ZenML are always welcome and can be a quick way to get your fix or
improvement slated for the next release. In
general, PRs should:
- Only fix/add the functionality in question **OR** address widespread
whitespace/style issues, not both.
- Add unit or integration tests for fixed or changed functionality (if a test
suite already exists).
- Address a single concern in the least number of changed lines as possible.
- Include documentation in the repo or in your Pull Request.
- Be accompanied by a filled-out Pull Request template (loaded automatically when
a PR is created).
For changes that address core functionality or would require breaking changes (e.g. a major release), it's best to open
an Issue to discuss your proposal first. This is not required but can save time
creating and reviewing changes.
### 💯 Pull Requests: Workflow to Contribute
