mlops-zoomcamp Documentation

Free MLOps course from DataTalks.Club. Register here 👇🏼 to get notified about the next cohort

15,463 stars Jupyter Notebook 34 files · ~26,081 tokens #machine-learning#mlops#model-deployment#model-monitoring#workflow-orchestration
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy mlops-zoomcamp on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name mlops-zoomcamp -p 5432:5432 --restart always postgres
Configured Services & Ports
⚙️ db
image: postgres
🔌 Ports: 5432:5432
⚙️ adminer
image: adminer
🔌 Ports: 8080:8080
⚙️ grafana
image: grafana/grafana-enterprise
🔌 Ports: 3000:3000
docker-compose.yml
yaml (docker-compose.yml) 
version: '3.7'

volumes: 
  grafana_data: {}

networks:
  front-tier:
  back-tier:

services:
  db:
    image: postgres
    restart: always
    environment:
      POSTGRES_PASSWORD: example
    ports:
      - "5432:5432"
    networks:
      - back-tier

  adminer:
    image: adminer
    restart: always
    ports:
      - "8080:8080"
    networks:
      - back-tier
      - front-tier  

  grafana:
    image: grafana/grafana-enterprise
    user: "472"
    ports:
      - "3000:3000"
    volumes:
      - ./config/grafana_datasources.yaml:/etc/grafana/provisioning/datasources/datasource.yaml:ro
      - ./config/grafana_dashboards.yaml:/etc/grafana/provisioning/dashboards/dashboards.yaml:ro
      - ./dashboards:/opt/grafana/dashboards
    networks:
      - back-tier
      - front-tier
    restart: always

Technical Documentation: DataTalksClub/mlops-zoomcamp

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`)

MLOps Zoomcamp: A Free 9-Week Course on Productionizing ML Services

Resource Link
Course materials GitHub repository
Video lectures YouTube playlist
Documentation Zoomcamp Logistics · MLOps Zoomcamp
Course platform (deadlines, homework) courses.datatalks.club
Slack channel #course-mlops-zoomcamp
Announcements Telegram
FAQ FAQ document

About the Course

MLOps (machine learning operations) is a must-know skill for many data professionals. This free 9-week course teaches the fundamentals of MLOps, from training and experimentation to deployment and monitoring, through structured modules, hands-on workshops, and a final project. Each module introduces core MLOps concepts and tools.

Who Should Join

This course is for data scientists, ML engineers, and software engineers who want to learn how to put machine learning models into production and operate them reliably.

Prerequisites

To get the most out of this course, you should have prior experience with:

  • Python
  • Docker
  • Command line basics
  • Machine learning (e.g., through ML Zoomcamp)
  • 1+ year of programming experience

How to Take the Course

There are two ways to follow the course: live and self-paced.

Live Cohort Self-Paced
Start Not currently scheduled Anytime
Lectures Pre-recorded Pre-recorded
Homework Graded Available but not scored
Leaderboard ✅ Yes ❌ No
Peer Review ✅ Yes ❌ No
Certificate ✅ Yes ❌ No
Cost Free Free
Register Get updates Just start learning!

[!NOTE]
We don't plan to run a live cohort in 2026. The course is fully available for self-paced study now. Register here to be notified if we run a live cohort again.

Self-paced steps:

  1. Follow the materials on GitHub
  2. Ask questions and share progress in Slack
  3. Do the homework (self-checked) and build a project for your portfolio

Syllabus

[Module 1: Introduction](01-intro)

  • What is MLOps?
  • MLOps maturity model
  • NY Taxi dataset (our running example)
  • Why MLOps is essential
  • Course structure & environment setup
  • Homework

[Module 2: Experiment Tracking & Model Management](02-experiment-tracking)

  • Introduction to experiment tracking
  • MLflow basics
  • Model saving and loading
  • Model registry
  • Hands-on MLflow exercises
  • Homework

[Module 3: Orchestration & ML Pipelines](03-orchestration)

  • Workflow orchestration
  • Homework

[Module 4: Model Deployment](04-deployment)

  • Deployment strategies: online (web, streaming) vs. offline (batch)
  • Deploying with Flask (web service)
  • Streaming deployment with AWS Kinesis & Lambda
  • Batch scoring for offline processing
  • Homework

[Module 5: Model Monitoring](05-monitoring)

  • Monitoring ML-based services
  • Web service monitoring with Prometheus, Evidently, and Grafana
  • Batch job monitoring with Prefect, MongoDB, and Evidently
  • Homework

[Module 6: Best Practices](06-best-practices)

  • Unit and integration testing
  • Linting, formatting, and pre-commit hooks
  • CI/CD with GitHub Actions
  • Infrastructure as Code (Terraform)
  • Homework

Final Project

The final project integrates everything covered in the course into an end-to-end MLOps pipeline.

Certificate

Certificates are awarded to learners who complete the final project during a live cohort. After the project is graded, the certificate appears on your course platform dashboard once the instructors issue it, and the release is announced in Slack and Telegram. See the certificate guide for how to get it and add it to LinkedIn.

Instructors

Community & Support

Getting Help on Slack

Join the #course-mlops-zoomcamp channel on DataTalks.Club Slack for discussions, troubleshooting, and networking.

To keep discussions organized:

Learning in Public

Share your progress as you go — see the learning in public guide.

Sponsors

Interested in supporting our community? Reach out to [email protected].

FAQ

A few common questions. For everything else, see the full MLOps Zoomcamp FAQ.

Q: Is this course really free?
A: Yes. All videos, materials, and homework are free and open-source.

Q: Do I need prior experience?
A: Yes, some. You should know Python, Docker basics, and the machine learning workflow (for example from ML Zoomcamp), plus about a year of programming experience.

Q: Is there a live cohort?
A: Not currently. The course is fully available for self-paced study. Register on the signup form to be notified if we run a live cohort again.

Q: What does "live cohort" mean? Are there live classes?
A: No mandatory live classes. All lectures are pre-recorded. "Live" means deadlines, scored homework, peer review, and certificate eligibility.

About DataTalks.Club

All the activity at DataTalks.Club mainly happens on Slack. We post updates there and discuss different aspects of data, career questions, and more.

At DataTalks.Club, we organize online events, community activities, and free courses. You can learn more about what we do at DataTalks.Club docs.

Overview (`01-intro/README.md`)

Module 1: Introduction

In this module, we introduce the taxi ride duration example and prepare a Linux-based development environment. We also build the baseline notebook that the rest of the course improves.

1.1 [Introduction](01-introduction.md)

MLOps connects design, training, deployment, and operation around a production ML problem.

1.2 [GitHub Codespaces](02-codespaces.md)

Set up a cloud development workspace with a repository, terminal, Docker, and Jupyter access.

1.3 [VM in AWS](03-aws-vm.md)

Create an optional Ubuntu VM and install the tools used by the course.

1.4 [Reading Parquet Data](04-read-parquet.md)

Read the current NYC taxi data format and install the Parquet engine required by pandas.

1.5 [Training a Ride Duration Prediction Model](05-train-duration-model.md)

Build and evaluate a first baseline for taxi ride duration.

1.6 [Course Overview](06-course-overview.md)

See how the modules turn a notebook into a tracked, deployed, monitored ML service.

1.7 [MLOps Maturity Model](07-maturity-model.md)

Choose an appropriate level of automation for a model and its operating context.

Homework

More information is available in the Module 1 homework.

Optional resources

Use these links when you want to explore the setup or the baseline notebook further.

Community Notes

Share community notes and resources below.

Overview (`02-experiment-tracking/README.md`)

Module 2: Experiment Tracking and Model Management

We record experiments from the taxi duration notebook and follow a model from its first run to a shared registry. We use MLflow for local tracking and model management. We also set up a remote AWS server.

2.1 [Experiment Tracking Intro](01-experiment-tracking-intro.md)

Define the tracking concepts and understand why we keep them together.

2.2 [Getting Started with MLflow](02-mlflow-getting-started.md)

Install MLflow, start the local UI, configure SQLite, and connect the notebook to an experiment.

2.3 [Experiment Tracking with MLflow](03-experiment-tracking.md)

Track hyperparameter search runs, then compare their parameters and metrics.

2.4 [Model Management](04-model-management.md)

Keep selected model artifacts and their lineage after the experiment ends.

2.5 [Model Registry](05-model-registry.md)

Register model versions, review lineage, and coordinate promotion with the deployment process.

2.6 [MLflow in Practice](06-mlflow-in-practice.md)

Choose between local tracking and a shared server with database-backed metadata and S3 artifacts.

2.7 [MLflow: Benefits, Limitations and Alternatives](07-mlflow-alternatives.md)

Evaluate the trade-offs of MLflow and compare the requirements that matter when choosing a tracking platform.

Homework

More information is available in the Module 2 homework.

Optional resources

Use these links to explore the presentation, AWS setup, and current MLflow documentation.

Use search_experiments in current MLflow versions instead of list_experiments, and follow the MLflow issue when you update older notebooks.

Community Notes

Share community notes and resources below.

Overview (`03-orchestration/README.md`)

Module 3: Orchestration and ML Pipelines

We turn the duration prediction notebook into a repeatable, parameterized training script and then connect it to the concepts of workflow orchestration.

3.1 [Introduction to ML Pipelines](01-ml-pipelines.md)

Define a training pipeline, make its dependencies explicit, and distinguish pipeline logic from orchestration.

3.2 [Turning the Notebook into a Python Script](02-notebook-to-script.md)

Refactor the notebook into functions, expose the training period as command-line parameters, and preserve MLflow metadata.

3.3 [Using an Orchestrator](03-orchestrator.md)

Schedule, parameterize, backfill, and optionally deploy the training workflow with an orchestrator.

Homework

More information is available in the Module 3 homework.

Optional resources

MLflow

To run MLflow with Docker, create an mlflow.dockerfile:

dockerfile
FROM python:3.10-slim

RUN pip install mlflow==2.12.1

EXPOSE 5000

CMD [ \
    "mlflow", "server", \
    "--backend-store-uri", "sqlite:///home/mlflow_data/mlflow.db", \
    "--host", "0.0.0.0", \
    "--port", "5000" \
]

Add it to docker-compose.yaml and persist the MLflow data directory:

yaml
mlflow:
    build:
      context: .
      dockerfile: mlflow.dockerfile
    ports:
      - "5000:5000"
    volumes:
      - "${PWD}/mlflow_data:/home/mlflow_data/"

Use the same MLflow version in the training environment. The service should then be reachable at http://localhost:5000 from the host or at http://mlflow:5000 from another Compose service.

Community Notes

Share community notes and resources below.

Previous-edition notes:

2025 notes:

Overview (`04-deployment/README.md`)

Module 4: Model Deployment

We package the duration model as a web service and connect serving to the model registry. We also compare online, streaming, and batch deployment modes.

4.1 [Three Ways of Deploying a Model](01-deployment-modes.md)

Compare batch, online, and event-driven streaming deployment and choose the boundary that fits the prediction requirement.

4.2 [Web Services: Deploying Models with Flask and Docker](02-flask-docker.md)

Build a Flask prediction endpoint, test it, and package it with its model artifacts in Docker.

4.3 [Web Services: Getting Models from the Model Registry](03-model-registry-serving.md)

Load a selected MLflow model version at serving time instead of bundling a fixed artifact into the image.

4.4 [Streaming: Deploying Models with Kinesis and Lambda](04-streaming-kinesis-lambda.md)

Optionally consume Kinesis events with a containerized Lambda function and produce predictions.

4.5 [Batch: Preparing a Scoring Script](05-batch-scoring.md)

Turn a scoring notebook into a parameterized script that can run scheduled batches and backfills.

4.6 [Batch Scoring with Mage](06-mage-batch-scoring.md)

Use a Mage transformation block to retrieve a model from MLflow and apply it to a batch.

Homework

More information is available in the Module 4 homework.

Code and resources

The main code examples are:

The streaming exercise can create AWS charges. Treat it as optional and remove test resources when finished. Use the local tests in Module 6 when you don't want to deploy cloud resources.

Community Notes

Share community notes and resources below.

Overview (`05-monitoring/README.md`)

Module 5: Model Monitoring

We define production monitoring metrics and calculate Evidently results. We store them in PostgreSQL, then expose them through Grafana dashboards and debugging reports.

5.1 [Intro to ML Monitoring](01-ml-monitoring.md)

Separate service health, model performance, data quality, and drift indicators.

5.2 [Environment Setup](02-monitoring-environment.md)

Create the Python and Docker Compose environment for metric calculation, PostgreSQL, Adminer, and Grafana.

5.3 [Prepare Reference and Model](03-reference-model.md)

Train a baseline model and save the reference dataset used for later comparisons.

5.4 [Evidently Metrics Calculation](04-evidently-metrics.md)

Compare reference and current batches, then persist selected metrics for time-series monitoring.

5.5 [Evidently Monitoring Dashboard](05-evidently-dashboard.md)

Create a workspace for reports and model/data behavior.

5.6 [Dummy Monitoring](06-dummy-monitoring.md)

Test the database and dashboard path with repeatable synthetic values before adding model metrics.

5.7 [Data Quality Monitoring](07-data-quality.md)

Add data-quality checks, schedule metric calculations, and choose thresholds with model owners.

5.8 [Save Grafana Dashboard](08-save-grafana-dashboard.md)

Provision a saved Grafana dashboard from versioned JSON and configuration files.

5.9 [Debugging with Test Suites and Reports](09-debugging-tests-reports.md)

Use Evidently reports and test suites to investigate a monitoring metric.

5.10 [Monitoring Example](10-monitoring-example.md)

Run the local end-to-end monitoring stack with Docker Compose, PostgreSQL, and Grafana.

Homework

More information is available in the Module 5 homework.

Code and resources

The current requirements set Evidently to 0.6.7. Compare this code with the Evidently 0.7 example when updating the API.

Community Notes

Share community notes and resources below.

Overview (`06-best-practices/README.md`)

Module 6: Best Practices

We make the deployed model easier to test, review, provision, and deliver. We move from Python tests through Terraform and GitHub Actions.

6.1 [Testing Python Code with pytest](01-pytest.md)

Separate model logic from cloud clients and cover the service with fast unit tests.

6.2 [Integration Tests with docker-compose](02-docker-compose-tests.md)

Start the service in Docker and verify its HTTP API with a deterministic test event.

6.3 [Testing Cloud Services with LocalStack](03-localstack.md)

Exercise Kinesis-compatible behavior locally without creating real AWS resources.

6.4 [Code Quality: Linting and Formatting](04-linting-formatting.md)

Use isort, Black, and Pylint to keep code consistent and catch common problems.

6.5 [Git Pre-commit Hooks](05-pre-commit.md)

Run project checks automatically before creating a commit.

6.6 [Makefiles and make](06-makefiles.md)

Give tests, quality checks, builds, and deployment steps stable command names.

6.7 [Terraform: Introduction](07-terraform-introduction.md)

Describe the AWS stream-based prediction infrastructure as version-controlled code.

6.8 [Terraform: Modules and Output Variables](08-terraform-modules.md)

Extract reusable infrastructure and pass resource identifiers between modules.

6.9 [Build an End-to-end Ride Prediction Workflow](09-terraform-pipeline.md)

Connect Kinesis, Lambda, S3, and ECR into a deployable prediction workflow.

6.10 [Test the Pipeline End to End](10-terraform-testing.md)

Send a ride event through the deployed pipeline and review the prediction output.

6.11 [CI/CD: Introduction](11-cicd-introduction.md)

Map pull-request checks and delivery jobs onto the ride-prediction architecture.

6.12 [Continuous Integration](12-continuous-integration.md)

Run tests and Terraform plan automatically for proposed changes.

6.13 [Continuous Delivery](13-continuous-delivery.md)

Build, publish, and deploy the service after an approved change.

Homework

More information is available in the Module 6 homework.

Code and resources

Resources:

Cloud resources can incur charges. Destroy temporary infrastructure after testing and keep credentials out of the repository.

Community Notes

Share community notes and resources below.

Overview (`07-project/README.md`)

Course Project

Watch the course project video

Objective

The goal of this project is to apply everything we have learned
in this course to build an end-to-end machine learning project.

Problem statement

For the project, we will ask you to build an end-to-end ML project.

For that, you will need:

  • Select a dataset that you're interested in (see Datasets)
  • Train a model on that dataset tracking your experiments
  • Create a model training pipeline
  • Deploy the model in batch, web service or streaming
  • Monitor the performance of your model
  • Follow the best practices

Datasets you cannot use

The NYC taxi dataset is used throughout the course modules and homework. It cannot be used for the project. Pick any other dataset.

Technologies

You don't have to limit yourself to technologies covered in the course. You can use alternatives as well:

  • Cloud: AWS, GCP, Azure, ...
  • Experiment tracking tools: MLFlow, Weights & Biases, ...
  • Workflow orchestration: Prefect, Airflow, Flyte, Kubeflow, Argo, ...
  • Monitoring: Evidently, WhyLabs/whylogs, ...
  • CI/CD: Github actions, Gitlab CI/CD, ...
  • Infrastructure as code (IaC): Terraform, Pulumi, Cloud Formation, ...

If you use a tool that wasn't covered in the course, be sure to explain what that tool does.

If you're not certain about some tools, ask in Slack.

Peer reviewing

[!IMPORTANT]
To evaluate the projects, we'll use peer reviewing. This is a great opportunity for you to learn from each other.

  • To get points for your project, you need to evaluate 3 projects of your peers
  • You get 3 extra points for each evaluation

Evaluation Criteria

  • Problem description
    • 0 points: The problem is not described
    • 1 point: The problem is described but shortly or not clearly
    • 2 points: The problem is well described and it's clear what the problem the project solves
  • Cloud
    • 0 points: Cloud is not used, things run only locally
    • 2 points: The project is developed on the cloud OR uses localstack (or similar tool) OR the project is deployed to Kubernetes or similar container management platforms
    • 4 points: The project is developed on the cloud and IaC tools are used for provisioning the infrastructure
  • Experiment tracking and model registry
    • 0 points: No experiment tracking or model registry
    • 2 points: Experiments are tracked or models are registered in the registry
    • 4 points: Both experiment tracking and model registry are used
  • Workflow orchestration
    • 0 points: No workflow orchestration
    • 2 points: Basic workflow orchestration
    • 4 points: Fully deployed workflow
  • Model deployment
    • 0 points: Model is not deployed
    • 2 points: Model is deployed but only locally
    • 4 points: The model deployment code is containerized and could be deployed to cloud or special tools for model deployment are used
  • Model monitoring
    • 0 points: No model monitoring
    • 2 points: Basic model monitoring that calculates and reports metrics
    • 4 points: Comprehensive model monitoring that sends alerts or runs a conditional workflow (e.g. retraining, generating debugging dashboard, switching to a different model) if the defined metrics threshold is violated
  • Reproducibility
    • 0 points: No instructions on how to run the code at all, the data is missing
    • 2 points: Some instructions are there, but they are not complete OR instructions are clear and complete, the code works, but the data is missing
    • 4 points: Instructions are clear, it's easy to run the code, and it works. The versions for all the dependencies are specified.
  • Best practices
    • There are unit tests (1 point)
    • There is an integration test (1 point)
    • Linter and/or code formatter are used (1 point)
    • There's a Makefile (1 point)
    • There are pre-commit hooks (1 point)
    • There's a CI/CD pipeline (2 points)

[!NOTE]
It's highly recommended to create a new repository for your project (not inside an existing repo) with a meaningful title, such as
"Car Price Prediction" or "Music Genre Classification" and include as many details as possible in the README file. ChatGPT can assist you with this. Doing so will not only make it easier to showcase your project for potential job opportunities but also have it featured on the Projects Gallery App.
If you leave the README file empty or with minimal details, there may be point deductions as per the Evaluation Criteria.

Resources

Datasets

Refer to the provided datasets for possible selection.

Explore a collection of projects completed by members of our community. The projects cover a wide range of topics and utilize different tools and techniques. Feel free to delve into any project and see how others have tackled real-world problems with data, structured their code, and presented their findings. It's a great resource to learn and get ideas for your own projects.

MLOps Zoomcamp 2023

MLOps Zoomcamp 2022

Overview (`cohorts/README.md`)

Cohorts

Each cohorts/<year>/ directory here is a frozen, self-contained copy of one
past live delivery of the course: 2022, 2023, 2024, 2025. Every one of
them has a cohort.yaml (curriculum: github_archive) recording its dates
and pointing at a notice file — added after the fact, since none of these
years originally shipped with any manifest at all. Their interior (module
directories, differing content per year) is untouched pre-migration content,
not reshaped to match any newer convention.

cohorts/2022/ has no README.md — its cohort.yaml points its
archive.notice_path at leaderboard.md instead, the closest thing it has
to a cohort-identifying document. That's a gap worth a human decision on
whether to add a real README there; it hasn't been treated as blocking.

The current cohort is `self-paced`, not a year

There is no live cohort scheduled for this course — see the root
README.md: "We don't plan to run a live cohort in
2026... fully available for self-paced study now." Root
(01-intro/ through 07-project/) holds
that actively-maintained self-paced material, and is what content: root
means in course.yaml.

cohorts/self-paced/cohort.yaml represents that: identifier: "self-paced",
delivery: self_paced, curriculum: current, start_date/end_date: null
(a self-paced delivery has no cohort-wide deadline), homework: [] (nothing
is currently graded or tracked). course.yaml:current_cohort names it the
same way a year would for a live delivery.

What's still deferred: each root module is presently a single long-form
README.md, not the module.yaml + numbered NN-kebab.md lesson files the
shared-root v2 contract expects. The checker (scripts/check-zoomcamp) knows
about this and reports it (numbered_module_required,
current_module_missing) — those are expected, not a sign something broke.
Restructuring each module's content into that shape is separate, real
editorial work, not done here.

The conventions themselves

DataTalksClub/zoomcamp-ops is the authority: STRUCTURE.md for the
repository layout and docs/shared-curriculum-v2.md for the shared-root
schema this repository's course.yaml now follows, plus the curriculum
contract documented beside it for the YAML schemas and the unit page rules.

The website's ingestion parser is the final authority and fails loudly on
push.

Overview (`04-deployment/batch/README.md`)

Batch deployment

  • Turn the notebook for training a model into a notebook for applying the model
  • Turn the notebook into a script
  • Clean it and parametrize

Overview (`04-deployment/streaming/README.md`)

Machine Learning for Streaming

  • Scenario
  • Creating the role
  • Create a Lambda function, test it
  • Create a Kinesis stream
  • Connect the function to the stream
  • Send the records

Links

Code snippets

Sending data

bash
KINESIS_STREAM_INPUT=ride_events
aws kinesis put-record \
    --stream-name ${KINESIS_STREAM_INPUT} \
    --partition-key 1 \
    --data "Hello, this is a test."

Decoding base64

python
base64.b64decode(data_encoded).decode('utf-8')

Record example

json
{
    "ride": {
        "PULocationID": 130,
        "DOLocationID": 205,
        "trip_distance": 3.66
    }, 
    "ride_id": 123
}

Sending this record

bash
aws kinesis put-record \
    --stream-name ${KINESIS_STREAM_INPUT} \
    --partition-key 1 \
    --data '{
        "ride": {
            "PULocationID": 130,
            "DOLocationID": 205,
            "trip_distance": 3.66
        }, 
        "ride_id": 156
    }'

Test event

json
{
    "Records": [
        {
            "kinesis": {
                "kinesisSchemaVersion": "1.0",
                "partitionKey": "1",
                "sequenceNumber": "49630081666084879290581185630324770398608704880802529282",
                "data": "ewogICAgICAgICJyaWRlIjogewogICAgICAgICAgICAiUFVMb2NhdGlvbklEIjogMTMwLAogICAgICAgICAgICAiRE9Mb2NhdGlvbklEIjogMjA1LAogICAgICAgICAgICAidHJpcF9kaXN0YW5jZSI6IDMuNjYKICAgICAgICB9LCAKICAgICAgICAicmlkZV9pZCI6IDI1NgogICAgfQ==",
                "approximateArrivalTimestamp": 1654161514.132
            },
            "eventSource": "aws:kinesis",
            "eventVersion": "1.0",
            "eventID": "shardId-000000000000:49630081666084879290581185630324770398608704880802529282",
            "eventName": "aws:kinesis:record",
            "invokeIdentityArn": "arn:aws:iam::XXXXXXXXX:role/lambda-kinesis-role",
            "awsRegion": "eu-west-1",
            "eventSourceARN": "arn:aws:kinesis:eu-west-1:XXXXXXXXX:stream/ride_events"
        }
    ]
}

Reading from the stream

bash
KINESIS_STREAM_OUTPUT='ride_predictions'
SHARD='shardId-000000000000'

SHARD_ITERATOR=$(aws kinesis \
    get-shard-iterator \
        --shard-id ${SHARD} \
        --shard-iterator-type TRIM_HORIZON \
        --stream-name ${KINESIS_STREAM_OUTPUT} \
        --query 'ShardIterator' \
)

RESULT=$(aws kinesis get-records --shard-iterator $SHARD_ITERATOR)

echo ${RESULT} | jq -r '.Records[0].Data' | base64 --decode

Running the test

bash
export PREDICTIONS_STREAM_NAME="ride_predictions"
export RUN_ID="e1efc53e9bd149078b0c12aeaa6365df"
export TEST_RUN="True"

python test.py

Putting everything to Docker

bash
docker build -t stream-model-duration:v1 .

docker run -it --rm \
    -p 8080:8080 \
    -e PREDICTIONS_STREAM_NAME="ride_predictions" \
    -e RUN_ID="e1efc53e9bd149078b0c12aeaa6365df" \
    -e TEST_RUN="True" \
    -e AWS_DEFAULT_REGION="eu-west-1" \
    stream-model-duration:v1

URL for testing:

Configuring AWS CLI to run in Docker

To use AWS CLI, you may need to set the env variables:

bash
docker run -it --rm \
    -p 8080:8080 \
    -e PREDICTIONS_STREAM_NAME="ride_predictions" \
    -e RUN_ID="e1efc53e9bd149078b0c12aeaa6365df" \
    -e TEST_RUN="True" \
    -e AWS_ACCESS_KEY_ID="${AWS_ACCESS_KEY_ID}" \
    -e AWS_SECRET_ACCESS_KEY="${AWS_SECRET_ACCESS_KEY}" \
    -e AWS_DEFAULT_REGION="${AWS_DEFAULT_REGION}" \
    stream-model-duration:v1

Alternatively, you can mount the .aws folder with your credentials to the .aws folder in the container:

bash
docker run -it --rm \
    -p 8080:8080 \
    -e PREDICTIONS_STREAM_NAME="ride_predictions" \
    -e RUN_ID="e1efc53e9bd149078b0c12aeaa6365df" \
    -e TEST_RUN="True" \
    -v c:/Users/alexe/.aws:/root/.aws \
    stream-model-duration:v1

Publishing Docker images

Creating an ECR repo

bash
aws ecr create-repository --repository-name duration-model

Logging in

bash
$(aws ecr get-login --no-include-email)

Pushing

bash
REMOTE_URI="387546586013.dkr.ecr.eu-west-1.amazonaws.com/duration-model"
REMOTE_TAG="v1"
REMOTE_IMAGE=${REMOTE_URI}:${REMOTE_TAG}

LOCAL_IMAGE="stream-model-duration:v1"
docker tag ${LOCAL_IMAGE} ${REMOTE_IMAGE}
docker push ${REMOTE_IMAGE}

Overview (`04-deployment/web-service-mlflow/README.md`)

Getting the model for deployment from MLflow

  • Take the code from the previous video
  • Train another model, register with MLflow
  • Put the model into a scikit-learn pipeline
  • Model deployment with tracking server
  • Model deployment without the tracking server

Starting the MLflow server with S3:

bash
mlflow server \
    --backend-store-uri=sqlite:///mlflow.db \
    --default-artifact-root=s3://mlflow-models-alexey/

Downloading the artifact

bash
export MLFLOW_TRACKING_URI="http://127.0.0.1:5000"
export MODEL_RUN_ID="6dd459b11b4e48dc862f4e1019d166f6"

mlflow artifacts download \
    --run-id ${MODEL_RUN_ID} \
    --artifact-path model \
    --dst-path .

Overview (`04-deployment/web-service/README.md`)

Deploying a model as a web-service

  • Creating a virtual environment with Pipenv
  • Creating a script for predictiong
  • Putting the script into a Flask app
  • Packaging the app to Docker
bash
docker build -t ride-duration-prediction-service:v1 .
bash
docker run -it --rm -p 9696:9696  ride-duration-prediction-service:v1

Overview (`05-monitoring/post-evidently-0.7/README.md`)

5. Model Monitoring

5.1 Intro to ML monitoring

Watch the monitoring introduction

5.2 Environment setup

Watch the monitoring environment setup

5.3 Prepare reference and model

Watch the reference and model preparation video

5.4 Evidently metrics calculation

Watch the Evidently metrics video

5.5 Evidently Monitoring Dashboard

Watch the Evidently dashboard video

5.6 Dummy monitoring

Watch the dummy monitoring video

5.7 Data quality monitoring

Watch the data quality monitoring video

Note: in this video we use Prefect (07:33-11:21). Feel free to skip this part. Also note that Prefect
is not officially supported in the 2024 edition of the course.

5.8 Save Grafana Dashboard

Watch the dashboard saving video

5.9 Debugging with test suites and reports

Watch the debugging and reports video

Homework

More information here

Notes

Did you take notes? Add them here:

Monitoring example

Prerequisites

You need following tools installed:

  • docker
  • docker-compose (included to Docker Desktop for Mac and Docker Desktop for Windows )

Preparation

Note: all actions expected to be executed in repo folder.

  • Create virtual environment and activate it (eg. python -m venv venv && source ./venv/bin/activate or conda create -n venv python=3.11 && conda activate venv)
  • Install required packages pip install -r requirements.txt
  • Run baseline_model_nyc_taxi_data.ipynb for downloading datasets, training model and creating reference dataset

Monitoring Example

Starting services

To start all required services, execute:

bash
docker-compose up

It will start following services:

  • db - PostgreSQL, for storing metrics data
  • adminer - database management tool
  • grafana - Visual dashboarding tool

Sending data

To calculate evidently metrics with prefect and send them to database, execute:

bash
python evidently_metrics_calculation.py

This script will simulate batch monitoring. Every 10 seconds it will collect data for a daily batch, calculate metrics and insert them into database. This metrics will be available in Grafana in preconfigured dashboard.

Accsess dashboard

  • In your browser go to a localhost:3000
    The default username and password are admin

  • Then navigate to General/Home menu and click on Home.

  • In the folder General you will see New Dashboard. Click on it to access preconfigured dashboard.

Ad-hoc debugging

Run debugging_nyc_taxi_data.ipynb to see how you can perform a debugging with help of Evidently TestSuites and Reports

Stopping services

To stop all services, execute:

bash
docker-compose down

Overview (`06-best-practices/code/README.md`)

Code snippets

Building and running Docker images

bash
docker build -t stream-model-duration:v2 .
bash
docker run -it --rm \
    -p 8080:8080 \
    -e PREDICTIONS_STREAM_NAME="ride_predictions" \
    -e RUN_ID="e1efc53e9bd149078b0c12aeaa6365df" \
    -e TEST_RUN="True" \
    -e AWS_DEFAULT_REGION="eu-west-1" \
    stream-model-duration:v2

Mounting the model folder:

text
docker run -it --rm \
    -p 8080:8080 \
    -e PREDICTIONS_STREAM_NAME="ride_predictions" \
    -e RUN_ID="Test123" \
    -e MODEL_LOCATION="/app/model" \
    -e TEST_RUN="True" \
    -e AWS_DEFAULT_REGION="eu-west-1" \
    -v $(pwd)/model:/app/model \
    stream-model-duration:v2

Specifying endpoint URL

bash
aws --endpoint-url=http://localhost:4566 \
    kinesis list-streams
bash
aws --endpoint-url=http://localhost:4566 \
    kinesis create-stream \
    --stream-name ride_predictions \
    --shard-count 1
bash
aws  --endpoint-url=http://localhost:4566 \
    kinesis     get-shard-iterator \
    --shard-id ${SHARD} \
    --shard-iterator-type TRIM_HORIZON \
    --stream-name ${PREDICTIONS_STREAM_NAME} \
    --query 'ShardIterator'

Unable to locate credentials

If you get 'Unable to locate credentials' error, add these
env variables to the docker-compose.yaml file:

yaml
- AWS_ACCESS_KEY_ID=abc
- AWS_SECRET_ACCESS_KEY=xyz

Make

Without make:

text
isort .
black .
pylint --recursive=y .
pytest tests/

With make:

text
make quality_checks
make test

To prepare the project, run

bash
make setup

IaC

w/ Terraform

Setup

Installation:

Configuration:

  1. If you've already created an AWS account, head to the IAM section, generate your secret-key, and download it locally.
    Instructions

  2. Configure aws-cli with your downloaded AWS secret keys:

    shell
    $ aws configure
   AWS Access Key ID [None]: xxx
   AWS Secret Access Key [None]: xxx
   Default region name [None]: eu-west-1
   Default output format [None]:

  3. Verify aws config:

    shell
    $ aws sts get-caller-identity

  4. (Optional) Configuring with aws profile: here and here

Execution

  1. To create infra (manually, in order to test on staging env)

    shell
    # Initialize state file (.tfstate)
terraform init

# Check changes to new infra plan
terraform plan -var-file=vars/stg.tfvars
    shell
    # Create new infra
terraform apply -var-file=vars/stg.tfvars

  2. To prepare aws env (copy model artifacts, set env-vars for lambda etc.):

    text
    . ./scripts/deploy_manual.sh

  3. To test the pipeline end-to-end with our new cloud infra:

    text
    . ./scripts/test_cloud_e2e.sh

  4. And then check on CloudWatch logs. Or try get-records on the output_kinesis_stream (refer to integration_test)

  5. Destroy infra after use:

    shell
    # Delete infra after your work, to avoid costs on any running services
terraform destroy

CI/CD

  1. Create a PR (feature branch): .github/workflows/ci-tests.yml
    • Env setup, Unit test, Integration test, Terraform plan
  2. Merge PR to develop: .github/workflows/cd-deploy.yml
    • Terraform plan, Terraform apply, Docker build & ECR push, Update Lambda config

Notes

  • Unfortunately, the RUN_ID (if set via the ENV or ARG in Dockerfile), disappears during lambda invocation.
    We'll set it via aws lambda update-function-configuration CLI command (refer to deploy_manual.sh or .github/workflows/cd-deploy.yml)

Overview (`cohorts/2023/README.md`)

MLOps Zoomcamp 2023 Cohort

Module 1: Introduction

Module 2: Experiment Tracking

Module 3: Orchestration and ML pipelines

Module 4: Model Deployment

Module 5: Model Monitoring

Module 6: Best Practices

Projects

Overview (`cohorts/2024/README.md`)

MLOps Zoomcamp 2024 Cohort

Module 1: Introduction

Module 2: Experiment Tracking

Module 3: Orchestration and ML pipelines

Module 4: Model Deployment

Module 5: Model Monitoring

Module 6: Best Practices

Project

Overview (`cohorts/2025/README.md`)

MLOps Zoomcamp 2025 Cohort

Module 1: Introduction

Module 2: Experiment Tracking

Module 3: Orchestration and ML pipelines

Module 4: Model Deployment

Module 5: Model Monitoring

Module 6: Best Practices

Project

Overview (`cohorts/self-paced/README.md`)

MLOps Zoomcamp — self-paced

There is no live cohort currently scheduled for this course. The curriculum
at the repository root is fully available for self-paced study: work through
the modules at your own pace, using the code and notebooks in each module
directory.

No homework or deadlines are tracked for self-paced study.

Overview (`cohorts/2022/03-orchestration/README.md`)

3. Orchestration and ML Pipelines

Note: orchestration.py is a ready final version. The rest of the files were worked on together during the video tutorials.

Note With Prefect version 2.2.1 or later DeploymentSpec's are now just Deployment's.

3.1 Negative engineering and workflow orchestration

Watch the video

3.2 Introduction to Prefect 2.0

Watch the video

3.3 First Prefect flow and basics

Watch the video

3.4 Remote Prefect Orion deployment

Watch the video

3.5 Deployment of Prefect flow

Note: There are several changes to deployment in Prefect 2.3.1 since 2.0b8:

  • DeploymentSpec in 2.0b8 now becomes Deployment.
  • work_queue_name is used instead of tags to submit the deployment to the a specific work queue.
  • You don't need to create a work queue before using the work queue. A work queue will be created if it doesn't exist.
python
from prefect.deployments import Deployment
from prefect.orion.schemas.schedules import IntervalSchedule
from datetime import timedelta

deployment = Deployment.build_from_flow(
    flow=main,
    name="model_training",
    schedule=IntervalSchedule(interval=timedelta(minutes=5)),
    work_queue_name="ml"
)

deployment.apply()

Watch the video

Links:

3.6 MLOps Zoomcamp 3.6 - (Optional) Work queues and agents

Watch the video

3.7 Homework

More information here: TBD

Notes

Did you take notes? Add them here:

Overview (`cohorts/2022/05-monitoring/README.md`)

5. Model Monitoring

5.1 Monitoring for ML-based services

Watch the monitoring introduction

Slides

5.2 Setting up the environment

Watch the monitoring environment setup

5.3 Creating a prediction service and simulating traffic

Watch the prediction service and traffic simulation video

5.4 Realtime monitoring walktrough (Prometheus, Evidently, Grafana)

Watch the realtime monitoring walkthrough

5.5 Batch monitoring walktrough (Prefect, MongoDB, Evidently)

Watch the batch monitoring walkthrough

5.6 Homework

More information here: TBD

Notes

Did you take notes? Add them here:

Monitoring example

Prerequisites

You need following tools installed:

  • docker
  • docker-compose (included to Docker Desktop for Mac and Docker Desktop for Windows )

Preparation

Note: all actions expected to be executed in repo folder.

  • Create virtual environment and activate it (eg. python -m venv venv && source ./venv/bin/activate)
  • Install required packages pip install -r requirements.txt
  • Run python prepare.py for downloading datasets

Monitoring Example

Starting services

To start all required services, execute:

bash
docker-compose up

It will start following services:

  • prometheus - TSDB for metrics
  • grafana - Visual tool for metrics
  • mongo - MongoDB, for storing raw data, predictions, targets and profile reports
  • evidently_service - Evindently RT-monitoring service (draft example)
  • prediction_service - main service, which makes predictions

Sending data

To start sending data to service, execute:

bash
python send_data.py

This script will send every second single row from dataset to prediction service along with creating file target.csv with actual results (so it can be loaded after)

Batch Monitoring Example

After you stop sending data to service, you can run batch monitoring pipeline (using Prefect) by running script:

bash
python prefect_example.py

This script will:

  • load target.csv to MongoDB
  • download dataset from MongoDB
  • Run Evidently Model Profile and Evidently Report on this data
  • Save Profile data back to MongoDB
  • Save Report to evidently_report_example.html

You can look at Prefect steps in Prefect Orion UI
(to start it execute prefect orion start)

Overview (`cohorts/2022/07-project/README.md`)

Course Project

The goal of this project is to apply everything we learned
in this course and build an end-to-end machine learning project.

Remember that to pass the project, you must evaluate 3 peers. If you don't do that, your project can't be considered compelete.

Submitting

Project Cohort #2

Project:

Peer reviewing:

Project feedback: link ("feedback-02" tab)

Project Cohort #1

Project:

Peer reviewing:

Project feedback: link ("feedback-02" tab)

Evaluation criteria

See here

Misc

To get the hash for your project, use this function to hash your email:

python
from hashlib import sha1

def compute_hash(email):
    return sha1(email.lower().encode('utf-8')).hexdigest()

Or use this website.

Overview (`cohorts/2023/07-project/README.md`)

Course Project

The goal of this project is to apply everything we learned
in this course and build an end-to-end machine learning project.

Remember that to pass the project, you must evaluate 3 peers. If you don't do that, your project can't be considered compelete.

Submitting

Project Cohort #1

Project:

Peer reviewing:

Project feedback: link ("feedback-01" tab)

Project Cohort #2

Project:

Peer reviewing:

Project feedback: link ("feedback-02" tab)

Evaluation criteria

See here

Misc

To get the hash for your project, use this function to hash your email:

python
from hashlib import sha1

def compute_hash(email):
    return sha1(email.lower().encode('utf-8')).hexdigest()

Or use this website.

Overview (`cohorts/2024/03-orchestration/README.md`)

3. Orchestration and ML Pipelines

[3.0 Introduction: ML pipelines and Mage](3.0/README.md)

[3.1 Data preparation: ETL and feature engineering](3.1/README.md)

[3.2 Training: sklearn models and XGBoost](3.2/README.md)

[3.3 Observability: Monitoring and alerting](3.3/README.md)

[3.4 Triggering: Inference and retraining](3.4/README.md)

[3.5 Deploying: Running operations in production](3.5/README.md)

[3.6 Homework](../cohorts/2024/03-orchestration/homework.md).

Quickstart

See the Unit 3.0 for a Quick Start guide

Need help?

  1. Developer documentation
  2. AI chat bot
  3. Live chat with the Mage team directly

Notes

Did you take notes? Add them here:

Notes previous editions

Overview (`cohorts/2025/competition/README.md`)

MLOps Zoomcamp 2025 Competition

Welcome to the MLOps Zoomcamp 2025 Competition!

We're hosting this competition to help you practice everything you learn in the course.

Unlike traditional Kaggle competitions that focus only on model accuracy, this competition emphasizes the complete MLOps lifecycle, from model development to deployment and monitoring.

Competition Overview

What You'll Build

Classifier: Build an ML model that detects whether a chat participant is a bot or human

Scoring

Classifiers: Evaluated on both:

  • ML metrics: accuracy, ROC, precision, recall, F-score
  • Technical metrics: latency, uptime, response rate

Timeline

  • Competition period: June 2025-August 2025
  • Final submission deadline: September 1st, 2025
  • Evaluation: Best consecutive 7-day performance
    • Deploy anytime during the competition period
    • No need to keep your service running the entire time

Benefits

  • Practical MLOps experience in a real-world scenario
  • Portfolio-worthy project showcasing end-to-end ML system
  • Course points for participation
  • $500 of AWS vouchers divided equally among top 10 participants

Implementation Guide

Start with these core components:

  • FastAPI/Flask for serving your model
  • PostgreSQL for message history (optional)
  • MLflow for experiment tracking
  • Prometheus/Grafana for monitoring

As you advance, consider adding:

  • CI/CD pipeline for automated deployment
  • Evidently for model quality monitoring
  • Airflow for automated retraining
  • Cloud deployment (AWS, Fly.io, etc.)

Getting Started

  1. Example Implementations:

  2. Deployment Options:

    • Testing: Use quickstart repo with SSH tunneling
    • Production: Deploy to any cloud provider
      • AWS (recommended, covered in course)
      • Fly.io
      • Your preferred cloud platform

Support

For technical questions, contact Misha:

Overview (`cohorts/2023/03-orchestration/prefect/README.md`)

3. Orchestration and ML Pipelines

This section of the repo contains Python code to accompany the videos that show how to use Prefect for MLOps. We will create workflows that you can orchestrate and observe.

3.1 Introdution to Workflow Orchestration

Watch the video

3.2 Introduction to Prefect

Watch the video

3.3 Prefect Workflow

Watch the video

3.4 Deploying Your Workflow

Watch the video

3.5 Working with Deployments

Watch the video

3.6 Prefect Cloud (optional)

Watch the video

3.7 Homework

More information here.

Quick setup

Install packages

In a conda environment with Python 3.10.12 or similar, install all package dependencies with

bash
pip install -r requirements.txt

Start the Prefect server locally

Create another window and activate your conda environment. Start the Prefect API server locally with

bash
prefect server start

Alternative to self-hosted server use Prefect Cloud for added capabilties

Signup and use for free at https://app.prefect.cloud

Authenticate through the terminal with

bash
prefect cloud login

Use your Prefect profile to switch between a self-hosted server and Cloud.

Notes

Did you take notes? Add them here:

Notes 2022 Edition

To read the notes from the previous edition, see here

Overview (`cohorts/2024/03-orchestration/3.0/README.md`)

3.0 Introduction: ML pipelines and Mage

3.0.1 ML Pipelines

3.0.2 Running Mage

In Codespaces

On Windows with Docker-Compose

What is MLOps

Operationalizing ML models involves moving them from development to production to drive business value.

Step 1

Preparing the model for deployment involves optimizing performance, ensuring it handles real-world data, and packaging it for integration into existing systems.

Step 2

Deploying the model involves moving it from development to production, making it accessible to users and applications.

Step 3

Once deployed, models must be continuously monitored for accuracy and reliability, and may need retraining on new data and updates to maintain effectiveness.

Step 4

The operationalized model must be integrated into existing workflows, applications, and decision-making processes to drive business impact.

Effective operationalization enables organizations to move beyond experimentation and drive tangible value from ML at scale, powering intelligent applications that personalize the customer experience and creates real business value.

Why we need to operationalize ML

1. Productivity

MLOps fosters collaboration between data scientists, ML engineers, and DevOps teams by providing a unified environment for experiment tracking, feature engineering, model management, and deployment. This breaks down silos and accelerates the entire machine learning lifecycle.

2. Reliability

MLOps ensures high-quality, reliable models in production through clean datasets, proper testing, validation, CI/CD practices, monitoring, and governance.

3. Reproducibility

MLOps enables reproducibility and compliance by versioning datasets, code, and models, providing transparency and auditability to ensure adherence to policies and regulations.

4. Time-to-value

MLOps streamlines the ML lifecycle, enabling organizations to successfully deploy more projects to production and derive tangible business value and ROI from AI/ML investments at scale.

How Mage helps MLOps

1. Data preparation

Mage offers features to build, run, and manage data pipelines for data transformation and integration, including pipeline orchestration, notebook environments, data integrations, and streaming pipelines for real-time data.

2. Training and deployment

Mage helps prepare data, train machine learning models, and deploy them with accessible API endpoints.

3. Standardize complex processes

Mage simplifies MLOps by providing a unified platform for data pipelining, model development, deployment, versioning, CI/CD, and maintenance, allowing developers to focus on model creation while improving efficiency and collaboration.

Example data pipeline: Quick Start

  1. Clone the following respository containing the complete code for this module:

    bash
    git clone https://github.com/mage-ai/mlops.git
cd mlops

  2. Launch Mage and the database service (PostgreSQL):

    bash
    ./scripts/start.sh

    If don't have bash in your enviroment, modify the following command and run it:

    bash
    PROJECT_NAME=mlops \
    MAGE_CODE_PATH=/home/src \
    SMTP_EMAIL=$SMTP_EMAIL \
    SMTP_PASSWORD=$SMTP_PASSWORD \
    docker compose up

    It is ok if you get this warning, you can ignore it
    The "PYTHONPATH" variable is not set. Defaulting to a blank string.

  3. The subproject that contains all the pipelines and code is named
    unit_3_observability

Run example pipeline

  1. Open http://localhost:6789 in your browser.

  2. In the top left corner of the screen next to the Mage logo and mlops project name,
    click the project selector dropdown and choose the unit_0_setup option.

  3. Click on the pipeline named example_pipeline.

  4. Click on the button labeled Run @once.

Resources

  1. Code for example data pipeline
  2. The definitive end-to-end machine learning (ML lifecycle) guide and tutorial for data engineers.

Overview (`cohorts/2024/03-orchestration/3.1/README.md`)

3.1 Data preparation: ETL and feature engineering

Ingestion

3.1.1 Creating a new project

In order to create a new project, firstly, you need to right-click on a folder in Text editor, and choose New Mage project. Secondly, you need to open Settings and click on Register project.

Video

Opening a text editor:

  • Go to the command center (At the top)
  • Type "text editor"

3.1.2 Data preparation - Ingestion

The project unit_1_data_preparation now has an empty pipeline, and it can be developed further using blocks. The first one we'll create is an ingestion block, which uses Python code to download the parquet files from January to March of the green taxi datasets and concatenate them. Done that, generate a series of graphs and charts useful for data profiling.

  • Note: If the time chart isn't displayed, insert the following snippet df['lpep_pickup_datetime_cleaned'] = df['lpep_pickup_datetime'].astype(np.int64) // 10**9 just above the dfs.append(df) line in ingest.py

Video

Code:

3.1.3 Utility helper functions

Utility functions are already created in the utils folder. They will be then imported into the transformer block.

Video

Code

2. Data Preparation

Videos

  1. Data preparation block
  2. Visualize prepared data

To see the correct histogram, change last two lines of the default code to:

python
col = 'trip_distance'
x = df_1[df_1[col] <= 20][col]

Code

3. Build training sets

Videos

  1. Encoding functions
  2. Training set block

Code

4. Data validations using built-in testing framework

Videos

  1. Writing data validations

Code

Code

  1. Complete code solution
  2. Pipeline configuration

Resources

  1. Global Data Products

  2. Data validations using built-in testing framework

  3. Data quality checks with Great Expectations integration

  4. Unit tests

  5. Feature encoding

Overview (`cohorts/2024/03-orchestration/3.2/README.md`)

3.2 Training: sklearn models and XGBoost

1. Training pipeline for sklearn models

Videos

  1. GDP training set
  2. Sklearn training GDP
  3. Load models
  4. Utility helper functions for loading models
  5. Hyperparameter tuning
  6. Train sklearn model

Code

2. Training pipeline for XGBoost model

Videos

  1. Hyperparameter tuning
  2. Train XGBoost model

Code

Code

  1. Complete code solution
  2. sklearn training pipeline configuration
  3. XGBoost training pipeline configuration

Resources

  1. Accuracy, precision, recall

  2. Regression model performance metrics

Overview (`cohorts/2024/03-orchestration/3.3/README.md`)

3.3 Observability: Monitoring and alerting

1. Pipeline health monitoring

  1. sklearn training pipeline health

2. Explainability

Videos

  1. Customize layout
  2. XGBoost explainability dashboard

Code

3. Model performance dashboard

Videos

  1. Overview dashboard for entire project
  2. RMSE and MSE using time series chart
  3. RMSE distribution using a histogram
  4. Training runs by model using bar chart
  5. Training runs by model using pie chart

Code

4. Alerting

Videos

  1. Setup email to send alerts
  2. Setup pipeline alerting

Code

Code

  1. sklearn training pipeline health
  2. XGBoost explainability dashboard
  3. Model performance dashboard configuration

Resources

  1. How to interpret ML models using SHAP values

  2. Customize dashboards for your project and pipelines

  3. Confusion matrix

  4. Set up alerts for pipeline run statuses:

    1. Email
    2. Opsgenie
    3. Slack
    4. Teams
    5. Discord
    6. Telegram

Overview (`cohorts/2024/03-orchestration/3.4/README.md`)

3.4 Triggering: Inference and retraining

1. Retraining pipeline

Videos

  1. Setup pipeline
  2. Trigger pipeline to run

Code

2. Inference pipeline

Videos

  1. Make a prediction
  2. Build pipeline
  3. Model inference playground part 1
  4. Model inference playground part 2
  5. Get prediction via API

Code

Code

  1. Retraining pipeline metadata.yaml
  2. Inference pipeline metadata.yaml
  3. Playground configuration settings

Resources

  1. No-code UI interactions

  2. Saving triggers in code

  3. Trigger another pipeline from a block

  4. Trigger pipeline via API endpoint

  5. Run pipelines on a recurring schedule

  6. Improving model performance through retraining

Overview (`cohorts/2024/03-orchestration/3.5/README.md`)

3.5 Deploying: Running operations in production

1. Permissions

Videos

  1. Configure permissions on AWS

Code

2. Deploy

Videos

  1. Setup and deploy using Terraform

Code

3. Continuous deployment and integration

Videos

  1. CI/CD with GitHub Actions
  2. Mage deployed

Code

Code

  1. Deployment pipeline metadata.yaml

Resources

  1. Repository setup

  2. AWS IAM policy permissions

    1. Terraform apply
    2. Terraform destroy

  3. Terraform setup

  4. Configure Terraform for AWS

  5. CI/CD overview

  6. Setup GitHub actions for CI/CD

02 Experiment Tracking Docs Mlflow On Aws (`02-experiment-tracking/docs/mlflow_on_aws.md`)

Basic AWS setup

This tutorials explains how to configure a remote tracking server on AWS. We will use an RDS database as the backend store and an s3 bucket as the artifact store.

  1. First, you need to create an AWS account. If you open a new account, AWS allows you to use some of their products for free but take into account that you may be charged for using the AWS services. More information here and here.

  2. Launch a new EC2 instance.

For this, you can select one of the instance types that are free tier eligible. For example, we will select an Amazon Linux OS (Amazon Linux 2 AMI (HVM) - Kernel 5.10, SSD Volume Type) and a t2.micro instance type, which are free tier eligible.

You'll also need to create a new key pair so later you can connect to the new instance using SSH. Click on "Create new key pair" and complete the details like in the image below:

Select the new key pair and then click on "Launch Instance".

Finally, you have to edit the security group so the EC2 instance accepts SSH (port 22) and HTTP connections (port 5000):

  1. Create an s3 bucket to be used as the artifact store.

Go to s3 and click on "Create bucket". Fill in the bucket name as in the image below and let all the other configurations with their default values.

Note: s3 bucket names must be unique across all AWS account in all the AWS Regions within a partition, that means that once a bucket is created, the name of that bucket cannot be used by another AWS account within the same region. If you get an error saying that the bucket name was already taken you can fix it easily by just changing the name to something like mlflow-artifacts-remote-2 or another name.

  1. Create a new PostgreSQL database to be used as the backend store

Go to the RDS Console and click on "Create database". Make sure to select "PostgreSQL" engine type and the "Free tier" template.

Select a name for your DB instance, set the master username as "mlflow" and tick the option "Auto generate a password" so Amazon RDS generate a password automatically.

Finally, on the section "Additional configuration" specify a database name so RDS automatically creates an initial database for you.

After clicking on "launch database" you will be able to check the newly generated password, but take into account that the automatically generated password will be shown only once!

The password value is intentionally redacted in this documentation image. In the AWS console, copy the generated password immediately and keep it private.

You can use the default values for all the other configurations.

Take note of the following information:

  • master username
  • password
  • initial database name
  • endpoint

Once the DB instance is created, go to the RDS console, select the new db and under "Connectivity & security" select the VPC security group. Modify the security group by adding a new inbound rule that allows postgreSQL connections on the port 5432 from the security group of the EC2 instance. This way, the server will be able to connect to the postgres database.

Note that the database name might be set to "postgres" by default.

  1. Connect to the EC2 instance and launch the tracking server.

Go to the EC2 Console and find the instance launched on the step 2. Click on "Connect" and then follow the steps described in the tab "SSH".

Run the following commands to install the dependencies, configure the environment and launch the server:

  • sudo yum update
  • pip3 install mlflow boto3 psycopg2-binary
  • aws configure # you'll need to input your AWS credentials here
  • mlflow server -h 0.0.0.0 -p 5000 --backend-store-uri postgresql://DB_USER:DB_PASSWORD@DB_ENDPOINT:5432/DB_NAME --default-artifact-root s3://S3_BUCKET_NAME

Note: before launching the server, check that the instance can access the s3 bucket created in the step number 3. To do that, just run this command from the EC2 instance: aws s3 ls. You should see the bucket listed in the result.

  1. Access the remote tracking server from your local machine.

Open a new tab on your web browser and go to this address: http://<EC2_PUBLIC_DNS>:5000 (you can find the instance's public DNS by checking the details of your instance in the EC2 Console).

06 Best Practices Docs Extra Material (`06-best-practices/docs/extra-material.md`)

Extra Material

Concepts of IaC and Terraform

Summary

Infrastructure-as-Code (IaC):

  • Define and automate operations around you application's infrastructure.
  • Can use version control to track changes made to infrastructure
  • Easy to replicate the configuration across different environments such as development, staging, and production.

Reference Material

We have already covered Terraform concepts at a deeper level in the Data Engineering Zoomcamp, and will not be repeating some of those basic concepts again. You can find the content here for your reference:

Notes:

Videos:

  1. For an introduction to Terraform and IaC concepts, please refer to this video
    (from the DE Zoomcamp), especially the sections in the time-codes:

    • 00:00 Introduction
    • 00:35 What is Terraform?
    • 01:10 What is IaC?
    • 01:43 Advantages of IaC
    • 14:48 Installing Terraform
    • 02:28 More on Installing Terraform

  2. For a quickstart tutorial, and understanding the main components of a basic Terraform script, please refer to this video
    (from the DE Zoomcamp). Please note that this example uses GCP as a cloud provider, while for MLOps Zoomcamp we are using AWS.

    • 00:00 Introduction
    • 00:20 .terraform-version
    • 01:04 main.tf
    • 01:23 terraform declaration
    • 03:25 provider plugins
    • 04:00 resource example - google_storage_bucket
    • 05:42 provider credentials
    • 06:34 variables.tf
    • 10:54 overview of terraform commands
    • 13:35 running terraform commands
    • 18:08 recap

In case you're using GCP instead of AWS, following is some setup material:

References

Concepts of CI/CD and GitHub Actions

Summary

  • Using GitHub Actions to create workflows to automatically test a pull request,
    build and push a Docker image, and deploy the updated lambda service to production.
  • Creating specific YAML files in GitHub repo, to automatically kick off a series of automation steps.
  • Motivation on automating your further tasks with GitHub Actions:
    • Orchestrating a continuous training pipeline (CT) to retrain your model and generate updated model artifacts in production
    • Integrating the model registry (MLflow, DVC etc.) to fetch the latest model version or experiment ID
    • and many more...

Reference Material

--- METRICS ---

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#machine-learning Browse machine-learning catalog → #mlops Browse mlops catalog → #model-deployment Browse model-deployment catalog → #model-monitoring Browse model-monitoring catalog → #workflow-orchestration Browse workflow-orchestration catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →