## 1. Project Overview & Quickstart (kubeflow/pipelines)

# Kubeflow Pipelines

[](https://kubeflow-pipelines.readthedocs.io/en/stable/?badge=latest)
[](https://pypi.org/project/kfp)
[](https://pypi.org/project/kfp)
[](https://www.bestpractices.dev/projects/9938)
[](https://deepwiki.com/kubeflow/pipelines)

## Overview of the Kubeflow pipelines service

[Kubeflow](https://www.kubeflow.org/) is a machine learning (ML) toolkit that is dedicated to making deployments of ML workflows on Kubernetes simple, portable, and scalable.

**Kubeflow pipelines** are reusable end-to-end ML workflows built using the Kubeflow Pipelines SDK.

The Kubeflow pipelines service has the following goals:

* End to end orchestration: enabling and simplifying the orchestration of end to end machine learning pipelines
* Easy experimentation: making it easy for you to try numerous ideas and techniques, and manage your various trials/experiments.
* Easy re-use: enabling you to re-use components and pipelines to quickly cobble together end to end solutions, without having to re-build each time.

## Installation

* Kubeflow Pipelines can be installed as part of the [Kubeflow Platform](https://www.kubeflow.org/docs/started/installing-kubeflow/#kubeflow-platform). Alternatively you can deploy [Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines/operator-guides/installation/) as a standalone service.

* The Docker container runtime has been deprecated on Kubernetes 1.20+. Kubeflow Pipelines has switched to use [Emissary Executor](https://www.kubeflow.org/docs/components/pipelines/legacy-v1/installation/choose-executor/#emissary-executor) by default from Kubeflow Pipelines 1.8. Emissary executor is Container runtime agnostic, meaning you are able to run Kubeflow Pipelines on Kubernetes cluster with any [Container runtimes](https://kubernetes.io/docs/setup/production-environment/container-runtimes/).

### Dependencies Compatibility Matrix

| Dependency     | Versions         |
| -------------- |------------------|
| Argo Workflows | v3.7, v4.1 |
| MySQL          | v8               |

> **KFP 3.0 compatibility notice:** Argo Workflows 3.x remains supported for KFP 2.x, but is
> deprecated and will not be supported by Kubeflow Pipelines 3.0. Before upgrading to KFP 3.0,
> operators must upgrade their Argo Workflows controller to a supported 4.x release listed in this
> matrix. Follow
> [the KFP 3.0 compatibility tracking issue](https://github.com/kubeflow/pipelines/issues/14139)
> for the removal and migration work.

## Documentation

Get started with your first pipeline and read further information in the [Kubeflow Pipelines overview](https://www.kubeflow.org/docs/components/pipelines/overview/).

See the various ways you can [use the Kubeflow Pipelines SDK](https://kubeflow-pipelines.readthedocs.io/en/stable/).

See the Kubeflow [Pipelines API doc](https://www.kubeflow.org/docs/components/pipelines/reference/api/kubeflow-pipeline-api-spec/) for API specification.

Consult the [Python SDK reference docs](https://kubeflow-pipelines.readthedocs.io/en/stable/) when writing pipelines using the Python SDK.

## Deep Wiki
Check out our AI Powered repo documentation on [DeepWiki](https://deepwiki.com/kubeflow/pipelines). 

> :warning: Please note, this is AI generated and may not have completely accurate information.

## Contributing to Kubeflow Pipelines

Before you start contributing to Kubeflow Pipelines, read the guidelines in [How to Contribute](./CONTRIBUTING.md). To learn how to build and deploy Kubeflow Pipelines from source code, read the [developer guide](./developer_guide.md).

### Optional `just` command runner

For local developer convenience, this repository includes an optional [just](https://github.com/casey/just) command runner at the repo root. It provides short aliases for existing `make` targets and does not replace any CI or release workflows.

To use it, install `just` and run, for example:

```bash
just           # list available recipes
just backend-test
just backend-images
```

Notes:

* All `just` recipes are thin wrappers around existing `make` targets (for example, `make -C backend/src/v2 test`).
* There is intentionally no generic `just build` or `just test` recipe; heavy or Docker-building flows are exposed only via explicitly named recipes such as `backend-images`.

## Kubeflow Pipelines Community

### Community Meeting

The Kubeflow Pipelines Community Meeting occurs every other Wed 10-11AM (PST).

[Calendar Invite](https://calendar.google.com/event?action=TEMPLATE&tmeid=NTdoNG5uMDBtcnJlYmdlOWt1c2lkY25jdmlfMjAxOTExMTNUMTgwMDAwWiBqZXNzaWV6aHVAZ29vZ2xlLmNvbQ&tmsrc=jessiezhu%40google.com&scp=ALL)

[Direct Meeting Link](https://zoom.us/j/92607298595?pwd%3DVlKLUbiguGkbT9oKbaoDmCxrhbRop7.1&sa=D&source=calendar&ust=1736264977415448&usg=AOvVaw1EIkjFsKy0d4yQPptIJS3x)

[Meeting notes](http://bit.ly/kfp-meeting-notes)

### Slack

We also have a slack channel (#kubeflow-pipelines) on the Cloud Native Computing Foundation Slack workspace. You can find more details at [https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels](https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels)

## Architecture

Details about the KFP Architecture can be found at [Architecture.md](docs/sdk/Architecture.md)

## Blog posts

* [From Raw Data to Model Serving: A Blueprint for the AI/ML Lifecycle with Kubeflow](https://blog.kubeflow.org/fraud-detection-e2e/) (By [Helber Belmiro](https://github.com/hbelmiro))
* [Getting started with Kubeflow Pipelines](https://cloud.google.com/blog/products/ai-machine-learning/getting-started-kubeflow-pipelines) (By Amy Unruh)
* How to create and deploy a Kubeflow Machine Learning Pipeline (By Lak Lakshmanan)
  * [Part 1: How to create and deploy a Kubeflow Machine Learning Pipeline](https://medium.com/data-science/how-to-create-and-deploy-a-kubeflow-machine-learning-pipeline-part-1-efea7a4b650f)
  * [Part 2: How to deploy Jupyter notebooks as components of a Kubeflow ML pipeline](https://medium.com/data-science/how-to-deploy-jupyter-notebooks-as-components-of-a-kubeflow-ml-pipeline-part-2-b1df77f4e5b3)
  * [Part 3: How to carry out CI/CD in Machine Learning (“MLOps”) using Kubeflow ML pipelines](https://medium.com/google-cloud/how-to-carry-out-ci-cd-in-machine-learning-mlops-using-kubeflow-ml-pipelines-part-3-bdaf68082112)

## Acknowledgments

Kubeflow pipelines uses [Argo Workflows](https://github.com/argoproj/argo-workflows) by default under the hood to orchestrate Kubernetes resources. The Argo community has been very supportive and we are very grateful.

## 2. Official Technical Reference & Guides (kubeflow/website)

# Kubeflow Website

<h4 align="center">

  [](https://app.netlify.com/sites/competent-brattain-de2d6d/deploys)
  <a href="https://landscape.cncf.io/?item=orchestration-management--scheduling-orchestration--kubeflow">
    
  </a>

  [](https://www.kubeflow.org/docs/about/community/#kubeflow-slack-channels)
  [](https://www.linkedin.com/company/kubeflow)
  [](https://www.youtube.com/kubeflow)
  [](https://x.com/kubeflow/)

</h4>

Welcome to the GitHub repository for Kubeflow's public website!

The docs website is hosted at https://www.kubeflow.org.

We use [Hugo](https://gohugo.io/) with the [google/docsy](https://github.com/google/docsy)
theme for styling and site structure, and [Netlify](https://www.netlify.com/) to manage the deployment of the site.

## Quickstart

Here's a quick guide to updating the docs:

1. Fork the [kubeflow/website repository](https://github.com/kubeflow/website) on GitHub.

2. Make your changes and send a pull request (PR).

3. If you're not yet ready for a review, add "WIP" to the PR name to indicate it's a work in progress.
   Alternatively, you use the `/hold` [prow command](https://prow.k8s.io/command-help) in a comment to mark the PR as not ready for merge.

4. Wait for the automated PR workflow to do some checks.
   When it's ready, you should see a comment like this: `deploy/netlify — Deploy preview ready!`

5. Click **Details** to the right of "Deploy preview ready" to see a preview of your updates.

6. Continue updating your doc and pushing your changes until you're happy with the content.

7. When you're ready for a review, add a comment to the PR, remove any holds or "WIP" markers, and assign a reviewer/approver.
   See the [Kubeflow contributor guide](https://www.kubeflow.org/docs/about/contributing/).

If you need more help with the GitHub workflow, follow
this [guide to a standard GitHub workflow](https://github.com/kubeflow/website/blob/master/quick-github-guide.md).

## Local development

This section will show you how to develop the website locally, by running a local Hugo server.

### Install Hugo

To install Hugo, follow the [instructions for your system type](https://gohugo.io/getting-started/installing/).

**NOTE:** we recommend that you use Hugo version `0.124.1`, as this is currently the version we deploy to Netlify.

For example, using homebrew to install hugo on macOS or linux:

```bash
# WARNING: using `brew install hugo` will install the latest version of hugo
#          which may not be compatible with the website

# TIP: to install hugo run the following commands
HOMEBREW_COMMIT="9d025105a8be086b2eeb3b1b2697974f848dbaac" # 0.124.1
curl -fL -o "hugo.rb" "https://raw.githubusercontent.com/Homebrew/homebrew-core/${HOMEBREW_COMMIT}/Formula/h/hugo.rb"
HOMEBREW_DEVELOPER="true" brew install ./hugo.rb
brew pin hugo
```

### Install Node Packages

If you plan to make changes to the site styling, you need to install some **node libraries** as well.
(See the [Docsy setup guide](https://www.docsy.dev/docs/getting-started/#install-postcss) for more information)

You can install the same versions we use in Netlify (defined in `package.json`) with the following command:

```bash
# NOTE: ensure you have node 18 installed, possibly using nvm
npm install
```

### Run local hugo server

Follow the usual GitHub workflow of forking the repository on GitHub and then cloning your fork to your local machine.

1. **Fork** the [kubeflow/website repository](https://github.com/kubeflow/website) in the GitHub UI.

2. Clone your fork locally:

   ```bash
   git clone git@github.com:<your-github-username>/website.git
   cd website/
   ```

3. Initialize the Docsy submodule:

   ```bash
   git submodule update --init --recursive
   ```

4. Install Docsy dependencies:

   ```bash
   # install the node packages (at the root of the repo)
   # NOTE: ensure you have node 18 installed, possibly using nvm
   npm install
   ```

5. Start your local Hugo server:

   ```bash
   # NOTE: You should ensure that you are in the root directory of the repository.
   hugo server -D
   ```

6. You can access your website at [http://localhost:1313/](http://localhost:1313/)

### Useful docs

- [User guide for the Docsy theme](https://www.docsy.dev/docs/getting-started/)
- [Hugo installation guide](https://gohugo.io/getting-started/installing/)
- [Hugo basic usage](https://gohugo.io/getting-started/usage/)
- [Hugo site directory structure](https://gohugo.io/getting-started/directory-structure/)
- [hugo server reference](https://gohugo.io/commands/hugo_server/)

## Menu structure

The site theme has one Hugo menu (`main`), which defines the top navigation bar. You can find and adjust the definition
of the menu in the [site configuration file](https://github.com/kubeflow/website/blob/master/config.toml).

The left-hand navigation panel is defined by the directory structure under the [`docs` directory](https://github.com/kubeflow/website/tree/master/content/en/docs).

A `weight` property in the _front matter_ of each page determines the position of the page relative to the others in the same directory.
The lower the weight, the earlier the page appears in the section.

Here is an example `_index.md` file:

```md
+++
title = "Getting Started with Kubeflow"
description = "Overview"
weight = 1
+++
```

## Docsy Theme

We use the [Docsy](https://www.docsy.dev/) theme for the website.
The theme files are managed with a [git submodule](https://git-scm.com/book/en/v2/Git-Tools-Submodules) in the `themes/docsy` directory.

**Do not change these files**, they are not actually inside this repo, but are part of the [google/docsy](https://github.com/google/docsy) repo.

To update referenced docsy commit, run the following command at the root of the repo:

```bash
# for example, to update docsy to v0.6.0
# WARNING: updating the docsy version will require you to update our overrides
#          check under: `layouts/partials` and `assets/scss`
#          also, we have [[module.mounts]] in `config.toml` that must align with the `config.yaml` from docsy
git -C themes/docsy fetch --tags
git -C themes/docsy checkout tags/v0.6.0
```

## Documentation style guide

For guidance on writing effective documentation, see
the [style guide for the Kubeflow docs](https://kubeflow.org/docs/about/style-guide/).

## Styling your content

The theme holds its styles in the [`assets/scss` directory](https://github.com/google/docsy/tree/5597d435dc74ce68240e0c3871addf24567493b0/assets/scss).
**Do not change these files**, they are not actually inside this repo, but are part of the [google/docsy](https://github.com/google/docsy) repo.

You can override the default styles and add new ones:

- In general, put your files in the project directory structure under `website` rather than in the theme directory.
  Use the same file name as the theme does, and put the file in the same relative position.
  Hugo looks first at the file in the main project directories, if present, then at the files under the theme directory.
  For example, the Kubeflow website's [`layouts/partials/navbar.html`](https://github.com/kubeflow/website/blob/master/layouts/partials/navbar.html)
  overrides the theme's [`layouts/partials/navbar.html`](https://github.com/kubeflow/website/blob/master/themes/docsy/layouts/partials/navbar.html)

- You can update the Kubeflow website's project variables in the [`_variables_project.scss` file](https://github.com/kubeflow/website/blob/master/assets/scss/_variables_project.scss).
  Values in that file override the [Docsy variables](https://github.com/google/docsy/blob/5597d435dc74ce68240e0c3871addf24567493b0/assets/scss/_variables.scss).
  You can also use `_variables_project.scss` to specify your own values for any of the default [Bootstrap 4 variables](https://getbootstrap.com/docs/4.0/getting-started/theming/).

- Custom styles [`_styles_project` file](https://github.com/kubeflow/website/blob/master/assets/scss/_styles_project.scss)

Styling of images:

- To see some examples of styled images, take a look at the [OAuth setup page](https://googlecloudplatform.github.io/kubeflow-gke-docs/docs/deploy/oauth-setup/) in the Kubeflow docs.
  Search for `.png` in the [page source](https://raw.githubusercontent.com/GoogleCloudPlatform/kubeflow-gke-docs/main/content/en/docs/deploy/oauth-setup.md).

- For more help, see the guide to
  [Bootstrap image styling](https://getbootstrap.com/docs/4.0/content/images/).

- Also see the Bootstrap utilities, such as
  [borders](https://getbootstrap.com/docs/4.0/utilities/borders/).

The site's [front page](https://www.kubeflow.org/):

- See the [page source](https://github.com/kubeflow/website/blob/master/content/en/_index.html).

- The CSS styles are in the [project variables file](https://github.com/kubeflow/website/blob/master/assets/scss/_variables_project.scss).

- The page uses the [cover block](https://www.docsy.dev/docs/content/shortcodes/#blocks-cover) defined by the theme.
  
- The page also uses the [linkdown block](https://www.docsy.dev/docs/content/shortcodes/#blocks-link-down).

## Using Hugo shortcodes

Sometimes it's useful to define a snippet of information in one place and reuse it wherever we need it.
For example, we want to be able to refer to the minimum version of various frameworks/libraries throughout the docs,
without causing a maintenance nightmare.

For this purpose, we use Hugo's "shortcodes".
Shortcodes are similar to Django variables. You define a shortcode in a file, then use a specific markup
to invoke the shortcode in the docs. That markup is replaced by the content of the shortcode file when the page is built.

To create a shortcode:

1. Add an HTML file in the `/website/layouts/shortcodes/` directory.
   The file name must be short and meaningful, as it determines the shortcode you and others use in the docs.

2. For the file content, add the text and HTML markup that should replace the shortcode markup when the web page is built.

To use a shortcode in a document, wrap the name of the shortcode in braces and percent signs like this:

```
{{% shortcode-name %}}
```

The shortcode name is the file name minus the `.html` file extension.

**Example:** The following shortcode defines the minimum required version of Kubernetes:

- File name of the shortcode:

  ```
  kubernetes-min-version.html
  ```

- Content of the shortcode:

  ```
  1.8
  ```

- Usage in a document:

  ```
  You need Kubernetes version {{% kubernetes-min-version %}} or later.
  ```

Useful Hugo docs:

- [Shortcode templates](https://gohugo.io/templates/shortcode-templates/)
- [Shortcodes](https://gohugo.io/content-management/shortcodes/)

## Versioning of the docs

For each stable release, we create a new branch for the relevant documentation.
For example, the documentation for the v0.2 stable release is maintained in the [v0.2-branch](https://github.com/kubeflow/website/tree/v0.2-branch).
Each branch has a corresponding Netlify website that automatically syncs each merged PR.

The versioned sites follow this convention:

- `www.kubeflow.org` always points to the current _master branch_
- `master.kubeflow.org` always points to GitHub head
- `vXXX-YYY.kubeflow.org` points to the release at vXXX.YYY-branch

We also hook up each version to the dropdown on the website menu bar.

Whenever any documents reference any source code, you should use the version shortcode in the links, like so:

```
https://github.com/kubeflow/kubeflow/blob/{{< params "githubbranch" >}}/scripts/gke/deploy.sh
```

This ensures that all the links in a versioned webpage point to the correct branch.