kuberay Documentation

A toolkit to run Ray applications on Kubernetes

2,520 stars Go Go 1 file · ~7,676 tokens #apache#deep-learning#kubernetes#machine-learning#ray
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy kuberay on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name kuberay --restart always ray-project/kuberay:latest

File: README.md

KubeRay



KubeRay is a powerful, open-source Kubernetes operator that simplifies the deployment and management of Ray applications on Kubernetes. It offers several key components:

KubeRay core: This is the official, fully-maintained component of KubeRay that provides three custom resource definitions, RayCluster, RayJob, and RayService. These resources are designed to help you run a wide range of workloads with ease.

  • RayCluster: KubeRay fully manages the lifecycle of RayCluster, including cluster creation/deletion, autoscaling, and ensuring fault tolerance.

  • RayJob: With RayJob, KubeRay automatically creates a RayCluster and submits a job when the cluster is ready. You can also configure RayJob to automatically delete the RayCluster once the job finishes.

  • RayService: RayService is made up of two parts: a RayCluster and a Ray Serve deployment graph. RayService offers zero-downtime upgrades for RayCluster and high availability.

KubeRay ecosystem: Some optional components.

  • Kubectl Plugin (Beta): Starting from KubeRay v1.3.0, you can use the kubectl ray plugin to simplify
    common workflows when deploying Ray on Kubernetes. If you aren’t familiar with Kubernetes, this
    plugin simplifies running Ray on Kubernetes. See kubectl-plugin for more details.

  • KubeRay APIServer (Alpha): It provides a layer of simplified configuration for KubeRay resources. The KubeRay API server is used internally
    by some organizations to back user interfaces for KubeRay resource management. See KubeRay APIServer V2 for more details.

  • KubeRay Dashboard (Experimental): Starting from KubeRay v1.4.0, we have introduced a new dashboard that enables users to view and manage KubeRay resources.
    While it is not yet production-ready, we welcome your feedback. See KubeRay Dashboard for more details.

Documentation

From September 2023, all user-facing KubeRay documentation will be hosted on the Ray documentation.
The KubeRay repository only contains documentation related to the development and maintenance of KubeRay.

Quick Start

Examples

KubeRay examples are hosted on the Ray documentation.
Examples span a wide range of use cases, including training, LLM online inference, batch inference, and more.

Kubernetes Ecosystem

KubeRay integrates with the Kubernetes ecosystem, including observability tools (e.g., Prometheus, Grafana, py-spy), queuing systems (e.g., Volcano, Apache YuniKorn, Kueue), ingress controllers (e.g., Nginx), and more.
See KubeRay Ecosystem for more details.

Blog Posts

Talks

Development

Please read our CONTRIBUTING guide before making a pull request. Refer to our DEVELOPMENT to build and run tests locally.

Getting Involved

Join Ray's Slack workspace, and search the following public channels:

  • #kuberay: This channel aims to help KubeRay users with their questions. The messages will be closely monitored by the Ray and KubeRay maintainers.

KubeRay contributors are welcome to join the bi-weekly KubeRay community meetings.

Security

If you discover a potential security issue in this project, or think you may
have discovered a security issue, we ask that you notify KubeRay Security via our
Slack Channel.
Please do not create a public GitHub issue.

License

This project is licensed under the Apache-2.0 License.

File: apiserver/README.md

KubeRay APIServer V1 (deprecated)

[!WARNING]
KubeRay APIServer V1 is deprecated and will be removed in the future.
Please use KubeRay APIServer V2 instead.

The KubeRay APIServer offers gRPC and HTTP APIs to manage KubeRay resources.

Introduction

The KubeRay APIServer is an optional component that provides a layer of simplified configuration for KubeRay resources. Some organizations use the KubeRay APIServer internally to support user interfaces for managing KubeRay resources.

Installation

Install with Helm

Ensure that the version of Helm is v3+. Currently, existing CI tests are based on Helm v3.4.1 and v3.9.4.

sh
helm version

Create a Kubernetes cluster

Create a local Kubernetes cluster using Kind. If you already
have a Kubernetes cluster, you can skip this step.

sh
kind create cluster --image=kindest/node:v1.29.0

Install KubeRay Operator

Refer to this document to install the latest stable KubeRay operator.

Install KubeRay APIServer

Refer to this document to install the latest
stable KubeRay operator and APIServer (without the security proxy) from the Helm
repository.

[!IMPORTANT]
If you install APIServer with security proxy, you may receive an "Unauthorized" error
when making a request. Please add an authorization header to the request: -H 'Authorization: 12345'
or install the APIServer without a security proxy.

Port-forwarding the APIServer service

Use the following command for port-forwarding to access the APIServer through port 31888:

sh
kubectl port-forward service/kuberay-apiserver-service 31888:8888

For Development: Start a Local APIServer

You can build and start the APIServer from scratch in your local environment with a single command. This will deploy all the necessary components to a local kind cluster.

sh
make start-local-apiserver

Verify the installation

The APIServer supports HTTP requests, so you can easily verify its successful startup by issuing two simple curl commands.

sh
# Create complete template.
curl --silent -X 'POST' \
    'http://localhost:31888/apis/v1/namespaces/ray-system/compute_templates' \
    -H 'accept: application/json' \
    -H 'Content-Type: application/json' \
    -d '{
        "name": "default-template",
        "namespace": "ray-system",
        "cpu": 2,
        "memory": 4
      }'

# Check whether compute template is created successfully.
curl --silent -X 'GET' \
    'http://localhost:31888/apis/v1/namespaces/ray-system/compute_templates' \
    -H 'accept: application/json'

Usage

After deployment, you can use the {{baseUrl}} to access the service. Refer to the Swagger support section for complete API definitions.

  • (default) for NodePort access, use port 31888 for connection

  • for ingress access, you will need to create your own ingress

Details of the request parameters can be found in KubeRay Swagger. This README provides only basic examples.

Setup a smoke test

The following steps allow you to validate the integration of the KubeRay APIServer components and the KubeRay Operator in your environment.

  1. (Optional) You may use your local kind cluster or minikube

    bash
    cat <<EOF | kind create cluster --name ray-test --config -
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:v1.29.0
  kubeadmConfigPatches:
    - |
      kind: InitConfiguration
      nodeRegistration:
        kubeletExtraArgs:
          node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 30265
    hostPort: 8265
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 30001
    hostPort: 10001
    listenAddress: "0.0.0.0"
    protocol: tcp
  - containerPort: 8000
    hostPort: 8000
    listenAddress: "0.0.0.0"
  - containerPort: 31888
    hostPort: 31888
    listenAddress: "0.0.0.0"
  - containerPort: 31887
    hostPort: 31887
    listenAddress: "0.0.0.0"
- role: worker
  image: kindest/node:v1.29.0
- role: worker
  image: kindest/node:v1.29.0
EOF

  2. Deploy the KubeRay APIServer within the same cluster of KubeRay operator

    sh
    helm repo add kuberay https://ray-project.github.io/kuberay-helm/
helm -n ray-system install kuberay-apiserver kuberay/kuberay-apiserver -n ray-system --create-namespace

  3. The APIServer exposes its service using NodePort by default. You can test access via your host and port; the default port is set to 31888. The examples below assume a kind (localhost) deployment. If the KubeRay APIServer is deployed on another type of cluster, you'll need to adjust the hostname to match your environment.

    sh
    curl localhost:31888
...
{
  "code": 5,
  "message": "Not Found"
}

  4. You can create RayCluster, RayJobs, or RayService by accessing the endpoints.

Swagger Support

The KubeRay APIServer supports Swagger UI. The Swagger page can be accessed at:

HTTP definition endpoints

The APIServer supports HTTP requests. For detailed specifications, check out the full spec document.

Advanced Usage

File: benchmark/perf-tests/README.md

KubeRay Performance Tests

This directory contains a collection of large scale KubeRay tests using clusterloader2.
clusterloader2 is a Kubernetes load testing tool by SIG Scalability used for Kubernetes scalability and performance testing.

Running clusterloader2 tests

First, install the perf-tests repository and compile the clusterloader2 binary

sh
git clone [email protected]:kubernetes/perf-tests.git
cd perf-tests/clusterloader2
go build -o clusterloader2 ./cmd

Run the following command to run clusterloader2 against one of the test folders. In this example we'll run the test configured in the 100-rayjob folder.

sh
clusterloader2 --provider= --kubeconfig= --testconfig=100-rayjob/config.yaml

Tests & Results

Each directory contains a test scenario and it's clusterloader2 configuraiton. Within the directories contains a results subdirectory containing junit.xml files generated by clusterloader2
for previously executed runs of the tests.

The current lists of tests are:

All published results are based on tests that ran on GKE clusters using KubeRay v1.1.1. Each test directory contains a
results/junit.xml file containing the Cluster Loader 2 steps that were successfully completed.
To learn more about the benchmark measurements, see Cluster Loader 2 Measurements.

Run a performance test with Kind

You can test clusterloader2 configs using Kind.

First create a kind cluster:

sh
kind create cluster --image=kindest/node:v1.29.0

Install KubeRay;

sh
helm install kuberay-operator kuberay/kuberay-operator --version 1.1.0

Run a clusterloader2 test:

sh
clusterloader2 --provider kind --kubeconfig ~/.kube/config --testconfig ./100-rayjob/config.yaml

Note: If you want to generate a number of RayJob custom resources other than 100, you need to make the following changes: (1) modify replicasPerNamespace in the "Creating RayJobs" step of the config.yaml file, and (2) adjust expect_succeeded in the wait-for-rayjobs.sh file.

File: clients/python-client/python_client_test/README.md

Overview

For developers

  1. pip install -U pip setuptools
  2. cd clients/python-client && pip install -e .

Uninstall with pip uninstall python-client.

For testing run

python -m unittest discover 'clients/python-client/python_client_test/'

Coverage report

Pre-requisites

  • sudo apt install libsqlite3-dev
  • pyenv install 3.6.5 # or your Python version
  • pip install db-sqlite3 coverage

To gather data
python -m coverage run -m unittest

to generate a coverage report
python -m coverage report

to generate the test coverage report in HTML format
python -m coverage html

File: clients/python-client/README.md

Overview

This python client library provide APIs to handle raycluster and rayjobs from your python application.

Prerequisites

It is assumed that your k8s cluster in already setup. Your kubectl configuration is expected to be
in ~/.kube/config if you are running the code directly from you terminal.

It is also expected that the kuberay operator is installed.
Installation instructions are here

Usage

There are multiple levels of using the API with increasing levels of complexity.

director

This is the easiest form of using the API to create rayclusters with predefined cluster sizes

python
my_kuberay_api = kuberay_cluster_api.RayClusterApi()

my_cluster_director = kuberay_cluster_builder.Director()

cluster0 = my_cluster_director.build_small_cluster(name="new-cluster0")

if cluster0:
    my_kuberay_api.create_ray_cluster(body=cluster0)

the director create the cluster definition, and the cluster_api acts as the HTTP client sending
the create (post) request to the k8s api-server

cluster_builder

The builder allows you to build the cluster piece by piece. You can customize the cluster more.

python
cluster1 = (
        my_cluster_builder.build_meta(name="new-cluster1")
        .build_head()
        .build_worker(group_name="workers", replicas=3)
        .get_cluster()
    )

if not my_cluster_builder.succeeded:
    return

my_kuberay_api.create_ray_cluster(body=cluster1)

cluster_utils

cluster_utils gives you even more options to modify your cluster definition, add/remove worker
groups, change replicas in a worker group, duplicate a worker group, etc.

python
my_Cluster_utils = kuberay_cluster_utils.ClusterUtils()

cluster_to_patch, succeeded = my_Cluster_utils.update_worker_group_replicas(
    cluster2, group_name="workers", max_replicas=4, min_replicas=1, replicas=2
)

if succeeded:
    my_kuberay_api.patch_ray_cluster(
        name=cluster_to_patch["metadata"]["name"], ray_patch=cluster_to_patch
    )

cluster_api

The cluster_api is the one you always use to implement your cluster change in k8s. You can
use it with raw JSON if you wish. The director/cluster_builder/cluster_utils are just tools to
shield the user from using raw JSON.

job_api

Finally, the job_api can be used to submit RayJobs to a pre-existing RayCluster.

Submitting to Existing Cluster

python
from python_client import kuberay_job_api, kuberay_cluster_api, constants

job_body = {
    "apiVersion": "ray.io/v1",
    "kind": "RayJob",
    "metadata": {...},
    "spec": {
        "clusterSelector": {
            "ray.io/cluster": "ray-cluster-name",
        },
        "entrypoint": 'python -c training_script.py',
        "submissionMode": "K8sJobMode",
    },
}

kuberay_job_api.submit_job(
    job=job_body,
    k8s_namespace=namespace,
)

Code Organization

text
clients/
└── python-client
    ├── examples
    │   ├── complete-example.py
    │   ├── use-builder.py
    │   ├── use-director.py
    │   ├── use-raw-config_map_with-api.py
    │   ├── use-raw-with-api.py
    │   └── use-utils.py
    ├── LICENSE
    ├── poetry.lock
    ├── pyproject.toml
    ├── python_client
    │   ├── __init__.py
    │   ├── constants.py
    │   ├── kuberay_cluster_api.py
    │   ├── kuberay_job_api.py
    │   └── utils
    │       ├── __init__.py
    │       ├── kuberay_cluster_builder.py
    │       └── kuberay_cluster_utils.py
    ├── python_client_test
    │   ├── README.md
    │   ├── test_cluster_api.py
    │   ├── test_director.py
    │   ├── test_job_api.py
    │   └── test_utils.py
    └── README.md

For developers

make sure you have installed setuptool

pip install -U pip setuptools

run the pip command

from the directory path/to/kuberay/clients/python-client

pip install -e .

to uninstall the module run

pip uninstall python-client

For testing run

python -m unittest discover 'path/to/kuberay/clients/python-client/python_client_test/'

File: clients/python-apiserver-client/README.md

Python client for KubeRay API server

This Python client is currently only supporting Ray cluster management through usage of the API server Ray API. It implements all of the current functionality of the API server (and the operator) and provide pythonic APIs to the capabilities.

The package supports Python objects (documented in the code) that can be used to build and receive payloads for creation, listing and deleting of template and Ray clusters.

The main class of the package is KubeRayAPIs that implements all of the functionality. It leverages templates and cluster definitions, allowing to specify all required parameters as straight Python classes. Additional (intermediate) definitions are provided (see cluster, environment variables, volumes, head group, worker group and job submission)

Prerequisites

It is also expected that the kuberay operator (Installation instructions are here.) and api server (Installation instructions are here.) are installed

Development

Start by installing setup-tools

shell
pip3 install -U setuptools

Now install our library locally. From the directory path/to/kuberay/clients/python_apiserver_client execute

shell
pip3 install -e .

Testing

To do testing first create the current images for operator and API server, create kind cluster and install operator and API server.
From apiserver directory, execute:

shell
make operator-image cluster load-operator-image deploy-operator docker-image load-image install

Now you can use the following test files:

  • parameters_test exercise parameter creation
  • api_test exercise overall package functionality and can also be used as a guide for API usage.

Clean up

From apiserver directory, execute:

shell
make clean-cluster

File: dashboard/README.md

Kuberay Dashboard

This is the repo for the open source dashboard for KubeRay

How to deploy with KubeRay Operator and API Server

First, clone the KubeRay repo and cd to the apiserver folder. Then, create a local cluster and install the KubeRay
components by running

bash
make cluster
make operator-image
make load-operator-image
make deploy-operator
make install

Now, you should be able to access the KubeRay apiserver at http://localhost:31888/apis/v1/namespaces/default/jobs.

Now, to deploy the dashboard, you can run yarn dev and go to localhost:3000/ray/jobs on your browser. Note that
you might need to disable CORS by launching Chrome with

bash
open -n -a /Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --args --user-data-dir="/tmp/chrome_dev_test"
--disable-web-security

If you have any ray jobs in the kind cluster, they will show up here. If not, you can create new ones by clicking on the
Create Job button, choosing a random compute template (they don't work yet), and clicking "Create Job".

If you run into an error about missing compute template, you can apply the demo/compute-template.yaml file.

What works

You can view the list of ray jobs and ray clusters. You can search and filter them using frontend components. You can
delete them using the select button. You can also create a test job, but the compute templates don't work yet.

What doesn't work

The Grafana and Logs link don't work since the KubeRay apiserver doesn't return the metrics/logs links. We should allow
users to customize their metrics/logs link templates based on their observability setup.

Create job doesn't work with compute templates yet. We don't have a detailed view of each job/cluster. We also don't
have links to the Ray head node dashboard as the user would need to expose them securely. We also need to add a
namespace selector since it's using the "default" namespace right now.

Note on open source

There are currently roblox-only components that are hidden with a flag in the codebase. We want to remove them
eventually while keeping the roblox fork easily synced with the OSS branch. Any suggestions for this are welcome.

Tech stack

Language ➜ Typescript

Should be more maintainable than JS!

Library ➜ React

We choose React as our javascript library since it’s the industry standard. It’s great for creating interactive Single
Page Applications (SPA).

Framework ➜ Next.js

We need a framework to handle the tooling and configuration needed for React. Next.js has replaced create-react-app as
the standard React framework.

Since we don’t need SEO, we can build it as a SPA instead of using SSR (server-side rendering), which Next.js is known
for. Nonetheless, Next.js is still good for building SPAs, since it provides a file-system based router, fast local
development with TurboPack, a performant Rust-based transpiler called SWC, and built-in optimizations for asset-loading
(such as fonts and images).

To disable SSR, we can use React Router and change server config like
this or use dynamic.

UI Framework ➜ MUI (Joy UI)

Between Material UI, which has an older Google-designed look that is also used by Kubeflow, and NextUI, which is more
modern, we choose Joy UI, which sits somewhere in the middle. Joy UI is a modified version of MUI with a clean and
modern look that is not too out-of-place with the Kubeflow interface.

Data Fetching ➜ SWR

While we can directly use JavaScript fetch() to fetch data, Vercel’s SWR hook is easier to use and more feature-rich.
SWR makes it easy to continuously fetch the latest data to keep the frontend up-to-date without manual refreshing, and
it also uses caching with the stale-while-revalidate strategy to provide a smooth user experience. In addition, SWR
simplifies our data-fetching code to be more maintainable.

Since our app is a dashboard, we want to revalidate the data every 5s like Grafana. Since we care more about accurate
information than UI latency, we won’t use the optimistic UI feature for POST requests, which mutates data immediately
and only rollback if the request didn’t go through. Instead, we will show a spinner for every API call like Kubeflow
does.

Next.js

This is a Next.js project bootstrapped with
create-next-app.

Getting Started

First, run the development server:

bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev

Open http://localhost:3000 with your browser to see the result.

You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.

This project uses next/font to automatically optimize and
load Inter, a custom Google Font.

Learn More

To learn more about Next.js, take a look at the following resources:

You can check out the Next.js GitHub repository - your feedback and contributions
are welcome!

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#apache Browse apache catalog → #deep-learning Browse deep-learning catalog → #kubernetes Browse kubernetes catalog → #machine-learning Browse machine-learning catalog → #ray Browse ray catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →