## File: README.md

Manage all your AI compute

SkyPilot is a system to run, manage, and scale AI workloads on any AI infrastructure. SkyPilot gives **AI teams** a simple interface to run jobs on any infra. **Infra teams** get a unified control plane to manage any AI compute — with advanced scheduling, scaling, and orchestration. ----- :fire: *News* :fire: - [Aug 2026] **RL is bottlenecked by inference**: scale it independently with SkyPilot: [**blog**](https://skypilot.ai/blog/slime-agentic-rl) - [Jul 2026] **Serving Kimi K3 on your own GPUs with SkyPilot**: [**blog**](https://skypilot.ai/blog/kimi-k3) - [Jul 2026] **SkyPilot v0.13.0** released: Hugging Face storage, batch inference abstractions, lifecycle hooks, governance & robustness on API server: [**Release notes**](https://github.com/skypilot-org/skypilot/releases#release-v0.13.0) - [Jun 2026] **SkyPilot Endpoints**: production-ready inference on every cluster you own: [**blog**](https://skypilot.ai/blog/skypilot-endpoints) - [Jun 2026] **Announcing SkyPilot Sandboxes**: run untrusted, LLM-generated code on the Kubernetes clusters you already own. [**Learn more**](https://skypilot.ai/blog/sandboxes), [**join early access**](https://forms.gle/o4keAryXsVazNjyGA) - [May 2026] **How Multiverse doubled their GPU utilization with SkyPilot**: [**case study**](https://multiversecomputing.com/papers/2x-gpu-utilization-same-hardware-discover-our-efficiency-playbook) - [Apr 2026] Introducing **GPU Compass**: One dashboard to browse, compare pricing, and launch across every GPU cloud. Try it at [**gpus.skypilot.co**](https://gpus.skypilot.co). - [Apr 2026] **Research-Driven Agents**: Agents read arxiv papers before coding, landed 5 llama.cpp kernel fusions and +15% faster flash attention in ~3 hours for ~$29: [**blog**](https://skypilot.ai/blog/research-driven-agents), [**HackerNews**](https://news.ycombinator.com/item?id=47706141) - [Mar 2026] **Scaling Karpathy's Autoresearch**: Autoresearch runs 1 experiment at a time. We gave it 16 GPUs and let it run in parallel: [**blog**](https://skypilot.ai/blog/scaling-autoresearch), [**HackerNews**](https://news.ycombinator.com/item?id=47442435) - [Mar 2026] **How H Company Unlocked Online RL and Unified their AI Platform**: [**case study**](https://hcompany.ai/unlocking-online-rl-skypilot) ## Overview SkyPilot **is easy to use for AI users**: - Quickly spin up compute on your own infra - Environment and job as code — simple and portable - Easy job management: queue, run, and auto-recover many jobs SkyPilot **makes Kubernetes easy for AI & Infra teams**: - Slurm-like ease of use, cloud-native robustness - Local dev experience on K8s: SSH into pods, sync code, or connect IDE - Turbocharge your clusters: gang scheduling, multi-cluster, and scaling SkyPilot **unifies multiple clusters, clouds, and hardware**: - One interface to use reserved GPUs, Kubernetes clusters, Slurm clusters, or 20+ clouds - [Flexible provisioning](https://docs.skypilot.co/en/latest/examples/auto-failover.html) of GPUs, TPUs, CPUs, with smart failover - [Team deployment](https://docs.skypilot.co/en/latest/reference/api-server/api-server.html) and resource sharing SkyPilot **maximizes GPU fleet utilization**: * Autostop: automatic cleanup of idle resources * Binpacking: workload binpacking on shared clusters * Intelligent scheduler: automatically schedule on the most available infra SkyPilot supports your existing GPU, TPU, and CPU workloads, with no code changes. Install with uv ([also supported](https://docs.skypilot.co/en/latest/getting-started/installation.html): pip, nightly, from source) ```bash # Choose your clouds: uv pip install "skypilot[kubernetes,aws,gcp,azure,oci,nebius,lambda,runpod,fluidstack,paperspace,cudo,ibm,scp,seeweb,shadeform,verda]" ``` To use SkyPilot directly with your agent (Claude Code, Codex, etc.), install the [SkyPilot Skill](https://docs.skypilot.co/en/latest/getting-started/skill.html). Tell your agent: ``` Fetch and follow https://github.com/skypilot-org/skypilot/blob/HEAD/agent/INSTALL.md to install the skypilot skill ``` Current supported infra: Kubernetes, Slurm, AWS, GCP, Azure, OCI, CoreWeave, Nebius, Lambda Cloud, RunPod, Fluidstack, Cudo, Digital Ocean, Paperspace, Cloudflare, Samsung, IBM, Vast.ai, VMware vSphere, Seeweb, Prime Intellect, Shadeform, Verda Cloud, VastData, Crusoe. ## Getting started [Install SkyPilot](https://docs.skypilot.co/en/latest/getting-started/installation.html) in 1 minute. Then, launch your first cluster in 2 minutes in [Quickstart](https://docs.skypilot.co/en/latest/getting-started/quickstart.html). SkyPilot is BYOC: Everything is launched within your cloud accounts, VPCs, and clusters. ## Benefits of SkyPilot on Kubernetes SkyPilot makes Kubernetes AI-native. It turbocharges your existing Kubernetes clusters by **accelerating AI/ML velocity**: - AI-friendly interface to launch jobs and deployments - Much simplified interactive dev for K8s (SSH / sync code / connect IDE to pods) ...and **optimizing GPU scheduling, utilization, and scaling**: - Advanced scheduling: Gang scheduling, multi-node jobs, and queueing - Multi-cluster support: Bring all your clusters under one control plane - Multi-cloud support: One consistent interface to manage many providers See [SkyPilot vs Vanilla Kubernetes](https://docs.skypilot.co/en/latest/reference/kubernetes/skypilot-and-vanilla-k8s.html) and this [blog post](https://skypilot.ai/blog/ai-on-kubernetes) for more details. ## SkyPilot in 1 minute A SkyPilot task specifies: resource requirements, data to be synced, setup commands, and the task commands. Once written in this [**unified interface**](https://docs.skypilot.co/en/latest/reference/yaml-spec.html) (YAML or Python API), the task can be launched on any available infra (Kubernetes, Slurm, cloud, etc.). This avoids vendor lock-in, and allows easily moving jobs to a different provider. Paste the following into a file `my_task.yaml`: ```yaml resources: accelerators: A100:8 # 8x NVIDIA A100 GPU num_nodes: 1 # Number of VMs to launch # Working directory (optional) containing the project codebase. # Its contents are synced to ~/sky_workdir/ on the cluster. workdir: ~/torch_examples # Commands to be run before executing the job. # Typical use: pip install -r requirements.txt, git clone, etc. setup: | cd mnist pip install -r requirements.txt # Commands to run as a job. # Typical use: launch the main program. run: | cd mnist python main.py --epochs 1 ``` Prepare the workdir by cloning: ```bash git clone https://github.com/pytorch/examples.git ~/torch_examples ``` Launch with `sky launch` (note: [access to GPU instances](https://docs.skypilot.co/en/latest/cloud-setup/quota.html) is needed for this example): ```bash sky launch my_task.yaml ``` SkyPilot then performs the heavy-lifting for you, including: 1. Find the cheapest & available infra across your clusters or clouds 2. Provision the GPUs (pods or VMs), with auto-failover if the infra returned capacity errors 3. Sync your local `workdir` to the provisioned cluster 4. Auto-install dependencies by running the task's `setup` commands 5. Run the task's `run` commands, and stream logs See [Quickstart](https://docs.skypilot.co/en/latest/getting-started/quickstart.html) to get started with SkyPilot. ## Runnable examples See [**SkyPilot examples**](https://docs.skypilot.co/en/docs-examples/examples/index.html) that cover: development, training, serving, LLM models, AI apps, and common frameworks. Latest featured examples: | Task | Examples | |----------|----------| | Training | [Verl](https://docs.skypilot.co/en/latest/examples/training/verl.html), [Finetune Llama 4](https://docs.skypilot.co/en/latest/examples/training/llama-4-finetuning.html), [TorchTitan](https://docs.skypilot.co/en/latest/examples/training/torchtitan.html), [PyTorch](https://docs.skypilot.co/en/latest/getting-started/tutorial.html), [DeepSpeed](https://docs.skypilot.co/en/latest/examples/training/deepspeed.html), [NeMo](https://docs.skypilot.co/en/latest/examples/training/nemo.html), [Ray](https://docs.skypilot.co/en/latest/examples/training/ray.html), [Unsloth](https://docs.skypilot.co/en/latest/examples/training/unsloth.html), [Jax/TPU](https://docs.skypilot.co/en/latest/examples/training/tpu.html), [OpenRLHF](https://docs.skypilot.co/en/latest/examples/training/openrlhf.html) | | Serving | [vLLM](https://docs.skypilot.co/en/latest/examples/serving/vllm.html), [SGLang](https://docs.skypilot.co/en/latest/examples/serving/sglang.html), [Ollama](https://docs.skypilot.co/en/latest/examples/serving/ollama.html) | | Models | [DeepSeek-R1](https://docs.skypilot.co/en/latest/examples/models/deepseek-r1.html), [Llama 4](https://docs.skypilot.co/en/latest/examples/models/llama-4.html), [Llama 3](https://docs.skypilot.co/en/latest/examples/models/llama-3.html), [CodeLlama](https://docs.skypilot.co/en/latest/examples/models/codellama.html), [Qwen](https://docs.skypilot.co/en/latest/examples/models/qwen.html), [Kimi-K2](https://docs.skypilot.co/en/latest/examples/models/kimi-k2.html), [Kimi-K2-Thinking](https://docs.skypilot.co/en/latest/examples/models/kimi-k2-thinking.html), [Mixtral](https://docs.skypilot.co/en/latest/examples/models/mixtral.html) | | AI apps | [RAG](https://docs.skypilot.co/en/latest/examples/applications/rag.html), [vector databases](https://docs.skypilot.co/en/latest/examples/applications/vector_database.html) (ChromaDB, CLIP) | | Common frameworks | [Airflow](https://docs.skypilot.co/en/latest/examples/frameworks/airflow.html), [Jupyter](https://docs.skypilot.co/en/latest/examples/frameworks/jupyter.html), [marimo](https://docs.skypilot.co/en/latest/examples/frameworks/marimo.html) | Source files can be found in [`llm/`](https://github.com/skypilot-org/skypilot/tree/master/llm) and [`examples/`](https://github.com/skypilot-org/skypilot/tree/master/examples). ## Learn more To learn more, see [SkyPilot Overview](https://docs.skypilot.co/en/latest/overview.html), [SkyPilot docs](https://docs.skypilot.co/en/latest/), and [SkyPilot blog](https://skypilot.ai/blog). SkyPilot adopters: [Testimonials and Case Studies](https://skypilot.ai/case-studies) Follow updates: - [Slack](http://slack.skypilot.co) - [X](https://twitter.com/skypilot_org) - [LinkedIn](https://www.linkedin.com/company/skypilot-oss/) - [YouTube](https://www.youtube.com/@skypilot-org) - [SkyPilot Blog](https://skypilot.ai/blog) ## Questions and feedback We are excited to hear your feedback: * For issues and feature requests, please [open a GitHub issue](https://github.com/skypilot-org/skypilot/issues/new). * For questions, please use [GitHub Discussions](https://github.com/skypilot-org/skypilot/discussions). For general discussions, join us on the [SkyPilot Slack](http://slack.skypilot.co). ## Contributing We welcome all contributions to the project! See [CONTRIBUTING](CONTRIBUTING.md) for how to get involved. --- ## File: addons/fuse-proxy/README.md # SkyPilot fuse-proxy A fuse-proxy that enables SkyPilot to run FUSE in containers without root privilege and `SYS_ADMIN` capability. ## Architecture The fuse-proxy consists of two components: 1. A client (`fusermount-shim`) that masks the `fusermount` binary to intercept `fusermount` calls; 2. A privileged server running as a DaemonSet on each Kubernetes node; The client and server communicate via unix domain socket, so SkyPilot Pods and the DaemonSet Pods must mount a shared directory from host. ## Usage This component can be used independently with some integration work. Here is how SkyPilot uses this: 1. Build the server image ```bash docker build . -t fusermount-server:latest ``` 2. Deploy the server as a privileged DaemonSet: [example manifest](https://raw.githubusercontent.com/skypilot-org/skypilot/master/sky/provision/kubernetes/manifests/fusermount-server-daemonset.yaml) 3. In the application Pod, mount the shared directory: ```yaml spec: containers: - name: main volumeMounts: - mountPath: /var/run/fusermount name: fusermount-shared-dir volumes: - hostPath: path: /var/run/fusermount type: DirectoryOrCreate name: fusermount-shared-dir ``` 4. Mask the `fusermount` binary in an init-container or in init script in your application container: ```bash # Install fuse2 and fuse3 apt-get install -y fuse fuse3 # Locate the original fusermount binary. FUSERMOUNT_PATH=$(which fusermount) if [ -z "$FUSERMOUNT_PATH" ]; then echo "Error: fusermount binary not found" exit 1 fi FOURSMOUNE3_PATH=$(which fusemount3) if [ -z "$FOURSMOUNE3_PATH" ]; then echo "Error: fusemount3 binary not found" exit 1 fi # The -original suffix is crucial: the server will enter the mnt namespace of application container and find the original fusermount binary # by searching `fusermount-original` executable in PATH. cp -p "$FUSERMOUNT_PATH" "${FUSERMOUNT_PATH}-original" # Mask the fusermount/fusemount3 binary by the shim provided by fusermount-server. ln -sf /var/run/fusermount/fusermount-shim "$FUSERMOUNT_PATH" ln -sf /var/run/fusermount/fusermount-shim "$FOURSMOUNE3_PATH" ``` ## How it Works ### Proxy Mode: fusermount-shim 1. When a FUSE mount is requested, SkyPilot starts a FUSE adapter process (e.g., gcsfuse) in the Sky container 2. The FUSE adapter executes `fusermount-shim` (instead of the regular `fusermount`) 3. `fusermount-shim` forwards the request to the privileged daemon pod 4. The daemon pod: - Identifies the caller's mnt namespace - Performs the mount operation using `nsenter` and real `fusermount` in the caller's mnt namespace - Returns the mounted file descriptor back to the FUSE adapter ```mermaid sequenceDiagram SkyPilot->>+Fuse Adapter: sync mount box beige Sky Pod (non-Privileged) participant Fuse Adapter participant fusermount-shim end Fuse Adapter->>+fusermount-shim: exec fusermount-shim->>+shim-server: RPC box pink Daemon Pod (Privileged) participant shim-server participant fusermount end Note over shim-server,fusermount: Set mnt namespace shim-server->>fusermount: exec fusermount->>fusermount: mount(/dev/fuse) fusermount->>shim-server: send fd shim-server->>fusermount-shim: return fd fusermount-shim->>Fuse Adapter: send fd Fuse Adapter->>Fuse Adapter: start working ``` ## Wrapper Mode: fusermount-wrapper For FUSE adapter that use `libfuse` to mount FUSE device directly (e.g. [blobfuse](https://github.com/Azure/azure-storage-fuse) for Azure Blob Storage), `fusermount-shim` will fail to work because `libfuse` only fallback to `fusermount` when mount operation fails. However, `open /dev/fuse` syscall also requires root privilege in container so that the mount process will fail before the fallback happens. `fusermount-wrapper` is used in this case to: 1. Call `fusermount-server` to mount the FUSE device before starting the FUSE adapter; 2. Pass the mounted file descriptor to the FUSE adapter, `libfuse` will [discover the mounted fd and use it directly](https://github.com/libfuse/libfuse/blob/a25fb9bd49ef56a2223262784f18dd9bbc2601dc/lib/fuse_lowlevel.c#L3435). Refer to [fusermount-wrapper](./cmd/fusermount-wrapper/main.go) for more implementation details. ## Acknowledgements This component is inspired by [k8s-fuse-csi-plugin](https://github.com/pfnet-research/meta-fuse-csi-plugin). Kudos to the author for the great work! --- ## File: docs/source/cloud-setup/cloud-permissions/aws.rst AWS ===== .. note:: By default, SkyPilot will use the credentials you have set up locally. For most cases, the :ref:`installation instructions ` are all you need to do. The steps below are **optional advanced configuration options**, aimed primarily at cloud admins and advanced users. .. _aws-sso: Using AWS SSO ------------- `AWS IAM Identity Center `_ (successor to AWS Single Sign-On, or SSO) is supported. .. note:: If you use AWS SSO and multiple clouds, check the :ref:`SSO multi-cloud compatibility notes `. To use SSO, ensure that your machine `has AWS CLI v2 installed `_. By default, ``pip install skypilot[aws]`` installs v1; v2 cannot be installed via pip. To use your newly installed AWS v2 CLI, use the absolute path to the CLI (by default, ``/usr/local/aws-cli/aws``) or create an alias ``alias awsv2=/usr/local/aws-cli/aws``. You can use the following to check version: .. code-block:: console $ aws --version aws-cli/2.25.6 ... Visit your SSO login portal (e.g. ``https://my-sso-portal.awsapps.com/start``), and click on **Access keys** under the corresponding account. Under "AWS IAM Identity Center credentials (Recommended)", copy these values: - the ``SSO start URL`` - the ``SSO Region`` Then, log into your SSO account: .. code-block:: console $ aws configure sso - **SSO session name**: should be set, but you can choose any name you want. - **SSO start URL**: copy from the SSO login portal - **SSO region**: copy from the SSO login portal - **SSO registration scopes**: leave blank to use the default Log in and approve the request in your web browser. Then back in the CLI, complete the remaining fields: - **Default client Region**: optional - **CLI default output format**: optional - **Profile name**: set to ``default`` if you want to use this profile by default with SkyPilot, otherwise see :ref:`several-aws-profiles`. If everything is set up correctly, :code:`sky check aws` should succeed! .. _aws-ssm: Using AWS Systems Manager (SSM) -------------------------------- `AWS Systems Manager Session Manager `_ provides secure shell access to EC2 instances without requiring direct network access through SSH ports or bastion hosts. SkyPilot can be configured to use SSM for SSH connections by setting ``use_ssm`` to true in your :ref:`config.yaml ` file under the ``aws`` section. One use case for SSM is to enable running SkyPilot for AWS instances that do not have public IPs. By also setting ``use_internal_ips`` to true in your :ref:`config.yaml ` file under the ``aws`` section SkyPilot will use private IPs to connect to the instances and will have access via SSM. Prerequisites ~~~~~~~~~~~~~ Before using SSM with SkyPilot, ensure the following: 1. **AWS CLI v2** and **Session Manager plugin** are installed. Follow the `installation guide `_ for your operating system. .. note:: If using a :ref:`remote API server `, install the Session Manager plugin on the API server instead of your local machine. 2. **IAM permissions** are configured to allow SSM access: - Your user/role needs ``ssm:StartSession`` permission - The ``skypilot-v1`` IAM role (used by EC2 instances) needs the ``AmazonSSMManagedInstanceCore`` managed policy attached To attach the SSM policy to the SkyPilot role: a. Open the `IAM console `_ and go to **Roles** b. Search for and select the ``skypilot-v1`` role c. Click **Add permissions** → **Attach policies** d. Search for ``AmazonSSMManagedInstanceCore`` and select it e. Click **Add permissions** Configuration ~~~~~~~~~~~~~ Add the following to your ``~/.sky/config.yaml`` file: .. code-block:: yaml aws: use_ssm: true Once configured, SkyPilot will automatically use SSM for all SSH connections to your AWS instances in the specified regions. .. _sso-feature-compat: Multi-cloud access with SSO login ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ SSO login has limited functionality *across multiple clouds*. If you use multiple clouds, you can :ref:`set up a dedicated IAM user and access key ` so that instances launched on other clouds can use AWS resources. .. tip:: If you are running SkyPilot on an EKS cluster and need S3 access without static credentials, see :ref:`aws-eks-iam-roles` for setting up IAM roles for EKS pods. .. list-table:: :header-rows: 1 * - *Supported features:* - SSO credentials - Static credentials * - Use S3 buckets on an AWS cluster - |:white_check_mark:| - |:white_check_mark:| * - Use S3 buckets on a cluster in another cloud - |:x:| - |:white_check_mark:| * - Run :ref:`managed jobs ` across multiple clouds - |:yellow_circle:| [1]_ - |:white_check_mark:| .. [1] To allow managed jobs to run on AWS instances, make sure your controller is also on AWS, by :ref:`specifying the controller resources `. .. _several-aws-profiles: Switching profiles or accounts ------------------------------ You can use different AWS profiles or accounts to launch different clusters. SkyPilot will remember the owner identity of each cluster and properly protects any "write" operations. All clusters are shown in ``sky status``. Example of mixing the default profile and another profile: .. code-block:: console $ # A cluster launched under the default AWS identity. $ sky launch --infra aws -c default $ # A cluster launched under a different profile. $ AWS_PROFILE=AdministratorAccess-12345 sky launch --infra aws -c other-profile-cluster If you are using a :ref:`remote API server `, the AWS credentials are configured on the remote server. Overriding ``AWS_PROFILE`` on the client side won't work. Using a specific VPC ----------------------- By default, SkyPilot uses the "default" VPC in each region. If a region does not have a `default VPC `_, SkyPilot will not be able to use the region. To instruct SkyPilot to use a specific VPC, you can use SkyPilot's global config file ``~/.sky/config.yaml`` to specify the VPC name in the ``aws.vpc_names`` field: .. code-block:: yaml aws: vpc_names: my-vpc-name See details in :ref:`config-yaml`. Example use cases include using a private VPC or a VPC with fine-grained constraints, typically created via Terraform or manually. To manually create a private VPC (i.e., all nodes will have internal IPs only), you can use the AWS console; see instructions `here `_. .. These two aren't currently used, but keep them so that old links like /aws.html#cloud-permissions-aws will still jump to here. .. _cloud-permissions-aws: .. _cloud-permissions-aws-user-creation: .. _dedicated-aws-user: Dedicated SkyPilot IAM user --------------------------- You can optionally create a dedicated IAM user for SkyPilot with specifically granted permissions. **Creating a dedicated user is not required** --- as long as you have AWS CLI credentials set up, SkyPilot will automatically use those credentials. However, using a dedicated IAM user can have some benefits: - Avoid using your personal credentials with SkyPilot. - Specify minimal permissions to avoid granting broad access to SkyPilot. - If you use SSO login, enable some :ref:`additional cross-cloud features `. Follow these steps to create a new AWS user: 1. Open the `IAM dashboard `_ in the AWS console and click on the **Users** tab. .. image:: ../../images/screenshots/aws/aws-add-user.png :alt: AWS Add User Then, click **Create User** and enter the user's name. Click **Next**. 2. In the **Permissions options** section, select "Attach existing policies directly". Depending on whether you want simplified or minimal permissions, follow the instructions below: .. tab-set:: .. tab-item:: Simplified permissions Search for the **AdministratorAccess** policy, and check the box to add it. Click **Next** to proceed. .. tip:: To use AWS for S3 but not for launching VMs, add **AmazonS3FullAccess** policy instead. .. tab-item:: Minimal permissions Click on **Create Policy**. .. image:: ../../images/screenshots/aws/aws-create-policy.png :alt: AWS Create Policy This will open a new window to define the minimal policy. Follow the instructions to create a new policy: :ref:`aws-minimal-policy`. Come back to this window, and in the **Permissions policies** box, click on the refresh button. You can now search for the policy you just created. .. image:: ../../images/screenshots/aws/aws-add-policy.png :alt: AWS Add Policy Check the box to add the policy, and click **Next** to proceed. 3. Click on **Next** and follow the instructions to create the user. 4. Select the newly created user from the dashboard, and go to the **Security credentials** tab. Click on **Create access key**. .. image:: ../../images/screenshots/aws/aws-create-access-key.png :alt: AWS Create access key 5. For "Use case", select **Other**. Click **Next**, then click **Create access key**. 6. Use the newly created access key to configure your credentials with the AWS CLI: .. code-block:: console :emphasize-lines: 13-14 $ # Configure your AWS credentials $ aws configure AWS Access Key ID [None]: AWS Secret Access Key [None]: Default region name [None]: Default output format [None]: $ # Check that AWS sees the shared-credentials-files $ aws configure list Name Value Type Location ---- ----- ---- -------- profile None None access_key ****************xxxx shared-credentials-file secret_key ****************xxxx shared-credentials-file region None None $ # Validate that credentials are working $ sky check aws -v Minimal permissions ----------------------- If you want to minimize the AWS permissions used by SkyPilot, you should set up the minimal permissions in two places: 1. :ref:`User Account `: the user account is the individual account of an user created by the administrator. 2. :ref:`IAM role `: the IAM role is assigned to all EC2 instances created by SkyPilot, which is used by the instances to access AWS resources, e.g., read/write S3 buckets or create other EC2 nodes. The IAM role is shared by all users under the same organization/root account. (If a user account has the permission to create IAM roles, SkyPilot can automatically create the role.) .. _aws-create-minimal-user: Create a user account with minimal permissions ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Follow the instructions above for :ref:`dedicated-aws-user`. When setting permissions for the user, use the :ref:`aws-minimal-policy` rules below. .. _iam-role-creation: Create the internal IAM role for SkyPilot ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: In most cases, the IAM role will be automatically created. You only need to manually create the IAM role if you have excluded the optional role creation permissions from your minimal skypilot policy. If you already have an IAM role called ``skypilot-v1`` in your AWS account, it is likely created by SkyPilot automatically, and you can skip this section. 1. If you haven't yet, :ref:`create a minimal IAM policy for SkyPilot `. If you previously created a dedicated IAM user with minimal permissions, you can re-use the same policy you used for the dedicated user. 2. In the `IAM dashboard `_, go to the "Roles" tab, and click on **Create role**. .. image:: ../../images/screenshots/aws/aws-add-role.png :alt: AWS Add Role 3. Select **Trusted entity type**: AWS service, and **Use case**: EC2, as seen in the image below. .. image:: ../../images/screenshots/aws/aws-add-role-entity.png :alt: AWS Role Entity, with "Trusted entity type" set to "AWS service", "Service or use case" set to "EC2", and "Use case" set to "EC2". Click **Next**. 4. Search for and select the IAM policy from step 1. 5. Click **Next**, and name your role exactly ``skypilot-v1``. Click **Create role**. .. _aws-minimal-policy: Minimal IAM policy ~~~~~~~~~~~~~~~~~~ To avoid giving SkyPilot administrator access, you can create a policy that limits the permissions of the account. When creating the policy, use the **JSON** policy editor. You can copy in the minimal policy rules and additional optional policy rules. These are the minimal policy rules required by SkyPilot: .. note:: **Replace the** ```` **with your AWS account ID**. You can find your AWS account ID by clicking on the upper right corner of the console. .. note:: There are **additional optional rules** below. It's likely that you will want to use some of these, so please take a look. .. code-block:: json :name: aws-policy-json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": "ec2:RunInstances", "Resource": "arn:aws:ec2:*::image/ami-*" }, { "Effect": "Allow", "Action": "ec2:RunInstances", "Resource": [ "arn:aws:ec2:*::instance/*", "arn:aws:ec2:*::network-interface/*", "arn:aws:ec2:*::subnet/*", "arn:aws:ec2:*::volume/*", "arn:aws:ec2:*::security-group/*" ] }, { "Effect": "Allow", "Action": [ "ec2:TerminateInstances", "ec2:DeleteTags", "ec2:StartInstances", "ec2:CreateTags", "ec2:StopInstances" ], "Resource": "arn:aws:ec2:*::instance/*" }, { "Effect": "Allow", "Action": [ "ec2:CreateTags" ], "Resource": "arn:aws:ec2:*::volume/*" }, { "Effect": "Allow", "Action": [ "ec2:Describe*" ], "Resource": "*" }, { "Effect": "Allow", "Action": [ "ec2:CreateSecurityGroup", "ec2:AuthorizeSecurityGroupIngress" ], "Resource": "arn:aws:ec2:*::*" }, { "Effect": "Allow", "Action": "iam:CreateServiceLinkedRole", "Resource": "*", "Condition": { "StringEquals": { "iam:AWSServiceName": "spot.amazonaws.com" } } }, { "Effect": "Allow", "Action": [ "iam:GetRole", "iam:PassRole" ], "Resource": [ "arn:aws:iam:::role/skypilot-v1" ] }, { "Effect": "Allow", "Action": [ "iam:GetInstanceProfile" ], "Resource": "arn:aws:iam:::instance-profile/skypilot-v1" } ] } **Optional**: If you would like SkyPilot to automatically set up an IAM role and instance profile for EC2 instances, modify the last two rules in the policy with the highlighted four lines: .. warning:: If you don't do this, you must manually set up the IAM role that SkyPilot will use: see :ref:`iam-role-creation`. .. code-block:: json :emphasize-lines: 6-7,17-18 { "Effect": "Allow", "Action": [ "iam:GetRole", "iam:PassRole", "iam:CreateRole", "iam:AttachRolePolicy" ], "Resource": [ "arn:aws:iam:::role/skypilot-v1" ] }, { "Effect": "Allow", "Action": [ "iam:GetInstanceProfile", "iam:CreateInstanceProfile", "iam:AddRoleToInstanceProfile" ], "Resource": "arn:aws:iam:::instance-profile/skypilot-v1" } **Optional**: To enable ``sky launch --clone-disk-from``, you need to add the following permissions to the policy above as well. .. code-block:: json { "Effect": "Allow", "Action": [ "ec2:CreateImage", "ec2:CopyImage", "ec2:DeregisterImage" ], "Resource": "*" } **Optional**: To enable opening ports on AWS cluster, you need to add the following permissions to the policy above as well. .. code-block:: json { "Effect": "Allow", "Action": [ "ec2:DeleteSecurityGroup", "ec2:ModifyInstanceAttribute" ], "Resource": "arn:aws:ec2:*::*" } **Optional**: If you would like to have your users access S3 buckets, you need to add the following permissions to the policy above as well. .. code-block:: json { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Resource": "arn:aws:s3:::*/*" }, { "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetBucketLocation" ], "Resource": "arn:aws:s3:::*" }, { "Effect": "Allow", "Action": "s3:ListAllMyBuckets", "Resource": "*" } **Optional**: If you also want to allow SkyPilot to create and delete S3 buckets (for ``sky storage`` commands), add these additional permissions: .. code-block:: json { "Effect": "Allow", "Action": [ "s3:CreateBucket", "s3:DeleteBucket", "s3:PutBucketTagging" ], "Resource": "arn:aws:s3:::*" } .. tip:: If you are using EKS and want to set up S3 access with IAM roles, see :ref:`aws-eks-iam-roles`. **Once you have added all needed policies, click Next** and follow the instructions to finish creating the policy. You can give the policy a descriptive name, such as ``minimal-skypilot-policy``. .. _aws-troubleshooting: Troubleshooting --------------- If your credentials are not being picked up, or you're seeing the wrong credentials in SkyPilot, here are some steps you can take to troubleshoot: 1. **Check** ``aws configure list``. This command should show the currently configured credentials. If you have static credentials set up correctly, you should see something like this: .. code-block:: console $ aws configure list Name Value Type Location ---- ----- ---- -------- profile None None access_key ****************xxxx shared-credentials-file secret_key ****************xxxx shared-credentials-file region None None If you have SSO credentials set up correctly, you should see something like this: .. code-block:: console $ aws configure list Name Value Type Location ---- ----- ---- -------- profile None None access_key ****************xxxx sso secret_key ****************xxxx sso region None None 2. **Check** ``sky check aws``. This should show whether SkyPilot is picking up the credentials and has the necessary permissions. .. code-block:: console $ sky check aws -v Checking credentials to enable clouds for SkyPilot. AWS: enabled [compute, storage] Activated account: VRSC9IFFYQI7THCKR5UVC [account=190763068689] ... Common issues ~~~~~~~~~~~~~ - **Wrong profile is enabled.** SkyPilot will respect the ``AWS_PROFILE`` environment variable if it is set; see :ref:`several-aws-profiles`. If ``AWS_PROFILE`` is not set, SkyPilot will use the profile named ``default``. You may have previously set ``AWS_PROFILE`` in your ``.bashrc`` file or similar. Try to double-check the value: .. code-block:: console :emphasize-lines: 13 $ # Check the account being used by skypilot $ sky check aws -v Checking credentials to enable clouds for SkyPilot. AWS: enabled [compute, storage] Activated account: XXXXXXXXXXXXXXXXXXXXX:user [account=123456789012] ... $ # AWS account 1234-5678-9012 is enabled via @user SSO login. $ # See the currently enabled profile. $ aws configure list Name Value Type Location ---- ----- ---- -------- profile AWSPowerUserAccess-123456789012 env ['AWS_DEFAULT_PROFILE', 'AWS_PROFILE'] access_key ****************xxxx sso secret_key ****************xxxx sso region None None $ # SSO profile AWSPowerUserAccess-123456789012 is enabled $ # via environment variable. $ # See details of the currently enabled AWS account and user/role. $ aws sts get-caller-identity $ # See if the environment variable has been set. $ echo $AWS_PROFILE AWSPowerUserAccess-123456789012 $ unset AWS_PROFILE $ # Delete from .bashrc/.zshrc to make the change permanent. $ # Now, default profile will be used. $ aws configure list Name Value Type Location ---- ----- ---- -------- profile None None ... $ sky check aws -v Checking credentials to enable clouds for SkyPilot. AWS: enabled [compute, storage] Activated account: XXXXXXXXXXXXXXXXXXXXX [account=987654321098] ... $ # Now AWS account 9876-5432-1098 is enabled via default profile. - **Profile is not set**. If ``sky check aws`` and ``aws configure list`` cannot find credentials, you may not have a default profile set. 1. If the environment variable ``AWS_PROFILE`` is set, this profile name will be used. 2. If there is a profile named ``default``, it will be used. 3. Otherwise, the profile will not be accessible. Even if there is only one profile, it will not be used unless ``AWS_PROFILE`` is set or the profile is named ``default``. In AWS CLI v1, you can check ``~/.aws/credentials`` and ``~/.aws/config`` to look for profile names. In AWS CLI v2, you can check from the CLI. .. code-block:: console $ # AWS CLI v2 only $ aws --version aws-cli/2.25.6 ... $ # List all profiles $ aws configure list-profiles AWSPowerUserAccess-xxxxxxx default If there is no ``default`` profile, you can edit the configuration directly: .. code-block:: cfg :emphasize-lines: 2 # ~/.aws/config [profile default] sso_session = my-skypilot-session sso_account_id = XXXXXXXXXX ... .. code-block:: cfg :emphasize-lines: 2 # ~/.aws/config [default] aws_access_key_id = XXXXXXXXXXXXXXXXXXXX aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Or, you can set the ``AWS_PROFILE`` environment variable in your shell config: .. code-block:: shell # .bashrc / .zshrc # Enable AWS profile named "AWSPowerUserAccess-123456789012" export AWS_PROFILE='AWSPowerUserAccess-123456789012' .. toctree:: :hidden: aws-eks-iam-roles --- ## File: docs/source/cloud-setup/cloud-permissions/gcp.rst GCP ============= .. note:: By default, SkyPilot will pick up the credentials you have set up. For most cases, the :ref:`installation instructions ` are all you need to do. The steps below are **optional advanced configuration options**, aimed primarily at cloud admins and advanced users. .. _gcp-service-account: Service account --------------- `GCP service accounts `__ are supported. .. tip:: A service account on your local machine can avoid the periodic ``google.auth.exceptions.RefreshError: Reauthentication is needed. Please run `gcloud auth application-default login` to reauthenticate.`` error. A service account is long-lived as it does not have an expiry time. Set up a service account as follows: 1. Follow the :ref:`instructions ` to create a service account with the appropriate roles/permissions. 2. In the "Service Accounts" tab in the `IAM & Admin console `__, click on the service account to go to its detailed page. Click on the **KEYS** tab, then click on **ADD KEY** to add a JSON key. The key will be downloaded automatically. 3. Set the environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` to the path of the key file, and configure the gcloud CLI tool: .. code-block:: console $ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json $ gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS $ gcloud config set project your-project-id You may want to add the export statement in your profile (e.g. ``~/.bashrc``, ``~/.zshrc``) so that it is set automatically in all new terminal sessions. .. _cloud-permissions-gcp: Setting permissions ------------------- Generally, the administrator can choose among three "levels" of permissions, from the most permissive and least setup effort, to the least permissive and more setup effort: * Default: no setup, give users Owner-level permissions (i.e., you do not need to follow the instructions in this section) * :ref:`Medium `: easy setup, with a medium set of permissions * :ref:`Minimal `: more setup, with the minimal set of permissions .. _gcp-medium-permissions: Medium permissions ~~~~~~~~~~~~~~~~~~ The easiest way to grant permissions to a user access your GCP project without the ``Owner`` role is to add the following roles to the user principals: .. code-block:: yaml roles/browser roles/compute.admin roles/iam.serviceAccountAdmin roles/iam.serviceAccountUser roles/serviceusage.serviceUsageAdmin roles/storage.admin roles/iam.securityAdmin .. note:: If the ``roles/iam.securityAdmin`` role is undesirable, you can do the following. First, include the role and have any user (e.g., the admin) run ``sky launch --infra gcp`` successfully once. This is to create the necessary service account. Then, replace the role ``roles/iam.securityAdmin`` with ``roles/iam.roleViewer`` in the list above. Optionally, to use TPUs, add the following role: .. code-block:: yaml roles/tpu.admin You can grant those accesses via GCP's `IAM & Admin console `__. .. _gcp-minimal-permissions: Minimal permissions ~~~~~~~~~~~~~~~~~~~ The :ref:`Medium Permissions ` assigns admin permissions for some GCP services to the user. If you would like to grant finer-grained and more minimal permissions to your users in your organization / project, you can create a custom role by following the steps below: 1. Go to GCP's `IAM & Admin console `__ and click on **Create Role**. .. image:: ../../images/screenshots/gcp/create-role.png :width: 80% :align: center :alt: GCP Create Role 2. Give the role a descriptive name, such as ``minimal-skypilot-role``. 3. Click **Add Permissions** and search for the following permissions and add them to the role: .. code-block:: text compute.disks.create compute.disks.list compute.firewalls.create compute.firewalls.delete compute.firewalls.get compute.instances.create compute.instances.delete compute.instances.get compute.instances.list compute.instances.setLabels compute.instances.setMetadata compute.instances.setServiceAccount compute.instances.start compute.instances.stop compute.networks.get compute.networks.list compute.networks.getEffectiveFirewalls compute.globalOperations.get compute.subnetworks.use compute.subnetworks.list compute.subnetworks.useExternalIp compute.projects.get compute.zoneOperations.get iam.roles.get iam.serviceAccounts.actAs iam.serviceAccounts.get serviceusage.services.enable serviceusage.services.list serviceusage.services.use resourcemanager.projects.get resourcemanager.projects.getIamPolicy .. note:: For custom VPC users (with :code:`gcp.vpc_name` specified in :code:`~/.sky/config.yaml`, check `here <#_gcp-bring-your-vpc>`_), :code:`compute.firewalls.create` and :code:`compute.firewalls.delete` are not necessary unless opening ports is needed via `resources.ports` in task yaml. .. note:: (Advanced) To further limit the ``iam.serviceAccounts.actAs`` permission to access SkyPilot's service account only, you can remove the permission from the list above and additionally grant your organization's users the ability to use the service account ``skypilot-v1`` created by the admin (see :ref:`Service Account `). This can be done by going to ``IAM & Admin console -> Service Accounts -> skypilot-v1 -> Permissions -> GRANT ACCESS`` and adding the users with role ``roles/iam.serviceAccountUser``. This permits the users to use the ``skypilot-v1`` service account required by SkyPilot. 4. **Optional**: If the user needs to access GCS buckets, you can additionally add the following permissions: .. code-block:: text storage.buckets.create storage.buckets.get storage.buckets.delete storage.objects.create storage.objects.update storage.objects.delete storage.objects.get storage.objects.list 5. **Optional**: If the user needs to access TPU VMs, you can additionally add the following permissions (the following may not be exhaustive, please file an issue if you find any missing permissions): .. code-block:: text tpu.nodes.create tpu.nodes.delete tpu.nodes.list tpu.nodes.get tpu.nodes.update tpu.operations.get 6. **Optional**: To enable ``sky launch --clone-disk-from``, you need to have the following permissions for the role as well: .. code-block:: text compute.disks.useReadOnly compute.images.create compute.images.get compute.images.delete 7. **Optional**: To enable opening ports on GCP cluster, you need to have the following permissions for the role as well: .. code-block:: text compute.instances.setTags compute.firewalls.list compute.firewalls.update 8. **Optional**: If the user needs to use custom machine images with ``sky launch --image-id``, you can additionally add the following permissions: .. code-block:: text compute.disks.get compute.disks.resize compute.images.get compute.images.useReadOnly 9. **Optional**: If your organization sets ``gcp.prioritize_reservations`` or ``gcp.specific_reservations`` in :ref:`~/.sky/config.yaml `, you can additionally add the following permissions: .. code-block:: text compute.reservations.list 9. Click **Create** to create the role. 10. Go back to the "IAM" tab and click on **GRANT ACCESS**. 11. Fill in the email address of the user in the “Add principals” section, and select ``minimal-skypilot-role`` in the “Assign roles” section. Click **Save**. .. image:: ../../images/screenshots/gcp/create-iam.png :width: 80% :align: center :alt: GCP Grant Access 12. The user should receive an invitation to the project and should be able to setup SkyPilot by following the instructions in :ref:`Installation `. .. note:: The user created with the above minimal permissions will not be able to create service accounts to be assigned to SkyPilot instances. The admin needs to follow the :ref:`instruction below ` to create a service account to be shared by all users in the project. .. _gcp-service-account-creation: Service account for SkyPilot system ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: If you already have an service account under "Service Accounts" tab with the email starting with ``skypilot-v1@``, it is likely created by SkyPilot automatically, and you can skip this section. 1. Click the "Service Accounts" tab in the `IAM & Admin console `__, and click on **CREATE SERVICE ACCOUNT**. .. image:: ../../images/screenshots/gcp/create-service-account.png :width: 80% :align: center :alt: GCP Create Service Account 2. Set the service account id to ``skypilot-v1`` and click **CREATE AND CONTINUE**. .. image:: ../../images/screenshots/gcp/service-account-name.png :width: 60% :align: center :alt: Set Service Account Name 3. Select the ``minimal-skypilot-role`` (or the name you set) created in the last section and click on **DONE**. You can also choose to use the Default or Medium Permissions roles as described in the previous sections. .. image:: ../../images/screenshots/gcp/service-account-grant-role.png :width: 60% :align: center :alt: Set Service Account Role .. _gcp-minimum-firewall-rules: Firewall rules ~~~~~~~~~~~~~~~~~~~ By default, users do not need to set up any special firewall rules to start using SkyPilot. If the default VPC does not satisfy the minimal required rules, a new VPC ``skypilot-vpc`` with sufficient rules will be automatically created and used. However, if you manually set up and instruct SkyPilot to use a custom VPC (see :ref:`below `), ensure it has the following required firewall rules: .. code-block:: python # Allow internal connections between SkyPilot VMs: # # controller -> head node of a cluster # head node of a cluster <-> worker node(s) of a cluster # # NOTE: these ports are more relaxed than absolute minimum, but the # sourceRanges restrict the traffic to internal IPs. { "direction": "INGRESS", "allowed": [ {"IPProtocol": "tcp", "ports": ["0-65535"]}, {"IPProtocol": "udp", "ports": ["0-65535"]}, ], "sourceRanges": ["10.128.0.0/9"], }, # Allow SSH connections from user machine(s) # # NOTE: This can be satisfied using the following relaxed sourceRanges # (0.0.0.0/0), but you can customize it if you want to restrict to certain # known public IPs (useful when using internal VPN or proxy solutions). { "direction": "INGRESS", "allowed": [ {"IPProtocol": "tcp", "ports": ["22"]}, ], "sourceRanges": ["0.0.0.0/0"], }, You can inspect and manage firewall rules at ``https://console.cloud.google.com/net-security/firewall-manager/firewall-policies/list?project=`` or using any of GCP's SDKs. .. _gcp-bring-your-vpc: Using a specific VPC ----------------------- By default, SkyPilot uses the following behavior to get a VPC to use for all GCP instances: - First, all existing VPCs in the project are checked against the minimal recommended firewall rules for SkyPilot to function. If any VPC satisfies these rules, it is used. - Otherwise, a new VPC named ``skypilot-vpc`` is automatically created with the minimal recommended firewall rules and will be used. It is an auto mode VPC that automatically starts with one subnet per region. To instruct SkyPilot to use a specific VPC, you can use SkyPilot's global config file ``~/.sky/config.yaml`` to specify the VPC name in the ``gcp.vpc_name`` field: .. code-block:: yaml gcp: vpc_name: my-vpc-name See details in :ref:`config-yaml`. Example use cases include using a private VPC or a VPC with fine-grained constraints, typically created via Terraform or manually. The custom VPC should contain the :ref:`required firewall rules `. .. _gcp-use-internal-ips: Using internal IPs ----------------------- For security reason, users may only want to use internal IPs for SkyPilot instances. To do so, you can use SkyPilot's global config file ``~/.sky/config.yaml`` to specify the ``gcp.use_internal_ips`` and ``gcp.ssh_proxy_command`` fields (to see the detailed syntax, see :ref:`config-yaml`): .. code-block:: yaml gcp: use_internal_ips: true # VPC with NAT setup, see below vpc_name: my-vpc-name ssh_proxy_command: ssh -W %h:%p -o StrictHostKeyChecking=no myself@my.proxy The ``gcp.ssh_proxy_command`` field is optional. If SkyPilot is run on a machine that can directly access the internal IPs of the instances, it can be omitted. Otherwise, it should be set to a command that can be used to proxy SSH connections to the internal IPs of the instances. Here are two ways to enable SkyPilot's access those private hosts: * **Option 1** – :ref:`SSH proxy / bastion host ` * **Option 2** – :ref:`IAP tunneling (Identity‑Aware Proxy) ` This is often simpler because you do **not** need to run or secure your own bastion; Google handles the tunnel at the edge. Choose whichever fits your environment. .. _gcp-proxy-host: Option 1: Simple single‑proxy example ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If you have a "jump host" that's directly accessible from the machine running SkyPilot, tell SkyPilot how to pipe SSH through it: .. code-block:: yaml gcp: use_internal_ips: true vpc_name: my-vpc-name ssh_proxy_command: ssh -W %h:%p -o StrictHostKeyChecking=no myself@my.proxy When **multiple regions** are involved you can map each region to its own proxy (just as before): .. code-block:: yaml gcp: use_internal_ips: true vpc_name: my-vpc-name ssh_proxy_command: us-west1: ssh -W %h:%p -o StrictHostKeyChecking=no myself@west.proxy us-east1: ssh -W %h:%p -o StrictHostKeyChecking=no myself@east.proxy .. _gcp-iap-tunnel: Option 2: IAP tunneling ~~~~~~~~~~~~~~~~~~~~~~~ `IAP TCP forwarding `__ creates an authenticated tunnel from your workstation to any private IP inside a GCP VPC—no public IP, VPN, or self-managed bastion required. SkyPilot implements the `IP-based (destination-group) mode `__; the per-instance mode is not supported yet. Setup ^^^^^ * Enable the API once per project: .. code-block:: console $ gcloud services enable iap.googleapis.com * This method requires **IAP tunnel permissions**. Adding the built-in role ``roles/iap.tunnelResourceAccessor`` on the destination group (or project-wide) is the simplest approach, in addition to the :ref:`other permissions required by SkyPilot `. * A **Cloud NAT** gateway must exist in every region where you launch internal-only VMs so they can reach the internet for package installs. See :ref:`gcp-cloud-nat`. The following script can be used to set up IAP tunneling in a specific VPC region: .. dropdown:: Automated setup script (per region) .. code-block:: bash #!/usr/bin/env bash # Set your region and VPC. REGION=us-east1 # e.g. us-west1, europe-west4, ... VPC=default # change if using a custom VPC ####################################################################### # 1. Cloud NAT (if none exists) # ####################################################################### nat_exists=false for r in $(gcloud compute routers list --regions="$REGION" \ --format='value(name)'); do if gcloud compute routers nats list --router="$r" --region="$REGION" \ --format='value(name)' | grep -q .; then nat_exists=true fi done if [ "$nat_exists" = false ]; then gcloud compute routers create nat-router --network="$VPC" --region="$REGION" gcloud compute routers nats create nat-config \ --router=nat-router --router-region="$REGION" \ --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips fi ####################################################################### # 2. IAP destination group covering the subnet CIDR # ####################################################################### CIDR=$(gcloud compute networks subnets list --network="$VPC" \ --regions="$REGION" --format='value(ipCidrRange)') gcloud iap tcp dest-groups create "internal-vpc-$VPC" \ --region="$REGION" --ip-range-list="$CIDR" ####################################################################### # 3. Show the SkyPilot config snippet # ####################################################################### echo "Update your config.yaml to add the new gcp.ssh_proxy_command.$REGION value:" echo cat <`. For instance, if you have - destination group: ``internal-vpc-default`` - region: ``us-east1`` - VPC: ``default`` you could use this configuration: .. code-block:: yaml gcp: use_internal_ips: true ssh_proxy_command: us-east1: gcloud compute start-iap-tunnel %h %p --listen-on-stdin --region=us-east1 --network=default --dest-group=internal-vpc-default How it works ^^^^^^^^^^^^ 1. ``sky launch`` creates VMs **without external IPs**. 2. When connecting to the instance, ``ssh`` uses the **IAP proxy command** shown above. ``gcloud compute start-iap-tunnel`` transparently forwards the connection through Google’s edge to the VM’s private address. 3. SSH packets travel over the authenticated IAP tunnel—no public ingress is ever exposed. Troubleshooting ^^^^^^^^^^^^^^^ * *Permission denied / 403* when opening the tunnel → the caller is missing ``roles/iap.tunnelResourceAccessor`` (or an equivalent custom role) on the destination group. * VM cannot reach the Internet → ensure the Cloud NAT gateway is in **the same region** as the subnet and that ``--nat-all-subnet-ip-ranges`` is enabled. * Multiple subnets or VPCs → create one destination group per *region × CIDR* and map each region to the correct group in ``ssh_proxy_command``. .. _gcp-cloud-nat: Cloud NAT setup ~~~~~~~~~~~~~~~~ Instances created with internal IPs only on GCP cannot access public internet by default. To make sure SkyPilot can install the dependencies correctly on the instances, cloud NAT needs to be setup for the VPC (see `GCP's documentation `__ for details). Cloud NAT is a regional resource, so it will need to be created in each region that SkyPilot will be used in. .. image:: ../../images/screenshots/gcp/cloud-nat.png :width: 80% :align: center :alt: GCP Cloud NAT To limit SkyPilot to use some specific regions only, you can specify the ``gcp.ssh_proxy_command`` to be a dict mapping from region to the SSH proxy command for that region (see :ref:`config-yaml` for details): .. code-block:: yaml gcp: use_internal_ips: true vpc_name: my-vpc-name ssh_proxy_command: us-west1: ssh -W %h:%p -o StrictHostKeyChecking=no myself@my.us-west1.proxy us-east1: ssh -W %h:%p -o StrictHostKeyChecking=no myself@my.us-west2.proxy If proxy is not needed, but the regions need to be limited, you can set the ``gcp.ssh_proxy_command`` to be a dict mapping from region to ``null``: .. code-block:: yaml gcp: use_internal_ips: true vpc_name: my-vpc-name ssh_proxy_command: us-west1: null us-east1: null Force enable external IPs ~~~~~~~~~~~~~~~~~~~~~~~~~ An alternative to setting up cloud NAT for instances that need to access the public internet but are in a VPC and communicated with via their internal IP is to force them to be created with an external IP address. .. code-block:: yaml gcp: use_internal_ips: true vpc_name: my-vpc-name force_enable_external_ips: true Using multiple GCP projects ---------------------------------- Admins can isolate different teams to use separate GCP projects. This enables fine-grained resource tracking, billing separation, and access control per team. For more information on configuring workspaces, see `Workspaces `__. Prerequisites ~~~~~~~~~~~~~ The service account must have access to all GCP projects intended to be used. To grant the service account access to additional projects: 1. Follow the :ref:`instructions ` to create a service account with the appropriate roles/permissions. 2. In the GCP Console, switch to the project you want to grant your service account to using the project dropdown at the top of the page, or use ``Cmd/Ctrl + O``. 3. Navigate to the `IAM & Admin console `__ and click on the **IAM** tab. 4. Click **GRANT ACCESS** at the top of the table: - In the **Add principals** field, enter your service account email (e.g., ``my-service-account@main-project.iam.gserviceaccount.com``). - In the **Assign roles** section, select the appropriate roles (see :ref:`Setting permissions ` for recommended roles). - Click **SAVE**. 5. Repeat steps 2-4 for each additional project you want to enable. Configuring workspaces with different projects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Once the service account has access to all the projects, configure the workspaces in the :ref:`SkyPilot config file ` to use different ``project_id`` values: .. code-block:: yaml workspaces: team-a: gcp: project_id: main-project-id team-b: gcp: project_id: secondary-project-id You can then launch resources in a specific workspace: .. code-block:: console $ sky launch --config active_workspace=team-a --infra gcp --- ## File: docs/source/cloud-setup/cloud-permissions/index.rst .. _cloud-auth: Cloud Accounts and Permissions ============================== Some providers in SkyPilot may offer identity and permission-related configuration options. These are **optional** and mostly targeted at cloud admins. **For most users, SkyPilot should work out of the box** with the permissions you have configured locally during :ref:`installation `. The subpages here document cloud configuration details that admins may want to use to limit access, set up advanced features, etc. .. _cloud-permissions: Minimal cloud permissions ------------------------- SkyPilot is designed to minimize cloud administration and setup steps. It should work out of the box for most users who own the root (admin) account of their clouds, if they follow the steps in :ref:`Installation ` section. However, in some cases, the cloud administrator may want to **limit the permissions** of the account used by SkyPilot. This is especially true if the user is using a shared cloud account. The docs linked below describe the permissions required by SkyPilot for each cloud provider. .. note:: Setting up minimal permissions is completely **optional** and is not needed to get started with SkyPilot. The instructions in these docs are only needed for users who want to make SkyPilot use non-default, restricted permissions. Table of contents ----------------- .. toctree:: aws gcp nebius vsphere kubernetes --- ## File: docs/source/cloud-setup/cloud-permissions/kubernetes.rst .. _cloud-permissions-kubernetes: Kubernetes ========== When running outside your Kubernetes cluster, SkyPilot uses your local ``~/.kube/config`` file for authentication and creating resources on your Kubernetes cluster. When running inside your Kubernetes cluster (e.g., as a remote API server, Job controller or Serve controller), SkyPilot can operate using either of the following three authentication methods: 1. **Automatically create a service account**: SkyPilot can automatically create the service account and roles for itself to manage resources in the Kubernetes cluster. This is the default method when running inside the cluster, and no additional configuration is required. For details on the permissions that are granted to the service account, refer to the `Minimum Permissions Required for SkyPilot`_ section below. 2. **Using a custom service account**: If you have a custom service account with the `necessary permissions `__, you can configure SkyPilot to use it by adding this to your :ref:`~/.sky/config.yaml ` file: .. code-block:: yaml kubernetes: remote_identity: your-service-account-name 3. **Using your local kubeconfig file**: In this case, SkyPilot will copy your local ``~/.kube/config`` file to the controller pod and use it for authentication. To use this method, set ``remote_identity: LOCAL_CREDENTIALS`` to your Kubernetes configuration in the :ref:`~/.sky/config.yaml ` file: .. code-block:: yaml kubernetes: remote_identity: LOCAL_CREDENTIALS .. note:: If your cluster uses exec based authentication in your ``~/.kube/config`` file (e.g., GKE uses exec auth by default), SkyPilot may not be able to authenticate using this method. In this case, consider using the service account methods below. .. note:: Service account based authentication applies only when the remote SkyPilot cluster (including spot and serve controller) is launched inside the Kubernetes cluster. When running outside the cluster (e.g., on AWS), SkyPilot will use the local ``~/.kube/config`` file for authentication. Below are the permissions required by SkyPilot and an example service account YAML that you can use to create a service account with the necessary permissions. .. _k8s-permissions: Minimum permissions required for SkyPilot ----------------------------------------- SkyPilot requires permissions equivalent to the following roles to be able to manage the resources in the Kubernetes cluster: .. code-block:: yaml # Namespaced role for the service account # Required for creating pods, services and other necessary resources in the namespace. # Note these permissions only apply in the namespace where SkyPilot is deployed, and the namespace can be changed below. kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sky-sa-role # Can be changed if needed namespace: default # Change to your namespace if using a different one. rules: # Required for managing pods and their lifecycle - apiGroups: [ "" ] resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ] verbs: [ "*" ] # Required for managing services for SkyPilot Pods - apiGroups: [ "" ] resources: [ "services" ] verbs: [ "*" ] # Required for managing SSH keys - apiGroups: [ "" ] resources: [ "secrets" ] verbs: [ "*" ] # Required for retrieving reason when Pod scheduling fails. - apiGroups: [ "" ] resources: [ "events" ] verbs: [ "get", "list", "watch" ] --- # ClusterRole for accessing cluster-wide resources. Details for each resource below: kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sky-sa-cluster-role # Can be changed if needed namespace: default # Change to your namespace if using a different one. labels: parent: skypilot rules: # Required for getting node resources. - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] # Required for autodetecting the runtime class of the nodes. - apiGroups: ["node.k8s.io"] resources: ["runtimeclasses"] verbs: ["get", "list", "watch"] # Required for accessing storage classes. - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] .. tip:: If you are using a different namespace than ``default``, make sure to change the namespace in the above manifests. These roles must apply to both the user account configured in the kubeconfig file and the service account used by SkyPilot (if configured). If you need to view real-time GPU availability with ``sky gpus list``, your tasks use object store mounting or your tasks require access to ingress resources, you will need to grant additional permissions as described below. Permissions for ``sky gpus list`` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``sky gpus list`` needs to list all pods across all namespaces to calculate GPU availability. To do this, SkyPilot needs the ``get`` and ``list`` permissions for pods in a ``ClusterRole``: .. code-block:: yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: sky-sa-cluster-role-pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] .. tip:: If this role is not granted to the service account, ``sky gpus list`` will still work but it will only show the total GPUs on the nodes, not the number of free GPUs. Permissions for GPU metrics ^^^^^^^^^^^^^^^^^^^^^^^^^^^ If the API server federates :ref:`GPU metrics ` from this cluster, the credentials used in the kubeconfig context should also be able to ``get`` the ``kube-system`` namespace. Service accounts created with ``generate_kubeconfig.sh`` and API servers deployed with the Helm chart already include this permission; if you manage permissions manually, add: .. code-block:: yaml # Required only if the API server federates GPU metrics from this cluster. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: sky-sa-cluster-role-namespace-reader rules: - apiGroups: [""] resources: ["namespaces"] resourceNames: ["kube-system"] verbs: ["get"] SkyPilot uses this to detect whether a kubeconfig context points back at the cluster the API server itself runs in (by reading the UID of the ``kube-system`` namespace, the de-facto cluster identity, and comparing it with the UID seen through the in-cluster credentials), which is required to avoid a metrics self-federation loop. .. tip:: If this permission is not granted, GPU metrics for remote clusters still work; only a context that points back at the API server's own cluster would be misdetected as remote. The local cluster can always be referenced via the ``in-cluster`` context instead, which does not need this permission. Permissions for object store mounting ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If your tasks use object store mounting (e.g., S3, GCS, etc.), SkyPilot will need to run a DaemonSet to expose the FUSE device as a Kubernetes resource to SkyPilot pods. To allow this, you will need to also create a ``skypilot-system`` namespace which will run the DaemonSet and grant the necessary permissions to the service account in that namespace. .. code-block:: yaml # Required only if using object store mounting # Create namespace for SkyPilot system apiVersion: v1 kind: Namespace metadata: name: skypilot-system # Do not change this labels: parent: skypilot --- # Role for the skypilot-system namespace to create fusermount-server and # any other system components required by SkyPilot. # This role must be bound in the skypilot-system namespace to the service account used for SkyPilot. kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: skypilot-system-service-account-role # Can be changed if needed namespace: skypilot-system # Do not change this namespace labels: parent: skypilot rules: - apiGroups: [ "*" ] resources: [ "apps" ] verbs: [ "daemonsets" ] Permissions for using Ingress ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If your tasks use :ref:`Ingress ` for exposing ports, you will need to grant the necessary permissions to the service account in the ``ingress-nginx`` namespace. .. code-block:: yaml # Required only if using ingresses # Role for accessing ingress service IP apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: ingress-nginx # Do not change this name: sky-sa-role-ingress-nginx # Can be changed if needed rules: - apiGroups: [""] resources: ["services"] verbs: ["list", "get"] .. _k8s-workload-sa-permissions: Minimum permissions for the workload service account ---------------------------------------------------- When SkyPilot launches a pod on Kubernetes, it creates a service account (``skypilot-service-account``) that is mounted inside the head pod. This **workload service account** is used by the skylet process running inside the pod for operations such as autodown (automatically tearing down idle clusters). By default, this service account is granted broad namespace permissions. If you want to restrict the workload service account to the minimum permissions needed for autodown, you can create a custom service account with the following role and configure SkyPilot to use it via :code:`remote_identity` in :ref:`~/.sky/config.yaml `. .. note:: These permissions are separate from the `Minimum Permissions Required for SkyPilot`_ described above, which apply to the service account used to *launch* clusters (e.g., the API server SA or the user's kubeconfig). The workload SA permissions below apply to the service account *inside* the head pod. .. code-block:: yaml # Namespace role for the workload service account # Only namespace-scoped permissions are needed (no ClusterRole required). kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sky-workload-sa-role # Can be changed if needed namespace: default # Change to your namespace if using a different one. rules: # Required for listing and deleting pods during autodown - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "delete"] # Required for listing and deleting services during autodown. # "deletecollection" is needed for bulk service cleanup. - apiGroups: [""] resources: ["services"] verbs: ["get", "list", "delete", "deletecollection"] # Required for checking if the cluster uses a HA deployment - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list"] # Required for emitting a breadcrumb event when the cluster autodowns # itself, which the API server reads back to record the autostop. - apiGroups: [""] resources: ["events"] verbs: ["create"] To use a custom workload service account, create the service account and role above, then set the following in :ref:`~/.sky/config.yaml `: .. code-block:: yaml # ~/.sky/config.yaml kubernetes: remote_identity: sky-workload-sa # Your custom service account name .. note:: If your workload pods need to perform additional operations beyond autodown (e.g., launching new SkyPilot clusters from within a pod, or running ``sky`` commands inside the pod), the workload service account will need broader permissions similar to the `Minimum Permissions Required for SkyPilot`_. .. _k8s-sa-example: Example using custom service account ------------------------------------ To create a service account that has all necessary permissions for SkyPilot (including for accessing object stores), you can use the following YAML. .. tip:: In this example, the service account is named ``sky-sa`` and is created in the ``default`` namespace. Change the namespace and service account name as needed. .. code-block:: yaml :linenos: # create-sky-sa.yaml kind: ServiceAccount apiVersion: v1 metadata: name: sky-sa # Change to your service account name namespace: default # Change to your namespace if using a different one. labels: parent: skypilot --- # Role for the service account kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sky-sa-role # Can be changed if needed namespace: default # Change to your namespace if using a different one. labels: parent: skypilot rules: # Required for managing pods and their lifecycle - apiGroups: [ "" ] resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ] verbs: [ "*" ] # Required for managing services for SkyPilot Pods - apiGroups: [ "" ] resources: [ "services" ] verbs: [ "*" ] # Required for managing SSH keys - apiGroups: [ "" ] resources: [ "secrets" ] verbs: [ "*" ] # Required for retrieving reason when Pod scheduling fails. - apiGroups: [ "" ] resources: [ "events" ] verbs: [ "get", "list", "watch" ] --- # RoleBinding for the service account kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sky-sa-rb # Can be changed if needed namespace: default # Change to your namespace if using a different one. labels: parent: skypilot subjects: - kind: ServiceAccount name: sky-sa # Change to your service account name roleRef: kind: Role name: sky-sa-role # Use the same name as the role at line 14 apiGroup: rbac.authorization.k8s.io --- # ClusterRole for the service account kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: sky-sa-cluster-role # Can be changed if needed labels: parent: skypilot rules: - apiGroups: [""] resources: ["nodes"] # Required for getting node resources. verbs: ["get", "list", "watch"] - apiGroups: ["node.k8s.io"] resources: ["runtimeclasses"] # Required for autodetecting the runtime class of the nodes. verbs: ["get", "list", "watch"] - apiGroups: ["networking.k8s.io"] # Required for exposing services through ingresses resources: ["ingressclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] # Required for `sky gpus list` command resources: ["pods"] verbs: ["get", "list"] - apiGroups: ["storage.k8s.io"] # Required for using volumes resources: ["storageclasses"] verbs: ["get", "list", "watch"] --- # ClusterRoleBinding for the service account apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: sky-sa-cluster-role-binding # Can be changed if needed labels: parent: skypilot subjects: - kind: ServiceAccount name: sky-sa # Change to your service account name namespace: default # Change to your namespace if using a different one. roleRef: kind: ClusterRole name: sky-sa-cluster-role # Use the same name as the cluster role at line 56 apiGroup: rbac.authorization.k8s.io --- # Optional: If using object store mounting, create the skypilot-system namespace apiVersion: v1 kind: Namespace metadata: name: skypilot-system # Do not change this labels: parent: skypilot --- # Optional: If using object store mounting, create role in the skypilot-system # namespace to create fusermount-server. kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: skypilot-system-service-account-role # Can be changed if needed namespace: skypilot-system # Do not change this namespace labels: parent: skypilot rules: - apiGroups: [ "apps" ] resources: [ "daemonsets" ] verbs: [ "*" ] --- # Optional: If using object store mounting, create rolebinding in the skypilot-system # namespace to create fusermount-server. apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: sky-sa-skypilot-system-role-binding namespace: skypilot-system # Do not change this namespace labels: parent: skypilot subjects: - kind: ServiceAccount name: sky-sa # Change to your service account name namespace: default # Change this to the namespace where the service account is created roleRef: kind: Role name: skypilot-system-service-account-role # Use the same name as the role above apiGroup: rbac.authorization.k8s.io --- # Optional: Role for accessing ingress resources apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: sky-sa-role-ingress-nginx # Can be changed if needed namespace: ingress-nginx # Do not change this namespace labels: parent: skypilot rules: - apiGroups: [""] resources: ["services"] verbs: ["list", "get", "watch"] --- # Optional: RoleBinding for accessing ingress resources apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: sky-sa-rolebinding-ingress-nginx # Can be changed if needed namespace: ingress-nginx # Do not change this namespace labels: parent: skypilot subjects: - kind: ServiceAccount name: sky-sa # Change to your service account name namespace: default # Change this to the namespace where the service account is created roleRef: kind: Role name: sky-sa-role-ingress-nginx # Use the same name as the role above apiGroup: rbac.authorization.k8s.io Create the service account using the following command: .. code-block:: bash $ kubectl apply -f create-sky-sa.yaml After creating the service account, the cluster admin may distribute kubeconfigs with the ``sky-sa`` service account to users who need to access the cluster. Users should also configure SkyPilot to use the ``sky-sa`` service account through ``~/.sky/config.yaml``: .. code-block:: yaml # ~/.sky/config.yaml kubernetes: remote_identity: sky-sa # Or your service account name --- ## File: docs/source/cloud-setup/cloud-permissions/vsphere.rst .. _cloud-prepare-vsphere: vSphere ======= This document is provided for users who use VMware vSphere provider and helps them set up basic configuration on VMware vSphere to meet the needs of running SkyPilot tasks. .. _cloud-prepare-vsphere-tags: Prepare category & tag ~~~~~~~~~~~~~~~~~~~~~~~ The Categories and Tags is needed when using the vSphere provider, please follow bellow steps to create them. **Create the Category:** 1. Open web browser and Login your vSphere Client. 2. Navigate to **Menu** -> **Tags & Custom Attributes**. .. image:: ../../images/screenshots/vsphere/vsphere-catagory-create_navigate.png :width: 80% :align: center :alt: vSphere Catagory Creation Navigate 3. Click **Tags** -> **CATEGORIES** -> **NEW**. .. image:: ../../images/screenshots/vsphere/vsphere-catagory-create_navigate_new.png :width: 80% :align: center :alt: vSphere Catagory Creation Navigate New 4. Fill in the **Create Category** form with follow content: .. code-block:: text Category Name: skypilot Tags Per Object: Many tags Associable Object Types: Datastore, Content Library, Library Item, VirtualMachine .. image:: ../../images/screenshots/vsphere/vsphere-catagory-create.png :width: 80% :align: center :alt: vSphere Catagory Creation 5. Click the **CREATE** to finish the creation. **Create the Tag:** 1. Open web browser and Login your vSphere Client. 2. Navigate to **Menu** -> **Tags & Custom Attributes**. .. image:: ../../images/screenshots/vsphere/vsphere-catagory-create_navigate.png :width: 80% :align: center :alt: vSphere Catagory Creation Navigate 3. Click **Tags** -> **TAGS** -> **NEW**. .. image:: ../../images/screenshots/vsphere/vsphere-tags-create_navigate.png :width: 80% :align: center :alt: vSphere Catagory Creation Navigate 4. Fill in the **Create Tag** form with follow content: .. code-block:: text Name: skypilot Category: skypilot .. image:: ../../images/screenshots/vsphere/vsphere-tags-create.png :width: 80% :align: center :alt: vSphere Tags Creation 5. Click the **CREATE** to finish the creation. .. _cloud-prepare-vsphere-storage-policy: Create VM storage policies ~~~~~~~~~~~~~~~~~~~~~~~~~~ The vSphere provider depends on the VM Storage Policies to place the VM. A Shared Datastore is recommended. 1. Open web browser and Login your vSphere Client. 2. Navigate to **Menu** -> **Inventory**. .. image:: ../../images/screenshots/vsphere/vsphere-vm-storage-policy-inventory.png :width: 80% :align: center :alt: Datastore Add Tag 3. Click the **Datastore Icon**. Select the datastore eligible for VM creation, assign the previously created ``skypilot`` tag to them. .. image:: ../../images/screenshots/vsphere/vsphere-datastore-tag-adding.png :width: 80% :align: center :alt: Datastore Add Tag 4. Navigate to **Menu** -> **Policies and Profiles**. .. image:: ../../images/screenshots/vsphere/vsphere-vm-storage-policy-navigate.png :width: 80% :align: center :alt: Storage Policy Navigate 5. Select **VM Storage Policies** , and click **CREATE** button. .. image:: ../../images/screenshots/vsphere/vsphere-vm-storage-policy-navigate-new.png :width: 80% :align: center :alt: Storage Policy Navigate New 6. In the **Name and description** step, specify the name as ``skypilot_policy``. .. image:: ../../images/screenshots/vsphere/vsphere-vm-storage-policy-name.png :width: 80% :align: center :alt: Policy Name 7. In the **Policy structure** step, select **Enable tag based placement rules**. .. image:: ../../images/screenshots/vsphere/vsphere-vm-storage-policy-rule.png :width: 80% :align: center :alt: Policy Rule 8. In the **Tag based placement** step, select the tag **skypilot**. .. image:: ../../images/screenshots/vsphere/vsphere-vm-storage-policy-tag.png :width: 80% :align: center :alt: Policy Tags 9. In the **Storage compatibility** step, review the datastores. .. image:: ../../images/screenshots/vsphere/vsphere-vm-storage-policy-review.png :width: 80% :align: center :alt: Policy Review 10. Review and Click the **FINISH** to create the policy. .. _cloud-prepare-vsphere-content-library-create: Prepare VM image ~~~~~~~~~~~~~~~~ The VM must be Linux-based; we take Ubuntu 20.04 as the base OS in this document. Other Linux distributions may also works but not promised. **Step 1. Prepare a linux-based VM**. Open web browser and Login your vSphere Client, create a linux-based Virtual Machine. If you're unfamiliar with the process of creating a virtual machine, please refer to this guide for assistance: `Deploying Virtual Machines `_ **Step 2. Create a user and Enable password-less sudo**. Login your prepared VM, Create a user named ``ubuntu``, and add to the sudo group. .. code-block:: shell sudo adduser ubuntu sudo usermod -aG sudo ubuntu Edit the sudoer file to enable passwordless sudo. .. code-block:: shell sudo visudo Add the following line: .. code-block:: text ubuntu ALL=(ALL:ALL) NOPASSWD: ALL **Step 3. Check VMtools version**. Execute the following command to check VMtools version 10.1.0 or later is required. .. code-block:: shell vmware-toolbox-cmd -v **Step 4. Enabling Custom Scripts in VMtools**. Execute the following command to check if the enable-custom-scripts option is enabled: .. code-block:: shell sudo vmware-toolbox-cmd config get deployPkg enable-custom-scripts If the enable-custom-scripts option is disabled, enable it by executing: .. code-block:: shell sudo vmware-toolbox-cmd config set deployPkg enable-custom-scripts true **Step 5. Install the GPU driver**. Different GPUs require different drivers; please choose the correct driver based on your GPU mode and OS version. Take Ubuntu and Nvidia GPU for example, the following driver works for most of the case: .. code-block:: shell sudo apt install nvidia-headless-535-server --no-install-recommends **Step 6. Convert the VM to template**. 1. Power off the VM. 2. Select the VM and right click -> **Clone**-> **Clone as Template to Library**. .. image:: ../../images/screenshots/vsphere/vm-clone-to-template.png :width: 80% :align: center :alt: VM Clone To Temple 3. Select Template type: **OVF**, provide the template name and click **Next**. .. image:: ../../images/screenshots/vsphere/vm-clone-to-template-ovf.png :width: 80% :align: center :alt: VM Clone To Temple OVF 4. Select a Content Library location and click **Next**. Create one if you haven't set up a **Local** content library. Please refer to `Create a Library `_ .. image:: ../../images/screenshots/vsphere/vm-clone-to-template-cl.png :width: 80% :align: center :alt: VM Clone To Temple Content Lib 5. Review and click **Finish**. **Step 7. Tag the VM template**. Follow the steps bellow to tag the VM template: 1. Navigate to **Menu** -> **Content Libraries**. .. image:: ../../images/screenshots/vsphere/content-libs-navigate.png :width: 80% :align: center :alt: Content Libs 2. Review the content libraries and click on the one that contains your target VM templates. .. image:: ../../images/screenshots/vsphere/content-libs.png :width: 80% :align: center :alt: Content Libs 3. Review the content library items and click on your target VM template. .. image:: ../../images/screenshots/vsphere/content-lib-item.png :width: 80% :align: center :alt: Content Lib Item 4. Assign the relevant tags to the VM template. If you no sure how to assign the tags, please refer to :ref:`Rules for tagging VM template ` .. image:: ../../images/screenshots/vsphere/content-lib-item-tag-adding.png :width: 80% :align: center :alt: Content Lib Item Tagging .. _cloud-prepare-vsphere-references: References ~~~~~~~~~~ **Rules for tagging VM template** The vSphere driver for SkyPilot uses the Tag to identify the VM images for GPU. VI admin needs to tag the VM template accordingly in the Content Library. Here are a few rules for tagging templates: - All workloads that do not need an accelerator will use the VM image tagged with ``skypilot-cpu``. If you have multiple VM templates (Content Item) tagged with ``skypilot-cpu`` only the first one will be used, so the best practice is only tagging one image with ``skypilot-cpu``. - Vi Admin should create a default VM Image for each accelerator vendor and tag it with the format ``skypilot-vendorname`` e.g., ``skypilot-nvidia``. The fallback logic is if the user requires a specific Nvidia GPU(e.g., k1200) and the vSphere driver cannot find an image with tag ``skypilot-K1200`` it will use the VM image tagged with ``skypilot-nvidia``. - All the tags should select the Tag category as: ``skypilot``. **Support new GPU** The vSphere cloud provider has a default support GPU list. You can first check the default one by executing the ``sky check`` command. Then the GPU list can be found in the ``~/.sky/catalogs/v5/vsphere/accelerators.csv`` file. If you want to support a new GPU, say ``Nvidia K1200`` you can add one more line at the end of the file: .. code-block:: text Model,Type,MemoryMB,vCPUs,fullNames ... K1200,GPU,4096,4,['GM107GL [Quadro K1200]'] If the GPU needs a particular driver, ask the VI admin to create a new VM template and upload it to the Content Library, then add a tag to the VM template, e.g., ``GPU-k1200``. If the new default VM template's driver supports the new GPU, you will not need to create a new one. For example, if the VM template with the Tag ``skypilot-nvidia`` supports the new GPU, then you will not need to create a new VM template. --- ## File: docs/source/cloud-setup/policy.rst .. _advanced-policy-config: Admin Policy Enforcement ======================== SkyPilot provides an **admin policy** mechanism that admins can use to enforce certain policies on users' SkyPilot usage. An admin policy applies custom validation and mutation logic to a user's tasks and SkyPilot config. Example usage: - :ref:`kubernetes-labels-policy` - :ref:`disable-public-ip-policy` - :ref:`use-spot-for-gpu-policy` - :ref:`enforce-autostop-policy` - :ref:`dynamic-kubernetes-contexts-update-policy` - :ref:`use-local-gcp-credentials-policy` - :ref:`add-volumes-policy` - :ref:`reject-old-clients-policy` - :ref:`slurm-partition-routing-policy` - :ref:`slurm-filesystem-routing-policy` Overview -------- SkyPilot has a :ref:`client-server architecture `, where a centralized API server can be deployed and users can interact with the server through a client. To deploy a admin policy, here are the steps: 1. :ref:`Implement the policy in a Python package ` or :ref:`host the policy as a RESTful server `. 2. Install the policy at the :ref:`server-side ` to enforce it for all users. 3. Optionally, an admin policy can also be installed at the :ref:`client-side` if it needs to access user's local environment. The order of policy application is demonstrated below: .. figure:: ../images/admin-policy.png :align: center :width: 80% :alt: Admin policy application order .. note:: Client-side policy lacks enforcement capability, i.e., end-user may modify them. However, client policy is useful for automation that can only be applied at client-side. Refer to :ref:`use-local-gcp-credentials-policy` as an example. Quickstart ---------- Install the example policy package: .. code-block:: bash git clone https://github.com/skypilot-org/skypilot.git cd skypilot pip install examples/admin_policy/example_policy Then, set the :ref:`admin_policy ` field in :ref:`the SkyPilot config ` to use an example policy. .. code-block:: yaml admin_policy: example_policy.DoNothingPolicy .. tip:: You can replace ``DoNothingPolicy`` with any of the :ref:`example policies `. Then, run a task: .. code-block:: bash $ sky launch You should see the admin policy is applied to the task. .. code-block:: bash $ sky launch Applying client admin policy: DoNothingPolicy Applying server admin policy: DoNothingPolicy ... You can also deploy an admin policy as a :ref:`RESTful server `. Deploy an admin policy ---------------------- .. _server-side-admin-policy: Server-side ~~~~~~~~~~~ If you have a :ref:`centralized API server ` deployed, you can enforce a policy for all users by setting it at the server-side. .. tab-set:: .. tab-item:: Use a RESTful policy Open :ref:`SkyPilot dashboard ` https://api.server.com/dashboard/config, and set the :ref:`admin_policy ` field to the URL of the RESTful policy. To host a RESTful policy, see :ref:`here `. .. code-block:: yaml admin_policy: https://example.com/policy .. tab-item:: Use a Python package First, install the Python package that implements the policy on the API server host: .. code-block:: bash pip install mypackage.subpackage For helm deployment, refer to :ref:`sky-api-server-admin-policy` to install the policy package. Then, open the server's dashboard, go to :ref:`the server's SkyPilot config ` and set the :ref:`admin_policy ` field to the path of the Python package that implements the policy. .. code-block:: yaml admin_policy: mypackage.subpackage.MyPolicy .. _client-side-admin-policy: Client-side ~~~~~~~~~~~ If the policy needs to access user's local environment, you can get the policy applied at the client-side by following the steps below. First, install the Python package that implements the policy: .. code-block:: bash pip install mypackage.subpackage Then, set the :ref:`admin_policy ` field in :ref:`the SkyPilot config ` to the path of the Python package that implements the policy. .. code-block:: yaml admin_policy: mypackage.subpackage.MyPolicy Optionally, you can also apply different policies in different projects by leveraging the :ref:`layered config `, e.g. set a different policy in ``$pwd/.sky.yaml`` for the current project: .. code-block:: yaml admin_policy: mypackage.subpackage.AnotherPolicy .. hint:: SkyPilot loads the policy from the given package in the same Python environment. You can test the existence of the policy by running: .. code-block:: bash python -c "from mypackage.subpackage import MyPolicy" .. note:: It is possible to call a RESTful policy at client-side. However, a RESTful policy is executed on the policy server host, i.e., cannot access user's local environment, e.g., local files. .. _host-admin-policy-as-server: Host admin policy as a RESTful server ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ You can host an admin policy as a RESTful API server and configure the SkyPilot to call the RESTful url to apply the policy. It is recommended to :ref:`inherit your implementation from the AdminPolicy interface ` to ensure the request and response body are correctly typed. You can also import existing policies at the server and composite these policies to fit your needs. Here is an example of implementing a policy server using Python and FastAPI: .. dropdown:: Example Policy Server .. literalinclude:: ../../../examples/admin_policy/example_server/policy_server.py :language: python :caption: `Policy Server `_ Optionally, the server can also be implemented in other languages as long as it follows the API convention: .. dropdown:: The Admin Policy API **POST /** Request body is a marshalled :ref:`sky.UserRequest ` in JSON format: .. code-block:: json { "task": { "name": "sky-cmd", "resources": { "cpus": "1+", }, "num_nodes": 1, }, "skypilot_config": {}, "request_options": { "cluster_name": "test", "idle_minutes_to_autostop": null, "down": false, "dryrun": false }, "at_client_side": false } Response body is a marshalled :ref:`sky.MutatedUserRequest ` in JSON format: .. code-block:: json { "task": { "name": "sky-cmd", "resources": { "cpus": "1+", }, "num_nodes": 1, }, "skypilot_config": {} } .. _implement-admin-policy: Implement an admin policy ------------------------- Admin policies are implemented by extending the ``sky.AdminPolicy`` `interface `_: .. dropdown:: AdminPolicy Interface .. literalinclude:: ../../../sky/admin_policy.py :language: python :pyobject: AdminPolicy :name: admin-policy-interface :caption: `AdminPolicy Interface `_ Your custom admin policy should look like this: .. code-block:: python import sky class MyPolicy(sky.AdminPolicy): @classmethod def validate_and_mutate(cls, user_request: sky.UserRequest) -> sky.MutatedUserRequest: # Logic for validate and modify user requests. ... return sky.MutatedUserRequest(user_request.task, user_request.skypilot_config) ``UserRequest`` and ``MutatedUserRequest`` are defined as follows (see `source code `_ for more details): .. dropdown:: UserRequest Class .. literalinclude:: ../../../sky/admin_policy.py :language: python :pyobject: UserRequest :name: user-request-class :caption: `UserRequest Class `_ .. dropdown:: MutatedUserRequest Class .. literalinclude:: ../../../sky/admin_policy.py :language: python :pyobject: MutatedUserRequest :name: mutated-user-request-class :caption: `MutatedUserRequest Class `_ In other words, an ``AdminPolicy`` can mutate any fields of a user request, including the :ref:`task ` and the :ref:`skypilot config ` for that specific user request, giving admins a lot of flexibility to control user's SkyPilot usage. An ``AdminPolicy`` can be used to both validate and mutate user requests. If a request should be rejected, the policy should raise an exception. The ``sky.Config`` and ``sky.RequestOptions`` classes are defined as follows: .. dropdown:: Config Class .. literalinclude:: ../../../sky/utils/config_utils.py :language: python :pyobject: Config :caption: `Config Class `_ .. dropdown:: RequestOptions Class .. literalinclude:: ../../../sky/admin_policy.py :language: python :pyobject: RequestOptions :caption: `RequestOptions Class `_ Tips for writing an admin policy -------------------------------- When writing an admin policy, you can leverage the following tips to make your policy more robust and flexible. Access user information ~~~~~~~~~~~~~~~~~~~~~~~ Use ``UserRequest.user`` to access the user who made the request. This field is only available when the policy is applied at the server-side. Useful for: - Logging the user who made a specific request. - Implementing per-user quotas or rate limits. - Changing the behavior of the policy based on the user. Access request name ~~~~~~~~~~~~~~~~~~~ Use ``UserRequest.request_name`` to access the request name. Useful for: - Activating a policy selectively for certain request types. - Implementing per-request rate limits. - Changing the behavior of the policy based on the request name. Access client version information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use ``UserRequest.client_api_version`` and ``UserRequest.client_version`` to access the version information of the client making the request. - ``client_api_version``: The API version of the client (an integer, e.g., ``31``). This version is incremented when breaking changes are made to the API. - ``client_version``: The package version string of the client (e.g., ``"1.0.0"`` or ``"1.0.0-dev0 (commit: abc1234)"``). These fields are only available when the policy is applied at the server-side. When applied at the client-side, these fields will be ``None``. .. note:: For very old clients that don't send version headers (pre-v0.11.0), ``client_api_version`` will be ``None`` and ``client_version`` will be ``"unknown"``. Useful for: - Enforcing minimum client versions to ensure compatibility. - Logging client version information for debugging. - Implementing version-specific behavior or deprecation warnings. See :ref:`reject-old-clients-policy` for an example policy that uses client version information. Inspect and modify resources ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Use ``UserRequest.task.get_resource_config()`` to get the resource configuration of a task. The resource configuration is a dictionary conforming to the :ref:`resource config schema `. Once the resource configuration is modified, you can use ``UserRequest.task.set_resources(resource_config)`` to set the modified resource configuration back to the task. .. code-block:: python resource_config = user_request.task.get_resource_config() resource_config['use_spot'] = True if 'any_of' in resource_config: for any_of_config in resource_config['any_of']: any_of_config['use_spot'] = True elif 'ordered' in resource_config: for ordered_config in resource_config['ordered']: ordered_config['use_spot'] = True user_request.task.set_resources(resource_config) Useful for: - Enforcing resource constraints (e.g. use spot instances for all GPU tasks, enforce autostop for all tasks). .. _example-policies: Example policies ---------------- We have provided a few example policies in `examples/admin_policy/example_policy `_. You can test these policies by installing the example policy package in your Python environment. .. code-block:: bash git clone https://github.com/skypilot-org/skypilot.git cd skypilot pip install examples/admin_policy/example_policy Reject all tasks ~~~~~~~~~~~~~~~~ .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: RejectAllPolicy :caption: `RejectAllPolicy `_ .. literalinclude:: ../../../examples/admin_policy/reject_all.yaml :language: yaml :caption: `Config YAML for using RejectAllPolicy `_ .. _kubernetes-labels-policy: Add labels for all tasks on Kubernetes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: AddLabelsPolicy :caption: `AddLabelsPolicy `_ .. literalinclude:: ../../../examples/admin_policy/add_labels.yaml :language: yaml :caption: `Config YAML for using AddLabelsPolicy `_ .. _disable-public-ip-policy: Always disable public IP for AWS tasks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: DisablePublicIpPolicy :caption: `DisablePublicIpPolicy `_ .. literalinclude:: ../../../examples/admin_policy/disable_public_ip.yaml :language: yaml :caption: `Config YAML for using DisablePublicIpPolicy `_ .. _use-spot-for-gpu-policy: Use spot for all GPU tasks ~~~~~~~~~~~~~~~~~~~~~~~~~~ .. .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: UseSpotForGpuPolicy :caption: `UseSpotForGpuPolicy `_ .. literalinclude:: ../../../examples/admin_policy/use_spot_for_gpu.yaml :language: yaml :caption: `Config YAML for using UseSpotForGpuPolicy `_ .. _enforce-autostop-policy: Enforce autostop for all tasks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: EnforceAutostopPolicy :caption: `EnforceAutostopPolicy `_ .. literalinclude:: ../../../examples/admin_policy/enforce_autostop.yaml :language: yaml :caption: `Config YAML for using EnforceAutostopPolicy `_ Set max autostop idle minutes for all tasks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: SetMaxAutostopIdleMinutesPolicy :caption: `SetMaxAutostopIdleMinutesPolicy `_ .. literalinclude:: ../../../examples/admin_policy/set_max_autostop_idle_minutes.yaml :language: yaml :caption: `Config YAML for using SetMaxAutostopIdleMinutesPolicy `_ .. _dynamic-kubernetes-contexts-update-policy: Dynamically update Kubernetes contexts to use ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: DynamicKubernetesContextsUpdatePolicy :caption: `DynamicKubernetesContextsUpdatePolicy `_ .. literalinclude:: ../../../examples/admin_policy/dynamic_kubernetes_contexts_update.yaml :language: yaml :caption: `Config YAML for using DynamicKubernetesContextsUpdatePolicy `_ .. _use-local-gcp-credentials-policy: Use local GCP credentials for all tasks ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/client_policy.py :language: python :pyobject: UseLocalGcpCredentialsPolicy :caption: `UseLocalGcpCredentialsPolicy `_ Specify the following config in :ref:`the SkyPilot config ` at the client: .. literalinclude:: ../../../examples/admin_policy/use_local_gcp_credentials.yaml :language: yaml :caption: `Config YAML for using UseLocalGcpCredentialsPolicy `_ Then specify the policy at the server with the same config, or call this policy in the :ref:`RESTful policy server `. This policy only take effects when applied at :ref:`client-side `. Use this policy at the :ref:`server-side ` will be a no-op. .. _add-volumes-policy: Add volumes to all tasks ~~~~~~~~~~~~~~~~~~~~~~~~ .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: AddVolumesPolicy :caption: `AddVolumesPolicy `_ .. literalinclude:: ../../../examples/admin_policy/add_volumes.yaml :language: yaml :caption: `Config YAML for using AddVolumesPolicy `_ .. _rate-limit-launch-policy: Rate limit cluster launch requests ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: RateLimitLaunchPolicy :caption: `RateLimitLaunchPolicy `_ The ``RateLimitLaunchPolicy`` uses ``TokenBucketRateLimiter`` class to rate limit cluster launch requests. .. dropdown:: TokenBucketRateLimiter Class .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: TokenBucketRateLimiter :caption: `TokenBucketRateLimiter `_ .. literalinclude:: ../../../examples/admin_policy/rate_limit_launch.yaml :language: yaml :caption: `Config YAML for using RateLimitLaunchPolicy `_ .. _gpu-static-quota-policy: Enforce a static GPU quota for each user on cluster launch requests ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .. note:: This policy calls sky.status() to get the total number of GPUs currently used by the user and therefore adds a few seconds of latency to every cluster launch request. This policy should be considered an educational example and not a production-ready policy. .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: GPUStaticQuotaPolicy :caption: `GPUStaticQuotaPolicy `_ .. literalinclude:: ../../../examples/admin_policy/gpu_static_quota.yaml :language: yaml :caption: `Config YAML for using GPUStaticQuotaPolicy `_ .. _reject-old-clients-policy: Reject requests from old clients ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This policy demonstrates how to use client version information to enforce minimum client versions. It rejects requests from clients with an API version below a specified threshold. .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: RejectOldClientsPolicy :caption: `RejectOldClientsPolicy `_ .. literalinclude:: ../../../examples/admin_policy/reject_old_clients.yaml :language: yaml :caption: `Config YAML for using RejectOldClientsPolicy `_ .. _slurm-partition-routing-policy: Route Slurm jobs to partitions based on resources ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This policy automatically routes Slurm jobs to the appropriate partition based on the requested resources. GPU tasks go to the ``gpu`` partition, high-memory CPU tasks go to the ``highmem`` partition, and all other tasks go to the ``cpu`` partition. If the user has already specified a partition (zone), the policy respects their choice. .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: SlurmPartitionRoutingPolicy :caption: `SlurmPartitionRoutingPolicy `_ .. literalinclude:: ../../../examples/admin_policy/slurm_partition_routing.yaml :language: yaml :caption: `Config YAML for using SlurmPartitionRoutingPolicy `_ .. _slurm-filesystem-routing-policy: Route Slurm jobs to clusters based on mounted filesystems ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This policy routes Slurm jobs only to clusters where the required filesystem paths are accessible. Users declare required paths via the ``SKYPILOT_REQUIRED_FILESYSTEMS`` environment variable in their task (comma-separated). The policy connects to each candidate cluster over SSH at optimization time and checks that all required paths exist. Results are cached per cluster to avoid repeated SSH calls. This is useful when the same dataset is stored at different paths on different clusters (e.g. ``/mnt/data/MNIST`` on one cluster and ``/home/$USER/data/MNIST`` on another). Users can set the environment variable via the ``envs`` field in a :ref:`task YAML `: .. code-block:: yaml envs: SKYPILOT_REQUIRED_FILESYSTEMS: /mnt/home,/mnt/data resources: cloud: slurm Or via the ``--env`` flag in ``sky launch`` :ref:`CLI `: .. code-block:: console $ sky launch --infra slurm --env SKYPILOT_REQUIRED_FILESYSTEMS=/home/ubuntu .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py :language: python :pyobject: SlurmFilesystemRoutingPolicy :caption: `SlurmFilesystemRoutingPolicy `_ .. literalinclude:: ../../../examples/admin_policy/slurm_filesystem_routing.yaml :language: yaml :caption: `Config YAML for using SlurmFilesystemRoutingPolicy `_ --- ## File: docs/source/cloud-setup/quota.rst .. _quota: Requesting Quota Increase ========================= Most cloud providers enforce a quota policy to limit the number of VM instances that can exist in a given region. Users may encounter `QuotaExceeded` or `VcpuLimitExceeded` errors during resources provisioning, especially for high end GPUs such as H100/A100. To check or increase your quota limits, please follow the below instructions. After submitting the request, it will usually take a few days for the support team to review. To increase chances of being approved, you may respond their inquiry emails on how the requested resources will be used your projects. .. _quota-aws: AWS --- 1. Go to the `EC2 Quotas console `_. 2. **Select a region** on the top right. 3. Choose an EC2 instance type from the list (e.g, ``Running On-Demand P instances`` or ``All P Spot Instance Requests``). Use ``sky gpus list --infra aws --all`` or check `here `__ for more instance types. 4. Click the quota name, and then choose **Request quota increase**. 5. For **Change quota value**, enter the new value. 6. Choose **Request**. Azure ----- 1. First go to `Azure's quota page `_. 2. Select **Request Increase** near the top of the screen. 3. For Quota type, select ``Compute-VM (cores-vCPUs) subscription limit increases``. Hint: note that a message "Get more quota now. You don’t need a support ticket to get more quota..." may pop up; feel free to skip it as requesting quotas for most GPU instances still requires creating support tickets (next steps). 4. Select **Next** to go to the Additional details screen, then select **Enter details**. - In the Quota details screen: - For Deployment model, ensure **Resource Manager** is selected. - For Locations, select all regions in which you want to increase quotas. - For each region you selected, select one or more VM series from the Quotas drop-down list. - For each VM Series you selected (e.g., ``ND_H100_v5`` for H100 instances), enter the new vCPU limit that you want for this subscription. You may check `for more VM Series `_. - When you're finished, select **Save and continue**. 5. Enter or confirm your contact details, then select **Next**. 6. Finally, ensure that everything looks correct on the Review + create page, then select **Create** to submit your request. GCP --- 1. In the Google Cloud Console, go to the `Quota page `_. 2. Click **Filter** and select ``Service: Compute Engine API``. 3. For H100 GPUs: choose ``metric: GPUS_PER_GPU_FAMILY`` and select dimension ``gpu_family: NVIDIA_H100``. 4. For all other GPUs: choose ``Limit Name: instance_name``. (e.g., ``NVIDIA-V100-GPUS-per-project-region``). You may check the `the compute GPU list `_. 5. Select the checkbox of the region whose quota you want to change. 6. Click **Edit Quotas** and fill out the new limit. 7. Click **Submit Request**. OCI ------------------------------- 1. Go to the `OCI Limits, Quotas and Usage console `_ to check your current resources status. 2. Click the **request a service limit increase** link on the page if you want to increase quotas. 3. Choose a **Service Category** from the list (e.g, ``Compute``). 4. Choose a **Resource** from the list (e.g, ``GPUs for GPU.A10 based VM and BM Instances``). Use ``sky gpus list --infra oci --all`` or check `here `__ for more instance types. 5. Enter the **Limit** field for your new limit and **Reason for request** for justification. 6. Click **Create Support Request** to submit. 7. You may check `OCI Service Limits `_ for more information.