skypilot Documentation

# Choose your clouds:
uv pip install "skypilot[kubernetes,aws,gcp,azure,oci,nebius,lambda,runpod,fluidstack,paperspace,cudo,ibm,scp,seeweb,shadeform,verda]"

Fetch and follow https://github.com/skypilot-org/skypilot/blob/HEAD/agent/INSTALL.md to install the skypilot skill

resources:
  accelerators: A100:8  # 8x NVIDIA A100 GPU

num_nodes: 1  # Number of VMs to launch

# Working directory (optional) containing the project codebase.
# Its contents are synced to ~/sky_workdir/ on the cluster.
workdir: ~/torch_examples

# Commands to be run before executing the job.
# Typical use: pip install -r requirements.txt, git clone, etc.
setup: |
  cd mnist
  pip install -r requirements.txt

# Commands to run as a job.
# Typical use: launch the main program.
run: |
  cd mnist
  python main.py --epochs 1

git clone https://github.com/pytorch/examples.git ~/torch_examples

sky launch my_task.yaml

docker build . -t fusermount-server:latest

spec:
  containers:
  - name: main
    volumeMounts:
      - mountPath: /var/run/fusermount
        name: fusermount-shared-dir
  volumes:
    - hostPath:
        path: /var/run/fusermount
        type: DirectoryOrCreate
      name: fusermount-shared-dir

# Install fuse2 and fuse3
apt-get install -y fuse fuse3
# Locate the original fusermount binary.
FUSERMOUNT_PATH=$(which fusermount)
if [ -z "$FUSERMOUNT_PATH" ]; then
    echo "Error: fusermount binary not found"
    exit 1
fi
FOURSMOUNE3_PATH=$(which fusemount3)
if [ -z "$FOURSMOUNE3_PATH" ]; then
    echo "Error: fusemount3 binary not found"
    exit 1
fi
# The -original suffix is crucial: the server will enter the mnt namespace of application container and find the original fusermount binary
# by searching `fusermount-original` executable in PATH.
cp -p "$FUSERMOUNT_PATH" "${FUSERMOUNT_PATH}-original"
# Mask the fusermount/fusemount3 binary by the shim provided by fusermount-server.
ln -sf /var/run/fusermount/fusermount-shim "$FUSERMOUNT_PATH"
ln -sf /var/run/fusermount/fusermount-shim "$FOURSMOUNE3_PATH"

sequenceDiagram
    SkyPilot->>+Fuse Adapter: sync mount
    box beige Sky Pod (non-Privileged)
    participant Fuse Adapter
    participant fusermount-shim
    end
    Fuse Adapter->>+fusermount-shim: exec
    fusermount-shim->>+shim-server: RPC
    box pink Daemon Pod (Privileged)
    participant shim-server
    participant fusermount
    end
    Note over shim-server,fusermount: Set mnt namespace
    shim-server->>fusermount: exec
    fusermount->>fusermount: mount(/dev/fuse)
    fusermount->>shim-server: send fd
    shim-server->>fusermount-shim: return fd
    fusermount-shim->>Fuse Adapter: send fd
    Fuse Adapter->>Fuse Adapter: start working

By default, SkyPilot will use the credentials you have set up locally. For most cases, the :ref:`installation instructions <aws-installation>` are all you need to do. The steps below are **optional advanced configuration options**, aimed primarily at cloud admins and advanced users.

If you use AWS SSO and multiple clouds, check the :ref:`SSO multi-cloud compatibility notes <sso-feature-compat>`.

$ aws --version
aws-cli/2.25.6 ...

$ aws configure sso

Before using SSM with SkyPilot, ensure the following:

1. **AWS CLI v2** and **Session Manager plugin** are installed. Follow the `installation guide <https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html>`_ for your operating system.

   .. note::
      If using a :ref:`remote API server <sky-api-server>`, install the Session Manager plugin on the API server instead of your local machine.

2. **IAM permissions** are configured to allow SSM access:

   - Your user/role needs ``ssm:StartSession`` permission
   - The ``skypilot-v1`` IAM role (used by EC2 instances) needs the ``AmazonSSMManagedInstanceCore`` managed policy attached

   To attach the SSM policy to the SkyPilot role:

   a. Open the `IAM console <https://console.aws.amazon.com/iam/>`_ and go to **Roles**
   b. Search for and select the ``skypilot-v1`` role
   c. Click **Add permissions** → **Attach policies**
   d. Search for ``AmazonSSMManagedInstanceCore`` and select it
   e. Click **Add permissions**

Configuration

aws:
    use_ssm: true

SSO login has limited functionality *across multiple clouds*. If you use multiple clouds, you can :ref:`set up a dedicated IAM user and access key <dedicated-aws-user>` so that instances launched on other clouds can use AWS resources.

.. tip::

    If you are running SkyPilot on an EKS cluster and need S3 access without static credentials, see :ref:`aws-eks-iam-roles` for setting up IAM roles for EKS pods.

.. list-table::
   :header-rows: 1

   * - *Supported features:*
     - SSO credentials
     - Static credentials
   * - Use S3 buckets on an AWS cluster
     - |:white_check_mark:|
     - |:white_check_mark:|
   * - Use S3 buckets on a cluster in another cloud
     - |:x:|
     - |:white_check_mark:|
   * - Run :ref:`managed jobs <managed-jobs>` across multiple clouds
     - |:yellow_circle:| [1]_
     - |:white_check_mark:|

.. [1] To allow managed jobs to run on AWS instances, make sure your controller is also on AWS, by :ref:`specifying the controller resources <jobs-controller-custom-resources>`.


.. _several-aws-profiles:

Switching profiles or accounts
------------------------------

You can use different AWS profiles or accounts to launch different clusters. SkyPilot will remember the owner identity of each cluster and properly protects any "write" operations. All clusters are shown in ``sky status``.

Example of mixing the default profile and another profile:

.. code-block:: console

    $ # A cluster launched under the default AWS identity.
    $ sky launch --infra aws -c default

    $ # A cluster launched under a different profile.
    $ AWS_PROFILE=AdministratorAccess-12345 sky launch --infra aws -c other-profile-cluster

If you are using a :ref:`remote API server <sky-api-server>`, the AWS credentials are configured on the remote server. Overriding ``AWS_PROFILE`` on the client side won't work.


Using a specific VPC
-----------------------
By default, SkyPilot uses the "default" VPC in each region. If a region does not have a `default VPC <https://docs.aws.amazon.com/vpc/latest/userguide/work-with-default-vpc.html#create-default-vpc>`_, SkyPilot will not be able to use the region.

To instruct SkyPilot to use a specific VPC, you can use SkyPilot's global config
file ``~/.sky/config.yaml`` to specify the VPC name in the ``aws.vpc_names``
field:

.. code-block:: yaml

    aws:
      vpc_names: my-vpc-name

See details in :ref:`config-yaml`.  Example use cases include using a private VPC or a
VPC with fine-grained constraints, typically created via Terraform or manually.

To manually create a private VPC (i.e., all nodes will have internal IPs only),
you can use the AWS console; see instructions `here
<https://github.com/skypilot-org/skypilot/pull/1512>`_.


..
    These two aren't currently used, but keep them so that old links like
    /aws.html#cloud-permissions-aws will still jump to here.
.. _cloud-permissions-aws:
.. _cloud-permissions-aws-user-creation:

.. _dedicated-aws-user:

Dedicated SkyPilot IAM user
---------------------------

You can optionally create a dedicated IAM user for SkyPilot with specifically granted permissions. **Creating a dedicated user is not required** --- as long as you have AWS CLI credentials set up, SkyPilot will automatically use those credentials.

However, using a dedicated IAM user can have some benefits:

- Avoid using your personal credentials with SkyPilot.
- Specify minimal permissions to avoid granting broad access to SkyPilot.
- If you use SSO login, enable some :ref:`additional cross-cloud features <sso-feature-compat>`.

Follow these steps to create a new AWS user:

1. Open the `IAM dashboard <https://us-east-1.console.aws.amazon.com/iamv2/home#/home>`_ in the AWS console and click on the **Users** tab.

   .. image:: ../../images/screenshots/aws/aws-add-user.png
       :alt: AWS Add User


   Then, click **Create User** and enter the user's name. Click **Next**.

2. In the **Permissions options** section, select "Attach existing policies directly". Depending on whether you want simplified or minimal permissions, follow the instructions below:

   .. tab-set::

       .. tab-item:: Simplified permissions

           Search for the **AdministratorAccess** policy, and check the box to add it. Click **Next** to proceed.

           .. tip::

            To use AWS for S3 but not for launching VMs, add **AmazonS3FullAccess** policy instead.

       .. tab-item:: Minimal permissions

           Click on **Create Policy**.

           .. image:: ../../images/screenshots/aws/aws-create-policy.png
               :alt: AWS Create Policy

           This will open a new window to define the minimal policy. Follow the instructions to create a new policy: :ref:`aws-minimal-policy`.

           Come back to this window, and in the **Permissions policies** box, click on the refresh button. You can now search for the policy you just created.

           .. image:: ../../images/screenshots/aws/aws-add-policy.png
               :alt: AWS Add Policy

           Check the box to add the policy, and click **Next** to proceed.

3. Click on **Next** and follow the instructions to create the user.

4. Select the newly created user from the dashboard, and go to the **Security credentials** tab. Click on **Create access key**.

   .. image:: ../../images/screenshots/aws/aws-create-access-key.png
       :alt: AWS Create access key

5. For "Use case", select **Other**. Click **Next**, then click **Create access key**.

6. Use the newly created access key to configure your credentials with the AWS CLI:

   .. code-block:: console
     :emphasize-lines: 13-14

     $ # Configure your AWS credentials
     $ aws configure
     AWS Access Key ID [None]: <Access key>
     AWS Secret Access Key [None]: <Secret access key>
     Default region name [None]:
     Default output format [None]:

     $ # Check that AWS sees the shared-credentials-files
     $ aws configure list
           Name                    Value             Type    Location
           ----                    -----             ----    --------
        profile                <not set>             None    None
     access_key     ****************xxxx shared-credentials-file
     secret_key     ****************xxxx shared-credentials-file
         region                <not set>             None    None

     $ # Validate that credentials are working
     $ sky check aws -v


Minimal permissions
-----------------------

If you want to minimize the AWS permissions used by SkyPilot, you should set up the minimal permissions in two places:

1. :ref:`User Account <aws-create-minimal-user>`: the user account is the individual account of an user created by the administrator.
2. :ref:`IAM role `: the IAM role is assigned to all EC2 instances created by SkyPilot, which is used by the instances to access AWS resources, e.g., read/write S3 buckets or create other EC2 nodes. The IAM role is shared by all users under the same organization/root account. (If a user account has the permission to create IAM roles, SkyPilot can automatically create the role.)

.. _aws-create-minimal-user:

Create a user account with minimal permissions

.. note::
    In most cases, the IAM role will be automatically created. You only need to manually create the IAM role if you have excluded the optional role creation permissions from your minimal skypilot policy.

    If you already have an IAM role called ``skypilot-v1`` in your AWS account, it is likely created by SkyPilot automatically, and you can skip this section.

1. If you haven't yet, :ref:`create a minimal IAM policy for SkyPilot <aws-minimal-policy>`. If you previously created a dedicated IAM user with minimal permissions, you can re-use the same policy you used for the dedicated user.

2. In the `IAM dashboard <https://us-east-1.console.aws.amazon.com/iamv2/home#/home>`_, go to the "Roles" tab, and click on **Create role**.

   .. image:: ../../images/screenshots/aws/aws-add-role.png
       :alt: AWS Add Role

3. Select **Trusted entity type**: AWS service, and **Use case**: EC2, as seen in the image below.

   .. image:: ../../images/screenshots/aws/aws-add-role-entity.png
       :alt: AWS Role Entity, with "Trusted entity type" set to "AWS service", "Service or use case" set to "EC2", and "Use case" set to "EC2".

   Click **Next**.

4. Search for and select the IAM policy from step 1.
5. Click **Next**, and name your role exactly ``skypilot-v1``. Click **Create role**.


.. _aws-minimal-policy:

Minimal IAM policy
~~~~~~~~~~~~~~~~~~

To avoid giving SkyPilot administrator access, you can create a policy that limits the permissions of the account.

When creating the policy, use the **JSON** policy editor. You can copy in the minimal policy rules and additional optional policy rules.

These are the minimal policy rules required by SkyPilot:

.. note::
    **Replace the** ``<account-ID-without-hyphens>`` **with your AWS account ID**. You can find your AWS account ID by clicking on the upper right corner of the console.

.. note::
    There are **additional optional rules** below. It's likely that you will want to use some of these, so please take a look.

.. code-block:: json
    :name: aws-policy-json

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": "ec2:RunInstances",
                "Resource": "arn:aws:ec2:*::image/ami-*"
            },
            {
                "Effect": "Allow",
                "Action": "ec2:RunInstances",
                "Resource": [
                    "arn:aws:ec2:*:<account-ID-without-hyphens>:instance/*",
                    "arn:aws:ec2:*:<account-ID-without-hyphens>:network-interface/*",
                    "arn:aws:ec2:*:<account-ID-without-hyphens>:subnet/*",
                    "arn:aws:ec2:*:<account-ID-without-hyphens>:volume/*",
                    "arn:aws:ec2:*:<account-ID-without-hyphens>:security-group/*"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:TerminateInstances",
                    "ec2:DeleteTags",
                    "ec2:StartInstances",
                    "ec2:CreateTags",
                    "ec2:StopInstances"
                ],
                "Resource": "arn:aws:ec2:*:<account-ID-without-hyphens>:instance/*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateTags"
                ],
                "Resource": "arn:aws:ec2:*:<account-ID-without-hyphens>:volume/*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:Describe*"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateSecurityGroup",
                    "ec2:AuthorizeSecurityGroupIngress"
                ],
                "Resource": "arn:aws:ec2:*:<account-ID-without-hyphens>:*"
            },
            {
                "Effect": "Allow",
                "Action": "iam:CreateServiceLinkedRole",
                "Resource": "*",
                "Condition": {
                    "StringEquals": {
                        "iam:AWSServiceName": "spot.amazonaws.com"
                    }
                }
            },
            {
                "Effect": "Allow",
                "Action": [
                    "iam:GetRole",
                    "iam:PassRole"
                ],
                "Resource": [
                    "arn:aws:iam::<account-ID-without-hyphens>:role/skypilot-v1"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "iam:GetInstanceProfile"
                ],
                "Resource": "arn:aws:iam::<account-ID-without-hyphens>:instance-profile/skypilot-v1"
            }
        ]
    }

**Optional**: If you would like SkyPilot to automatically set up an IAM role and instance profile for EC2 instances, modify the last two rules in the policy with the highlighted four lines:

.. warning::

    If you don't do this, you must manually set up the IAM role that SkyPilot will use: see :ref:`iam-role-creation`.

.. code-block:: json
    :emphasize-lines: 6-7,17-18

            {
                "Effect": "Allow",
                "Action": [
                    "iam:GetRole",
                    "iam:PassRole",
                    "iam:CreateRole",
                    "iam:AttachRolePolicy"
                ],
                "Resource": [
                    "arn:aws:iam::<account-ID-without-hyphens>:role/skypilot-v1"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "iam:GetInstanceProfile",
                    "iam:CreateInstanceProfile",
                    "iam:AddRoleToInstanceProfile"
                ],
                "Resource": "arn:aws:iam::<account-ID-without-hyphens>:instance-profile/skypilot-v1"
            }

**Optional**: To enable ``sky launch --clone-disk-from``, you need to add the following permissions to the policy above as well.

.. code-block:: json

           {
                "Effect": "Allow",
                "Action": [
                    "ec2:CreateImage",
                    "ec2:CopyImage",
                    "ec2:DeregisterImage"
                ],
                "Resource": "*"
            }

**Optional**: To enable opening ports on AWS cluster, you need to add the following permissions to the policy above as well.

.. code-block:: json

           {
                "Effect": "Allow",
                "Action": [
                    "ec2:DeleteSecurityGroup",
                    "ec2:ModifyInstanceAttribute"
                ],
                "Resource": "arn:aws:ec2:*:<account-ID-without-hyphens>:*"
            }


**Optional**: If you would like to have your users access S3 buckets, you need to add the following permissions to the policy above as well.

.. code-block:: json

           {
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject"
                ],
                "Resource": "arn:aws:s3:::*/*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket",
                    "s3:GetBucketLocation"
                ],
                "Resource": "arn:aws:s3:::*"
            },
            {
                "Effect": "Allow",
                "Action": "s3:ListAllMyBuckets",
                "Resource": "*"
            }

**Optional**: If you also want to allow SkyPilot to create and delete S3 buckets (for ``sky storage`` commands), add these additional permissions:

.. code-block:: json

           {
                "Effect": "Allow",
                "Action": [
                    "s3:CreateBucket",
                    "s3:DeleteBucket",
                    "s3:PutBucketTagging"
                ],
                "Resource": "arn:aws:s3:::*"
            }

.. tip::

    If you are using EKS and want to set up S3 access with IAM roles, see :ref:`aws-eks-iam-roles`.

**Once you have added all needed policies, click Next** and follow the instructions to finish creating the policy. You can give the policy a descriptive name, such as ``minimal-skypilot-policy``.


.. _aws-troubleshooting:

Troubleshooting
---------------

If your credentials are not being picked up, or you're seeing the wrong credentials in SkyPilot, here are some steps you can take to troubleshoot:

1. **Check** ``aws configure list``. This command should show the currently configured credentials.

   If you have static credentials set up correctly, you should see something like this:

   .. code-block:: console

       $ aws configure list
             Name                    Value             Type    Location
             ----                    -----             ----    --------
          profile                <not set>             None    None
       access_key     ****************xxxx shared-credentials-file
       secret_key     ****************xxxx shared-credentials-file
           region                <not set>             None    None

   If you have SSO credentials set up correctly, you should see something like this:

   .. code-block:: console

       $ aws configure list
             Name                    Value             Type    Location
             ----                    -----             ----    --------
          profile                <not set>             None    None
       access_key     ****************xxxx              sso
       secret_key     ****************xxxx              sso
           region                <not set>             None    None

2. **Check** ``sky check aws``. This should show whether SkyPilot is picking up the credentials and has the necessary permissions.

   .. code-block:: console

       $ sky check aws -v
       Checking credentials to enable clouds for SkyPilot.
         AWS: enabled [compute, storage]
           Activated account: VRSC9IFFYQI7THCKR5UVC [account=190763068689]
       ...

Common issues
~~~~~~~~~~~~~

- **Wrong profile is enabled.** SkyPilot will respect the ``AWS_PROFILE`` environment variable if it is set; see :ref:`several-aws-profiles`. If ``AWS_PROFILE`` is not set, SkyPilot will use the profile named ``default``.

  You may have previously set ``AWS_PROFILE`` in your ``.bashrc`` file or similar. Try to double-check the value:

  .. code-block:: console
      :emphasize-lines: 13

      $ # Check the account being used by skypilot
      $ sky check aws -v
      Checking credentials to enable clouds for SkyPilot.
        AWS: enabled [compute, storage]
          Activated account: XXXXXXXXXXXXXXXXXXXXX:user [account=123456789012]
        ...
      $ # AWS account 1234-5678-9012 is enabled via @user SSO login.

      $ # See the currently enabled profile.
      $ aws configure list
            Name                    Value             Type    Location
            ----                    -----             ----    --------
         profile AWSPowerUserAccess-123456789012              env    ['AWS_DEFAULT_PROFILE', 'AWS_PROFILE']
      access_key     ****************xxxx              sso
      secret_key     ****************xxxx              sso
          region                <not set>             None    None
      $ # SSO profile AWSPowerUserAccess-123456789012 is enabled
      $ # via environment variable.

      $ # See details of the currently enabled AWS account and user/role.
      $ aws sts get-caller-identity

      $ # See if the environment variable has been set.
      $ echo $AWS_PROFILE
      AWSPowerUserAccess-123456789012

      $ unset AWS_PROFILE
      $ # Delete from .bashrc/.zshrc to make the change permanent.
      $ # Now, default profile will be used.
      $ aws configure list
            Name                    Value             Type    Location
            ----                    -----             ----    --------
         profile                <not set>             None    None
         ...
      $ sky check aws -v
      Checking credentials to enable clouds for SkyPilot.
        AWS: enabled [compute, storage]
          Activated account: XXXXXXXXXXXXXXXXXXXXX [account=987654321098]
        ...
      $ # Now AWS account 9876-5432-1098 is enabled via default profile.


- **Profile is not set**. If ``sky check aws`` and ``aws configure list`` cannot find credentials, you may not have a default profile set.

  1. If the environment variable ``AWS_PROFILE`` is set, this profile name will be used.
  2. If there is a profile named ``default``, it will be used.
  3. Otherwise, the profile will not be accessible.

  Even if there is only one profile, it will not be used unless ``AWS_PROFILE`` is set or the profile is named ``default``.

  In AWS CLI v1, you can check ``~/.aws/credentials`` and ``~/.aws/config`` to look for profile names. In AWS CLI v2, you can check from the CLI.

  .. code-block:: console

      $ # AWS CLI v2 only
      $ aws --version
      aws-cli/2.25.6 ...

      $ # List all profiles
      $ aws configure list-profiles
      AWSPowerUserAccess-xxxxxxx
      default

  If there is no ``default`` profile, you can edit the configuration directly:

  .. code-block:: cfg
      :emphasize-lines: 2

      # ~/.aws/config
      [profile default]
      sso_session = my-skypilot-session
      sso_account_id = XXXXXXXXXX
      ...

  .. code-block:: cfg
      :emphasize-lines: 2

      # ~/.aws/config
      [default]
      aws_access_key_id = XXXXXXXXXXXXXXXXXXXX
      aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

  Or, you can set the ``AWS_PROFILE`` environment variable in your shell config:

  .. code-block:: shell

      # .bashrc / .zshrc
      # Enable AWS profile named "AWSPowerUserAccess-123456789012"
      export AWS_PROFILE='AWSPowerUserAccess-123456789012'


.. toctree::
   :hidden:

   aws-eks-iam-roles

---

## File: docs/source/cloud-setup/cloud-permissions/gcp.rst

GCP
=============

.. note::

    By default, SkyPilot will pick up the credentials you have set up. For most cases, the :ref:`installation instructions ` are all you need to do. The steps below are **optional advanced configuration options**, aimed primarily at cloud admins and advanced users.


.. _gcp-service-account:

Service account
---------------

`GCP service accounts
<https://cloud.google.com/iam/docs/service-account-overview>`__ are supported.

.. tip::
    A service account on your local machine can avoid the periodic
    ``google.auth.exceptions.RefreshError: Reauthentication is needed. Please
    run `gcloud auth application-default login` to reauthenticate.`` error. A
    service account is long-lived as it does not have an expiry time.

Set up a service account as follows:

1. Follow the :ref:`instructions <gcp-service-account-creation>` to create a service account with the appropriate roles/permissions.
2. In the "Service Accounts" tab in the `IAM & Admin console
   <https://console.cloud.google.com/iam-admin/iam>`__, click on the service
   account to go to its detailed page. Click on the **KEYS** tab, then click on
   **ADD KEY** to add a JSON key.  The key will be downloaded automatically.
3. Set the environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` to the path of the key file, and configure the gcloud CLI tool:

   .. code-block:: console

       $ export GOOGLE_APPLICATION_CREDENTIALS=/path/to/key.json
       $ gcloud auth activate-service-account --key-file=$GOOGLE_APPLICATION_CREDENTIALS
       $ gcloud config set project your-project-id

   You may want to add the export statement in your profile (e.g. ``~/.bashrc``, ``~/.zshrc``) so that it is set automatically in all new terminal sessions.



.. _cloud-permissions-gcp:

Setting permissions
-------------------

Generally, the administrator can choose among three "levels" of permissions, from the most permissive and least setup effort, to the least permissive and more setup effort:

* Default: no setup, give users Owner-level permissions (i.e., you do not need to follow the instructions in this section)
* :ref:`Medium <gcp-medium-permissions>`: easy setup, with a medium set of permissions
* :ref:`Minimal <gcp-minimal-permissions>`: more setup, with the minimal set of permissions


.. _gcp-medium-permissions:

Medium permissions
~~~~~~~~~~~~~~~~~~

The easiest way to grant permissions to a user access your GCP project without the ``Owner`` role is to add the following roles to the user principals:

.. code-block:: yaml

  roles/browser
  roles/compute.admin
  roles/iam.serviceAccountAdmin
  roles/iam.serviceAccountUser
  roles/serviceusage.serviceUsageAdmin
  roles/storage.admin
  roles/iam.securityAdmin

.. note::
    If the ``roles/iam.securityAdmin`` role is undesirable, you can do the following. First, include the role and have any user (e.g., the admin) run ``sky launch --infra gcp`` successfully once. This is to create the necessary service account. Then, replace the role ``roles/iam.securityAdmin`` with ``roles/iam.roleViewer`` in the list above.


Optionally, to use TPUs, add the following role:

.. code-block:: yaml

  roles/tpu.admin

You can grant those accesses via GCP's `IAM & Admin console <https://console.cloud.google.com/iam-admin/iam>`__.

.. _gcp-minimal-permissions:

Minimal permissions
~~~~~~~~~~~~~~~~~~~

The :ref:`Medium Permissions <gcp-medium-permissions>` assigns admin permissions for some GCP services to the user.  If you would like to grant finer-grained and more minimal permissions to your users in your organization / project, you can create a custom role by following the steps below:

1. Go to GCP's `IAM & Admin console <https://console.cloud.google.com/iam-admin/roles>`__ and click on **Create Role**.

.. image:: ../../images/screenshots/gcp/create-role.png
    :width: 80%
    :align: center
    :alt: GCP Create Role

2. Give the role a descriptive name, such as ``minimal-skypilot-role``.
3. Click **Add Permissions** and search for the following permissions and add them to the role:

.. code-block:: text

    compute.disks.create
    compute.disks.list
    compute.firewalls.create
    compute.firewalls.delete
    compute.firewalls.get
    compute.instances.create
    compute.instances.delete
    compute.instances.get
    compute.instances.list
    compute.instances.setLabels
    compute.instances.setMetadata
    compute.instances.setServiceAccount
    compute.instances.start
    compute.instances.stop
    compute.networks.get
    compute.networks.list
    compute.networks.getEffectiveFirewalls
    compute.globalOperations.get
    compute.subnetworks.use
    compute.subnetworks.list
    compute.subnetworks.useExternalIp
    compute.projects.get
    compute.zoneOperations.get
    iam.roles.get
    iam.serviceAccounts.actAs
    iam.serviceAccounts.get
    serviceusage.services.enable
    serviceusage.services.list
    serviceusage.services.use
    resourcemanager.projects.get
    resourcemanager.projects.getIamPolicy

.. note::

    For custom VPC users (with :code:`gcp.vpc_name` specified in :code:`~/.sky/config.yaml`, check `here <#_gcp-bring-your-vpc>`_),  :code:`compute.firewalls.create` and :code:`compute.firewalls.delete` are not necessary unless opening ports is needed via `resources.ports` in task yaml.

.. note::

     (Advanced) To further limit the ``iam.serviceAccounts.actAs`` permission to access SkyPilot's service account only, you can remove the permission from the list above and additionally grant your organization's users the ability to use the service account ``skypilot-v1`` created by the admin (see :ref:`Service Account <gcp-service-account-creation>`). This can be done by going to ``IAM & Admin console -> Service Accounts -> skypilot-v1 -> Permissions -> GRANT ACCESS`` and adding the users with role ``roles/iam.serviceAccountUser``. This permits the users to use the ``skypilot-v1`` service account required by SkyPilot.

4. **Optional**: If the user needs to access GCS buckets, you can additionally add the following permissions:

.. code-block:: text

    storage.buckets.create
    storage.buckets.get
    storage.buckets.delete
    storage.objects.create
    storage.objects.update
    storage.objects.delete
    storage.objects.get
    storage.objects.list

5. **Optional**: If the user needs to access TPU VMs, you can additionally add the following permissions (the following may not be exhaustive, please file an issue if you find any missing permissions):

.. code-block:: text

    tpu.nodes.create
    tpu.nodes.delete
    tpu.nodes.list
    tpu.nodes.get
    tpu.nodes.update
    tpu.operations.get

6. **Optional**: To enable ``sky launch --clone-disk-from``, you need to have the following permissions for the role as well:

.. code-block:: text

    compute.disks.useReadOnly
    compute.images.create
    compute.images.get
    compute.images.delete

7. **Optional**: To enable opening ports on GCP cluster, you need to have the following permissions for the role as well:

.. code-block:: text

    compute.instances.setTags
    compute.firewalls.list
    compute.firewalls.update

8. **Optional**: If the user needs to use custom machine images with ``sky launch --image-id``, you can additionally add the following permissions:

.. code-block:: text

    compute.disks.get
    compute.disks.resize
    compute.images.get
    compute.images.useReadOnly

9. **Optional**: If your organization sets ``gcp.prioritize_reservations`` or ``gcp.specific_reservations`` in :ref:`~/.sky/config.yaml <config-yaml>`, you can additionally add the following permissions:

.. code-block:: text

    compute.reservations.list

9. Click **Create** to create the role.
10. Go back to the "IAM" tab and click on **GRANT ACCESS**.
11. Fill in the email address of the user in the “Add principals” section, and select ``minimal-skypilot-role`` in the “Assign roles” section. Click **Save**.


.. image:: ../../images/screenshots/gcp/create-iam.png
    :width: 80%
    :align: center
    :alt: GCP Grant Access

12. The user should receive an invitation to the project and should be able to setup SkyPilot by following the instructions in :ref:`Installation `.

.. note::

    The user created with the above minimal permissions will not be able to create service accounts to be assigned to SkyPilot instances.

    The admin needs to follow the :ref:`instruction below <gcp-service-account-creation>` to create a service account to be shared by all users in the project.


.. _gcp-service-account-creation:

Service account for SkyPilot system
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. note::

    If you already have an service account under "Service Accounts" tab with the email starting with ``skypilot-v1@``, it is likely created by SkyPilot automatically, and you can skip this section.

1. Click the "Service Accounts" tab in the `IAM & Admin console <https://console.cloud.google.com/iam-admin/iam>`__, and click on **CREATE SERVICE ACCOUNT**.

.. image:: ../../images/screenshots/gcp/create-service-account.png
    :width: 80%
    :align: center
    :alt: GCP Create Service Account

2. Set the service account id to ``skypilot-v1`` and click **CREATE AND CONTINUE**.

.. image:: ../../images/screenshots/gcp/service-account-name.png
    :width: 60%
    :align: center
    :alt: Set Service Account Name

3. Select the ``minimal-skypilot-role`` (or the name you set) created in the
last section and click on **DONE**. You can also choose to use the Default or
Medium Permissions roles as described in the previous sections.

.. image:: ../../images/screenshots/gcp/service-account-grant-role.png
    :width: 60%
    :align: center
    :alt: Set Service Account Role


.. _gcp-minimum-firewall-rules:

Firewall rules
~~~~~~~~~~~~~~~~~~~

By default, users do not need to set up any special firewall rules to start
using SkyPilot. If the default VPC does not satisfy the minimal required rules,
a new VPC ``skypilot-vpc`` with sufficient rules will be automatically created
and used.

However, if you manually set up and instruct SkyPilot to use a custom VPC (see
:ref:`below <gcp-bring-your-vpc>`), ensure it has the following required firewall rules:

.. code-block:: python

    # Allow internal connections between SkyPilot VMs:
    #
    #   controller -> head node of a cluster
    #   head node of a cluster <-> worker node(s) of a cluster
    #
    # NOTE: these ports are more relaxed than absolute minimum, but the
    # sourceRanges restrict the traffic to internal IPs.
    {
        "direction": "INGRESS",
        "allowed": [
            {"IPProtocol": "tcp", "ports": ["0-65535"]},
            {"IPProtocol": "udp", "ports": ["0-65535"]},
        ],
        "sourceRanges": ["10.128.0.0/9"],
    },

    # Allow SSH connections from user machine(s)
    #
    # NOTE: This can be satisfied using the following relaxed sourceRanges
    # (0.0.0.0/0), but you can customize it if you want to restrict to certain
    # known public IPs (useful when using internal VPN or proxy solutions).
    {
        "direction": "INGRESS",
        "allowed": [
            {"IPProtocol": "tcp", "ports": ["22"]},
        ],
        "sourceRanges": ["0.0.0.0/0"],
    },

You can inspect and manage firewall rules at
``https://console.cloud.google.com/net-security/firewall-manager/firewall-policies/list?project=<your-project-id>``
or using any of GCP's SDKs.

.. _gcp-bring-your-vpc:

Using a specific VPC
-----------------------
By default, SkyPilot uses the following behavior to get a VPC to use for all GCP instances:

- First, all existing VPCs in the project are checked against the minimal
  recommended firewall rules for SkyPilot to function. If any VPC satisfies these
  rules, it is used.
- Otherwise, a new VPC named ``skypilot-vpc`` is automatically created with the
  minimal recommended firewall rules and will be used. It is an auto mode VPC that
  automatically starts with one subnet per region.

To instruct SkyPilot to use a specific VPC, you can use SkyPilot's global config
file ``~/.sky/config.yaml`` to specify the VPC name in the ``gcp.vpc_name`` field:

.. code-block:: yaml

    gcp:
      vpc_name: my-vpc-name

See details in :ref:`config-yaml`.  Example use cases include using a private VPC or a
VPC with fine-grained constraints, typically created via Terraform or manually.

The custom VPC should contain the :ref:`required firewall rules <gcp-minimum-firewall-rules>`.


.. _gcp-use-internal-ips:


Using internal IPs
-----------------------
For security reason, users may only want to use internal IPs for SkyPilot instances.
To do so, you can use SkyPilot's global config file ``~/.sky/config.yaml`` to specify the ``gcp.use_internal_ips`` and ``gcp.ssh_proxy_command`` fields (to see the detailed syntax, see :ref:`config-yaml`):

.. code-block:: yaml

    gcp:
      use_internal_ips: true
      # VPC with NAT setup, see below
      vpc_name: my-vpc-name
      ssh_proxy_command: ssh -W %h:%p -o StrictHostKeyChecking=no [email protected]

The ``gcp.ssh_proxy_command`` field is optional. If SkyPilot is run on a machine that can directly access the internal IPs of the instances, it can be omitted. Otherwise, it should be set to a command that can be used to proxy SSH connections to the internal IPs of the instances.

Here are two ways to enable SkyPilot's access those private hosts:

* **Option 1** – :ref:`SSH proxy / bastion host <gcp-proxy-host>`
* **Option 2** – :ref:`IAP tunneling (Identity‑Aware Proxy) <gcp-iap-tunnel>`
  This is often simpler because you do **not** need to run or secure your own
  bastion; Google handles the tunnel at the edge.

Choose whichever fits your environment.

.. _gcp-proxy-host:

Option 1: Simple single‑proxy example
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you have a "jump host" that's directly accessible from the machine running SkyPilot, tell SkyPilot how to pipe SSH through it:

.. code-block:: yaml

    gcp:
      use_internal_ips: true
      vpc_name: my-vpc-name
      ssh_proxy_command: ssh -W %h:%p -o StrictHostKeyChecking=no [email protected]

When **multiple regions** are involved you can map each region to its own proxy
(just as before):

.. code-block:: yaml

    gcp:
      use_internal_ips: true
      vpc_name: my-vpc-name
      ssh_proxy_command:
        us-west1: ssh -W %h:%p -o StrictHostKeyChecking=no [email protected]
        us-east1: ssh -W %h:%p -o StrictHostKeyChecking=no [email protected]


.. _gcp-iap-tunnel:

Option 2: IAP tunneling
~~~~~~~~~~~~~~~~~~~~~~~
`IAP TCP forwarding <https://cloud.google.com/iap/docs/using-tcp-forwarding>`__
creates an authenticated tunnel from your workstation to any private IP inside
a GCP VPC—no public IP, VPN, or self-managed bastion required. SkyPilot
implements the `IP-based (destination-group) mode <https://cloud.google.com/iap/docs/tcp-by-host>`__; the per-instance mode is not supported yet.

Setup
^^^^^
* Enable the API once per project:

  .. code-block:: console

     $ gcloud services enable iap.googleapis.com

* This method requires **IAP tunnel permissions**.
  Adding the built-in role ``roles/iap.tunnelResourceAccessor`` on the destination
  group (or project-wide) is the simplest approach, in addition to the :ref:`other permissions required by SkyPilot <cloud-permissions-gcp>`.
* A **Cloud NAT** gateway must exist in every region where you launch
  internal-only VMs so they can reach the internet for package installs.
  See :ref:`gcp-cloud-nat`.

The following script can be used to set up IAP tunneling in a specific VPC region:

.. dropdown:: Automated setup script (per region)

    .. code-block:: bash

        #!/usr/bin/env bash
        # Set your region and VPC.
        REGION=us-east1        # e.g. us-west1, europe-west4, ...
        VPC=default            # change if using a custom VPC

        #######################################################################
        # 1. Cloud NAT (if none exists)                                       #
        #######################################################################
        nat_exists=false
        for r in $(gcloud compute routers list --regions="$REGION" \
                     --format='value(name)'); do
          if gcloud compute routers nats list --router="$r" --region="$REGION" \
                 --format='value(name)' | grep -q .; then
            nat_exists=true
          fi
        done
        if [ "$nat_exists" = false ]; then
          gcloud compute routers create nat-router --network="$VPC" --region="$REGION"
          gcloud compute routers nats create nat-config \
            --router=nat-router --router-region="$REGION" \
            --nat-all-subnet-ip-ranges --auto-allocate-nat-external-ips
        fi

        #######################################################################
        # 2. IAP destination group covering the subnet CIDR                   #
        #######################################################################
        CIDR=$(gcloud compute networks subnets list --network="$VPC" \
                 --regions="$REGION" --format='value(ipCidrRange)')
        gcloud iap tcp dest-groups create "internal-vpc-$VPC" \
            --region="$REGION" --ip-range-list="$CIDR"

        #######################################################################
        # 3. Show the SkyPilot config snippet                                 #
        #######################################################################
        echo "Update your config.yaml to add the new gcp.ssh_proxy_command.$REGION value:"
        echo
        cat <<EOF
        gcp:
          use_internal_ips: true
          ssh_proxy_command:
            $REGION: gcloud compute start-iap-tunnel %h %p --listen-on-stdin --region=$REGION --network=$VPC --dest-group=internal-vpc-$VPC
        EOF

        # Automatically do it
        read -p 'Automatically update ~/.sky/config.yaml? [y/N] ' choice
        if echo "$choice" | grep -xqE '[yY]'; then
          yq -Yi ".gcp.use_internal_ips=true | .gcp.ssh_proxy_command[\"$REGION\"]=\"gcloud compute start-iap-tunnel %h %p --listen-on-stdin --region=$REGION --network=$VPC --dest-group=internal-vpc-$VPC\"" ~/.sky/config.yaml
        else
          echo 'Aborting.'
        fi

    Paste the printed snippet into your ``~/.sky/config.yaml`` (or let the script
    modify it automatically).

Once the destination group is created, make sure to set the ``ssh_proxy_command`` :ref:`in the SkyPilot config <config-yaml-gcp-ssh-proxy-command>`. For instance, if you have

- destination group: ``internal-vpc-default``
- region: ``us-east1``
- VPC: ``default``

you could use this configuration:

.. code-block:: yaml

    gcp:
      use_internal_ips: true
      ssh_proxy_command:
        us-east1: gcloud compute start-iap-tunnel %h %p --listen-on-stdin --region=us-east1 --network=default --dest-group=internal-vpc-default


How it works
^^^^^^^^^^^^
1. ``sky launch`` creates VMs **without external IPs**.
2. When connecting to the instance, ``ssh`` uses the **IAP proxy command** shown above.
   ``gcloud compute start-iap-tunnel`` transparently
   forwards the connection through Google’s edge to the VM’s private address.
3. SSH packets travel over the authenticated IAP
   tunnel—no public ingress is ever exposed.

Troubleshooting
^^^^^^^^^^^^^^^
* *Permission denied / 403* when opening the tunnel → the caller is missing
  ``roles/iap.tunnelResourceAccessor`` (or an equivalent custom role) on the
  destination group.
* VM cannot reach the Internet → ensure the Cloud NAT gateway is in **the same
  region** as the subnet and that ``--nat-all-subnet-ip-ranges`` is enabled.
* Multiple subnets or VPCs → create one destination group per
  *region × CIDR* and map each region to the correct group in
  ``ssh_proxy_command``.


.. _gcp-cloud-nat:

Cloud NAT setup
~~~~~~~~~~~~~~~~

Instances created with internal IPs only on GCP cannot access public internet by default. To make sure SkyPilot can install the dependencies correctly on the instances,
cloud NAT needs to be setup for the VPC (see `GCP's documentation <https://cloud.google.com/nat/docs/overview>`__ for details).


Cloud NAT is a regional resource, so it will need to be created in each region that SkyPilot will be used in.


.. image:: ../../images/screenshots/gcp/cloud-nat.png
    :width: 80%
    :align: center
    :alt: GCP Cloud NAT

To limit SkyPilot to use some specific regions only, you can specify the ``gcp.ssh_proxy_command`` to be a dict mapping from region to the SSH proxy command for that region (see :ref:`config-yaml` for details):

.. code-block:: yaml

    gcp:
      use_internal_ips: true
      vpc_name: my-vpc-name
      ssh_proxy_command:
        us-west1: ssh -W %h:%p -o StrictHostKeyChecking=no [email protected]
        us-east1: ssh -W %h:%p -o StrictHostKeyChecking=no [email protected]

If proxy is not needed, but the regions need to be limited, you can set the ``gcp.ssh_proxy_command`` to be a dict mapping from region to ``null``:

.. code-block:: yaml

    gcp:
      use_internal_ips: true
      vpc_name: my-vpc-name
      ssh_proxy_command:
        us-west1: null
        us-east1: null


Force enable external IPs
~~~~~~~~~~~~~~~~~~~~~~~~~

An alternative to setting up cloud NAT for instances that need to access the public internet but are in a VPC and communicated with via their internal IP is to force them to be created with an external IP address.

.. code-block:: yaml

    gcp:
      use_internal_ips: true
      vpc_name: my-vpc-name
      force_enable_external_ips: true


Using multiple GCP projects
----------------------------------

Admins can isolate different teams to use separate GCP projects. This enables fine-grained resource tracking, billing separation, and access control per team.
For more information on configuring workspaces, see `Workspaces <https://docs.skypilot.co/en/latest/admin/workspaces.html>`__.

Prerequisites
~~~~~~~~~~~~~

The service account must have access to all GCP projects intended to be used. To grant the service account access to additional projects:

1. Follow the :ref:`instructions <gcp-service-account-creation>` to create a service account with the appropriate roles/permissions.
2. In the GCP Console, switch to the project you want to grant your service account to using the project dropdown at the top of the page, or use ``Cmd/Ctrl + O``.
3. Navigate to the `IAM & Admin console <https://console.cloud.google.com/iam-admin/iam>`__ and click on the **IAM** tab.
4. Click **GRANT ACCESS** at the top of the table:

   - In the **Add principals** field, enter your service account email (e.g., ``[email protected]``).
   - In the **Assign roles** section, select the appropriate roles (see :ref:`Setting permissions <cloud-permissions-gcp>` for recommended roles).
   - Click **SAVE**.

5. Repeat steps 2-4 for each additional project you want to enable.

Configuring workspaces with different projects

workspaces:
  team-a:
    gcp:
      project_id: main-project-id
  team-b:
    gcp:
      project_id: secondary-project-id

$ sky launch --config active_workspace=team-a --infra gcp

Setting up minimal permissions is completely **optional** and is not needed to get started with SkyPilot. The instructions in these docs are only needed for users who want to make SkyPilot use non-default, restricted permissions.

aws
gcp
nebius
vsphere
kubernetes

Service account based authentication applies only when the remote SkyPilot
cluster (including spot and serve controller) is launched inside the
Kubernetes cluster. When running outside the cluster (e.g., on AWS),
SkyPilot will use the local ``~/.kube/config`` file for authentication.

# Namespaced role for the service account
# Required for creating pods, services and other necessary resources in the namespace.
# Note these permissions only apply in the namespace where SkyPilot is deployed, and the namespace can be changed below.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-sa-role  # Can be changed if needed
  namespace: default  # Change to your namespace if using a different one.
rules:
  # Required for managing pods and their lifecycle
  - apiGroups: [ "" ]
    resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ]
    verbs: [ "*" ]
  # Required for managing services for SkyPilot Pods
  - apiGroups: [ "" ]
    resources: [ "services" ]
    verbs: [ "*" ]
  # Required for managing SSH keys
  - apiGroups: [ "" ]
    resources: [ "secrets" ]
    verbs: [ "*" ]
  # Required for retrieving reason when Pod scheduling fails.
  - apiGroups: [ "" ]
    resources: [ "events" ]
    verbs: [ "get", "list", "watch" ]
---
# ClusterRole for accessing cluster-wide resources. Details for each resource below:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-sa-cluster-role  # Can be changed if needed
  namespace: default  # Change to your namespace if using a different one.
  labels:
    parent: skypilot
rules:
  # Required for getting node resources.
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  # Required for autodetecting the runtime class of the nodes.
  - apiGroups: ["node.k8s.io"]
    resources: ["runtimeclasses"]
    verbs: ["get", "list", "watch"]
  # Required for accessing storage classes.
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]

If you are using a different namespace than ``default``, make sure to change the namespace in the above manifests.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
    name: sky-sa-cluster-role-pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list"]

If this role is not granted to the service account, ``sky gpus list`` will still work but it will only show the total GPUs on the nodes, not the number of free GPUs.

# Required only if the API server federates GPU metrics from this cluster.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
    name: sky-sa-cluster-role-namespace-reader
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    resourceNames: ["kube-system"]
    verbs: ["get"]

If this permission is not granted, GPU metrics for remote clusters still work; only a context that points back at the API server's own cluster would be misdetected as remote. The local cluster can always be referenced via the ``in-cluster`` context instead, which does not need this permission.

# Required only if using object store mounting
# Create namespace for SkyPilot system
apiVersion: v1
kind: Namespace
metadata:
  name: skypilot-system  # Do not change this
  labels:
    parent: skypilot
---
# Role for the skypilot-system namespace to create fusermount-server and
# any other system components required by SkyPilot.
# This role must be bound in the skypilot-system namespace to the service account used for SkyPilot.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: skypilot-system-service-account-role  # Can be changed if needed
  namespace: skypilot-system  # Do not change this namespace
  labels:
    parent: skypilot
rules:
  - apiGroups: [ "*" ]
    resources: [ "apps" ]
    verbs: [ "daemonsets" ]

# Required only if using ingresses
# Role for accessing ingress service IP
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: ingress-nginx  # Do not change this
  name: sky-sa-role-ingress-nginx  # Can be changed if needed
rules:
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["list", "get"]

These permissions are separate from the `Minimum Permissions Required for SkyPilot`_
described above, which apply to the service account used to *launch* clusters
(e.g., the API server SA or the user's kubeconfig). The workload SA permissions
below apply to the service account *inside* the head pod.

# Namespace role for the workload service account
# Only namespace-scoped permissions are needed (no ClusterRole required).
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-workload-sa-role  # Can be changed if needed
  namespace: default  # Change to your namespace if using a different one.
rules:
  # Required for listing and deleting pods during autodown
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "delete"]
  # Required for listing and deleting services during autodown.
  # "deletecollection" is needed for bulk service cleanup.
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["get", "list", "delete", "deletecollection"]
  # Required for checking if the cluster uses a HA deployment
  - apiGroups: ["apps"]
    resources: ["deployments"]
    verbs: ["get", "list"]
  # Required for emitting a breadcrumb event when the cluster autodowns
  # itself, which the API server reads back to record the autostop.
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create"]

# ~/.sky/config.yaml
kubernetes:
  remote_identity: sky-workload-sa  # Your custom service account name

If your workload pods need to perform additional operations beyond autodown
(e.g., launching new SkyPilot clusters from within a pod, or running
``sky`` commands inside the pod), the workload service account will need
broader permissions similar to the `Minimum Permissions Required for SkyPilot`_.

In this example, the service account is named ``sky-sa`` and is created in the ``default`` namespace.
Change the namespace and service account name as needed.

# create-sky-sa.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
  name: sky-sa  # Change to your service account name
  namespace: default  # Change to your namespace if using a different one.
  labels:
    parent: skypilot
---
# Role for the service account
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-sa-role  # Can be changed if needed
  namespace: default  # Change to your namespace if using a different one.
  labels:
    parent: skypilot
rules:
  # Required for managing pods and their lifecycle
  - apiGroups: [ "" ]
    resources: [ "pods", "pods/status", "pods/exec", "pods/portforward" ]
    verbs: [ "*" ]
  # Required for managing services for SkyPilot Pods
  - apiGroups: [ "" ]
    resources: [ "services" ]
    verbs: [ "*" ]
  # Required for managing SSH keys
  - apiGroups: [ "" ]
    resources: [ "secrets" ]
    verbs: [ "*" ]
  # Required for retrieving reason when Pod scheduling fails.
  - apiGroups: [ "" ]
    resources: [ "events" ]
    verbs: [ "get", "list", "watch" ]
---
# RoleBinding for the service account
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-sa-rb  # Can be changed if needed
  namespace: default  # Change to your namespace if using a different one.
  labels:
    parent: skypilot
subjects:
  - kind: ServiceAccount
    name: sky-sa  # Change to your service account name
roleRef:
  kind: Role
  name: sky-sa-role  # Use the same name as the role at line 14
  apiGroup: rbac.authorization.k8s.io
---
# ClusterRole for the service account
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: sky-sa-cluster-role  # Can be changed if needed
  labels:
    parent: skypilot
rules:
  - apiGroups: [""]
    resources: ["nodes"]  # Required for getting node resources.
    verbs: ["get", "list", "watch"]
  - apiGroups: ["node.k8s.io"]
    resources: ["runtimeclasses"]   # Required for autodetecting the runtime class of the nodes.
    verbs: ["get", "list", "watch"]
  - apiGroups: ["networking.k8s.io"]   # Required for exposing services through ingresses
    resources: ["ingressclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]                 # Required for `sky gpus list` command
    resources: ["pods"]
    verbs: ["get", "list"]
  - apiGroups: ["storage.k8s.io"]   # Required for using volumes
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
---
# ClusterRoleBinding for the service account
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: sky-sa-cluster-role-binding  # Can be changed if needed
  labels:
    parent: skypilot
subjects:
  - kind: ServiceAccount
    name: sky-sa  # Change to your service account name
    namespace: default  # Change to your namespace if using a different one.
roleRef:
  kind: ClusterRole
  name: sky-sa-cluster-role  # Use the same name as the cluster role at line 56
  apiGroup: rbac.authorization.k8s.io
---
# Optional: If using object store mounting, create the skypilot-system namespace
apiVersion: v1
kind: Namespace
metadata:
  name: skypilot-system  # Do not change this
  labels:
    parent: skypilot
---
# Optional: If using object store mounting, create role in the skypilot-system
# namespace to create fusermount-server.
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: skypilot-system-service-account-role  # Can be changed if needed
  namespace: skypilot-system  # Do not change this namespace
  labels:
    parent: skypilot
rules:
  - apiGroups: [ "apps" ]
    resources: [ "daemonsets" ]
    verbs: [ "*" ]
---
# Optional: If using object store mounting, create rolebinding in the skypilot-system
# namespace to create fusermount-server.
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sky-sa-skypilot-system-role-binding
  namespace: skypilot-system  # Do not change this namespace
  labels:
    parent: skypilot
subjects:
  - kind: ServiceAccount
    name: sky-sa  # Change to your service account name
    namespace: default  # Change this to the namespace where the service account is created
roleRef:
  kind: Role
  name: skypilot-system-service-account-role  # Use the same name as the role above
  apiGroup: rbac.authorization.k8s.io
---
# Optional: Role for accessing ingress resources
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: sky-sa-role-ingress-nginx  # Can be changed if needed
  namespace: ingress-nginx  # Do not change this namespace
  labels:
    parent: skypilot
rules:
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["list", "get", "watch"]
---
# Optional: RoleBinding for accessing ingress resources
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: sky-sa-rolebinding-ingress-nginx  # Can be changed if needed
  namespace: ingress-nginx  # Do not change this namespace
  labels:
    parent: skypilot
subjects:
  - kind: ServiceAccount
    name: sky-sa  # Change to your service account name
    namespace: default  # Change this to the namespace where the service account is created
roleRef:
  kind: Role
  name: sky-sa-role-ingress-nginx  # Use the same name as the role above
  apiGroup: rbac.authorization.k8s.io

$ kubectl apply -f create-sky-sa.yaml

# ~/.sky/config.yaml
kubernetes:
  remote_identity: sky-sa   # Or your service account name

The Categories and Tags is needed when using the vSphere provider, please follow bellow steps to create them.

**Create the Category:**

1. Open web browser and Login your vSphere Client.

2. Navigate to **Menu** -> **Tags & Custom Attributes**.

.. image:: ../../images/screenshots/vsphere/vsphere-catagory-create_navigate.png
    :width: 80%
    :align: center
    :alt: vSphere Catagory Creation Navigate

3. Click **Tags** -> **CATEGORIES** -> **NEW**.

.. image:: ../../images/screenshots/vsphere/vsphere-catagory-create_navigate_new.png
    :width: 80%
    :align: center
    :alt: vSphere Catagory Creation Navigate New

4. Fill in the **Create Category** form with follow content:

.. code-block:: text

    Category Name: skypilot
    Tags Per Object: Many tags
    Associable Object Types: Datastore, Content Library, Library Item, VirtualMachine

.. image:: ../../images/screenshots/vsphere/vsphere-catagory-create.png
    :width: 80%
    :align: center
    :alt: vSphere Catagory Creation

5. Click the **CREATE** to finish the creation.

**Create the Tag:**

1. Open web browser and Login your vSphere Client.

2. Navigate to **Menu** -> **Tags & Custom Attributes**.

.. image:: ../../images/screenshots/vsphere/vsphere-catagory-create_navigate.png
    :width: 80%
    :align: center
    :alt: vSphere Catagory Creation Navigate

3. Click **Tags** -> **TAGS** -> **NEW**.

.. image:: ../../images/screenshots/vsphere/vsphere-tags-create_navigate.png
    :width: 80%
    :align: center
    :alt: vSphere Catagory Creation Navigate

4. Fill in the **Create Tag** form with follow content:

.. code-block:: text

    Name: skypilot
    Category: skypilot

.. image:: ../../images/screenshots/vsphere/vsphere-tags-create.png
    :width: 80%
    :align: center
    :alt: vSphere Tags Creation

5. Click the **CREATE** to finish the creation.

.. _cloud-prepare-vsphere-storage-policy:

Create VM storage policies

The VM must be Linux-based; we take Ubuntu 20.04 as the base OS in this document. Other Linux distributions may also works but not promised.

**Step 1. Prepare a linux-based VM**.

Open web browser and Login your vSphere Client, create a linux-based Virtual Machine.

If you're unfamiliar with the process of creating a virtual machine, please refer to this guide for assistance: `Deploying Virtual Machines <https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-39D19B2B-A11C-42AE-AC80-DDA8682AB42C.html>`_

**Step 2. Create a user and Enable password-less sudo**.

Login your prepared VM, Create a user named ``ubuntu``, and add to the sudo group.

.. code-block:: shell

    sudo adduser ubuntu
    sudo usermod -aG sudo ubuntu

Edit the sudoer file to enable passwordless sudo.

.. code-block:: shell

    sudo visudo

Add the following line:

.. code-block:: text

    ubuntu   ALL=(ALL:ALL) NOPASSWD: ALL

**Step 3. Check VMtools version**.

Execute the following command to check VMtools version 10.1.0 or later is required.

.. code-block:: shell

    vmware-toolbox-cmd -v


**Step 4. Enabling Custom Scripts in VMtools**.

Execute the following command to check if the enable-custom-scripts option is enabled:

.. code-block:: shell

    sudo vmware-toolbox-cmd config get deployPkg enable-custom-scripts

If the enable-custom-scripts option is disabled, enable it by executing:

.. code-block:: shell

    sudo vmware-toolbox-cmd config set deployPkg enable-custom-scripts true

**Step 5. Install the GPU driver**.

Different GPUs require different drivers; please choose the correct driver based on your GPU mode and OS version.
Take Ubuntu and Nvidia GPU for example, the following driver works for most of the case:

.. code-block:: shell

    sudo apt install nvidia-headless-535-server --no-install-recommends

**Step 6. Convert the VM to template**.

1. Power off the VM.

2. Select the VM and right click -> **Clone**-> **Clone as Template to Library**.

.. image:: ../../images/screenshots/vsphere/vm-clone-to-template.png
    :width: 80%
    :align: center
    :alt: VM Clone To Temple

3. Select Template type: **OVF**, provide the template name and click **Next**.

.. image:: ../../images/screenshots/vsphere/vm-clone-to-template-ovf.png
    :width: 80%
    :align: center
    :alt: VM Clone To Temple OVF

4. Select a Content Library location and click **Next**.

   Create one if you haven't set up a **Local** content library. Please refer to `Create a Library <https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.vm_admin.doc/GUID-2A0F1C13-7336-45CE-B211-610D39A6E1F4.html>`_

.. image:: ../../images/screenshots/vsphere/vm-clone-to-template-cl.png
    :width: 80%
    :align: center
    :alt: VM Clone To Temple Content Lib

5. Review and click **Finish**.

**Step 7. Tag the VM template**.

Follow the steps bellow to tag the VM template:

1. Navigate to **Menu** -> **Content Libraries**.

.. image:: ../../images/screenshots/vsphere/content-libs-navigate.png
    :width: 80%
    :align: center
    :alt: Content Libs

2. Review the content libraries and click on the one that contains your target VM templates.

.. image:: ../../images/screenshots/vsphere/content-libs.png
    :width: 80%
    :align: center
    :alt: Content Libs

3. Review the content library items and click on your target VM template.

.. image:: ../../images/screenshots/vsphere/content-lib-item.png
    :width: 80%
    :align: center
    :alt: Content Lib Item

4. Assign the relevant tags to the VM template.

   If you no sure how to assign the tags, please refer to :ref:`Rules for tagging VM template <cloud-prepare-vsphere-references>`

.. image:: ../../images/screenshots/vsphere/content-lib-item-tag-adding.png
    :width: 80%
    :align: center
    :alt: Content Lib Item Tagging

.. _cloud-prepare-vsphere-references:

References
~~~~~~~~~~

**Rules for tagging VM template**

The vSphere driver for SkyPilot uses the Tag to identify the VM images for GPU. VI admin needs to tag the VM template accordingly in the Content Library.
Here are a few rules for tagging templates:

- All workloads that do not need an accelerator will use the VM image tagged with ``skypilot-cpu``. If you have multiple VM templates (Content Item) tagged with ``skypilot-cpu`` only the first one will be used, so the best practice is only tagging one image with ``skypilot-cpu``.

- Vi Admin should create a default VM Image for each accelerator vendor and tag it with the format ``skypilot-vendorname``  e.g., ``skypilot-nvidia``. The fallback logic is if the user requires a specific Nvidia GPU(e.g., k1200) and the vSphere driver cannot find an image with tag ``skypilot-K1200`` it will use the VM image tagged with ``skypilot-nvidia``.

- All the tags should select the Tag category as: ``skypilot``.

**Support new GPU**

The vSphere cloud provider has a default support GPU list. You can first check the default one by executing the ``sky check`` command.
Then the GPU list can be found in the ``~/.sky/catalogs/v5/vsphere/accelerators.csv`` file. If you want to support a new GPU, say ``Nvidia K1200``
you can add one more line at the end of the file:

.. code-block:: text

    Model,Type,MemoryMB,vCPUs,fullNames
    ...
    K1200,GPU,4096,4,['GM107GL [Quadro K1200]']

If the GPU needs a particular driver, ask the VI admin to create a new VM template and upload it to the Content Library, then add a tag to the VM template, e.g., ``GPU-k1200``. If the new default VM template's driver supports the new GPU, you will not need to create a new one. For example, if the VM template with the Tag ``skypilot-nvidia`` supports the new GPU, then you will not need to create a new VM template.

---

## File: docs/source/cloud-setup/policy.rst

.. _advanced-policy-config:

Admin Policy Enforcement
========================


SkyPilot provides an **admin policy** mechanism that admins can use to enforce certain policies on users' SkyPilot usage. An admin policy applies
custom validation and mutation logic to a user's tasks and SkyPilot config.

Example usage:

- :ref:`kubernetes-labels-policy`
- :ref:`disable-public-ip-policy`
- :ref:`use-spot-for-gpu-policy`
- :ref:`enforce-autostop-policy`
- :ref:`dynamic-kubernetes-contexts-update-policy`
- :ref:`use-local-gcp-credentials-policy`
- :ref:`add-volumes-policy`
- :ref:`reject-old-clients-policy`
- :ref:`slurm-partition-routing-policy`
- :ref:`slurm-filesystem-routing-policy`

Overview
--------

SkyPilot has a :ref:`client-server architecture <sky-api-server>`, where a centralized API server can be deployed and users can interact with the server through a client.

To deploy a admin policy, here are the steps:

1. :ref:`Implement the policy in a Python package ` or :ref:`host the policy as a RESTful server <host-admin-policy-as-server>`.
2. Install the policy at the :ref:`server-side <server-side-admin-policy>` to enforce it for all users.
3. Optionally, an admin policy can also be installed at the :ref:`client-side<client-side-admin-policy>` if it needs to access user's local environment.

The order of policy application is demonstrated below:

.. figure:: ../images/admin-policy.png
    :align: center
    :width: 80%
    :alt: Admin policy application order

.. note::

    Client-side policy lacks enforcement capability, i.e., end-user may modify them. However, client policy is useful for automation that can only be applied at client-side. Refer to :ref:`use-local-gcp-credentials-policy` as an example.

Quickstart
----------

Install the example policy package:

.. code-block:: bash

    git clone https://github.com/skypilot-org/skypilot.git
    cd skypilot
    pip install examples/admin_policy/example_policy

Then, set the :ref:`admin_policy <config-yaml-admin-policy>` field in :ref:`the SkyPilot config <config-yaml>` to use an example policy.

.. code-block:: yaml

    admin_policy: example_policy.DoNothingPolicy


.. tip::

   You can replace ``DoNothingPolicy`` with any of the :ref:`example policies <example-policies>`.


Then, run a task:

.. code-block:: bash

    $ sky launch

You should see the admin policy is applied to the task.


.. code-block:: bash

    $ sky launch
    Applying client admin policy: DoNothingPolicy
    Applying server admin policy: DoNothingPolicy
    ...


You can also deploy an admin policy as a :ref:`RESTful server <host-admin-policy-as-server>`.

Deploy an admin policy
----------------------

.. _server-side-admin-policy:

Server-side
~~~~~~~~~~~

If you have a :ref:`centralized API server <sky-api-server>` deployed, you can enforce a policy for all users by setting it at the server-side.

.. tab-set::

  .. tab-item:: Use a RESTful policy

    Open :ref:`SkyPilot dashboard <sky-api-server-config>` https://api.server.com/dashboard/config,  and set the :ref:`admin_policy <config-yaml-admin-policy>` field to the URL of the RESTful policy. To host a RESTful policy, see :ref:`here <host-admin-policy-as-server>`.

    .. code-block:: yaml

        admin_policy: https://example.com/policy

  .. tab-item:: Use a Python package

    First, install the Python package that implements the policy on the API server host:

    .. code-block:: bash

        pip install mypackage.subpackage

    For helm deployment, refer to :ref:`sky-api-server-admin-policy` to install the policy package.

    Then, open the server's dashboard, go to :ref:`the server's SkyPilot config <sky-api-server-config>` and set the :ref:`admin_policy <config-yaml-admin-policy>` field to the path of the Python package that implements the policy.

    .. code-block:: yaml

        admin_policy: mypackage.subpackage.MyPolicy

.. _client-side-admin-policy:

Client-side
~~~~~~~~~~~

If the policy needs to access user's local environment, you can get the policy applied at the client-side by following the steps below.

First, install the Python package that implements the policy:

.. code-block:: bash

    pip install mypackage.subpackage

Then, set the :ref:`admin_policy <config-yaml-admin-policy>` field in :ref:`the SkyPilot config <config-yaml>` to the path of the Python package that implements the policy.

.. code-block:: yaml

    admin_policy: mypackage.subpackage.MyPolicy

Optionally, you can also apply different policies in different projects by leveraging the :ref:`layered config <config-sources-and-overrides>`, e.g. set a different policy in ``$pwd/.sky.yaml`` for the current project:

.. code-block:: yaml

    admin_policy: mypackage.subpackage.AnotherPolicy

.. hint::

    SkyPilot loads the policy from the given package in the same Python environment.
    You can test the existence of the policy by running:

    .. code-block:: bash

        python -c "from mypackage.subpackage import MyPolicy"

.. note::

    It is possible to call a RESTful policy at client-side. However, a RESTful policy is executed on the policy server host, i.e., cannot access user's local environment, e.g., local files.

.. _host-admin-policy-as-server:

Host admin policy as a RESTful server

.. literalinclude:: ../../../examples/admin_policy/example_server/policy_server.py
    :language: python
    :caption: `Policy Server <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_server/policy_server.py>`_

**POST /<api-path>**

Request body is a marshalled :ref:`sky.UserRequest ` in JSON format:

.. code-block:: json

    {
      "task": {
        "name": "sky-cmd",
        "resources": {
          "cpus": "1+",
        },
        "num_nodes": 1,
      },
      "skypilot_config": {},
      "request_options": {
        "cluster_name": "test",
        "idle_minutes_to_autostop": null,
        "down": false,
        "dryrun": false
      },
      "at_client_side": false
    }

Response body is a marshalled :ref:`sky.MutatedUserRequest <mutated-user-request-class>` in JSON format:

.. code-block:: json

    {
      "task": {
        "name": "sky-cmd",
        "resources": {
          "cpus": "1+",
        },
        "num_nodes": 1,
      },
      "skypilot_config": {}
    }

.. literalinclude:: ../../../sky/admin_policy.py
    :language: python
    :pyobject: AdminPolicy
    :name: admin-policy-interface
    :caption: `AdminPolicy Interface <https://github.com/skypilot-org/skypilot/blob/master/sky/admin_policy.py>`_

import sky

class MyPolicy(sky.AdminPolicy):
    @classmethod
    def validate_and_mutate(cls, user_request: sky.UserRequest) -> sky.MutatedUserRequest:
        # Logic for validate and modify user requests.
        ...
        return sky.MutatedUserRequest(user_request.task,
                                      user_request.skypilot_config)

.. literalinclude:: ../../../sky/admin_policy.py
    :language: python
    :pyobject: UserRequest
    :name: user-request-class
    :caption: `UserRequest Class <https://github.com/skypilot-org/skypilot/blob/master/sky/admin_policy.py>`_

.. literalinclude:: ../../../sky/admin_policy.py
    :language: python
    :pyobject: MutatedUserRequest
    :name: mutated-user-request-class
    :caption: `MutatedUserRequest Class <https://github.com/skypilot-org/skypilot/blob/master/sky/admin_policy.py>`_

.. literalinclude:: ../../../sky/utils/config_utils.py
    :language: python
    :pyobject: Config
    :caption: `Config Class <https://github.com/skypilot-org/skypilot/blob/master/sky/utils/config_utils.py>`_

.. literalinclude:: ../../../sky/admin_policy.py
    :language: python
    :pyobject: RequestOptions
    :caption: `RequestOptions Class <https://github.com/skypilot-org/skypilot/blob/master/sky/admin_policy.py>`_

Use ``UserRequest.user`` to access the user who made the request.
This field is only available when the policy is applied at the server-side.

Useful for:

- Logging the user who made a specific request.
- Implementing per-user quotas or rate limits.
- Changing the behavior of the policy based on the user.

Access request name
~~~~~~~~~~~~~~~~~~~

Use ``UserRequest.request_name`` to access the request name.

Useful for:

- Activating a policy selectively for certain request types.
- Implementing per-request rate limits.
- Changing the behavior of the policy based on the request name.

Access client version information

For very old clients that don't send version headers (pre-v0.11.0), ``client_api_version`` will be ``None`` and ``client_version`` will be ``"unknown"``.

Use ``UserRequest.task.get_resource_config()`` to get the resource configuration of a task.

The resource configuration is a dictionary conforming to the :ref:`resource config schema <yaml-spec>`.

Once the resource configuration is modified, you can use
``UserRequest.task.set_resources(resource_config)``
to set the modified resource configuration back to the task.

.. code-block:: python

    resource_config = user_request.task.get_resource_config()
    resource_config['use_spot'] = True
    if 'any_of' in resource_config:
        for any_of_config in resource_config['any_of']:
            any_of_config['use_spot'] = True
    elif 'ordered' in resource_config:
        for ordered_config in resource_config['ordered']:
            ordered_config['use_spot'] = True
    user_request.task.set_resources(resource_config)

Useful for:

- Enforcing resource constraints (e.g. use spot instances for all GPU tasks, enforce autostop for all tasks).

.. _example-policies:

Example policies
----------------

We have provided a few example policies in `examples/admin_policy/example_policy <https://github.com/skypilot-org/skypilot/tree/master/examples/admin_policy/example_policy>`_. You can test these policies by installing the example policy package in your Python environment.

.. code-block:: bash

    git clone https://github.com/skypilot-org/skypilot.git
    cd skypilot
    pip install examples/admin_policy/example_policy

Reject all tasks
~~~~~~~~~~~~~~~~

.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
    :language: python
    :pyobject: RejectAllPolicy
    :caption: `RejectAllPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

.. literalinclude:: ../../../examples/admin_policy/reject_all.yaml
    :language: yaml
    :caption: `Config YAML for using RejectAllPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/reject_all.yaml>`_

.. _kubernetes-labels-policy:

Add labels for all tasks on Kubernetes

.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
    :language: python
    :pyobject: DisablePublicIpPolicy
    :caption: `DisablePublicIpPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

.. literalinclude:: ../../../examples/admin_policy/disable_public_ip.yaml
    :language: yaml
    :caption: `Config YAML for using DisablePublicIpPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/disable_public_ip.yaml>`_

.. _use-spot-for-gpu-policy:

Use spot for all GPU tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~

..
.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
    :language: python
    :pyobject: UseSpotForGpuPolicy
    :caption: `UseSpotForGpuPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

.. literalinclude:: ../../../examples/admin_policy/use_spot_for_gpu.yaml
    :language: yaml
    :caption: `Config YAML for using UseSpotForGpuPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/use_spot_for_gpu.yaml>`_

.. _enforce-autostop-policy:

Enforce autostop for all tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
    :language: python
    :pyobject: EnforceAutostopPolicy
    :caption: `EnforceAutostopPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

.. literalinclude:: ../../../examples/admin_policy/enforce_autostop.yaml
    :language: yaml
    :caption: `Config YAML for using EnforceAutostopPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/enforce_autostop.yaml>`_


Set max autostop idle minutes for all tasks

.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
    :language: python
    :pyobject: DynamicKubernetesContextsUpdatePolicy
    :caption: `DynamicKubernetesContextsUpdatePolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

.. literalinclude:: ../../../examples/admin_policy/dynamic_kubernetes_contexts_update.yaml
    :language: yaml
    :caption: `Config YAML for using DynamicKubernetesContextsUpdatePolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/dynamic_kubernetes_contexts_update.yaml>`_

.. _use-local-gcp-credentials-policy:

Use local GCP credentials for all tasks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/client_policy.py
    :language: python
    :pyobject: UseLocalGcpCredentialsPolicy
    :caption: `UseLocalGcpCredentialsPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/client_policy.py>`_

Specify the following config in :ref:`the SkyPilot config <config-yaml>` at the client:

.. literalinclude:: ../../../examples/admin_policy/use_local_gcp_credentials.yaml
    :language: yaml
    :caption: `Config YAML for using UseLocalGcpCredentialsPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/use_local_gcp_credentials.yaml>`_

Then specify the policy at the server with the same config, or call this policy in the :ref:`RESTful policy server <host-admin-policy-as-server>`.

    This policy only take effects when applied at :ref:`client-side <client-side-admin-policy>`. Use this policy at the :ref:`server-side <server-side-admin-policy>` will be a no-op.

.. _add-volumes-policy:

Add volumes to all tasks
~~~~~~~~~~~~~~~~~~~~~~~~

.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
    :language: python
    :pyobject: AddVolumesPolicy
    :caption: `AddVolumesPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

.. literalinclude:: ../../../examples/admin_policy/add_volumes.yaml
    :language: yaml
    :caption: `Config YAML for using AddVolumesPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/add_volumes.yaml>`_


.. _rate-limit-launch-policy:

Rate limit cluster launch requests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
    :language: python
    :pyobject: RateLimitLaunchPolicy
    :caption: `RateLimitLaunchPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

The ``RateLimitLaunchPolicy`` uses ``TokenBucketRateLimiter`` class to rate limit cluster launch requests.

.. dropdown:: TokenBucketRateLimiter Class

    .. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
        :language: python
        :pyobject: TokenBucketRateLimiter
        :caption: `TokenBucketRateLimiter <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

.. literalinclude:: ../../../examples/admin_policy/rate_limit_launch.yaml
    :language: yaml
    :caption: `Config YAML for using RateLimitLaunchPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/rate_limit_launch.yaml>`_

.. _gpu-static-quota-policy:

Enforce a static GPU quota for each user on cluster launch requests

This policy calls sky.status() to get the total number
of GPUs currently used by the user and therefore adds a
few seconds of latency to every cluster launch request.

This policy should be considered an educational example and not a
production-ready policy.

This policy demonstrates how to use client version information to enforce minimum client versions. It rejects requests from clients with an API version below a specified threshold.

.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
    :language: python
    :pyobject: RejectOldClientsPolicy
    :caption: `RejectOldClientsPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

.. literalinclude:: ../../../examples/admin_policy/reject_old_clients.yaml
    :language: yaml
    :caption: `Config YAML for using RejectOldClientsPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/reject_old_clients.yaml>`_

.. _slurm-partition-routing-policy:

Route Slurm jobs to partitions based on resources

This policy routes Slurm jobs only to clusters where the required filesystem
paths are accessible. Users declare required paths via the
``SKYPILOT_REQUIRED_FILESYSTEMS`` environment variable in their task
(comma-separated). The policy connects to each candidate cluster over SSH
at optimization time and checks that all required paths exist. Results are
cached per cluster to avoid repeated SSH calls.

This is useful when the same dataset is stored at different paths on
different clusters (e.g. ``/mnt/data/MNIST`` on one cluster and
``/home/$USER/data/MNIST`` on another).

Users can set the environment variable via the ``envs`` field in a :ref:`task YAML <yaml-spec>`:

.. code-block:: yaml

   envs:
     SKYPILOT_REQUIRED_FILESYSTEMS: /mnt/home,/mnt/data

   resources:
     cloud: slurm

Or via the ``--env`` flag in ``sky launch`` :ref:`CLI <cli>`:

.. code-block:: console

   $ sky launch --infra slurm --env SKYPILOT_REQUIRED_FILESYSTEMS=/home/ubuntu

.. literalinclude:: ../../../examples/admin_policy/example_policy/example_policy/skypilot_policy.py
    :language: python
    :pyobject: SlurmFilesystemRoutingPolicy
    :caption: `SlurmFilesystemRoutingPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/example_policy/example_policy/skypilot_policy.py>`_

.. literalinclude:: ../../../examples/admin_policy/slurm_filesystem_routing.yaml
    :language: yaml
    :caption: `Config YAML for using SlurmFilesystemRoutingPolicy <https://github.com/skypilot-org/skypilot/blob/master/examples/admin_policy/slurm_filesystem_routing.yaml>`_

---

## File: docs/source/cloud-setup/quota.rst

.. _quota:

Requesting Quota Increase
=========================


Most cloud providers enforce a quota policy to limit the number of VM instances that can exist in a given region.
Users may encounter `QuotaExceeded` or `VcpuLimitExceeded` errors during resources provisioning, especially for high end GPUs such as H100/A100.
To check or increase your quota limits, please follow the below instructions.
After submitting the request, it will usually take a few days for the support team to review.
To increase chances of being approved, you may respond their inquiry emails on how the requested resources will be used your projects.

.. _quota-aws:

AWS
---

1. Go to the `EC2 Quotas console <https://console.aws.amazon.com/servicequotas/home/services/ec2/quotas>`_.
2. **Select a region** on the top right.
3. Choose an EC2 instance type from the list (e.g, ``Running On-Demand P instances`` or ``All P Spot Instance Requests``). Use ``sky gpus list --infra aws --all`` or check `here <https://aws.amazon.com/ec2/instance-types/>`__ for more instance types.
4. Click the quota name, and then choose **Request quota increase**.
5. For **Change quota value**, enter the new value.
6. Choose **Request**.

Azure
-----

1. First go to `Azure's quota page <https://portal.azure.com/#blade/Microsoft_Azure_Capacity/QuotaMenuBlade/myQuotas>`_.
2. Select **Request Increase** near the top of the screen.
3. For Quota type, select ``Compute-VM (cores-vCPUs) subscription limit increases``. Hint: note that a message "Get more quota now. You don’t need a support ticket to get more quota..." may pop up; feel free to skip it as requesting quotas for most GPU instances still requires creating support tickets (next steps).
4. Select **Next** to go to the Additional details screen, then select **Enter details**.

  - In the Quota details screen:
  - For Deployment model, ensure **Resource Manager** is selected.
  - For Locations, select all regions in which you want to increase quotas.
  - For each region you selected, select one or more VM series from the Quotas drop-down list.
  - For each VM Series you selected (e.g., ``ND_H100_v5`` for H100 instances), enter the new vCPU limit that you want for this subscription. You may check `for more VM Series <https://docs.microsoft.com/en-us/azure/virtual-machines/sizes-gpu>`_.
  - When you're finished, select **Save and continue**.

5. Enter or confirm your contact details, then select **Next**.
6. Finally, ensure that everything looks correct on the Review + create page, then select **Create** to submit your request.

GCP
---

1. In the Google Cloud Console, go to the `Quota page <https://console.cloud.google.com/iam-admin/quotas/>`_.
2. Click **Filter** and select ``Service: Compute Engine API``.
3. For H100 GPUs: choose ``metric: GPUS_PER_GPU_FAMILY`` and select dimension ``gpu_family: NVIDIA_H100``.
4. For all other GPUs: choose ``Limit Name: instance_name``. (e.g., ``NVIDIA-V100-GPUS-per-project-region``). You may check the `the compute GPU list <https://cloud.google.com/compute/quotas#gpu_quota>`_.
5. Select the checkbox of the region whose quota you want to change.
6. Click **Edit Quotas** and fill out the new limit.
7. Click **Submit Request**.

OCI
-------------------------------

1. Go to the `OCI Limits, Quotas and Usage console <https://cloud.oracle.com/limits>`_ to check your current resources status.
2. Click the **request a service limit increase** link on the page if you want to increase quotas.
3. Choose a **Service Category** from the list (e.g, ``Compute``).
4. Choose a **Resource** from the list (e.g, ``GPUs for GPU.A10 based VM and BM Instances``). Use ``sky gpus list --infra oci --all`` or check `here <https://docs.oracle.com/en-us/iaas/Content/Compute/References/computeshapes.htm>`__ for more instance types.
5. Enter the **Limit** field for your new limit and **Reason for request** for justification.
6. Click **Create Support Request** to submit.
7. You may check `OCI Service Limits <https://docs.oracle.com/en-us/iaas/Content/General/Concepts/servicelimits.htm#computelimits>`_ for more information.