{"owner":"skypilot-org","repo":"skypilot","hasSkills":true,"totalSkillsCount":16,"totalTokensCount":67765,"categories":["marketplace","anthropic-skill","plugin-manifest","root-instruction","claude-rule","subagent-persona"],"hasMcp":false,"mcpConfig":null,"found":[".claude-plugin/marketplace.json","agent/skills/skypilot/SKILL.md","agent/.claude-plugin/plugin.json","AGENTS.md","agent/skills/skypilot/evals/evals.json","agent/skills/skypilot/references/advanced-patterns.md","CLAUDE.md","agent/skills/skypilot/references/examples.md","agent/skills/skypilot/references/python-sdk.md","agent/skills/skypilot/references/job-investigation.md","agent/skills/skypilot/references/cli-reference.md","agent/skills/skypilot/references/migrating-from-slurm.md","agent/skills/skypilot/references/troubleshooting.md","agent/skills/skypilot/references/yaml-spec.md","docs/source/examples/agents/autonomous-code-optimization.md","docs/source/examples/agents/autoresearch.md"],"skills":{".claude-plugin/marketplace.json":"{\n \"name\": \"skypilot\",\n \"owner\": {\n \"name\": \"SkyPilot Team\"\n },\n \"metadata\": {\n \"description\": \"Official SkyPilot skills for AI coding agents\",\n \"version\": \"1.0.0\"\n },\n \"plugins\": [\n {\n \"name\": \"skypilot\",\n \"source\": \"./agent\",\n \"description\": \"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds\",\n \"version\": \"1.0.0\",\n \"author\": {\n \"name\": \"SkyPilot Team\"\n },\n \"homepage\": \"https://docs.skypilot.co/\",\n \"repository\": \"https://github.com/skypilot-org/skypilot\",\n \"license\": \"Apache-2.0\",\n \"keywords\": [\"AI Infrastructure\", \"Multi-Cloud\", \"GPU\", \"Kubernetes\", \"Slurm\", \"Training\", \"Serving\", \"Cost Optimization\", \"SkyPilot\"]\n }\n ]\n}\n","agent/skills/skypilot/SKILL.md":"---\nname: skypilot\ndescription: \"Use when launching cloud VMs, Kubernetes pods, or Slurm jobs for GPU/TPU/CPU workloads, training or fine-tuning models on cloud GPUs, deploying inference servers (vllm, TGI, etc.) with autoscaling, writing or debugging SkyPilot task YAML files, using spot/preemptible instances for cost savings, comparing GPU prices across clouds, managing compute across 25+ clouds, Kubernetes, Slurm, and on-prem clusters with failover between them, troubleshooting resource availability or SkyPilot errors, optimizing cost and GPU availability, or migrating an existing Slurm workload (converting sbatch scripts, salloc sessions, job arrays or srun invocations into SkyPilot task YAMLs, and mapping Slurm commands, directives and SLURM_* environment variables to their SkyPilot equivalents).\"\n---\n\n# SkyPilot Skill\n\nSkyPilot is a unified framework to run AI workloads on any cloud, Slurm or Kubernetes. It provides a single interface to launch clusters, run jobs, and serve models across 25+ clouds (AWS, GCP, Azure, Coreweave, Nebius, Lambda, Together AI, RunPod, and more), Kubernetes clusters, and Slurm clusters.\n\n## When to Use SkyPilot\n\n**Use SkyPilot when you need to:**\n- Manage compute resources on any cloud, Slurm, or Kubernetes cluster\n- Launch CPU/GPU/TPU (GB300, GB200, B200, H200, H100, etc.) on any cloud, Kubernetes or Slurm\n- Run training, fine-tuning, or batch inference jobs\n- Serve models with autoscaling and multi-cloud replicas (SkyServe)\n- Run long-running jobs with automatic lifecycle management and recovery (managed jobs)\n- Find the cheapest or most available GPU across clouds\n\n**Don't use SkyPilot for:**\n- Local-only workloads (use Docker/conda directly)\n\n## Capabilities: When to Use What\n\nSkyPilot has three core abstractions. Use the right one for each stage of your workflow:\n\n**1. SkyPilot Clusters** (`sky launch` / `sky exec`) — Interactive development and debugging\n- Use during initial development, debugging, and experimentation\n- Launch a cluster, SSH in or connect VSCode/Cursor (`code --remote ssh-remote+CLUSTER`), iterate quickly\n- Cluster stays up until you stop/down it or autostop triggers\n- Best for: prototyping, debugging, short experiments\n\n**2. Managed Jobs** (`sky jobs launch`) — Long-running training and batch jobs\n- Use when submitting long-running jobs that should run unattended\n- Manages the full lifecycle: provisioning, execution, recovery, and teardown\n- Automatically recovers from spot preemptions, quota limits, and transient failures\n- Works across clouds, Kubernetes, and Slurm (handles preemptions and quota)\n- Best for: training runs, fine-tuning, hyperparameter sweeps, batch inference\n\n**3. SkyServe** (`sky serve up`) — Production model serving\n- Use when serving models at scale with autoscaling\n- Start with `sky launch` + open port to test your serving setup, then use `sky serve up` to scale\n- Provides load balancing, autoscaling, and multi-cloud replicas\n- Best for: model serving endpoints, API services\n\n## Before You Start (Agent Bootstrap)\n\nBootstrap to confirm SkyPilot is installed, connected to an API server, and has cloud credentials. Once confirmed, skip straight to the user's task.\n\n**Step 1: Check installation and API server connectivity**\n\n```bash\nsky api info\n```\n| Output contains | Meaning | Next action |\n|-----------------|---------|-------------|\n| Server version and status | Server is running and connected | **Bootstrap done.** Skip to user's task. |\n| `No SkyPilot API server is connected` | No server connected | Go to \"Start or connect a server\" below. |\n| `Could not connect to SkyPilot API server` | Remote server unreachable or auth expired | Tell the user and suggest `sky api login --relogin -e ` to reconnect. |\n| `command not found: sky` | SkyPilot not installed | Go to \"Install SkyPilot\" below. |\n\n**Install SkyPilot** (only if `sky` command not found):\n```bash\npip install \"skypilot[aws,gcp,kubernetes]\" # Pick clouds the user needs\n```\nAsk the user which clouds they need if unclear, then re-run `sky api info`.\n\n**Start or connect a server** (only if \"not running\"):\n\nAsk the user:\n> Do you have an existing SkyPilot API server to connect to, or should I start one locally?\n\n- **Connect to existing server:** `sky api login -e ` — get the URL from the user.\n- **Start locally:** `sky api start`\n\nAfter either path, re-run `sky api info` to confirm the server is reachable.\n\n**Step 2: Check cloud credentials** (only for fresh setups — skip if the server was already running)\n```bash\nsky check -o json\n```\nThis shows which clouds are enabled or disabled. If the user's target cloud is not enabled, guide them through credential setup (see [Troubleshooting](references/troubleshooting.md#1-installation-and-credentials)).\n\n## Essential Commands\n\nUse `-o json` with status/query commands to get structured JSON output instead of tables.\n\n**Clusters** — interactive development and debugging:\n\n| Command | Description |\n|---------|-------------|\n| `sky launch -c NAME task.yaml` | Launch a cluster or run a task |\n| `sky exec NAME task.yaml` | Run task on existing cluster (skips provisioning); syncs workdir each time |\n| `sky exec NAME task.yaml -d` | Same, but detach immediately (don't stream logs) |\n| `sky status -o json` | Show all clusters |\n| `sky logs NAME` | Stream job logs from a cluster |\n| `sky logs NAME --no-follow` | Print existing logs and exit immediately |\n| `sky logs NAME --tail 50` | Print last 50 lines of logs and exit |\n| `sky logs NAME --status` | Exit with code 0=succeeded, 100=failed, 101=not finished, 102=not found, 103=cancelled |\n| `sky queue NAME -o json` | List jobs on a cluster with status (structured JSON) |\n| `sky stop NAME` / `sky start NAME` | Stop/restart to save costs (preserves disk) |\n| `sky down NAME` | Tear down a cluster completely |\n| `sky gpus list -o json` | List available GPU types across clouds |\n\n**Managed Jobs** — long-running unattended workloads:\n\n| Command | Description |\n|---------|-------------|\n| `sky jobs launch task.yaml` | Launch a managed job (auto lifecycle + recovery) |\n| `sky jobs queue -o json` | Show all managed jobs and their status |\n| `sky jobs logs JOB_ID` | Stream logs from a managed job |\n| `sky jobs cancel JOB_ID` | Cancel a managed job |\n\n**SkyServe** — model serving with autoscaling:\n\n| Command | Description |\n|---------|-------------|\n| `sky serve up serve.yaml -n NAME` | Start a model serving service |\n| `sky serve status NAME` | Show service status and endpoint URL |\n| `sky serve update NAME new.yaml` | Update a running service (rolling) |\n| `sky serve down NAME` | Tear down a service |\n\nFor complete CLI reference, see [CLI Reference](references/cli-reference.md).\n\n## Quick Start\n\n```bash\n# Launch a GPU cluster\nsky launch -c mycluster --gpus H100 -- nvidia-smi\n\n# Run a task from YAML\nsky launch -c mycluster task.yaml\n\n# SSH into cluster\nssh mycluster\n\n# Connect VSCode or Cursor to the cluster for interactive development\ncode --remote ssh-remote+mycluster /home/user/sky_workdir\n# or: cursor --remote ssh-remote+mycluster /home/user/sky_workdir\n\n# Tear down\nsky down mycluster\n```\n\n## Task YAML Structure\n\nThe task YAML is SkyPilot's primary interface. All fields are optional.\n\n```yaml\n# task.yaml\nname: my-training-job\n\n# Local directory to sync to remote ~/sky_workdir\nworkdir: .\n\n# Number of nodes (for distributed training)\nnum_nodes: 1\n\nresources:\n # GPU/TPU accelerators (SkyPilot auto-selects the cheapest cloud/region)\n accelerators: H200:8\n # Optional: pin to a specific cloud/region/infra\n # infra: aws # or aws/us-east-1, k8s, ssh/my-pool\n # If infra is left out, SkyPilot automatically fails over across all\n # enabled clouds/regions to find the cheapest available option.\n # Use spot instances for cost savings\n use_spot: false\n # Disk size in GB\n disk_size: 256\n # Open ports for serving\n ports: 8080\n\n# Environment variables (accessible in file_mounts, setup, and run)\nenvs:\n MODEL_NAME: my-model\n BATCH_SIZE: 32\n\n# Setup: runs once on cluster creation, cached on reuse\nsetup: |\n pip install torch transformers\n\n# Run: the main command\nrun: |\n python train.py --model $MODEL_NAME --batch-size $BATCH_SIZE\n```\n\nFor complete YAML schema including file mounts, environment variables set by SkyPilot, and advanced fields, see [YAML Specification](references/yaml-spec.md).\n\n## GPU and Cloud Selection\n\n**IMPORTANT: Let SkyPilot choose the cloud and region.** Do NOT manually pick a cloud/region/instance by parsing `sky gpus list` output. SkyPilot's optimizer automatically selects the cheapest available option across all enabled clouds. Only specify `infra:` when the user explicitly requests a specific cloud or region.\n\n**Default behavior (recommended):** Just specify the GPU type. SkyPilot finds the cheapest cloud/region automatically:\n\n```yaml\nresources:\n accelerators: H200:8 # SkyPilot picks the cheapest cloud/region with H200:8\n```\n\nIf the user doesn't specify a GPU type, ask them what GPU they need (or what model/workload they're running so you can recommend one). Do NOT run `sky gpus list` and pick for them — present options and let the user decide, or use `any_of` to let SkyPilot maximize availability:\n\n```yaml\n# Let SkyPilot choose from multiple acceptable GPU types (cheapest wins)\nresources:\n any_of:\n - accelerators: H100:8\n - accelerators: A100-80GB:8\n - accelerators: A100:8\n```\n\nUse `ordered` only when the user has a strict preference:\n\n```yaml\n# Try H100 first on AWS, fall back to GCP, then A100\nresources:\n ordered:\n - infra: aws/us-east-1\n accelerators: H100:8\n - infra: gcp/us-central1\n accelerators: H100:8\n - infra: aws/us-west-2\n accelerators: A100-80GB:8\n```\n\nOnly set `infra:` when the user explicitly says something like \"use AWS\" or \"run on GCP us-central1\":\n\n```yaml\nresources:\n infra: aws # User asked for AWS specifically\n accelerators: H100:8\n```\n\n## Cluster Lifecycle\n\n```bash\n# Launch and run a task\nsky launch -c mycluster task.yaml\n\n# Launch with autostop at launch time (preferred: saves cost, no follow-up command needed)\nsky launch -c mycluster task.yaml -i 30 # stop after 30 min idle\nsky launch -c mycluster task.yaml -i 30 --down # tear down after 30 min idle\n\n# Override or pass environment variables via CLI\nsky launch -c mycluster task.yaml --env MODEL_NAME=llama3 --env BATCH_SIZE=64\n\n# Re-run a different task on the same cluster (fast, skips provisioning)\nsky exec mycluster another_task.yaml\n\n# Run an inline command\nsky exec mycluster -- python train.py --epochs 10\n\n# Set autostop after launch (use if you forgot to set -i at launch time)\nsky autostop mycluster -i 30 # stop after 30 min idle, preserving disk (can restart with sky start)\nsky autostop mycluster -i 30 --down # tear down after 30 min idle (disk is deleted, cannot restart)\n\n# Stop to save costs, restart later\nsky stop mycluster\nsky start mycluster\n\n# Tear down completely\nsky down mycluster\n```\n\n## Workdir Sync Behavior\n\n`workdir:` is synced to `~/sky_workdir` on the remote via `rsync` before every `sky exec`. **rsync is additive — deleted local files are NOT removed from the remote.** This can cause experiments to run against stale build artifacts or old configs.\n\nTo ensure a clean slate, SSH and wipe before `sky exec`:\n```bash\nssh mycluster \"rm -rf ~/sky_workdir\"\nsky exec mycluster task.yaml\n```\n\nOr clean inside `run:` if only specific artifacts need removal:\n```yaml\nrun: |\n find ~/sky_workdir/build -name '*.o' -delete 2>/dev/null || true\n cd ~/sky_workdir && make\n```\n\n## Managed Jobs\n\nUse `sky jobs launch` for long-running jobs that should run unattended. SkyPilot manages the full lifecycle — provisioning, execution, recovery from preemptions/quota/failures, and teardown:\n\n```yaml\n# managed-job.yaml\nname: training-job\n\nresources:\n accelerators: A100:8\n\nrun: |\n python train.py --resume-from-checkpoint\n```\n\n```bash\n# Launch as managed job\nsky jobs launch managed-job.yaml\n\n# Check status\nsky jobs queue -o json\n\n# Stream logs\nsky jobs logs \n\n# Cancel\nsky jobs cancel \n```\n\n**Checkpoint pattern**: Your training script should save checkpoints to persistent storage (cloud bucket or volume) and resume from the latest checkpoint on restart. SkyPilot handles the cluster recovery; your script handles the state recovery.\n\n## SkyServe: Model Serving\n\n```yaml\n# serve.yaml\nresources:\n accelerators: A100:1\n ports: 8080\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-3.1-8B-Instruct \\\n --port 8080\n\nservice:\n readiness_probe: /v1/models\n replica_policy:\n min_replicas: 1\n max_replicas: 3\n target_qps_per_replica: 5\n```\n\n```bash\n# Start service\nsky serve up serve.yaml -n my-llm\n\n# Check status / get endpoint\nsky serve status my-llm\nsky serve status my-llm --endpoint\n\n# Update (rolling)\nsky serve update my-llm new-serve.yaml\n\n# Tear down\nsky serve down my-llm\n```\n\n## Common Workflows\n\n### Fine-Tuning Workflow\n1. Write task YAML with `setup` (install deps) and `run` (training command)\n2. Use `file_mounts` or `workdir` to sync code\n3. `sky launch -c train task.yaml` to launch\n4. `sky logs train` to monitor\n5. `sky exec train -- python eval.py` to evaluate on same cluster\n6. `sky down train` when done\n\n### Hyperparameter Sweep\n1. Create parameterized YAML with `envs`\n2. Launch multiple managed jobs:\n ```bash\n for lr in 1e-4 1e-5 1e-6; do\n sky jobs launch sweep.yaml --env LR=$lr --name sweep-lr-$lr\n done\n ```\n3. Monitor with `sky jobs queue -o json`\n\n### Model Serving Deployment\n1. Write serve YAML with `service:` section\n2. `sky serve up serve.yaml -n my-service`\n3. Get endpoint: `sky serve status my-service --endpoint`\n4. Update model: `sky serve update my-service updated.yaml`\n\n### Parallel Experiment Submission\n\nUse `sky exec -d` to submit jobs to multiple VMs without blocking, then collect results:\n\n```bash\n# Submit all experiments (detached, returns after job is queued)\nfor i in 1 2 3 4; do\n sky exec exp-vm-0$i task.yaml --env LR=1e-$i -d\ndone\n\n# Get the latest job ID from a cluster\njob_id=$(sky queue exp-vm-01 -o json \\\n | python3 -c \"import sys, json; jobs = json.load(sys.stdin).get('exp-vm-01', []); print(max(j['job_id'] for j in jobs) if jobs else '')\")\n\n# Wait for a specific job and fetch last 50 lines\nsky logs exp-vm-01 $job_id --status && sky logs exp-vm-01 $job_id --tail 50\n\n# Check all jobs across a cluster at once\nsky queue exp-vm-01 -o json\n```\n\n### Converting a Slurm Workload\n\nWhen the user has an existing `sbatch` script, `salloc` workflow, or asks how a\nSlurm command maps to SkyPilot, **read\n[Migrating from Slurm](references/migrating-from-slurm.md) before writing any\nYAML.** The mapping has non-obvious parts that are easy to get wrong:\n\n- `--time` does **not** map to `autostop` — autostop is unsupported on Slurm.\n Use `config.slurm.sbatch_options.time`.\n- A bare `srun ` should usually be **dropped**, not translated: `run`\n already executes on every node. Only MPI/PMIx launchers keep `srun`, and\n then they need `--overlap` and a rank-0 guard.\n- `--account` / `--qos` / `--exclusive` and other unmodeled directives go\n through `config.slurm.sbatch_options`, which passes them to `sbatch`\n verbatim.\n- `sbatch --array` maps to `sky jobs launch --num-jobs` with\n `$SKYPILOT_JOB_RANK`, not to a shell loop.\n\nValidate the result with `sky launch --dryrun ` — note there is no\n`--dryrun` on `sky jobs launch`, so dry-run via `sky launch` even when the\nfinal command will be `sky jobs launch`.\n\nDo not assume the user wants to leave Slurm. SkyPilot submits to an existing\nSlurm cluster through its login node; migrating *to Kubernetes* is a separate\nquestion, and the constraints differ.\n\n## Agent Feedback Loop\n\nWhen using SkyPilot programmatically, follow this loop:\n\n1. **Validate**: `sky launch --dryrun task.yaml` (check resource availability/cost)\n2. **Launch**: `sky launch -c mycluster task.yaml`\n3. **Monitor**: `sky status -o json` and `sky queue mycluster -o json`\n4. **Wait for completion**: `sky logs mycluster ` (streams logs so you can observe progress and react to stalls; blocks until job finishes; get JOB_ID from `sky queue mycluster -o json`). For long-running jobs where you don't need intermediate output, use `sky logs mycluster --status` instead (blocks silently, exits 0 on success).\n5. **Inspect output**: `sky logs mycluster --no-follow` or `sky logs mycluster --tail 100`\n6. **Debug**: `ssh mycluster` (interactive)\n7. **Iterate**: `sky exec mycluster updated_task.yaml` (run on existing cluster)\n8. **Cleanup**: `sky down mycluster`\n\n> **Never poll with `sleep` + `sky queue`** — use `sky logs CLUSTER JOB_ID` to stream logs and block until done. Use `--status` if you only need the exit code, or `--tail N` to fetch recent output after completion.\n\n## Investigating a Managed Job\n\nWhen the user asks \"what is my job doing\" or \"why is it pending/failing\", run\nthese in order and stop as soon as the question is answered:\n\n1. `sky jobs queue -v -o json` — `status`, `details`, `failure_reason` (`-v` is what adds `details`). For a job that has not started, `details` is the answer: it says what the cluster is waiting on, e.g. a Slurm `QOSGrpGRES` or `Dependency` reason with the partition it is queued in\n2. `sky jobs logs JOB_ID --tail 100 --no-follow` — recent task output; `--controller` for provisioning/recovery logs\n\nSee [Job Investigation](references/job-investigation.md) for how to read `details` and the Slurm pending reasons.\n\n## Common Agent Mistakes\n\n| Mistake | Why it's wrong | Do this instead |\n|---------|---------------|-----------------|\n| Manually picking cloud/region from `sky gpus list` output | SkyPilot optimizer does this automatically and better | Just set `accelerators:` and let SkyPilot choose |\n| Using `sky launch` for long-running unattended jobs | No recovery if preempted or interrupted | Use `sky jobs launch` for unattended work |\n| Forgetting `sky down` or autostop after work is done | Wastes money on idle clusters | Always clean up, or use `-i --down` at launch |\n| Hardcoding `infra: aws` without user asking | Limits availability and increases cost | Only set `infra:` when user explicitly requests a cloud |\n| Not using `envs:` for configurable values | Hard to reuse or override from CLI | Use `envs:` in YAML + `--env KEY=VAL` for parameterization |\n| Running `sky launch` without `-c ` | Creates randomly-named cluster, hard to reference | Always name clusters with `-c` |\n| Parsing table output from status commands | Table formatting is for humans, fragile to parse | Use `-o json` for structured output |\n| Using deprecated `cloud:`/`region:`/`zone:` fields | Deprecated in favor of `infra:` | Use `infra: aws/us-east-1` instead |\n| Polling job status with `sleep` + `sky queue` | Wastes tokens, introduces timing bugs, fragile | Use `sky logs CLUSTER JOB_ID --status` to block until done |\n| Assuming workdir sync removes remote files | rsync is additive; old remote files persist across `sky exec` calls | SSH and manually clean `~/sky_workdir`, or clean in `run:` script |\n| Not using `--tail` when only last output matters | Streaming full logs wastes tokens for long jobs | Use `sky logs CLUSTER JOB_ID --tail 50` for last N lines |\n\n## Common Issues Quick Reference\n\n| Issue | Solution |\n|-------|----------|\n| GPU not available | Use `any_of` for fallback, or try different regions/clouds |\n| Setup takes too long | SkyPilot caches setup; use `sky exec` to skip it on reruns |\n| Task fails silently | Check `sky logs ` or `ssh ` to debug |\n| Cluster stuck in INIT | `sky down ` and relaunch |\n| Preemption/quota | Use `sky jobs launch` for automatic recovery and lifecycle management |\n| Port not accessible | Ensure `ports:` is set in resources and security groups allow traffic |\n| File sync slow | Use cloud bucket mounts instead of `workdir` for large datasets |\n| Credentials error | Run `sky check -o json` and inspect which clouds are disabled |\n\n## References\n\nFor detailed reference documentation:\n\n- [CLI Reference](references/cli-reference.md) — All commands and flags\n- [YAML Specification](references/yaml-spec.md) — Complete task YAML schema, file mounts, environment variables\n- [Python SDK](references/python-sdk.md) — Programmatic API and SDK usage\n- [Advanced Patterns](references/advanced-patterns.md) — Multi-cloud, distributed training, production patterns\n- [Migrating from Slurm](references/migrating-from-slurm.md) — Converting `sbatch` scripts to task YAMLs, Slurm command/env-var mapping, Slurm-specific constraints\n- [Job Investigation](references/job-investigation.md) — Which command answers \"what is my job doing and why\"; reading `details`, events, Slurm pending reasons\n- [Troubleshooting](references/troubleshooting.md) — Error diagnosis and solutions\n- [Examples](references/examples.md) — Copy-paste task YAML examples\n","agent/.claude-plugin/plugin.json":"{\n \"name\": \"skypilot\",\n \"description\": \"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds\",\n \"version\": \"1.0.0\",\n \"author\": {\n \"name\": \"SkyPilot Team\"\n },\n \"homepage\": \"https://docs.skypilot.co/\",\n \"repository\": \"https://github.com/skypilot-org/skypilot\",\n \"license\": \"Apache-2.0\",\n \"keywords\": [\"AI Infrastructure\", \"Multi-Cloud\", \"GPU\", \"Kubernetes\", \"Slurm\", \"Training\", \"Serving\", \"Cost Optimization\", \"SkyPilot\"]\n}\n","AGENTS.md":"# CLAUDE.md - SkyPilot Development Guide\n\nThis document provides guidance for AI assistants working with the SkyPilot codebase.\n\n## Project Overview\n\nSkyPilot is a system to run, manage, and scale AI workloads on any AI infrastructure. It provides a unified interface across 25+ cloud providers (AWS, GCP, Azure, Kubernetes, Slurm, and many others), enabling users to launch compute resources, run jobs, and serve models without vendor lock-in.\n\n**Key capabilities:**\n- **Clusters**: Launch and manage compute clusters across clouds and Kubernetes\n- **Managed jobs**: Run jobs with automatic recovery from preemptions and failures\n- **Multi-cloud and multi-Kubernetes**: Unified interface across 25+ clouds and multiple K8s clusters\n- Cost optimization and GPU availability maximization\n\n## Repository Structure\n\n```\nskypilot/\n├── sky/ # Main source code\n│ ├── __init__.py # Package exports and version\n│ ├── core.py # Core orchestration logic\n│ ├── task.py # Task definition and YAML parsing\n│ ├── resources.py # Cloud resource specifications\n│ ├── optimizer.py # Resource allocation optimizer\n│ ├── execution.py # Task execution pipeline\n│ ├── exceptions.py # Custom exceptions\n│ ├── skypilot_config.py # Configuration management\n│ ├── clouds/ # Cloud provider abstractions (25+ providers)\n│ ├── backends/ # Execution backends (Ray-based VM, Docker)\n│ ├── provision/ # Cloud resource provisioning\n│ ├── jobs/ # Managed job lifecycle\n│ ├── serve/ # Model serving (SkyServe)\n│ ├── skylet/ # On-cluster execution agent\n│ ├── client/ # SDK and CLI\n│ ├── server/ # API server and dashboard backend\n│ ├── dashboard/ # Web UI (Next.js)\n│ ├── utils/ # 50+ utility modules\n│ ├── catalog/ # Cloud pricing and instance catalogs\n│ ├── data/ # Storage and data handling\n│ ├── adaptors/ # Cloud-specific metadata adapters\n│ └── schemas/ # Protobuf definitions and API schemas\n├── tests/ # Test suite\n│ ├── unit_tests/ # Unit tests with subdirectories per module\n│ ├── smoke_tests/ # Quick validation tests\n│ ├── integration_tests/ # End-to-end tests\n│ ├── kubernetes/ # K8s-specific tests\n│ └── conftest.py # Pytest fixtures\n├── examples/ # 50+ usage examples\n├── llm/ # 45+ LLM training/serving examples\n├── docs/ # Sphinx documentation\n├── charts/ # Helm charts for K8s deployment\n└── format.sh # Code formatting script\n```\n\n## Development Setup\n\n### Environment Setup\n\n```bash\n# Create virtual environment with uv (Python 3.8-3.11 supported)\n# --seed is required to ensure pip is installed (needed for building wheels)\nuv venv --seed --python 3.11\nsource .venv/bin/activate\n\n# Install in editable mode with all cloud support\nuv pip install -e \".[all]\"\n# Or specific clouds only:\n# uv pip install -e \".[aws,gcp,kubernetes]\"\n\n# Install development dependencies\nuv pip install -r requirements-dev.txt\n\n# Optional: Install pre-commit hooks\nuv pip install pre-commit\npre-commit install\n```\n\n### Environment Variables\n\n```bash\nexport SKYPILOT_DEV=1 # Enable development mode\nexport SKYPILOT_DEBUG=1 # Enable debug logging\n```\n\n## Code Formatting and Linting\n\n**Always run `format.sh` before committing:**\n\n```bash\nbash format.sh # Format changed files (vs origin/master)\nbash format.sh --all # Format entire codebase\nbash format.sh --files path/to/file.py # Format specific files\n```\n\nThe script runs:\n1. **YAPF** - Google style for all Python code\n2. **isort** - Import sorting (Google profile)\n3. **mypy** - Type checking\n4. **pylint** - Linting with custom rules\n\n### Tool Versions (must match exactly)\n\nFrom `requirements-dev.txt`:\n- yapf==0.32.0\n- pylint==2.14.5\n- mypy==1.19.1\n- isort==5.12.0\n- pylint-quotes==0.2.3\n\n### Excluded from Formatting\n\n- `sky/schemas/generated/` - Auto-generated protobuf files\n- `build/` - Build artifacts\n\n## Testing\n\n### Running Tests\n\n```bash\n# Unit tests (fast, no cloud resources)\npytest tests/unit_tests/\n\n# Specific test file\npytest tests/unit_tests/test_resources.py\n```\n\n### CI Tests via PR Comments\n\nTrigger CI tests on pull requests using comments:\n- `/quicktest-core` - Run quick core tests\n- `/smoke-test` - Run smoke tests (launches cloud clusters)\n- `/smoke-test --kubernetes --postgres` - Test with PostgreSQL backend on Kubernetes\n- `/smoke-test --kubernetes --remote-server --postgres` - Test remote API server with PostgreSQL\n\n### Test Configuration\n\nFrom `pyproject.toml`:\n- Uses pytest-xdist with 16 parallel workers\n- Environment: `SKYPILOT_DEBUG=1`, `SKYPILOT_DEV=1`\n- Buildkite integration for CI\n\n### Checking Buildkite CI Status\n\nTo check CI test results for a PR:\n\n1. **Get Buildkite URL from GitHub**: Check the PR's commit status checks to find the Buildkite build URL (may take a moment to appear after triggering)\n2. **Fetch build logs via Buildkite API**: Use the Buildkite API to retrieve detailed logs and test results\n\n```bash\n# Example: Get build details (requires BUILDKITE_TOKEN)\ncurl -H \"Authorization: Bearer $BUILDKITE_TOKEN\" \\\n \"https://api.buildkite.com/v2/organizations/skypilot/pipelines/skypilot/builds/\"\n```\n\n## Code Style Guidelines\n\n### General Principles\n\n- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)\n- Use f-strings over `.format()` for readability\n- Use `class MyClass:` not `class MyClass(object):`\n- Use `abc` module for abstract classes\n- Use Python typing with `if typing.TYPE_CHECKING:` for type-only imports\n- Always place imports at the top of the file, not inside function definitions. If this causes a circular dependency, try to resolve it (e.g., restructure modules or use `if typing.TYPE_CHECKING:`). Only as a last resort, place the import inside the function with a comment explaining why. Import performance is not a valid reason for in-function imports; use `LazyImport` instead (see below).\n\n### TODOs and FIXMEs\n\nAlways include author attribution:\n```python\n# TODO(username): Description of what needs to be done\n# FIXME(username): Description of what needs fixing\n```\n\n### Exceptions vs Assertions\n\n- Use exceptions for errors (better error messages)\n- Use `assert` only for debugging/proof-checking\n\n### Lazy Imports\n\nFor modules with significant import time (>100ms) that are imported during `import sky`, use `LazyImport` from `sky/adaptors/common.py`:\n\n```python\nfrom sky.adaptors.common import LazyImport\nheavy_module = LazyImport('heavy_module')\n```\n\nMeasure import time with:\n```bash\npython -X importtime -c \"import sky\" 2> import.log\ntuna import.log # pip install tuna\n```\n\n## Architecture Patterns\n\n### Cloud Provider Abstraction\n\nAll cloud providers inherit from `sky.clouds.Cloud` (in `sky/clouds/cloud.py`). Cloud objects are **lightweight and stateless**. Key design principles:\n\n- Methods should be inexpensive to call\n- Don't store heavy state in cloud objects\n- Cache cloud-specific queries in `sky/clouds/utils/` modules\n- Each cloud implements feature flags via `CloudImplementationFeatures`\n\nAdding a new cloud provider:\n1. Create `sky/clouds/.py` inheriting from `Cloud`\n2. Implement required abstract methods\n3. Add provisioning logic in `sky/provision//`\n4. Register in `sky/clouds/__init__.py`\n\n### Client-Server Architecture\n\nSkyPilot uses a client-server model with API versioning:\n\n- Client (SDK/CLI) in `sky/client/`\n- Server in `sky/server/`\n- API version in `sky/server/constants.py`\n\n**Backward Compatibility Rules (from v0.10.0+):**\n- Changes must be backward compatible\n- Bump `API_VERSION` when introducing API changes\n- Use `@versions.minimal_api_version(N)` decorator for new SDK methods\n- Handle version differences with `versions.get_remote_api_version()`\n\n### Protobuf Regeneration\n\nWhen modifying `.proto` files in `sky/schemas/proto/`:\n\n```bash\npython -m grpc_tools.protoc \\\n --proto_path=sky/schemas/generated=sky/schemas/proto \\\n --python_out=. \\\n --grpc_python_out=. \\\n --pyi_out=. \\\n sky/schemas/proto/*.proto\n```\n\n### Dependency Management\n\nDependencies are defined in `sky/setup_files/dependencies.py`:\n\n- **Core dependencies**: Listed in `install_requires`\n- **Cloud-specific**: Defined in `extras_require` (e.g., `aws`, `gcp`, `kubernetes`)\n- **Development**: Listed in `requirements-dev.txt`\n\nWhen updating dependencies:\n\n1. Check version constraints carefully - some packages have breaking changes\n2. Consider Python version compatibility (3.8-3.11)\n3. Test with both minimum and latest allowed versions\n4. Document version constraints with comments explaining why\n\n## Key Modules Reference\n\n| Module | Purpose |\n|--------|---------|\n| `sky/core.py` | Core orchestration (launch, exec, stop, down) |\n| `sky/task.py` | Task YAML parsing and specification |\n| `sky/resources.py` | Resource requirements (GPUs, memory, disk) |\n| `sky/optimizer.py` | Cloud selection and cost optimization |\n| `sky/client/sdk.py` | Python SDK implementation |\n| `sky/client/cli/` | CLI commands |\n| `sky/backends/cloud_vm_ray_backend.py` | Main execution backend for clusters |\n| `sky/provision/provisioner.py` | Resource provisioning |\n| `sky/jobs/` | Managed jobs with recovery and scheduling |\n\n## Pull Request Guidelines\n\n1. **Branch from master**, create descriptive branch name\n2. **Run `format.sh`** before committing\n3. **Add tests** for core system changes\n4. **Run smoke tests** for significant changes\n5. **Include `Tested:` section** in PR description with test plan\n6. **Delete branch** after merging\n\n### PR Description Format\n\nPRs should include:\n- **Summary**: Brief description of changes (1-3 bullet points)\n- **Test plan**: How the changes were tested (commands run, manual verification steps)\n\n**Important**: Always generate a manual test plan describing how to verify the changes work correctly. Include specific commands, expected outputs, or UI verification steps. Whenever possible, add unit tests and smoke tests for the changes.\n\n### Commit Message Format\n\nUse the `[Area] Description` format:\n- `[Core]` for core system changes\n- `[CLI]` for CLI changes\n- `[API]` for API server changes\n- `[Docs]` for documentation\n- `[AWS]`, `[GCP]`, `[Azure]`, `[Kubernetes]` for cloud-specific changes\n- `[Jobs]` for managed jobs\n- `[Dashboard]` for web UI changes\n- `[Serve]` for model serving (SkyServe)\n- `[Test]` for testing changes\n- `[CI]` for CI/CD changes\n\nExamples:\n- `[Core] Fix cluster status refresh logic`\n- `[AWS] Add support for new instance types`\n- `[Docs] Update installation guide`\n\n## API Server Testing\n\n### Local API Server (Recommended for Development)\n\n```bash\n# Always restart API server after code changes to pick up changes\nsky api stop\nsky api start\n\n# Verify server is running\nsky api status\n```\n\n### Dashboard Development\n\n**For local API server development**, rebuild the dashboard before restarting:\n\n```bash\n# Install dependencies (first time or after package.json changes)\nnpm --prefix sky/dashboard install\n\n# Rebuild the dashboard\nnpm --prefix sky/dashboard run build\n\n# Then restart the API server\nsky api stop\nsky api start\n```\n\n**For remote API server (Docker/Kubernetes)**, the Dockerfile automatically builds the dashboard - no manual build needed before `docker build`.\n\nThe dashboard is a Next.js application. For development with hot reloading:\n\n```bash\n# Run dashboard in development mode (separate from API server)\ncd sky/dashboard\nnpm run dev\n```\n\n### Mocking Remote API Server Locally\n\nTo test remote API server behavior locally:\n\n```bash\n# Start local API server (runs on port 46580 by default)\nsky api stop\nsky api start\n\n# Forward to a different port to simulate remote server\nsocat TCP-LISTEN:46590,fork TCP:127.0.0.1:46580 &\n\n# Connect to the forwarded port as if it were a remote server\nsky api login -e http://127.0.0.1:46590\n```\n\n### Remote API Server (Kubernetes Deployment)\n\nFor testing with a remote API server on Kubernetes:\n\n```bash\n# Build local changes\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\nhelm dependency build ./charts/skypilot\n\n# Build Docker image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy (NEW installation)\nNAMESPACE=skypilot\nRELEASE_NAME=skypilot\nhelm upgrade --install $RELEASE_NAME ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n#### Upgrading Existing Deployments\n\n**CRITICAL:** Always use `--reuse-values` to preserve database/credential config:\n\n```bash\n# Upgrade existing deployment (keeps PostgreSQL, auth, etc.)\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n\n# Check current values / rollback if needed\nhelm get values skypilot -n skypilot\nhelm rollback skypilot -n skypilot\n```\n\n#### PostgreSQL Backend\n\n```bash\n# Create connection secret\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/db\"\n\n# Deploy with PostgreSQL\nhelm upgrade --install skypilot ./charts/skypilot -n skypilot \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\n## Critical Code Paths (Handle with Care)\n\nThe following modules contain complex, stateful logic that requires careful review when modifying:\n\n**Managed Jobs Recovery:**\n- `sky/jobs/controller.py` - Job lifecycle management, state transitions, async coordination\n- `sky/jobs/recovery_strategy.py` - Preemption recovery, retry logic for managed jobs\n- `sky/backends/cloud_vm_ray_backend.py` - Execution backend with complex state handling\n\nThese modules handle edge cases like preemption during job submission, controller failures mid-recovery, and race conditions between concurrent operations. Changes can have subtle effects on job reliability.\n\n**API Server Performance & Robustness:**\n- `sky/server/` - API server with memory efficiency, low latency requirements\n- `sky/backends/backend_utils.py` - Cluster status caching, network resilience, SSH handling\n\nThese modules are optimized for performance. Be cautious about adding blocking calls, memory-heavy operations, or changing caching behavior.\n\n**CLI/SDK Interface Design:**\n- `sky/client/cli/` - Command-line interface\n- `sky/client/sdk.py` - Python SDK\n\nBe cautious about adding new interfaces or changing existing UX significantly. Keep the interface clean and minimal for both CLI and SDK.\n\n**Backward Compatibility:**\n- Test with old server + new client, and old client + new server\n- Ensure existing clusters and jobs continue working after server upgrades\n- `/quicktest-core` tests some backward compatibility scenarios\n- See `docs/source/developers/CONTRIBUTING.md` for API versioning guidelines\n\n## Common Pitfalls\n\n1. **Always restart API server after code changes** - Run `sky api stop; sky api start` to pick up changes\n2. **Don't modify `sky/schemas/generated/`** - These are auto-generated\n3. **Match formatter versions exactly** - Version mismatches cause CI failures\n4. **Consider import time** - Heavy imports slow down CLI responsiveness\n5. **API versioning** - Always maintain backward compatibility\n\n## Useful Commands\n\n```bash\n# Profile CLI performance\nuv pip install py-spy\npy-spy record -t -o sky.svg -- python -m sky.cli status\n\n# Check cloud credentials\nsky check\n\n# View cluster status\nsky status\n\n# Launch a cluster\nsky launch \n\n# Execute a command on the cluster\nsky exec -- bash\n\n# SSH into the cluster\nssh \n\n# Launch a managed job\nsky jobs launch \n```\n\n## Documentation\n\n- **User docs**: https://docs.skypilot.co/\n- **Source**: `docs/source/` (Sphinx)\n- **Build docs**: See `.readthedocs.yml`\n\n## Additional Resources\n\n- **Full contributing guide**: `docs/source/developers/CONTRIBUTING.md`\n- **User docs**: https://docs.skypilot.co/\n- **GitHub Issues**: https://github.com/skypilot-org/skypilot/issues\n- **Discussions**: https://github.com/skypilot-org/skypilot/discussions\n- **Slack**: http://slack.skypilot.co\n","agent/skills/skypilot/evals/evals.json":"{\n \"skill_name\": \"skypilot\",\n \"description\": \"Evaluates the 'Before You Start (Agent Bootstrap)', 'Converting a Slurm Workload' and 'Investigating a Managed Job' section behavior\",\n \"evals\": [\n {\n \"id\": 1,\n \"name\": \"happy-path-server-running\",\n \"prompt\": \"I want to launch an 8xH100 cluster for training a LLaMA model. My SkyPilot is already set up.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, sees the server is running (table output with request data), skips rest of bootstrap (no `sky check`, no install, no `sky api start`), and proceeds directly to help write the task YAML or run `sky launch`.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` (or `sky api status -l 1`) as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT run `sky check` when the server is already running\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent does NOT run `sky api start` or `sky api login` when the server is already running\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent proceeds to help with the user's actual task (launching H100 cluster) without unnecessary bootstrap steps\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 2,\n \"name\": \"server-not-running-local-setup\",\n \"prompt\": \"I just installed SkyPilot and want to try launching a GPU instance on AWS. I don't have a remote API server.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, detects 'SkyPilot API server is not running.', asks about local vs remote, user indicates local, agent runs `sky api start`, re-checks with `sky api status -l 1`, then runs `sky check` for credentials since this is a fresh setup.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent asks the user whether they have a remote server or want to start locally\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent runs `sky api start` after user confirms local setup\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent re-runs `sky api status` after starting the server to confirm it's reachable\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent runs `sky check` since this is a fresh setup\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 3,\n \"name\": \"server-not-running-remote-connect\",\n \"prompt\": \"I need to connect to our team's SkyPilot API server at https://skypilot.internal.company.com and launch a fine-tuning job.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, detects server not running, recognizes user already provided a remote URL, runs `sky api login -e https://skypilot.internal.company.com` without asking unnecessary questions, re-checks status, then proceeds to help with the fine-tuning job.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent uses `sky api login -e https://skypilot.internal.company.com` (not `sky api start`) since the user provided a remote endpoint\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT ask the user for the server URL again since they already provided it\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent re-runs `sky api status` after connecting to confirm it's reachable\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 4,\n \"name\": \"slurm-sbatch-time-not-autostop\",\n \"prompt\": \"Convert this to SkyPilot for our Slurm cluster:\\n#!/bin/bash\\n#SBATCH --nodes=2\\n#SBATCH --gpus-per-node=8\\n#SBATCH --time=24:00:00\\n#SBATCH --account=ai-research\\nsrun python train.py\",\n \"expected_output\": \"Agent produces a task YAML with num_nodes: 2, resources.accelerators: H100:8 (or asks for the GRES name), and puts both --time and --account under config.slurm.sbatch_options. The bare `srun` is dropped because `run` already executes on every node. Agent recommends `sky jobs launch` for this batch script.\",\n \"assertions\": [\n {\n \"text\": \"Agent maps --time to config.slurm.sbatch_options.time\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT map --time to autostop, which is unsupported on Slurm\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent maps --account to config.slurm.sbatch_options.account\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent puts num_nodes at task level, not under resources\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT wrap the run command in `srun` again\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent recommends `sky jobs launch` (not `sky launch`) for this batch script\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 5,\n \"name\": \"slurm-job-array-to-num-jobs\",\n \"prompt\": \"On Slurm I run `sbatch --array=0-99 shard.sh` and the script reads $SLURM_ARRAY_TASK_ID to pick its shard. What's the SkyPilot equivalent?\",\n \"expected_output\": \"Agent answers `sky jobs launch --num-jobs 100` and maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK (with $SKYPILOT_NUM_JOBS for the count). May also mention pools for reusing workers across submissions.\",\n \"assertions\": [\n {\n \"text\": \"Agent uses `sky jobs launch --num-jobs 100`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT propose a shell for-loop as the primary answer\",\n \"type\": \"negative\"\n }\n ]\n },\n {\n \"id\": 6,\n \"name\": \"why-is-my-managed-job-pending\",\n \"prompt\": \"My managed job 42 has been PENDING for 20 minutes on our Slurm cluster. Why? SkyPilot is already set up.\",\n \"expected_output\": \"Agent reads `details` from `sky jobs queue -v -o json` and finds the Slurm pending reason with its partition (e.g. `Launching (pending: QOSGrpGRES; partition: h200)`), explains that category (quota vs no free capacity vs dependency vs held) and what to do about it. Does not claim the job is stuck in SkyPilot's own queue when the reason is Slurm's.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky jobs queue -v -o json` (`-v` is required for the `details` field) and reads the job's details\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent reads the Slurm pending reason and its partition from `details`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent explains the pending reason in plain terms (quota, capacity, dependency, or held) with a next step\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT poll with `sleep` + `sky jobs queue`\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent does NOT tear down the jobs controller as a first step\",\n \"type\": \"negative\"\n }\n ]\n },\n {\n \"id\": 7,\n \"name\": \"what-happened-to-my-job\",\n \"prompt\": \"Managed job 17 keeps flipping between RUNNING and RECOVERING. Show me what happened.\",\n \"expected_output\": \"Agent reads `sky jobs queue -v -o json` for `details` / `failure_reason` / `recovery_count`, then pulls `sky jobs logs 17 --controller --no-follow` for the recovery history, and points out that a repeated identical reason means a persistent cause the controller cannot fix by retrying.\",\n \"assertions\": [\n {\n \"text\": \"Agent reads `details`, `failure_reason` and `recovery_count` from `sky jobs queue -v -o json`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent identifies the recurring RECOVERING reason from the events\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent uses `--tail` or `--no-follow` when fetching logs rather than streaming the full log\",\n \"type\": \"required_action\"\n }\n ]\n }\n ]\n}\n","agent/skills/skypilot/references/advanced-patterns.md":"# SkyPilot Advanced Patterns Reference\n\nThis reference covers advanced SkyPilot patterns for multi-cloud strategies,\ndistributed training, managed jobs, serving, Kubernetes integration, API server\nsetup, and cost optimization.\n\n---\n\n## 1. Multi-Cloud Strategies\n\nSkyPilot can automatically select the cheapest cloud/region/zone for a\nworkload, or follow explicit priority ordering. The key mechanisms are:\n- `any_of`: try all listed resource options, optimize for cost (unordered)\n- `ordered`: try resource options in strict priority order (first match wins)\n- `infra:` field with wildcard patterns for flexible cloud/region/zone targeting\n\n### 1.1 any_of -- Cheapest Available GPU\n\nUse `any_of` when you have multiple acceptable resource configurations and\nwant SkyPilot to pick the cheapest one available across all of them.\n\nNote: If all options use the same accelerator, you can simply specify\n`accelerators:` without `any_of` — SkyPilot will automatically search\nacross all enabled clouds. Use `any_of` when mixing different GPU types:\n\n```yaml\n# cheapest-gpu.yaml — mix different GPU types, SkyPilot picks cheapest\nname: cheapest-gpu-training\n\nresources:\n any_of:\n - accelerators: H100:8\n - accelerators: A100-80GB:8\n - accelerators: A100:8\n\nrun: |\n torchrun --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE train.py\n```\n\nYou can also mix different GPU types across clouds:\n\n```yaml\n# mixed-gpus.yaml\nname: mixed-gpu-training\n\nresources:\n any_of:\n - infra: aws\n accelerators: A10g:4\n - infra: gcp\n accelerators: L4:4\n - infra: azure\n accelerators: A100:1\n - infra: lambda\n accelerators: A100:1\n\nrun: |\n python train.py\n```\n\nThe optimizer evaluates all options together, considering current prices,\navailability, and any quotas, then launches on the cheapest option.\n\n### 1.2 ordered -- Strict Priority Fallback\n\nUse `ordered` when you have a strict preference. SkyPilot tries each option\nin order and uses the first one that succeeds.\n\n```yaml\n# priority-fallback.yaml\nname: priority-training\n\nresources:\n ordered:\n - infra: aws/us-east-1\n accelerators: A100:8\n - infra: gcp/us-central1\n accelerators: A100:8\n - infra: aws/us-west-2\n accelerators: A100:4\n - infra: gcp/europe-west4\n accelerators: A100:4\n\nrun: |\n python train.py\n```\n\nThis is useful when you have reserved instances or negotiated pricing on a\nspecific cloud/region and want to use those first.\n\n### 1.3 Kubernetes + Cloud Fallback\n\nA common pattern is to try on-prem Kubernetes first, then fall back to cloud:\n\n```yaml\n# k8s-cloud-fallback.yaml\nname: k8s-with-cloud-fallback\n\nresources:\n ordered:\n # Try on-prem K8s cluster first (free/cheaper)\n - infra: k8s/my-onprem-context\n accelerators: A100:4\n # Fall back to cloud K8s\n - infra: k8s/gke-prod-cluster\n accelerators: A100:4\n # Fall back to cloud VMs\n - infra: aws\n accelerators: A100:4\n - infra: gcp\n accelerators: A100:4\n\nrun: |\n python train.py\n```\n\n### 1.4 SSH Node Pools\n\nSkyPilot can use your own bare-metal machines or VMs via SSH node pools.\nFirst, define node pools in `~/.sky/ssh_node_pools.yaml`:\n\n```yaml\n# ~/.sky/ssh_node_pools.yaml\nmy-gpu-cluster:\n hosts:\n - 10.0.1.10\n - 10.0.1.11\n - 10.0.1.12\n\nmy-inference-box:\n hosts:\n - gpu-server.internal.example.com\n```\n\nThen reference them in your task YAML:\n\n```yaml\n# ssh-nodes.yaml\nname: train-on-ssh-nodes\n\nresources:\n infra: ssh/my-gpu-cluster\n accelerators: A100:8\n\nrun: |\n python train.py\n```\n\nSSH node pools can also participate in `any_of` or `ordered` fallback:\n\n```yaml\nresources:\n ordered:\n - infra: ssh/my-gpu-cluster\n accelerators: A100:8\n - infra: aws\n accelerators: A100:8\n```\n\n### 1.5 Wildcard Patterns in infra\n\nThe `infra` field supports `*` as a wildcard for any component\n(cloud, region, zone):\n\n```yaml\n# Any cloud, specific zone\nresources:\n infra: \"*/us-east-1/us-east-1a\"\n accelerators: A100:4\n```\n\n```yaml\n# Specific cloud, any region, specific zone suffix\nresources:\n infra: \"aws/*/us-east-1a\"\n accelerators: A100:4\n```\n\n```yaml\n# Any cloud, any region (equivalent to not specifying infra)\nresources:\n infra: \"*\"\n accelerators: A100:4\n```\n\nThe infra format is `cloud[/region[/zone]]`. For Kubernetes, the format is\n`k8s/context-name` (context names may contain slashes).\n\n### 1.6 Multi-Region Strategies\n\nTarget a specific region across any cloud, or multiple specific regions:\n\n```yaml\n# Multi-region with any_of\nresources:\n any_of:\n - infra: aws/us-east-1\n accelerators: H100:8\n - infra: aws/us-west-2\n accelerators: H100:8\n - infra: gcp/us-central1\n accelerators: H100:8\n - infra: gcp/us-east4\n accelerators: H100:8\n```\n\nFor single-cloud multi-region with the same GPU:\n\n```yaml\n# Will automatically try all available regions for this cloud\nresources:\n infra: aws\n accelerators: H100:8\n```\n\nSkyPilot's optimizer automatically considers all regions within the specified\ncloud and picks the cheapest available one. You only need `any_of` or\n`ordered` when you want to cross cloud boundaries or mix GPU types.\n\n### 1.7 Accelerator Sets and Lists\n\nFor single-resource blocks, you can use accelerator sets (unordered, cost-\noptimized) or lists (ordered, priority-based):\n\n```yaml\n# Unordered set: SkyPilot picks the cheapest available\nresources:\n accelerators: {A100:1, V100:1, L4:1, T4:1}\n```\n\n```yaml\n# Ordered list: SkyPilot tries in order\nresources:\n accelerators: [A100:1, V100:1, L4:1, T4:1]\n```\n\n---\n\n## 2. Distributed Training\n\nSkyPilot supports multi-node distributed training with automatic cluster\nsetup. It provides environment variables on every node for coordination.\n\n### 2.1 SkyPilot Environment Variables for Distributed Training\n\nOn each node, SkyPilot sets:\n\n| Variable | Description |\n|---|---|\n| `SKYPILOT_NODE_IPS` | Newline-separated list of all node IPs (head node first) |\n| `SKYPILOT_NODE_RANK` | 0-indexed rank of the current node (head = 0) |\n| `SKYPILOT_NUM_NODES` | Total number of nodes in the cluster |\n| `SKYPILOT_NUM_GPUS_PER_NODE` | Number of GPUs on the current node |\n\n### 2.2 PyTorch DDP Multi-Node Setup\n\nThe standard pattern uses `torchrun` with SkyPilot's environment variables:\n\n```yaml\n# ddp-training.yaml\nname: pytorch-ddp\n\nresources:\n accelerators: A100:8\n\nnum_nodes: 4\n\nsetup: |\n pip install torch torchvision\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n\n torchrun \\\n --nnodes=$NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$HEAD_IP \\\n --master_port=12355 \\\n train_ddp.py \\\n --epochs 100 \\\n --batch-size 256\n```\n\nA minimal `train_ddp.py` would use:\n\n```python\nimport os\n\nimport torch\nimport torch.distributed as dist\n\ndist.init_process_group(backend='nccl')\nlocal_rank = int(os.environ['LOCAL_RANK'])\ntorch.cuda.set_device(local_rank)\n# ... training loop\n```\n\n### 2.3 DeepSpeed ZeRO-3 Multi-Node\n\n```yaml\n# deepspeed-zero3.yaml\nname: deepspeed-training\n\nresources:\n accelerators: A100-80GB:8\n\nnum_nodes: 2\n\nfile_mounts:\n /workspace: .\n\nsetup: |\n pip install deepspeed torch transformers accelerate\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n\n # Create hostfile for DeepSpeed\n echo \"$SKYPILOT_NODE_IPS\" | while read ip; do\n echo \"$ip slots=$SKYPILOT_NUM_GPUS_PER_NODE\"\n done > /tmp/hostfile\n\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n deepspeed \\\n --hostfile /tmp/hostfile \\\n --master_addr $HEAD_IP \\\n --master_port 29500 \\\n --num_nodes $NUM_NODES \\\n --num_gpus $SKYPILOT_NUM_GPUS_PER_NODE \\\n /workspace/train.py \\\n --deepspeed /workspace/ds_config_zero3.json \\\n --model_name_or_path meta-llama/Llama-2-7b-hf \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 4\n fi\n```\n\nExample `ds_config_zero3.json`:\n\n```json\n{\n \"bf16\": { \"enabled\": true },\n \"zero_optimization\": {\n \"stage\": 3,\n \"offload_optimizer\": { \"device\": \"none\" },\n \"offload_param\": { \"device\": \"none\" },\n \"overlap_comm\": true,\n \"contiguous_gradients\": true,\n \"reduce_bucket_size\": 5e7,\n \"stage3_prefetch_bucket_size\": 5e7,\n \"stage3_param_persistence_threshold\": 1e5\n },\n \"gradient_accumulation_steps\": 4,\n \"train_micro_batch_size_per_gpu\": 2,\n \"wall_clock_breakdown\": false\n}\n```\n\n### 2.4 Ray Train Integration\n\nSkyPilot includes a helper script to start a Ray cluster across nodes. Use it\nfor Ray Train, Ray Tune, and other Ray-based distributed workloads:\n\n```yaml\n# ray-train.yaml\nname: ray-distributed-training\n\nresources:\n accelerators: A100:8\n ports:\n - 8280 # Ray dashboard\n\nnum_nodes: 2\n\nenvs:\n HF_HUB_ENABLE_HF_TRANSFER: \"1\"\n\nsetup: |\n pip install \"ray[default,train]\" torch transformers\n\nrun: |\n head_ip=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n\n # Use the built-in SkyPilot Ray start script\n export RAY_HEAD_PORT=6385\n export RAY_DASHBOARD_PORT=8280\n export RAY_DASHBOARD_HOST=0.0.0.0\n export RAY_HEAD_IP_ADDRESS=\"$head_ip\"\n ~/sky_templates/ray/start_cluster\n\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n export RAY_ADDRESS=\"http://localhost:8280\"\n ray job submit --address=\"$RAY_ADDRESS\" \\\n -- python train_with_ray.py\n fi\n```\n\n### 2.5 FSDP Configuration\n\nPyTorch Fully Sharded Data Parallel (FSDP) with SkyPilot:\n\n```yaml\n# fsdp-training.yaml\nname: fsdp-llama\n\nresources:\n accelerators: A100-80GB:8\n\nnum_nodes: 2\n\nsetup: |\n pip install torch transformers accelerate\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n TOTAL_GPUS=$((NUM_NODES * SKYPILOT_NUM_GPUS_PER_NODE))\n\n accelerate launch \\\n --num_processes $TOTAL_GPUS \\\n --num_machines $NUM_NODES \\\n --machine_rank $SKYPILOT_NODE_RANK \\\n --main_process_ip $HEAD_IP \\\n --main_process_port 29500 \\\n --use_fsdp \\\n --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP \\\n --fsdp_sharding_strategy FULL_SHARD \\\n --fsdp_state_dict_type SHARDED_STATE_DICT \\\n --mixed_precision bf16 \\\n train_fsdp.py \\\n --model_name meta-llama/Llama-2-7b-hf \\\n --per_device_train_batch_size 2\n```\n\n### 2.6 NCCL Environment Tuning\n\nFor multi-node GPU training, NCCL configuration is critical for performance.\nSet these via `envs:` in your YAML:\n\n```yaml\n# nccl-tuned.yaml\nname: nccl-optimized-training\n\nresources:\n accelerators: H100:8\n\nnum_nodes: 4\n\nenvs:\n # Debug: set to INFO or WARN for troubleshooting\n NCCL_DEBUG: WARN\n # Network interface selection (check with `ip addr` on your nodes)\n NCCL_SOCKET_IFNAME: eth0\n # Use tree algorithm for better all-reduce on large clusters\n NCCL_ALGO: Tree\n # Avoid CUDA stream recording overhead\n TORCH_NCCL_AVOID_RECORD_STREAMS: \"1\"\n # InfiniBand settings (for clouds with IB/RDMA)\n NCCL_IB_DISABLE: \"0\"\n NCCL_IB_GID_INDEX: 3\n # Cross-node buffer sizes\n NCCL_BUFFSIZE: \"8388608\"\n # P2P transport (NVLink within node, network across nodes)\n NCCL_P2P_LEVEL: NVL\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$HEAD_IP \\\n --master_port=12355 \\\n train.py\n```\n\n### 2.7 Common Distributed Training Gotchas\n\n1. **Port conflicts**: Always use a non-default master port (not 29500)\n if running multiple training jobs on the same cluster. Or use SkyPilot's\n port allocation.\n\n2. **Head node vs worker pattern**: Only the head node (`SKYPILOT_NODE_RANK\n == 0`) should launch the training coordinator (e.g., DeepSpeed launcher,\n Ray job submit). Worker nodes run the setup + the Ray/torchrun rendezvous.\n\n3. **Hanging at initialization**: Usually caused by NCCL not finding the right\n network interface. Set `NCCL_SOCKET_IFNAME` to the correct interface.\n Use `NCCL_DEBUG=INFO` to diagnose.\n\n4. **OOM on multi-node**: Gradient synchronization buffers consume extra\n memory. Reduce per-device batch size or use gradient accumulation.\n\n5. **Run section runs on ALL nodes**: The `run:` section executes on every\n node in the cluster. Use `if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]` to gate\n head-only logic (like launching the coordinator or downloading data).\n\n6. **Setup runs on ALL nodes**: The `setup:` section also runs on all nodes.\n This is typically correct (install dependencies everywhere), but be aware\n of it when doing one-time data preparation.\n\n---\n\n## 3. Managed Jobs Production Patterns\n\nManaged jobs (`sky jobs launch`) provide automatic recovery from preemptions,\nspot instance failures, and transient errors. They run on a lightweight jobs\ncontroller that monitors and relaunches your workload.\n\n### 3.1 job_recovery Configuration\n\nThe `job_recovery` field controls how managed jobs handle failures.\n\n**Strategy options:**\n- `EAGER_NEXT_REGION` (default): On preemption, immediately try a different\n region/cloud before retrying the same region.\n- `FAILOVER`: On preemption, first retry in the same region, then fail over\n to other regions.\n\n**Full configuration as a dict:**\n\n```yaml\n# managed-job-recovery.yaml\nname: resilient-training\n\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n recover_on_exit_codes: [1, 137]\n\nrun: |\n python train.py --checkpoint-dir /ckpts\n```\n\n**Shorthand (strategy only):**\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery: FAILOVER\n```\n\n**Field reference:**\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `strategy` | string | `EAGER_NEXT_REGION` | Recovery strategy name |\n| `max_restarts_on_errors` | int | 0 | Max restarts on non-preemption errors |\n| `recover_on_exit_codes` | int or list[int] | none | Exit codes that always trigger recovery (ignores max_restarts_on_errors counter) |\n\nWhen `recover_on_exit_codes` is set, any job failure with a matching exit code\ntriggers recovery without counting against `max_restarts_on_errors`. This is\nuseful for known transient failures.\n\nTo give up on a job that cannot get resources, set the top-level\n`job.wait_for_scheduling_timeout` (not part of `job_recovery`). The job ends\n`CANCELLED` if it has not started within that long of being submitted, whether\nit is retrying cloud VM launches or waiting in a Kubernetes queue such as\nKueue. Later recoveries do not re-arm it.\n\n```yaml\njob:\n wait_for_scheduling_timeout: 2h # duration (e.g. 90s, 30m, 2h) or int seconds\nresources:\n accelerators: A100:8\n job_recovery: FAILOVER\n```\n\n### 3.2 Checkpointing Patterns\n\nFor spot instances, checkpointing to cloud storage is essential. The pattern\nis: save checkpoints periodically to a mounted bucket, and on restart, resume\nfrom the latest checkpoint.\n\n```yaml\n# checkpointed-training.yaml\nname: checkpointed-llm-training\n\nresources:\n accelerators: A100-80GB:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 5\n\nfile_mounts:\n /ckpts:\n name: my-training-checkpoints # S3/GCS/R2 bucket\n mode: MOUNT\n\nrun: |\n # Resume from latest checkpoint if it exists\n CKPT_ARG=\"\"\n LATEST=$(ls -td /ckpts/checkpoint-* 2>/dev/null | head -1)\n if [ -n \"$LATEST\" ]; then\n echo \"Resuming from checkpoint: $LATEST\"\n CKPT_ARG=\"--resume_from_checkpoint $LATEST\"\n fi\n\n python train.py \\\n --output_dir /ckpts \\\n --save_steps 500 \\\n $CKPT_ARG\n```\n\nLaunch as a managed job:\n\n```bash\nsky jobs launch checkpointed-training.yaml\n```\n\n### 3.3 Hyperparameter Sweeps as Managed Jobs\n\nLaunch multiple managed jobs for a hyperparameter sweep. Each job runs\nindependently with automatic recovery:\n\n```bash\n# Launch sweep via CLI\nfor lr in 1e-4 5e-5 1e-5; do\n for bs in 16 32 64; do\n sky jobs launch sweep.yaml \\\n --env LR=$lr \\\n --env BATCH_SIZE=$bs \\\n -n \"sweep-lr${lr}-bs${bs}\" \\\n -y\n done\ndone\n```\n\n```yaml\n# sweep.yaml\nname: hp-sweep\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 2\n\nenvs:\n LR: 1e-4\n BATCH_SIZE: 32\n\nfile_mounts:\n /results:\n name: sweep-results-bucket\n mode: MOUNT\n\nrun: |\n python train.py \\\n --learning-rate $LR \\\n --batch-size $BATCH_SIZE \\\n --output-dir \"/results/lr${LR}_bs${BATCH_SIZE}\"\n```\n\nMonitor all running jobs:\n\n```bash\nsky jobs queue -o json\n```\n\n### 3.4 Job Pools (Experimental)\n\nJob pools pre-provision clusters that multiple jobs can be submitted to,\navoiding cold-start overhead. Pools are defined with a `service:` section\ncontaining `pool:` configuration.\n\n**Create a pool:**\n\n```yaml\n# gpu-pool.yaml\nname: gpu-pool\n\nresources:\n accelerators: A100:8\n\nservice:\n pool:\n workers: 3\n```\n\n```bash\n# Apply the pool configuration\nsky jobs pool apply gpu-pool.yaml -p my-gpu-pool\n```\n\n**Submit jobs to the pool:**\n\n```bash\nsky jobs launch job.yaml --pool my-gpu-pool\n```\n\n**Pool with autoscaling:**\n\n```yaml\n# autoscaling-pool.yaml\nname: autoscaling-pool\n\nresources:\n accelerators: A100:4\n\nservice:\n pool:\n min_workers: 1\n max_workers: 10\n queue_length_threshold: 2\n upscale_delay_seconds: 60\n downscale_delay_seconds: 300\n```\n\n**Pool management commands:**\n\n```bash\n# Check pool status\nsky jobs pool status\n\n# Check a specific pool\nsky jobs pool status my-gpu-pool\n\n# Update number of workers\nsky jobs pool apply -p my-gpu-pool --workers 5\n\n# Delete a pool\nsky jobs pool down my-gpu-pool\n\n# Delete all pools\nsky jobs pool down -a\n```\n\n---\n\n## 4. SkyServe Advanced\n\nSkyServe deploys model serving endpoints with autoscaling, multi-cloud\nreplicas, rolling updates, and spot instance support.\n\n### 4.1 Autoscaling Configuration\n\n```yaml\n# autoscaling-service.yaml\nname: llm-service\n\nresources:\n accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe:\n path: /health\n initial_delay_seconds: 300\n timeout_seconds: 10\n replica_policy:\n min_replicas: 2\n max_replicas: 10\n target_qps_per_replica: 5.0\n upscale_delay_seconds: 120\n downscale_delay_seconds: 600\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-2-7b-hf \\\n --port 8080\n```\n\n**Autoscaling field reference:**\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `min_replicas` | int | 1 | Minimum number of replicas |\n| `max_replicas` | int | same as min | Maximum number of replicas |\n| `target_qps_per_replica` | float | required if max != min | Target QPS per replica for scaling |\n| `upscale_delay_seconds` | int | 300 | Seconds target must exceed before scaling up |\n| `downscale_delay_seconds` | int | 1200 | Seconds target must be below before scaling down |\n| `num_overprovision` | int | none | Extra replicas beyond what autoscaling calculates |\n\nDeploy and manage:\n\n```bash\n# Deploy the service\nsky serve up autoscaling-service.yaml -n my-llm\n\n# Check service status\nsky serve status my-llm\n\n# Get the service endpoint\nsky serve status my-llm --endpoint\n\n# Tail controller logs\nsky serve logs --controller my-llm\n```\n\n### 4.2 Rolling Updates vs Blue-Green Updates\n\n**Rolling update** (default): Gradually replaces replicas one by one. Some\nold and new replicas serve traffic simultaneously during the transition.\n\n```bash\n# Update with rolling strategy (default)\nsky serve update my-llm updated-service.yaml\n\n# Explicitly specify rolling\nsky serve update --mode rolling my-llm updated-service.yaml\n```\n\n**Blue-green update**: Provisions all new replicas first, then switches\ntraffic atomically. No mixed versions serve traffic simultaneously.\n\n```bash\n# Blue-green update\nsky serve update --mode blue_green my-llm updated-service.yaml\n```\n\nBlue-green requires enough quota to run both old and new replicas\nsimultaneously during the transition.\n\n### 4.3 Spot Replicas for Cost Savings\n\nUse spot instances for serving replicas with automatic on-demand fallback:\n\n```yaml\n# spot-serving.yaml\nname: spot-llm-service\n\nresources:\n accelerators: A100:1\n use_spot: true\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 3\n max_replicas: 8\n target_qps_per_replica: 5.0\n # Maintain 1 on-demand replica as a baseline\n base_ondemand_fallback_replicas: 1\n # Dynamically add on-demand replicas if spot is unavailable\n dynamic_ondemand_fallback: true\n\nrun: |\n python serve.py --port 8080\n```\n\n### 4.4 TLS/HTTPS Configuration\n\nEnable TLS termination at the load balancer:\n\n```yaml\n# tls-service.yaml\nname: secure-service\n\nresources:\n accelerators: L4:1\n ports:\n - 8080\n\nfile_mounts:\n /certs/server.key: ./certs/server.key\n /certs/server.crt: ./certs/server.crt\n\nservice:\n readiness_probe: /health\n replicas: 2\n tls:\n keyfile: /certs/server.key\n certfile: /certs/server.crt\n\nrun: |\n python serve.py --port 8080\n```\n\n### 4.5 Readiness Probe Configuration\n\nThe readiness probe determines when a replica is ready to receive traffic.\n\n**Simple GET probe (most common):**\n\n```yaml\nservice:\n readiness_probe: /health\n replicas: 2\n```\n\n**Full probe configuration:**\n\n```yaml\nservice:\n readiness_probe:\n path: /v1/models\n initial_delay_seconds: 600\n timeout_seconds: 30\n headers:\n Authorization: \"Bearer test-token\"\n replicas: 1\n```\n\n**POST probe (useful for LLMs that need a generation test):**\n\n```yaml\nservice:\n readiness_probe:\n path: /v1/completions\n post_data:\n model: llama-2-7b\n prompt: \"Hello\"\n max_tokens: 1\n initial_delay_seconds: 900\n timeout_seconds: 60\n replicas: 1\n```\n\n**Default values:**\n- `initial_delay_seconds`: 1200 (20 minutes) -- readiness failures during\n this period are ignored\n- `timeout_seconds`: 15\n\n### 4.6 Multi-Cloud Replicas\n\nDeploy replicas across multiple clouds for availability and cost optimization:\n\n```yaml\n# multi-cloud-serving.yaml\nname: multi-cloud-llm\n\nresources:\n any_of:\n - infra: aws\n accelerators: A100:1\n - infra: gcp\n accelerators: A100:1\n - infra: azure\n accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 3\n max_replicas: 10\n target_qps_per_replica: 5.0\n\nrun: |\n python serve.py --port 8080\n```\n\nSkyPilot distributes replicas across available clouds for redundancy and\npicks the cheapest options.\n\n### 4.7 Load Balancing Policies\n\nSkyServe supports different load balancing policies:\n\n```yaml\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 2\n max_replicas: 8\n target_qps_per_replica: 5.0\n load_balancing_policy: round_robin\n```\n\nFor heterogeneous GPU deployments, use instance-aware load balancing:\n\n```yaml\n# Instance-aware balancing across different GPU types\nresources:\n any_of:\n - accelerators: H100:1\n - accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 2\n max_replicas: 10\n target_qps_per_replica:\n H100: 10.0\n A100: 5.0\n load_balancing_policy: instance_aware_least_load\n```\n\n---\n\n## 5. Kubernetes Integration\n\nSkyPilot treats Kubernetes clusters as first-class infrastructure alongside\ncloud providers.\n\n### 5.1 Using K8s Contexts\n\nSkyPilot uses your kubeconfig contexts. Reference them with `infra: k8s/`:\n\n```yaml\n# Use a specific K8s context\nresources:\n infra: k8s/my-cluster-context\n accelerators: A100:4\n```\n\n```yaml\n# Use the default K8s context (current-context in kubeconfig)\nresources:\n infra: kubernetes\n accelerators: A100:4\n```\n\n```yaml\n# EKS context (contexts can contain special characters)\nresources:\n infra: k8s/arn:aws:eks:us-east-1:123456789012:cluster/my-cluster\n accelerators: A100:4\n```\n\n### 5.2 Pod Config Overrides\n\nCustomize the pod spec for SkyPilot pods via `~/.sky/config.yaml`. This\nallows setting tolerations, nodeSelectors, labels, and other Kubernetes-\nspecific configurations.\n\n```yaml\n# ~/.sky/config.yaml\nkubernetes:\n allowed_contexts:\n - my-production-cluster\n - my-dev-cluster\n context_configs:\n my-production-cluster:\n pod_config:\n spec:\n tolerations:\n - key: \"nvidia.com/gpu\"\n operator: \"Exists\"\n effect: \"NoSchedule\"\n nodeSelector:\n node-type: gpu\n team: ml-training\n containers:\n - name: skypilot\n resources:\n limits:\n nvidia.com/gpu: \"1\"\n metadata:\n labels:\n team: ml-platform\n environment: production\n custom_metadata:\n labels:\n app: skypilot\n managed-by: skypilot\n my-dev-cluster:\n pod_config:\n spec:\n nodeSelector:\n node-type: dev-gpu\n```\n\n### 5.3 Multi-Cluster K8s with any_of\n\nSpread workloads across multiple Kubernetes clusters:\n\n```yaml\n# multi-k8s.yaml\nname: multi-cluster-training\n\nresources:\n any_of:\n - infra: k8s/us-east-cluster\n accelerators: A100:8\n - infra: k8s/eu-west-cluster\n accelerators: A100:8\n - infra: k8s/asia-cluster\n accelerators: H100:8\n\nrun: |\n python train.py\n```\n\nCombined with cloud fallback:\n\n```yaml\nresources:\n ordered:\n - infra: k8s/onprem-cluster\n accelerators: A100:8\n - infra: k8s/gke-cluster\n accelerators: A100:8\n - infra: aws\n accelerators: A100:8\n```\n\n### 5.4 GPU Detection and Labeling\n\nSkyPilot needs GPU labels on Kubernetes nodes to schedule GPU workloads.\nUse `sky gpus label` to automatically detect and label GPUs:\n\n```bash\n# Label GPUs in the current Kubernetes context\nsky gpus label\n\n# Label GPUs in a specific context\nsky gpus label --context my-k8s-cluster\n\n# Start labeling without waiting for completion\nsky gpus label --async\n\n# Cleanup labeling resources when done\nsky gpus label --cleanup\n```\n\nCheck available GPUs across all configured infrastructure:\n\n```bash\n# List all available GPUs\nsky gpus list\n\n# Show GPUs for a specific type\nsky gpus list A100\n\n# Show GPUs available on Kubernetes\nsky gpus list --infra kubernetes\n```\n\n### 5.5 Kubernetes Configuration in ~/.sky/config.yaml\n\n```yaml\n# ~/.sky/config.yaml\nkubernetes:\n # Restrict which contexts SkyPilot can use\n allowed_contexts:\n - gke-prod\n - eks-staging\n\n # Port exposure mode\n ports: loadbalancer # or nodeport, ingress, podip\n\n # Cluster autoscaler integration\n autoscaler: gke # or karpenter, generic\n\n # Per-context configuration\n context_configs:\n gke-prod:\n provision_timeout: 600\n pod_config:\n spec:\n nodeSelector:\n cloud.google.com/gke-accelerator: nvidia-tesla-a100\n```\n\n---\n\n## 6. API Server and Team Setup\n\nThe SkyPilot API server enables team collaboration by providing a shared\ncontrol plane for launching and managing clusters, jobs, and services.\n\n### 6.1 Local API Server\n\nFor individual use or development:\n\n```bash\n# Start the local API server\nsky api start\n\n# Check server status\nsky api status\n\n# Stop the server\nsky api stop\n```\n\nThe local API server runs on port 46580 by default. It starts automatically\nwhen you run any `sky` command.\n\n### 6.2 Remote API Server with Helm\n\nDeploy a shared API server on Kubernetes for team use:\n\n```bash\n# Add the SkyPilot Helm repo\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\n\n# Build chart dependencies\nhelm dependency build ./charts/skypilot\n\n# Deploy the API server\nNAMESPACE=skypilot\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace\n```\n\nFor custom images (e.g., testing local changes):\n\n```bash\n# Build and push custom image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 \\\n -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy with custom image\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n --namespace skypilot \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\nUpgrade an existing deployment (always use `--reuse-values`):\n\n```bash\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n### 6.3 PostgreSQL Backend\n\nFor production deployments, use PostgreSQL instead of SQLite:\n\n```bash\n# Create a secret with the database connection string\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/skypilot\"\n\n# Deploy with PostgreSQL backend\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n -n skypilot --create-namespace \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\nOr set the connection string directly (less secure, not recommended for\nproduction):\n\n```bash\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n -n skypilot --create-namespace \\\n --set apiService.dbConnectionString=\"postgresql://user:pass@host:5432/db\"\n```\n\n### 6.4 Authentication\n\nConnect to a remote API server:\n\n```bash\n# OAuth2 browser login\nsky api login -e https://skypilot.example.com\n\n# Service account token login\nsky api login -e https://skypilot.example.com --token sky_abc123...\n```\n\nCheck the current connection:\n\n```bash\nsky api status\n```\n\n---\n\n## 7. Cost Optimization\n\nSkyPilot provides multiple mechanisms to minimize cloud spending while\nmaximizing GPU availability.\n\n### 7.1 Automatic Cloud Selection (Cheapest)\n\nBy default, SkyPilot's optimizer picks the cheapest cloud/region for your\nrequested resources:\n\n```yaml\n# SkyPilot automatically finds the cheapest option\nresources:\n accelerators: A100:4\n```\n\n```bash\n# Preview the optimizer's choice without launching\nsky launch --dryrun task.yaml\n```\n\nThe `--dryrun` flag shows you which cloud, region, instance type, and price\nSkyPilot would choose, without actually provisioning anything.\n\n### 7.2 Spot Instances\n\nSpot/preemptible instances offer 60-90% savings over on-demand:\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n```\n\nFor production spot workloads, combine with managed jobs for automatic\nrecovery:\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n```\n\n```bash\nsky jobs launch spot-training.yaml\n```\n\n### 7.3 Dryrun to Preview Costs\n\nAlways preview costs before launching expensive workloads:\n\n```bash\n# Show what would be launched and estimated cost\nsky launch --dryrun gpu-training.yaml\n\n# Show cost for a specific cloud\nsky launch --dryrun --infra aws gpu-training.yaml\n```\n\n### 7.4 Cost Report\n\nTrack spending across all clusters:\n\n```bash\n# Show cost report for active clusters\nsky cost-report\n\n# Show all clusters including terminated (last 30 days)\nsky cost-report --all\n\n# Show costs for specific number of days\nsky cost-report --days 7\n```\n\n### 7.5 Autostop Configuration\n\nAutomatically stop or terminate idle clusters to avoid waste. Autostop is\nconfigured in the `resources:` section of your YAML.\n\n**Simple autostop (idle minutes):**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop: 30 # Stop after 30 minutes idle\n```\n\n**Full autostop configuration:**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop:\n idle_minutes: 30\n down: true # Terminate instead of just stopping\n wait_for: jobs_and_ssh # Reset timer while jobs or SSH sessions active\n```\n\n**wait_for options:**\n\n| Value | Description |\n|---|---|\n| `jobs_and_ssh` | (default) Wait for running jobs AND open SSH sessions |\n| `jobs` | Only wait for running jobs to finish |\n| `none` | Stop immediately after idle_minutes, regardless of activity |\n\n**Disable autostop:**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop: false\n```\n\n**CLI overrides:**\n\n```bash\n# Set autostop on an existing cluster\nsky autostop my-cluster -i 60\n\n# Set autodown (terminate when idle)\nsky autostop my-cluster -i 30 --down\n\n# Cancel autostop\nsky autostop my-cluster --cancel\n```\n\n### 7.6 Autostop Hooks for Pre-Shutdown Actions\n\nRun a custom script before the cluster stops or terminates. This is useful\nfor saving state, uploading logs, or graceful shutdown procedures:\n\n```yaml\nresources:\n accelerators: A100:4\n autostop:\n idle_minutes: 30\n down: true\n hook: |\n echo \"Cluster shutting down, saving state...\"\n cp -r /tmp/logs /data/saved-logs/\n python cleanup.py\n hook_timeout: 120 # Seconds to wait for hook to complete\n```\n\nThe hook runs before the cluster is stopped or terminated. If the hook\nexceeds `hook_timeout` seconds, it is killed and the shutdown proceeds.\n\n### 7.7 Combining Cost Optimization Strategies\n\nFor maximum savings, combine multiple strategies:\n\n```yaml\n# maximum-savings.yaml\nname: cost-optimized-training\n\nresources:\n any_of:\n - infra: aws\n accelerators: A100:8\n - infra: gcp\n accelerators: A100:8\n - infra: lambda\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 5\n autostop:\n idle_minutes: 10\n down: true\n\nfile_mounts:\n /ckpts:\n name: training-checkpoints\n mode: MOUNT\n\nrun: |\n python train.py --checkpoint-dir /ckpts --resume-latest\n```\n\nLaunch as a managed job for automatic recovery:\n\n```bash\nsky jobs launch maximum-savings.yaml\n```\n\nThis configuration:\n1. Uses `any_of` to find the cheapest cloud\n2. Uses spot instances for 60-90% savings\n3. Recovers automatically from preemptions via managed jobs\n4. Auto-terminates idle clusters\n5. Checkpoints to cloud storage for seamless recovery\n","CLAUDE.md":"# CLAUDE.md - SkyPilot Development Guide\n\nThis document provides guidance for AI assistants working with the SkyPilot codebase.\n\n## Project Overview\n\nSkyPilot is a system to run, manage, and scale AI workloads on any AI infrastructure. It provides a unified interface across 25+ cloud providers (AWS, GCP, Azure, Kubernetes, Slurm, and many others), enabling users to launch compute resources, run jobs, and serve models without vendor lock-in.\n\n**Key capabilities:**\n- **Clusters**: Launch and manage compute clusters across clouds and Kubernetes\n- **Managed jobs**: Run jobs with automatic recovery from preemptions and failures\n- **Multi-cloud and multi-Kubernetes**: Unified interface across 25+ clouds and multiple K8s clusters\n- Cost optimization and GPU availability maximization\n\n## Repository Structure\n\n```\nskypilot/\n├── sky/ # Main source code\n│ ├── __init__.py # Package exports and version\n│ ├── core.py # Core orchestration logic\n│ ├── task.py # Task definition and YAML parsing\n│ ├── resources.py # Cloud resource specifications\n│ ├── optimizer.py # Resource allocation optimizer\n│ ├── execution.py # Task execution pipeline\n│ ├── exceptions.py # Custom exceptions\n│ ├── skypilot_config.py # Configuration management\n│ ├── clouds/ # Cloud provider abstractions (25+ providers)\n│ ├── backends/ # Execution backends (Ray-based VM, Docker)\n│ ├── provision/ # Cloud resource provisioning\n│ ├── jobs/ # Managed job lifecycle\n│ ├── serve/ # Model serving (SkyServe)\n│ ├── skylet/ # On-cluster execution agent\n│ ├── client/ # SDK and CLI\n│ ├── server/ # API server and dashboard backend\n│ ├── dashboard/ # Web UI (Next.js)\n│ ├── utils/ # 50+ utility modules\n│ ├── catalog/ # Cloud pricing and instance catalogs\n│ ├── data/ # Storage and data handling\n│ ├── adaptors/ # Cloud-specific metadata adapters\n│ └── schemas/ # Protobuf definitions and API schemas\n├── tests/ # Test suite\n│ ├── unit_tests/ # Unit tests with subdirectories per module\n│ ├── smoke_tests/ # Quick validation tests\n│ ├── integration_tests/ # End-to-end tests\n│ ├── kubernetes/ # K8s-specific tests\n│ └── conftest.py # Pytest fixtures\n├── examples/ # 50+ usage examples\n├── llm/ # 45+ LLM training/serving examples\n├── docs/ # Sphinx documentation\n├── charts/ # Helm charts for K8s deployment\n└── format.sh # Code formatting script\n```\n\n## Development Setup\n\n### Environment Setup\n\n```bash\n# Create virtual environment with uv (Python 3.8-3.11 supported)\n# --seed is required to ensure pip is installed (needed for building wheels)\nuv venv --seed --python 3.11\nsource .venv/bin/activate\n\n# Install in editable mode with all cloud support\nuv pip install -e \".[all]\"\n# Or specific clouds only:\n# uv pip install -e \".[aws,gcp,kubernetes]\"\n\n# Install development dependencies\nuv pip install -r requirements-dev.txt\n\n# Optional: Install pre-commit hooks\nuv pip install pre-commit\npre-commit install\n```\n\n### Environment Variables\n\n```bash\nexport SKYPILOT_DEV=1 # Enable development mode\nexport SKYPILOT_DEBUG=1 # Enable debug logging\n```\n\n## Code Formatting and Linting\n\n**Always run `format.sh` before committing:**\n\n```bash\nbash format.sh # Format changed files (vs origin/master)\nbash format.sh --all # Format entire codebase\nbash format.sh --files path/to/file.py # Format specific files\n```\n\nThe script runs:\n1. **YAPF** - Google style for all Python code\n2. **isort** - Import sorting (Google profile)\n3. **mypy** - Type checking\n4. **pylint** - Linting with custom rules\n\n### Tool Versions (must match exactly)\n\nFrom `requirements-dev.txt`:\n- yapf==0.32.0\n- pylint==2.14.5\n- mypy==1.19.1\n- isort==5.12.0\n- pylint-quotes==0.2.3\n\n### Excluded from Formatting\n\n- `sky/schemas/generated/` - Auto-generated protobuf files\n- `build/` - Build artifacts\n\n## Testing\n\n### Running Tests\n\n```bash\n# Unit tests (fast, no cloud resources)\npytest tests/unit_tests/\n\n# Specific test file\npytest tests/unit_tests/test_resources.py\n```\n\n### CI Tests via PR Comments\n\nTrigger CI tests on pull requests using comments:\n- `/quicktest-core` - Run quick core tests\n- `/smoke-test` - Run smoke tests (launches cloud clusters)\n- `/smoke-test --kubernetes --postgres` - Test with PostgreSQL backend on Kubernetes\n- `/smoke-test --kubernetes --remote-server --postgres` - Test remote API server with PostgreSQL\n\n### Test Configuration\n\nFrom `pyproject.toml`:\n- Uses pytest-xdist with 16 parallel workers\n- Environment: `SKYPILOT_DEBUG=1`, `SKYPILOT_DEV=1`\n- Buildkite integration for CI\n\n### Checking Buildkite CI Status\n\nTo check CI test results for a PR:\n\n1. **Get Buildkite URL from GitHub**: Check the PR's commit status checks to find the Buildkite build URL (may take a moment to appear after triggering)\n2. **Fetch build logs via Buildkite API**: Use the Buildkite API to retrieve detailed logs and test results\n\n```bash\n# Example: Get build details (requires BUILDKITE_TOKEN)\ncurl -H \"Authorization: Bearer $BUILDKITE_TOKEN\" \\\n \"https://api.buildkite.com/v2/organizations/skypilot/pipelines/skypilot/builds/\"\n```\n\n## Code Style Guidelines\n\n### General Principles\n\n- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)\n- Use f-strings over `.format()` for readability\n- Use `class MyClass:` not `class MyClass(object):`\n- Use `abc` module for abstract classes\n- Use Python typing with `if typing.TYPE_CHECKING:` for type-only imports\n- Always place imports at the top of the file, not inside function definitions. If this causes a circular dependency, try to resolve it (e.g., restructure modules or use `if typing.TYPE_CHECKING:`). Only as a last resort, place the import inside the function with a comment explaining why. Import performance is not a valid reason for in-function imports; use `LazyImport` instead (see below).\n\n### TODOs and FIXMEs\n\nAlways include author attribution:\n```python\n# TODO(username): Description of what needs to be done\n# FIXME(username): Description of what needs fixing\n```\n\n### Exceptions vs Assertions\n\n- Use exceptions for errors (better error messages)\n- Use `assert` only for debugging/proof-checking\n\n### Lazy Imports\n\nFor modules with significant import time (>100ms) that are imported during `import sky`, use `LazyImport` from `sky/adaptors/common.py`:\n\n```python\nfrom sky.adaptors.common import LazyImport\nheavy_module = LazyImport('heavy_module')\n```\n\nMeasure import time with:\n```bash\npython -X importtime -c \"import sky\" 2> import.log\ntuna import.log # pip install tuna\n```\n\n## Architecture Patterns\n\n### Cloud Provider Abstraction\n\nAll cloud providers inherit from `sky.clouds.Cloud` (in `sky/clouds/cloud.py`). Cloud objects are **lightweight and stateless**. Key design principles:\n\n- Methods should be inexpensive to call\n- Don't store heavy state in cloud objects\n- Cache cloud-specific queries in `sky/clouds/utils/` modules\n- Each cloud implements feature flags via `CloudImplementationFeatures`\n\nAdding a new cloud provider:\n1. Create `sky/clouds/.py` inheriting from `Cloud`\n2. Implement required abstract methods\n3. Add provisioning logic in `sky/provision//`\n4. Register in `sky/clouds/__init__.py`\n\n### Client-Server Architecture\n\nSkyPilot uses a client-server model with API versioning:\n\n- Client (SDK/CLI) in `sky/client/`\n- Server in `sky/server/`\n- API version in `sky/server/constants.py`\n\n**Backward Compatibility Rules (from v0.10.0+):**\n- Changes must be backward compatible\n- Bump `API_VERSION` when introducing API changes\n- Use `@versions.minimal_api_version(N)` decorator for new SDK methods\n- Handle version differences with `versions.get_remote_api_version()`\n\n### Protobuf Regeneration\n\nWhen modifying `.proto` files in `sky/schemas/proto/`:\n\n```bash\npython -m grpc_tools.protoc \\\n --proto_path=sky/schemas/generated=sky/schemas/proto \\\n --python_out=. \\\n --grpc_python_out=. \\\n --pyi_out=. \\\n sky/schemas/proto/*.proto\n```\n\n### Dependency Management\n\nDependencies are defined in `sky/setup_files/dependencies.py`:\n\n- **Core dependencies**: Listed in `install_requires`\n- **Cloud-specific**: Defined in `extras_require` (e.g., `aws`, `gcp`, `kubernetes`)\n- **Development**: Listed in `requirements-dev.txt`\n\nWhen updating dependencies:\n\n1. Check version constraints carefully - some packages have breaking changes\n2. Consider Python version compatibility (3.8-3.11)\n3. Test with both minimum and latest allowed versions\n4. Document version constraints with comments explaining why\n\n## Key Modules Reference\n\n| Module | Purpose |\n|--------|---------|\n| `sky/core.py` | Core orchestration (launch, exec, stop, down) |\n| `sky/task.py` | Task YAML parsing and specification |\n| `sky/resources.py` | Resource requirements (GPUs, memory, disk) |\n| `sky/optimizer.py` | Cloud selection and cost optimization |\n| `sky/client/sdk.py` | Python SDK implementation |\n| `sky/client/cli/` | CLI commands |\n| `sky/backends/cloud_vm_ray_backend.py` | Main execution backend for clusters |\n| `sky/provision/provisioner.py` | Resource provisioning |\n| `sky/jobs/` | Managed jobs with recovery and scheduling |\n\n## Pull Request Guidelines\n\n1. **Branch from master**, create descriptive branch name\n2. **Run `format.sh`** before committing\n3. **Add tests** for core system changes\n4. **Run smoke tests** for significant changes\n5. **Include `Tested:` section** in PR description with test plan\n6. **Delete branch** after merging\n\n### PR Description Format\n\nPRs should include:\n- **Summary**: Brief description of changes (1-3 bullet points)\n- **Test plan**: How the changes were tested (commands run, manual verification steps)\n\n**Important**: Always generate a manual test plan describing how to verify the changes work correctly. Include specific commands, expected outputs, or UI verification steps. Whenever possible, add unit tests and smoke tests for the changes.\n\n### Commit Message Format\n\nUse the `[Area] Description` format:\n- `[Core]` for core system changes\n- `[CLI]` for CLI changes\n- `[API]` for API server changes\n- `[Docs]` for documentation\n- `[AWS]`, `[GCP]`, `[Azure]`, `[Kubernetes]` for cloud-specific changes\n- `[Jobs]` for managed jobs\n- `[Dashboard]` for web UI changes\n- `[Serve]` for model serving (SkyServe)\n- `[Test]` for testing changes\n- `[CI]` for CI/CD changes\n\nExamples:\n- `[Core] Fix cluster status refresh logic`\n- `[AWS] Add support for new instance types`\n- `[Docs] Update installation guide`\n\n## API Server Testing\n\n### Local API Server (Recommended for Development)\n\n```bash\n# Always restart API server after code changes to pick up changes\nsky api stop\nsky api start\n\n# Verify server is running\nsky api status\n```\n\n### Dashboard Development\n\n**For local API server development**, rebuild the dashboard before restarting:\n\n```bash\n# Install dependencies (first time or after package.json changes)\nnpm --prefix sky/dashboard install\n\n# Rebuild the dashboard\nnpm --prefix sky/dashboard run build\n\n# Then restart the API server\nsky api stop\nsky api start\n```\n\n**For remote API server (Docker/Kubernetes)**, the Dockerfile automatically builds the dashboard - no manual build needed before `docker build`.\n\nThe dashboard is a Next.js application. For development with hot reloading:\n\n```bash\n# Run dashboard in development mode (separate from API server)\ncd sky/dashboard\nnpm run dev\n```\n\n### Mocking Remote API Server Locally\n\nTo test remote API server behavior locally:\n\n```bash\n# Start local API server (runs on port 46580 by default)\nsky api stop\nsky api start\n\n# Forward to a different port to simulate remote server\nsocat TCP-LISTEN:46590,fork TCP:127.0.0.1:46580 &\n\n# Connect to the forwarded port as if it were a remote server\nsky api login -e http://127.0.0.1:46590\n```\n\n### Remote API Server (Kubernetes Deployment)\n\nFor testing with a remote API server on Kubernetes:\n\n```bash\n# Build local changes\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\nhelm dependency build ./charts/skypilot\n\n# Build Docker image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy (NEW installation)\nNAMESPACE=skypilot\nRELEASE_NAME=skypilot\nhelm upgrade --install $RELEASE_NAME ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n#### Upgrading Existing Deployments\n\n**CRITICAL:** Always use `--reuse-values` to preserve database/credential config:\n\n```bash\n# Upgrade existing deployment (keeps PostgreSQL, auth, etc.)\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n\n# Check current values / rollback if needed\nhelm get values skypilot -n skypilot\nhelm rollback skypilot -n skypilot\n```\n\n#### PostgreSQL Backend\n\n```bash\n# Create connection secret\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/db\"\n\n# Deploy with PostgreSQL\nhelm upgrade --install skypilot ./charts/skypilot -n skypilot \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\n## Critical Code Paths (Handle with Care)\n\nThe following modules contain complex, stateful logic that requires careful review when modifying:\n\n**Managed Jobs Recovery:**\n- `sky/jobs/controller.py` - Job lifecycle management, state transitions, async coordination\n- `sky/jobs/recovery_strategy.py` - Preemption recovery, retry logic for managed jobs\n- `sky/backends/cloud_vm_ray_backend.py` - Execution backend with complex state handling\n\nThese modules handle edge cases like preemption during job submission, controller failures mid-recovery, and race conditions between concurrent operations. Changes can have subtle effects on job reliability.\n\n**API Server Performance & Robustness:**\n- `sky/server/` - API server with memory efficiency, low latency requirements\n- `sky/backends/backend_utils.py` - Cluster status caching, network resilience, SSH handling\n\nThese modules are optimized for performance. Be cautious about adding blocking calls, memory-heavy operations, or changing caching behavior.\n\n**CLI/SDK Interface Design:**\n- `sky/client/cli/` - Command-line interface\n- `sky/client/sdk.py` - Python SDK\n\nBe cautious about adding new interfaces or changing existing UX significantly. Keep the interface clean and minimal for both CLI and SDK.\n\n**Backward Compatibility:**\n- Test with old server + new client, and old client + new server\n- Ensure existing clusters and jobs continue working after server upgrades\n- `/quicktest-core` tests some backward compatibility scenarios\n- See `docs/source/developers/CONTRIBUTING.md` for API versioning guidelines\n\n## Common Pitfalls\n\n1. **Always restart API server after code changes** - Run `sky api stop; sky api start` to pick up changes\n2. **Don't modify `sky/schemas/generated/`** - These are auto-generated\n3. **Match formatter versions exactly** - Version mismatches cause CI failures\n4. **Consider import time** - Heavy imports slow down CLI responsiveness\n5. **API versioning** - Always maintain backward compatibility\n\n## Useful Commands\n\n```bash\n# Profile CLI performance\nuv pip install py-spy\npy-spy record -t -o sky.svg -- python -m sky.cli status\n\n# Check cloud credentials\nsky check\n\n# View cluster status\nsky status\n\n# Launch a cluster\nsky launch \n\n# Execute a command on the cluster\nsky exec -- bash\n\n# SSH into the cluster\nssh \n\n# Launch a managed job\nsky jobs launch \n```\n\n## Documentation\n\n- **User docs**: https://docs.skypilot.co/\n- **Source**: `docs/source/` (Sphinx)\n- **Build docs**: See `.readthedocs.yml`\n\n## Additional Resources\n\n- **Full contributing guide**: `docs/source/developers/CONTRIBUTING.md`\n- **User docs**: https://docs.skypilot.co/\n- **GitHub Issues**: https://github.com/skypilot-org/skypilot/issues\n- **Discussions**: https://github.com/skypilot-org/skypilot/discussions\n- **Slack**: http://slack.skypilot.co\n","agent/skills/skypilot/references/examples.md":"# SkyPilot Examples Reference\n\nCopy-paste-ready YAML examples and CLI commands for common SkyPilot workflows.\n\n---\n\n## 1. Basic Examples\n\n### Minimal Task (Just a Run Command)\n\nThe simplest possible SkyPilot task -- just a command to run.\n\n```yaml\n# minimal.yaml\nname: hello-world\n\nrun: |\n echo \"Hello from SkyPilot!\"\n hostname\n uname -a\n```\n\n```bash\n# Launch on any available cloud (cheapest option selected automatically)\nsky launch -c hello minimal.yaml\n\n# Tear down when done\nsky down hello\n```\n\n### GPU Task with Specific Accelerator\n\nRequest a specific GPU type and count.\n\n```yaml\n# gpu-task.yaml\nname: gpu-check\n\nresources:\n # Request 1x A100 GPU\n accelerators: A100:1\n\nsetup: |\n pip install torch\n\nrun: |\n nvidia-smi\n python -c \"import torch; print(f'CUDA available: {torch.cuda.is_available()}'); print(f'GPU: {torch.cuda.get_device_name(0)}')\"\n```\n\n```bash\n# Launch on any cloud that has A100\nsky launch -c gpu-test gpu-task.yaml\n\n# Or pin to a specific cloud\nsky launch -c gpu-test gpu-task.yaml --infra aws\n\n# Tear down\nsky down gpu-test\n```\n\n### Task with Setup and Workdir\n\nSync local code and install dependencies before running.\n\n```yaml\n# train-task.yaml\nname: my-training\n\n# Sync current directory to ~/sky_workdir on the remote machine\nworkdir: .\n\nresources:\n accelerators: L4:1\n\n# Setup runs once on cluster creation, cached on subsequent runs\nsetup: |\n pip install torch torchvision\n pip install -r requirements.txt\n\n# Run executes your main command\nrun: |\n python train.py --epochs 10\n```\n\n```bash\n# Launch -- setup runs first, then the run command\nsky launch -c train train-task.yaml\n\n# Re-run with different args (setup is cached, only run executes)\nsky exec train train-task.yaml\n```\n\n### Task with Environment Variables and Secrets\n\nPass configuration via envs and sensitive values via secrets.\n\n```yaml\n# env-task.yaml\nname: training-with-config\n\nresources:\n accelerators: A100:1\n\n# Environment variables -- visible in logs and dashboard\nenvs:\n MODEL_NAME: bert-base-uncased\n BATCH_SIZE: \"32\"\n LEARNING_RATE: \"1e-4\"\n\n# Secrets -- redacted in logs and dashboard\nsecrets:\n HF_TOKEN: null # Will be passed via CLI\n WANDB_API_KEY: null # Will be passed via CLI\n\nsetup: |\n pip install transformers datasets wandb\n\nrun: |\n echo \"Training model: $MODEL_NAME with batch size $BATCH_SIZE\"\n wandb login $WANDB_API_KEY\n python train.py \\\n --model $MODEL_NAME \\\n --batch-size $BATCH_SIZE \\\n --lr $LEARNING_RATE\n```\n\n```bash\n# Pass secrets and override envs from the CLI\nHF_TOKEN=hf_abc123 WANDB_API_KEY=wk_xyz789 \\\n sky launch -c train env-task.yaml \\\n --secret HF_TOKEN --secret WANDB_API_KEY \\\n --env BATCH_SIZE=64\n\n# Secrets can also be specified inline\nsky launch -c train env-task.yaml \\\n --secret HF_TOKEN=hf_abc123 \\\n --secret WANDB_API_KEY=wk_xyz789\n```\n\n### Inline Command (No YAML File)\n\nRun a quick command without writing a YAML file.\n\n```bash\n# Launch a GPU instance and run nvidia-smi\nsky launch -c quick --gpus H100 -- nvidia-smi\n\n# Run a Python script on an existing cluster\nsky exec quick -- python -c \"import torch; print(torch.cuda.device_count())\"\n\n# Launch with specific cloud and region\nsky launch -c quick --gpus A100 --infra gcp/us-central1 -- nvidia-smi\n```\n\n### Multi-Node Task\n\nLaunch a task across multiple nodes.\n\n```yaml\n# multi-node.yaml\nname: multi-node-check\n\n# Launch 2 nodes (1 head + 1 worker)\nnum_nodes: 2\n\nresources:\n accelerators: L4:1\n\nrun: |\n echo \"Node rank: $SKYPILOT_NODE_RANK\"\n echo \"Total nodes: $SKYPILOT_NUM_NODES\"\n echo \"All node IPs:\"\n echo \"$SKYPILOT_NODE_IPS\"\n echo \"GPUs per node: $SKYPILOT_NUM_GPUS_PER_NODE\"\n nvidia-smi\n```\n\n```bash\nsky launch -c multi multi-node.yaml\n```\n\n---\n\n## 2. Training Examples\n\n### Single-GPU PyTorch Training\n\nA complete single-GPU training example with local code sync.\n\n```yaml\n# single-gpu-train.yaml\nname: pytorch-single-gpu\n\nworkdir: .\n\nresources:\n accelerators: A100:1\n disk_size: 256\n\nenvs:\n NUM_EPOCHS: \"10\"\n BATCH_SIZE: \"32\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n python train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c train single-gpu-train.yaml\nsky logs train # Stream logs\nsky down train # Tear down when done\n```\n\n### Multi-GPU Single-Node Training\n\nUse all GPUs on a single node with torchrun.\n\n```yaml\n# multi-gpu-train.yaml\nname: pytorch-multi-gpu\n\nworkdir: .\n\nresources:\n # 4x A100 GPUs on a single node\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n NUM_EPOCHS: \"10\"\n BATCH_SIZE: \"64\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_port=12345 \\\n train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c multi-gpu multi-gpu-train.yaml\n```\n\n### Multi-Node Distributed Training (PyTorch DDP)\n\nDistributed training across 2 nodes with 8 GPUs each using torchrun.\n\n```yaml\n# multi-node-ddp.yaml\nname: pytorch-ddp-training\n\nworkdir: .\n\n# 2 nodes, each with 8 GPUs = 16 GPUs total\nnum_nodes: 2\n\nresources:\n accelerators: A100-80GB:8\n disk_size: 512\n\nenvs:\n NUM_EPOCHS: \"20\"\n BATCH_SIZE: \"256\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n # SKYPILOT_NODE_IPS contains all node IPs, one per line\n # The first IP is always the head node (rank 0)\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=12345 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c ddp-train multi-node-ddp.yaml\n\n# SSH into the head node to check progress\nssh ddp-train\n```\n\n### Training with Checkpointing to Cloud Storage\n\nSave checkpoints to S3/GCS for persistence across runs.\n\n```yaml\n# train-with-checkpoints.yaml\nname: training-with-checkpoints\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n CHECKPOINT_BUCKET: my-training-checkpoints\n\nfile_mounts:\n # Mount an S3 bucket for writing checkpoints\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\n # Mount a dataset bucket (read-only streaming)\n /data:\n source: s3://my-training-data/imagenet\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision boto3\n\nrun: |\n # Training script should save checkpoints to /checkpoints\n # and load dataset from /data\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n train.py \\\n --data-dir /data \\\n --checkpoint-dir /checkpoints \\\n --resume-from-checkpoint\n```\n\n```bash\n# Launch training\nsky launch -c train train-with-checkpoints.yaml\n\n# Re-launch after preemption or failure -- resumes from last checkpoint\nsky launch -c train train-with-checkpoints.yaml\n```\n\n### Training on Spot Instances with Auto-Recovery (Managed Job)\n\nRun long training on cheap spot instances with automatic recovery from preemptions.\n\n```yaml\n# spot-train.yaml\nname: spot-training-resnet\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n # Use spot instances for ~3x cost savings\n use_spot: true\n disk_size: 512\n\n # Recovery strategy when spot instance is preempted\n job_recovery:\n strategy: EAGER_NEXT_REGION # Move to next region on preemption\n max_restarts_on_errors: 3 # Retry up to 3 times on user code errors\n\nenvs:\n CHECKPOINT_BUCKET: my-spot-training-ckpts\n\nfile_mounts:\n # Persistent checkpoint storage -- survives preemptions\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision\n\nrun: |\n # IMPORTANT: Your training script must handle checkpoint resume.\n # SkyPilot recovers the cluster; your script recovers the state.\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n train.py \\\n --checkpoint-dir /checkpoints \\\n --resume-from-checkpoint\n```\n\n```bash\n# Launch as a managed job (NOT sky launch -- use sky jobs launch)\nsky jobs launch spot-train.yaml\n\n# Monitor the job\nsky jobs queue\n\n# Stream logs\nsky jobs logs \n```\n\n### Fine-Tuning a HuggingFace Model\n\nFine-tune a HuggingFace model with LoRA, using secrets for authentication.\n\n```yaml\n# finetune-hf.yaml\nname: llama3-finetune\n\nworkdir: .\n\nresources:\n accelerators: A100-80GB:4\n disk_size: 512\n disk_tier: best\n\nenvs:\n MODEL_ID: meta-llama/Meta-Llama-3.1-8B-Instruct\n DATASET: yahma/alpaca-cleaned\n CHECKPOINT_BUCKET: my-finetune-checkpoints\n\n# HuggingFace token is a secret -- redacted in logs/dashboard\nsecrets:\n HF_TOKEN: null # Pass via CLI: --secret HF_TOKEN=hf_xxx\n\nfile_mounts:\n /output:\n name: ${CHECKPOINT_BUCKET}\n mode: MOUNT\n\nsetup: |\n pip install torch==2.4.0 torchvision\n pip install transformers peft trl datasets accelerate\n pip install bitsandbytes\n\n # Download the model (needs HF_TOKEN for gated models)\n huggingface-cli download $MODEL_ID \\\n --token $HF_TOKEN \\\n --local-dir /tmp/model\n\nrun: |\n python finetune_lora.py \\\n --model-path /tmp/model \\\n --dataset $DATASET \\\n --output-dir /output/lora-adapter \\\n --num-epochs 3 \\\n --batch-size 4 \\\n --learning-rate 2e-4\n```\n\n```bash\n# Launch with HF_TOKEN secret\nHF_TOKEN=hf_abc123 sky launch -c finetune finetune-hf.yaml --secret HF_TOKEN\n```\n\n---\n\n## 3. Model Serving Examples\n\n### Basic vLLM Serving (Single GPU)\n\nServe a model with vLLM behind a SkyServe endpoint.\n\n```yaml\n# vllm-serve.yaml\nname: vllm-llama\n\nresources:\n accelerators: A100:1\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\n# SkyServe configuration\nservice:\n # Readiness check endpoint (must return 200 when ready)\n readiness_probe: /v1/models\n # Fixed number of replicas\n replicas: 1\n```\n\n```bash\n# Deploy as a service\nHF_TOKEN=hf_xxx sky serve up vllm-serve.yaml -n llama-svc --secret HF_TOKEN\n\n# Get the endpoint URL\nsky serve status llama-svc --endpoint\n\n# Test the endpoint\nENDPOINT=$(sky serve status llama-svc --endpoint)\ncurl $ENDPOINT/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n }'\n\n# Tear down\nsky serve down llama-svc\n```\n\n### vLLM with Autoscaling\n\nScale replicas automatically based on query load.\n\n```yaml\n# vllm-autoscale.yaml\nname: vllm-autoscaling\n\nresources:\n accelerators: A100:1\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe:\n path: /v1/models\n # Wait 20 minutes for model loading before checking readiness\n initial_delay_seconds: 1200\n replica_policy:\n min_replicas: 1\n max_replicas: 5\n # Scale up when each replica handles more than 5 QPS\n target_qps_per_replica: 5\n # Wait 5 minutes before adding replicas\n upscale_delay_seconds: 300\n # Wait 20 minutes before removing replicas\n downscale_delay_seconds: 1200\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up vllm-autoscale.yaml -n llama-auto --secret HF_TOKEN\n```\n\n### Serving on Spot Instances\n\nUse spot instances for serving replicas to reduce cost.\n\n```yaml\n# vllm-spot-serve.yaml\nname: vllm-spot-serving\n\nresources:\n accelerators: A100:1\n ports: 8000\n # Use spot instances for replicas -- SkyServe handles recovery\n use_spot: true\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe: /v1/models\n replica_policy:\n # Keep at least 2 replicas for high availability with spot\n min_replicas: 2\n max_replicas: 6\n target_qps_per_replica: 5\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up vllm-spot-serve.yaml -n llama-spot --secret HF_TOKEN\n```\n\n### Multi-Model Serving\n\nServe multiple models by deploying separate services.\n\n```yaml\n# serve-llama-70b.yaml\nname: llama-70b-service\n\nresources:\n # 70B model needs multi-GPU with tensor parallelism\n accelerators: A100-80GB:4\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-70B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --tensor-parallel-size 4 \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe: /v1/models\n replicas: 2\n```\n\n```bash\n# Deploy multiple model services\nHF_TOKEN=hf_xxx sky serve up vllm-serve.yaml -n llama-8b --secret HF_TOKEN\nHF_TOKEN=hf_xxx sky serve up serve-llama-70b.yaml -n llama-70b --secret HF_TOKEN\n\n# Check all services\nsky serve status\n\n# Get endpoints\nsky serve status llama-8b --endpoint\nsky serve status llama-70b --endpoint\n```\n\n### TGI (Text Generation Inference) Example\n\nServe a model using HuggingFace TGI with Docker.\n\n```yaml\n# tgi-serve.yaml\nname: tgi-service\n\nresources:\n accelerators: A100:1\n ports: 8080\n\nenvs:\n MODEL_ID: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nrun: |\n docker run --gpus all --shm-size 1g \\\n -p 8080:80 \\\n -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \\\n -v ~/data:/data \\\n ghcr.io/huggingface/text-generation-inference \\\n --model-id $MODEL_ID\n\nservice:\n readiness_probe: /health\n replicas: 2\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up tgi-serve.yaml -n tgi-llama --secret HF_TOKEN\nsky serve status tgi-llama --endpoint\n```\n\n### Service with Custom Readiness Probe (POST)\n\nUse a POST request with custom data for the readiness check.\n\n```yaml\n# custom-probe-serve.yaml\nname: custom-model-service\n\nresources:\n accelerators: A100:1\n ports: 8080\n\nsetup: |\n pip install my-model-server\n\nrun: |\n python -m my_model_server --port 8080\n\nservice:\n readiness_probe:\n path: /v1/predict\n # Use POST with custom payload for readiness checking\n post_data: '{\"text\": \"health check\", \"max_tokens\": 1}'\n # Wait 30 minutes for model loading\n initial_delay_seconds: 1800\n # Timeout for each probe request\n timeout_seconds: 30\n replica_policy:\n min_replicas: 1\n max_replicas: 3\n target_qps_per_replica: 10\n```\n\n```bash\nsky serve up custom-probe-serve.yaml -n custom-svc\n```\n\n---\n\n## 4. File Mounts and Storage Examples\n\n### Sync Local Directory (workdir)\n\nThe workdir syncs your local directory to `~/sky_workdir` on the remote cluster.\n\n```yaml\n# workdir-example.yaml\nname: local-code-sync\n\n# Sync current directory to ~/sky_workdir\nworkdir: .\n\nresources:\n accelerators: L4:1\n\nsetup: |\n pip install -r requirements.txt\n\nrun: |\n # Commands run under ~/sky_workdir\n python train.py\n```\n\n```bash\n# Any changes to local files are synced on each sky launch/exec\nsky launch -c dev workdir-example.yaml\n\n# Re-sync and re-run (setup is cached)\nsky exec dev workdir-example.yaml\n```\n\n### Mount S3 Bucket (mode: MOUNT)\n\nStream data from S3 without downloading -- read-only, low disk usage.\n\n```yaml\n# s3-mount.yaml\nname: s3-streaming\n\nresources:\n accelerators: A100:1\n\nfile_mounts:\n # MOUNT mode: read-only, streams data on access, no local copy\n /data/imagenet:\n source: s3://my-imagenet-bucket/train\n mode: MOUNT\n\nrun: |\n python train.py --data-dir /data/imagenet\n```\n\n```bash\nsky launch -c train s3-mount.yaml\n```\n\n### Mount with Cache (mode: MOUNT_CACHED)\n\nCache accessed data locally for fast random access -- ideal for model weights.\n\n```yaml\n# mount-cached.yaml\nname: cached-model-weights\n\nresources:\n accelerators: A100:1\n disk_size: 512 # Need enough disk for cached data\n\nfile_mounts:\n # MOUNT_CACHED: caches accessed files locally for fast random access\n # Good for model weights that are read repeatedly\n /models/llama:\n source: s3://my-model-weights/llama-3.1-8b\n mode: MOUNT_CACHED\n\nrun: |\n python serve.py --model-path /models/llama\n```\n\n```bash\nsky launch -c serve mount-cached.yaml\n```\n\n### Copy from Cloud Storage (mode: COPY)\n\nDownload data fully -- read-write access, full local copy.\n\n```yaml\n# copy-mount.yaml\nname: full-copy\n\nresources:\n accelerators: A100:1\n # Need enough disk for the full dataset\n disk_size: 1024\n\nfile_mounts:\n # COPY mode: full download, read-write, supports modification\n /data/dataset:\n source: s3://my-dataset-bucket/training-data\n mode: COPY\n\nrun: |\n # Can read AND write to the copied data\n python preprocess.py --input /data/dataset --output /data/dataset/processed\n python train.py --data-dir /data/dataset/processed\n```\n\n```bash\nsky launch -c train copy-mount.yaml\n```\n\n### Mount GCS Bucket\n\nMount a Google Cloud Storage bucket.\n\n```yaml\n# gcs-mount.yaml\nname: gcs-data\n\nresources:\n infra: gcp\n accelerators: A100:1\n\nfile_mounts:\n # GCS bucket mount\n /data:\n source: gs://my-gcs-bucket/datasets\n mode: MOUNT\n\n # Create a new GCS bucket for output\n /output:\n name: my-training-output-bucket\n store: gcs\n mode: MOUNT\n\nrun: |\n python train.py --data /data --output /output\n```\n\n```bash\nsky launch -c gcp-train gcs-mount.yaml\n```\n\n### Using Volumes on Kubernetes\n\nKubernetes persistent and ephemeral volumes.\n\n```yaml\n# k8s-volumes.yaml\nname: k8s-with-volumes\n\nresources:\n infra: k8s\n accelerators: A100:1\n\n# Volumes (Kubernetes only)\nvolumes:\n # Persistent volume -- data survives cluster restarts\n /mnt/data: my-persistent-volume\n\n # Ephemeral volume -- fast scratch space, lost on restart\n /mnt/scratch:\n size: 200Gi\n\nrun: |\n # /mnt/data persists across restarts\n # /mnt/scratch is fast ephemeral storage\n python train.py \\\n --data-dir /mnt/data \\\n --cache-dir /mnt/scratch\n```\n\n```bash\nsky launch -c k8s-train k8s-volumes.yaml\n```\n\n### Multiple Mounts in One Task\n\nCombine different mount types in a single task.\n\n```yaml\n# multi-mount.yaml\nname: multi-mount-training\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n OUTPUT_BUCKET: my-training-outputs\n\nfile_mounts:\n # Sync local config files\n /configs: ./configs\n\n # Stream training data from S3 (read-only)\n /data/train:\n source: s3://my-dataset/train\n mode: MOUNT\n\n # Cache model weights locally for fast loading\n /models/pretrained:\n source: s3://my-models/llama-3.1-8b\n mode: MOUNT_CACHED\n\n # Writable output bucket for checkpoints\n /output:\n name: ${OUTPUT_BUCKET}\n store: s3\n mode: MOUNT\n\n # Copy evaluation data (small, needs read-write)\n /data/eval:\n source: s3://my-dataset/eval\n mode: COPY\n\nsetup: |\n pip install torch transformers datasets\n\nrun: |\n python train.py \\\n --config /configs/train_config.yaml \\\n --data-dir /data/train \\\n --eval-dir /data/eval \\\n --pretrained-model /models/pretrained \\\n --output-dir /output\n```\n\n```bash\nsky launch -c train multi-mount.yaml\n```\n\n---\n\n## 5. Multi-Cloud and Fallback Examples\n\n### any_of: Multiple GPU Types\n\nLet SkyPilot pick the cheapest available GPU from a set.\n\n```yaml\n# any-gpu.yaml\nname: flexible-gpu-training\n\nresources:\n # SkyPilot picks the cheapest available option\n any_of:\n - accelerators: H100:1\n - accelerators: A100-80GB:1\n - accelerators: A100:1\n - accelerators: L40S:1\n\nrun: |\n nvidia-smi\n python train.py\n```\n\n```bash\nsky launch -c train any-gpu.yaml\n```\n\n### any_of: Multiple Clouds for Same GPU\n\nTry different clouds for the same GPU type.\n\n```yaml\n# any-cloud.yaml\nname: multi-cloud-a100\n\nresources:\n accelerators: A100-80GB:8\n any_of:\n - infra: aws/us-east-1\n - infra: gcp/us-central1\n - infra: azure/eastus\n\nrun: |\n nvidia-smi\n python train.py\n```\n\n```bash\nsky launch -c train any-cloud.yaml\n```\n\n### ordered: Strict Priority\n\nTry clouds/regions in a specific order -- first match wins.\n\n```yaml\n# ordered-priority.yaml\nname: priority-training\n\nresources:\n ordered:\n # Try AWS us-east-1 first (our reserved instances)\n - infra: aws/us-east-1\n accelerators: H100:8\n # Then try GCP us-central1\n - infra: gcp/us-central1\n accelerators: H100:8\n # Fall back to A100s on any cloud\n - accelerators: A100-80GB:8\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train ordered-priority.yaml\n```\n\n### K8s + Cloud Fallback\n\nTry Kubernetes first, fall back to cloud if K8s has no capacity.\n\n```yaml\n# k8s-fallback.yaml\nname: k8s-or-cloud\n\nresources:\n accelerators: A100:1\n ordered:\n # Try on-prem K8s cluster first (free)\n - infra: k8s/my-onprem-cluster\n # Fall back to AWS\n - infra: aws\n # Fall back to GCP\n - infra: gcp\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train k8s-fallback.yaml\n```\n\n### Region Preference with Wildcards\n\nUse wildcards to match multiple regions or zones.\n\n```yaml\n# wildcard-region.yaml\nname: us-region-only\n\nresources:\n accelerators: A100:8\n any_of:\n # Any zone in us-east-1\n - infra: aws/us-east-1\n # Any zone in us-west-2\n - infra: aws/us-west-2\n # Any US region on GCP\n - infra: gcp/us-*\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train wildcard-region.yaml\n```\n\n### Cost Optimization: Cheapest Across All Clouds\n\nOmit `infra` to let SkyPilot search all enabled clouds for the cheapest option.\n\n```yaml\n# cheapest.yaml\nname: cheapest-option\n\nresources:\n # No infra specified -- SkyPilot searches all enabled clouds\n accelerators: A100:1\n\nrun: |\n python train.py\n```\n\n```bash\n# Dry run to see the cheapest option without launching\nsky launch cheapest.yaml --dryrun\n\n# Launch on the cheapest cloud/region\nsky launch -c train cheapest.yaml\n\n# Check GPU prices across clouds\nsky gpus list A100\n```\n\n---\n\n## 6. Managed Job Examples\n\n### Basic Managed Job\n\nManaged jobs run on a dedicated controller that monitors and manages your job.\n\n```yaml\n# basic-job.yaml\nname: data-processing\n\nresources:\n cpus: 16+\n memory: 64+\n\nsetup: |\n pip install pandas pyarrow\n\nrun: |\n python process_data.py --output /tmp/results\n```\n\n```bash\n# Launch as a managed job (runs on a separate controller)\nsky jobs launch basic-job.yaml\n\n# Check status\nsky jobs queue\n\n# Stream logs\nsky jobs logs 1\n\n# Cancel the job\nsky jobs cancel 1\n```\n\n### Spot Job with FAILOVER Recovery\n\nAutomatically retry in the same region first, then try other regions.\n\n```yaml\n# spot-failover.yaml\nname: training-failover\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n # FAILOVER: retry same region first, then move to next region\n strategy: FAILOVER\n # Retry up to 3 times on user code errors\n max_restarts_on_errors: 3\n\nfile_mounts:\n /checkpoints:\n name: my-training-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch spot-failover.yaml\n```\n\n### Spot Job with EAGER_NEXT_REGION Recovery\n\nMove to a different region immediately on preemption -- good for spot shortages.\n\n```yaml\n# spot-eager.yaml\nname: training-eager-recovery\n\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n # EAGER_NEXT_REGION: immediately try a different region on preemption\n # Best for spot instances where preemption signals regional shortage\n strategy: EAGER_NEXT_REGION\n\nfile_mounts:\n /checkpoints:\n name: my-training-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch spot-eager.yaml\n```\n\n### Hyperparameter Sweep\n\nLaunch multiple managed jobs with different hyperparameters.\n\n```yaml\n# sweep.yaml\nname: hp-sweep\n\nresources:\n accelerators: A100:1\n\nenvs:\n LR: \"1e-4\"\n WEIGHT_DECAY: \"0.01\"\n BATCH_SIZE: \"32\"\n\nfile_mounts:\n /results:\n name: sweep-results\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers wandb\n\nrun: |\n python train.py \\\n --lr $LR \\\n --weight-decay $WEIGHT_DECAY \\\n --batch-size $BATCH_SIZE \\\n --output-dir /results/lr-${LR}_wd-${WEIGHT_DECAY}_bs-${BATCH_SIZE}\n```\n\n```bash\n# Launch multiple jobs with different hyperparameters\nfor lr in 1e-3 1e-4 1e-5; do\n for wd in 0.01 0.1; do\n sky jobs launch sweep.yaml \\\n --env LR=$lr --env WEIGHT_DECAY=$wd \\\n --name sweep-lr${lr}-wd${wd}\n done\ndone\n\n# Monitor all sweep jobs\nsky jobs queue\n```\n\n### Job with max_restarts_on_errors\n\nAutomatically restart on user code errors (non-zero exit codes).\n\n```yaml\n# auto-restart.yaml\nname: resilient-training\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n strategy: FAILOVER\n # Restart up to 5 times if the training script crashes\n max_restarts_on_errors: 5\n\nfile_mounts:\n /checkpoints:\n name: my-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch auto-restart.yaml\n```\n\n### Job with recover_on_exit_codes for NCCL Errors\n\nAlways retry on specific exit codes (such as NCCL timeouts).\n\n```yaml\n# nccl-recovery.yaml\nname: nccl-resilient-training\n\nnum_nodes: 4\n\nresources:\n accelerators: H100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n # Exit code 33 = NCCL timeout (convention in many training scripts)\n # These restarts do NOT count toward max_restarts_on_errors\n recover_on_exit_codes: [33, 34]\n\nfile_mounts:\n /checkpoints:\n name: my-distributed-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=12345 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch nccl-recovery.yaml\n```\n\n### Job Pool Example\n\nUse a job pool for batch processing with pre-provisioned workers.\n\n```yaml\n# pool.yaml\n# Pool configuration: pre-provisions workers with setup\nworkdir: .\n\nresources:\n accelerators: H100:1\n disk_size: 100\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B\n\nsetup: |\n pip install vllm transformers datasets\n # Pre-download model so all jobs can use it immediately\n huggingface-cli download $MODEL_NAME --local-dir /tmp/model\n\npool:\n workers: 3\n```\n\n```yaml\n# job.yaml\n# Job that runs on the pool\nname: batch-inference\n\nresources:\n accelerators: H100:1\n\nrun: |\n echo \"Processing partition ${SKYPILOT_JOB_RANK} of ${SKYPILOT_NUM_JOBS}\"\n python classify.py \\\n --job-rank ${SKYPILOT_JOB_RANK} \\\n --num-jobs ${SKYPILOT_NUM_JOBS} \\\n --model-path /tmp/model \\\n --output-dir /results\n```\n\n```bash\n# Create the pool with 3 workers\nsky jobs pool apply -p my-pool pool.yaml\n\n# Submit 10 jobs to the pool (distributed across 3 workers)\nsky jobs launch -p my-pool --num-jobs 10 job.yaml\n\n# Check pool status\nsky jobs pool status my-pool\n\n# Check job status\nsky jobs queue\n\n# Tear down pool when done\nsky jobs pool down my-pool\n```\n\n---\n\n## 7. Distributed Training Examples\n\n### PyTorch DDP on 4 Nodes (Complete Example)\n\nFull distributed training with torchrun and NCCL configuration.\n\n```yaml\n# ddp-4node.yaml\nname: ddp-4node-training\n\nworkdir: .\n\nnum_nodes: 4\n\nresources:\n accelerators: H100:8\n disk_size: 1024\n\nenvs:\n # NCCL configuration for optimal multi-node communication\n NCCL_DEBUG: INFO\n NCCL_SOCKET_IFNAME: eth0\n # Training hyperparameters\n GLOBAL_BATCH_SIZE: \"512\"\n NUM_EPOCHS: \"50\"\n CHECKPOINT_BUCKET: my-ddp-checkpoints\n\nfile_mounts:\n # Training data (streaming)\n /data:\n source: s3://my-training-data/preprocessed\n mode: MOUNT\n\n # Checkpoint storage (persistent, writable)\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision\n pip install wandb tensorboard\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_GPUS=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))\n\n echo \"Node rank: $SKYPILOT_NODE_RANK / $SKYPILOT_NUM_NODES\"\n echo \"Master: $MASTER_ADDR\"\n echo \"Total GPUs: $NUM_GPUS\"\n\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=29500 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train_ddp.py \\\n --data-dir /data \\\n --checkpoint-dir /checkpoints \\\n --global-batch-size $GLOBAL_BATCH_SIZE \\\n --epochs $NUM_EPOCHS \\\n --resume\n```\n\n```bash\n# Launch distributed training\nsky launch -c ddp4 ddp-4node.yaml\n\n# SSH into head node\nssh ddp4\n\n# Check worker node\nssh ddp4-worker1\n\n# Monitor logs\nsky logs ddp4\n```\n\n### DeepSpeed ZeRO-3 on 2 Nodes\n\nDistributed training with DeepSpeed, including hostfile generation.\n\n```yaml\n# deepspeed-2node.yaml\nname: deepspeed-zero3\n\nnum_nodes: 2\n\nresources:\n accelerators: A100-80GB:8\n disk_size: 1024\n image_id: docker:nvidia/cuda:12.1.1-devel-ubuntu20.04\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B\n DEEPSPEED_ENVS: \"MODEL_NAME,SKYPILOT_NODE_RANK\"\n\nsetup: |\n pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121\n pip install deepspeed==0.14.4\n pip install transformers datasets accelerate\n sudo apt-get update && sudo apt-get -y install pdsh\n\nrun: |\n # Only the head node (rank 0) launches DeepSpeed\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n # Generate hostfile for DeepSpeed\n HOSTFILE=/tmp/hostfile\n python -c \"\n import os\n n_gpus = os.environ['SKYPILOT_NUM_GPUS_PER_NODE']\n ips = os.environ['SKYPILOT_NODE_IPS'].splitlines()\n with open('$HOSTFILE', 'w') as f:\n for ip in ips:\n f.write(f'{ip} slots={n_gpus}\\n')\n \"\n echo \"Hostfile:\"\n cat $HOSTFILE\n\n # Generate .deepspeed_env for environment propagation\n python -c \"\n import os\n envs = os.environ.get('DEEPSPEED_ENVS', '').split(',')\n with open('.deepspeed_env', 'w') as f:\n for var in envs:\n f.write(f'{var}=\\\"{os.getenv(var, \\\"\\\")}\\\"\\n')\n \"\n\n deepspeed \\\n --hostfile $HOSTFILE \\\n train_deepspeed.py \\\n --model-name $MODEL_NAME \\\n --deepspeed \\\n --deepspeed-config ds_config_zero3.json\n fi\n```\n\n```bash\nsky launch -c ds2 deepspeed-2node.yaml\n```\n\n### FSDP Training Example\n\nMulti-node Fully Sharded Data Parallel training with HuggingFace Accelerate.\n\n```yaml\n# fsdp-train.yaml\nname: fsdp-training\n\nworkdir: .\n\nnum_nodes: 2\n\nresources:\n accelerators: H100:8\n disk_size: 1024\n\nenvs:\n MODEL_ID: meta-llama/Llama-3.1-8B\n CHECKPOINT_BUCKET: my-fsdp-checkpoints\n\nsecrets:\n HF_TOKEN: null\n\nfile_mounts:\n /output:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch --index-url https://download.pytorch.org/whl/cu128\n pip install transformers accelerate trl peft datasets\n pip install deepspeed # Needed for some FSDP optimizations\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_PROCESSES=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))\n\n accelerate launch \\\n --config_file configs/fsdp_config.yaml \\\n --num_machines $SKYPILOT_NUM_NODES \\\n --num_processes $NUM_PROCESSES \\\n --machine_rank $SKYPILOT_NODE_RANK \\\n --main_process_ip $MASTER_ADDR \\\n --main_process_port 29500 \\\n train_fsdp.py \\\n --model-id $MODEL_ID \\\n --output-dir /output \\\n --resume-from-checkpoint\n```\n\n```bash\nHF_TOKEN=hf_xxx sky launch -c fsdp fsdp-train.yaml --secret HF_TOKEN\n```\n\n### Ray Train Multi-Node Example\n\nDistributed training using Ray Train across multiple nodes.\n\n```yaml\n# ray-train.yaml\nname: ray-train-distributed\n\nworkdir: .\n\nnum_nodes: 2\n\nresources:\n accelerators: A100:4\n\nsetup: |\n pip install \"ray[train]\" torch torchvision\n pip install transformers datasets\n\nrun: |\n # SkyPilot sets up a Ray cluster across all nodes automatically.\n # The head node's Ray address is available at the standard port.\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n # Only run training script on the head node;\n # Ray distributes the work to all nodes\n python train_ray.py \\\n --num-workers $((SKYPILOT_NUM_NODES * SKYPILOT_NUM_GPUS_PER_NODE)) \\\n --gpus-per-worker 1\n fi\n```\n\nThe `train_ray.py` script uses `ray.train`:\n\n```python\n# train_ray.py (referenced by the YAML above)\nimport ray\nfrom ray import train\nfrom ray.train.torch import TorchTrainer\nfrom ray.train import ScalingConfig\n\ndef train_func(config):\n import torch\n # Your PyTorch training code here\n # ray.train handles distributed setup automatically\n model = ...\n optimizer = ...\n for epoch in range(config[\"epochs\"]):\n train_epoch(model, optimizer)\n # Report metrics\n train.report({\"loss\": loss, \"epoch\": epoch})\n\nscaling_config = ScalingConfig(\n num_workers=args.num_workers,\n use_gpu=True,\n resources_per_worker={\"GPU\": args.gpus_per_worker},\n)\n\ntrainer = TorchTrainer(\n train_func,\n train_loop_config={\"epochs\": 10},\n scaling_config=scaling_config,\n)\nresult = trainer.fit()\n```\n\n```bash\nsky launch -c ray-train ray-train.yaml\n```\n\n---\n\n## 8. Python SDK Examples\n\n### Basic: Create Task, Set Resources, Launch\n\n```python\nimport sky\n\n# Create a task programmatically\ntask = sky.Task(\n name='hello-gpu',\n run='nvidia-smi && python -c \"import torch; print(torch.cuda.device_count())\"',\n)\ntask.set_resources(sky.Resources(\n accelerators='A100:1',\n))\n\n# Launch returns a request ID (async by default)\nrequest_id = sky.launch(task, cluster_name='my-cluster')\n\n# Wait for the launch to complete\njob_id, handle = sky.get(request_id)\nprint(f'Job ID: {job_id}')\n```\n\n### Launch from YAML File\n\n```python\nimport sky\n\n# Load task from a YAML file\ntask = sky.Task.from_yaml('train.yaml')\n\n# Launch with options\nrequest_id = sky.launch(\n task,\n cluster_name='train-cluster',\n idle_minutes_to_autostop=30, # Auto-stop after 30 min idle\n retry_until_up=True, # Keep trying if resources unavailable\n)\nresult = sky.get(request_id)\n```\n\n### Check Status and Get Cluster Info\n\n```python\nimport sky\n\n# Get all cluster statuses\nrequest_id = sky.status()\nclusters = sky.get(request_id)\n\nfor cluster in clusters:\n print(f\"Cluster: {cluster['name']}\")\n print(f\" Status: {cluster['status']}\")\n print(f\" Resources: {cluster['resources_str']}\")\n print(f\" Autostop: {cluster['autostop']} min\")\n\n# Get status for a specific cluster\nrequest_id = sky.status(cluster_names=['my-cluster'])\nclusters = sky.get(request_id)\n\n# Get job queue for a cluster\nrequest_id = sky.queue('my-cluster')\njobs = sky.get(request_id)\nfor job in jobs:\n print(f\"Job {job['job_id']}: {job['status']}\")\n```\n\n### Batch Submission (Launch Multiple Tasks)\n\n```python\nimport sky\n\n# Launch multiple training runs with different configs\nconfigs = [\n {'lr': 1e-3, 'batch_size': 32},\n {'lr': 1e-4, 'batch_size': 64},\n {'lr': 1e-5, 'batch_size': 128},\n]\n\nrequest_ids = []\nfor i, cfg in enumerate(configs):\n task = sky.Task(\n name=f'sweep-{i}',\n setup='pip install torch transformers',\n run=f'python train.py --lr {cfg[\"lr\"]} --batch-size {cfg[\"batch_size\"]}',\n )\n task.set_resources(sky.Resources(accelerators='A100:1'))\n\n request_id = sky.launch(\n task,\n cluster_name=f'sweep-{i}',\n idle_minutes_to_autostop=10,\n down=True, # Auto-teardown after idle\n )\n request_ids.append(request_id)\n print(f'Launched sweep-{i}: {request_id}')\n\n# Wait for all to complete\nfor rid in request_ids:\n result = sky.get(rid)\n print(f'Completed: {rid}')\n```\n\n### Wait for Completion and Get Results\n\n```python\nimport sky\n\n# Launch a task\ntask = sky.Task.from_yaml('train.yaml')\nrequest_id = sky.launch(task, cluster_name='train')\n\n# stream_and_get streams logs AND waits for completion\nresult = sky.stream_and_get(request_id)\n\n# Check job status\nrequest_id = sky.job_status('train', job_ids=[1])\nstatuses = sky.get(request_id)\nprint(f'Job 1 status: {statuses[1]}')\n\n# Tail logs (blocking, streams to console)\nexit_code = sky.tail_logs('train', job_id=1, follow=True)\nprint(f'Job exit code: {exit_code}')\n```\n\n### Programmatic Managed Job Launch\n\n```python\nimport sky\n\n# Create a managed job task\ntask = sky.Task(\n name='managed-training',\n setup='pip install torch transformers',\n run='python train.py --resume',\n)\ntask.set_resources(sky.Resources(\n accelerators='A100:4',\n use_spot=True,\n job_recovery={\n 'strategy': 'EAGER_NEXT_REGION',\n 'max_restarts_on_errors': 3,\n },\n))\n\n# Launch as a managed job\nrequest_id = sky.jobs.launch(task)\njob_id = sky.get(request_id)\nprint(f'Managed job ID: {job_id}')\n\n# Check managed job queue\nrequest_id = sky.jobs.queue()\njobs = sky.get(request_id)\nfor job in jobs:\n print(f\"Job {job['job_id']}: {job['job_name']} - {job['status']}\")\n```\n\n### DAG Workflow (Task Dependencies)\n\n```python\nimport sky\n\n# Create a DAG with task dependencies\nwith sky.Dag() as dag:\n # Step 1: Data preprocessing\n preprocess = sky.Task(\n name='preprocess',\n setup='pip install pandas pyarrow',\n run='python preprocess.py --output /tmp/data',\n )\n preprocess.set_resources(sky.Resources(cpus='8+', memory='32+'))\n\n # Step 2: Training (depends on preprocessing)\n train = sky.Task(\n name='train',\n setup='pip install torch transformers',\n run='python train.py --data /tmp/data',\n )\n train.set_resources(sky.Resources(accelerators='A100:4'))\n\n # Define dependency: preprocess must finish before train\n preprocess >> train\n\n# Launch the DAG (experimental)\nrequest_id = sky.launch(dag, cluster_name='pipeline')\nsky.get(request_id)\n```\n\n### Autostop Management\n\n```python\nimport sky\n\n# Set autostop: stop after 30 minutes of idleness\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=30,\n)\nsky.get(request_id)\n\n# Set autodown: tear down (not just stop) after 60 minutes idle\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=60,\n down=True,\n)\nsky.get(request_id)\n\n# Cancel autostop\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=-1, # Negative value cancels autostop\n)\nsky.get(request_id)\n\n# Stop and start a cluster\nsky.get(sky.stop('my-cluster'))\nsky.get(sky.start('my-cluster'))\n\n# Tear down a cluster\nsky.get(sky.down('my-cluster'))\n```\n\n### List GPUs Programmatically\n\n```python\nimport sky\n\n# List all available accelerators\nrequest_id = sky.list_accelerators(\n gpus_only=True,\n name_filter='A100',\n)\naccelerators = sky.get(request_id)\n\nfor acc_name, offerings in accelerators.items():\n print(f'\\n{acc_name}:')\n for offering in offerings:\n print(f' {offering}')\n\n# List accelerator counts (how many GPUs per node)\nrequest_id = sky.list_accelerator_counts(\n gpus_only=True,\n name_filter='H100',\n)\ncounts = sky.get(request_id)\nfor acc_name, available_counts in counts.items():\n print(f'{acc_name}: available in counts {available_counts}')\n\n# Check which clouds are enabled\nrequest_id = sky.enabled_clouds()\nclouds = sky.get(request_id)\nprint(f'Enabled clouds: {clouds}')\n```\n\n---\n\n## Quick Reference: SkyPilot Environment Variables\n\nThese variables are automatically set by SkyPilot on every node and are available\nin `setup` and `run` commands:\n\n| Variable | Description | Example Value |\n|----------|-------------|---------------|\n| `SKYPILOT_NODE_RANK` | Rank of the current node (0 = head) | `0` |\n| `SKYPILOT_NODE_IPS` | Newline-separated IPs of all nodes | `10.0.0.1\\n10.0.0.2` |\n| `SKYPILOT_NUM_NODES` | Total number of nodes in the cluster | `2` |\n| `SKYPILOT_NUM_GPUS_PER_NODE` | Number of GPUs on this node | `8` |\n| `SKYPILOT_TASK_ID` | Unique task identifier | `sky-2024-01-15-abc123` |\n| `SKYPILOT_CLUSTER_INFO` | JSON with cluster metadata | `{\"cloud\": \"aws\", ...}` |\n| `SKYPILOT_JOB_RANK` | Job rank within a pool (pool jobs only) | `3` |\n| `SKYPILOT_NUM_JOBS` | Total number of jobs in batch (pool jobs only) | `10` |\n\n## Quick Reference: Common CLI Patterns\n\n```bash\n# Launch and auto-stop after 30 minutes idle\nsky launch -c mycluster task.yaml -i 30\n\n# Launch with auto-teardown (not just stop) after idle\nsky launch -c mycluster task.yaml -i 30 --down\n\n# Launch detached (don't stream logs)\nsky launch -c mycluster task.yaml -d\n\n# Retry until resources are available\nsky launch -c mycluster task.yaml -r\n\n# Override GPU from CLI\nsky launch -c mycluster task.yaml --gpus H100:4\n\n# Override cloud from CLI\nsky launch -c mycluster task.yaml --infra aws/us-east-1\n\n# Execute on existing cluster (skip setup)\nsky exec mycluster another-task.yaml\n\n# Execute inline command on existing cluster\nsky exec mycluster -- python eval.py --checkpoint /tmp/best.pt\n\n# Check GPU pricing\nsky gpus list H100\nsky gpus list A100 --all-regions\nsky gpus list --infra aws\n```\n","agent/skills/skypilot/references/python-sdk.md":"\n\n\n# SkyPilot Python SDK Reference\n\nAll SDK functions return a request ID (future). Use `sky.get(request_id)` to await the result.\n\n```python\nimport sky\n\nrequest_id = sky.launch(sky.Task.from_yaml(\"task.yaml\"))\nresult = sky.get(request_id)\n```\n\n## Cluster Operations\n\n### `sky.launch`\n\n```python\nsky.launch(task: Union['sky.Task', 'sky.Dag'], cluster_name: Optional[str] = None, retry_until_up: bool = False, idle_minutes_to_autostop: Optional[int] = None, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, dryrun: bool = False, down: bool = False, backend: Optional['backends.Backend'] = None, optimize_target: common.OptimizeTarget = common.OptimizeTarget.COST, no_setup: bool = False, clone_disk_from: Optional[str] = None, fast: bool = False, resize: bool = False, _need_confirmation: bool = False, _is_launched_by_jobs_controller: bool = False, _is_launched_by_sky_serve_controller: bool = False, _file_mounts_blob_id: Optional[str] = None, _extra_launch_context: Optional[Dict[str, Any]] = None, _include_credentials: bool = False) -> server_common.RequestId[Tuple[Optional[int], Optional['backends.ResourceHandle']]]\n```\n\nLaunches a cluster or task.\n\nThe task's setup and run commands are executed under the task's workdir\n(when specified, it is synced to remote cluster). The task undergoes job\nqueue scheduling on the cluster.\n\nCurrently, the first argument must be a sky.Task, or (EXPERIMENTAL advanced\nusage) a sky.Dag. In the latter case, currently it must contain a single\ntask; support for pipelines/general DAGs are in experimental branches.\n\n**Example:**\n ```python\n\n import sky\n task = sky.Task(run='echo hello SkyPilot')\n task.set_resources(\n sky.Resources(infra='aws', accelerators='V100:4'))\n sky.launch(task, cluster_name='my-cluster')\n\n\n**Args:**\n task: sky.Task, or sky.Dag (experimental; 1-task only) to launch.\n cluster_name: name of the cluster to create/reuse. If None,\n auto-generate a name.\n retry_until_up: whether to retry launching the cluster until it is\n up.\n idle_minutes_to_autostop: automatically stop the cluster after this\n many minute of idleness, i.e., no running or pending jobs in the\n cluster's job queue. Idleness gets reset whenever setting-up/\n running/pending jobs are found in the job queue. Setting this\n flag is equivalent to running\n ``sky.launch(...)`` and then\n ``sky.autostop(idle_minutes=)``. If set, the autostop\n config specified in the task' resources will be overridden by\n this parameter.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes_to_autostop``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after\n ``idle_minutes_to_autostop``.\n dryrun: if True, do not actually launch the cluster.\n down: Tear down the cluster after all jobs finish (successfully or\n abnormally). If --idle-minutes-to-autostop is also set, the\n cluster will be torn down after the specified idle time.\n Note that if errors occur during provisioning/data syncing/setting\n up, the cluster will not be torn down for debugging purposes. If\n set, the autostop config specified in the task' resources will be\n overridden by this parameter.\n backend: backend to use. If None, use the default backend\n (CloudVMRayBackend).\n optimize_target: target to optimize for. Choices: OptimizeTarget.COST,\n OptimizeTarget.TIME.\n no_setup: if True, do not re-run setup commands.\n clone_disk_from: [Experimental] if set, clone the disk from the\n specified cluster. This is useful to migrate the cluster to a\n different availability zone or region.\n fast: [Experimental] If the cluster is already up and available,\n skip provisioning and setup steps.\n resize: if True, resize the existing cluster to the ``num_nodes``\n specified in the task. Supports both scaling up (adding workers)\n and scaling down (removing workers). Scale-down requires no\n running jobs on the cluster. If True, requires ``cluster_name``\n to be set.\n _need_confirmation: (Internal only) If True, show the confirmation\n prompt.\n\n**Returns:**\n The request ID of the launch request.\n\n**Request Returns:**\n job_id (Optional[int]): the job ID of the submitted job. None if the\n backend is not ``CloudVmRayBackend``, or no job is submitted to the\n cluster.\n handle (Optional[backends.ResourceHandle]): the handle to the cluster.\n None if dryrun.\n\n**Request Raises:**\n exceptions.ClusterOwnerIdentityMismatchError: if the cluster is owned\n by another user.\n exceptions.InvalidClusterNameError: if the cluster name is invalid.\n exceptions.ResourcesMismatchError: if the requested resources\n do not match the existing cluster.\n exceptions.NotSupportedError: if required features are not supported\n by the backend/cloud/cluster.\n exceptions.ResourcesUnavailableError: if the requested resources\n cannot be satisfied. The failover_history of the exception will be set\n as:\n\n 1. Empty: iff the first-ever sky.optimize() fails to find a feasible\n resource; no pre-check or actual launch is attempted.\n\n 2. Non-empty: iff at least 1 exception from either our pre-checks\n (e.g., cluster name invalid) or a region/zone throwing resource\n unavailability.\n\n exceptions.CommandError: any ssh command error.\n exceptions.NoCloudAccessError: if all clouds are disabled.\n\nOther exceptions may be raised depending on the backend.\n\n### `sky.exec`\n\n```python\nsky.exec(task: Union['sky.Task', 'sky.Dag'], cluster_name: Optional[str] = None, dryrun: bool = False, down: bool = False, backend: Optional['backends.Backend'] = None) -> server_common.RequestId[Tuple[Optional[int], Optional['backends.ResourceHandle']]]\n```\n\nExecutes a task on an existing cluster.\n\n**This function performs two actions:**\n\n(1) workdir syncing, if the task has a workdir specified;\n(2) executing the task's ``run`` commands.\n\nAll other steps (provisioning, setup commands, file mounts syncing) are\nskipped. If any of those specifications changed in the task, this function\nwill not reflect those changes. To ensure a cluster's setup is up to date,\nuse ``sky.launch()`` instead.\n\n**Execution and scheduling behavior:**\n\n- The task will undergo job queue scheduling, respecting any specified\n resource requirement. It can be executed on any node of the cluster with\n enough resources.\n- The task is run under the workdir (if specified).\n- The task is run non-interactively (without a pseudo-terminal or\n pty), so interactive commands such as ``htop`` do not work.\n Use ``ssh my_cluster`` instead.\n\n**Args:**\n task: sky.Task, or sky.Dag (experimental; 1-task only) containing the\n task to execute.\n cluster_name: name of an existing cluster to execute the task.\n dryrun: if True, do not actually execute the task.\n down: Tear down the cluster after all jobs finish (successfully or\n abnormally). If --idle-minutes-to-autostop is also set, the\n cluster will be torn down after the specified idle time.\n Note that if errors occur during provisioning/data syncing/setting\n up, the cluster will not be torn down for debugging purposes.\n backend: backend to use. If None, use the default backend\n (CloudVMRayBackend).\n\n**Returns:**\n The request ID of the exec request.\n\n\n**Request Returns:**\n job_id (Optional[int]): the job ID of the submitted job. None if the\n backend is not CloudVmRayBackend, or no job is submitted to\n the cluster.\n handle (Optional[backends.ResourceHandle]): the handle to the cluster.\n None if dryrun.\n\n**Request Raises:**\n ValueError: if the specified cluster is not in UP status.\n sky.exceptions.ClusterDoesNotExist: if the specified cluster does not\n exist.\n sky.exceptions.NotSupportedError: if the specified cluster is a\n controller that does not support this operation.\n\n### `sky.stop`\n\n```python\nsky.stop(cluster_name: str, purge: bool = False, graceful: bool = False, graceful_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nStops a cluster.\n\nData on attached disks is not lost when a cluster is stopped. Billing for\nthe instances will stop, while the disks will still be charged. Those\ndisks will be reattached when restarting the cluster.\n\nCurrently, spot instance clusters cannot be stopped (except for GCP, which\ndoes allow disk contents to be preserved when stopping spot VMs).\n\n**Args:**\n cluster_name: name of the cluster to stop.\n purge: (Advanced) Forcefully mark the cluster as stopped in SkyPilot's\n cluster table, even if the actual cluster stop operation failed on\n the cloud. WARNING: This flag should only be set sparingly in\n certain manual troubleshooting scenarios; with it set, it is the\n user's responsibility to ensure there are no leaked instances and\n related resources.\n\n**Returns:**\n The request ID of the stop request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n RuntimeError: failed to stop the cluster.\n sky.exceptions.NotSupportedError: if the specified cluster is a spot\n cluster, or a TPU VM Pod cluster, or the managed jobs controller.\n\n### `sky.start`\n\n```python\nsky.start(cluster_name: str, idle_minutes_to_autostop: Optional[int] = None, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, retry_until_up: bool = False, down: bool = False, force: bool = False) -> server_common.RequestId['backends.CloudVmRayResourceHandle']\n```\n\nRestart a cluster.\n\nIf a cluster is previously stopped (status is STOPPED) or failed in\nprovisioning/runtime installation (status is INIT), this function will\nattempt to start the cluster. In the latter case, provisioning and runtime\ninstallation will be retried.\n\nAuto-failover provisioning is not used when restarting a stopped\ncluster. It will be started on the same cloud, region, and zone that were\nchosen before.\n\nIf a cluster is already in the UP status, this function has no effect.\n\n**Args:**\n cluster_name: name of the cluster to start.\n idle_minutes_to_autostop: automatically stop the cluster after this\n many minute of idleness, i.e., no running or pending jobs in the\n cluster's job queue. Idleness gets reset whenever setting-up/\n running/pending jobs are found in the job queue. Setting this\n flag is equivalent to running ``sky.launch()`` and then\n ``sky.autostop(idle_minutes=)``. If not set, the\n cluster will not be autostopped.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes_to_autostop``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after\n ``idle_minutes_to_autostop``.\n retry_until_up: whether to retry launching the cluster until it is\n up.\n down: Autodown the cluster: tear down the cluster after specified\n minutes of idle time after all jobs finish (successfully or\n abnormally). Requires ``idle_minutes_to_autostop`` to be set.\n force: whether to force start the cluster even if it is already up.\n Useful for upgrading SkyPilot runtime.\n\n**Returns:**\n The request ID of the start request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: argument values are invalid: (1) if ``down`` is set to True\n but ``idle_minutes_to_autostop`` is None; (2) if the specified\n cluster is the managed jobs controller, and either\n ``idle_minutes_to_autostop`` is not None or ``down`` is True (omit\n them to use the default autostop settings).\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n sky.exceptions.NotSupportedError: if the cluster to restart was\n launched using a non-default backend that does not support this\n operation.\n sky.exceptions.ClusterOwnerIdentitiesMismatchError: if the cluster to\n restart was launched by a different user.\n\n### `sky.down`\n\n```python\nsky.down(cluster_name: str, purge: bool = False, graceful: bool = False, graceful_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nTears down a cluster.\n\nTearing down a cluster will delete all associated resources (all billing\nstops), and any data on the attached disks will be lost. Accelerators\n(e.g., TPUs) that are part of the cluster will be deleted too.\n\n**Args:**\n cluster_name: name of the cluster to down.\n purge: (Advanced) Forcefully remove the cluster from SkyPilot's cluster\n table, even if the actual cluster termination failed on the cloud.\n WARNING: This flag should only be set sparingly in certain manual\n troubleshooting scenarios; with it set, it is the user's\n responsibility to ensure there are no leaked instances and related\n resources.\n graceful: Cancel the user's task but block until MOUNT_CACHED data is\n fully uploaded. This helps with preserving user data integrity.\n graceful_timeout: If not None, sets a timeout for the graceful option\n above (in seconds).\n\n**Returns:**\n The request ID of the down request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n RuntimeError: failed to tear down the cluster.\n sky.exceptions.NotSupportedError: the specified cluster is the managed\n jobs controller.\n\n### `sky.status`\n\n```python\nsky.status(cluster_names: Optional[List[str]] = None, refresh: common.StatusRefreshMode = common.StatusRefreshMode.NONE, all_users: bool = False, *, _include_credentials: bool = False, _summary_response: bool = False) -> server_common.RequestId[List[responses.StatusResponse]]\n```\n\nGets cluster statuses.\n\nIf cluster_names is given, return those clusters. Otherwise, return all\nclusters.\n\n**Each cluster can have one of the following statuses:**\n\n- ``INIT``: The cluster may be live or down. It can happen in the following\n cases:\n\n - Ongoing provisioning or runtime setup. (A ``sky.launch()`` has started\n but has not completed.)\n - Or, the cluster is in an abnormal state, e.g., some cluster nodes are\n down, or the SkyPilot runtime is unhealthy. (To recover the cluster,\n try ``sky launch`` again on it.)\n\n- ``UP``: Provisioning and runtime setup have succeeded and the cluster is\n live. (The most recent ``sky.launch()`` has completed successfully.)\n\n- ``STOPPED``: The cluster is stopped and the storage is persisted. Use\n ``sky.start()`` to restart the cluster.\n\n**Autostop column:**\n\n- The autostop column indicates how long the cluster will be autostopped\n after minutes of idling (no jobs running). If ``to_down`` is True, the\n cluster will be autodowned, rather than autostopped.\n\nGetting up-to-date cluster statuses:\n\n- In normal cases where clusters are entirely managed by SkyPilot (i.e., no\n manual operations in cloud consoles) and no autostopping is used, the\n table returned by this command will accurately reflect the cluster\n statuses.\n\n- In cases where the clusters are changed outside of SkyPilot (e.g., manual\n operations in cloud consoles; unmanaged spot clusters getting preempted)\n or for autostop-enabled clusters, use ``refresh=True`` to query the\n latest cluster statuses from the cloud providers.\n\n**Args:**\n cluster_names: a list of cluster names to query. If not\n provided, all clusters will be queried.\n refresh: whether to query the latest cluster statuses from the cloud\n provider(s).\n all_users: whether to include all users' clusters. By default, only\n the current user's clusters are included.\n _include_credentials: (internal) whether to include cluster ssh\n credentials in the response (default: False).\n\n**Returns:**\n The request ID of the status request.\n\n**Request Returns:**\n cluster_records (List[Dict[str, Any]]): A list of dicts, with each dict\n containing the information of a cluster. If a cluster is found to be\n terminated or not found, it will be omitted from the returned list.\n\n ```python\n\n {\n 'name': (str) cluster name,\n 'launched_at': (int) timestamp of last launch on this cluster,\n 'handle': (ResourceHandle) an internal handle to the cluster,\n 'last_use': (str) the last command/entrypoint that affected this\n cluster,\n 'status': (sky.ClusterStatus) cluster status,\n 'autostop': (int) idle time before autostop,\n 'to_down': (bool) whether autodown is used instead of autostop,\n 'metadata': (dict) metadata of the cluster,\n 'user_hash': (str) user hash of the cluster owner,\n 'user_name': (str) user name of the cluster owner,\n 'resources_str': (str) the resource string representation of the\n cluster,\n }\n\n### `sky.autostop`\n\n```python\nsky.autostop(cluster_name: str, idle_minutes: int, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, down: bool = False, hook: Optional[str] = None, hook_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nSchedules an autostop/autodown for a cluster.\n\nAutostop/autodown will automatically stop or teardown a cluster when it\nbecomes idle for a specified duration. Idleness means there are no\nin-progress (pending/running) jobs in a cluster's job queue.\n\nIdleness time of a cluster is reset to zero, whenever:\n\n- A job is submitted (``sky.launch()`` or ``sky.exec()``).\n\n- The cluster has restarted.\n\n- An autostop is set when there is no active setting. (Namely, either\n there's never any autostop setting set, or the previous autostop setting\n was canceled.) This is useful for restarting the autostop timer.\n\nExample: say a cluster without any autostop set has been idle for 1 hour,\nthen an autostop of 30 minutes is set. The cluster will not be immediately\nautostopped. Instead, the idleness timer only starts counting after the\nautostop setting was set.\n\nWhen multiple autostop settings are specified for the same cluster, the\nlast setting takes precedence.\n\n**Args:**\n cluster_name: name of the cluster.\n idle_minutes: the number of minutes of idleness (no pending/running\n jobs) after which the cluster will be stopped automatically. Setting\n to a negative number cancels any autostop/autodown setting.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after ``idle_minutes``.\n down: if true, use autodown (tear down the cluster; non-restartable),\n rather than autostop (restartable).\n hook: optional script to execute on the remote cluster before autostop.\n The script runs before the cluster is stopped or torn down. If the\n hook fails, autostop will still proceed but a warning will be\n logged.\n hook_timeout: timeout in seconds for hook execution. If None, uses\n DEFAULT_HOOK_TIMEOUT_SECONDS (3600 = 1 hour). The hook will\n be terminated if it exceeds this timeout.\n\n**Returns:**\n The request ID of the autostop request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: if arguments are invalid.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend or the cluster is TPU VM Pod.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.cost_report`\n\n```python\nsky.cost_report(days: Optional[int] = None) -> server_common.RequestId[List[Dict[str, Any]]]\n```\n\nGets all cluster cost reports, including those that have been downed.\n\nThe estimated cost column indicates price for the cluster based on the type\nof resources being used and the duration of use up until the call to\nstatus. This means if the cluster is UP, successive calls to report will\nshow increasing price. The estimated cost is calculated based on the local\ncache of the cluster status, and may not be accurate for the cluster with\nautostop/use_spot set or terminated/stopped on the cloud console.\n\n**Args:**\n days: The number of days to get the cost report for. If not provided,\n the default is 30 days.\n\n**Returns:**\n The request ID of the cost report request.\n\n**Request Returns:**\n cluster_cost_records (List[Dict[str, Any]]): A list of dicts, with each\n dict containing the cost information of a cluster.\n\n ```python\n\n {\n 'name': (str) cluster name,\n 'launched_at': (int) timestamp of last launch on this cluster,\n 'duration': (int) total seconds that cluster was up and running,\n 'last_use': (str) the last command/entrypoint that affected this\n 'num_nodes': (int) number of nodes launched for cluster,\n 'resources': (resources.Resources) type of resource launched,\n 'cluster_hash': (str) unique hash identifying cluster,\n 'usage_intervals': (List[Tuple[int, int]]) cluster usage times,\n 'total_cost': (float) cost given resources and usage intervals,\n }\n\n### `sky.endpoints`\n\n```python\nsky.endpoints(cluster: str, port: Optional[Union[int, str]] = None) -> server_common.RequestId[Dict[int, str]]\n```\n\nGets the endpoint for a given cluster and port number (endpoint).\n\n**Example:**\n ```python\n\n import sky\n request_id = sky.endpoints('test-cluster')\n sky.get(request_id)\n\n\n**Args:**\n cluster: The name of the cluster.\n port: The port number to get the endpoint for. If None, endpoints\n for all ports are returned.\n\n**Returns:**\n The request ID of the endpoints request.\n\n**Request Returns:**\n A dictionary of port numbers to endpoints.\n If port is None, the dictionary contains all\n ports:endpoints exposed on the cluster.\n\n**Request Raises:**\n ValueError: if the cluster is not UP or the endpoint is not exposed.\n RuntimeError: if the cluster has no ports to be exposed or no endpoints\n are exposed yet.\n\n## Job Operations\n\n### `sky.queue`\n\n```python\nsky.queue(cluster_name: str, skip_finished: bool = False, all_users: bool = False) -> server_common.RequestId[List[responses.ClusterJobRecord]]\n```\n\nGets the job queue of a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n skip_finished: if True, skip finished jobs.\n all_users: if True, return jobs from all users.\n\n\n**Returns:**\n The request ID of the queue request.\n\n**Request Returns:**\n job_records (List[responses.ClusterJobRecord]): A list of job records\n for each job in the queue.\n\n ```python\n\n [\n {\n 'job_id': (int) job id,\n 'job_name': (str) job name,\n 'username': (str) username,\n 'user_hash': (str) user hash,\n 'submitted_at': (int) timestamp of submitted,\n 'start_at': (int) timestamp of started,\n 'end_at': (int) timestamp of ended,\n 'resources': (str) resources,\n 'status': (job_lib.JobStatus) job status,\n 'log_path': (str) log path,\n }\n ]\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n ``CloudVmRayBackend``.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n sky.exceptions.CommandError: if failed to get the job queue with ssh.\n\n### `sky.job_status`\n\n```python\nsky.job_status(cluster_name: str, job_ids: Optional[List[int]] = None) -> server_common.RequestId[Dict[Optional[int], Optional['job_lib.JobStatus']]]\n```\n\nGets the status of jobs on a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n job_ids: job ids. If None, get the status of the last job.\n\n**Returns:**\n The request ID of the job status request.\n\n**Request Returns:**\n job_statuses (Dict[Optional[int], Optional[job_lib.JobStatus]]): A\n mapping of job_id to job statuses. The status will be None if the\n job does not exist. If job_ids is None and there is no job on the\n cluster, it will return {None: None}.\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n ``CloudVmRayBackend``.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.cluster_job_spec`\n\n```python\nsky.cluster_job_spec(cluster_name: str, job_id: int) -> server_common.RequestId[Optional[Dict[str, Any]]]\n```\n\nGets the task YAML and entrypoint a cluster job was submitted with.\n\n**Args:**\n cluster_name: name of the cluster.\n job_id: id of the job on the cluster.\n\n**Returns:**\n The request ID of the cluster job spec request.\n\n**Request Returns:**\n spec (Optional[Dict[str, Any]]): None if the cluster does not exist or\n the job's spec was not recorded (e.g. it was submitted by an older\n API server version), otherwise:\n\n ```python\n\n {\n 'task_yaml': (str) the task YAML, with secrets redacted,\n 'entrypoint': (str) the command that submitted the job,\n }\n\n### `sky.cancel`\n\n```python\nsky.cancel(cluster_name: str, all: bool = False, all_users: bool = False, job_ids: Optional[List[int]] = None, _try_cancel_if_cluster_is_init: bool = False) -> server_common.RequestId[None]\n```\n\nCancels jobs on a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n all: if True, cancel all jobs.\n all_users: if True, cancel all jobs from all users.\n job_ids: a list of job IDs to cancel.\n _try_cancel_if_cluster_is_init: (bool) whether to try cancelling the job\n even if the cluster is not UP, but the head node is still alive.\n This is used by the jobs controller to cancel the job when the\n worker node is preempted in the spot cluster.\n\n**Returns:**\n The request ID of the cancel request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: if arguments are invalid.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the specified cluster is a\n controller that does not support this operation.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.tail_logs`\n\n```python\nsky.tail_logs(cluster_name: str, job_id: Optional[int], follow: bool, tail: int = 0, output_stream: Optional['io.TextIOBase'] = None, *, preload_content: bool = True) -> Union[int, Iterator[Optional[str]]]\n```\n\nTails the logs of a job.\n\n**Args:**\n cluster_name: name of the cluster.\n job_id: job id.\n follow: if True, follow the logs. Otherwise, return the logs\n immediately.\n tail: if > 0, tail the last N lines of the logs.\n output_stream: the stream to write the logs to. If None, print to the\n console. Cannot be used with preload_content=False.\n preload_content: if False, returns an Iterator[str | None] containing\n the logs without the function blocking on the retrieval of entire\n log. Iterator returns None when the log has been completely\n streamed. Default True. Cannot be used with output_stream.\n\n**Returns:**\n If preload_content is True:\n Exit code based on success or failure of the job. 0 if success,\n 100 if the job failed. See exceptions.JobExitCode for possible exit\n codes.\n If preload_content is False:\n Iterator[str | None] containing the logs without the function\n blocking on the retrieval of entire log. Iterator returns None\n when the log has been completely streamed.\n\n**Request Raises:**\n ValueError: if arguments are invalid or the cluster is not supported.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.tail_provision_logs`\n\n```python\nsky.tail_provision_logs(cluster_name: str, worker: Optional[int] = None, follow: bool = True, tail: int = 0, output_stream: Optional['io.TextIOBase'] = None) -> int\n```\n\nTails the provisioning logs (provision.log) for a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n worker: worker id in multi-node cluster.\n If None, stream the logs of the head node.\n follow: follow the logs.\n tail: lines from end to tail.\n output_stream: optional stream to write logs.\n**Returns:**\n Exit code 0 on streaming success; raises on HTTP error.\n\n### `sky.tail_hook_logs`\n\n```python\nsky.tail_hook_logs(cluster_name: str, event: Optional[str] = None, follow: bool = True, tail: int = 0) -> int\n```\n\nTails a per-event lifecycle-hook log on the cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n event: one of ``stop``, ``preemption``, ``down``. When None,\n auto-selects whichever log exists on the cluster.\n follow: whether to follow the logs.\n tail: number of lines to display from the end of the log file.\n\n**Returns:**\n Exit code 0 on streaming success; non-zero on failure.\n\n### `sky.download_logs`\n\n```python\nsky.download_logs(cluster_name: str, job_ids: Optional[List[str]]) -> Dict[str, str]\n```\n\nDownloads the logs of jobs.\n\n**Args:**\n cluster_name: (str) name of the cluster.\n job_ids: (List[str]) job ids.\n\n**Returns:**\n The request ID of the download_logs request.\n\n**Request Returns:**\n job_log_paths (Dict[str, str]): a mapping of job_id to local log path.\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n## Infrastructure & Configuration\n\n### `sky.check`\n\n```python\nsky.check(infra_list: Optional[Tuple[str, ...]], verbose: bool, workspace: Optional[str] = None) -> server_common.RequestId[Dict[str, Dict[str, List[str]]]]\n```\n\nChecks the credentials to enable clouds.\n\n**Args:**\n infra: The infra to check.\n verbose: Whether to show verbose output.\n workspace: The workspace to check. If None, all workspaces will be\n checked.\n\n**Returns:**\n The request ID of the check request.\n\n**Request Returns:**\n Dict mapping workspace name to a dict of cloud name to list of\n enabled capability strings (e.g. 'compute', 'storage').\n\n### `sky.enabled_clouds`\n\n```python\nsky.enabled_clouds(workspace: Optional[str] = None, expand: bool = False) -> server_common.RequestId[List[str]]\n```\n\nGets the enabled clouds.\n\n**Args:**\n workspace: The workspace to get the enabled clouds for. If None, the\n active workspace will be used.\n expand: Whether to expand Kubernetes and SSH to list of resource pools.\n\n**Returns:**\n The request ID of the enabled clouds request.\n\n**Request Returns:**\n A list of enabled clouds in string format.\n\n### `sky.list_accelerators`\n\n```python\nsky.list_accelerators(gpus_only: bool = True, name_filter: Optional[str] = None, region_filter: Optional[str] = None, quantity_filter: Optional[int] = None, clouds: Optional[Union[List[str], str]] = None, all_regions: bool = False, require_price: bool = True, case_sensitive: bool = True) -> server_common.RequestId[Dict[str, List['catalog.common.InstanceTypeInfo']]]\n```\n\nLists the names of all accelerators offered by Sky.\n\nThis will include all accelerators offered by Sky, including those\nthat may not be available in the user's account.\n\n**Args:**\n gpus_only: Whether to only list GPU accelerators.\n name_filter: The name filter.\n region_filter: The region filter.\n quantity_filter: The quantity filter.\n clouds: The clouds to list.\n all_regions: Whether to list all regions.\n require_price: Whether to require price.\n case_sensitive: Whether to case sensitive.\n\n**Returns:**\n The request ID of the list accelerator counts request.\n\n**Request Returns:**\n acc_to_instance_type_dict (Dict[str, List[InstanceTypeInfo]]): A\n dictionary of canonical accelerator names mapped to a list of\n instance type offerings. See usage in cli.py.\n\n### `sky.kubernetes_node_info`\n\n```python\nsky.kubernetes_node_info(context: Optional[str] = None) -> server_common.RequestId['models.KubernetesNodesInfo']\n```\n\nGets the resource information for all the nodes in the cluster.\n\nCurrently only GPU resources are supported. The function returns the total\nnumber of GPUs available on the node and the number of free GPUs on the\nnode.\n\nIf the user does not have sufficient permissions to list pods in all\nnamespaces, the function will return free GPUs as -1.\n\n**Args:**\n context: The Kubernetes context. If None, the default context is used.\n\n**Returns:**\n The request ID of the Kubernetes node info request.\n\n**Request Returns:**\n KubernetesNodesInfo: A model that contains the node info map and other\n information.\n\n### `sky.realtime_kubernetes_gpu_availability`\n\n```python\nsky.realtime_kubernetes_gpu_availability(context: Optional[str] = None, name_filter: Optional[str] = None, quantity_filter: Optional[int] = None, is_ssh: Optional[bool] = None) -> server_common.RequestId[List[Tuple[str, List['models.RealtimeGpuAvailability']]]]\n```\n\nGets the real-time Kubernetes GPU availability.\n\n**Returns:**\n The request ID of the real-time Kubernetes GPU availability request.\n\n### `sky.optimize`\n\n```python\nsky.optimize(dag: 'sky.Dag', minimize: common.OptimizeTarget = common.OptimizeTarget.COST, admin_policy_request_options: Optional[admin_policy.RequestOptions] = None) -> server_common.RequestId['sky.Dag']\n```\n\nFinds the best execution plan for the given DAG.\n\n**Args:**\n dag: the DAG to optimize.\n minimize: whether to minimize cost or time.\n admin_policy_request_options: Request options used for admin policy\n validation. This is only required when a admin policy is in use,\n see: https://docs.skypilot.co/en/latest/cloud-setup/policy.html\n\n**Returns:**\n The request ID of the optimize request.\n\n**Request Returns:**\n optimized_dag (str): The optimized DAG in YAML format.\n\n**Request Raises:**\n exceptions.ResourcesUnavailableError: if no resources are available\n for a task.\n exceptions.NoCloudAccessError: if no public clouds are enabled.\n\n### `sky.validate`\n\n```python\nsky.validate(dag: 'sky.Dag', workdir_only: bool = False, admin_policy_request_options: Optional[admin_policy.RequestOptions] = None, _skip_local_paths: bool = False) -> None\n```\n\nValidates the tasks.\n\nThe file paths (workdir and file_mounts) are validated on the client side\nwhile the rest (e.g. resource) are validated on server side.\n\nRaises exceptions if the DAG is invalid.\n\n**Args:**\n dag: the DAG to validate.\n workdir_only: whether to only validate the workdir. This is used for\n `exec` as it does not need other files/folders in file_mounts.\n admin_policy_request_options: Request options used for admin policy\n validation. This is only required when a admin policy is in use,\n see: https://docs.skypilot.co/en/latest/cloud-setup/policy.html\n _skip_local_paths: Do not check the workdir and file mount sources on\n this machine. For callers whose sources are API server paths\n inside an upload blob; the server resolves the blob and checks\n the paths when it runs the request.\n\n### `sky.reload_config`\n\n```python\nsky.reload_config() -> None\n```\n\nReloads the client-side config.\n\n## Storage & Volumes\n\n### `sky.storage_ls`\n\n```python\nsky.storage_ls() -> server_common.RequestId[List[responses.StorageRecord]]\n```\n\nGets the storages.\n\n**Returns:**\n The request ID of the storage list request.\n\n**Request Returns:**\n storage_records (List[responses.StorageRecord]):\n A list of storage records.\n\n### `sky.storage_delete`\n\n```python\nsky.storage_delete(name: str) -> server_common.RequestId[None]\n```\n\nDeletes a storage.\n\n**Args:**\n name: The name of the storage to delete.\n\n**Returns:**\n The request ID of the storage delete request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: If the storage does not exist.\n\n## API Server\n\n### `sky.api_start`\n\n```python\nsky.api_start(*, deploy: bool = False, host: str = '127.0.0.1', foreground: bool = False, metrics: bool = False, metrics_port: Optional[int] = None, enable_basic_auth: bool = False, port: Optional[int] = None) -> None\n```\n\nStarts the API server.\n\nIt checks the existence of the API server and starts it if it does not\nexist.\n\n**Args:**\n deploy: Whether to deploy the API server, i.e. fully utilize the\n resources of the machine.\n host: The host to bind the API server to. Under deploy, the server\n always binds a wildcard address for remote access: ``::`` when an\n IPv6 host is given, otherwise ``0.0.0.0``. Without deploy the host\n is used as-is.\n foreground: Whether to run the API server in the foreground (run in\n the current process).\n metrics: Whether to export metrics of the API server.\n metrics_port: The port to export metrics of the API server.\n enable_basic_auth: Whether to enable basic authentication\n in the API server.\n port: The port to bind the API server to. Defaults to the\n SKYPILOT_API_SERVER_LOCAL_PORT environment variable, or 46580.\n Other client commands only find a server on a non-default port\n if the same environment variable is exported.\n**Returns:**\n None\n\n### `sky.api_stop`\n\n```python\nsky.api_stop() -> None\n```\n\nStops the API server.\n\nIt will do nothing if the API server is remotely hosted.\n\n**Returns:**\n None\n\n### `sky.api_status`\n\n```python\nsky.api_status(request_ids: Optional[List[Union[server_common.RequestId[T], str]]] = None, all_status: bool = False, limit: Optional[int] = None, fields: Optional[List[str]] = None, cluster_name: Optional[str] = None) -> List[payloads.RequestPayload]\n```\n\nLists all requests.\n\n**Args:**\n request_ids: The prefixes of the request IDs of the requests to query.\n If None, all requests are queried.\n all_status: Whether to list all finished requests as well. This argument\n is ignored if request_ids is not None.\n limit: The number of requests to show. If None, show all requests.\n fields: The fields to get. If None, get all fields except\n return_value and error. This argument is ignored if request_ids\n is not None.\n cluster_name: Filter requests by cluster name.\n If None, show all requests.\n\n**Returns:**\n A list of request payloads.\n\n### `sky.api_info`\n\n```python\nsky.api_info() -> responses.APIHealthResponse\n```\n\nGets the server's status, commit and version.\n\n**Returns:**\n A dictionary containing the server's status, commit and version.\n\n ```python\n\n {\n 'status': 'healthy',\n 'api_version': '1',\n 'commit': 'abc1234567890',\n 'version': '1.0.0',\n 'version_on_disk': '1.0.0',\n 'user': {\n 'name': 'test@example.com',\n 'id': '12345abcd',\n },\n }\n\n Note that user may be None if we are not using an auth proxy.\n\n### `sky.api_cancel`\n\n```python\nsky.api_cancel(request_ids: Optional[Union[server_common.RequestId[T], List[server_common.RequestId[T]], str, List[str]]] = None, all_users: bool = False, silent: bool = False) -> server_common.RequestId[List[str]]\n```\n\nAborts a request or all requests.\n\nWith neither request_ids nor all_users, aborts all of the caller's own\nrequests.\n\n**Args:**\n request_ids: The request ID(s) to abort. Can be a single string or a\n list of strings. On a server with per-user auth, an admin can\n abort any request by id and other users only their own; on a\n server without it, anyone can abort any request by id.\n all_users: Whether to abort all requests from all users. On a server\n with per-user auth, a non-admin aborts only their own. Cannot be\n combined with request_ids.\n silent: Whether to suppress the output.\n\n**Returns:**\n The request ID of the abort request itself.\n\n**Request Returns:**\n A list of request IDs that were cancelled.\n\n**Raises:**\n ValueError: If both request_ids and all_users are given.\n\n### `sky.api_server_logs`\n\n```python\nsky.api_server_logs(follow: bool = True, tail: Optional[int] = None) -> None\n```\n\nStreams the API server logs.\n\n**Args:**\n follow: Whether to follow the logs.\n tail: the number of lines to show from the end of the logs.\n If None, show all logs.\n\n**Returns:**\n None\n\n### `sky.api_login`\n\n```python\nsky.api_login(endpoint: Optional[str] = None, relogin: bool = False, service_account_token: Optional[str] = None, no_browser: bool = False) -> None\n```\n\nLogs into a SkyPilot API server.\n\nThis sets the endpoint globally, i.e., all SkyPilot CLI and SDK calls will\nuse this endpoint.\n\nTo temporarily override the endpoint, use the environment variable\n`SKYPILOT_API_SERVER_ENDPOINT` instead.\n\n**Args:**\n endpoint: The endpoint of the SkyPilot API server, e.g.,\n http://1.2.3.4:46580 or https://skypilot.mydomain.com.\n relogin: Whether to force relogin with OAuth2 when enabled.\n service_account_token: Service account token for authentication.\n no_browser: If True, do not attempt to open a browser locally; print\n the auth URL and let the user open it themselves. Skips the\n localhost-callback flow, which requires a local browser.\n\n**Returns:**\n None\n\n### `sky.api_logout`\n\n```python\nsky.api_logout() -> None\n```\n\nLogout of the API server.\n\nClears all cookies and settings stored in ~/.sky/config.yaml\n\n## Utilities\n\n### `sky.dashboard`\n\n```python\nsky.dashboard(starting_page: Optional[str] = None) -> None\n```\n\nStarts the dashboard for SkyPilot.\n\n### `sky.get`\n\n```python\nsky.get(request_id: server_common.RequestId[T]) -> T\n```\n\nWaits for and gets the result of a request.\n\nThis function will not check the server health since /api/get is typically\nnot the first API call in an SDK session and checking the server health\nmay cause GET /api/get being sent to a restarted API server.\n\n**Args:**\n request_id: The request ID of the request to get. May be a full request\n ID or a prefix.\n\n**Returns:**\n The ``Request Returns`` of the specified request. See the documentation\n of the specific requests above for more details.\n\n**Raises:**\n Exception: It raises the same exceptions as the specific requests,\n see ``Request Raises`` in the documentation of the specific requests\n above.\n\n### `sky.stream_and_get`\n\n```python\nsky.stream_and_get(request_id: Optional[server_common.RequestId[T]] = None, log_path: Optional[str] = None, tail: Optional[int] = None, follow: bool = True, output_stream: Optional['io.TextIOBase'] = None, relay_rich_status: bool = False) -> Optional[T]\n```\n\nStreams the logs of a request or a log file and gets the final result.\n\nThis will block until the request is finished. The request id can be a\nprefix of the full request id.\n\n**Args:**\n request_id: The request ID of the request to stream. May be a full\n request ID or a prefix.\n If None, the latest request submitted to the API server is streamed.\n Using None request_id is not recommended in multi-user environments.\n log_path: The path to the log file to stream.\n tail: The number of lines to show from the end of the logs.\n If None, show all logs.\n follow: Whether to follow the logs.\n output_stream: The output stream to write to. If None, print to the\n console.\n relay_rich_status: If True, forward encoded rich-status control payloads\n verbatim to the output instead of rendering a local spinner. Used by\n the managed jobs controller to preserve provisioning spinner codes\n in its per-job log. See\n :func:`sky.utils.rich_utils.decode_rich_status`.\n\n**Returns:**\n The ``Request Returns`` of the specified request. See the documentation\n of the specific requests above for more details.\n If follow is False, will always return None. See note on\n stream_response.\n\n**Raises:**\n Exception: It raises the same exceptions as the specific requests,\n see ``Request Raises`` in the documentation of the specific requests\n above.\n\n### `sky.workspaces`\n\n```python\nsky.workspaces() -> server_common.RequestId[Dict[str, Any]]\n```\n\nGets the workspaces.\n\n### `sky.set_preferred_workspace`\n\n```python\nsky.set_preferred_workspace(preferred: Optional[str]) -> Dict[str, Any]\n```\n\nSets (or clears with None) the user's preferred workspace.\n\n**Args:**\n preferred: workspace name to set as default, or None to clear.\n\n**Returns:**\n ``{'preferred': }`` echoing what was set. Callers that\n need the resolved workspace + accessible list should follow up\n with :func:`get_user_workspace`. Raises if the server rejects\n the change (workspace does not exist, or user lacks permission\n to it).\n\n### `sky.get_user_workspace`\n\n```python\nsky.get_user_workspace(requested: Optional[str] = None) -> Dict[str, Any]\n```\n\nReturns workspace state for the calling user.\n\nMirrors the launch-path precedence — if the caller has an explicit\n``active_workspace``, the server returns that with ``source='explicit'``;\notherwise the resolver runs (preferred / default-fallback /\nsingle-membership).\n\n**Args:**\n requested: explicit active workspace to ask about. ``None`` (the\n default) — the SDK reads your locally-configured\n ``active_workspace`` (the value `skypilot_config` merges\n from ``~/.sky/config.yaml`` + ``./.sky.yaml`` + any\n ``--config active_workspace=X`` override) and forwards it\n on the wire as ``?requested=``. Pass a non-None value to\n query the resolver as if ``active_workspace`` were that\n value, without changing your local config — useful for\n previewing \"what would land if I switched to X\".\n\n**Returns:**\n ``{workspace, source, note, preferred, accessible}``.\n\n * ``workspace``: the workspace the launch path would pick. Can\n be ``None`` when the resolver couldn't pick (no access /\n ambiguous / explicit ``requested`` rejected by RBAC); the\n reason is then in ``note``.\n * ``source``: one of ``WORKSPACE_SOURCE_*`` on success, ``None``\n when ``workspace`` is ``None``.\n * ``note``: optional message — drift on success\n (``preferred 'team-x' not accessible``) or the resolver error\n when ``workspace`` is ``None``.\n * ``preferred``: the persisted preferred workspace (``None`` if\n unset).\n * ``accessible``: sorted list of workspaces the user can launch\n into.\n\n## Other Functions\n\n### `sky.create_debug_dump`\n\n```python\nsky.create_debug_dump(request_ids: Optional[List[str]] = None, cluster_names: Optional[List[str]] = None, managed_job_ids: Optional[List[int]] = None, recent_minutes: Optional[float] = None) -> server_common.RequestId[str]\n```\n\nCreate a debug dump for troubleshooting.\n\nAt least one of ``request_ids``, ``cluster_names``, ``managed_job_ids``,\nor ``recent_minutes`` must be provided.\n\n**Args:**\n request_ids: List of request IDs or prefixes to include in the\n dump. Prefixes are resolved to all matching request IDs on\n the server.\n cluster_names: List of cluster names to include in the dump.\n managed_job_ids: List of managed job IDs to include in the dump.\n recent_minutes: If specified, include all resources active within\n this many minutes.\n\n**Returns:**\n The request ID of the debug dump creation request.\n\n**Request Returns:**\n Path to the created zip file on the server.\n\n### `sky.download_debug_dump`\n\n```python\nsky.do","agent/skills/skypilot/references/job-investigation.md":"# Investigating a Managed Job\n\nHow to answer \"what is my job doing and why\" with as few commands as\npossible. Every command here supports `-o json`; prefer it over parsing\ntable output.\n\n## Which command answers which question\n\n| Question | Command | Look at |\n|----------|---------|---------|\n| What state is job N in, and why? | `sky jobs queue -v -o json` | `status`, `details`, `failure_reason`, `schedule_state`, `recovery_count` |\n| Why is job N still provisioning / pending on the cluster? | `sky jobs queue -v -o json` | `details`, when it starts with `Launching (` |\n| How long has job N been at this? | `sky jobs queue -v -o json` | `submitted_at`, and the duration columns in the table |\n| What did job N print recently? | `sky jobs logs N --tail 100 --no-follow` | stdout/stderr of the task |\n| What did the controller do (provision, recover, retry)? | `sky jobs logs N --controller --no-follow` | controller log |\n| Wait until job N finishes | `sky jobs logs N` (blocks) | exit when the job ends |\n\nDo not poll with `sleep` + `sky jobs queue`. Stream logs to block, or\nfetch `--tail` once after the fact.\n\n`details` is the current explanation, not a history. For the sequence of\ntransitions a job went through — and, on Slurm, how long the allocation\nwaited and on what — the timeline lives behind the `jobs.events` API and is\nsurfaced by the enterprise client.\n\n## Reading `details`\n\n`details` in `sky jobs queue -v -o json` is the one-line explanation of the\ncurrent state. Common shapes:\n\n| `details` | Meaning | Next step |\n|-----------|---------|-----------|\n| `Waiting for other jobs to launch` | Controller is at its concurrent-launch limit | Nothing to fix; the duration columns say how long it has waited |\n| `Waiting for higher priority jobs to launch` | A higher-priority managed job is ahead | Raise `priority` in the task YAML if this job matters more |\n| `In backoff, waiting for resources` | Provisioning failed and is retrying | `sky jobs logs N --controller --no-follow` shows the last failure |\n| `Recovering: ` | Cluster was lost or the task crashed (e.g. OOMKilled) | Fix the cause; recovery is automatic |\n| `Failure: ` | Terminal failure | Read the reason; `sky jobs logs N --no-follow` for the task output |\n| `Cancellation requested by ...` | Someone cancelled it | The requester and request ID are in the text |\n| `Launching (pending: ; partition:

)` | Job is STARTING and Slurm has not allocated nodes yet | See \"Slurm pending reasons\" below |\n| `Launching (nodes allocated; Slurm job on )` | The queue wait is over; the job is still bootstrapping | Nothing to fix. That id is what `sacct -j ` takes if you have login-node access |\n\n## Slurm pending reasons\n\nWhile a managed job's Slurm allocation is queued, `details` carries the\n`squeue` reason and the partition it is queued in — the partition matters\nbecause the same reason means different things in different partitions, and\nby the time anyone looks the allocation may be gone. The common codes:\n\n| Reason | Category | What to do |\n|--------|----------|------------|\n| `Resources`, `Priority`, `ReqNodeNotAvail` | No free capacity in the partition | Wait, or resubmit to another partition via `config.slurm` |\n| `QOSGrpGRES`, `QOSMax*`, `AssocGrp*` | A QoS / account quota is full | Wait for the group's jobs to finish, or use a different QoS/partition |\n| `Dependency`, `DependencyNeverSatisfied` | Waiting on another Slurm job; `NeverSatisfied` will pend forever | Check the dependency job; cancel and resubmit if it failed |\n| `JobHeldUser`, `JobHeldAdmin`, `launch failed requeued held` | Job is held | Ask the admin to `scontrol release`, or cancel and relaunch |\n| `BeginTime` | Scheduled start time not reached | Nothing to fix |\n\nThe reason is recorded each time it changes, so a job that moved from\n`Resources` to `Priority` and back leaves all three behind it — `details`\nshows only the latest.\n\n## Recognising a stuck job\n\n- Repeated `Recovering: ` means a persistent problem (bad node,\n OOM, quota). The controller keeps retrying and will not fix it; read\n `sky jobs logs N --controller --no-follow`.\n- A long gap between STARTING and RUNNING is provisioning time. On Slurm\n that is queue time, and `details` says what the queue is waiting on.\n- `failure_reason` distinguishes the user's program failing from an\n infrastructure failure; do not report the latter as a code bug.\n","agent/skills/skypilot/references/cli-reference.md":"\n\n\n# SkyPilot CLI Reference\n\nComplete reference for all `sky` commands and options.\n\n## Core Cluster Commands\n\n### `sky launch`\n\nLaunch a cluster or task.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--cluster`, `-c` — A cluster name. If provided, either reuse an existing cluster with that name or provision a new cluster with that name. Otherwise provision a new cluster with an autogenerated name.\n- `--dryrun` — If True, do not actually run the job.\n- `--detach-run`, `-d`, `-no-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--docker` — (Deprecated) Local docker support is deprecated. To run locally, create a local Kubernetes cluster with ``sky local up``.\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--priority` — Priority for this task. Accepts either an integer (from -1000 to 1000, or a string priority class name. Passing \"none\" clears both.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--idle-minutes-to-autostop`, `-i` — Automatically stop the cluster after this many minutes of idleness, i.e., no running or pending jobs in the cluster's job queue. Idleness gets reset depending on the ``--wait-for`` flag. Setting this ...\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes-to-autostop``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs an...\n- `--down` — Autodown the cluster: tear down the cluster after all jobs finish (successfully or abnormally). If --idle-minutes-to-autostop is also set, the cluster will be torn down after the specified idle time. ...\n- `--retry-until-up`, `-r` — Whether to retry provisioning infinitely until the cluster is up, if we fail to launch the cluster on any possible region/cloud due to unavailability errors.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--no-setup` — Skip setup phase when (re-)launching cluster.\n- `--clone-disk-from`, `--clone` — [Experimental] Clone disk from an existing cluster to launch a new one. This is useful when the new cluster needs to have the same data on the boot disk as an existing cluster.\n- `--fast` — [Experimental] If the cluster is already up and available, skip provisioning and setup steps.\n- `--resize` — Resize an existing cluster to --num-nodes. Supports both scale-up (adding workers) and scale-down (removing workers). Scale-down requires no running jobs. Requires -c to specify an existing cluster.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workspace`, `-w` — Workspace to launch into. Shorthand for `--config active_workspace=`.\n\n### `sky exec`\n\nExecute a task or command on an existing cluster.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTER` — text\n- `--cluster`, `-c` — This is the same as the positional argument, just for consistency.\n- `ENTRYPOINT` — text\n- `--detach-run`, `-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n\n### `sky status`\n\nShow clusters.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--refresh`, `-r` — Query the latest cluster statuses from the cloud provider(s).\n- `--ip` — Get the IP address of the head node of a cluster. This option will override all other options. For Kubernetes clusters, the returned IP address is the internal IP of the head pod, and may not be acces...\n- `--endpoints` — Get all exposed endpoints and corresponding URLs for acluster. This option will override all other options.\n- `--endpoint` — Get the endpoint URL for the specified port number on the cluster. This option will override all other options.\n- `--show-managed-jobs`, `--no-show-managed-jobs` — Also show recent in-progress managed jobs, if any.\n- `--show-services`, `--no-show-services` — Also show sky serve services, if any.\n- `--show-pools`, `--no-show-pools` — Also show pools, if any.\n- `CLUSTERS` — text\n- `--all-users`, `-u` — Show all clusters, including those not owned by the current user.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n### `sky stop`\n\nStop cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Stop all existing clusters.\n- `--all-users`, `-u` — Stop all existing clusters for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky start`\n\nRestart cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Start all existing clusters.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--idle-minutes-to-autostop`, `-i` — Automatically stop the cluster after this many minutes of idleness, i.e., no running or pending jobs in the cluster's job queue. Idleness gets reset depending on the ``--wait-for`` flag. Setting this ...\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes-to-autostop``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs an...\n- `--down` — Autodown the cluster: tear down the cluster after specified minutes of idle time after all jobs finish (successfully or abnormally). Requires --idle-minutes-to-autostop to be set.\n- `--retry-until-up`, `-r` — Retry provisioning infinitely until the cluster is up, if we fail to start the cluster due to unavailability errors.\n- `--force`, `-f` — Force start the cluster even if it is already UP. Useful for upgrading the SkyPilot runtime on the cluster.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky down`\n\nTear down cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Tear down all existing clusters.\n- `--all-users`, `-u` — Tear down all existing clusters for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--purge`, `-p` — (Advanced) Forcefully remove the cluster(s) from SkyPilot's cluster table, even if the actual cluster termination failed on the cloud. WARNING: This flag should only be set sparingly in certain manual...\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky autostop`\n\nSchedule an autostop or autodown for cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Autostop all existing clusters.\n- `--all-users`, `-u` — Autostop all existing clusters for all users.\n- `--idle-minutes`, `-i` — Set the idle minutes before autostopping the cluster. See the doc above for detailed semantics.\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs and SSH connec...\n- `--cancel` — Cancel any currently active auto{stop,down} setting for the cluster. No-op if there is no active setting.\n- `--down` — Use autodown (tear down the cluster; non-restartable), instead of autostop (restartable).\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky cost-report`\n\nShow estimated costs for launched clusters.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--all`, `-a` — Show all cluster information.\n- `--days` (default: `30`) — Show clusters from the last N days. Default is 30 days. If set to 0, show all clusters.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Job Queue Commands\n\n### `sky queue`\n\nShow the job queue for cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--all-users`, `-u` — Show all users' information in full.\n- `--skip-finished`, `-s` — Show only pending/running jobs' information.\n- `CLUSTERS` — text\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n### `sky logs`\n\nTail the log of a job.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--provision` — Stream the cluster provisioning logs (provision.log).\n- `--hook` — Stream a per-event lifecycle-hook log from the cluster. Omit the event name to auto-select whichever log exists.\n- `--autostop` — [DEPRECATED] Alias for `--hook stop`. The autostop event was renamed to `stop` in the lifecycle-hooks framework.\n- `--worker`, `-w` — The worker ID to stream the logs from. If not set, stream the logs of the head node.\n- `--sync-down`, `-s` — Sync down the logs of a job to the local machine. For a distributed job, a separate log file from each worker will be downloaded.\n- `--status` — If specified, do not show logs but exit with a status code for the job's status: 0 for succeeded, or 1 for all other statuses.\n- `--follow`, `--no-follow` — Follow the logs of a job. If --no-follow is specified, print the log so far and exit. [default: --follow]\n- `--tail` (default: `-1`) — Number of lines to display from the end of the log file. Default is the last 1000 lines — sensible for multi-GB logs where downloading the full file is slow. Pass --tail 0 to print the entire log.\n- `CLUSTER` — text\n- `JOB_IDS` — text\n\n### `sky cancel`\n\nCancel job(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTER` — text\n- `--all`, `-a` — Cancel all jobs from current user on the specified cluster.\n- `--all-users`, `-u` — Cancel all jobs on the specified cluster for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `JOBS` — integer\n\n## Infrastructure Commands\n\n### `sky check`\n\nCheck which clouds are available to use.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `INFRA_LIST` — text\n- `--verbose`, `-v` — Show the activated account for each cloud.\n- `--workspace`, `-w` — The workspace to check. If None, all workspaces will be checked.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Dashboard\n\n### `sky dashboard`\n\nOpens the SkyPilot dashboard.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n\n## GPU/Accelerator Commands\n\nSkyPilot GPU/Accelerator CLI.\n\n### `sky gpus label`\n\nLabel GPU nodes in a Kubernetes cluster for use with SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--context`, `-c` — Kubernetes context to use. If not specified, uses the current context from the API server.\n- `--cleanup` — Only cleanup existing GPU labeler resources.\n- `--async` — Do not wait for GPU labeling to complete.\n\n### `sky gpus list`\n\nShow supported GPU/TPU/accelerators and their prices.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ACCELERATOR_STR` — text\n- `--all`, `-a` — Show details of all GPU/TPU/accelerator offerings.\n- `--infra` — Infrastructure to query. Examples: \"aws\", \"aws/us-east-1\"\n- `--cloud` — Cloud provider to query.\n- `--region` — The region to use. If not specified, shows accelerators from all regions.\n- `--all-regions` — Show pricing and instance details for a specified accelerator across all regions and clouds.\n- `--verbose`, `-v` — Show all information in full.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Managed Jobs Commands\n\nManaged Jobs CLI (jobs with auto-recovery).\n\n### `sky jobs cancel`\n\nCancel managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--name`, `-n` — Managed job name to cancel.\n- `--pool`, `-p` — Pool name to cancel.\n- `JOB_IDS` — integer\n- `--task` — Cancel one dynamic task of the job (a job launched from inside it), by the index shown in `sky jobs queue` or by name. A declared task cannot be cancelled alone.\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--all`, `-a` — Cancel all managed jobs for the current user.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--all-users`, `-u` — Cancel all managed jobs from all users.\n\n### `sky jobs dashboard`\n\nOpens a dashboard for managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n\n### `sky jobs launch`\n\nLaunch a managed job from a YAML or a command.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--priority` — Priority for this task. Accepts either an integer (from -1000 to 1000, or a string priority class name. Passing \"none\" clears both.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--cluster`, `-c` — Alias for --name, the name of the managed job.\n- `--job-recovery` — Recovery strategy to use for managed jobs.\n- `--detach-run`, `-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--pool`, `-p` — (Experimental; optional) Pool to use for jobs submission.\n- `--num-jobs` — Number of jobs to submit.\n- `--job-group` — Attach to an existing job group, by job id or unique running job name. The job is shown under it and cancelled with it. Defaults to the surrounding job group when launched from inside one.\n- `--no-job-group` — Launch a top-level job even when running inside a job group (do not attach to it).\n- `--depends-on` — Comma-separated managed job IDs to wait for. The job starts once all of them succeed, and is cancelled if any of them ends otherwise. The launch fails if one of them has already ended without succeedi...\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workspace`, `-w` — Workspace to submit the managed job into. Shorthand for `--config active_workspace=`.\n- `--output`, `-o` — Print only the submitted job IDs, for scripts: `id` prints one per line, `json` prints {\"job_ids\": [...]}. Everything else goes to stderr. Implies --detach-run.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky jobs logs`\n\nTail or sync down the log of a managed job.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--name`, `-n` — Managed job name.\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this job; useful for debugging launching/recoveries, etc.\n- `--refresh`, `-r` — Query the latest job logs, restarting the jobs controller if stopped.\n- `--sync-down`, `-s` — Download logs for all jobs shown in the queue.\n- `--tail` (default: `-1`) — Number of lines to display from the end of the log file. Default is the last 1000 lines — sensible for multi-GB logs where downloading the full file is slow. Pass --tail 0 to print the entire log.\n- `JOB_ID` — integer\n- `TASK` — text\n\n### `sky jobs pool` (subgroup)\n\n(Experimental) Pool management commands.\n\n### `sky jobs pool apply`\n\nEither apply a config to a pool for managed jobs submission or update the number of workers in the pool. One of POOL_YAML or --workers must be provided.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `POOL_YAML` — text\n- `--pool`, `-p` — A pool name. Unique for each pool. If not provided, a unique name is autogenerated.\n- `--mode` (default: `rolling`) — Update mode. If \"rolling\", pool will be updated with rolling update. If \"blue_green\", pool will be updated with blue-green update. This option is only valid when the pool is already running.\n- `--workers` — Can be used to update the number of workers in the pool.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky jobs pool down`\n\nDelete pool(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `POOL_NAMES` — text\n- `--all`, `-a` — Delete all pools.\n- `--purge`, `-p` — Tear down pools in failed status.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky jobs pool logs`\n\nTail or sync down logs of a pool.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this pool.\n- `--sync-down`, `-s` — Sync down logs to the local machine. Can be combined with --controller or worker ID to narrow scope.\n- `--tail` — The number of lines to display from the end of the log file. Default is None, which means print all lines.\n- `POOL_NAME` — text\n- `WORKER_IDS` — integer\n\n### `sky jobs pool status`\n\nShow statuses of pools.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `POOL_NAMES` — text\n- `--all`, `-a` — Show all workers.\n\n### `sky jobs queue`\n\nShow statuses of managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--limit`, `-l` (default: `50`) — Number of jobs to show, default is 50, use \"-a/--all\" to show all jobs.\n- `--refresh`, `-r` — Query the latest statuses, restarting the jobs controller if stopped.\n- `--skip-finished` — Show only pending/running jobs' information.\n- `-s`, `--status` — Filter by status, comma-separated (e.g. -s FAILED,FAILED_SETUP). A bare -s (no value) is a deprecated alias for --skip-finished.\n- `--since` — Show only jobs submitted within this time window, relative to now (e.g. \"30m\", \"48h\", \"7d\", \"2w\"). A bare number is seconds. Mutually exclusive with --after.\n- `--after` — Show only jobs submitted at or after this absolute local time (e.g. \"2026-01-13\" or \"2026-01-13 15:30:00\"). Mutually exclusive with --since.\n- `--before` — Show only jobs submitted at or before this absolute local time (e.g. \"2026-01-13\" or \"2026-01-13 15:30:00\").\n- `--infra` — Show only jobs running on this infrastructure. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/u...\n- `--all-users`, `-u` — Show jobs from all users.\n- `--all`, `-a` — Show all jobs.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## SkyServe Commands\n\nSkyServe CLI (multi-region, multi-cloud serving).\n\n### `sky serve down`\n\nTeardown service(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_NAMES` — text\n- `--all`, `-a` — Tear down all services.\n- `--purge`, `-p` — Tear down services in failed status.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--replica-id` — Tear down a given replica\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky serve logs`\n\nTail or sync down logs of a service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this service.\n- `--load-balancer` — Show the load balancer logs of this service.\n- `--sync-down`, `-s` — Sync down logs to the local machine. Can be combined with --controller, --load-balancer, or a replica ID to narrow scope.\n- `--tail` — The number of lines to display from the end of the log file. Default is None, which means print all lines.\n- `SERVICE_NAME` — text\n- `REPLICA_IDS` — integer\n\n### `sky serve status`\n\nShow statuses of SkyServe services.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--endpoint` — Show service endpoint.\n- `SERVICE_NAMES` — text\n\n### `sky serve up`\n\nLaunch a SkyServe service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_YAML` — text\n- `--service-name`, `-n` — A service name. Unique for each service. If not provided, a unique name is autogenerated.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky serve update`\n\nUpdate a SkyServe service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_NAME` — text\n- `SERVICE_YAML` — text\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--mode` (default: `rolling`) — Update mode. If \"rolling\", SkyServe will update the service with rolling update. If \"blue_green\", SkyServe will update the service with blue-green update.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n## Storage Commands\n\nSkyPilot Storage CLI.\n\n### `sky storage delete`\n\nDelete storage objects.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--all`, `-a` — Delete all storage objects.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky storage ls`\n\nList storage objects managed by SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n\n## Volume Commands\n\nSkyPilot Volumes CLI.\n\n### `sky volumes apply`\n\nApply a volume.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--name`, `-n` — Volume name. Override the name defined in the YAML.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, or k8s/context-name.Examples: k8s, k8s/my-context, runpod/US/US-CA-2. Override the infra defined in the YAML.\n- `--type` — Volume type. Override the type defined in the YAML.\n- `--size` — Volume size. Override the size defined in the YAML.\n- `--use-existing`, `--no-use-existing` — Whether to use an existing volume. Override the use_existing defined in the YAML.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky volumes delete`\n\nDelete volumes.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--all`, `-a` — Delete all volumes.\n- `--purge`, `-p` — Forcibly delete the volume from the volumes table even if the deletion API fails.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky volumes ls`\n\nList volumes managed by SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--verbose`, `-v` — Show all information in full.\n- `--refresh`, `-r` — Refresh volume state from cloud APIs before listing. Without this flag, cached data is returned which is updated periodically by the background daemon.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## API Server Commands\n\nSkyPilot API server commands.\n\n### `sky api cancel`\n\nCancel a request running on SkyPilot API server.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `REQUEST_IDS` — text\n- `--all`, `-a` — Cancel all your requests.\n- `--all-users`, `-u` — Cancel all requests from all users. On a server with per-user auth, a non-admin cancels only their own.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky api info`\n\nShows the SkyPilot API server URL.\n\n**Options:**\n\n- `--output`, `-o` (default: `table`) — Output forma","agent/skills/skypilot/references/migrating-from-slurm.md":"# Migrating from Slurm Reference\n\nThis reference is for converting existing Slurm workloads into SkyPilot task\nYAMLs, and for answering \"how do I do X, which I used to do with Slurm?\".\n\nTwo distinct situations, which need different answers:\n\n- **SkyPilot on the user's existing Slurm cluster.** SkyPilot submits to it\n through the login node via `sbatch`. The cluster is unchanged. Most users\n start here — do not assume they want to leave Slurm.\n- **SkyPilot on Kubernetes or a cloud.** The workload moves off Slurm. The\n YAML is the same; the constraints are different (see section 8).\n\nAsk which one applies if it is not obvious. The conversion is nearly identical;\nthe difference shows up in lifecycle (section 7) and what is unsupported\n(section 8).\n\n---\n\n## 1. Conversion Procedure\n\nFollow this order. Do not skip step 5.\n\n1. **Read the whole `sbatch` script**, not just the `#SBATCH` block. The\n `srun` invocations and `module load` lines carry as much information as the\n directives.\n2. **Map the `#SBATCH` directives** to `resources` / `num_nodes` (section 2).\n3. **Map the body**: `module load` → `setup`, environment activation → `setup`,\n the actual work → `run` (section 6).\n4. **Translate `srun`** — this is the step most likely to be done wrong\n (section 5).\n5. **Validate**: `sky launch --dryrun `. This resolves resources and\n runs the optimizer without provisioning anything. Note there is **no**\n `--dryrun` on `sky jobs launch`, so dry-run against `sky launch` even when\n the final command will be `sky jobs launch`.\n6. **Tell the user which submit command to use**: `sky jobs launch` for batch\n work (an `sbatch` script), `sky launch` for an interactive allocation (an\n `salloc` session).\n\n### 1.1 Choosing the submit command\n\n| The user's script is | Use | Why |\n|---|---|---|\n| `sbatch` batch job | `sky jobs launch` | Allocation released when the job ends, and the job is recovered if the node fails |\n| `salloc` / `srun --pty` session | `sky launch` + `ssh ` | Long-lived allocation to work in |\n| `sbatch --array` | `sky jobs launch --num-jobs N` | Section 7.1 |\n| A chain with `--dependency` | One managed pipeline | Section 7.2 |\n\n---\n\n## 2. `#SBATCH` Directive Mapping\n\n| `#SBATCH` directive | SkyPilot YAML | Notes |\n|---|---|---|\n| `--nodes=2` | `num_nodes: 2` | Task-level field, **not** under `resources` |\n| `--gpus-per-node=8`, `--gres=gpu:h100:8` | `resources.accelerators: H100:8` | Emitted as `--gres=gpu::`. The name must match the cluster's GRES config |\n| `--cpus-per-task=32` | `resources.cpus: 32+` | `+` means \"at least\". Emitted as `--cpus-per-task` |\n| `--mem=256G` | `resources.memory: 256+` | In GB. Emitted as `--mem` |\n| `--partition=gpu` | `resources.infra: slurm//gpu` | Omit to let the optimizer choose a partition |\n| `--time=24:00:00` | `config.slurm.sbatch_options.time: \"24:00:00\"` | Defaults to the partition's `MaxTime` (or `DefaultTime` when `MaxTime` is `UNLIMITED`) if unset. **Not** `autostop` — see section 8 |\n| `--job-name=train` | `name: train` | |\n| `--output=`, `--error=` | *(nothing)* | SkyPilot manages job logs; see section 7.4 |\n| `--account`, `--qos`, `--reservation`, `--exclusive`, `--constraint`, ... | `config.slurm.sbatch_options.` | Section 3 |\n\n### 2.1 Worked example\n\nInput:\n\n```bash\n#!/bin/bash\n#SBATCH --job-name=train\n#SBATCH --nodes=2\n#SBATCH --gpus-per-node=8\n#SBATCH --cpus-per-task=32\n#SBATCH --mem=256G\n#SBATCH --partition=gpu\n#SBATCH --time=24:00:00\n#SBATCH --account=ai-research\n\nmodule load cuda/12.1\nsource ~/venv/bin/activate\n\nsrun python train.py --epochs 100\n```\n\nOutput:\n\n```yaml\n# train.yaml\nname: train\n\nnum_nodes: 2\n\nresources:\n infra: slurm/mycluster/gpu\n accelerators: H100:8\n cpus: 32+\n memory: 256+\n\nconfig:\n slurm:\n sbatch_options:\n time: \"24:00:00\"\n account: ai-research\n\nsetup: |\n pip install -r requirements.txt\n\nrun: |\n python train.py --epochs 100\n```\n\nSubmit with `sky jobs launch -n train train.yaml`.\n\nNote what happened to the `srun`: it disappeared. `run` already executes on\nevery node of the allocation, so wrapping the command in `srun` again would\nnest a job step inside SkyPilot's own. See section 5.\n\n---\n\n## 3. Directives SkyPilot Does Not Model\n\nAnything without a `resources` equivalent goes through\n`slurm.sbatch_options`, which is emitted verbatim as `#SBATCH` lines. Keys are\n`sbatch` long option names; underscores become hyphens; `true` emits a bare\nflag; `null` / `false` omit the line.\n\n```yaml\nconfig:\n slurm:\n sbatch_options:\n account: ai-research\n qos: high\n time: \"24:00:00\"\n exclusive: true\n constraint: ib\n```\n\nThe same block works in `~/.sky/config.yaml` under a top-level `slurm:` key to\napply fleet-wide, and can be scoped per cluster or per partition:\n`slurm.cluster_configs..sbatch_options` and\n`slurm.cluster_configs..partition_configs..sbatch_options`.\n\n**Do not put these in `sbatch_options`** — SkyPilot manages them and will drop\nthem with a warning:\n\n```\njob-name output error nodes wait-all-nodes\nno-requeue requeue cpus-per-task mem gres partition\n```\n\nIf a user asks for one of those, set it through `resources` / `name` instead.\n\n---\n\n## 4. Environment Variable Mapping\n\n| Slurm | SkyPilot | Notes |\n|---|---|---|\n| `$SLURM_JOB_NODELIST` | `$SKYPILOT_NODE_IPS` | Newline-separated IPs, **not** a Slurm hostlist expression. `$(echo \"$SKYPILOT_NODE_IPS\" \\| head -n1)` is the head node |\n| `$SLURM_NNODES` | `$SKYPILOT_NUM_NODES` | |\n| `$SLURM_NODEID`, `$SLURM_PROCID` | `$SKYPILOT_NODE_RANK` | `0` to `num_nodes-1` |\n| `$SLURM_GPUS_PER_NODE` | `$SKYPILOT_NUM_GPUS_PER_NODE` | |\n| `$SLURM_JOB_ID` | `$SKYPILOT_TASK_ID` | |\n| `$SLURM_ARRAY_TASK_ID` | `$SKYPILOT_JOB_RANK` | Only set under `--num-jobs`; section 7.1 |\n| `$SLURM_ARRAY_TASK_COUNT` | `$SKYPILOT_NUM_JOBS` | |\n\n**On Slurm specifically**, the *job-scoped* `SLURM_*` variables of the\nunderlying allocation are still present inside `run` — `SLURM_JOB_ID`,\n`SLURM_JOB_NODELIST`, `SLURM_GPUS_ON_NODE`, etc. Scripts that read them keep\nworking. The *step-scoped* variables from SkyPilot's own job step\n(`SLURM_CPUS_PER_TASK`, `SLURM_CPU_BIND*`, `SLURM_STEP_*`, `SLURM_PROCID`, ...)\nare deliberately unset before the user script runs, so a nested `srun` targets\nthe full allocation instead of inheriting SkyPilot's step shape.\n\nPrefer the `SKYPILOT_*` variables in generated YAML — they are what makes the\nsame task portable to Kubernetes and clouds. Use the `SLURM_*` ones only when\ntranslating a script that must keep working unchanged.\n\n---\n\n## 5. Translating `srun`\n\nThree cases. Pick by what the `srun` was *for*.\n\n### 5.1 `srun ` launching one process per node\n\nDrop the `srun`. `run` already runs on every node.\n\n```bash\n# Slurm\nsrun python preprocess.py\n```\n\n```yaml\nrun: |\n python preprocess.py\n```\n\n### 5.2 `srun` launching distributed training ranks\n\nUse the rank variables and the framework's own launcher. This is the portable\nform and should be the default for training workloads.\n\n```yaml\nnum_nodes: 2\n\nresources:\n accelerators: H100:8\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$MASTER_ADDR \\\n --master_port=8008 \\\n train.py\n```\n\n### 5.3 `srun` launching an MPI/PMIx program\n\nSome binaries must be launched by Slurm's own PMI (NCCL tests, anything\nMPI-linked, `--mpi=pmix`). Keep `srun`, but:\n\n- **Gate it on rank 0**, or every node will launch the whole job.\n- **Pass `--overlap`**, so the step can share the allocation with SkyPilot's\n own step.\n\n```yaml\nnum_nodes: 2\n\nresources:\n infra: slurm\n accelerators: H100:8\n\nrun: |\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n srun --overlap --mpi=pmix \\\n --ntasks-per-node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n ./my_mpi_program\n fi\n```\n\nThis form is Slurm-only by construction — it will not run on Kubernetes or a\ncloud. Say so when generating it.\n\n---\n\n## 6. Replacing the Module System\n\n`module load` reads a cluster-wide tree shared by all jobs. SkyPilot gives\neach job its own environment. In increasing order of isolation:\n\n```yaml\n# 1. setup commands — runs once per cluster, before the job\nsetup: |\n pip install -r requirements.txt\n```\n\n```yaml\n# 2. a container image — the closest thing to a reproducible module set.\n# On Slurm this needs the Pyxis SPANK plugin on the cluster.\nresources:\n image_id: docker:pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime\n```\n\n```yaml\n# 3. conda on the shared filesystem, if the user wants to keep their layout\nsetup: |\n conda create -n myenv python=3.10 -y\n conda activate myenv\n conda install pytorch pytorch-cuda=12.1 -c pytorch -c nvidia -y\n\nrun: |\n conda activate myenv\n python train.py\n```\n\n`module load` itself still works inside `setup` / `run` on a Slurm cluster,\nas long as the task is not running in a container. Keeping it is a legitimate\nfirst migration step — it just does not port to Kubernetes or clouds.\n\n---\n\n## 7. Patterns Without a One-to-One Field\n\n### 7.1 Job arrays → `--num-jobs`\n\n`sbatch --array=0-99` becomes:\n\n```bash\nsky jobs launch --num-jobs 100 -y -d task.yaml\n```\n\nEach job gets `$SKYPILOT_JOB_RANK` (the index) and `$SKYPILOT_NUM_JOBS`:\n\n```yaml\nrun: |\n python train.py --shard $SKYPILOT_JOB_RANK --num-shards $SKYPILOT_NUM_JOBS\n```\n\nEach job provisions its own allocation. To reuse workers across submissions\ninstead — closer to how a long-lived Slurm allocation behaves, and much\ncheaper when `setup` is expensive — submit to a pool (note `pool.yaml` is a\npool spec: it needs a `pool:` section, not a bare task YAML):\n\n```bash\nsky jobs pool apply -p mypool pool.yaml\nsky jobs launch -p mypool --num-jobs 100 task.yaml\n```\n\nFor sweeps over *named* parameters rather than an index, loop with `--env`:\n\n```bash\nfor lr in 0.001 0.01 0.1; do\n sky jobs launch --env LR=$lr -y -d task.yaml\ndone\n```\n\n### 7.2 Job dependencies → a managed pipeline\n\n`sbatch --dependency=afterok:` becomes a sequence of tasks in one\nmanaged job, separated by `---`. Each task can request different resources.\n\n```yaml\nname: train-then-eval\n\n---\n\nname: train\nresources:\n accelerators: H100:8\nrun: python train.py\n\n---\n\nname: eval\nresources:\n accelerators: L4:1\nrun: python eval.py\n```\n\nFor tasks that should run *in parallel* rather than in sequence, use a job\ngroup instead of a pipeline. A job group is the same multi-document layout,\nbut the header must set `execution: parallel` — without it the file is a\n*serial* pipeline:\n\n```yaml\nname: train-and-eval\nexecution: parallel\n\n---\n\nname: train\nresources:\n accelerators: H100:8\nrun: python train.py\n\n---\n\nname: eval\nresources:\n accelerators: L4:1\nrun: python eval.py\n```\n\n### 7.3 Login node workflow\n\nThere is no login node in the SkyPilot workflow. Submit from the user's laptop\nor CI; SkyPilot makes the SSH hop to the login node itself. Sync code with\n`workdir` (a local path or a git URL) rather than editing on NFS. For\ninteractive work, `ssh ` lands on the *allocated compute node*, not a\nshared login node.\n\n### 7.4 Job output files\n\nSlurm writes `slurm-.out`. SkyPilot streams logs instead:\n\n```bash\nsky jobs logs # managed job\nsky logs # latest job on a cluster\nsky logs 2 # a specific job\n```\n\nDo not translate `--output` / `--error` into anything. If the script writes\nits own result files, point them at the shared filesystem or a bucket.\n\n---\n\n## 8. Constraints When Running on Slurm\n\nDo not generate YAML that uses these on a Slurm target — they will fail or be\nrejected:\n\n| Feature | Status on Slurm | What to do instead |\n|---|---|---|\n| `autostop` / `sky stop` | Unsupported | `sky down` explicitly; bound wall-clock with `sbatch_options.time` |\n| `ports:` / `sky serve` | Unsupported | Run serving on Kubernetes or a cloud |\n| `use_spot: true` | Unsupported | Omit; there is no spot tier |\n| Jobs/serve controllers | Not hosted on Slurm | With a remote API server, consolidation mode runs the controller inside the API server, so `sky jobs launch` works against a Slurm-only fleet. With a local API server the controller needs Kubernetes or a cloud |\n| `image_id: docker:...` | Needs Pyxis on the cluster | Fall back to `setup` commands |\n| Bucket mounting (`mode: MOUNT`) | Needs FUSE on compute nodes | Use the shared filesystem, or `mode: COPY` |\n\nBecause autostop does not exist on Slurm, an idle `sky launch` cluster holds\nits allocation until `sky down`. Prefer `sky jobs launch` for anything\nbatch-like so the allocation is released automatically.\n\n### 8.1 If the target is Kubernetes instead\n\nThe assumption that breaks first is shared storage: Kubernetes does not mount\nhome directories. A Slurm script that reads `~/data` or writes `~/checkpoints`\nneeds one of:\n\n- an NFS mount via `config.kubernetes.pod_config` (if the nodes can reach the\n existing NFS server),\n- a `ReadWriteMany` PVC through SkyPilot volumes,\n- a bucket under `file_mounts` with `mode: MOUNT`,\n- `workdir` for code only.\n\nPartitions map to Kubernetes contexts (`infra: kubernetes/`), and\nSlurm's fairshare/QoS has no built-in equivalent — that is Kueue or priority\nclasses.\n\n---\n\n## 9. Validation Checklist\n\nBefore handing a converted YAML to the user:\n\n- [ ] `sky launch --dryrun ` resolves without error.\n- [ ] `num_nodes` is at task level, not under `resources`.\n- [ ] No `autostop` / `ports` / `use_spot` if the target is Slurm.\n- [ ] Every `srun` is accounted for by section 5 — dropped, replaced with\n rank variables, or kept with `--overlap` and gated on rank 0.\n- [ ] GPU name matches the cluster's GRES names (check `sky gpus list --infra\n slurm`).\n- [ ] Protected `sbatch` options are not in `sbatch_options` (section 3).\n- [ ] The user is told whether to run `sky launch` or `sky jobs launch`.\n","agent/skills/skypilot/references/troubleshooting.md":"# References Skill\n\nPath: `agent/skills/skypilot/references/troubleshooting.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/troubleshooting.md)","agent/skills/skypilot/references/yaml-spec.md":"# References Skill\n\nPath: `agent/skills/skypilot/references/yaml-spec.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/yaml-spec.md)","docs/source/examples/agents/autonomous-code-optimization.md":"# Subagent: autonomous-code-optimization\n\nPath: `docs/source/examples/agents/autonomous-code-optimization.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autonomous-code-optimization.md)","docs/source/examples/agents/autoresearch.md":"# Subagent: autoresearch\n\nPath: `docs/source/examples/agents/autoresearch.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autoresearch.md)"},"files":{".claude-plugin/marketplace.json":"{\n \"name\": \"skypilot\",\n \"owner\": {\n \"name\": \"SkyPilot Team\"\n },\n \"metadata\": {\n \"description\": \"Official SkyPilot skills for AI coding agents\",\n \"version\": \"1.0.0\"\n },\n \"plugins\": [\n {\n \"name\": \"skypilot\",\n \"source\": \"./agent\",\n \"description\": \"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds\",\n \"version\": \"1.0.0\",\n \"author\": {\n \"name\": \"SkyPilot Team\"\n },\n \"homepage\": \"https://docs.skypilot.co/\",\n \"repository\": \"https://github.com/skypilot-org/skypilot\",\n \"license\": \"Apache-2.0\",\n \"keywords\": [\"AI Infrastructure\", \"Multi-Cloud\", \"GPU\", \"Kubernetes\", \"Slurm\", \"Training\", \"Serving\", \"Cost Optimization\", \"SkyPilot\"]\n }\n ]\n}\n","agent/skills/skypilot/SKILL.md":"---\nname: skypilot\ndescription: \"Use when launching cloud VMs, Kubernetes pods, or Slurm jobs for GPU/TPU/CPU workloads, training or fine-tuning models on cloud GPUs, deploying inference servers (vllm, TGI, etc.) with autoscaling, writing or debugging SkyPilot task YAML files, using spot/preemptible instances for cost savings, comparing GPU prices across clouds, managing compute across 25+ clouds, Kubernetes, Slurm, and on-prem clusters with failover between them, troubleshooting resource availability or SkyPilot errors, optimizing cost and GPU availability, or migrating an existing Slurm workload (converting sbatch scripts, salloc sessions, job arrays or srun invocations into SkyPilot task YAMLs, and mapping Slurm commands, directives and SLURM_* environment variables to their SkyPilot equivalents).\"\n---\n\n# SkyPilot Skill\n\nSkyPilot is a unified framework to run AI workloads on any cloud, Slurm or Kubernetes. It provides a single interface to launch clusters, run jobs, and serve models across 25+ clouds (AWS, GCP, Azure, Coreweave, Nebius, Lambda, Together AI, RunPod, and more), Kubernetes clusters, and Slurm clusters.\n\n## When to Use SkyPilot\n\n**Use SkyPilot when you need to:**\n- Manage compute resources on any cloud, Slurm, or Kubernetes cluster\n- Launch CPU/GPU/TPU (GB300, GB200, B200, H200, H100, etc.) on any cloud, Kubernetes or Slurm\n- Run training, fine-tuning, or batch inference jobs\n- Serve models with autoscaling and multi-cloud replicas (SkyServe)\n- Run long-running jobs with automatic lifecycle management and recovery (managed jobs)\n- Find the cheapest or most available GPU across clouds\n\n**Don't use SkyPilot for:**\n- Local-only workloads (use Docker/conda directly)\n\n## Capabilities: When to Use What\n\nSkyPilot has three core abstractions. Use the right one for each stage of your workflow:\n\n**1. SkyPilot Clusters** (`sky launch` / `sky exec`) — Interactive development and debugging\n- Use during initial development, debugging, and experimentation\n- Launch a cluster, SSH in or connect VSCode/Cursor (`code --remote ssh-remote+CLUSTER`), iterate quickly\n- Cluster stays up until you stop/down it or autostop triggers\n- Best for: prototyping, debugging, short experiments\n\n**2. Managed Jobs** (`sky jobs launch`) — Long-running training and batch jobs\n- Use when submitting long-running jobs that should run unattended\n- Manages the full lifecycle: provisioning, execution, recovery, and teardown\n- Automatically recovers from spot preemptions, quota limits, and transient failures\n- Works across clouds, Kubernetes, and Slurm (handles preemptions and quota)\n- Best for: training runs, fine-tuning, hyperparameter sweeps, batch inference\n\n**3. SkyServe** (`sky serve up`) — Production model serving\n- Use when serving models at scale with autoscaling\n- Start with `sky launch` + open port to test your serving setup, then use `sky serve up` to scale\n- Provides load balancing, autoscaling, and multi-cloud replicas\n- Best for: model serving endpoints, API services\n\n## Before You Start (Agent Bootstrap)\n\nBootstrap to confirm SkyPilot is installed, connected to an API server, and has cloud credentials. Once confirmed, skip straight to the user's task.\n\n**Step 1: Check installation and API server connectivity**\n\n```bash\nsky api info\n```\n| Output contains | Meaning | Next action |\n|-----------------|---------|-------------|\n| Server version and status | Server is running and connected | **Bootstrap done.** Skip to user's task. |\n| `No SkyPilot API server is connected` | No server connected | Go to \"Start or connect a server\" below. |\n| `Could not connect to SkyPilot API server` | Remote server unreachable or auth expired | Tell the user and suggest `sky api login --relogin -e ` to reconnect. |\n| `command not found: sky` | SkyPilot not installed | Go to \"Install SkyPilot\" below. |\n\n**Install SkyPilot** (only if `sky` command not found):\n```bash\npip install \"skypilot[aws,gcp,kubernetes]\" # Pick clouds the user needs\n```\nAsk the user which clouds they need if unclear, then re-run `sky api info`.\n\n**Start or connect a server** (only if \"not running\"):\n\nAsk the user:\n> Do you have an existing SkyPilot API server to connect to, or should I start one locally?\n\n- **Connect to existing server:** `sky api login -e ` — get the URL from the user.\n- **Start locally:** `sky api start`\n\nAfter either path, re-run `sky api info` to confirm the server is reachable.\n\n**Step 2: Check cloud credentials** (only for fresh setups — skip if the server was already running)\n```bash\nsky check -o json\n```\nThis shows which clouds are enabled or disabled. If the user's target cloud is not enabled, guide them through credential setup (see [Troubleshooting](references/troubleshooting.md#1-installation-and-credentials)).\n\n## Essential Commands\n\nUse `-o json` with status/query commands to get structured JSON output instead of tables.\n\n**Clusters** — interactive development and debugging:\n\n| Command | Description |\n|---------|-------------|\n| `sky launch -c NAME task.yaml` | Launch a cluster or run a task |\n| `sky exec NAME task.yaml` | Run task on existing cluster (skips provisioning); syncs workdir each time |\n| `sky exec NAME task.yaml -d` | Same, but detach immediately (don't stream logs) |\n| `sky status -o json` | Show all clusters |\n| `sky logs NAME` | Stream job logs from a cluster |\n| `sky logs NAME --no-follow` | Print existing logs and exit immediately |\n| `sky logs NAME --tail 50` | Print last 50 lines of logs and exit |\n| `sky logs NAME --status` | Exit with code 0=succeeded, 100=failed, 101=not finished, 102=not found, 103=cancelled |\n| `sky queue NAME -o json` | List jobs on a cluster with status (structured JSON) |\n| `sky stop NAME` / `sky start NAME` | Stop/restart to save costs (preserves disk) |\n| `sky down NAME` | Tear down a cluster completely |\n| `sky gpus list -o json` | List available GPU types across clouds |\n\n**Managed Jobs** — long-running unattended workloads:\n\n| Command | Description |\n|---------|-------------|\n| `sky jobs launch task.yaml` | Launch a managed job (auto lifecycle + recovery) |\n| `sky jobs queue -o json` | Show all managed jobs and their status |\n| `sky jobs logs JOB_ID` | Stream logs from a managed job |\n| `sky jobs cancel JOB_ID` | Cancel a managed job |\n\n**SkyServe** — model serving with autoscaling:\n\n| Command | Description |\n|---------|-------------|\n| `sky serve up serve.yaml -n NAME` | Start a model serving service |\n| `sky serve status NAME` | Show service status and endpoint URL |\n| `sky serve update NAME new.yaml` | Update a running service (rolling) |\n| `sky serve down NAME` | Tear down a service |\n\nFor complete CLI reference, see [CLI Reference](references/cli-reference.md).\n\n## Quick Start\n\n```bash\n# Launch a GPU cluster\nsky launch -c mycluster --gpus H100 -- nvidia-smi\n\n# Run a task from YAML\nsky launch -c mycluster task.yaml\n\n# SSH into cluster\nssh mycluster\n\n# Connect VSCode or Cursor to the cluster for interactive development\ncode --remote ssh-remote+mycluster /home/user/sky_workdir\n# or: cursor --remote ssh-remote+mycluster /home/user/sky_workdir\n\n# Tear down\nsky down mycluster\n```\n\n## Task YAML Structure\n\nThe task YAML is SkyPilot's primary interface. All fields are optional.\n\n```yaml\n# task.yaml\nname: my-training-job\n\n# Local directory to sync to remote ~/sky_workdir\nworkdir: .\n\n# Number of nodes (for distributed training)\nnum_nodes: 1\n\nresources:\n # GPU/TPU accelerators (SkyPilot auto-selects the cheapest cloud/region)\n accelerators: H200:8\n # Optional: pin to a specific cloud/region/infra\n # infra: aws # or aws/us-east-1, k8s, ssh/my-pool\n # If infra is left out, SkyPilot automatically fails over across all\n # enabled clouds/regions to find the cheapest available option.\n # Use spot instances for cost savings\n use_spot: false\n # Disk size in GB\n disk_size: 256\n # Open ports for serving\n ports: 8080\n\n# Environment variables (accessible in file_mounts, setup, and run)\nenvs:\n MODEL_NAME: my-model\n BATCH_SIZE: 32\n\n# Setup: runs once on cluster creation, cached on reuse\nsetup: |\n pip install torch transformers\n\n# Run: the main command\nrun: |\n python train.py --model $MODEL_NAME --batch-size $BATCH_SIZE\n```\n\nFor complete YAML schema including file mounts, environment variables set by SkyPilot, and advanced fields, see [YAML Specification](references/yaml-spec.md).\n\n## GPU and Cloud Selection\n\n**IMPORTANT: Let SkyPilot choose the cloud and region.** Do NOT manually pick a cloud/region/instance by parsing `sky gpus list` output. SkyPilot's optimizer automatically selects the cheapest available option across all enabled clouds. Only specify `infra:` when the user explicitly requests a specific cloud or region.\n\n**Default behavior (recommended):** Just specify the GPU type. SkyPilot finds the cheapest cloud/region automatically:\n\n```yaml\nresources:\n accelerators: H200:8 # SkyPilot picks the cheapest cloud/region with H200:8\n```\n\nIf the user doesn't specify a GPU type, ask them what GPU they need (or what model/workload they're running so you can recommend one). Do NOT run `sky gpus list` and pick for them — present options and let the user decide, or use `any_of` to let SkyPilot maximize availability:\n\n```yaml\n# Let SkyPilot choose from multiple acceptable GPU types (cheapest wins)\nresources:\n any_of:\n - accelerators: H100:8\n - accelerators: A100-80GB:8\n - accelerators: A100:8\n```\n\nUse `ordered` only when the user has a strict preference:\n\n```yaml\n# Try H100 first on AWS, fall back to GCP, then A100\nresources:\n ordered:\n - infra: aws/us-east-1\n accelerators: H100:8\n - infra: gcp/us-central1\n accelerators: H100:8\n - infra: aws/us-west-2\n accelerators: A100-80GB:8\n```\n\nOnly set `infra:` when the user explicitly says something like \"use AWS\" or \"run on GCP us-central1\":\n\n```yaml\nresources:\n infra: aws # User asked for AWS specifically\n accelerators: H100:8\n```\n\n## Cluster Lifecycle\n\n```bash\n# Launch and run a task\nsky launch -c mycluster task.yaml\n\n# Launch with autostop at launch time (preferred: saves cost, no follow-up command needed)\nsky launch -c mycluster task.yaml -i 30 # stop after 30 min idle\nsky launch -c mycluster task.yaml -i 30 --down # tear down after 30 min idle\n\n# Override or pass environment variables via CLI\nsky launch -c mycluster task.yaml --env MODEL_NAME=llama3 --env BATCH_SIZE=64\n\n# Re-run a different task on the same cluster (fast, skips provisioning)\nsky exec mycluster another_task.yaml\n\n# Run an inline command\nsky exec mycluster -- python train.py --epochs 10\n\n# Set autostop after launch (use if you forgot to set -i at launch time)\nsky autostop mycluster -i 30 # stop after 30 min idle, preserving disk (can restart with sky start)\nsky autostop mycluster -i 30 --down # tear down after 30 min idle (disk is deleted, cannot restart)\n\n# Stop to save costs, restart later\nsky stop mycluster\nsky start mycluster\n\n# Tear down completely\nsky down mycluster\n```\n\n## Workdir Sync Behavior\n\n`workdir:` is synced to `~/sky_workdir` on the remote via `rsync` before every `sky exec`. **rsync is additive — deleted local files are NOT removed from the remote.** This can cause experiments to run against stale build artifacts or old configs.\n\nTo ensure a clean slate, SSH and wipe before `sky exec`:\n```bash\nssh mycluster \"rm -rf ~/sky_workdir\"\nsky exec mycluster task.yaml\n```\n\nOr clean inside `run:` if only specific artifacts need removal:\n```yaml\nrun: |\n find ~/sky_workdir/build -name '*.o' -delete 2>/dev/null || true\n cd ~/sky_workdir && make\n```\n\n## Managed Jobs\n\nUse `sky jobs launch` for long-running jobs that should run unattended. SkyPilot manages the full lifecycle — provisioning, execution, recovery from preemptions/quota/failures, and teardown:\n\n```yaml\n# managed-job.yaml\nname: training-job\n\nresources:\n accelerators: A100:8\n\nrun: |\n python train.py --resume-from-checkpoint\n```\n\n```bash\n# Launch as managed job\nsky jobs launch managed-job.yaml\n\n# Check status\nsky jobs queue -o json\n\n# Stream logs\nsky jobs logs \n\n# Cancel\nsky jobs cancel \n```\n\n**Checkpoint pattern**: Your training script should save checkpoints to persistent storage (cloud bucket or volume) and resume from the latest checkpoint on restart. SkyPilot handles the cluster recovery; your script handles the state recovery.\n\n## SkyServe: Model Serving\n\n```yaml\n# serve.yaml\nresources:\n accelerators: A100:1\n ports: 8080\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-3.1-8B-Instruct \\\n --port 8080\n\nservice:\n readiness_probe: /v1/models\n replica_policy:\n min_replicas: 1\n max_replicas: 3\n target_qps_per_replica: 5\n```\n\n```bash\n# Start service\nsky serve up serve.yaml -n my-llm\n\n# Check status / get endpoint\nsky serve status my-llm\nsky serve status my-llm --endpoint\n\n# Update (rolling)\nsky serve update my-llm new-serve.yaml\n\n# Tear down\nsky serve down my-llm\n```\n\n## Common Workflows\n\n### Fine-Tuning Workflow\n1. Write task YAML with `setup` (install deps) and `run` (training command)\n2. Use `file_mounts` or `workdir` to sync code\n3. `sky launch -c train task.yaml` to launch\n4. `sky logs train` to monitor\n5. `sky exec train -- python eval.py` to evaluate on same cluster\n6. `sky down train` when done\n\n### Hyperparameter Sweep\n1. Create parameterized YAML with `envs`\n2. Launch multiple managed jobs:\n ```bash\n for lr in 1e-4 1e-5 1e-6; do\n sky jobs launch sweep.yaml --env LR=$lr --name sweep-lr-$lr\n done\n ```\n3. Monitor with `sky jobs queue -o json`\n\n### Model Serving Deployment\n1. Write serve YAML with `service:` section\n2. `sky serve up serve.yaml -n my-service`\n3. Get endpoint: `sky serve status my-service --endpoint`\n4. Update model: `sky serve update my-service updated.yaml`\n\n### Parallel Experiment Submission\n\nUse `sky exec -d` to submit jobs to multiple VMs without blocking, then collect results:\n\n```bash\n# Submit all experiments (detached, returns after job is queued)\nfor i in 1 2 3 4; do\n sky exec exp-vm-0$i task.yaml --env LR=1e-$i -d\ndone\n\n# Get the latest job ID from a cluster\njob_id=$(sky queue exp-vm-01 -o json \\\n | python3 -c \"import sys, json; jobs = json.load(sys.stdin).get('exp-vm-01', []); print(max(j['job_id'] for j in jobs) if jobs else '')\")\n\n# Wait for a specific job and fetch last 50 lines\nsky logs exp-vm-01 $job_id --status && sky logs exp-vm-01 $job_id --tail 50\n\n# Check all jobs across a cluster at once\nsky queue exp-vm-01 -o json\n```\n\n### Converting a Slurm Workload\n\nWhen the user has an existing `sbatch` script, `salloc` workflow, or asks how a\nSlurm command maps to SkyPilot, **read\n[Migrating from Slurm](references/migrating-from-slurm.md) before writing any\nYAML.** The mapping has non-obvious parts that are easy to get wrong:\n\n- `--time` does **not** map to `autostop` — autostop is unsupported on Slurm.\n Use `config.slurm.sbatch_options.time`.\n- A bare `srun ` should usually be **dropped**, not translated: `run`\n already executes on every node. Only MPI/PMIx launchers keep `srun`, and\n then they need `--overlap` and a rank-0 guard.\n- `--account` / `--qos` / `--exclusive` and other unmodeled directives go\n through `config.slurm.sbatch_options`, which passes them to `sbatch`\n verbatim.\n- `sbatch --array` maps to `sky jobs launch --num-jobs` with\n `$SKYPILOT_JOB_RANK`, not to a shell loop.\n\nValidate the result with `sky launch --dryrun ` — note there is no\n`--dryrun` on `sky jobs launch`, so dry-run via `sky launch` even when the\nfinal command will be `sky jobs launch`.\n\nDo not assume the user wants to leave Slurm. SkyPilot submits to an existing\nSlurm cluster through its login node; migrating *to Kubernetes* is a separate\nquestion, and the constraints differ.\n\n## Agent Feedback Loop\n\nWhen using SkyPilot programmatically, follow this loop:\n\n1. **Validate**: `sky launch --dryrun task.yaml` (check resource availability/cost)\n2. **Launch**: `sky launch -c mycluster task.yaml`\n3. **Monitor**: `sky status -o json` and `sky queue mycluster -o json`\n4. **Wait for completion**: `sky logs mycluster ` (streams logs so you can observe progress and react to stalls; blocks until job finishes; get JOB_ID from `sky queue mycluster -o json`). For long-running jobs where you don't need intermediate output, use `sky logs mycluster --status` instead (blocks silently, exits 0 on success).\n5. **Inspect output**: `sky logs mycluster --no-follow` or `sky logs mycluster --tail 100`\n6. **Debug**: `ssh mycluster` (interactive)\n7. **Iterate**: `sky exec mycluster updated_task.yaml` (run on existing cluster)\n8. **Cleanup**: `sky down mycluster`\n\n> **Never poll with `sleep` + `sky queue`** — use `sky logs CLUSTER JOB_ID` to stream logs and block until done. Use `--status` if you only need the exit code, or `--tail N` to fetch recent output after completion.\n\n## Investigating a Managed Job\n\nWhen the user asks \"what is my job doing\" or \"why is it pending/failing\", run\nthese in order and stop as soon as the question is answered:\n\n1. `sky jobs queue -v -o json` — `status`, `details`, `failure_reason` (`-v` is what adds `details`). For a job that has not started, `details` is the answer: it says what the cluster is waiting on, e.g. a Slurm `QOSGrpGRES` or `Dependency` reason with the partition it is queued in\n2. `sky jobs logs JOB_ID --tail 100 --no-follow` — recent task output; `--controller` for provisioning/recovery logs\n\nSee [Job Investigation](references/job-investigation.md) for how to read `details` and the Slurm pending reasons.\n\n## Common Agent Mistakes\n\n| Mistake | Why it's wrong | Do this instead |\n|---------|---------------|-----------------|\n| Manually picking cloud/region from `sky gpus list` output | SkyPilot optimizer does this automatically and better | Just set `accelerators:` and let SkyPilot choose |\n| Using `sky launch` for long-running unattended jobs | No recovery if preempted or interrupted | Use `sky jobs launch` for unattended work |\n| Forgetting `sky down` or autostop after work is done | Wastes money on idle clusters | Always clean up, or use `-i --down` at launch |\n| Hardcoding `infra: aws` without user asking | Limits availability and increases cost | Only set `infra:` when user explicitly requests a cloud |\n| Not using `envs:` for configurable values | Hard to reuse or override from CLI | Use `envs:` in YAML + `--env KEY=VAL` for parameterization |\n| Running `sky launch` without `-c ` | Creates randomly-named cluster, hard to reference | Always name clusters with `-c` |\n| Parsing table output from status commands | Table formatting is for humans, fragile to parse | Use `-o json` for structured output |\n| Using deprecated `cloud:`/`region:`/`zone:` fields | Deprecated in favor of `infra:` | Use `infra: aws/us-east-1` instead |\n| Polling job status with `sleep` + `sky queue` | Wastes tokens, introduces timing bugs, fragile | Use `sky logs CLUSTER JOB_ID --status` to block until done |\n| Assuming workdir sync removes remote files | rsync is additive; old remote files persist across `sky exec` calls | SSH and manually clean `~/sky_workdir`, or clean in `run:` script |\n| Not using `--tail` when only last output matters | Streaming full logs wastes tokens for long jobs | Use `sky logs CLUSTER JOB_ID --tail 50` for last N lines |\n\n## Common Issues Quick Reference\n\n| Issue | Solution |\n|-------|----------|\n| GPU not available | Use `any_of` for fallback, or try different regions/clouds |\n| Setup takes too long | SkyPilot caches setup; use `sky exec` to skip it on reruns |\n| Task fails silently | Check `sky logs ` or `ssh ` to debug |\n| Cluster stuck in INIT | `sky down ` and relaunch |\n| Preemption/quota | Use `sky jobs launch` for automatic recovery and lifecycle management |\n| Port not accessible | Ensure `ports:` is set in resources and security groups allow traffic |\n| File sync slow | Use cloud bucket mounts instead of `workdir` for large datasets |\n| Credentials error | Run `sky check -o json` and inspect which clouds are disabled |\n\n## References\n\nFor detailed reference documentation:\n\n- [CLI Reference](references/cli-reference.md) — All commands and flags\n- [YAML Specification](references/yaml-spec.md) — Complete task YAML schema, file mounts, environment variables\n- [Python SDK](references/python-sdk.md) — Programmatic API and SDK usage\n- [Advanced Patterns](references/advanced-patterns.md) — Multi-cloud, distributed training, production patterns\n- [Migrating from Slurm](references/migrating-from-slurm.md) — Converting `sbatch` scripts to task YAMLs, Slurm command/env-var mapping, Slurm-specific constraints\n- [Job Investigation](references/job-investigation.md) — Which command answers \"what is my job doing and why\"; reading `details`, events, Slurm pending reasons\n- [Troubleshooting](references/troubleshooting.md) — Error diagnosis and solutions\n- [Examples](references/examples.md) — Copy-paste task YAML examples\n","agent/.claude-plugin/plugin.json":"{\n \"name\": \"skypilot\",\n \"description\": \"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds\",\n \"version\": \"1.0.0\",\n \"author\": {\n \"name\": \"SkyPilot Team\"\n },\n \"homepage\": \"https://docs.skypilot.co/\",\n \"repository\": \"https://github.com/skypilot-org/skypilot\",\n \"license\": \"Apache-2.0\",\n \"keywords\": [\"AI Infrastructure\", \"Multi-Cloud\", \"GPU\", \"Kubernetes\", \"Slurm\", \"Training\", \"Serving\", \"Cost Optimization\", \"SkyPilot\"]\n}\n","AGENTS.md":"# CLAUDE.md - SkyPilot Development Guide\n\nThis document provides guidance for AI assistants working with the SkyPilot codebase.\n\n## Project Overview\n\nSkyPilot is a system to run, manage, and scale AI workloads on any AI infrastructure. It provides a unified interface across 25+ cloud providers (AWS, GCP, Azure, Kubernetes, Slurm, and many others), enabling users to launch compute resources, run jobs, and serve models without vendor lock-in.\n\n**Key capabilities:**\n- **Clusters**: Launch and manage compute clusters across clouds and Kubernetes\n- **Managed jobs**: Run jobs with automatic recovery from preemptions and failures\n- **Multi-cloud and multi-Kubernetes**: Unified interface across 25+ clouds and multiple K8s clusters\n- Cost optimization and GPU availability maximization\n\n## Repository Structure\n\n```\nskypilot/\n├── sky/ # Main source code\n│ ├── __init__.py # Package exports and version\n│ ├── core.py # Core orchestration logic\n│ ├── task.py # Task definition and YAML parsing\n│ ├── resources.py # Cloud resource specifications\n│ ├── optimizer.py # Resource allocation optimizer\n│ ├── execution.py # Task execution pipeline\n│ ├── exceptions.py # Custom exceptions\n│ ├── skypilot_config.py # Configuration management\n│ ├── clouds/ # Cloud provider abstractions (25+ providers)\n│ ├── backends/ # Execution backends (Ray-based VM, Docker)\n│ ├── provision/ # Cloud resource provisioning\n│ ├── jobs/ # Managed job lifecycle\n│ ├── serve/ # Model serving (SkyServe)\n│ ├── skylet/ # On-cluster execution agent\n│ ├── client/ # SDK and CLI\n│ ├── server/ # API server and dashboard backend\n│ ├── dashboard/ # Web UI (Next.js)\n│ ├── utils/ # 50+ utility modules\n│ ├── catalog/ # Cloud pricing and instance catalogs\n│ ├── data/ # Storage and data handling\n│ ├── adaptors/ # Cloud-specific metadata adapters\n│ └── schemas/ # Protobuf definitions and API schemas\n├── tests/ # Test suite\n│ ├── unit_tests/ # Unit tests with subdirectories per module\n│ ├── smoke_tests/ # Quick validation tests\n│ ├── integration_tests/ # End-to-end tests\n│ ├── kubernetes/ # K8s-specific tests\n│ └── conftest.py # Pytest fixtures\n├── examples/ # 50+ usage examples\n├── llm/ # 45+ LLM training/serving examples\n├── docs/ # Sphinx documentation\n├── charts/ # Helm charts for K8s deployment\n└── format.sh # Code formatting script\n```\n\n## Development Setup\n\n### Environment Setup\n\n```bash\n# Create virtual environment with uv (Python 3.8-3.11 supported)\n# --seed is required to ensure pip is installed (needed for building wheels)\nuv venv --seed --python 3.11\nsource .venv/bin/activate\n\n# Install in editable mode with all cloud support\nuv pip install -e \".[all]\"\n# Or specific clouds only:\n# uv pip install -e \".[aws,gcp,kubernetes]\"\n\n# Install development dependencies\nuv pip install -r requirements-dev.txt\n\n# Optional: Install pre-commit hooks\nuv pip install pre-commit\npre-commit install\n```\n\n### Environment Variables\n\n```bash\nexport SKYPILOT_DEV=1 # Enable development mode\nexport SKYPILOT_DEBUG=1 # Enable debug logging\n```\n\n## Code Formatting and Linting\n\n**Always run `format.sh` before committing:**\n\n```bash\nbash format.sh # Format changed files (vs origin/master)\nbash format.sh --all # Format entire codebase\nbash format.sh --files path/to/file.py # Format specific files\n```\n\nThe script runs:\n1. **YAPF** - Google style for all Python code\n2. **isort** - Import sorting (Google profile)\n3. **mypy** - Type checking\n4. **pylint** - Linting with custom rules\n\n### Tool Versions (must match exactly)\n\nFrom `requirements-dev.txt`:\n- yapf==0.32.0\n- pylint==2.14.5\n- mypy==1.19.1\n- isort==5.12.0\n- pylint-quotes==0.2.3\n\n### Excluded from Formatting\n\n- `sky/schemas/generated/` - Auto-generated protobuf files\n- `build/` - Build artifacts\n\n## Testing\n\n### Running Tests\n\n```bash\n# Unit tests (fast, no cloud resources)\npytest tests/unit_tests/\n\n# Specific test file\npytest tests/unit_tests/test_resources.py\n```\n\n### CI Tests via PR Comments\n\nTrigger CI tests on pull requests using comments:\n- `/quicktest-core` - Run quick core tests\n- `/smoke-test` - Run smoke tests (launches cloud clusters)\n- `/smoke-test --kubernetes --postgres` - Test with PostgreSQL backend on Kubernetes\n- `/smoke-test --kubernetes --remote-server --postgres` - Test remote API server with PostgreSQL\n\n### Test Configuration\n\nFrom `pyproject.toml`:\n- Uses pytest-xdist with 16 parallel workers\n- Environment: `SKYPILOT_DEBUG=1`, `SKYPILOT_DEV=1`\n- Buildkite integration for CI\n\n### Checking Buildkite CI Status\n\nTo check CI test results for a PR:\n\n1. **Get Buildkite URL from GitHub**: Check the PR's commit status checks to find the Buildkite build URL (may take a moment to appear after triggering)\n2. **Fetch build logs via Buildkite API**: Use the Buildkite API to retrieve detailed logs and test results\n\n```bash\n# Example: Get build details (requires BUILDKITE_TOKEN)\ncurl -H \"Authorization: Bearer $BUILDKITE_TOKEN\" \\\n \"https://api.buildkite.com/v2/organizations/skypilot/pipelines/skypilot/builds/\"\n```\n\n## Code Style Guidelines\n\n### General Principles\n\n- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)\n- Use f-strings over `.format()` for readability\n- Use `class MyClass:` not `class MyClass(object):`\n- Use `abc` module for abstract classes\n- Use Python typing with `if typing.TYPE_CHECKING:` for type-only imports\n- Always place imports at the top of the file, not inside function definitions. If this causes a circular dependency, try to resolve it (e.g., restructure modules or use `if typing.TYPE_CHECKING:`). Only as a last resort, place the import inside the function with a comment explaining why. Import performance is not a valid reason for in-function imports; use `LazyImport` instead (see below).\n\n### TODOs and FIXMEs\n\nAlways include author attribution:\n```python\n# TODO(username): Description of what needs to be done\n# FIXME(username): Description of what needs fixing\n```\n\n### Exceptions vs Assertions\n\n- Use exceptions for errors (better error messages)\n- Use `assert` only for debugging/proof-checking\n\n### Lazy Imports\n\nFor modules with significant import time (>100ms) that are imported during `import sky`, use `LazyImport` from `sky/adaptors/common.py`:\n\n```python\nfrom sky.adaptors.common import LazyImport\nheavy_module = LazyImport('heavy_module')\n```\n\nMeasure import time with:\n```bash\npython -X importtime -c \"import sky\" 2> import.log\ntuna import.log # pip install tuna\n```\n\n## Architecture Patterns\n\n### Cloud Provider Abstraction\n\nAll cloud providers inherit from `sky.clouds.Cloud` (in `sky/clouds/cloud.py`). Cloud objects are **lightweight and stateless**. Key design principles:\n\n- Methods should be inexpensive to call\n- Don't store heavy state in cloud objects\n- Cache cloud-specific queries in `sky/clouds/utils/` modules\n- Each cloud implements feature flags via `CloudImplementationFeatures`\n\nAdding a new cloud provider:\n1. Create `sky/clouds/.py` inheriting from `Cloud`\n2. Implement required abstract methods\n3. Add provisioning logic in `sky/provision//`\n4. Register in `sky/clouds/__init__.py`\n\n### Client-Server Architecture\n\nSkyPilot uses a client-server model with API versioning:\n\n- Client (SDK/CLI) in `sky/client/`\n- Server in `sky/server/`\n- API version in `sky/server/constants.py`\n\n**Backward Compatibility Rules (from v0.10.0+):**\n- Changes must be backward compatible\n- Bump `API_VERSION` when introducing API changes\n- Use `@versions.minimal_api_version(N)` decorator for new SDK methods\n- Handle version differences with `versions.get_remote_api_version()`\n\n### Protobuf Regeneration\n\nWhen modifying `.proto` files in `sky/schemas/proto/`:\n\n```bash\npython -m grpc_tools.protoc \\\n --proto_path=sky/schemas/generated=sky/schemas/proto \\\n --python_out=. \\\n --grpc_python_out=. \\\n --pyi_out=. \\\n sky/schemas/proto/*.proto\n```\n\n### Dependency Management\n\nDependencies are defined in `sky/setup_files/dependencies.py`:\n\n- **Core dependencies**: Listed in `install_requires`\n- **Cloud-specific**: Defined in `extras_require` (e.g., `aws`, `gcp`, `kubernetes`)\n- **Development**: Listed in `requirements-dev.txt`\n\nWhen updating dependencies:\n\n1. Check version constraints carefully - some packages have breaking changes\n2. Consider Python version compatibility (3.8-3.11)\n3. Test with both minimum and latest allowed versions\n4. Document version constraints with comments explaining why\n\n## Key Modules Reference\n\n| Module | Purpose |\n|--------|---------|\n| `sky/core.py` | Core orchestration (launch, exec, stop, down) |\n| `sky/task.py` | Task YAML parsing and specification |\n| `sky/resources.py` | Resource requirements (GPUs, memory, disk) |\n| `sky/optimizer.py` | Cloud selection and cost optimization |\n| `sky/client/sdk.py` | Python SDK implementation |\n| `sky/client/cli/` | CLI commands |\n| `sky/backends/cloud_vm_ray_backend.py` | Main execution backend for clusters |\n| `sky/provision/provisioner.py` | Resource provisioning |\n| `sky/jobs/` | Managed jobs with recovery and scheduling |\n\n## Pull Request Guidelines\n\n1. **Branch from master**, create descriptive branch name\n2. **Run `format.sh`** before committing\n3. **Add tests** for core system changes\n4. **Run smoke tests** for significant changes\n5. **Include `Tested:` section** in PR description with test plan\n6. **Delete branch** after merging\n\n### PR Description Format\n\nPRs should include:\n- **Summary**: Brief description of changes (1-3 bullet points)\n- **Test plan**: How the changes were tested (commands run, manual verification steps)\n\n**Important**: Always generate a manual test plan describing how to verify the changes work correctly. Include specific commands, expected outputs, or UI verification steps. Whenever possible, add unit tests and smoke tests for the changes.\n\n### Commit Message Format\n\nUse the `[Area] Description` format:\n- `[Core]` for core system changes\n- `[CLI]` for CLI changes\n- `[API]` for API server changes\n- `[Docs]` for documentation\n- `[AWS]`, `[GCP]`, `[Azure]`, `[Kubernetes]` for cloud-specific changes\n- `[Jobs]` for managed jobs\n- `[Dashboard]` for web UI changes\n- `[Serve]` for model serving (SkyServe)\n- `[Test]` for testing changes\n- `[CI]` for CI/CD changes\n\nExamples:\n- `[Core] Fix cluster status refresh logic`\n- `[AWS] Add support for new instance types`\n- `[Docs] Update installation guide`\n\n## API Server Testing\n\n### Local API Server (Recommended for Development)\n\n```bash\n# Always restart API server after code changes to pick up changes\nsky api stop\nsky api start\n\n# Verify server is running\nsky api status\n```\n\n### Dashboard Development\n\n**For local API server development**, rebuild the dashboard before restarting:\n\n```bash\n# Install dependencies (first time or after package.json changes)\nnpm --prefix sky/dashboard install\n\n# Rebuild the dashboard\nnpm --prefix sky/dashboard run build\n\n# Then restart the API server\nsky api stop\nsky api start\n```\n\n**For remote API server (Docker/Kubernetes)**, the Dockerfile automatically builds the dashboard - no manual build needed before `docker build`.\n\nThe dashboard is a Next.js application. For development with hot reloading:\n\n```bash\n# Run dashboard in development mode (separate from API server)\ncd sky/dashboard\nnpm run dev\n```\n\n### Mocking Remote API Server Locally\n\nTo test remote API server behavior locally:\n\n```bash\n# Start local API server (runs on port 46580 by default)\nsky api stop\nsky api start\n\n# Forward to a different port to simulate remote server\nsocat TCP-LISTEN:46590,fork TCP:127.0.0.1:46580 &\n\n# Connect to the forwarded port as if it were a remote server\nsky api login -e http://127.0.0.1:46590\n```\n\n### Remote API Server (Kubernetes Deployment)\n\nFor testing with a remote API server on Kubernetes:\n\n```bash\n# Build local changes\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\nhelm dependency build ./charts/skypilot\n\n# Build Docker image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy (NEW installation)\nNAMESPACE=skypilot\nRELEASE_NAME=skypilot\nhelm upgrade --install $RELEASE_NAME ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n#### Upgrading Existing Deployments\n\n**CRITICAL:** Always use `--reuse-values` to preserve database/credential config:\n\n```bash\n# Upgrade existing deployment (keeps PostgreSQL, auth, etc.)\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n\n# Check current values / rollback if needed\nhelm get values skypilot -n skypilot\nhelm rollback skypilot -n skypilot\n```\n\n#### PostgreSQL Backend\n\n```bash\n# Create connection secret\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/db\"\n\n# Deploy with PostgreSQL\nhelm upgrade --install skypilot ./charts/skypilot -n skypilot \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\n## Critical Code Paths (Handle with Care)\n\nThe following modules contain complex, stateful logic that requires careful review when modifying:\n\n**Managed Jobs Recovery:**\n- `sky/jobs/controller.py` - Job lifecycle management, state transitions, async coordination\n- `sky/jobs/recovery_strategy.py` - Preemption recovery, retry logic for managed jobs\n- `sky/backends/cloud_vm_ray_backend.py` - Execution backend with complex state handling\n\nThese modules handle edge cases like preemption during job submission, controller failures mid-recovery, and race conditions between concurrent operations. Changes can have subtle effects on job reliability.\n\n**API Server Performance & Robustness:**\n- `sky/server/` - API server with memory efficiency, low latency requirements\n- `sky/backends/backend_utils.py` - Cluster status caching, network resilience, SSH handling\n\nThese modules are optimized for performance. Be cautious about adding blocking calls, memory-heavy operations, or changing caching behavior.\n\n**CLI/SDK Interface Design:**\n- `sky/client/cli/` - Command-line interface\n- `sky/client/sdk.py` - Python SDK\n\nBe cautious about adding new interfaces or changing existing UX significantly. Keep the interface clean and minimal for both CLI and SDK.\n\n**Backward Compatibility:**\n- Test with old server + new client, and old client + new server\n- Ensure existing clusters and jobs continue working after server upgrades\n- `/quicktest-core` tests some backward compatibility scenarios\n- See `docs/source/developers/CONTRIBUTING.md` for API versioning guidelines\n\n## Common Pitfalls\n\n1. **Always restart API server after code changes** - Run `sky api stop; sky api start` to pick up changes\n2. **Don't modify `sky/schemas/generated/`** - These are auto-generated\n3. **Match formatter versions exactly** - Version mismatches cause CI failures\n4. **Consider import time** - Heavy imports slow down CLI responsiveness\n5. **API versioning** - Always maintain backward compatibility\n\n## Useful Commands\n\n```bash\n# Profile CLI performance\nuv pip install py-spy\npy-spy record -t -o sky.svg -- python -m sky.cli status\n\n# Check cloud credentials\nsky check\n\n# View cluster status\nsky status\n\n# Launch a cluster\nsky launch \n\n# Execute a command on the cluster\nsky exec -- bash\n\n# SSH into the cluster\nssh \n\n# Launch a managed job\nsky jobs launch \n```\n\n## Documentation\n\n- **User docs**: https://docs.skypilot.co/\n- **Source**: `docs/source/` (Sphinx)\n- **Build docs**: See `.readthedocs.yml`\n\n## Additional Resources\n\n- **Full contributing guide**: `docs/source/developers/CONTRIBUTING.md`\n- **User docs**: https://docs.skypilot.co/\n- **GitHub Issues**: https://github.com/skypilot-org/skypilot/issues\n- **Discussions**: https://github.com/skypilot-org/skypilot/discussions\n- **Slack**: http://slack.skypilot.co\n","agent/skills/skypilot/evals/evals.json":"{\n \"skill_name\": \"skypilot\",\n \"description\": \"Evaluates the 'Before You Start (Agent Bootstrap)', 'Converting a Slurm Workload' and 'Investigating a Managed Job' section behavior\",\n \"evals\": [\n {\n \"id\": 1,\n \"name\": \"happy-path-server-running\",\n \"prompt\": \"I want to launch an 8xH100 cluster for training a LLaMA model. My SkyPilot is already set up.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, sees the server is running (table output with request data), skips rest of bootstrap (no `sky check`, no install, no `sky api start`), and proceeds directly to help write the task YAML or run `sky launch`.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` (or `sky api status -l 1`) as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT run `sky check` when the server is already running\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent does NOT run `sky api start` or `sky api login` when the server is already running\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent proceeds to help with the user's actual task (launching H100 cluster) without unnecessary bootstrap steps\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 2,\n \"name\": \"server-not-running-local-setup\",\n \"prompt\": \"I just installed SkyPilot and want to try launching a GPU instance on AWS. I don't have a remote API server.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, detects 'SkyPilot API server is not running.', asks about local vs remote, user indicates local, agent runs `sky api start`, re-checks with `sky api status -l 1`, then runs `sky check` for credentials since this is a fresh setup.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent asks the user whether they have a remote server or want to start locally\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent runs `sky api start` after user confirms local setup\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent re-runs `sky api status` after starting the server to confirm it's reachable\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent runs `sky check` since this is a fresh setup\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 3,\n \"name\": \"server-not-running-remote-connect\",\n \"prompt\": \"I need to connect to our team's SkyPilot API server at https://skypilot.internal.company.com and launch a fine-tuning job.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, detects server not running, recognizes user already provided a remote URL, runs `sky api login -e https://skypilot.internal.company.com` without asking unnecessary questions, re-checks status, then proceeds to help with the fine-tuning job.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent uses `sky api login -e https://skypilot.internal.company.com` (not `sky api start`) since the user provided a remote endpoint\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT ask the user for the server URL again since they already provided it\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent re-runs `sky api status` after connecting to confirm it's reachable\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 4,\n \"name\": \"slurm-sbatch-time-not-autostop\",\n \"prompt\": \"Convert this to SkyPilot for our Slurm cluster:\\n#!/bin/bash\\n#SBATCH --nodes=2\\n#SBATCH --gpus-per-node=8\\n#SBATCH --time=24:00:00\\n#SBATCH --account=ai-research\\nsrun python train.py\",\n \"expected_output\": \"Agent produces a task YAML with num_nodes: 2, resources.accelerators: H100:8 (or asks for the GRES name), and puts both --time and --account under config.slurm.sbatch_options. The bare `srun` is dropped because `run` already executes on every node. Agent recommends `sky jobs launch` for this batch script.\",\n \"assertions\": [\n {\n \"text\": \"Agent maps --time to config.slurm.sbatch_options.time\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT map --time to autostop, which is unsupported on Slurm\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent maps --account to config.slurm.sbatch_options.account\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent puts num_nodes at task level, not under resources\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT wrap the run command in `srun` again\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent recommends `sky jobs launch` (not `sky launch`) for this batch script\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 5,\n \"name\": \"slurm-job-array-to-num-jobs\",\n \"prompt\": \"On Slurm I run `sbatch --array=0-99 shard.sh` and the script reads $SLURM_ARRAY_TASK_ID to pick its shard. What's the SkyPilot equivalent?\",\n \"expected_output\": \"Agent answers `sky jobs launch --num-jobs 100` and maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK (with $SKYPILOT_NUM_JOBS for the count). May also mention pools for reusing workers across submissions.\",\n \"assertions\": [\n {\n \"text\": \"Agent uses `sky jobs launch --num-jobs 100`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT propose a shell for-loop as the primary answer\",\n \"type\": \"negative\"\n }\n ]\n },\n {\n \"id\": 6,\n \"name\": \"why-is-my-managed-job-pending\",\n \"prompt\": \"My managed job 42 has been PENDING for 20 minutes on our Slurm cluster. Why? SkyPilot is already set up.\",\n \"expected_output\": \"Agent reads `details` from `sky jobs queue -v -o json` and finds the Slurm pending reason with its partition (e.g. `Launching (pending: QOSGrpGRES; partition: h200)`), explains that category (quota vs no free capacity vs dependency vs held) and what to do about it. Does not claim the job is stuck in SkyPilot's own queue when the reason is Slurm's.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky jobs queue -v -o json` (`-v` is required for the `details` field) and reads the job's details\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent reads the Slurm pending reason and its partition from `details`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent explains the pending reason in plain terms (quota, capacity, dependency, or held) with a next step\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT poll with `sleep` + `sky jobs queue`\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent does NOT tear down the jobs controller as a first step\",\n \"type\": \"negative\"\n }\n ]\n },\n {\n \"id\": 7,\n \"name\": \"what-happened-to-my-job\",\n \"prompt\": \"Managed job 17 keeps flipping between RUNNING and RECOVERING. Show me what happened.\",\n \"expected_output\": \"Agent reads `sky jobs queue -v -o json` for `details` / `failure_reason` / `recovery_count`, then pulls `sky jobs logs 17 --controller --no-follow` for the recovery history, and points out that a repeated identical reason means a persistent cause the controller cannot fix by retrying.\",\n \"assertions\": [\n {\n \"text\": \"Agent reads `details`, `failure_reason` and `recovery_count` from `sky jobs queue -v -o json`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent identifies the recurring RECOVERING reason from the events\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent uses `--tail` or `--no-follow` when fetching logs rather than streaming the full log\",\n \"type\": \"required_action\"\n }\n ]\n }\n ]\n}\n","agent/skills/skypilot/references/advanced-patterns.md":"# SkyPilot Advanced Patterns Reference\n\nThis reference covers advanced SkyPilot patterns for multi-cloud strategies,\ndistributed training, managed jobs, serving, Kubernetes integration, API server\nsetup, and cost optimization.\n\n---\n\n## 1. Multi-Cloud Strategies\n\nSkyPilot can automatically select the cheapest cloud/region/zone for a\nworkload, or follow explicit priority ordering. The key mechanisms are:\n- `any_of`: try all listed resource options, optimize for cost (unordered)\n- `ordered`: try resource options in strict priority order (first match wins)\n- `infra:` field with wildcard patterns for flexible cloud/region/zone targeting\n\n### 1.1 any_of -- Cheapest Available GPU\n\nUse `any_of` when you have multiple acceptable resource configurations and\nwant SkyPilot to pick the cheapest one available across all of them.\n\nNote: If all options use the same accelerator, you can simply specify\n`accelerators:` without `any_of` — SkyPilot will automatically search\nacross all enabled clouds. Use `any_of` when mixing different GPU types:\n\n```yaml\n# cheapest-gpu.yaml — mix different GPU types, SkyPilot picks cheapest\nname: cheapest-gpu-training\n\nresources:\n any_of:\n - accelerators: H100:8\n - accelerators: A100-80GB:8\n - accelerators: A100:8\n\nrun: |\n torchrun --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE train.py\n```\n\nYou can also mix different GPU types across clouds:\n\n```yaml\n# mixed-gpus.yaml\nname: mixed-gpu-training\n\nresources:\n any_of:\n - infra: aws\n accelerators: A10g:4\n - infra: gcp\n accelerators: L4:4\n - infra: azure\n accelerators: A100:1\n - infra: lambda\n accelerators: A100:1\n\nrun: |\n python train.py\n```\n\nThe optimizer evaluates all options together, considering current prices,\navailability, and any quotas, then launches on the cheapest option.\n\n### 1.2 ordered -- Strict Priority Fallback\n\nUse `ordered` when you have a strict preference. SkyPilot tries each option\nin order and uses the first one that succeeds.\n\n```yaml\n# priority-fallback.yaml\nname: priority-training\n\nresources:\n ordered:\n - infra: aws/us-east-1\n accelerators: A100:8\n - infra: gcp/us-central1\n accelerators: A100:8\n - infra: aws/us-west-2\n accelerators: A100:4\n - infra: gcp/europe-west4\n accelerators: A100:4\n\nrun: |\n python train.py\n```\n\nThis is useful when you have reserved instances or negotiated pricing on a\nspecific cloud/region and want to use those first.\n\n### 1.3 Kubernetes + Cloud Fallback\n\nA common pattern is to try on-prem Kubernetes first, then fall back to cloud:\n\n```yaml\n# k8s-cloud-fallback.yaml\nname: k8s-with-cloud-fallback\n\nresources:\n ordered:\n # Try on-prem K8s cluster first (free/cheaper)\n - infra: k8s/my-onprem-context\n accelerators: A100:4\n # Fall back to cloud K8s\n - infra: k8s/gke-prod-cluster\n accelerators: A100:4\n # Fall back to cloud VMs\n - infra: aws\n accelerators: A100:4\n - infra: gcp\n accelerators: A100:4\n\nrun: |\n python train.py\n```\n\n### 1.4 SSH Node Pools\n\nSkyPilot can use your own bare-metal machines or VMs via SSH node pools.\nFirst, define node pools in `~/.sky/ssh_node_pools.yaml`:\n\n```yaml\n# ~/.sky/ssh_node_pools.yaml\nmy-gpu-cluster:\n hosts:\n - 10.0.1.10\n - 10.0.1.11\n - 10.0.1.12\n\nmy-inference-box:\n hosts:\n - gpu-server.internal.example.com\n```\n\nThen reference them in your task YAML:\n\n```yaml\n# ssh-nodes.yaml\nname: train-on-ssh-nodes\n\nresources:\n infra: ssh/my-gpu-cluster\n accelerators: A100:8\n\nrun: |\n python train.py\n```\n\nSSH node pools can also participate in `any_of` or `ordered` fallback:\n\n```yaml\nresources:\n ordered:\n - infra: ssh/my-gpu-cluster\n accelerators: A100:8\n - infra: aws\n accelerators: A100:8\n```\n\n### 1.5 Wildcard Patterns in infra\n\nThe `infra` field supports `*` as a wildcard for any component\n(cloud, region, zone):\n\n```yaml\n# Any cloud, specific zone\nresources:\n infra: \"*/us-east-1/us-east-1a\"\n accelerators: A100:4\n```\n\n```yaml\n# Specific cloud, any region, specific zone suffix\nresources:\n infra: \"aws/*/us-east-1a\"\n accelerators: A100:4\n```\n\n```yaml\n# Any cloud, any region (equivalent to not specifying infra)\nresources:\n infra: \"*\"\n accelerators: A100:4\n```\n\nThe infra format is `cloud[/region[/zone]]`. For Kubernetes, the format is\n`k8s/context-name` (context names may contain slashes).\n\n### 1.6 Multi-Region Strategies\n\nTarget a specific region across any cloud, or multiple specific regions:\n\n```yaml\n# Multi-region with any_of\nresources:\n any_of:\n - infra: aws/us-east-1\n accelerators: H100:8\n - infra: aws/us-west-2\n accelerators: H100:8\n - infra: gcp/us-central1\n accelerators: H100:8\n - infra: gcp/us-east4\n accelerators: H100:8\n```\n\nFor single-cloud multi-region with the same GPU:\n\n```yaml\n# Will automatically try all available regions for this cloud\nresources:\n infra: aws\n accelerators: H100:8\n```\n\nSkyPilot's optimizer automatically considers all regions within the specified\ncloud and picks the cheapest available one. You only need `any_of` or\n`ordered` when you want to cross cloud boundaries or mix GPU types.\n\n### 1.7 Accelerator Sets and Lists\n\nFor single-resource blocks, you can use accelerator sets (unordered, cost-\noptimized) or lists (ordered, priority-based):\n\n```yaml\n# Unordered set: SkyPilot picks the cheapest available\nresources:\n accelerators: {A100:1, V100:1, L4:1, T4:1}\n```\n\n```yaml\n# Ordered list: SkyPilot tries in order\nresources:\n accelerators: [A100:1, V100:1, L4:1, T4:1]\n```\n\n---\n\n## 2. Distributed Training\n\nSkyPilot supports multi-node distributed training with automatic cluster\nsetup. It provides environment variables on every node for coordination.\n\n### 2.1 SkyPilot Environment Variables for Distributed Training\n\nOn each node, SkyPilot sets:\n\n| Variable | Description |\n|---|---|\n| `SKYPILOT_NODE_IPS` | Newline-separated list of all node IPs (head node first) |\n| `SKYPILOT_NODE_RANK` | 0-indexed rank of the current node (head = 0) |\n| `SKYPILOT_NUM_NODES` | Total number of nodes in the cluster |\n| `SKYPILOT_NUM_GPUS_PER_NODE` | Number of GPUs on the current node |\n\n### 2.2 PyTorch DDP Multi-Node Setup\n\nThe standard pattern uses `torchrun` with SkyPilot's environment variables:\n\n```yaml\n# ddp-training.yaml\nname: pytorch-ddp\n\nresources:\n accelerators: A100:8\n\nnum_nodes: 4\n\nsetup: |\n pip install torch torchvision\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n\n torchrun \\\n --nnodes=$NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$HEAD_IP \\\n --master_port=12355 \\\n train_ddp.py \\\n --epochs 100 \\\n --batch-size 256\n```\n\nA minimal `train_ddp.py` would use:\n\n```python\nimport os\n\nimport torch\nimport torch.distributed as dist\n\ndist.init_process_group(backend='nccl')\nlocal_rank = int(os.environ['LOCAL_RANK'])\ntorch.cuda.set_device(local_rank)\n# ... training loop\n```\n\n### 2.3 DeepSpeed ZeRO-3 Multi-Node\n\n```yaml\n# deepspeed-zero3.yaml\nname: deepspeed-training\n\nresources:\n accelerators: A100-80GB:8\n\nnum_nodes: 2\n\nfile_mounts:\n /workspace: .\n\nsetup: |\n pip install deepspeed torch transformers accelerate\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n\n # Create hostfile for DeepSpeed\n echo \"$SKYPILOT_NODE_IPS\" | while read ip; do\n echo \"$ip slots=$SKYPILOT_NUM_GPUS_PER_NODE\"\n done > /tmp/hostfile\n\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n deepspeed \\\n --hostfile /tmp/hostfile \\\n --master_addr $HEAD_IP \\\n --master_port 29500 \\\n --num_nodes $NUM_NODES \\\n --num_gpus $SKYPILOT_NUM_GPUS_PER_NODE \\\n /workspace/train.py \\\n --deepspeed /workspace/ds_config_zero3.json \\\n --model_name_or_path meta-llama/Llama-2-7b-hf \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 4\n fi\n```\n\nExample `ds_config_zero3.json`:\n\n```json\n{\n \"bf16\": { \"enabled\": true },\n \"zero_optimization\": {\n \"stage\": 3,\n \"offload_optimizer\": { \"device\": \"none\" },\n \"offload_param\": { \"device\": \"none\" },\n \"overlap_comm\": true,\n \"contiguous_gradients\": true,\n \"reduce_bucket_size\": 5e7,\n \"stage3_prefetch_bucket_size\": 5e7,\n \"stage3_param_persistence_threshold\": 1e5\n },\n \"gradient_accumulation_steps\": 4,\n \"train_micro_batch_size_per_gpu\": 2,\n \"wall_clock_breakdown\": false\n}\n```\n\n### 2.4 Ray Train Integration\n\nSkyPilot includes a helper script to start a Ray cluster across nodes. Use it\nfor Ray Train, Ray Tune, and other Ray-based distributed workloads:\n\n```yaml\n# ray-train.yaml\nname: ray-distributed-training\n\nresources:\n accelerators: A100:8\n ports:\n - 8280 # Ray dashboard\n\nnum_nodes: 2\n\nenvs:\n HF_HUB_ENABLE_HF_TRANSFER: \"1\"\n\nsetup: |\n pip install \"ray[default,train]\" torch transformers\n\nrun: |\n head_ip=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n\n # Use the built-in SkyPilot Ray start script\n export RAY_HEAD_PORT=6385\n export RAY_DASHBOARD_PORT=8280\n export RAY_DASHBOARD_HOST=0.0.0.0\n export RAY_HEAD_IP_ADDRESS=\"$head_ip\"\n ~/sky_templates/ray/start_cluster\n\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n export RAY_ADDRESS=\"http://localhost:8280\"\n ray job submit --address=\"$RAY_ADDRESS\" \\\n -- python train_with_ray.py\n fi\n```\n\n### 2.5 FSDP Configuration\n\nPyTorch Fully Sharded Data Parallel (FSDP) with SkyPilot:\n\n```yaml\n# fsdp-training.yaml\nname: fsdp-llama\n\nresources:\n accelerators: A100-80GB:8\n\nnum_nodes: 2\n\nsetup: |\n pip install torch transformers accelerate\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n TOTAL_GPUS=$((NUM_NODES * SKYPILOT_NUM_GPUS_PER_NODE))\n\n accelerate launch \\\n --num_processes $TOTAL_GPUS \\\n --num_machines $NUM_NODES \\\n --machine_rank $SKYPILOT_NODE_RANK \\\n --main_process_ip $HEAD_IP \\\n --main_process_port 29500 \\\n --use_fsdp \\\n --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP \\\n --fsdp_sharding_strategy FULL_SHARD \\\n --fsdp_state_dict_type SHARDED_STATE_DICT \\\n --mixed_precision bf16 \\\n train_fsdp.py \\\n --model_name meta-llama/Llama-2-7b-hf \\\n --per_device_train_batch_size 2\n```\n\n### 2.6 NCCL Environment Tuning\n\nFor multi-node GPU training, NCCL configuration is critical for performance.\nSet these via `envs:` in your YAML:\n\n```yaml\n# nccl-tuned.yaml\nname: nccl-optimized-training\n\nresources:\n accelerators: H100:8\n\nnum_nodes: 4\n\nenvs:\n # Debug: set to INFO or WARN for troubleshooting\n NCCL_DEBUG: WARN\n # Network interface selection (check with `ip addr` on your nodes)\n NCCL_SOCKET_IFNAME: eth0\n # Use tree algorithm for better all-reduce on large clusters\n NCCL_ALGO: Tree\n # Avoid CUDA stream recording overhead\n TORCH_NCCL_AVOID_RECORD_STREAMS: \"1\"\n # InfiniBand settings (for clouds with IB/RDMA)\n NCCL_IB_DISABLE: \"0\"\n NCCL_IB_GID_INDEX: 3\n # Cross-node buffer sizes\n NCCL_BUFFSIZE: \"8388608\"\n # P2P transport (NVLink within node, network across nodes)\n NCCL_P2P_LEVEL: NVL\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$HEAD_IP \\\n --master_port=12355 \\\n train.py\n```\n\n### 2.7 Common Distributed Training Gotchas\n\n1. **Port conflicts**: Always use a non-default master port (not 29500)\n if running multiple training jobs on the same cluster. Or use SkyPilot's\n port allocation.\n\n2. **Head node vs worker pattern**: Only the head node (`SKYPILOT_NODE_RANK\n == 0`) should launch the training coordinator (e.g., DeepSpeed launcher,\n Ray job submit). Worker nodes run the setup + the Ray/torchrun rendezvous.\n\n3. **Hanging at initialization**: Usually caused by NCCL not finding the right\n network interface. Set `NCCL_SOCKET_IFNAME` to the correct interface.\n Use `NCCL_DEBUG=INFO` to diagnose.\n\n4. **OOM on multi-node**: Gradient synchronization buffers consume extra\n memory. Reduce per-device batch size or use gradient accumulation.\n\n5. **Run section runs on ALL nodes**: The `run:` section executes on every\n node in the cluster. Use `if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]` to gate\n head-only logic (like launching the coordinator or downloading data).\n\n6. **Setup runs on ALL nodes**: The `setup:` section also runs on all nodes.\n This is typically correct (install dependencies everywhere), but be aware\n of it when doing one-time data preparation.\n\n---\n\n## 3. Managed Jobs Production Patterns\n\nManaged jobs (`sky jobs launch`) provide automatic recovery from preemptions,\nspot instance failures, and transient errors. They run on a lightweight jobs\ncontroller that monitors and relaunches your workload.\n\n### 3.1 job_recovery Configuration\n\nThe `job_recovery` field controls how managed jobs handle failures.\n\n**Strategy options:**\n- `EAGER_NEXT_REGION` (default): On preemption, immediately try a different\n region/cloud before retrying the same region.\n- `FAILOVER`: On preemption, first retry in the same region, then fail over\n to other regions.\n\n**Full configuration as a dict:**\n\n```yaml\n# managed-job-recovery.yaml\nname: resilient-training\n\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n recover_on_exit_codes: [1, 137]\n\nrun: |\n python train.py --checkpoint-dir /ckpts\n```\n\n**Shorthand (strategy only):**\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery: FAILOVER\n```\n\n**Field reference:**\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `strategy` | string | `EAGER_NEXT_REGION` | Recovery strategy name |\n| `max_restarts_on_errors` | int | 0 | Max restarts on non-preemption errors |\n| `recover_on_exit_codes` | int or list[int] | none | Exit codes that always trigger recovery (ignores max_restarts_on_errors counter) |\n\nWhen `recover_on_exit_codes` is set, any job failure with a matching exit code\ntriggers recovery without counting against `max_restarts_on_errors`. This is\nuseful for known transient failures.\n\nTo give up on a job that cannot get resources, set the top-level\n`job.wait_for_scheduling_timeout` (not part of `job_recovery`). The job ends\n`CANCELLED` if it has not started within that long of being submitted, whether\nit is retrying cloud VM launches or waiting in a Kubernetes queue such as\nKueue. Later recoveries do not re-arm it.\n\n```yaml\njob:\n wait_for_scheduling_timeout: 2h # duration (e.g. 90s, 30m, 2h) or int seconds\nresources:\n accelerators: A100:8\n job_recovery: FAILOVER\n```\n\n### 3.2 Checkpointing Patterns\n\nFor spot instances, checkpointing to cloud storage is essential. The pattern\nis: save checkpoints periodically to a mounted bucket, and on restart, resume\nfrom the latest checkpoint.\n\n```yaml\n# checkpointed-training.yaml\nname: checkpointed-llm-training\n\nresources:\n accelerators: A100-80GB:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 5\n\nfile_mounts:\n /ckpts:\n name: my-training-checkpoints # S3/GCS/R2 bucket\n mode: MOUNT\n\nrun: |\n # Resume from latest checkpoint if it exists\n CKPT_ARG=\"\"\n LATEST=$(ls -td /ckpts/checkpoint-* 2>/dev/null | head -1)\n if [ -n \"$LATEST\" ]; then\n echo \"Resuming from checkpoint: $LATEST\"\n CKPT_ARG=\"--resume_from_checkpoint $LATEST\"\n fi\n\n python train.py \\\n --output_dir /ckpts \\\n --save_steps 500 \\\n $CKPT_ARG\n```\n\nLaunch as a managed job:\n\n```bash\nsky jobs launch checkpointed-training.yaml\n```\n\n### 3.3 Hyperparameter Sweeps as Managed Jobs\n\nLaunch multiple managed jobs for a hyperparameter sweep. Each job runs\nindependently with automatic recovery:\n\n```bash\n# Launch sweep via CLI\nfor lr in 1e-4 5e-5 1e-5; do\n for bs in 16 32 64; do\n sky jobs launch sweep.yaml \\\n --env LR=$lr \\\n --env BATCH_SIZE=$bs \\\n -n \"sweep-lr${lr}-bs${bs}\" \\\n -y\n done\ndone\n```\n\n```yaml\n# sweep.yaml\nname: hp-sweep\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 2\n\nenvs:\n LR: 1e-4\n BATCH_SIZE: 32\n\nfile_mounts:\n /results:\n name: sweep-results-bucket\n mode: MOUNT\n\nrun: |\n python train.py \\\n --learning-rate $LR \\\n --batch-size $BATCH_SIZE \\\n --output-dir \"/results/lr${LR}_bs${BATCH_SIZE}\"\n```\n\nMonitor all running jobs:\n\n```bash\nsky jobs queue -o json\n```\n\n### 3.4 Job Pools (Experimental)\n\nJob pools pre-provision clusters that multiple jobs can be submitted to,\navoiding cold-start overhead. Pools are defined with a `service:` section\ncontaining `pool:` configuration.\n\n**Create a pool:**\n\n```yaml\n# gpu-pool.yaml\nname: gpu-pool\n\nresources:\n accelerators: A100:8\n\nservice:\n pool:\n workers: 3\n```\n\n```bash\n# Apply the pool configuration\nsky jobs pool apply gpu-pool.yaml -p my-gpu-pool\n```\n\n**Submit jobs to the pool:**\n\n```bash\nsky jobs launch job.yaml --pool my-gpu-pool\n```\n\n**Pool with autoscaling:**\n\n```yaml\n# autoscaling-pool.yaml\nname: autoscaling-pool\n\nresources:\n accelerators: A100:4\n\nservice:\n pool:\n min_workers: 1\n max_workers: 10\n queue_length_threshold: 2\n upscale_delay_seconds: 60\n downscale_delay_seconds: 300\n```\n\n**Pool management commands:**\n\n```bash\n# Check pool status\nsky jobs pool status\n\n# Check a specific pool\nsky jobs pool status my-gpu-pool\n\n# Update number of workers\nsky jobs pool apply -p my-gpu-pool --workers 5\n\n# Delete a pool\nsky jobs pool down my-gpu-pool\n\n# Delete all pools\nsky jobs pool down -a\n```\n\n---\n\n## 4. SkyServe Advanced\n\nSkyServe deploys model serving endpoints with autoscaling, multi-cloud\nreplicas, rolling updates, and spot instance support.\n\n### 4.1 Autoscaling Configuration\n\n```yaml\n# autoscaling-service.yaml\nname: llm-service\n\nresources:\n accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe:\n path: /health\n initial_delay_seconds: 300\n timeout_seconds: 10\n replica_policy:\n min_replicas: 2\n max_replicas: 10\n target_qps_per_replica: 5.0\n upscale_delay_seconds: 120\n downscale_delay_seconds: 600\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-2-7b-hf \\\n --port 8080\n```\n\n**Autoscaling field reference:**\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `min_replicas` | int | 1 | Minimum number of replicas |\n| `max_replicas` | int | same as min | Maximum number of replicas |\n| `target_qps_per_replica` | float | required if max != min | Target QPS per replica for scaling |\n| `upscale_delay_seconds` | int | 300 | Seconds target must exceed before scaling up |\n| `downscale_delay_seconds` | int | 1200 | Seconds target must be below before scaling down |\n| `num_overprovision` | int | none | Extra replicas beyond what autoscaling calculates |\n\nDeploy and manage:\n\n```bash\n# Deploy the service\nsky serve up autoscaling-service.yaml -n my-llm\n\n# Check service status\nsky serve status my-llm\n\n# Get the service endpoint\nsky serve status my-llm --endpoint\n\n# Tail controller logs\nsky serve logs --controller my-llm\n```\n\n### 4.2 Rolling Updates vs Blue-Green Updates\n\n**Rolling update** (default): Gradually replaces replicas one by one. Some\nold and new replicas serve traffic simultaneously during the transition.\n\n```bash\n# Update with rolling strategy (default)\nsky serve update my-llm updated-service.yaml\n\n# Explicitly specify rolling\nsky serve update --mode rolling my-llm updated-service.yaml\n```\n\n**Blue-green update**: Provisions all new replicas first, then switches\ntraffic atomically. No mixed versions serve traffic simultaneously.\n\n```bash\n# Blue-green update\nsky serve update --mode blue_green my-llm updated-service.yaml\n```\n\nBlue-green requires enough quota to run both old and new replicas\nsimultaneously during the transition.\n\n### 4.3 Spot Replicas for Cost Savings\n\nUse spot instances for serving replicas with automatic on-demand fallback:\n\n```yaml\n# spot-serving.yaml\nname: spot-llm-service\n\nresources:\n accelerators: A100:1\n use_spot: true\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 3\n max_replicas: 8\n target_qps_per_replica: 5.0\n # Maintain 1 on-demand replica as a baseline\n base_ondemand_fallback_replicas: 1\n # Dynamically add on-demand replicas if spot is unavailable\n dynamic_ondemand_fallback: true\n\nrun: |\n python serve.py --port 8080\n```\n\n### 4.4 TLS/HTTPS Configuration\n\nEnable TLS termination at the load balancer:\n\n```yaml\n# tls-service.yaml\nname: secure-service\n\nresources:\n accelerators: L4:1\n ports:\n - 8080\n\nfile_mounts:\n /certs/server.key: ./certs/server.key\n /certs/server.crt: ./certs/server.crt\n\nservice:\n readiness_probe: /health\n replicas: 2\n tls:\n keyfile: /certs/server.key\n certfile: /certs/server.crt\n\nrun: |\n python serve.py --port 8080\n```\n\n### 4.5 Readiness Probe Configuration\n\nThe readiness probe determines when a replica is ready to receive traffic.\n\n**Simple GET probe (most common):**\n\n```yaml\nservice:\n readiness_probe: /health\n replicas: 2\n```\n\n**Full probe configuration:**\n\n```yaml\nservice:\n readiness_probe:\n path: /v1/models\n initial_delay_seconds: 600\n timeout_seconds: 30\n headers:\n Authorization: \"Bearer test-token\"\n replicas: 1\n```\n\n**POST probe (useful for LLMs that need a generation test):**\n\n```yaml\nservice:\n readiness_probe:\n path: /v1/completions\n post_data:\n model: llama-2-7b\n prompt: \"Hello\"\n max_tokens: 1\n initial_delay_seconds: 900\n timeout_seconds: 60\n replicas: 1\n```\n\n**Default values:**\n- `initial_delay_seconds`: 1200 (20 minutes) -- readiness failures during\n this period are ignored\n- `timeout_seconds`: 15\n\n### 4.6 Multi-Cloud Replicas\n\nDeploy replicas across multiple clouds for availability and cost optimization:\n\n```yaml\n# multi-cloud-serving.yaml\nname: multi-cloud-llm\n\nresources:\n any_of:\n - infra: aws\n accelerators: A100:1\n - infra: gcp\n accelerators: A100:1\n - infra: azure\n accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 3\n max_replicas: 10\n target_qps_per_replica: 5.0\n\nrun: |\n python serve.py --port 8080\n```\n\nSkyPilot distributes replicas across available clouds for redundancy and\npicks the cheapest options.\n\n### 4.7 Load Balancing Policies\n\nSkyServe supports different load balancing policies:\n\n```yaml\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 2\n max_replicas: 8\n target_qps_per_replica: 5.0\n load_balancing_policy: round_robin\n```\n\nFor heterogeneous GPU deployments, use instance-aware load balancing:\n\n```yaml\n# Instance-aware balancing across different GPU types\nresources:\n any_of:\n - accelerators: H100:1\n - accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 2\n max_replicas: 10\n target_qps_per_replica:\n H100: 10.0\n A100: 5.0\n load_balancing_policy: instance_aware_least_load\n```\n\n---\n\n## 5. Kubernetes Integration\n\nSkyPilot treats Kubernetes clusters as first-class infrastructure alongside\ncloud providers.\n\n### 5.1 Using K8s Contexts\n\nSkyPilot uses your kubeconfig contexts. Reference them with `infra: k8s/`:\n\n```yaml\n# Use a specific K8s context\nresources:\n infra: k8s/my-cluster-context\n accelerators: A100:4\n```\n\n```yaml\n# Use the default K8s context (current-context in kubeconfig)\nresources:\n infra: kubernetes\n accelerators: A100:4\n```\n\n```yaml\n# EKS context (contexts can contain special characters)\nresources:\n infra: k8s/arn:aws:eks:us-east-1:123456789012:cluster/my-cluster\n accelerators: A100:4\n```\n\n### 5.2 Pod Config Overrides\n\nCustomize the pod spec for SkyPilot pods via `~/.sky/config.yaml`. This\nallows setting tolerations, nodeSelectors, labels, and other Kubernetes-\nspecific configurations.\n\n```yaml\n# ~/.sky/config.yaml\nkubernetes:\n allowed_contexts:\n - my-production-cluster\n - my-dev-cluster\n context_configs:\n my-production-cluster:\n pod_config:\n spec:\n tolerations:\n - key: \"nvidia.com/gpu\"\n operator: \"Exists\"\n effect: \"NoSchedule\"\n nodeSelector:\n node-type: gpu\n team: ml-training\n containers:\n - name: skypilot\n resources:\n limits:\n nvidia.com/gpu: \"1\"\n metadata:\n labels:\n team: ml-platform\n environment: production\n custom_metadata:\n labels:\n app: skypilot\n managed-by: skypilot\n my-dev-cluster:\n pod_config:\n spec:\n nodeSelector:\n node-type: dev-gpu\n```\n\n### 5.3 Multi-Cluster K8s with any_of\n\nSpread workloads across multiple Kubernetes clusters:\n\n```yaml\n# multi-k8s.yaml\nname: multi-cluster-training\n\nresources:\n any_of:\n - infra: k8s/us-east-cluster\n accelerators: A100:8\n - infra: k8s/eu-west-cluster\n accelerators: A100:8\n - infra: k8s/asia-cluster\n accelerators: H100:8\n\nrun: |\n python train.py\n```\n\nCombined with cloud fallback:\n\n```yaml\nresources:\n ordered:\n - infra: k8s/onprem-cluster\n accelerators: A100:8\n - infra: k8s/gke-cluster\n accelerators: A100:8\n - infra: aws\n accelerators: A100:8\n```\n\n### 5.4 GPU Detection and Labeling\n\nSkyPilot needs GPU labels on Kubernetes nodes to schedule GPU workloads.\nUse `sky gpus label` to automatically detect and label GPUs:\n\n```bash\n# Label GPUs in the current Kubernetes context\nsky gpus label\n\n# Label GPUs in a specific context\nsky gpus label --context my-k8s-cluster\n\n# Start labeling without waiting for completion\nsky gpus label --async\n\n# Cleanup labeling resources when done\nsky gpus label --cleanup\n```\n\nCheck available GPUs across all configured infrastructure:\n\n```bash\n# List all available GPUs\nsky gpus list\n\n# Show GPUs for a specific type\nsky gpus list A100\n\n# Show GPUs available on Kubernetes\nsky gpus list --infra kubernetes\n```\n\n### 5.5 Kubernetes Configuration in ~/.sky/config.yaml\n\n```yaml\n# ~/.sky/config.yaml\nkubernetes:\n # Restrict which contexts SkyPilot can use\n allowed_contexts:\n - gke-prod\n - eks-staging\n\n # Port exposure mode\n ports: loadbalancer # or nodeport, ingress, podip\n\n # Cluster autoscaler integration\n autoscaler: gke # or karpenter, generic\n\n # Per-context configuration\n context_configs:\n gke-prod:\n provision_timeout: 600\n pod_config:\n spec:\n nodeSelector:\n cloud.google.com/gke-accelerator: nvidia-tesla-a100\n```\n\n---\n\n## 6. API Server and Team Setup\n\nThe SkyPilot API server enables team collaboration by providing a shared\ncontrol plane for launching and managing clusters, jobs, and services.\n\n### 6.1 Local API Server\n\nFor individual use or development:\n\n```bash\n# Start the local API server\nsky api start\n\n# Check server status\nsky api status\n\n# Stop the server\nsky api stop\n```\n\nThe local API server runs on port 46580 by default. It starts automatically\nwhen you run any `sky` command.\n\n### 6.2 Remote API Server with Helm\n\nDeploy a shared API server on Kubernetes for team use:\n\n```bash\n# Add the SkyPilot Helm repo\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\n\n# Build chart dependencies\nhelm dependency build ./charts/skypilot\n\n# Deploy the API server\nNAMESPACE=skypilot\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace\n```\n\nFor custom images (e.g., testing local changes):\n\n```bash\n# Build and push custom image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 \\\n -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy with custom image\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n --namespace skypilot \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\nUpgrade an existing deployment (always use `--reuse-values`):\n\n```bash\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n### 6.3 PostgreSQL Backend\n\nFor production deployments, use PostgreSQL instead of SQLite:\n\n```bash\n# Create a secret with the database connection string\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/skypilot\"\n\n# Deploy with PostgreSQL backend\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n -n skypilot --create-namespace \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\nOr set the connection string directly (less secure, not recommended for\nproduction):\n\n```bash\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n -n skypilot --create-namespace \\\n --set apiService.dbConnectionString=\"postgresql://user:pass@host:5432/db\"\n```\n\n### 6.4 Authentication\n\nConnect to a remote API server:\n\n```bash\n# OAuth2 browser login\nsky api login -e https://skypilot.example.com\n\n# Service account token login\nsky api login -e https://skypilot.example.com --token sky_abc123...\n```\n\nCheck the current connection:\n\n```bash\nsky api status\n```\n\n---\n\n## 7. Cost Optimization\n\nSkyPilot provides multiple mechanisms to minimize cloud spending while\nmaximizing GPU availability.\n\n### 7.1 Automatic Cloud Selection (Cheapest)\n\nBy default, SkyPilot's optimizer picks the cheapest cloud/region for your\nrequested resources:\n\n```yaml\n# SkyPilot automatically finds the cheapest option\nresources:\n accelerators: A100:4\n```\n\n```bash\n# Preview the optimizer's choice without launching\nsky launch --dryrun task.yaml\n```\n\nThe `--dryrun` flag shows you which cloud, region, instance type, and price\nSkyPilot would choose, without actually provisioning anything.\n\n### 7.2 Spot Instances\n\nSpot/preemptible instances offer 60-90% savings over on-demand:\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n```\n\nFor production spot workloads, combine with managed jobs for automatic\nrecovery:\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n```\n\n```bash\nsky jobs launch spot-training.yaml\n```\n\n### 7.3 Dryrun to Preview Costs\n\nAlways preview costs before launching expensive workloads:\n\n```bash\n# Show what would be launched and estimated cost\nsky launch --dryrun gpu-training.yaml\n\n# Show cost for a specific cloud\nsky launch --dryrun --infra aws gpu-training.yaml\n```\n\n### 7.4 Cost Report\n\nTrack spending across all clusters:\n\n```bash\n# Show cost report for active clusters\nsky cost-report\n\n# Show all clusters including terminated (last 30 days)\nsky cost-report --all\n\n# Show costs for specific number of days\nsky cost-report --days 7\n```\n\n### 7.5 Autostop Configuration\n\nAutomatically stop or terminate idle clusters to avoid waste. Autostop is\nconfigured in the `resources:` section of your YAML.\n\n**Simple autostop (idle minutes):**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop: 30 # Stop after 30 minutes idle\n```\n\n**Full autostop configuration:**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop:\n idle_minutes: 30\n down: true # Terminate instead of just stopping\n wait_for: jobs_and_ssh # Reset timer while jobs or SSH sessions active\n```\n\n**wait_for options:**\n\n| Value | Description |\n|---|---|\n| `jobs_and_ssh` | (default) Wait for running jobs AND open SSH sessions |\n| `jobs` | Only wait for running jobs to finish |\n| `none` | Stop immediately after idle_minutes, regardless of activity |\n\n**Disable autostop:**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop: false\n```\n\n**CLI overrides:**\n\n```bash\n# Set autostop on an existing cluster\nsky autostop my-cluster -i 60\n\n# Set autodown (terminate when idle)\nsky autostop my-cluster -i 30 --down\n\n# Cancel autostop\nsky autostop my-cluster --cancel\n```\n\n### 7.6 Autostop Hooks for Pre-Shutdown Actions\n\nRun a custom script before the cluster stops or terminates. This is useful\nfor saving state, uploading logs, or graceful shutdown procedures:\n\n```yaml\nresources:\n accelerators: A100:4\n autostop:\n idle_minutes: 30\n down: true\n hook: |\n echo \"Cluster shutting down, saving state...\"\n cp -r /tmp/logs /data/saved-logs/\n python cleanup.py\n hook_timeout: 120 # Seconds to wait for hook to complete\n```\n\nThe hook runs before the cluster is stopped or terminated. If the hook\nexceeds `hook_timeout` seconds, it is killed and the shutdown proceeds.\n\n### 7.7 Combining Cost Optimization Strategies\n\nFor maximum savings, combine multiple strategies:\n\n```yaml\n# maximum-savings.yaml\nname: cost-optimized-training\n\nresources:\n any_of:\n - infra: aws\n accelerators: A100:8\n - infra: gcp\n accelerators: A100:8\n - infra: lambda\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 5\n autostop:\n idle_minutes: 10\n down: true\n\nfile_mounts:\n /ckpts:\n name: training-checkpoints\n mode: MOUNT\n\nrun: |\n python train.py --checkpoint-dir /ckpts --resume-latest\n```\n\nLaunch as a managed job for automatic recovery:\n\n```bash\nsky jobs launch maximum-savings.yaml\n```\n\nThis configuration:\n1. Uses `any_of` to find the cheapest cloud\n2. Uses spot instances for 60-90% savings\n3. Recovers automatically from preemptions via managed jobs\n4. Auto-terminates idle clusters\n5. Checkpoints to cloud storage for seamless recovery\n","CLAUDE.md":"# CLAUDE.md - SkyPilot Development Guide\n\nThis document provides guidance for AI assistants working with the SkyPilot codebase.\n\n## Project Overview\n\nSkyPilot is a system to run, manage, and scale AI workloads on any AI infrastructure. It provides a unified interface across 25+ cloud providers (AWS, GCP, Azure, Kubernetes, Slurm, and many others), enabling users to launch compute resources, run jobs, and serve models without vendor lock-in.\n\n**Key capabilities:**\n- **Clusters**: Launch and manage compute clusters across clouds and Kubernetes\n- **Managed jobs**: Run jobs with automatic recovery from preemptions and failures\n- **Multi-cloud and multi-Kubernetes**: Unified interface across 25+ clouds and multiple K8s clusters\n- Cost optimization and GPU availability maximization\n\n## Repository Structure\n\n```\nskypilot/\n├── sky/ # Main source code\n│ ├── __init__.py # Package exports and version\n│ ├── core.py # Core orchestration logic\n│ ├── task.py # Task definition and YAML parsing\n│ ├── resources.py # Cloud resource specifications\n│ ├── optimizer.py # Resource allocation optimizer\n│ ├── execution.py # Task execution pipeline\n│ ├── exceptions.py # Custom exceptions\n│ ├── skypilot_config.py # Configuration management\n│ ├── clouds/ # Cloud provider abstractions (25+ providers)\n│ ├── backends/ # Execution backends (Ray-based VM, Docker)\n│ ├── provision/ # Cloud resource provisioning\n│ ├── jobs/ # Managed job lifecycle\n│ ├── serve/ # Model serving (SkyServe)\n│ ├── skylet/ # On-cluster execution agent\n│ ├── client/ # SDK and CLI\n│ ├── server/ # API server and dashboard backend\n│ ├── dashboard/ # Web UI (Next.js)\n│ ├── utils/ # 50+ utility modules\n│ ├── catalog/ # Cloud pricing and instance catalogs\n│ ├── data/ # Storage and data handling\n│ ├── adaptors/ # Cloud-specific metadata adapters\n│ └── schemas/ # Protobuf definitions and API schemas\n├── tests/ # Test suite\n│ ├── unit_tests/ # Unit tests with subdirectories per module\n│ ├── smoke_tests/ # Quick validation tests\n│ ├── integration_tests/ # End-to-end tests\n│ ├── kubernetes/ # K8s-specific tests\n│ └── conftest.py # Pytest fixtures\n├── examples/ # 50+ usage examples\n├── llm/ # 45+ LLM training/serving examples\n├── docs/ # Sphinx documentation\n├── charts/ # Helm charts for K8s deployment\n└── format.sh # Code formatting script\n```\n\n## Development Setup\n\n### Environment Setup\n\n```bash\n# Create virtual environment with uv (Python 3.8-3.11 supported)\n# --seed is required to ensure pip is installed (needed for building wheels)\nuv venv --seed --python 3.11\nsource .venv/bin/activate\n\n# Install in editable mode with all cloud support\nuv pip install -e \".[all]\"\n# Or specific clouds only:\n# uv pip install -e \".[aws,gcp,kubernetes]\"\n\n# Install development dependencies\nuv pip install -r requirements-dev.txt\n\n# Optional: Install pre-commit hooks\nuv pip install pre-commit\npre-commit install\n```\n\n### Environment Variables\n\n```bash\nexport SKYPILOT_DEV=1 # Enable development mode\nexport SKYPILOT_DEBUG=1 # Enable debug logging\n```\n\n## Code Formatting and Linting\n\n**Always run `format.sh` before committing:**\n\n```bash\nbash format.sh # Format changed files (vs origin/master)\nbash format.sh --all # Format entire codebase\nbash format.sh --files path/to/file.py # Format specific files\n```\n\nThe script runs:\n1. **YAPF** - Google style for all Python code\n2. **isort** - Import sorting (Google profile)\n3. **mypy** - Type checking\n4. **pylint** - Linting with custom rules\n\n### Tool Versions (must match exactly)\n\nFrom `requirements-dev.txt`:\n- yapf==0.32.0\n- pylint==2.14.5\n- mypy==1.19.1\n- isort==5.12.0\n- pylint-quotes==0.2.3\n\n### Excluded from Formatting\n\n- `sky/schemas/generated/` - Auto-generated protobuf files\n- `build/` - Build artifacts\n\n## Testing\n\n### Running Tests\n\n```bash\n# Unit tests (fast, no cloud resources)\npytest tests/unit_tests/\n\n# Specific test file\npytest tests/unit_tests/test_resources.py\n```\n\n### CI Tests via PR Comments\n\nTrigger CI tests on pull requests using comments:\n- `/quicktest-core` - Run quick core tests\n- `/smoke-test` - Run smoke tests (launches cloud clusters)\n- `/smoke-test --kubernetes --postgres` - Test with PostgreSQL backend on Kubernetes\n- `/smoke-test --kubernetes --remote-server --postgres` - Test remote API server with PostgreSQL\n\n### Test Configuration\n\nFrom `pyproject.toml`:\n- Uses pytest-xdist with 16 parallel workers\n- Environment: `SKYPILOT_DEBUG=1`, `SKYPILOT_DEV=1`\n- Buildkite integration for CI\n\n### Checking Buildkite CI Status\n\nTo check CI test results for a PR:\n\n1. **Get Buildkite URL from GitHub**: Check the PR's commit status checks to find the Buildkite build URL (may take a moment to appear after triggering)\n2. **Fetch build logs via Buildkite API**: Use the Buildkite API to retrieve detailed logs and test results\n\n```bash\n# Example: Get build details (requires BUILDKITE_TOKEN)\ncurl -H \"Authorization: Bearer $BUILDKITE_TOKEN\" \\\n \"https://api.buildkite.com/v2/organizations/skypilot/pipelines/skypilot/builds/\"\n```\n\n## Code Style Guidelines\n\n### General Principles\n\n- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)\n- Use f-strings over `.format()` for readability\n- Use `class MyClass:` not `class MyClass(object):`\n- Use `abc` module for abstract classes\n- Use Python typing with `if typing.TYPE_CHECKING:` for type-only imports\n- Always place imports at the top of the file, not inside function definitions. If this causes a circular dependency, try to resolve it (e.g., restructure modules or use `if typing.TYPE_CHECKING:`). Only as a last resort, place the import inside the function with a comment explaining why. Import performance is not a valid reason for in-function imports; use `LazyImport` instead (see below).\n\n### TODOs and FIXMEs\n\nAlways include author attribution:\n```python\n# TODO(username): Description of what needs to be done\n# FIXME(username): Description of what needs fixing\n```\n\n### Exceptions vs Assertions\n\n- Use exceptions for errors (better error messages)\n- Use `assert` only for debugging/proof-checking\n\n### Lazy Imports\n\nFor modules with significant import time (>100ms) that are imported during `import sky`, use `LazyImport` from `sky/adaptors/common.py`:\n\n```python\nfrom sky.adaptors.common import LazyImport\nheavy_module = LazyImport('heavy_module')\n```\n\nMeasure import time with:\n```bash\npython -X importtime -c \"import sky\" 2> import.log\ntuna import.log # pip install tuna\n```\n\n## Architecture Patterns\n\n### Cloud Provider Abstraction\n\nAll cloud providers inherit from `sky.clouds.Cloud` (in `sky/clouds/cloud.py`). Cloud objects are **lightweight and stateless**. Key design principles:\n\n- Methods should be inexpensive to call\n- Don't store heavy state in cloud objects\n- Cache cloud-specific queries in `sky/clouds/utils/` modules\n- Each cloud implements feature flags via `CloudImplementationFeatures`\n\nAdding a new cloud provider:\n1. Create `sky/clouds/.py` inheriting from `Cloud`\n2. Implement required abstract methods\n3. Add provisioning logic in `sky/provision//`\n4. Register in `sky/clouds/__init__.py`\n\n### Client-Server Architecture\n\nSkyPilot uses a client-server model with API versioning:\n\n- Client (SDK/CLI) in `sky/client/`\n- Server in `sky/server/`\n- API version in `sky/server/constants.py`\n\n**Backward Compatibility Rules (from v0.10.0+):**\n- Changes must be backward compatible\n- Bump `API_VERSION` when introducing API changes\n- Use `@versions.minimal_api_version(N)` decorator for new SDK methods\n- Handle version differences with `versions.get_remote_api_version()`\n\n### Protobuf Regeneration\n\nWhen modifying `.proto` files in `sky/schemas/proto/`:\n\n```bash\npython -m grpc_tools.protoc \\\n --proto_path=sky/schemas/generated=sky/schemas/proto \\\n --python_out=. \\\n --grpc_python_out=. \\\n --pyi_out=. \\\n sky/schemas/proto/*.proto\n```\n\n### Dependency Management\n\nDependencies are defined in `sky/setup_files/dependencies.py`:\n\n- **Core dependencies**: Listed in `install_requires`\n- **Cloud-specific**: Defined in `extras_require` (e.g., `aws`, `gcp`, `kubernetes`)\n- **Development**: Listed in `requirements-dev.txt`\n\nWhen updating dependencies:\n\n1. Check version constraints carefully - some packages have breaking changes\n2. Consider Python version compatibility (3.8-3.11)\n3. Test with both minimum and latest allowed versions\n4. Document version constraints with comments explaining why\n\n## Key Modules Reference\n\n| Module | Purpose |\n|--------|---------|\n| `sky/core.py` | Core orchestration (launch, exec, stop, down) |\n| `sky/task.py` | Task YAML parsing and specification |\n| `sky/resources.py` | Resource requirements (GPUs, memory, disk) |\n| `sky/optimizer.py` | Cloud selection and cost optimization |\n| `sky/client/sdk.py` | Python SDK implementation |\n| `sky/client/cli/` | CLI commands |\n| `sky/backends/cloud_vm_ray_backend.py` | Main execution backend for clusters |\n| `sky/provision/provisioner.py` | Resource provisioning |\n| `sky/jobs/` | Managed jobs with recovery and scheduling |\n\n## Pull Request Guidelines\n\n1. **Branch from master**, create descriptive branch name\n2. **Run `format.sh`** before committing\n3. **Add tests** for core system changes\n4. **Run smoke tests** for significant changes\n5. **Include `Tested:` section** in PR description with test plan\n6. **Delete branch** after merging\n\n### PR Description Format\n\nPRs should include:\n- **Summary**: Brief description of changes (1-3 bullet points)\n- **Test plan**: How the changes were tested (commands run, manual verification steps)\n\n**Important**: Always generate a manual test plan describing how to verify the changes work correctly. Include specific commands, expected outputs, or UI verification steps. Whenever possible, add unit tests and smoke tests for the changes.\n\n### Commit Message Format\n\nUse the `[Area] Description` format:\n- `[Core]` for core system changes\n- `[CLI]` for CLI changes\n- `[API]` for API server changes\n- `[Docs]` for documentation\n- `[AWS]`, `[GCP]`, `[Azure]`, `[Kubernetes]` for cloud-specific changes\n- `[Jobs]` for managed jobs\n- `[Dashboard]` for web UI changes\n- `[Serve]` for model serving (SkyServe)\n- `[Test]` for testing changes\n- `[CI]` for CI/CD changes\n\nExamples:\n- `[Core] Fix cluster status refresh logic`\n- `[AWS] Add support for new instance types`\n- `[Docs] Update installation guide`\n\n## API Server Testing\n\n### Local API Server (Recommended for Development)\n\n```bash\n# Always restart API server after code changes to pick up changes\nsky api stop\nsky api start\n\n# Verify server is running\nsky api status\n```\n\n### Dashboard Development\n\n**For local API server development**, rebuild the dashboard before restarting:\n\n```bash\n# Install dependencies (first time or after package.json changes)\nnpm --prefix sky/dashboard install\n\n# Rebuild the dashboard\nnpm --prefix sky/dashboard run build\n\n# Then restart the API server\nsky api stop\nsky api start\n```\n\n**For remote API server (Docker/Kubernetes)**, the Dockerfile automatically builds the dashboard - no manual build needed before `docker build`.\n\nThe dashboard is a Next.js application. For development with hot reloading:\n\n```bash\n# Run dashboard in development mode (separate from API server)\ncd sky/dashboard\nnpm run dev\n```\n\n### Mocking Remote API Server Locally\n\nTo test remote API server behavior locally:\n\n```bash\n# Start local API server (runs on port 46580 by default)\nsky api stop\nsky api start\n\n# Forward to a different port to simulate remote server\nsocat TCP-LISTEN:46590,fork TCP:127.0.0.1:46580 &\n\n# Connect to the forwarded port as if it were a remote server\nsky api login -e http://127.0.0.1:46590\n```\n\n### Remote API Server (Kubernetes Deployment)\n\nFor testing with a remote API server on Kubernetes:\n\n```bash\n# Build local changes\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\nhelm dependency build ./charts/skypilot\n\n# Build Docker image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy (NEW installation)\nNAMESPACE=skypilot\nRELEASE_NAME=skypilot\nhelm upgrade --install $RELEASE_NAME ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n#### Upgrading Existing Deployments\n\n**CRITICAL:** Always use `--reuse-values` to preserve database/credential config:\n\n```bash\n# Upgrade existing deployment (keeps PostgreSQL, auth, etc.)\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n\n# Check current values / rollback if needed\nhelm get values skypilot -n skypilot\nhelm rollback skypilot -n skypilot\n```\n\n#### PostgreSQL Backend\n\n```bash\n# Create connection secret\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/db\"\n\n# Deploy with PostgreSQL\nhelm upgrade --install skypilot ./charts/skypilot -n skypilot \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\n## Critical Code Paths (Handle with Care)\n\nThe following modules contain complex, stateful logic that requires careful review when modifying:\n\n**Managed Jobs Recovery:**\n- `sky/jobs/controller.py` - Job lifecycle management, state transitions, async coordination\n- `sky/jobs/recovery_strategy.py` - Preemption recovery, retry logic for managed jobs\n- `sky/backends/cloud_vm_ray_backend.py` - Execution backend with complex state handling\n\nThese modules handle edge cases like preemption during job submission, controller failures mid-recovery, and race conditions between concurrent operations. Changes can have subtle effects on job reliability.\n\n**API Server Performance & Robustness:**\n- `sky/server/` - API server with memory efficiency, low latency requirements\n- `sky/backends/backend_utils.py` - Cluster status caching, network resilience, SSH handling\n\nThese modules are optimized for performance. Be cautious about adding blocking calls, memory-heavy operations, or changing caching behavior.\n\n**CLI/SDK Interface Design:**\n- `sky/client/cli/` - Command-line interface\n- `sky/client/sdk.py` - Python SDK\n\nBe cautious about adding new interfaces or changing existing UX significantly. Keep the interface clean and minimal for both CLI and SDK.\n\n**Backward Compatibility:**\n- Test with old server + new client, and old client + new server\n- Ensure existing clusters and jobs continue working after server upgrades\n- `/quicktest-core` tests some backward compatibility scenarios\n- See `docs/source/developers/CONTRIBUTING.md` for API versioning guidelines\n\n## Common Pitfalls\n\n1. **Always restart API server after code changes** - Run `sky api stop; sky api start` to pick up changes\n2. **Don't modify `sky/schemas/generated/`** - These are auto-generated\n3. **Match formatter versions exactly** - Version mismatches cause CI failures\n4. **Consider import time** - Heavy imports slow down CLI responsiveness\n5. **API versioning** - Always maintain backward compatibility\n\n## Useful Commands\n\n```bash\n# Profile CLI performance\nuv pip install py-spy\npy-spy record -t -o sky.svg -- python -m sky.cli status\n\n# Check cloud credentials\nsky check\n\n# View cluster status\nsky status\n\n# Launch a cluster\nsky launch \n\n# Execute a command on the cluster\nsky exec -- bash\n\n# SSH into the cluster\nssh \n\n# Launch a managed job\nsky jobs launch \n```\n\n## Documentation\n\n- **User docs**: https://docs.skypilot.co/\n- **Source**: `docs/source/` (Sphinx)\n- **Build docs**: See `.readthedocs.yml`\n\n## Additional Resources\n\n- **Full contributing guide**: `docs/source/developers/CONTRIBUTING.md`\n- **User docs**: https://docs.skypilot.co/\n- **GitHub Issues**: https://github.com/skypilot-org/skypilot/issues\n- **Discussions**: https://github.com/skypilot-org/skypilot/discussions\n- **Slack**: http://slack.skypilot.co\n","agent/skills/skypilot/references/examples.md":"# SkyPilot Examples Reference\n\nCopy-paste-ready YAML examples and CLI commands for common SkyPilot workflows.\n\n---\n\n## 1. Basic Examples\n\n### Minimal Task (Just a Run Command)\n\nThe simplest possible SkyPilot task -- just a command to run.\n\n```yaml\n# minimal.yaml\nname: hello-world\n\nrun: |\n echo \"Hello from SkyPilot!\"\n hostname\n uname -a\n```\n\n```bash\n# Launch on any available cloud (cheapest option selected automatically)\nsky launch -c hello minimal.yaml\n\n# Tear down when done\nsky down hello\n```\n\n### GPU Task with Specific Accelerator\n\nRequest a specific GPU type and count.\n\n```yaml\n# gpu-task.yaml\nname: gpu-check\n\nresources:\n # Request 1x A100 GPU\n accelerators: A100:1\n\nsetup: |\n pip install torch\n\nrun: |\n nvidia-smi\n python -c \"import torch; print(f'CUDA available: {torch.cuda.is_available()}'); print(f'GPU: {torch.cuda.get_device_name(0)}')\"\n```\n\n```bash\n# Launch on any cloud that has A100\nsky launch -c gpu-test gpu-task.yaml\n\n# Or pin to a specific cloud\nsky launch -c gpu-test gpu-task.yaml --infra aws\n\n# Tear down\nsky down gpu-test\n```\n\n### Task with Setup and Workdir\n\nSync local code and install dependencies before running.\n\n```yaml\n# train-task.yaml\nname: my-training\n\n# Sync current directory to ~/sky_workdir on the remote machine\nworkdir: .\n\nresources:\n accelerators: L4:1\n\n# Setup runs once on cluster creation, cached on subsequent runs\nsetup: |\n pip install torch torchvision\n pip install -r requirements.txt\n\n# Run executes your main command\nrun: |\n python train.py --epochs 10\n```\n\n```bash\n# Launch -- setup runs first, then the run command\nsky launch -c train train-task.yaml\n\n# Re-run with different args (setup is cached, only run executes)\nsky exec train train-task.yaml\n```\n\n### Task with Environment Variables and Secrets\n\nPass configuration via envs and sensitive values via secrets.\n\n```yaml\n# env-task.yaml\nname: training-with-config\n\nresources:\n accelerators: A100:1\n\n# Environment variables -- visible in logs and dashboard\nenvs:\n MODEL_NAME: bert-base-uncased\n BATCH_SIZE: \"32\"\n LEARNING_RATE: \"1e-4\"\n\n# Secrets -- redacted in logs and dashboard\nsecrets:\n HF_TOKEN: null # Will be passed via CLI\n WANDB_API_KEY: null # Will be passed via CLI\n\nsetup: |\n pip install transformers datasets wandb\n\nrun: |\n echo \"Training model: $MODEL_NAME with batch size $BATCH_SIZE\"\n wandb login $WANDB_API_KEY\n python train.py \\\n --model $MODEL_NAME \\\n --batch-size $BATCH_SIZE \\\n --lr $LEARNING_RATE\n```\n\n```bash\n# Pass secrets and override envs from the CLI\nHF_TOKEN=hf_abc123 WANDB_API_KEY=wk_xyz789 \\\n sky launch -c train env-task.yaml \\\n --secret HF_TOKEN --secret WANDB_API_KEY \\\n --env BATCH_SIZE=64\n\n# Secrets can also be specified inline\nsky launch -c train env-task.yaml \\\n --secret HF_TOKEN=hf_abc123 \\\n --secret WANDB_API_KEY=wk_xyz789\n```\n\n### Inline Command (No YAML File)\n\nRun a quick command without writing a YAML file.\n\n```bash\n# Launch a GPU instance and run nvidia-smi\nsky launch -c quick --gpus H100 -- nvidia-smi\n\n# Run a Python script on an existing cluster\nsky exec quick -- python -c \"import torch; print(torch.cuda.device_count())\"\n\n# Launch with specific cloud and region\nsky launch -c quick --gpus A100 --infra gcp/us-central1 -- nvidia-smi\n```\n\n### Multi-Node Task\n\nLaunch a task across multiple nodes.\n\n```yaml\n# multi-node.yaml\nname: multi-node-check\n\n# Launch 2 nodes (1 head + 1 worker)\nnum_nodes: 2\n\nresources:\n accelerators: L4:1\n\nrun: |\n echo \"Node rank: $SKYPILOT_NODE_RANK\"\n echo \"Total nodes: $SKYPILOT_NUM_NODES\"\n echo \"All node IPs:\"\n echo \"$SKYPILOT_NODE_IPS\"\n echo \"GPUs per node: $SKYPILOT_NUM_GPUS_PER_NODE\"\n nvidia-smi\n```\n\n```bash\nsky launch -c multi multi-node.yaml\n```\n\n---\n\n## 2. Training Examples\n\n### Single-GPU PyTorch Training\n\nA complete single-GPU training example with local code sync.\n\n```yaml\n# single-gpu-train.yaml\nname: pytorch-single-gpu\n\nworkdir: .\n\nresources:\n accelerators: A100:1\n disk_size: 256\n\nenvs:\n NUM_EPOCHS: \"10\"\n BATCH_SIZE: \"32\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n python train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c train single-gpu-train.yaml\nsky logs train # Stream logs\nsky down train # Tear down when done\n```\n\n### Multi-GPU Single-Node Training\n\nUse all GPUs on a single node with torchrun.\n\n```yaml\n# multi-gpu-train.yaml\nname: pytorch-multi-gpu\n\nworkdir: .\n\nresources:\n # 4x A100 GPUs on a single node\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n NUM_EPOCHS: \"10\"\n BATCH_SIZE: \"64\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_port=12345 \\\n train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c multi-gpu multi-gpu-train.yaml\n```\n\n### Multi-Node Distributed Training (PyTorch DDP)\n\nDistributed training across 2 nodes with 8 GPUs each using torchrun.\n\n```yaml\n# multi-node-ddp.yaml\nname: pytorch-ddp-training\n\nworkdir: .\n\n# 2 nodes, each with 8 GPUs = 16 GPUs total\nnum_nodes: 2\n\nresources:\n accelerators: A100-80GB:8\n disk_size: 512\n\nenvs:\n NUM_EPOCHS: \"20\"\n BATCH_SIZE: \"256\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n # SKYPILOT_NODE_IPS contains all node IPs, one per line\n # The first IP is always the head node (rank 0)\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=12345 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c ddp-train multi-node-ddp.yaml\n\n# SSH into the head node to check progress\nssh ddp-train\n```\n\n### Training with Checkpointing to Cloud Storage\n\nSave checkpoints to S3/GCS for persistence across runs.\n\n```yaml\n# train-with-checkpoints.yaml\nname: training-with-checkpoints\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n CHECKPOINT_BUCKET: my-training-checkpoints\n\nfile_mounts:\n # Mount an S3 bucket for writing checkpoints\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\n # Mount a dataset bucket (read-only streaming)\n /data:\n source: s3://my-training-data/imagenet\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision boto3\n\nrun: |\n # Training script should save checkpoints to /checkpoints\n # and load dataset from /data\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n train.py \\\n --data-dir /data \\\n --checkpoint-dir /checkpoints \\\n --resume-from-checkpoint\n```\n\n```bash\n# Launch training\nsky launch -c train train-with-checkpoints.yaml\n\n# Re-launch after preemption or failure -- resumes from last checkpoint\nsky launch -c train train-with-checkpoints.yaml\n```\n\n### Training on Spot Instances with Auto-Recovery (Managed Job)\n\nRun long training on cheap spot instances with automatic recovery from preemptions.\n\n```yaml\n# spot-train.yaml\nname: spot-training-resnet\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n # Use spot instances for ~3x cost savings\n use_spot: true\n disk_size: 512\n\n # Recovery strategy when spot instance is preempted\n job_recovery:\n strategy: EAGER_NEXT_REGION # Move to next region on preemption\n max_restarts_on_errors: 3 # Retry up to 3 times on user code errors\n\nenvs:\n CHECKPOINT_BUCKET: my-spot-training-ckpts\n\nfile_mounts:\n # Persistent checkpoint storage -- survives preemptions\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision\n\nrun: |\n # IMPORTANT: Your training script must handle checkpoint resume.\n # SkyPilot recovers the cluster; your script recovers the state.\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n train.py \\\n --checkpoint-dir /checkpoints \\\n --resume-from-checkpoint\n```\n\n```bash\n# Launch as a managed job (NOT sky launch -- use sky jobs launch)\nsky jobs launch spot-train.yaml\n\n# Monitor the job\nsky jobs queue\n\n# Stream logs\nsky jobs logs \n```\n\n### Fine-Tuning a HuggingFace Model\n\nFine-tune a HuggingFace model with LoRA, using secrets for authentication.\n\n```yaml\n# finetune-hf.yaml\nname: llama3-finetune\n\nworkdir: .\n\nresources:\n accelerators: A100-80GB:4\n disk_size: 512\n disk_tier: best\n\nenvs:\n MODEL_ID: meta-llama/Meta-Llama-3.1-8B-Instruct\n DATASET: yahma/alpaca-cleaned\n CHECKPOINT_BUCKET: my-finetune-checkpoints\n\n# HuggingFace token is a secret -- redacted in logs/dashboard\nsecrets:\n HF_TOKEN: null # Pass via CLI: --secret HF_TOKEN=hf_xxx\n\nfile_mounts:\n /output:\n name: ${CHECKPOINT_BUCKET}\n mode: MOUNT\n\nsetup: |\n pip install torch==2.4.0 torchvision\n pip install transformers peft trl datasets accelerate\n pip install bitsandbytes\n\n # Download the model (needs HF_TOKEN for gated models)\n huggingface-cli download $MODEL_ID \\\n --token $HF_TOKEN \\\n --local-dir /tmp/model\n\nrun: |\n python finetune_lora.py \\\n --model-path /tmp/model \\\n --dataset $DATASET \\\n --output-dir /output/lora-adapter \\\n --num-epochs 3 \\\n --batch-size 4 \\\n --learning-rate 2e-4\n```\n\n```bash\n# Launch with HF_TOKEN secret\nHF_TOKEN=hf_abc123 sky launch -c finetune finetune-hf.yaml --secret HF_TOKEN\n```\n\n---\n\n## 3. Model Serving Examples\n\n### Basic vLLM Serving (Single GPU)\n\nServe a model with vLLM behind a SkyServe endpoint.\n\n```yaml\n# vllm-serve.yaml\nname: vllm-llama\n\nresources:\n accelerators: A100:1\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\n# SkyServe configuration\nservice:\n # Readiness check endpoint (must return 200 when ready)\n readiness_probe: /v1/models\n # Fixed number of replicas\n replicas: 1\n```\n\n```bash\n# Deploy as a service\nHF_TOKEN=hf_xxx sky serve up vllm-serve.yaml -n llama-svc --secret HF_TOKEN\n\n# Get the endpoint URL\nsky serve status llama-svc --endpoint\n\n# Test the endpoint\nENDPOINT=$(sky serve status llama-svc --endpoint)\ncurl $ENDPOINT/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n }'\n\n# Tear down\nsky serve down llama-svc\n```\n\n### vLLM with Autoscaling\n\nScale replicas automatically based on query load.\n\n```yaml\n# vllm-autoscale.yaml\nname: vllm-autoscaling\n\nresources:\n accelerators: A100:1\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe:\n path: /v1/models\n # Wait 20 minutes for model loading before checking readiness\n initial_delay_seconds: 1200\n replica_policy:\n min_replicas: 1\n max_replicas: 5\n # Scale up when each replica handles more than 5 QPS\n target_qps_per_replica: 5\n # Wait 5 minutes before adding replicas\n upscale_delay_seconds: 300\n # Wait 20 minutes before removing replicas\n downscale_delay_seconds: 1200\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up vllm-autoscale.yaml -n llama-auto --secret HF_TOKEN\n```\n\n### Serving on Spot Instances\n\nUse spot instances for serving replicas to reduce cost.\n\n```yaml\n# vllm-spot-serve.yaml\nname: vllm-spot-serving\n\nresources:\n accelerators: A100:1\n ports: 8000\n # Use spot instances for replicas -- SkyServe handles recovery\n use_spot: true\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe: /v1/models\n replica_policy:\n # Keep at least 2 replicas for high availability with spot\n min_replicas: 2\n max_replicas: 6\n target_qps_per_replica: 5\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up vllm-spot-serve.yaml -n llama-spot --secret HF_TOKEN\n```\n\n### Multi-Model Serving\n\nServe multiple models by deploying separate services.\n\n```yaml\n# serve-llama-70b.yaml\nname: llama-70b-service\n\nresources:\n # 70B model needs multi-GPU with tensor parallelism\n accelerators: A100-80GB:4\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-70B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --tensor-parallel-size 4 \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe: /v1/models\n replicas: 2\n```\n\n```bash\n# Deploy multiple model services\nHF_TOKEN=hf_xxx sky serve up vllm-serve.yaml -n llama-8b --secret HF_TOKEN\nHF_TOKEN=hf_xxx sky serve up serve-llama-70b.yaml -n llama-70b --secret HF_TOKEN\n\n# Check all services\nsky serve status\n\n# Get endpoints\nsky serve status llama-8b --endpoint\nsky serve status llama-70b --endpoint\n```\n\n### TGI (Text Generation Inference) Example\n\nServe a model using HuggingFace TGI with Docker.\n\n```yaml\n# tgi-serve.yaml\nname: tgi-service\n\nresources:\n accelerators: A100:1\n ports: 8080\n\nenvs:\n MODEL_ID: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nrun: |\n docker run --gpus all --shm-size 1g \\\n -p 8080:80 \\\n -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \\\n -v ~/data:/data \\\n ghcr.io/huggingface/text-generation-inference \\\n --model-id $MODEL_ID\n\nservice:\n readiness_probe: /health\n replicas: 2\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up tgi-serve.yaml -n tgi-llama --secret HF_TOKEN\nsky serve status tgi-llama --endpoint\n```\n\n### Service with Custom Readiness Probe (POST)\n\nUse a POST request with custom data for the readiness check.\n\n```yaml\n# custom-probe-serve.yaml\nname: custom-model-service\n\nresources:\n accelerators: A100:1\n ports: 8080\n\nsetup: |\n pip install my-model-server\n\nrun: |\n python -m my_model_server --port 8080\n\nservice:\n readiness_probe:\n path: /v1/predict\n # Use POST with custom payload for readiness checking\n post_data: '{\"text\": \"health check\", \"max_tokens\": 1}'\n # Wait 30 minutes for model loading\n initial_delay_seconds: 1800\n # Timeout for each probe request\n timeout_seconds: 30\n replica_policy:\n min_replicas: 1\n max_replicas: 3\n target_qps_per_replica: 10\n```\n\n```bash\nsky serve up custom-probe-serve.yaml -n custom-svc\n```\n\n---\n\n## 4. File Mounts and Storage Examples\n\n### Sync Local Directory (workdir)\n\nThe workdir syncs your local directory to `~/sky_workdir` on the remote cluster.\n\n```yaml\n# workdir-example.yaml\nname: local-code-sync\n\n# Sync current directory to ~/sky_workdir\nworkdir: .\n\nresources:\n accelerators: L4:1\n\nsetup: |\n pip install -r requirements.txt\n\nrun: |\n # Commands run under ~/sky_workdir\n python train.py\n```\n\n```bash\n# Any changes to local files are synced on each sky launch/exec\nsky launch -c dev workdir-example.yaml\n\n# Re-sync and re-run (setup is cached)\nsky exec dev workdir-example.yaml\n```\n\n### Mount S3 Bucket (mode: MOUNT)\n\nStream data from S3 without downloading -- read-only, low disk usage.\n\n```yaml\n# s3-mount.yaml\nname: s3-streaming\n\nresources:\n accelerators: A100:1\n\nfile_mounts:\n # MOUNT mode: read-only, streams data on access, no local copy\n /data/imagenet:\n source: s3://my-imagenet-bucket/train\n mode: MOUNT\n\nrun: |\n python train.py --data-dir /data/imagenet\n```\n\n```bash\nsky launch -c train s3-mount.yaml\n```\n\n### Mount with Cache (mode: MOUNT_CACHED)\n\nCache accessed data locally for fast random access -- ideal for model weights.\n\n```yaml\n# mount-cached.yaml\nname: cached-model-weights\n\nresources:\n accelerators: A100:1\n disk_size: 512 # Need enough disk for cached data\n\nfile_mounts:\n # MOUNT_CACHED: caches accessed files locally for fast random access\n # Good for model weights that are read repeatedly\n /models/llama:\n source: s3://my-model-weights/llama-3.1-8b\n mode: MOUNT_CACHED\n\nrun: |\n python serve.py --model-path /models/llama\n```\n\n```bash\nsky launch -c serve mount-cached.yaml\n```\n\n### Copy from Cloud Storage (mode: COPY)\n\nDownload data fully -- read-write access, full local copy.\n\n```yaml\n# copy-mount.yaml\nname: full-copy\n\nresources:\n accelerators: A100:1\n # Need enough disk for the full dataset\n disk_size: 1024\n\nfile_mounts:\n # COPY mode: full download, read-write, supports modification\n /data/dataset:\n source: s3://my-dataset-bucket/training-data\n mode: COPY\n\nrun: |\n # Can read AND write to the copied data\n python preprocess.py --input /data/dataset --output /data/dataset/processed\n python train.py --data-dir /data/dataset/processed\n```\n\n```bash\nsky launch -c train copy-mount.yaml\n```\n\n### Mount GCS Bucket\n\nMount a Google Cloud Storage bucket.\n\n```yaml\n# gcs-mount.yaml\nname: gcs-data\n\nresources:\n infra: gcp\n accelerators: A100:1\n\nfile_mounts:\n # GCS bucket mount\n /data:\n source: gs://my-gcs-bucket/datasets\n mode: MOUNT\n\n # Create a new GCS bucket for output\n /output:\n name: my-training-output-bucket\n store: gcs\n mode: MOUNT\n\nrun: |\n python train.py --data /data --output /output\n```\n\n```bash\nsky launch -c gcp-train gcs-mount.yaml\n```\n\n### Using Volumes on Kubernetes\n\nKubernetes persistent and ephemeral volumes.\n\n```yaml\n# k8s-volumes.yaml\nname: k8s-with-volumes\n\nresources:\n infra: k8s\n accelerators: A100:1\n\n# Volumes (Kubernetes only)\nvolumes:\n # Persistent volume -- data survives cluster restarts\n /mnt/data: my-persistent-volume\n\n # Ephemeral volume -- fast scratch space, lost on restart\n /mnt/scratch:\n size: 200Gi\n\nrun: |\n # /mnt/data persists across restarts\n # /mnt/scratch is fast ephemeral storage\n python train.py \\\n --data-dir /mnt/data \\\n --cache-dir /mnt/scratch\n```\n\n```bash\nsky launch -c k8s-train k8s-volumes.yaml\n```\n\n### Multiple Mounts in One Task\n\nCombine different mount types in a single task.\n\n```yaml\n# multi-mount.yaml\nname: multi-mount-training\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n OUTPUT_BUCKET: my-training-outputs\n\nfile_mounts:\n # Sync local config files\n /configs: ./configs\n\n # Stream training data from S3 (read-only)\n /data/train:\n source: s3://my-dataset/train\n mode: MOUNT\n\n # Cache model weights locally for fast loading\n /models/pretrained:\n source: s3://my-models/llama-3.1-8b\n mode: MOUNT_CACHED\n\n # Writable output bucket for checkpoints\n /output:\n name: ${OUTPUT_BUCKET}\n store: s3\n mode: MOUNT\n\n # Copy evaluation data (small, needs read-write)\n /data/eval:\n source: s3://my-dataset/eval\n mode: COPY\n\nsetup: |\n pip install torch transformers datasets\n\nrun: |\n python train.py \\\n --config /configs/train_config.yaml \\\n --data-dir /data/train \\\n --eval-dir /data/eval \\\n --pretrained-model /models/pretrained \\\n --output-dir /output\n```\n\n```bash\nsky launch -c train multi-mount.yaml\n```\n\n---\n\n## 5. Multi-Cloud and Fallback Examples\n\n### any_of: Multiple GPU Types\n\nLet SkyPilot pick the cheapest available GPU from a set.\n\n```yaml\n# any-gpu.yaml\nname: flexible-gpu-training\n\nresources:\n # SkyPilot picks the cheapest available option\n any_of:\n - accelerators: H100:1\n - accelerators: A100-80GB:1\n - accelerators: A100:1\n - accelerators: L40S:1\n\nrun: |\n nvidia-smi\n python train.py\n```\n\n```bash\nsky launch -c train any-gpu.yaml\n```\n\n### any_of: Multiple Clouds for Same GPU\n\nTry different clouds for the same GPU type.\n\n```yaml\n# any-cloud.yaml\nname: multi-cloud-a100\n\nresources:\n accelerators: A100-80GB:8\n any_of:\n - infra: aws/us-east-1\n - infra: gcp/us-central1\n - infra: azure/eastus\n\nrun: |\n nvidia-smi\n python train.py\n```\n\n```bash\nsky launch -c train any-cloud.yaml\n```\n\n### ordered: Strict Priority\n\nTry clouds/regions in a specific order -- first match wins.\n\n```yaml\n# ordered-priority.yaml\nname: priority-training\n\nresources:\n ordered:\n # Try AWS us-east-1 first (our reserved instances)\n - infra: aws/us-east-1\n accelerators: H100:8\n # Then try GCP us-central1\n - infra: gcp/us-central1\n accelerators: H100:8\n # Fall back to A100s on any cloud\n - accelerators: A100-80GB:8\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train ordered-priority.yaml\n```\n\n### K8s + Cloud Fallback\n\nTry Kubernetes first, fall back to cloud if K8s has no capacity.\n\n```yaml\n# k8s-fallback.yaml\nname: k8s-or-cloud\n\nresources:\n accelerators: A100:1\n ordered:\n # Try on-prem K8s cluster first (free)\n - infra: k8s/my-onprem-cluster\n # Fall back to AWS\n - infra: aws\n # Fall back to GCP\n - infra: gcp\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train k8s-fallback.yaml\n```\n\n### Region Preference with Wildcards\n\nUse wildcards to match multiple regions or zones.\n\n```yaml\n# wildcard-region.yaml\nname: us-region-only\n\nresources:\n accelerators: A100:8\n any_of:\n # Any zone in us-east-1\n - infra: aws/us-east-1\n # Any zone in us-west-2\n - infra: aws/us-west-2\n # Any US region on GCP\n - infra: gcp/us-*\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train wildcard-region.yaml\n```\n\n### Cost Optimization: Cheapest Across All Clouds\n\nOmit `infra` to let SkyPilot search all enabled clouds for the cheapest option.\n\n```yaml\n# cheapest.yaml\nname: cheapest-option\n\nresources:\n # No infra specified -- SkyPilot searches all enabled clouds\n accelerators: A100:1\n\nrun: |\n python train.py\n```\n\n```bash\n# Dry run to see the cheapest option without launching\nsky launch cheapest.yaml --dryrun\n\n# Launch on the cheapest cloud/region\nsky launch -c train cheapest.yaml\n\n# Check GPU prices across clouds\nsky gpus list A100\n```\n\n---\n\n## 6. Managed Job Examples\n\n### Basic Managed Job\n\nManaged jobs run on a dedicated controller that monitors and manages your job.\n\n```yaml\n# basic-job.yaml\nname: data-processing\n\nresources:\n cpus: 16+\n memory: 64+\n\nsetup: |\n pip install pandas pyarrow\n\nrun: |\n python process_data.py --output /tmp/results\n```\n\n```bash\n# Launch as a managed job (runs on a separate controller)\nsky jobs launch basic-job.yaml\n\n# Check status\nsky jobs queue\n\n# Stream logs\nsky jobs logs 1\n\n# Cancel the job\nsky jobs cancel 1\n```\n\n### Spot Job with FAILOVER Recovery\n\nAutomatically retry in the same region first, then try other regions.\n\n```yaml\n# spot-failover.yaml\nname: training-failover\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n # FAILOVER: retry same region first, then move to next region\n strategy: FAILOVER\n # Retry up to 3 times on user code errors\n max_restarts_on_errors: 3\n\nfile_mounts:\n /checkpoints:\n name: my-training-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch spot-failover.yaml\n```\n\n### Spot Job with EAGER_NEXT_REGION Recovery\n\nMove to a different region immediately on preemption -- good for spot shortages.\n\n```yaml\n# spot-eager.yaml\nname: training-eager-recovery\n\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n # EAGER_NEXT_REGION: immediately try a different region on preemption\n # Best for spot instances where preemption signals regional shortage\n strategy: EAGER_NEXT_REGION\n\nfile_mounts:\n /checkpoints:\n name: my-training-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch spot-eager.yaml\n```\n\n### Hyperparameter Sweep\n\nLaunch multiple managed jobs with different hyperparameters.\n\n```yaml\n# sweep.yaml\nname: hp-sweep\n\nresources:\n accelerators: A100:1\n\nenvs:\n LR: \"1e-4\"\n WEIGHT_DECAY: \"0.01\"\n BATCH_SIZE: \"32\"\n\nfile_mounts:\n /results:\n name: sweep-results\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers wandb\n\nrun: |\n python train.py \\\n --lr $LR \\\n --weight-decay $WEIGHT_DECAY \\\n --batch-size $BATCH_SIZE \\\n --output-dir /results/lr-${LR}_wd-${WEIGHT_DECAY}_bs-${BATCH_SIZE}\n```\n\n```bash\n# Launch multiple jobs with different hyperparameters\nfor lr in 1e-3 1e-4 1e-5; do\n for wd in 0.01 0.1; do\n sky jobs launch sweep.yaml \\\n --env LR=$lr --env WEIGHT_DECAY=$wd \\\n --name sweep-lr${lr}-wd${wd}\n done\ndone\n\n# Monitor all sweep jobs\nsky jobs queue\n```\n\n### Job with max_restarts_on_errors\n\nAutomatically restart on user code errors (non-zero exit codes).\n\n```yaml\n# auto-restart.yaml\nname: resilient-training\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n strategy: FAILOVER\n # Restart up to 5 times if the training script crashes\n max_restarts_on_errors: 5\n\nfile_mounts:\n /checkpoints:\n name: my-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch auto-restart.yaml\n```\n\n### Job with recover_on_exit_codes for NCCL Errors\n\nAlways retry on specific exit codes (such as NCCL timeouts).\n\n```yaml\n# nccl-recovery.yaml\nname: nccl-resilient-training\n\nnum_nodes: 4\n\nresources:\n accelerators: H100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n # Exit code 33 = NCCL timeout (convention in many training scripts)\n # These restarts do NOT count toward max_restarts_on_errors\n recover_on_exit_codes: [33, 34]\n\nfile_mounts:\n /checkpoints:\n name: my-distributed-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=12345 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch nccl-recovery.yaml\n```\n\n### Job Pool Example\n\nUse a job pool for batch processing with pre-provisioned workers.\n\n```yaml\n# pool.yaml\n# Pool configuration: pre-provisions workers with setup\nworkdir: .\n\nresources:\n accelerators: H100:1\n disk_size: 100\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B\n\nsetup: |\n pip install vllm transformers datasets\n # Pre-download model so all jobs can use it immediately\n huggingface-cli download $MODEL_NAME --local-dir /tmp/model\n\npool:\n workers: 3\n```\n\n```yaml\n# job.yaml\n# Job that runs on the pool\nname: batch-inference\n\nresources:\n accelerators: H100:1\n\nrun: |\n echo \"Processing partition ${SKYPILOT_JOB_RANK} of ${SKYPILOT_NUM_JOBS}\"\n python classify.py \\\n --job-rank ${SKYPILOT_JOB_RANK} \\\n --num-jobs ${SKYPILOT_NUM_JOBS} \\\n --model-path /tmp/model \\\n --output-dir /results\n```\n\n```bash\n# Create the pool with 3 workers\nsky jobs pool apply -p my-pool pool.yaml\n\n# Submit 10 jobs to the pool (distributed across 3 workers)\nsky jobs launch -p my-pool --num-jobs 10 job.yaml\n\n# Check pool status\nsky jobs pool status my-pool\n\n# Check job status\nsky jobs queue\n\n# Tear down pool when done\nsky jobs pool down my-pool\n```\n\n---\n\n## 7. Distributed Training Examples\n\n### PyTorch DDP on 4 Nodes (Complete Example)\n\nFull distributed training with torchrun and NCCL configuration.\n\n```yaml\n# ddp-4node.yaml\nname: ddp-4node-training\n\nworkdir: .\n\nnum_nodes: 4\n\nresources:\n accelerators: H100:8\n disk_size: 1024\n\nenvs:\n # NCCL configuration for optimal multi-node communication\n NCCL_DEBUG: INFO\n NCCL_SOCKET_IFNAME: eth0\n # Training hyperparameters\n GLOBAL_BATCH_SIZE: \"512\"\n NUM_EPOCHS: \"50\"\n CHECKPOINT_BUCKET: my-ddp-checkpoints\n\nfile_mounts:\n # Training data (streaming)\n /data:\n source: s3://my-training-data/preprocessed\n mode: MOUNT\n\n # Checkpoint storage (persistent, writable)\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision\n pip install wandb tensorboard\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_GPUS=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))\n\n echo \"Node rank: $SKYPILOT_NODE_RANK / $SKYPILOT_NUM_NODES\"\n echo \"Master: $MASTER_ADDR\"\n echo \"Total GPUs: $NUM_GPUS\"\n\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=29500 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train_ddp.py \\\n --data-dir /data \\\n --checkpoint-dir /checkpoints \\\n --global-batch-size $GLOBAL_BATCH_SIZE \\\n --epochs $NUM_EPOCHS \\\n --resume\n```\n\n```bash\n# Launch distributed training\nsky launch -c ddp4 ddp-4node.yaml\n\n# SSH into head node\nssh ddp4\n\n# Check worker node\nssh ddp4-worker1\n\n# Monitor logs\nsky logs ddp4\n```\n\n### DeepSpeed ZeRO-3 on 2 Nodes\n\nDistributed training with DeepSpeed, including hostfile generation.\n\n```yaml\n# deepspeed-2node.yaml\nname: deepspeed-zero3\n\nnum_nodes: 2\n\nresources:\n accelerators: A100-80GB:8\n disk_size: 1024\n image_id: docker:nvidia/cuda:12.1.1-devel-ubuntu20.04\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B\n DEEPSPEED_ENVS: \"MODEL_NAME,SKYPILOT_NODE_RANK\"\n\nsetup: |\n pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121\n pip install deepspeed==0.14.4\n pip install transformers datasets accelerate\n sudo apt-get update && sudo apt-get -y install pdsh\n\nrun: |\n # Only the head node (rank 0) launches DeepSpeed\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n # Generate hostfile for DeepSpeed\n HOSTFILE=/tmp/hostfile\n python -c \"\n import os\n n_gpus = os.environ['SKYPILOT_NUM_GPUS_PER_NODE']\n ips = os.environ['SKYPILOT_NODE_IPS'].splitlines()\n with open('$HOSTFILE', 'w') as f:\n for ip in ips:\n f.write(f'{ip} slots={n_gpus}\\n')\n \"\n echo \"Hostfile:\"\n cat $HOSTFILE\n\n # Generate .deepspeed_env for environment propagation\n python -c \"\n import os\n envs = os.environ.get('DEEPSPEED_ENVS', '').split(',')\n with open('.deepspeed_env', 'w') as f:\n for var in envs:\n f.write(f'{var}=\\\"{os.getenv(var, \\\"\\\")}\\\"\\n')\n \"\n\n deepspeed \\\n --hostfile $HOSTFILE \\\n train_deepspeed.py \\\n --model-name $MODEL_NAME \\\n --deepspeed \\\n --deepspeed-config ds_config_zero3.json\n fi\n```\n\n```bash\nsky launch -c ds2 deepspeed-2node.yaml\n```\n\n### FSDP Training Example\n\nMulti-node Fully Sharded Data Parallel training with HuggingFace Accelerate.\n\n```yaml\n# fsdp-train.yaml\nname: fsdp-training\n\nworkdir: .\n\nnum_nodes: 2\n\nresources:\n accelerators: H100:8\n disk_size: 1024\n\nenvs:\n MODEL_ID: meta-llama/Llama-3.1-8B\n CHECKPOINT_BUCKET: my-fsdp-checkpoints\n\nsecrets:\n HF_TOKEN: null\n\nfile_mounts:\n /output:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch --index-url https://download.pytorch.org/whl/cu128\n pip install transformers accelerate trl peft datasets\n pip install deepspeed # Needed for some FSDP optimizations\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_PROCESSES=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))\n\n accelerate launch \\\n --config_file configs/fsdp_config.yaml \\\n --num_machines $SKYPILOT_NUM_NODES \\\n --num_processes $NUM_PROCESSES \\\n --machine_rank $SKYPILOT_NODE_RANK \\\n --main_process_ip $MASTER_ADDR \\\n --main_process_port 29500 \\\n train_fsdp.py \\\n --model-id $MODEL_ID \\\n --output-dir /output \\\n --resume-from-checkpoint\n```\n\n```bash\nHF_TOKEN=hf_xxx sky launch -c fsdp fsdp-train.yaml --secret HF_TOKEN\n```\n\n### Ray Train Multi-Node Example\n\nDistributed training using Ray Train across multiple nodes.\n\n```yaml\n# ray-train.yaml\nname: ray-train-distributed\n\nworkdir: .\n\nnum_nodes: 2\n\nresources:\n accelerators: A100:4\n\nsetup: |\n pip install \"ray[train]\" torch torchvision\n pip install transformers datasets\n\nrun: |\n # SkyPilot sets up a Ray cluster across all nodes automatically.\n # The head node's Ray address is available at the standard port.\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n # Only run training script on the head node;\n # Ray distributes the work to all nodes\n python train_ray.py \\\n --num-workers $((SKYPILOT_NUM_NODES * SKYPILOT_NUM_GPUS_PER_NODE)) \\\n --gpus-per-worker 1\n fi\n```\n\nThe `train_ray.py` script uses `ray.train`:\n\n```python\n# train_ray.py (referenced by the YAML above)\nimport ray\nfrom ray import train\nfrom ray.train.torch import TorchTrainer\nfrom ray.train import ScalingConfig\n\ndef train_func(config):\n import torch\n # Your PyTorch training code here\n # ray.train handles distributed setup automatically\n model = ...\n optimizer = ...\n for epoch in range(config[\"epochs\"]):\n train_epoch(model, optimizer)\n # Report metrics\n train.report({\"loss\": loss, \"epoch\": epoch})\n\nscaling_config = ScalingConfig(\n num_workers=args.num_workers,\n use_gpu=True,\n resources_per_worker={\"GPU\": args.gpus_per_worker},\n)\n\ntrainer = TorchTrainer(\n train_func,\n train_loop_config={\"epochs\": 10},\n scaling_config=scaling_config,\n)\nresult = trainer.fit()\n```\n\n```bash\nsky launch -c ray-train ray-train.yaml\n```\n\n---\n\n## 8. Python SDK Examples\n\n### Basic: Create Task, Set Resources, Launch\n\n```python\nimport sky\n\n# Create a task programmatically\ntask = sky.Task(\n name='hello-gpu',\n run='nvidia-smi && python -c \"import torch; print(torch.cuda.device_count())\"',\n)\ntask.set_resources(sky.Resources(\n accelerators='A100:1',\n))\n\n# Launch returns a request ID (async by default)\nrequest_id = sky.launch(task, cluster_name='my-cluster')\n\n# Wait for the launch to complete\njob_id, handle = sky.get(request_id)\nprint(f'Job ID: {job_id}')\n```\n\n### Launch from YAML File\n\n```python\nimport sky\n\n# Load task from a YAML file\ntask = sky.Task.from_yaml('train.yaml')\n\n# Launch with options\nrequest_id = sky.launch(\n task,\n cluster_name='train-cluster',\n idle_minutes_to_autostop=30, # Auto-stop after 30 min idle\n retry_until_up=True, # Keep trying if resources unavailable\n)\nresult = sky.get(request_id)\n```\n\n### Check Status and Get Cluster Info\n\n```python\nimport sky\n\n# Get all cluster statuses\nrequest_id = sky.status()\nclusters = sky.get(request_id)\n\nfor cluster in clusters:\n print(f\"Cluster: {cluster['name']}\")\n print(f\" Status: {cluster['status']}\")\n print(f\" Resources: {cluster['resources_str']}\")\n print(f\" Autostop: {cluster['autostop']} min\")\n\n# Get status for a specific cluster\nrequest_id = sky.status(cluster_names=['my-cluster'])\nclusters = sky.get(request_id)\n\n# Get job queue for a cluster\nrequest_id = sky.queue('my-cluster')\njobs = sky.get(request_id)\nfor job in jobs:\n print(f\"Job {job['job_id']}: {job['status']}\")\n```\n\n### Batch Submission (Launch Multiple Tasks)\n\n```python\nimport sky\n\n# Launch multiple training runs with different configs\nconfigs = [\n {'lr': 1e-3, 'batch_size': 32},\n {'lr': 1e-4, 'batch_size': 64},\n {'lr': 1e-5, 'batch_size': 128},\n]\n\nrequest_ids = []\nfor i, cfg in enumerate(configs):\n task = sky.Task(\n name=f'sweep-{i}',\n setup='pip install torch transformers',\n run=f'python train.py --lr {cfg[\"lr\"]} --batch-size {cfg[\"batch_size\"]}',\n )\n task.set_resources(sky.Resources(accelerators='A100:1'))\n\n request_id = sky.launch(\n task,\n cluster_name=f'sweep-{i}',\n idle_minutes_to_autostop=10,\n down=True, # Auto-teardown after idle\n )\n request_ids.append(request_id)\n print(f'Launched sweep-{i}: {request_id}')\n\n# Wait for all to complete\nfor rid in request_ids:\n result = sky.get(rid)\n print(f'Completed: {rid}')\n```\n\n### Wait for Completion and Get Results\n\n```python\nimport sky\n\n# Launch a task\ntask = sky.Task.from_yaml('train.yaml')\nrequest_id = sky.launch(task, cluster_name='train')\n\n# stream_and_get streams logs AND waits for completion\nresult = sky.stream_and_get(request_id)\n\n# Check job status\nrequest_id = sky.job_status('train', job_ids=[1])\nstatuses = sky.get(request_id)\nprint(f'Job 1 status: {statuses[1]}')\n\n# Tail logs (blocking, streams to console)\nexit_code = sky.tail_logs('train', job_id=1, follow=True)\nprint(f'Job exit code: {exit_code}')\n```\n\n### Programmatic Managed Job Launch\n\n```python\nimport sky\n\n# Create a managed job task\ntask = sky.Task(\n name='managed-training',\n setup='pip install torch transformers',\n run='python train.py --resume',\n)\ntask.set_resources(sky.Resources(\n accelerators='A100:4',\n use_spot=True,\n job_recovery={\n 'strategy': 'EAGER_NEXT_REGION',\n 'max_restarts_on_errors': 3,\n },\n))\n\n# Launch as a managed job\nrequest_id = sky.jobs.launch(task)\njob_id = sky.get(request_id)\nprint(f'Managed job ID: {job_id}')\n\n# Check managed job queue\nrequest_id = sky.jobs.queue()\njobs = sky.get(request_id)\nfor job in jobs:\n print(f\"Job {job['job_id']}: {job['job_name']} - {job['status']}\")\n```\n\n### DAG Workflow (Task Dependencies)\n\n```python\nimport sky\n\n# Create a DAG with task dependencies\nwith sky.Dag() as dag:\n # Step 1: Data preprocessing\n preprocess = sky.Task(\n name='preprocess',\n setup='pip install pandas pyarrow',\n run='python preprocess.py --output /tmp/data',\n )\n preprocess.set_resources(sky.Resources(cpus='8+', memory='32+'))\n\n # Step 2: Training (depends on preprocessing)\n train = sky.Task(\n name='train',\n setup='pip install torch transformers',\n run='python train.py --data /tmp/data',\n )\n train.set_resources(sky.Resources(accelerators='A100:4'))\n\n # Define dependency: preprocess must finish before train\n preprocess >> train\n\n# Launch the DAG (experimental)\nrequest_id = sky.launch(dag, cluster_name='pipeline')\nsky.get(request_id)\n```\n\n### Autostop Management\n\n```python\nimport sky\n\n# Set autostop: stop after 30 minutes of idleness\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=30,\n)\nsky.get(request_id)\n\n# Set autodown: tear down (not just stop) after 60 minutes idle\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=60,\n down=True,\n)\nsky.get(request_id)\n\n# Cancel autostop\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=-1, # Negative value cancels autostop\n)\nsky.get(request_id)\n\n# Stop and start a cluster\nsky.get(sky.stop('my-cluster'))\nsky.get(sky.start('my-cluster'))\n\n# Tear down a cluster\nsky.get(sky.down('my-cluster'))\n```\n\n### List GPUs Programmatically\n\n```python\nimport sky\n\n# List all available accelerators\nrequest_id = sky.list_accelerators(\n gpus_only=True,\n name_filter='A100',\n)\naccelerators = sky.get(request_id)\n\nfor acc_name, offerings in accelerators.items():\n print(f'\\n{acc_name}:')\n for offering in offerings:\n print(f' {offering}')\n\n# List accelerator counts (how many GPUs per node)\nrequest_id = sky.list_accelerator_counts(\n gpus_only=True,\n name_filter='H100',\n)\ncounts = sky.get(request_id)\nfor acc_name, available_counts in counts.items():\n print(f'{acc_name}: available in counts {available_counts}')\n\n# Check which clouds are enabled\nrequest_id = sky.enabled_clouds()\nclouds = sky.get(request_id)\nprint(f'Enabled clouds: {clouds}')\n```\n\n---\n\n## Quick Reference: SkyPilot Environment Variables\n\nThese variables are automatically set by SkyPilot on every node and are available\nin `setup` and `run` commands:\n\n| Variable | Description | Example Value |\n|----------|-------------|---------------|\n| `SKYPILOT_NODE_RANK` | Rank of the current node (0 = head) | `0` |\n| `SKYPILOT_NODE_IPS` | Newline-separated IPs of all nodes | `10.0.0.1\\n10.0.0.2` |\n| `SKYPILOT_NUM_NODES` | Total number of nodes in the cluster | `2` |\n| `SKYPILOT_NUM_GPUS_PER_NODE` | Number of GPUs on this node | `8` |\n| `SKYPILOT_TASK_ID` | Unique task identifier | `sky-2024-01-15-abc123` |\n| `SKYPILOT_CLUSTER_INFO` | JSON with cluster metadata | `{\"cloud\": \"aws\", ...}` |\n| `SKYPILOT_JOB_RANK` | Job rank within a pool (pool jobs only) | `3` |\n| `SKYPILOT_NUM_JOBS` | Total number of jobs in batch (pool jobs only) | `10` |\n\n## Quick Reference: Common CLI Patterns\n\n```bash\n# Launch and auto-stop after 30 minutes idle\nsky launch -c mycluster task.yaml -i 30\n\n# Launch with auto-teardown (not just stop) after idle\nsky launch -c mycluster task.yaml -i 30 --down\n\n# Launch detached (don't stream logs)\nsky launch -c mycluster task.yaml -d\n\n# Retry until resources are available\nsky launch -c mycluster task.yaml -r\n\n# Override GPU from CLI\nsky launch -c mycluster task.yaml --gpus H100:4\n\n# Override cloud from CLI\nsky launch -c mycluster task.yaml --infra aws/us-east-1\n\n# Execute on existing cluster (skip setup)\nsky exec mycluster another-task.yaml\n\n# Execute inline command on existing cluster\nsky exec mycluster -- python eval.py --checkpoint /tmp/best.pt\n\n# Check GPU pricing\nsky gpus list H100\nsky gpus list A100 --all-regions\nsky gpus list --infra aws\n```\n","agent/skills/skypilot/references/python-sdk.md":"\n\n\n# SkyPilot Python SDK Reference\n\nAll SDK functions return a request ID (future). Use `sky.get(request_id)` to await the result.\n\n```python\nimport sky\n\nrequest_id = sky.launch(sky.Task.from_yaml(\"task.yaml\"))\nresult = sky.get(request_id)\n```\n\n## Cluster Operations\n\n### `sky.launch`\n\n```python\nsky.launch(task: Union['sky.Task', 'sky.Dag'], cluster_name: Optional[str] = None, retry_until_up: bool = False, idle_minutes_to_autostop: Optional[int] = None, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, dryrun: bool = False, down: bool = False, backend: Optional['backends.Backend'] = None, optimize_target: common.OptimizeTarget = common.OptimizeTarget.COST, no_setup: bool = False, clone_disk_from: Optional[str] = None, fast: bool = False, resize: bool = False, _need_confirmation: bool = False, _is_launched_by_jobs_controller: bool = False, _is_launched_by_sky_serve_controller: bool = False, _file_mounts_blob_id: Optional[str] = None, _extra_launch_context: Optional[Dict[str, Any]] = None, _include_credentials: bool = False) -> server_common.RequestId[Tuple[Optional[int], Optional['backends.ResourceHandle']]]\n```\n\nLaunches a cluster or task.\n\nThe task's setup and run commands are executed under the task's workdir\n(when specified, it is synced to remote cluster). The task undergoes job\nqueue scheduling on the cluster.\n\nCurrently, the first argument must be a sky.Task, or (EXPERIMENTAL advanced\nusage) a sky.Dag. In the latter case, currently it must contain a single\ntask; support for pipelines/general DAGs are in experimental branches.\n\n**Example:**\n ```python\n\n import sky\n task = sky.Task(run='echo hello SkyPilot')\n task.set_resources(\n sky.Resources(infra='aws', accelerators='V100:4'))\n sky.launch(task, cluster_name='my-cluster')\n\n\n**Args:**\n task: sky.Task, or sky.Dag (experimental; 1-task only) to launch.\n cluster_name: name of the cluster to create/reuse. If None,\n auto-generate a name.\n retry_until_up: whether to retry launching the cluster until it is\n up.\n idle_minutes_to_autostop: automatically stop the cluster after this\n many minute of idleness, i.e., no running or pending jobs in the\n cluster's job queue. Idleness gets reset whenever setting-up/\n running/pending jobs are found in the job queue. Setting this\n flag is equivalent to running\n ``sky.launch(...)`` and then\n ``sky.autostop(idle_minutes=)``. If set, the autostop\n config specified in the task' resources will be overridden by\n this parameter.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes_to_autostop``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after\n ``idle_minutes_to_autostop``.\n dryrun: if True, do not actually launch the cluster.\n down: Tear down the cluster after all jobs finish (successfully or\n abnormally). If --idle-minutes-to-autostop is also set, the\n cluster will be torn down after the specified idle time.\n Note that if errors occur during provisioning/data syncing/setting\n up, the cluster will not be torn down for debugging purposes. If\n set, the autostop config specified in the task' resources will be\n overridden by this parameter.\n backend: backend to use. If None, use the default backend\n (CloudVMRayBackend).\n optimize_target: target to optimize for. Choices: OptimizeTarget.COST,\n OptimizeTarget.TIME.\n no_setup: if True, do not re-run setup commands.\n clone_disk_from: [Experimental] if set, clone the disk from the\n specified cluster. This is useful to migrate the cluster to a\n different availability zone or region.\n fast: [Experimental] If the cluster is already up and available,\n skip provisioning and setup steps.\n resize: if True, resize the existing cluster to the ``num_nodes``\n specified in the task. Supports both scaling up (adding workers)\n and scaling down (removing workers). Scale-down requires no\n running jobs on the cluster. If True, requires ``cluster_name``\n to be set.\n _need_confirmation: (Internal only) If True, show the confirmation\n prompt.\n\n**Returns:**\n The request ID of the launch request.\n\n**Request Returns:**\n job_id (Optional[int]): the job ID of the submitted job. None if the\n backend is not ``CloudVmRayBackend``, or no job is submitted to the\n cluster.\n handle (Optional[backends.ResourceHandle]): the handle to the cluster.\n None if dryrun.\n\n**Request Raises:**\n exceptions.ClusterOwnerIdentityMismatchError: if the cluster is owned\n by another user.\n exceptions.InvalidClusterNameError: if the cluster name is invalid.\n exceptions.ResourcesMismatchError: if the requested resources\n do not match the existing cluster.\n exceptions.NotSupportedError: if required features are not supported\n by the backend/cloud/cluster.\n exceptions.ResourcesUnavailableError: if the requested resources\n cannot be satisfied. The failover_history of the exception will be set\n as:\n\n 1. Empty: iff the first-ever sky.optimize() fails to find a feasible\n resource; no pre-check or actual launch is attempted.\n\n 2. Non-empty: iff at least 1 exception from either our pre-checks\n (e.g., cluster name invalid) or a region/zone throwing resource\n unavailability.\n\n exceptions.CommandError: any ssh command error.\n exceptions.NoCloudAccessError: if all clouds are disabled.\n\nOther exceptions may be raised depending on the backend.\n\n### `sky.exec`\n\n```python\nsky.exec(task: Union['sky.Task', 'sky.Dag'], cluster_name: Optional[str] = None, dryrun: bool = False, down: bool = False, backend: Optional['backends.Backend'] = None) -> server_common.RequestId[Tuple[Optional[int], Optional['backends.ResourceHandle']]]\n```\n\nExecutes a task on an existing cluster.\n\n**This function performs two actions:**\n\n(1) workdir syncing, if the task has a workdir specified;\n(2) executing the task's ``run`` commands.\n\nAll other steps (provisioning, setup commands, file mounts syncing) are\nskipped. If any of those specifications changed in the task, this function\nwill not reflect those changes. To ensure a cluster's setup is up to date,\nuse ``sky.launch()`` instead.\n\n**Execution and scheduling behavior:**\n\n- The task will undergo job queue scheduling, respecting any specified\n resource requirement. It can be executed on any node of the cluster with\n enough resources.\n- The task is run under the workdir (if specified).\n- The task is run non-interactively (without a pseudo-terminal or\n pty), so interactive commands such as ``htop`` do not work.\n Use ``ssh my_cluster`` instead.\n\n**Args:**\n task: sky.Task, or sky.Dag (experimental; 1-task only) containing the\n task to execute.\n cluster_name: name of an existing cluster to execute the task.\n dryrun: if True, do not actually execute the task.\n down: Tear down the cluster after all jobs finish (successfully or\n abnormally). If --idle-minutes-to-autostop is also set, the\n cluster will be torn down after the specified idle time.\n Note that if errors occur during provisioning/data syncing/setting\n up, the cluster will not be torn down for debugging purposes.\n backend: backend to use. If None, use the default backend\n (CloudVMRayBackend).\n\n**Returns:**\n The request ID of the exec request.\n\n\n**Request Returns:**\n job_id (Optional[int]): the job ID of the submitted job. None if the\n backend is not CloudVmRayBackend, or no job is submitted to\n the cluster.\n handle (Optional[backends.ResourceHandle]): the handle to the cluster.\n None if dryrun.\n\n**Request Raises:**\n ValueError: if the specified cluster is not in UP status.\n sky.exceptions.ClusterDoesNotExist: if the specified cluster does not\n exist.\n sky.exceptions.NotSupportedError: if the specified cluster is a\n controller that does not support this operation.\n\n### `sky.stop`\n\n```python\nsky.stop(cluster_name: str, purge: bool = False, graceful: bool = False, graceful_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nStops a cluster.\n\nData on attached disks is not lost when a cluster is stopped. Billing for\nthe instances will stop, while the disks will still be charged. Those\ndisks will be reattached when restarting the cluster.\n\nCurrently, spot instance clusters cannot be stopped (except for GCP, which\ndoes allow disk contents to be preserved when stopping spot VMs).\n\n**Args:**\n cluster_name: name of the cluster to stop.\n purge: (Advanced) Forcefully mark the cluster as stopped in SkyPilot's\n cluster table, even if the actual cluster stop operation failed on\n the cloud. WARNING: This flag should only be set sparingly in\n certain manual troubleshooting scenarios; with it set, it is the\n user's responsibility to ensure there are no leaked instances and\n related resources.\n\n**Returns:**\n The request ID of the stop request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n RuntimeError: failed to stop the cluster.\n sky.exceptions.NotSupportedError: if the specified cluster is a spot\n cluster, or a TPU VM Pod cluster, or the managed jobs controller.\n\n### `sky.start`\n\n```python\nsky.start(cluster_name: str, idle_minutes_to_autostop: Optional[int] = None, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, retry_until_up: bool = False, down: bool = False, force: bool = False) -> server_common.RequestId['backends.CloudVmRayResourceHandle']\n```\n\nRestart a cluster.\n\nIf a cluster is previously stopped (status is STOPPED) or failed in\nprovisioning/runtime installation (status is INIT), this function will\nattempt to start the cluster. In the latter case, provisioning and runtime\ninstallation will be retried.\n\nAuto-failover provisioning is not used when restarting a stopped\ncluster. It will be started on the same cloud, region, and zone that were\nchosen before.\n\nIf a cluster is already in the UP status, this function has no effect.\n\n**Args:**\n cluster_name: name of the cluster to start.\n idle_minutes_to_autostop: automatically stop the cluster after this\n many minute of idleness, i.e., no running or pending jobs in the\n cluster's job queue. Idleness gets reset whenever setting-up/\n running/pending jobs are found in the job queue. Setting this\n flag is equivalent to running ``sky.launch()`` and then\n ``sky.autostop(idle_minutes=)``. If not set, the\n cluster will not be autostopped.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes_to_autostop``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after\n ``idle_minutes_to_autostop``.\n retry_until_up: whether to retry launching the cluster until it is\n up.\n down: Autodown the cluster: tear down the cluster after specified\n minutes of idle time after all jobs finish (successfully or\n abnormally). Requires ``idle_minutes_to_autostop`` to be set.\n force: whether to force start the cluster even if it is already up.\n Useful for upgrading SkyPilot runtime.\n\n**Returns:**\n The request ID of the start request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: argument values are invalid: (1) if ``down`` is set to True\n but ``idle_minutes_to_autostop`` is None; (2) if the specified\n cluster is the managed jobs controller, and either\n ``idle_minutes_to_autostop`` is not None or ``down`` is True (omit\n them to use the default autostop settings).\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n sky.exceptions.NotSupportedError: if the cluster to restart was\n launched using a non-default backend that does not support this\n operation.\n sky.exceptions.ClusterOwnerIdentitiesMismatchError: if the cluster to\n restart was launched by a different user.\n\n### `sky.down`\n\n```python\nsky.down(cluster_name: str, purge: bool = False, graceful: bool = False, graceful_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nTears down a cluster.\n\nTearing down a cluster will delete all associated resources (all billing\nstops), and any data on the attached disks will be lost. Accelerators\n(e.g., TPUs) that are part of the cluster will be deleted too.\n\n**Args:**\n cluster_name: name of the cluster to down.\n purge: (Advanced) Forcefully remove the cluster from SkyPilot's cluster\n table, even if the actual cluster termination failed on the cloud.\n WARNING: This flag should only be set sparingly in certain manual\n troubleshooting scenarios; with it set, it is the user's\n responsibility to ensure there are no leaked instances and related\n resources.\n graceful: Cancel the user's task but block until MOUNT_CACHED data is\n fully uploaded. This helps with preserving user data integrity.\n graceful_timeout: If not None, sets a timeout for the graceful option\n above (in seconds).\n\n**Returns:**\n The request ID of the down request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n RuntimeError: failed to tear down the cluster.\n sky.exceptions.NotSupportedError: the specified cluster is the managed\n jobs controller.\n\n### `sky.status`\n\n```python\nsky.status(cluster_names: Optional[List[str]] = None, refresh: common.StatusRefreshMode = common.StatusRefreshMode.NONE, all_users: bool = False, *, _include_credentials: bool = False, _summary_response: bool = False) -> server_common.RequestId[List[responses.StatusResponse]]\n```\n\nGets cluster statuses.\n\nIf cluster_names is given, return those clusters. Otherwise, return all\nclusters.\n\n**Each cluster can have one of the following statuses:**\n\n- ``INIT``: The cluster may be live or down. It can happen in the following\n cases:\n\n - Ongoing provisioning or runtime setup. (A ``sky.launch()`` has started\n but has not completed.)\n - Or, the cluster is in an abnormal state, e.g., some cluster nodes are\n down, or the SkyPilot runtime is unhealthy. (To recover the cluster,\n try ``sky launch`` again on it.)\n\n- ``UP``: Provisioning and runtime setup have succeeded and the cluster is\n live. (The most recent ``sky.launch()`` has completed successfully.)\n\n- ``STOPPED``: The cluster is stopped and the storage is persisted. Use\n ``sky.start()`` to restart the cluster.\n\n**Autostop column:**\n\n- The autostop column indicates how long the cluster will be autostopped\n after minutes of idling (no jobs running). If ``to_down`` is True, the\n cluster will be autodowned, rather than autostopped.\n\nGetting up-to-date cluster statuses:\n\n- In normal cases where clusters are entirely managed by SkyPilot (i.e., no\n manual operations in cloud consoles) and no autostopping is used, the\n table returned by this command will accurately reflect the cluster\n statuses.\n\n- In cases where the clusters are changed outside of SkyPilot (e.g., manual\n operations in cloud consoles; unmanaged spot clusters getting preempted)\n or for autostop-enabled clusters, use ``refresh=True`` to query the\n latest cluster statuses from the cloud providers.\n\n**Args:**\n cluster_names: a list of cluster names to query. If not\n provided, all clusters will be queried.\n refresh: whether to query the latest cluster statuses from the cloud\n provider(s).\n all_users: whether to include all users' clusters. By default, only\n the current user's clusters are included.\n _include_credentials: (internal) whether to include cluster ssh\n credentials in the response (default: False).\n\n**Returns:**\n The request ID of the status request.\n\n**Request Returns:**\n cluster_records (List[Dict[str, Any]]): A list of dicts, with each dict\n containing the information of a cluster. If a cluster is found to be\n terminated or not found, it will be omitted from the returned list.\n\n ```python\n\n {\n 'name': (str) cluster name,\n 'launched_at': (int) timestamp of last launch on this cluster,\n 'handle': (ResourceHandle) an internal handle to the cluster,\n 'last_use': (str) the last command/entrypoint that affected this\n cluster,\n 'status': (sky.ClusterStatus) cluster status,\n 'autostop': (int) idle time before autostop,\n 'to_down': (bool) whether autodown is used instead of autostop,\n 'metadata': (dict) metadata of the cluster,\n 'user_hash': (str) user hash of the cluster owner,\n 'user_name': (str) user name of the cluster owner,\n 'resources_str': (str) the resource string representation of the\n cluster,\n }\n\n### `sky.autostop`\n\n```python\nsky.autostop(cluster_name: str, idle_minutes: int, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, down: bool = False, hook: Optional[str] = None, hook_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nSchedules an autostop/autodown for a cluster.\n\nAutostop/autodown will automatically stop or teardown a cluster when it\nbecomes idle for a specified duration. Idleness means there are no\nin-progress (pending/running) jobs in a cluster's job queue.\n\nIdleness time of a cluster is reset to zero, whenever:\n\n- A job is submitted (``sky.launch()`` or ``sky.exec()``).\n\n- The cluster has restarted.\n\n- An autostop is set when there is no active setting. (Namely, either\n there's never any autostop setting set, or the previous autostop setting\n was canceled.) This is useful for restarting the autostop timer.\n\nExample: say a cluster without any autostop set has been idle for 1 hour,\nthen an autostop of 30 minutes is set. The cluster will not be immediately\nautostopped. Instead, the idleness timer only starts counting after the\nautostop setting was set.\n\nWhen multiple autostop settings are specified for the same cluster, the\nlast setting takes precedence.\n\n**Args:**\n cluster_name: name of the cluster.\n idle_minutes: the number of minutes of idleness (no pending/running\n jobs) after which the cluster will be stopped automatically. Setting\n to a negative number cancels any autostop/autodown setting.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after ``idle_minutes``.\n down: if true, use autodown (tear down the cluster; non-restartable),\n rather than autostop (restartable).\n hook: optional script to execute on the remote cluster before autostop.\n The script runs before the cluster is stopped or torn down. If the\n hook fails, autostop will still proceed but a warning will be\n logged.\n hook_timeout: timeout in seconds for hook execution. If None, uses\n DEFAULT_HOOK_TIMEOUT_SECONDS (3600 = 1 hour). The hook will\n be terminated if it exceeds this timeout.\n\n**Returns:**\n The request ID of the autostop request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: if arguments are invalid.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend or the cluster is TPU VM Pod.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.cost_report`\n\n```python\nsky.cost_report(days: Optional[int] = None) -> server_common.RequestId[List[Dict[str, Any]]]\n```\n\nGets all cluster cost reports, including those that have been downed.\n\nThe estimated cost column indicates price for the cluster based on the type\nof resources being used and the duration of use up until the call to\nstatus. This means if the cluster is UP, successive calls to report will\nshow increasing price. The estimated cost is calculated based on the local\ncache of the cluster status, and may not be accurate for the cluster with\nautostop/use_spot set or terminated/stopped on the cloud console.\n\n**Args:**\n days: The number of days to get the cost report for. If not provided,\n the default is 30 days.\n\n**Returns:**\n The request ID of the cost report request.\n\n**Request Returns:**\n cluster_cost_records (List[Dict[str, Any]]): A list of dicts, with each\n dict containing the cost information of a cluster.\n\n ```python\n\n {\n 'name': (str) cluster name,\n 'launched_at': (int) timestamp of last launch on this cluster,\n 'duration': (int) total seconds that cluster was up and running,\n 'last_use': (str) the last command/entrypoint that affected this\n 'num_nodes': (int) number of nodes launched for cluster,\n 'resources': (resources.Resources) type of resource launched,\n 'cluster_hash': (str) unique hash identifying cluster,\n 'usage_intervals': (List[Tuple[int, int]]) cluster usage times,\n 'total_cost': (float) cost given resources and usage intervals,\n }\n\n### `sky.endpoints`\n\n```python\nsky.endpoints(cluster: str, port: Optional[Union[int, str]] = None) -> server_common.RequestId[Dict[int, str]]\n```\n\nGets the endpoint for a given cluster and port number (endpoint).\n\n**Example:**\n ```python\n\n import sky\n request_id = sky.endpoints('test-cluster')\n sky.get(request_id)\n\n\n**Args:**\n cluster: The name of the cluster.\n port: The port number to get the endpoint for. If None, endpoints\n for all ports are returned.\n\n**Returns:**\n The request ID of the endpoints request.\n\n**Request Returns:**\n A dictionary of port numbers to endpoints.\n If port is None, the dictionary contains all\n ports:endpoints exposed on the cluster.\n\n**Request Raises:**\n ValueError: if the cluster is not UP or the endpoint is not exposed.\n RuntimeError: if the cluster has no ports to be exposed or no endpoints\n are exposed yet.\n\n## Job Operations\n\n### `sky.queue`\n\n```python\nsky.queue(cluster_name: str, skip_finished: bool = False, all_users: bool = False) -> server_common.RequestId[List[responses.ClusterJobRecord]]\n```\n\nGets the job queue of a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n skip_finished: if True, skip finished jobs.\n all_users: if True, return jobs from all users.\n\n\n**Returns:**\n The request ID of the queue request.\n\n**Request Returns:**\n job_records (List[responses.ClusterJobRecord]): A list of job records\n for each job in the queue.\n\n ```python\n\n [\n {\n 'job_id': (int) job id,\n 'job_name': (str) job name,\n 'username': (str) username,\n 'user_hash': (str) user hash,\n 'submitted_at': (int) timestamp of submitted,\n 'start_at': (int) timestamp of started,\n 'end_at': (int) timestamp of ended,\n 'resources': (str) resources,\n 'status': (job_lib.JobStatus) job status,\n 'log_path': (str) log path,\n }\n ]\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n ``CloudVmRayBackend``.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n sky.exceptions.CommandError: if failed to get the job queue with ssh.\n\n### `sky.job_status`\n\n```python\nsky.job_status(cluster_name: str, job_ids: Optional[List[int]] = None) -> server_common.RequestId[Dict[Optional[int], Optional['job_lib.JobStatus']]]\n```\n\nGets the status of jobs on a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n job_ids: job ids. If None, get the status of the last job.\n\n**Returns:**\n The request ID of the job status request.\n\n**Request Returns:**\n job_statuses (Dict[Optional[int], Optional[job_lib.JobStatus]]): A\n mapping of job_id to job statuses. The status will be None if the\n job does not exist. If job_ids is None and there is no job on the\n cluster, it will return {None: None}.\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n ``CloudVmRayBackend``.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.cluster_job_spec`\n\n```python\nsky.cluster_job_spec(cluster_name: str, job_id: int) -> server_common.RequestId[Optional[Dict[str, Any]]]\n```\n\nGets the task YAML and entrypoint a cluster job was submitted with.\n\n**Args:**\n cluster_name: name of the cluster.\n job_id: id of the job on the cluster.\n\n**Returns:**\n The request ID of the cluster job spec request.\n\n**Request Returns:**\n spec (Optional[Dict[str, Any]]): None if the cluster does not exist or\n the job's spec was not recorded (e.g. it was submitted by an older\n API server version), otherwise:\n\n ```python\n\n {\n 'task_yaml': (str) the task YAML, with secrets redacted,\n 'entrypoint': (str) the command that submitted the job,\n }\n\n### `sky.cancel`\n\n```python\nsky.cancel(cluster_name: str, all: bool = False, all_users: bool = False, job_ids: Optional[List[int]] = None, _try_cancel_if_cluster_is_init: bool = False) -> server_common.RequestId[None]\n```\n\nCancels jobs on a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n all: if True, cancel all jobs.\n all_users: if True, cancel all jobs from all users.\n job_ids: a list of job IDs to cancel.\n _try_cancel_if_cluster_is_init: (bool) whether to try cancelling the job\n even if the cluster is not UP, but the head node is still alive.\n This is used by the jobs controller to cancel the job when the\n worker node is preempted in the spot cluster.\n\n**Returns:**\n The request ID of the cancel request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: if arguments are invalid.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the specified cluster is a\n controller that does not support this operation.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.tail_logs`\n\n```python\nsky.tail_logs(cluster_name: str, job_id: Optional[int], follow: bool, tail: int = 0, output_stream: Optional['io.TextIOBase'] = None, *, preload_content: bool = True) -> Union[int, Iterator[Optional[str]]]\n```\n\nTails the logs of a job.\n\n**Args:**\n cluster_name: name of the cluster.\n job_id: job id.\n follow: if True, follow the logs. Otherwise, return the logs\n immediately.\n tail: if > 0, tail the last N lines of the logs.\n output_stream: the stream to write the logs to. If None, print to the\n console. Cannot be used with preload_content=False.\n preload_content: if False, returns an Iterator[str | None] containing\n the logs without the function blocking on the retrieval of entire\n log. Iterator returns None when the log has been completely\n streamed. Default True. Cannot be used with output_stream.\n\n**Returns:**\n If preload_content is True:\n Exit code based on success or failure of the job. 0 if success,\n 100 if the job failed. See exceptions.JobExitCode for possible exit\n codes.\n If preload_content is False:\n Iterator[str | None] containing the logs without the function\n blocking on the retrieval of entire log. Iterator returns None\n when the log has been completely streamed.\n\n**Request Raises:**\n ValueError: if arguments are invalid or the cluster is not supported.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.tail_provision_logs`\n\n```python\nsky.tail_provision_logs(cluster_name: str, worker: Optional[int] = None, follow: bool = True, tail: int = 0, output_stream: Optional['io.TextIOBase'] = None) -> int\n```\n\nTails the provisioning logs (provision.log) for a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n worker: worker id in multi-node cluster.\n If None, stream the logs of the head node.\n follow: follow the logs.\n tail: lines from end to tail.\n output_stream: optional stream to write logs.\n**Returns:**\n Exit code 0 on streaming success; raises on HTTP error.\n\n### `sky.tail_hook_logs`\n\n```python\nsky.tail_hook_logs(cluster_name: str, event: Optional[str] = None, follow: bool = True, tail: int = 0) -> int\n```\n\nTails a per-event lifecycle-hook log on the cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n event: one of ``stop``, ``preemption``, ``down``. When None,\n auto-selects whichever log exists on the cluster.\n follow: whether to follow the logs.\n tail: number of lines to display from the end of the log file.\n\n**Returns:**\n Exit code 0 on streaming success; non-zero on failure.\n\n### `sky.download_logs`\n\n```python\nsky.download_logs(cluster_name: str, job_ids: Optional[List[str]]) -> Dict[str, str]\n```\n\nDownloads the logs of jobs.\n\n**Args:**\n cluster_name: (str) name of the cluster.\n job_ids: (List[str]) job ids.\n\n**Returns:**\n The request ID of the download_logs request.\n\n**Request Returns:**\n job_log_paths (Dict[str, str]): a mapping of job_id to local log path.\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n## Infrastructure & Configuration\n\n### `sky.check`\n\n```python\nsky.check(infra_list: Optional[Tuple[str, ...]], verbose: bool, workspace: Optional[str] = None) -> server_common.RequestId[Dict[str, Dict[str, List[str]]]]\n```\n\nChecks the credentials to enable clouds.\n\n**Args:**\n infra: The infra to check.\n verbose: Whether to show verbose output.\n workspace: The workspace to check. If None, all workspaces will be\n checked.\n\n**Returns:**\n The request ID of the check request.\n\n**Request Returns:**\n Dict mapping workspace name to a dict of cloud name to list of\n enabled capability strings (e.g. 'compute', 'storage').\n\n### `sky.enabled_clouds`\n\n```python\nsky.enabled_clouds(workspace: Optional[str] = None, expand: bool = False) -> server_common.RequestId[List[str]]\n```\n\nGets the enabled clouds.\n\n**Args:**\n workspace: The workspace to get the enabled clouds for. If None, the\n active workspace will be used.\n expand: Whether to expand Kubernetes and SSH to list of resource pools.\n\n**Returns:**\n The request ID of the enabled clouds request.\n\n**Request Returns:**\n A list of enabled clouds in string format.\n\n### `sky.list_accelerators`\n\n```python\nsky.list_accelerators(gpus_only: bool = True, name_filter: Optional[str] = None, region_filter: Optional[str] = None, quantity_filter: Optional[int] = None, clouds: Optional[Union[List[str], str]] = None, all_regions: bool = False, require_price: bool = True, case_sensitive: bool = True) -> server_common.RequestId[Dict[str, List['catalog.common.InstanceTypeInfo']]]\n```\n\nLists the names of all accelerators offered by Sky.\n\nThis will include all accelerators offered by Sky, including those\nthat may not be available in the user's account.\n\n**Args:**\n gpus_only: Whether to only list GPU accelerators.\n name_filter: The name filter.\n region_filter: The region filter.\n quantity_filter: The quantity filter.\n clouds: The clouds to list.\n all_regions: Whether to list all regions.\n require_price: Whether to require price.\n case_sensitive: Whether to case sensitive.\n\n**Returns:**\n The request ID of the list accelerator counts request.\n\n**Request Returns:**\n acc_to_instance_type_dict (Dict[str, List[InstanceTypeInfo]]): A\n dictionary of canonical accelerator names mapped to a list of\n instance type offerings. See usage in cli.py.\n\n### `sky.kubernetes_node_info`\n\n```python\nsky.kubernetes_node_info(context: Optional[str] = None) -> server_common.RequestId['models.KubernetesNodesInfo']\n```\n\nGets the resource information for all the nodes in the cluster.\n\nCurrently only GPU resources are supported. The function returns the total\nnumber of GPUs available on the node and the number of free GPUs on the\nnode.\n\nIf the user does not have sufficient permissions to list pods in all\nnamespaces, the function will return free GPUs as -1.\n\n**Args:**\n context: The Kubernetes context. If None, the default context is used.\n\n**Returns:**\n The request ID of the Kubernetes node info request.\n\n**Request Returns:**\n KubernetesNodesInfo: A model that contains the node info map and other\n information.\n\n### `sky.realtime_kubernetes_gpu_availability`\n\n```python\nsky.realtime_kubernetes_gpu_availability(context: Optional[str] = None, name_filter: Optional[str] = None, quantity_filter: Optional[int] = None, is_ssh: Optional[bool] = None) -> server_common.RequestId[List[Tuple[str, List['models.RealtimeGpuAvailability']]]]\n```\n\nGets the real-time Kubernetes GPU availability.\n\n**Returns:**\n The request ID of the real-time Kubernetes GPU availability request.\n\n### `sky.optimize`\n\n```python\nsky.optimize(dag: 'sky.Dag', minimize: common.OptimizeTarget = common.OptimizeTarget.COST, admin_policy_request_options: Optional[admin_policy.RequestOptions] = None) -> server_common.RequestId['sky.Dag']\n```\n\nFinds the best execution plan for the given DAG.\n\n**Args:**\n dag: the DAG to optimize.\n minimize: whether to minimize cost or time.\n admin_policy_request_options: Request options used for admin policy\n validation. This is only required when a admin policy is in use,\n see: https://docs.skypilot.co/en/latest/cloud-setup/policy.html\n\n**Returns:**\n The request ID of the optimize request.\n\n**Request Returns:**\n optimized_dag (str): The optimized DAG in YAML format.\n\n**Request Raises:**\n exceptions.ResourcesUnavailableError: if no resources are available\n for a task.\n exceptions.NoCloudAccessError: if no public clouds are enabled.\n\n### `sky.validate`\n\n```python\nsky.validate(dag: 'sky.Dag', workdir_only: bool = False, admin_policy_request_options: Optional[admin_policy.RequestOptions] = None, _skip_local_paths: bool = False) -> None\n```\n\nValidates the tasks.\n\nThe file paths (workdir and file_mounts) are validated on the client side\nwhile the rest (e.g. resource) are validated on server side.\n\nRaises exceptions if the DAG is invalid.\n\n**Args:**\n dag: the DAG to validate.\n workdir_only: whether to only validate the workdir. This is used for\n `exec` as it does not need other files/folders in file_mounts.\n admin_policy_request_options: Request options used for admin policy\n validation. This is only required when a admin policy is in use,\n see: https://docs.skypilot.co/en/latest/cloud-setup/policy.html\n _skip_local_paths: Do not check the workdir and file mount sources on\n this machine. For callers whose sources are API server paths\n inside an upload blob; the server resolves the blob and checks\n the paths when it runs the request.\n\n### `sky.reload_config`\n\n```python\nsky.reload_config() -> None\n```\n\nReloads the client-side config.\n\n## Storage & Volumes\n\n### `sky.storage_ls`\n\n```python\nsky.storage_ls() -> server_common.RequestId[List[responses.StorageRecord]]\n```\n\nGets the storages.\n\n**Returns:**\n The request ID of the storage list request.\n\n**Request Returns:**\n storage_records (List[responses.StorageRecord]):\n A list of storage records.\n\n### `sky.storage_delete`\n\n```python\nsky.storage_delete(name: str) -> server_common.RequestId[None]\n```\n\nDeletes a storage.\n\n**Args:**\n name: The name of the storage to delete.\n\n**Returns:**\n The request ID of the storage delete request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: If the storage does not exist.\n\n## API Server\n\n### `sky.api_start`\n\n```python\nsky.api_start(*, deploy: bool = False, host: str = '127.0.0.1', foreground: bool = False, metrics: bool = False, metrics_port: Optional[int] = None, enable_basic_auth: bool = False, port: Optional[int] = None) -> None\n```\n\nStarts the API server.\n\nIt checks the existence of the API server and starts it if it does not\nexist.\n\n**Args:**\n deploy: Whether to deploy the API server, i.e. fully utilize the\n resources of the machine.\n host: The host to bind the API server to. Under deploy, the server\n always binds a wildcard address for remote access: ``::`` when an\n IPv6 host is given, otherwise ``0.0.0.0``. Without deploy the host\n is used as-is.\n foreground: Whether to run the API server in the foreground (run in\n the current process).\n metrics: Whether to export metrics of the API server.\n metrics_port: The port to export metrics of the API server.\n enable_basic_auth: Whether to enable basic authentication\n in the API server.\n port: The port to bind the API server to. Defaults to the\n SKYPILOT_API_SERVER_LOCAL_PORT environment variable, or 46580.\n Other client commands only find a server on a non-default port\n if the same environment variable is exported.\n**Returns:**\n None\n\n### `sky.api_stop`\n\n```python\nsky.api_stop() -> None\n```\n\nStops the API server.\n\nIt will do nothing if the API server is remotely hosted.\n\n**Returns:**\n None\n\n### `sky.api_status`\n\n```python\nsky.api_status(request_ids: Optional[List[Union[server_common.RequestId[T], str]]] = None, all_status: bool = False, limit: Optional[int] = None, fields: Optional[List[str]] = None, cluster_name: Optional[str] = None) -> List[payloads.RequestPayload]\n```\n\nLists all requests.\n\n**Args:**\n request_ids: The prefixes of the request IDs of the requests to query.\n If None, all requests are queried.\n all_status: Whether to list all finished requests as well. This argument\n is ignored if request_ids is not None.\n limit: The number of requests to show. If None, show all requests.\n fields: The fields to get. If None, get all fields except\n return_value and error. This argument is ignored if request_ids\n is not None.\n cluster_name: Filter requests by cluster name.\n If None, show all requests.\n\n**Returns:**\n A list of request payloads.\n\n### `sky.api_info`\n\n```python\nsky.api_info() -> responses.APIHealthResponse\n```\n\nGets the server's status, commit and version.\n\n**Returns:**\n A dictionary containing the server's status, commit and version.\n\n ```python\n\n {\n 'status': 'healthy',\n 'api_version': '1',\n 'commit': 'abc1234567890',\n 'version': '1.0.0',\n 'version_on_disk': '1.0.0',\n 'user': {\n 'name': 'test@example.com',\n 'id': '12345abcd',\n },\n }\n\n Note that user may be None if we are not using an auth proxy.\n\n### `sky.api_cancel`\n\n```python\nsky.api_cancel(request_ids: Optional[Union[server_common.RequestId[T], List[server_common.RequestId[T]], str, List[str]]] = None, all_users: bool = False, silent: bool = False) -> server_common.RequestId[List[str]]\n```\n\nAborts a request or all requests.\n\nWith neither request_ids nor all_users, aborts all of the caller's own\nrequests.\n\n**Args:**\n request_ids: The request ID(s) to abort. Can be a single string or a\n list of strings. On a server with per-user auth, an admin can\n abort any request by id and other users only their own; on a\n server without it, anyone can abort any request by id.\n all_users: Whether to abort all requests from all users. On a server\n with per-user auth, a non-admin aborts only their own. Cannot be\n combined with request_ids.\n silent: Whether to suppress the output.\n\n**Returns:**\n The request ID of the abort request itself.\n\n**Request Returns:**\n A list of request IDs that were cancelled.\n\n**Raises:**\n ValueError: If both request_ids and all_users are given.\n\n### `sky.api_server_logs`\n\n```python\nsky.api_server_logs(follow: bool = True, tail: Optional[int] = None) -> None\n```\n\nStreams the API server logs.\n\n**Args:**\n follow: Whether to follow the logs.\n tail: the number of lines to show from the end of the logs.\n If None, show all logs.\n\n**Returns:**\n None\n\n### `sky.api_login`\n\n```python\nsky.api_login(endpoint: Optional[str] = None, relogin: bool = False, service_account_token: Optional[str] = None, no_browser: bool = False) -> None\n```\n\nLogs into a SkyPilot API server.\n\nThis sets the endpoint globally, i.e., all SkyPilot CLI and SDK calls will\nuse this endpoint.\n\nTo temporarily override the endpoint, use the environment variable\n`SKYPILOT_API_SERVER_ENDPOINT` instead.\n\n**Args:**\n endpoint: The endpoint of the SkyPilot API server, e.g.,\n http://1.2.3.4:46580 or https://skypilot.mydomain.com.\n relogin: Whether to force relogin with OAuth2 when enabled.\n service_account_token: Service account token for authentication.\n no_browser: If True, do not attempt to open a browser locally; print\n the auth URL and let the user open it themselves. Skips the\n localhost-callback flow, which requires a local browser.\n\n**Returns:**\n None\n\n### `sky.api_logout`\n\n```python\nsky.api_logout() -> None\n```\n\nLogout of the API server.\n\nClears all cookies and settings stored in ~/.sky/config.yaml\n\n## Utilities\n\n### `sky.dashboard`\n\n```python\nsky.dashboard(starting_page: Optional[str] = None) -> None\n```\n\nStarts the dashboard for SkyPilot.\n\n### `sky.get`\n\n```python\nsky.get(request_id: server_common.RequestId[T]) -> T\n```\n\nWaits for and gets the result of a request.\n\nThis function will not check the server health since /api/get is typically\nnot the first API call in an SDK session and checking the server health\nmay cause GET /api/get being sent to a restarted API server.\n\n**Args:**\n request_id: The request ID of the request to get. May be a full request\n ID or a prefix.\n\n**Returns:**\n The ``Request Returns`` of the specified request. See the documentation\n of the specific requests above for more details.\n\n**Raises:**\n Exception: It raises the same exceptions as the specific requests,\n see ``Request Raises`` in the documentation of the specific requests\n above.\n\n### `sky.stream_and_get`\n\n```python\nsky.stream_and_get(request_id: Optional[server_common.RequestId[T]] = None, log_path: Optional[str] = None, tail: Optional[int] = None, follow: bool = True, output_stream: Optional['io.TextIOBase'] = None, relay_rich_status: bool = False) -> Optional[T]\n```\n\nStreams the logs of a request or a log file and gets the final result.\n\nThis will block until the request is finished. The request id can be a\nprefix of the full request id.\n\n**Args:**\n request_id: The request ID of the request to stream. May be a full\n request ID or a prefix.\n If None, the latest request submitted to the API server is streamed.\n Using None request_id is not recommended in multi-user environments.\n log_path: The path to the log file to stream.\n tail: The number of lines to show from the end of the logs.\n If None, show all logs.\n follow: Whether to follow the logs.\n output_stream: The output stream to write to. If None, print to the\n console.\n relay_rich_status: If True, forward encoded rich-status control payloads\n verbatim to the output instead of rendering a local spinner. Used by\n the managed jobs controller to preserve provisioning spinner codes\n in its per-job log. See\n :func:`sky.utils.rich_utils.decode_rich_status`.\n\n**Returns:**\n The ``Request Returns`` of the specified request. See the documentation\n of the specific requests above for more details.\n If follow is False, will always return None. See note on\n stream_response.\n\n**Raises:**\n Exception: It raises the same exceptions as the specific requests,\n see ``Request Raises`` in the documentation of the specific requests\n above.\n\n### `sky.workspaces`\n\n```python\nsky.workspaces() -> server_common.RequestId[Dict[str, Any]]\n```\n\nGets the workspaces.\n\n### `sky.set_preferred_workspace`\n\n```python\nsky.set_preferred_workspace(preferred: Optional[str]) -> Dict[str, Any]\n```\n\nSets (or clears with None) the user's preferred workspace.\n\n**Args:**\n preferred: workspace name to set as default, or None to clear.\n\n**Returns:**\n ``{'preferred': }`` echoing what was set. Callers that\n need the resolved workspace + accessible list should follow up\n with :func:`get_user_workspace`. Raises if the server rejects\n the change (workspace does not exist, or user lacks permission\n to it).\n\n### `sky.get_user_workspace`\n\n```python\nsky.get_user_workspace(requested: Optional[str] = None) -> Dict[str, Any]\n```\n\nReturns workspace state for the calling user.\n\nMirrors the launch-path precedence — if the caller has an explicit\n``active_workspace``, the server returns that with ``source='explicit'``;\notherwise the resolver runs (preferred / default-fallback /\nsingle-membership).\n\n**Args:**\n requested: explicit active workspace to ask about. ``None`` (the\n default) — the SDK reads your locally-configured\n ``active_workspace`` (the value `skypilot_config` merges\n from ``~/.sky/config.yaml`` + ``./.sky.yaml`` + any\n ``--config active_workspace=X`` override) and forwards it\n on the wire as ``?requested=``. Pass a non-None value to\n query the resolver as if ``active_workspace`` were that\n value, without changing your local config — useful for\n previewing \"what would land if I switched to X\".\n\n**Returns:**\n ``{workspace, source, note, preferred, accessible}``.\n\n * ``workspace``: the workspace the launch path would pick. Can\n be ``None`` when the resolver couldn't pick (no access /\n ambiguous / explicit ``requested`` rejected by RBAC); the\n reason is then in ``note``.\n * ``source``: one of ``WORKSPACE_SOURCE_*`` on success, ``None``\n when ``workspace`` is ``None``.\n * ``note``: optional message — drift on success\n (``preferred 'team-x' not accessible``) or the resolver error\n when ``workspace`` is ``None``.\n * ``preferred``: the persisted preferred workspace (``None`` if\n unset).\n * ``accessible``: sorted list of workspaces the user can launch\n into.\n\n## Other Functions\n\n### `sky.create_debug_dump`\n\n```python\nsky.create_debug_dump(request_ids: Optional[List[str]] = None, cluster_names: Optional[List[str]] = None, managed_job_ids: Optional[List[int]] = None, recent_minutes: Optional[float] = None) -> server_common.RequestId[str]\n```\n\nCreate a debug dump for troubleshooting.\n\nAt least one of ``request_ids``, ``cluster_names``, ``managed_job_ids``,\nor ``recent_minutes`` must be provided.\n\n**Args:**\n request_ids: List of request IDs or prefixes to include in the\n dump. Prefixes are resolved to all matching request IDs on\n the server.\n cluster_names: List of cluster names to include in the dump.\n managed_job_ids: List of managed job IDs to include in the dump.\n recent_minutes: If specified, include all resources active within\n this many minutes.\n\n**Returns:**\n The request ID of the debug dump creation request.\n\n**Request Returns:**\n Path to the created zip file on the server.\n\n### `sky.download_debug_dump`\n\n```python\nsky.do","agent/skills/skypilot/references/job-investigation.md":"# Investigating a Managed Job\n\nHow to answer \"what is my job doing and why\" with as few commands as\npossible. Every command here supports `-o json`; prefer it over parsing\ntable output.\n\n## Which command answers which question\n\n| Question | Command | Look at |\n|----------|---------|---------|\n| What state is job N in, and why? | `sky jobs queue -v -o json` | `status`, `details`, `failure_reason`, `schedule_state`, `recovery_count` |\n| Why is job N still provisioning / pending on the cluster? | `sky jobs queue -v -o json` | `details`, when it starts with `Launching (` |\n| How long has job N been at this? | `sky jobs queue -v -o json` | `submitted_at`, and the duration columns in the table |\n| What did job N print recently? | `sky jobs logs N --tail 100 --no-follow` | stdout/stderr of the task |\n| What did the controller do (provision, recover, retry)? | `sky jobs logs N --controller --no-follow` | controller log |\n| Wait until job N finishes | `sky jobs logs N` (blocks) | exit when the job ends |\n\nDo not poll with `sleep` + `sky jobs queue`. Stream logs to block, or\nfetch `--tail` once after the fact.\n\n`details` is the current explanation, not a history. For the sequence of\ntransitions a job went through — and, on Slurm, how long the allocation\nwaited and on what — the timeline lives behind the `jobs.events` API and is\nsurfaced by the enterprise client.\n\n## Reading `details`\n\n`details` in `sky jobs queue -v -o json` is the one-line explanation of the\ncurrent state. Common shapes:\n\n| `details` | Meaning | Next step |\n|-----------|---------|-----------|\n| `Waiting for other jobs to launch` | Controller is at its concurrent-launch limit | Nothing to fix; the duration columns say how long it has waited |\n| `Waiting for higher priority jobs to launch` | A higher-priority managed job is ahead | Raise `priority` in the task YAML if this job matters more |\n| `In backoff, waiting for resources` | Provisioning failed and is retrying | `sky jobs logs N --controller --no-follow` shows the last failure |\n| `Recovering: ` | Cluster was lost or the task crashed (e.g. OOMKilled) | Fix the cause; recovery is automatic |\n| `Failure: ` | Terminal failure | Read the reason; `sky jobs logs N --no-follow` for the task output |\n| `Cancellation requested by ...` | Someone cancelled it | The requester and request ID are in the text |\n| `Launching (pending: ; partition:

)` | Job is STARTING and Slurm has not allocated nodes yet | See \"Slurm pending reasons\" below |\n| `Launching (nodes allocated; Slurm job on )` | The queue wait is over; the job is still bootstrapping | Nothing to fix. That id is what `sacct -j ` takes if you have login-node access |\n\n## Slurm pending reasons\n\nWhile a managed job's Slurm allocation is queued, `details` carries the\n`squeue` reason and the partition it is queued in — the partition matters\nbecause the same reason means different things in different partitions, and\nby the time anyone looks the allocation may be gone. The common codes:\n\n| Reason | Category | What to do |\n|--------|----------|------------|\n| `Resources`, `Priority`, `ReqNodeNotAvail` | No free capacity in the partition | Wait, or resubmit to another partition via `config.slurm` |\n| `QOSGrpGRES`, `QOSMax*`, `AssocGrp*` | A QoS / account quota is full | Wait for the group's jobs to finish, or use a different QoS/partition |\n| `Dependency`, `DependencyNeverSatisfied` | Waiting on another Slurm job; `NeverSatisfied` will pend forever | Check the dependency job; cancel and resubmit if it failed |\n| `JobHeldUser`, `JobHeldAdmin`, `launch failed requeued held` | Job is held | Ask the admin to `scontrol release`, or cancel and relaunch |\n| `BeginTime` | Scheduled start time not reached | Nothing to fix |\n\nThe reason is recorded each time it changes, so a job that moved from\n`Resources` to `Priority` and back leaves all three behind it — `details`\nshows only the latest.\n\n## Recognising a stuck job\n\n- Repeated `Recovering: ` means a persistent problem (bad node,\n OOM, quota). The controller keeps retrying and will not fix it; read\n `sky jobs logs N --controller --no-follow`.\n- A long gap between STARTING and RUNNING is provisioning time. On Slurm\n that is queue time, and `details` says what the queue is waiting on.\n- `failure_reason` distinguishes the user's program failing from an\n infrastructure failure; do not report the latter as a code bug.\n","agent/skills/skypilot/references/cli-reference.md":"\n\n\n# SkyPilot CLI Reference\n\nComplete reference for all `sky` commands and options.\n\n## Core Cluster Commands\n\n### `sky launch`\n\nLaunch a cluster or task.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--cluster`, `-c` — A cluster name. If provided, either reuse an existing cluster with that name or provision a new cluster with that name. Otherwise provision a new cluster with an autogenerated name.\n- `--dryrun` — If True, do not actually run the job.\n- `--detach-run`, `-d`, `-no-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--docker` — (Deprecated) Local docker support is deprecated. To run locally, create a local Kubernetes cluster with ``sky local up``.\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--priority` — Priority for this task. Accepts either an integer (from -1000 to 1000, or a string priority class name. Passing \"none\" clears both.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--idle-minutes-to-autostop`, `-i` — Automatically stop the cluster after this many minutes of idleness, i.e., no running or pending jobs in the cluster's job queue. Idleness gets reset depending on the ``--wait-for`` flag. Setting this ...\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes-to-autostop``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs an...\n- `--down` — Autodown the cluster: tear down the cluster after all jobs finish (successfully or abnormally). If --idle-minutes-to-autostop is also set, the cluster will be torn down after the specified idle time. ...\n- `--retry-until-up`, `-r` — Whether to retry provisioning infinitely until the cluster is up, if we fail to launch the cluster on any possible region/cloud due to unavailability errors.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--no-setup` — Skip setup phase when (re-)launching cluster.\n- `--clone-disk-from`, `--clone` — [Experimental] Clone disk from an existing cluster to launch a new one. This is useful when the new cluster needs to have the same data on the boot disk as an existing cluster.\n- `--fast` — [Experimental] If the cluster is already up and available, skip provisioning and setup steps.\n- `--resize` — Resize an existing cluster to --num-nodes. Supports both scale-up (adding workers) and scale-down (removing workers). Scale-down requires no running jobs. Requires -c to specify an existing cluster.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workspace`, `-w` — Workspace to launch into. Shorthand for `--config active_workspace=`.\n\n### `sky exec`\n\nExecute a task or command on an existing cluster.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTER` — text\n- `--cluster`, `-c` — This is the same as the positional argument, just for consistency.\n- `ENTRYPOINT` — text\n- `--detach-run`, `-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n\n### `sky status`\n\nShow clusters.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--refresh`, `-r` — Query the latest cluster statuses from the cloud provider(s).\n- `--ip` — Get the IP address of the head node of a cluster. This option will override all other options. For Kubernetes clusters, the returned IP address is the internal IP of the head pod, and may not be acces...\n- `--endpoints` — Get all exposed endpoints and corresponding URLs for acluster. This option will override all other options.\n- `--endpoint` — Get the endpoint URL for the specified port number on the cluster. This option will override all other options.\n- `--show-managed-jobs`, `--no-show-managed-jobs` — Also show recent in-progress managed jobs, if any.\n- `--show-services`, `--no-show-services` — Also show sky serve services, if any.\n- `--show-pools`, `--no-show-pools` — Also show pools, if any.\n- `CLUSTERS` — text\n- `--all-users`, `-u` — Show all clusters, including those not owned by the current user.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n### `sky stop`\n\nStop cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Stop all existing clusters.\n- `--all-users`, `-u` — Stop all existing clusters for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky start`\n\nRestart cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Start all existing clusters.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--idle-minutes-to-autostop`, `-i` — Automatically stop the cluster after this many minutes of idleness, i.e., no running or pending jobs in the cluster's job queue. Idleness gets reset depending on the ``--wait-for`` flag. Setting this ...\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes-to-autostop``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs an...\n- `--down` — Autodown the cluster: tear down the cluster after specified minutes of idle time after all jobs finish (successfully or abnormally). Requires --idle-minutes-to-autostop to be set.\n- `--retry-until-up`, `-r` — Retry provisioning infinitely until the cluster is up, if we fail to start the cluster due to unavailability errors.\n- `--force`, `-f` — Force start the cluster even if it is already UP. Useful for upgrading the SkyPilot runtime on the cluster.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky down`\n\nTear down cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Tear down all existing clusters.\n- `--all-users`, `-u` — Tear down all existing clusters for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--purge`, `-p` — (Advanced) Forcefully remove the cluster(s) from SkyPilot's cluster table, even if the actual cluster termination failed on the cloud. WARNING: This flag should only be set sparingly in certain manual...\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky autostop`\n\nSchedule an autostop or autodown for cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Autostop all existing clusters.\n- `--all-users`, `-u` — Autostop all existing clusters for all users.\n- `--idle-minutes`, `-i` — Set the idle minutes before autostopping the cluster. See the doc above for detailed semantics.\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs and SSH connec...\n- `--cancel` — Cancel any currently active auto{stop,down} setting for the cluster. No-op if there is no active setting.\n- `--down` — Use autodown (tear down the cluster; non-restartable), instead of autostop (restartable).\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky cost-report`\n\nShow estimated costs for launched clusters.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--all`, `-a` — Show all cluster information.\n- `--days` (default: `30`) — Show clusters from the last N days. Default is 30 days. If set to 0, show all clusters.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Job Queue Commands\n\n### `sky queue`\n\nShow the job queue for cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--all-users`, `-u` — Show all users' information in full.\n- `--skip-finished`, `-s` — Show only pending/running jobs' information.\n- `CLUSTERS` — text\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n### `sky logs`\n\nTail the log of a job.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--provision` — Stream the cluster provisioning logs (provision.log).\n- `--hook` — Stream a per-event lifecycle-hook log from the cluster. Omit the event name to auto-select whichever log exists.\n- `--autostop` — [DEPRECATED] Alias for `--hook stop`. The autostop event was renamed to `stop` in the lifecycle-hooks framework.\n- `--worker`, `-w` — The worker ID to stream the logs from. If not set, stream the logs of the head node.\n- `--sync-down`, `-s` — Sync down the logs of a job to the local machine. For a distributed job, a separate log file from each worker will be downloaded.\n- `--status` — If specified, do not show logs but exit with a status code for the job's status: 0 for succeeded, or 1 for all other statuses.\n- `--follow`, `--no-follow` — Follow the logs of a job. If --no-follow is specified, print the log so far and exit. [default: --follow]\n- `--tail` (default: `-1`) — Number of lines to display from the end of the log file. Default is the last 1000 lines — sensible for multi-GB logs where downloading the full file is slow. Pass --tail 0 to print the entire log.\n- `CLUSTER` — text\n- `JOB_IDS` — text\n\n### `sky cancel`\n\nCancel job(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTER` — text\n- `--all`, `-a` — Cancel all jobs from current user on the specified cluster.\n- `--all-users`, `-u` — Cancel all jobs on the specified cluster for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `JOBS` — integer\n\n## Infrastructure Commands\n\n### `sky check`\n\nCheck which clouds are available to use.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `INFRA_LIST` — text\n- `--verbose`, `-v` — Show the activated account for each cloud.\n- `--workspace`, `-w` — The workspace to check. If None, all workspaces will be checked.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Dashboard\n\n### `sky dashboard`\n\nOpens the SkyPilot dashboard.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n\n## GPU/Accelerator Commands\n\nSkyPilot GPU/Accelerator CLI.\n\n### `sky gpus label`\n\nLabel GPU nodes in a Kubernetes cluster for use with SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--context`, `-c` — Kubernetes context to use. If not specified, uses the current context from the API server.\n- `--cleanup` — Only cleanup existing GPU labeler resources.\n- `--async` — Do not wait for GPU labeling to complete.\n\n### `sky gpus list`\n\nShow supported GPU/TPU/accelerators and their prices.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ACCELERATOR_STR` — text\n- `--all`, `-a` — Show details of all GPU/TPU/accelerator offerings.\n- `--infra` — Infrastructure to query. Examples: \"aws\", \"aws/us-east-1\"\n- `--cloud` — Cloud provider to query.\n- `--region` — The region to use. If not specified, shows accelerators from all regions.\n- `--all-regions` — Show pricing and instance details for a specified accelerator across all regions and clouds.\n- `--verbose`, `-v` — Show all information in full.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Managed Jobs Commands\n\nManaged Jobs CLI (jobs with auto-recovery).\n\n### `sky jobs cancel`\n\nCancel managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--name`, `-n` — Managed job name to cancel.\n- `--pool`, `-p` — Pool name to cancel.\n- `JOB_IDS` — integer\n- `--task` — Cancel one dynamic task of the job (a job launched from inside it), by the index shown in `sky jobs queue` or by name. A declared task cannot be cancelled alone.\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--all`, `-a` — Cancel all managed jobs for the current user.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--all-users`, `-u` — Cancel all managed jobs from all users.\n\n### `sky jobs dashboard`\n\nOpens a dashboard for managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n\n### `sky jobs launch`\n\nLaunch a managed job from a YAML or a command.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--priority` — Priority for this task. Accepts either an integer (from -1000 to 1000, or a string priority class name. Passing \"none\" clears both.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--cluster`, `-c` — Alias for --name, the name of the managed job.\n- `--job-recovery` — Recovery strategy to use for managed jobs.\n- `--detach-run`, `-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--pool`, `-p` — (Experimental; optional) Pool to use for jobs submission.\n- `--num-jobs` — Number of jobs to submit.\n- `--job-group` — Attach to an existing job group, by job id or unique running job name. The job is shown under it and cancelled with it. Defaults to the surrounding job group when launched from inside one.\n- `--no-job-group` — Launch a top-level job even when running inside a job group (do not attach to it).\n- `--depends-on` — Comma-separated managed job IDs to wait for. The job starts once all of them succeed, and is cancelled if any of them ends otherwise. The launch fails if one of them has already ended without succeedi...\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workspace`, `-w` — Workspace to submit the managed job into. Shorthand for `--config active_workspace=`.\n- `--output`, `-o` — Print only the submitted job IDs, for scripts: `id` prints one per line, `json` prints {\"job_ids\": [...]}. Everything else goes to stderr. Implies --detach-run.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky jobs logs`\n\nTail or sync down the log of a managed job.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--name`, `-n` — Managed job name.\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this job; useful for debugging launching/recoveries, etc.\n- `--refresh`, `-r` — Query the latest job logs, restarting the jobs controller if stopped.\n- `--sync-down`, `-s` — Download logs for all jobs shown in the queue.\n- `--tail` (default: `-1`) — Number of lines to display from the end of the log file. Default is the last 1000 lines — sensible for multi-GB logs where downloading the full file is slow. Pass --tail 0 to print the entire log.\n- `JOB_ID` — integer\n- `TASK` — text\n\n### `sky jobs pool` (subgroup)\n\n(Experimental) Pool management commands.\n\n### `sky jobs pool apply`\n\nEither apply a config to a pool for managed jobs submission or update the number of workers in the pool. One of POOL_YAML or --workers must be provided.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `POOL_YAML` — text\n- `--pool`, `-p` — A pool name. Unique for each pool. If not provided, a unique name is autogenerated.\n- `--mode` (default: `rolling`) — Update mode. If \"rolling\", pool will be updated with rolling update. If \"blue_green\", pool will be updated with blue-green update. This option is only valid when the pool is already running.\n- `--workers` — Can be used to update the number of workers in the pool.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky jobs pool down`\n\nDelete pool(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `POOL_NAMES` — text\n- `--all`, `-a` — Delete all pools.\n- `--purge`, `-p` — Tear down pools in failed status.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky jobs pool logs`\n\nTail or sync down logs of a pool.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this pool.\n- `--sync-down`, `-s` — Sync down logs to the local machine. Can be combined with --controller or worker ID to narrow scope.\n- `--tail` — The number of lines to display from the end of the log file. Default is None, which means print all lines.\n- `POOL_NAME` — text\n- `WORKER_IDS` — integer\n\n### `sky jobs pool status`\n\nShow statuses of pools.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `POOL_NAMES` — text\n- `--all`, `-a` — Show all workers.\n\n### `sky jobs queue`\n\nShow statuses of managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--limit`, `-l` (default: `50`) — Number of jobs to show, default is 50, use \"-a/--all\" to show all jobs.\n- `--refresh`, `-r` — Query the latest statuses, restarting the jobs controller if stopped.\n- `--skip-finished` — Show only pending/running jobs' information.\n- `-s`, `--status` — Filter by status, comma-separated (e.g. -s FAILED,FAILED_SETUP). A bare -s (no value) is a deprecated alias for --skip-finished.\n- `--since` — Show only jobs submitted within this time window, relative to now (e.g. \"30m\", \"48h\", \"7d\", \"2w\"). A bare number is seconds. Mutually exclusive with --after.\n- `--after` — Show only jobs submitted at or after this absolute local time (e.g. \"2026-01-13\" or \"2026-01-13 15:30:00\"). Mutually exclusive with --since.\n- `--before` — Show only jobs submitted at or before this absolute local time (e.g. \"2026-01-13\" or \"2026-01-13 15:30:00\").\n- `--infra` — Show only jobs running on this infrastructure. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/u...\n- `--all-users`, `-u` — Show jobs from all users.\n- `--all`, `-a` — Show all jobs.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## SkyServe Commands\n\nSkyServe CLI (multi-region, multi-cloud serving).\n\n### `sky serve down`\n\nTeardown service(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_NAMES` — text\n- `--all`, `-a` — Tear down all services.\n- `--purge`, `-p` — Tear down services in failed status.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--replica-id` — Tear down a given replica\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky serve logs`\n\nTail or sync down logs of a service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this service.\n- `--load-balancer` — Show the load balancer logs of this service.\n- `--sync-down`, `-s` — Sync down logs to the local machine. Can be combined with --controller, --load-balancer, or a replica ID to narrow scope.\n- `--tail` — The number of lines to display from the end of the log file. Default is None, which means print all lines.\n- `SERVICE_NAME` — text\n- `REPLICA_IDS` — integer\n\n### `sky serve status`\n\nShow statuses of SkyServe services.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--endpoint` — Show service endpoint.\n- `SERVICE_NAMES` — text\n\n### `sky serve up`\n\nLaunch a SkyServe service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_YAML` — text\n- `--service-name`, `-n` — A service name. Unique for each service. If not provided, a unique name is autogenerated.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky serve update`\n\nUpdate a SkyServe service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_NAME` — text\n- `SERVICE_YAML` — text\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--mode` (default: `rolling`) — Update mode. If \"rolling\", SkyServe will update the service with rolling update. If \"blue_green\", SkyServe will update the service with blue-green update.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n## Storage Commands\n\nSkyPilot Storage CLI.\n\n### `sky storage delete`\n\nDelete storage objects.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--all`, `-a` — Delete all storage objects.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky storage ls`\n\nList storage objects managed by SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n\n## Volume Commands\n\nSkyPilot Volumes CLI.\n\n### `sky volumes apply`\n\nApply a volume.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--name`, `-n` — Volume name. Override the name defined in the YAML.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, or k8s/context-name.Examples: k8s, k8s/my-context, runpod/US/US-CA-2. Override the infra defined in the YAML.\n- `--type` — Volume type. Override the type defined in the YAML.\n- `--size` — Volume size. Override the size defined in the YAML.\n- `--use-existing`, `--no-use-existing` — Whether to use an existing volume. Override the use_existing defined in the YAML.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky volumes delete`\n\nDelete volumes.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--all`, `-a` — Delete all volumes.\n- `--purge`, `-p` — Forcibly delete the volume from the volumes table even if the deletion API fails.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky volumes ls`\n\nList volumes managed by SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--verbose`, `-v` — Show all information in full.\n- `--refresh`, `-r` — Refresh volume state from cloud APIs before listing. Without this flag, cached data is returned which is updated periodically by the background daemon.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## API Server Commands\n\nSkyPilot API server commands.\n\n### `sky api cancel`\n\nCancel a request running on SkyPilot API server.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `REQUEST_IDS` — text\n- `--all`, `-a` — Cancel all your requests.\n- `--all-users`, `-u` — Cancel all requests from all users. On a server with per-user auth, a non-admin cancels only their own.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky api info`\n\nShows the SkyPilot API server URL.\n\n**Options:**\n\n- `--output`, `-o` (default: `table`) — Output forma","agent/skills/skypilot/references/migrating-from-slurm.md":"# Migrating from Slurm Reference\n\nThis reference is for converting existing Slurm workloads into SkyPilot task\nYAMLs, and for answering \"how do I do X, which I used to do with Slurm?\".\n\nTwo distinct situations, which need different answers:\n\n- **SkyPilot on the user's existing Slurm cluster.** SkyPilot submits to it\n through the login node via `sbatch`. The cluster is unchanged. Most users\n start here — do not assume they want to leave Slurm.\n- **SkyPilot on Kubernetes or a cloud.** The workload moves off Slurm. The\n YAML is the same; the constraints are different (see section 8).\n\nAsk which one applies if it is not obvious. The conversion is nearly identical;\nthe difference shows up in lifecycle (section 7) and what is unsupported\n(section 8).\n\n---\n\n## 1. Conversion Procedure\n\nFollow this order. Do not skip step 5.\n\n1. **Read the whole `sbatch` script**, not just the `#SBATCH` block. The\n `srun` invocations and `module load` lines carry as much information as the\n directives.\n2. **Map the `#SBATCH` directives** to `resources` / `num_nodes` (section 2).\n3. **Map the body**: `module load` → `setup`, environment activation → `setup`,\n the actual work → `run` (section 6).\n4. **Translate `srun`** — this is the step most likely to be done wrong\n (section 5).\n5. **Validate**: `sky launch --dryrun `. This resolves resources and\n runs the optimizer without provisioning anything. Note there is **no**\n `--dryrun` on `sky jobs launch`, so dry-run against `sky launch` even when\n the final command will be `sky jobs launch`.\n6. **Tell the user which submit command to use**: `sky jobs launch` for batch\n work (an `sbatch` script), `sky launch` for an interactive allocation (an\n `salloc` session).\n\n### 1.1 Choosing the submit command\n\n| The user's script is | Use | Why |\n|---|---|---|\n| `sbatch` batch job | `sky jobs launch` | Allocation released when the job ends, and the job is recovered if the node fails |\n| `salloc` / `srun --pty` session | `sky launch` + `ssh ` | Long-lived allocation to work in |\n| `sbatch --array` | `sky jobs launch --num-jobs N` | Section 7.1 |\n| A chain with `--dependency` | One managed pipeline | Section 7.2 |\n\n---\n\n## 2. `#SBATCH` Directive Mapping\n\n| `#SBATCH` directive | SkyPilot YAML | Notes |\n|---|---|---|\n| `--nodes=2` | `num_nodes: 2` | Task-level field, **not** under `resources` |\n| `--gpus-per-node=8`, `--gres=gpu:h100:8` | `resources.accelerators: H100:8` | Emitted as `--gres=gpu::`. The name must match the cluster's GRES config |\n| `--cpus-per-task=32` | `resources.cpus: 32+` | `+` means \"at least\". Emitted as `--cpus-per-task` |\n| `--mem=256G` | `resources.memory: 256+` | In GB. Emitted as `--mem` |\n| `--partition=gpu` | `resources.infra: slurm//gpu` | Omit to let the optimizer choose a partition |\n| `--time=24:00:00` | `config.slurm.sbatch_options.time: \"24:00:00\"` | Defaults to the partition's `MaxTime` (or `DefaultTime` when `MaxTime` is `UNLIMITED`) if unset. **Not** `autostop` — see section 8 |\n| `--job-name=train` | `name: train` | |\n| `--output=`, `--error=` | *(nothing)* | SkyPilot manages job logs; see section 7.4 |\n| `--account`, `--qos`, `--reservation`, `--exclusive`, `--constraint`, ... | `config.slurm.sbatch_options.` | Section 3 |\n\n### 2.1 Worked example\n\nInput:\n\n```bash\n#!/bin/bash\n#SBATCH --job-name=train\n#SBATCH --nodes=2\n#SBATCH --gpus-per-node=8\n#SBATCH --cpus-per-task=32\n#SBATCH --mem=256G\n#SBATCH --partition=gpu\n#SBATCH --time=24:00:00\n#SBATCH --account=ai-research\n\nmodule load cuda/12.1\nsource ~/venv/bin/activate\n\nsrun python train.py --epochs 100\n```\n\nOutput:\n\n```yaml\n# train.yaml\nname: train\n\nnum_nodes: 2\n\nresources:\n infra: slurm/mycluster/gpu\n accelerators: H100:8\n cpus: 32+\n memory: 256+\n\nconfig:\n slurm:\n sbatch_options:\n time: \"24:00:00\"\n account: ai-research\n\nsetup: |\n pip install -r requirements.txt\n\nrun: |\n python train.py --epochs 100\n```\n\nSubmit with `sky jobs launch -n train train.yaml`.\n\nNote what happened to the `srun`: it disappeared. `run` already executes on\nevery node of the allocation, so wrapping the command in `srun` again would\nnest a job step inside SkyPilot's own. See section 5.\n\n---\n\n## 3. Directives SkyPilot Does Not Model\n\nAnything without a `resources` equivalent goes through\n`slurm.sbatch_options`, which is emitted verbatim as `#SBATCH` lines. Keys are\n`sbatch` long option names; underscores become hyphens; `true` emits a bare\nflag; `null` / `false` omit the line.\n\n```yaml\nconfig:\n slurm:\n sbatch_options:\n account: ai-research\n qos: high\n time: \"24:00:00\"\n exclusive: true\n constraint: ib\n```\n\nThe same block works in `~/.sky/config.yaml` under a top-level `slurm:` key to\napply fleet-wide, and can be scoped per cluster or per partition:\n`slurm.cluster_configs..sbatch_options` and\n`slurm.cluster_configs..partition_configs..sbatch_options`.\n\n**Do not put these in `sbatch_options`** — SkyPilot manages them and will drop\nthem with a warning:\n\n```\njob-name output error nodes wait-all-nodes\nno-requeue requeue cpus-per-task mem gres partition\n```\n\nIf a user asks for one of those, set it through `resources` / `name` instead.\n\n---\n\n## 4. Environment Variable Mapping\n\n| Slurm | SkyPilot | Notes |\n|---|---|---|\n| `$SLURM_JOB_NODELIST` | `$SKYPILOT_NODE_IPS` | Newline-separated IPs, **not** a Slurm hostlist expression. `$(echo \"$SKYPILOT_NODE_IPS\" \\| head -n1)` is the head node |\n| `$SLURM_NNODES` | `$SKYPILOT_NUM_NODES` | |\n| `$SLURM_NODEID`, `$SLURM_PROCID` | `$SKYPILOT_NODE_RANK` | `0` to `num_nodes-1` |\n| `$SLURM_GPUS_PER_NODE` | `$SKYPILOT_NUM_GPUS_PER_NODE` | |\n| `$SLURM_JOB_ID` | `$SKYPILOT_TASK_ID` | |\n| `$SLURM_ARRAY_TASK_ID` | `$SKYPILOT_JOB_RANK` | Only set under `--num-jobs`; section 7.1 |\n| `$SLURM_ARRAY_TASK_COUNT` | `$SKYPILOT_NUM_JOBS` | |\n\n**On Slurm specifically**, the *job-scoped* `SLURM_*` variables of the\nunderlying allocation are still present inside `run` — `SLURM_JOB_ID`,\n`SLURM_JOB_NODELIST`, `SLURM_GPUS_ON_NODE`, etc. Scripts that read them keep\nworking. The *step-scoped* variables from SkyPilot's own job step\n(`SLURM_CPUS_PER_TASK`, `SLURM_CPU_BIND*`, `SLURM_STEP_*`, `SLURM_PROCID`, ...)\nare deliberately unset before the user script runs, so a nested `srun` targets\nthe full allocation instead of inheriting SkyPilot's step shape.\n\nPrefer the `SKYPILOT_*` variables in generated YAML — they are what makes the\nsame task portable to Kubernetes and clouds. Use the `SLURM_*` ones only when\ntranslating a script that must keep working unchanged.\n\n---\n\n## 5. Translating `srun`\n\nThree cases. Pick by what the `srun` was *for*.\n\n### 5.1 `srun ` launching one process per node\n\nDrop the `srun`. `run` already runs on every node.\n\n```bash\n# Slurm\nsrun python preprocess.py\n```\n\n```yaml\nrun: |\n python preprocess.py\n```\n\n### 5.2 `srun` launching distributed training ranks\n\nUse the rank variables and the framework's own launcher. This is the portable\nform and should be the default for training workloads.\n\n```yaml\nnum_nodes: 2\n\nresources:\n accelerators: H100:8\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$MASTER_ADDR \\\n --master_port=8008 \\\n train.py\n```\n\n### 5.3 `srun` launching an MPI/PMIx program\n\nSome binaries must be launched by Slurm's own PMI (NCCL tests, anything\nMPI-linked, `--mpi=pmix`). Keep `srun`, but:\n\n- **Gate it on rank 0**, or every node will launch the whole job.\n- **Pass `--overlap`**, so the step can share the allocation with SkyPilot's\n own step.\n\n```yaml\nnum_nodes: 2\n\nresources:\n infra: slurm\n accelerators: H100:8\n\nrun: |\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n srun --overlap --mpi=pmix \\\n --ntasks-per-node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n ./my_mpi_program\n fi\n```\n\nThis form is Slurm-only by construction — it will not run on Kubernetes or a\ncloud. Say so when generating it.\n\n---\n\n## 6. Replacing the Module System\n\n`module load` reads a cluster-wide tree shared by all jobs. SkyPilot gives\neach job its own environment. In increasing order of isolation:\n\n```yaml\n# 1. setup commands — runs once per cluster, before the job\nsetup: |\n pip install -r requirements.txt\n```\n\n```yaml\n# 2. a container image — the closest thing to a reproducible module set.\n# On Slurm this needs the Pyxis SPANK plugin on the cluster.\nresources:\n image_id: docker:pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime\n```\n\n```yaml\n# 3. conda on the shared filesystem, if the user wants to keep their layout\nsetup: |\n conda create -n myenv python=3.10 -y\n conda activate myenv\n conda install pytorch pytorch-cuda=12.1 -c pytorch -c nvidia -y\n\nrun: |\n conda activate myenv\n python train.py\n```\n\n`module load` itself still works inside `setup` / `run` on a Slurm cluster,\nas long as the task is not running in a container. Keeping it is a legitimate\nfirst migration step — it just does not port to Kubernetes or clouds.\n\n---\n\n## 7. Patterns Without a One-to-One Field\n\n### 7.1 Job arrays → `--num-jobs`\n\n`sbatch --array=0-99` becomes:\n\n```bash\nsky jobs launch --num-jobs 100 -y -d task.yaml\n```\n\nEach job gets `$SKYPILOT_JOB_RANK` (the index) and `$SKYPILOT_NUM_JOBS`:\n\n```yaml\nrun: |\n python train.py --shard $SKYPILOT_JOB_RANK --num-shards $SKYPILOT_NUM_JOBS\n```\n\nEach job provisions its own allocation. To reuse workers across submissions\ninstead — closer to how a long-lived Slurm allocation behaves, and much\ncheaper when `setup` is expensive — submit to a pool (note `pool.yaml` is a\npool spec: it needs a `pool:` section, not a bare task YAML):\n\n```bash\nsky jobs pool apply -p mypool pool.yaml\nsky jobs launch -p mypool --num-jobs 100 task.yaml\n```\n\nFor sweeps over *named* parameters rather than an index, loop with `--env`:\n\n```bash\nfor lr in 0.001 0.01 0.1; do\n sky jobs launch --env LR=$lr -y -d task.yaml\ndone\n```\n\n### 7.2 Job dependencies → a managed pipeline\n\n`sbatch --dependency=afterok:` becomes a sequence of tasks in one\nmanaged job, separated by `---`. Each task can request different resources.\n\n```yaml\nname: train-then-eval\n\n---\n\nname: train\nresources:\n accelerators: H100:8\nrun: python train.py\n\n---\n\nname: eval\nresources:\n accelerators: L4:1\nrun: python eval.py\n```\n\nFor tasks that should run *in parallel* rather than in sequence, use a job\ngroup instead of a pipeline. A job group is the same multi-document layout,\nbut the header must set `execution: parallel` — without it the file is a\n*serial* pipeline:\n\n```yaml\nname: train-and-eval\nexecution: parallel\n\n---\n\nname: train\nresources:\n accelerators: H100:8\nrun: python train.py\n\n---\n\nname: eval\nresources:\n accelerators: L4:1\nrun: python eval.py\n```\n\n### 7.3 Login node workflow\n\nThere is no login node in the SkyPilot workflow. Submit from the user's laptop\nor CI; SkyPilot makes the SSH hop to the login node itself. Sync code with\n`workdir` (a local path or a git URL) rather than editing on NFS. For\ninteractive work, `ssh ` lands on the *allocated compute node*, not a\nshared login node.\n\n### 7.4 Job output files\n\nSlurm writes `slurm-.out`. SkyPilot streams logs instead:\n\n```bash\nsky jobs logs # managed job\nsky logs # latest job on a cluster\nsky logs 2 # a specific job\n```\n\nDo not translate `--output` / `--error` into anything. If the script writes\nits own result files, point them at the shared filesystem or a bucket.\n\n---\n\n## 8. Constraints When Running on Slurm\n\nDo not generate YAML that uses these on a Slurm target — they will fail or be\nrejected:\n\n| Feature | Status on Slurm | What to do instead |\n|---|---|---|\n| `autostop` / `sky stop` | Unsupported | `sky down` explicitly; bound wall-clock with `sbatch_options.time` |\n| `ports:` / `sky serve` | Unsupported | Run serving on Kubernetes or a cloud |\n| `use_spot: true` | Unsupported | Omit; there is no spot tier |\n| Jobs/serve controllers | Not hosted on Slurm | With a remote API server, consolidation mode runs the controller inside the API server, so `sky jobs launch` works against a Slurm-only fleet. With a local API server the controller needs Kubernetes or a cloud |\n| `image_id: docker:...` | Needs Pyxis on the cluster | Fall back to `setup` commands |\n| Bucket mounting (`mode: MOUNT`) | Needs FUSE on compute nodes | Use the shared filesystem, or `mode: COPY` |\n\nBecause autostop does not exist on Slurm, an idle `sky launch` cluster holds\nits allocation until `sky down`. Prefer `sky jobs launch` for anything\nbatch-like so the allocation is released automatically.\n\n### 8.1 If the target is Kubernetes instead\n\nThe assumption that breaks first is shared storage: Kubernetes does not mount\nhome directories. A Slurm script that reads `~/data` or writes `~/checkpoints`\nneeds one of:\n\n- an NFS mount via `config.kubernetes.pod_config` (if the nodes can reach the\n existing NFS server),\n- a `ReadWriteMany` PVC through SkyPilot volumes,\n- a bucket under `file_mounts` with `mode: MOUNT`,\n- `workdir` for code only.\n\nPartitions map to Kubernetes contexts (`infra: kubernetes/`), and\nSlurm's fairshare/QoS has no built-in equivalent — that is Kueue or priority\nclasses.\n\n---\n\n## 9. Validation Checklist\n\nBefore handing a converted YAML to the user:\n\n- [ ] `sky launch --dryrun ` resolves without error.\n- [ ] `num_nodes` is at task level, not under `resources`.\n- [ ] No `autostop` / `ports` / `use_spot` if the target is Slurm.\n- [ ] Every `srun` is accounted for by section 5 — dropped, replaced with\n rank variables, or kept with `--overlap` and gated on rank 0.\n- [ ] GPU name matches the cluster's GRES names (check `sky gpus list --infra\n slurm`).\n- [ ] Protected `sbatch` options are not in `sbatch_options` (section 3).\n- [ ] The user is told whether to run `sky launch` or `sky jobs launch`.\n","agent/skills/skypilot/references/troubleshooting.md":"# References Skill\n\nPath: `agent/skills/skypilot/references/troubleshooting.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/troubleshooting.md)","agent/skills/skypilot/references/yaml-spec.md":"# References Skill\n\nPath: `agent/skills/skypilot/references/yaml-spec.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/yaml-spec.md)","docs/source/examples/agents/autonomous-code-optimization.md":"# Subagent: autonomous-code-optimization\n\nPath: `docs/source/examples/agents/autonomous-code-optimization.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autonomous-code-optimization.md)","docs/source/examples/agents/autoresearch.md":"# Subagent: autoresearch\n\nPath: `docs/source/examples/agents/autoresearch.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autoresearch.md)"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# CLAUDE.md - SkyPilot Development Guide\n\nThis document provides guidance for AI assistants working with the SkyPilot codebase.\n\n## Project Overview\n\nSkyPilot is a system to run, manage, and scale AI workloads on any AI infrastructure. It provides a unified interface across 25+ cloud providers (AWS, GCP, Azure, Kubernetes, Slurm, and many others), enabling users to launch compute resources, run jobs, and serve models without vendor lock-in.\n\n**Key capabilities:**\n- **Clusters**: Launch and manage compute clusters across clouds and Kubernetes\n- **Managed jobs**: Run jobs with automatic recovery from preemptions and failures\n- **Multi-cloud and multi-Kubernetes**: Unified interface across 25+ clouds and multiple K8s clusters\n- Cost optimization and GPU availability maximization\n\n## Repository Structure\n\n```\nskypilot/\n├── sky/ # Main source code\n│ ├── __init__.py # Package exports and version\n│ ├── core.py # Core orchestration logic\n│ ├── task.py # Task definition and YAML parsing\n│ ├── resources.py # Cloud resource specifications\n│ ├── optimizer.py # Resource allocation optimizer\n│ ├── execution.py # Task execution pipeline\n│ ├── exceptions.py # Custom exceptions\n│ ├── skypilot_config.py # Configuration management\n│ ├── clouds/ # Cloud provider abstractions (25+ providers)\n│ ├── backends/ # Execution backends (Ray-based VM, Docker)\n│ ├── provision/ # Cloud resource provisioning\n│ ├── jobs/ # Managed job lifecycle\n│ ├── serve/ # Model serving (SkyServe)\n│ ├── skylet/ # On-cluster execution agent\n│ ├── client/ # SDK and CLI\n│ ├── server/ # API server and dashboard backend\n│ ├── dashboard/ # Web UI (Next.js)\n│ ├── utils/ # 50+ utility modules\n│ ├── catalog/ # Cloud pricing and instance catalogs\n│ ├── data/ # Storage and data handling\n│ ├── adaptors/ # Cloud-specific metadata adapters\n│ └── schemas/ # Protobuf definitions and API schemas\n├── tests/ # Test suite\n│ ├── unit_tests/ # Unit tests with subdirectories per module\n│ ├── smoke_tests/ # Quick validation tests\n│ ├── integration_tests/ # End-to-end tests\n│ ├── kubernetes/ # K8s-specific tests\n│ └── conftest.py # Pytest fixtures\n├── examples/ # 50+ usage examples\n├── llm/ # 45+ LLM training/serving examples\n├── docs/ # Sphinx documentation\n├── charts/ # Helm charts for K8s deployment\n└── format.sh # Code formatting script\n```\n\n## Development Setup\n\n### Environment Setup\n\n```bash\n# Create virtual environment with uv (Python 3.8-3.11 supported)\n# --seed is required to ensure pip is installed (needed for building wheels)\nuv venv --seed --python 3.11\nsource .venv/bin/activate\n\n# Install in editable mode with all cloud support\nuv pip install -e \".[all]\"\n# Or specific clouds only:\n# uv pip install -e \".[aws,gcp,kubernetes]\"\n\n# Install development dependencies\nuv pip install -r requirements-dev.txt\n\n# Optional: Install pre-commit hooks\nuv pip install pre-commit\npre-commit install\n```\n\n### Environment Variables\n\n```bash\nexport SKYPILOT_DEV=1 # Enable development mode\nexport SKYPILOT_DEBUG=1 # Enable debug logging\n```\n\n## Code Formatting and Linting\n\n**Always run `format.sh` before committing:**\n\n```bash\nbash format.sh # Format changed files (vs origin/master)\nbash format.sh --all # Format entire codebase\nbash format.sh --files path/to/file.py # Format specific files\n```\n\nThe script runs:\n1. **YAPF** - Google style for all Python code\n2. **isort** - Import sorting (Google profile)\n3. **mypy** - Type checking\n4. **pylint** - Linting with custom rules\n\n### Tool Versions (must match exactly)\n\nFrom `requirements-dev.txt`:\n- yapf==0.32.0\n- pylint==2.14.5\n- mypy==1.19.1\n- isort==5.12.0\n- pylint-quotes==0.2.3\n\n### Excluded from Formatting\n\n- `sky/schemas/generated/` - Auto-generated protobuf files\n- `build/` - Build artifacts\n\n## Testing\n\n### Running Tests\n\n```bash\n# Unit tests (fast, no cloud resources)\npytest tests/unit_tests/\n\n# Specific test file\npytest tests/unit_tests/test_resources.py\n```\n\n### CI Tests via PR Comments\n\nTrigger CI tests on pull requests using comments:\n- `/quicktest-core` - Run quick core tests\n- `/smoke-test` - Run smoke tests (launches cloud clusters)\n- `/smoke-test --kubernetes --postgres` - Test with PostgreSQL backend on Kubernetes\n- `/smoke-test --kubernetes --remote-server --postgres` - Test remote API server with PostgreSQL\n\n### Test Configuration\n\nFrom `pyproject.toml`:\n- Uses pytest-xdist with 16 parallel workers\n- Environment: `SKYPILOT_DEBUG=1`, `SKYPILOT_DEV=1`\n- Buildkite integration for CI\n\n### Checking Buildkite CI Status\n\nTo check CI test results for a PR:\n\n1. **Get Buildkite URL from GitHub**: Check the PR's commit status checks to find the Buildkite build URL (may take a moment to appear after triggering)\n2. **Fetch build logs via Buildkite API**: Use the Buildkite API to retrieve detailed logs and test results\n\n```bash\n# Example: Get build details (requires BUILDKITE_TOKEN)\ncurl -H \"Authorization: Bearer $BUILDKITE_TOKEN\" \\\n \"https://api.buildkite.com/v2/organizations/skypilot/pipelines/skypilot/builds/\"\n```\n\n## Code Style Guidelines\n\n### General Principles\n\n- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)\n- Use f-strings over `.format()` for readability\n- Use `class MyClass:` not `class MyClass(object):`\n- Use `abc` module for abstract classes\n- Use Python typing with `if typing.TYPE_CHECKING:` for type-only imports\n- Always place imports at the top of the file, not inside function definitions. If this causes a circular dependency, try to resolve it (e.g., restructure modules or use `if typing.TYPE_CHECKING:`). Only as a last resort, place the import inside the function with a comment explaining why. Import performance is not a valid reason for in-function imports; use `LazyImport` instead (see below).\n\n### TODOs and FIXMEs\n\nAlways include author attribution:\n```python\n# TODO(username): Description of what needs to be done\n# FIXME(username): Description of what needs fixing\n```\n\n### Exceptions vs Assertions\n\n- Use exceptions for errors (better error messages)\n- Use `assert` only for debugging/proof-checking\n\n### Lazy Imports\n\nFor modules with significant import time (>100ms) that are imported during `import sky`, use `LazyImport` from `sky/adaptors/common.py`:\n\n```python\nfrom sky.adaptors.common import LazyImport\nheavy_module = LazyImport('heavy_module')\n```\n\nMeasure import time with:\n```bash\npython -X importtime -c \"import sky\" 2> import.log\ntuna import.log # pip install tuna\n```\n\n## Architecture Patterns\n\n### Cloud Provider Abstraction\n\nAll cloud providers inherit from `sky.clouds.Cloud` (in `sky/clouds/cloud.py`). Cloud objects are **lightweight and stateless**. Key design principles:\n\n- Methods should be inexpensive to call\n- Don't store heavy state in cloud objects\n- Cache cloud-specific queries in `sky/clouds/utils/` modules\n- Each cloud implements feature flags via `CloudImplementationFeatures`\n\nAdding a new cloud provider:\n1. Create `sky/clouds/.py` inheriting from `Cloud`\n2. Implement required abstract methods\n3. Add provisioning logic in `sky/provision//`\n4. Register in `sky/clouds/__init__.py`\n\n### Client-Server Architecture\n\nSkyPilot uses a client-server model with API versioning:\n\n- Client (SDK/CLI) in `sky/client/`\n- Server in `sky/server/`\n- API version in `sky/server/constants.py`\n\n**Backward Compatibility Rules (from v0.10.0+):**\n- Changes must be backward compatible\n- Bump `API_VERSION` when introducing API changes\n- Use `@versions.minimal_api_version(N)` decorator for new SDK methods\n- Handle version differences with `versions.get_remote_api_version()`\n\n### Protobuf Regeneration\n\nWhen modifying `.proto` files in `sky/schemas/proto/`:\n\n```bash\npython -m grpc_tools.protoc \\\n --proto_path=sky/schemas/generated=sky/schemas/proto \\\n --python_out=. \\\n --grpc_python_out=. \\\n --pyi_out=. \\\n sky/schemas/proto/*.proto\n```\n\n### Dependency Management\n\nDependencies are defined in `sky/setup_files/dependencies.py`:\n\n- **Core dependencies**: Listed in `install_requires`\n- **Cloud-specific**: Defined in `extras_require` (e.g., `aws`, `gcp`, `kubernetes`)\n- **Development**: Listed in `requirements-dev.txt`\n\nWhen updating dependencies:\n\n1. Check version constraints carefully - some packages have breaking changes\n2. Consider Python version compatibility (3.8-3.11)\n3. Test with both minimum and latest allowed versions\n4. Document version constraints with comments explaining why\n\n## Key Modules Reference\n\n| Module | Purpose |\n|--------|---------|\n| `sky/core.py` | Core orchestration (launch, exec, stop, down) |\n| `sky/task.py` | Task YAML parsing and specification |\n| `sky/resources.py` | Resource requirements (GPUs, memory, disk) |\n| `sky/optimizer.py` | Cloud selection and cost optimization |\n| `sky/client/sdk.py` | Python SDK implementation |\n| `sky/client/cli/` | CLI commands |\n| `sky/backends/cloud_vm_ray_backend.py` | Main execution backend for clusters |\n| `sky/provision/provisioner.py` | Resource provisioning |\n| `sky/jobs/` | Managed jobs with recovery and scheduling |\n\n## Pull Request Guidelines\n\n1. **Branch from master**, create descriptive branch name\n2. **Run `format.sh`** before committing\n3. **Add tests** for core system changes\n4. **Run smoke tests** for significant changes\n5. **Include `Tested:` section** in PR description with test plan\n6. **Delete branch** after merging\n\n### PR Description Format\n\nPRs should include:\n- **Summary**: Brief description of changes (1-3 bullet points)\n- **Test plan**: How the changes were tested (commands run, manual verification steps)\n\n**Important**: Always generate a manual test plan describing how to verify the changes work correctly. Include specific commands, expected outputs, or UI verification steps. Whenever possible, add unit tests and smoke tests for the changes.\n\n### Commit Message Format\n\nUse the `[Area] Description` format:\n- `[Core]` for core system changes\n- `[CLI]` for CLI changes\n- `[API]` for API server changes\n- `[Docs]` for documentation\n- `[AWS]`, `[GCP]`, `[Azure]`, `[Kubernetes]` for cloud-specific changes\n- `[Jobs]` for managed jobs\n- `[Dashboard]` for web UI changes\n- `[Serve]` for model serving (SkyServe)\n- `[Test]` for testing changes\n- `[CI]` for CI/CD changes\n\nExamples:\n- `[Core] Fix cluster status refresh logic`\n- `[AWS] Add support for new instance types`\n- `[Docs] Update installation guide`\n\n## API Server Testing\n\n### Local API Server (Recommended for Development)\n\n```bash\n# Always restart API server after code changes to pick up changes\nsky api stop\nsky api start\n\n# Verify server is running\nsky api status\n```\n\n### Dashboard Development\n\n**For local API server development**, rebuild the dashboard before restarting:\n\n```bash\n# Install dependencies (first time or after package.json changes)\nnpm --prefix sky/dashboard install\n\n# Rebuild the dashboard\nnpm --prefix sky/dashboard run build\n\n# Then restart the API server\nsky api stop\nsky api start\n```\n\n**For remote API server (Docker/Kubernetes)**, the Dockerfile automatically builds the dashboard - no manual build needed before `docker build`.\n\nThe dashboard is a Next.js application. For development with hot reloading:\n\n```bash\n# Run dashboard in development mode (separate from API server)\ncd sky/dashboard\nnpm run dev\n```\n\n### Mocking Remote API Server Locally\n\nTo test remote API server behavior locally:\n\n```bash\n# Start local API server (runs on port 46580 by default)\nsky api stop\nsky api start\n\n# Forward to a different port to simulate remote server\nsocat TCP-LISTEN:46590,fork TCP:127.0.0.1:46580 &\n\n# Connect to the forwarded port as if it were a remote server\nsky api login -e http://127.0.0.1:46590\n```\n\n### Remote API Server (Kubernetes Deployment)\n\nFor testing with a remote API server on Kubernetes:\n\n```bash\n# Build local changes\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\nhelm dependency build ./charts/skypilot\n\n# Build Docker image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy (NEW installation)\nNAMESPACE=skypilot\nRELEASE_NAME=skypilot\nhelm upgrade --install $RELEASE_NAME ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n#### Upgrading Existing Deployments\n\n**CRITICAL:** Always use `--reuse-values` to preserve database/credential config:\n\n```bash\n# Upgrade existing deployment (keeps PostgreSQL, auth, etc.)\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n\n# Check current values / rollback if needed\nhelm get values skypilot -n skypilot\nhelm rollback skypilot -n skypilot\n```\n\n#### PostgreSQL Backend\n\n```bash\n# Create connection secret\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/db\"\n\n# Deploy with PostgreSQL\nhelm upgrade --install skypilot ./charts/skypilot -n skypilot \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\n## Critical Code Paths (Handle with Care)\n\nThe following modules contain complex, stateful logic that requires careful review when modifying:\n\n**Managed Jobs Recovery:**\n- `sky/jobs/controller.py` - Job lifecycle management, state transitions, async coordination\n- `sky/jobs/recovery_strategy.py` - Preemption recovery, retry logic for managed jobs\n- `sky/backends/cloud_vm_ray_backend.py` - Execution backend with complex state handling\n\nThese modules handle edge cases like preemption during job submission, controller failures mid-recovery, and race conditions between concurrent operations. Changes can have subtle effects on job reliability.\n\n**API Server Performance & Robustness:**\n- `sky/server/` - API server with memory efficiency, low latency requirements\n- `sky/backends/backend_utils.py` - Cluster status caching, network resilience, SSH handling\n\nThese modules are optimized for performance. Be cautious about adding blocking calls, memory-heavy operations, or changing caching behavior.\n\n**CLI/SDK Interface Design:**\n- `sky/client/cli/` - Command-line interface\n- `sky/client/sdk.py` - Python SDK\n\nBe cautious about adding new interfaces or changing existing UX significantly. Keep the interface clean and minimal for both CLI and SDK.\n\n**Backward Compatibility:**\n- Test with old server + new client, and old client + new server\n- Ensure existing clusters and jobs continue working after server upgrades\n- `/quicktest-core` tests some backward compatibility scenarios\n- See `docs/source/developers/CONTRIBUTING.md` for API versioning guidelines\n\n## Common Pitfalls\n\n1. **Always restart API server after code changes** - Run `sky api stop; sky api start` to pick up changes\n2. **Don't modify `sky/schemas/generated/`** - These are auto-generated\n3. **Match formatter versions exactly** - Version mismatches cause CI failures\n4. **Consider import time** - Heavy imports slow down CLI responsiveness\n5. **API versioning** - Always maintain backward compatibility\n\n## Useful Commands\n\n```bash\n# Profile CLI performance\nuv pip install py-spy\npy-spy record -t -o sky.svg -- python -m sky.cli status\n\n# Check cloud credentials\nsky check\n\n# View cluster status\nsky status\n\n# Launch a cluster\nsky launch \n\n# Execute a command on the cluster\nsky exec -- bash\n\n# SSH into the cluster\nssh \n\n# Launch a managed job\nsky jobs launch \n```\n\n## Documentation\n\n- **User docs**: https://docs.skypilot.co/\n- **Source**: `docs/source/` (Sphinx)\n- **Build docs**: See `.readthedocs.yml`\n\n## Additional Resources\n\n- **Full contributing guide**: `docs/source/developers/CONTRIBUTING.md`\n- **User docs**: https://docs.skypilot.co/\n- **GitHub Issues**: https://github.com/skypilot-org/skypilot/issues\n- **Discussions**: https://github.com/skypilot-org/skypilot/discussions\n- **Slack**: http://slack.skypilot.co\n","isInternal":false,"tokens":3889,"sizeBytes":16618},{"name":"evals.json","path":"agent/skills/skypilot/evals/evals.json","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/evals/evals.json","title":"Evals Skill","category":"anthropic-skill","format":"json","content":"{\n \"skill_name\": \"skypilot\",\n \"description\": \"Evaluates the 'Before You Start (Agent Bootstrap)', 'Converting a Slurm Workload' and 'Investigating a Managed Job' section behavior\",\n \"evals\": [\n {\n \"id\": 1,\n \"name\": \"happy-path-server-running\",\n \"prompt\": \"I want to launch an 8xH100 cluster for training a LLaMA model. My SkyPilot is already set up.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, sees the server is running (table output with request data), skips rest of bootstrap (no `sky check`, no install, no `sky api start`), and proceeds directly to help write the task YAML or run `sky launch`.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` (or `sky api status -l 1`) as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT run `sky check` when the server is already running\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent does NOT run `sky api start` or `sky api login` when the server is already running\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent proceeds to help with the user's actual task (launching H100 cluster) without unnecessary bootstrap steps\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 2,\n \"name\": \"server-not-running-local-setup\",\n \"prompt\": \"I just installed SkyPilot and want to try launching a GPU instance on AWS. I don't have a remote API server.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, detects 'SkyPilot API server is not running.', asks about local vs remote, user indicates local, agent runs `sky api start`, re-checks with `sky api status -l 1`, then runs `sky check` for credentials since this is a fresh setup.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent asks the user whether they have a remote server or want to start locally\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent runs `sky api start` after user confirms local setup\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent re-runs `sky api status` after starting the server to confirm it's reachable\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent runs `sky check` since this is a fresh setup\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 3,\n \"name\": \"server-not-running-remote-connect\",\n \"prompt\": \"I need to connect to our team's SkyPilot API server at https://skypilot.internal.company.com and launch a fine-tuning job.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, detects server not running, recognizes user already provided a remote URL, runs `sky api login -e https://skypilot.internal.company.com` without asking unnecessary questions, re-checks status, then proceeds to help with the fine-tuning job.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent uses `sky api login -e https://skypilot.internal.company.com` (not `sky api start`) since the user provided a remote endpoint\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT ask the user for the server URL again since they already provided it\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent re-runs `sky api status` after connecting to confirm it's reachable\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 4,\n \"name\": \"slurm-sbatch-time-not-autostop\",\n \"prompt\": \"Convert this to SkyPilot for our Slurm cluster:\\n#!/bin/bash\\n#SBATCH --nodes=2\\n#SBATCH --gpus-per-node=8\\n#SBATCH --time=24:00:00\\n#SBATCH --account=ai-research\\nsrun python train.py\",\n \"expected_output\": \"Agent produces a task YAML with num_nodes: 2, resources.accelerators: H100:8 (or asks for the GRES name), and puts both --time and --account under config.slurm.sbatch_options. The bare `srun` is dropped because `run` already executes on every node. Agent recommends `sky jobs launch` for this batch script.\",\n \"assertions\": [\n {\n \"text\": \"Agent maps --time to config.slurm.sbatch_options.time\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT map --time to autostop, which is unsupported on Slurm\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent maps --account to config.slurm.sbatch_options.account\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent puts num_nodes at task level, not under resources\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT wrap the run command in `srun` again\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent recommends `sky jobs launch` (not `sky launch`) for this batch script\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 5,\n \"name\": \"slurm-job-array-to-num-jobs\",\n \"prompt\": \"On Slurm I run `sbatch --array=0-99 shard.sh` and the script reads $SLURM_ARRAY_TASK_ID to pick its shard. What's the SkyPilot equivalent?\",\n \"expected_output\": \"Agent answers `sky jobs launch --num-jobs 100` and maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK (with $SKYPILOT_NUM_JOBS for the count). May also mention pools for reusing workers across submissions.\",\n \"assertions\": [\n {\n \"text\": \"Agent uses `sky jobs launch --num-jobs 100`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT propose a shell for-loop as the primary answer\",\n \"type\": \"negative\"\n }\n ]\n },\n {\n \"id\": 6,\n \"name\": \"why-is-my-managed-job-pending\",\n \"prompt\": \"My managed job 42 has been PENDING for 20 minutes on our Slurm cluster. Why? SkyPilot is already set up.\",\n \"expected_output\": \"Agent reads `details` from `sky jobs queue -v -o json` and finds the Slurm pending reason with its partition (e.g. `Launching (pending: QOSGrpGRES; partition: h200)`), explains that category (quota vs no free capacity vs dependency vs held) and what to do about it. Does not claim the job is stuck in SkyPilot's own queue when the reason is Slurm's.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky jobs queue -v -o json` (`-v` is required for the `details` field) and reads the job's details\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent reads the Slurm pending reason and its partition from `details`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent explains the pending reason in plain terms (quota, capacity, dependency, or held) with a next step\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT poll with `sleep` + `sky jobs queue`\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent does NOT tear down the jobs controller as a first step\",\n \"type\": \"negative\"\n }\n ]\n },\n {\n \"id\": 7,\n \"name\": \"what-happened-to-my-job\",\n \"prompt\": \"Managed job 17 keeps flipping between RUNNING and RECOVERING. Show me what happened.\",\n \"expected_output\": \"Agent reads `sky jobs queue -v -o json` for `details` / `failure_reason` / `recovery_count`, then pulls `sky jobs logs 17 --controller --no-follow` for the recovery history, and points out that a repeated identical reason means a persistent cause the controller cannot fix by retrying.\",\n \"assertions\": [\n {\n \"text\": \"Agent reads `details`, `failure_reason` and `recovery_count` from `sky jobs queue -v -o json`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent identifies the recurring RECOVERING reason from the events\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent uses `--tail` or `--no-follow` when fetching logs rather than streaming the full log\",\n \"type\": \"required_action\"\n }\n ]\n }\n ]\n}\n","frontmatter":{"skill_name":"skypilot","description":"Evaluates the 'Before You Start (Agent Bootstrap)', 'Converting a Slurm Workload' and 'Investigating a Managed Job' section behavior","evals":[{"id":1,"name":"happy-path-server-running","prompt":"I want to launch an 8xH100 cluster for training a LLaMA model. My SkyPilot is already set up.","expected_output":"Agent runs `sky api status -l 1`, sees the server is running (table output with request data), skips rest of bootstrap (no `sky check`, no install, no `sky api start`), and proceeds directly to help write the task YAML or run `sky launch`.","assertions":[{"text":"Agent runs `sky api status` (or `sky api status -l 1`) as the first SkyPilot command","type":"required_action"},{"text":"Agent does NOT run `sky check` when the server is already running","type":"negative"},{"text":"Agent does NOT run `sky api start` or `sky api login` when the server is already running","type":"negative"},{"text":"Agent proceeds to help with the user's actual task (launching H100 cluster) without unnecessary bootstrap steps","type":"required_action"}]},{"id":2,"name":"server-not-running-local-setup","prompt":"I just installed SkyPilot and want to try launching a GPU instance on AWS. I don't have a remote API server.","expected_output":"Agent runs `sky api status -l 1`, detects 'SkyPilot API server is not running.', asks about local vs remote, user indicates local, agent runs `sky api start`, re-checks with `sky api status -l 1`, then runs `sky check` for credentials since this is a fresh setup.","assertions":[{"text":"Agent runs `sky api status` as the first SkyPilot command","type":"required_action"},{"text":"Agent asks the user whether they have a remote server or want to start locally","type":"required_action"},{"text":"Agent runs `sky api start` after user confirms local setup","type":"required_action"},{"text":"Agent re-runs `sky api status` after starting the server to confirm it's reachable","type":"required_action"},{"text":"Agent runs `sky check` since this is a fresh setup","type":"required_action"}]},{"id":3,"name":"server-not-running-remote-connect","prompt":"I need to connect to our team's SkyPilot API server at https://skypilot.internal.company.com and launch a fine-tuning job.","expected_output":"Agent runs `sky api status -l 1`, detects server not running, recognizes user already provided a remote URL, runs `sky api login -e https://skypilot.internal.company.com` without asking unnecessary questions, re-checks status, then proceeds to help with the fine-tuning job.","assertions":[{"text":"Agent runs `sky api status` as the first SkyPilot command","type":"required_action"},{"text":"Agent uses `sky api login -e https://skypilot.internal.company.com` (not `sky api start`) since the user provided a remote endpoint","type":"required_action"},{"text":"Agent does NOT ask the user for the server URL again since they already provided it","type":"negative"},{"text":"Agent re-runs `sky api status` after connecting to confirm it's reachable","type":"required_action"}]},{"id":4,"name":"slurm-sbatch-time-not-autostop","prompt":"Convert this to SkyPilot for our Slurm cluster:\n#!/bin/bash\n#SBATCH --nodes=2\n#SBATCH --gpus-per-node=8\n#SBATCH --time=24:00:00\n#SBATCH --account=ai-research\nsrun python train.py","expected_output":"Agent produces a task YAML with num_nodes: 2, resources.accelerators: H100:8 (or asks for the GRES name), and puts both --time and --account under config.slurm.sbatch_options. The bare `srun` is dropped because `run` already executes on every node. Agent recommends `sky jobs launch` for this batch script.","assertions":[{"text":"Agent maps --time to config.slurm.sbatch_options.time","type":"required_action"},{"text":"Agent does NOT map --time to autostop, which is unsupported on Slurm","type":"negative"},{"text":"Agent maps --account to config.slurm.sbatch_options.account","type":"required_action"},{"text":"Agent puts num_nodes at task level, not under resources","type":"required_action"},{"text":"Agent does NOT wrap the run command in `srun` again","type":"negative"},{"text":"Agent recommends `sky jobs launch` (not `sky launch`) for this batch script","type":"required_action"}]},{"id":5,"name":"slurm-job-array-to-num-jobs","prompt":"On Slurm I run `sbatch --array=0-99 shard.sh` and the script reads $SLURM_ARRAY_TASK_ID to pick its shard. What's the SkyPilot equivalent?","expected_output":"Agent answers `sky jobs launch --num-jobs 100` and maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK (with $SKYPILOT_NUM_JOBS for the count). May also mention pools for reusing workers across submissions.","assertions":[{"text":"Agent uses `sky jobs launch --num-jobs 100`","type":"required_action"},{"text":"Agent maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK","type":"required_action"},{"text":"Agent does NOT propose a shell for-loop as the primary answer","type":"negative"}]},{"id":6,"name":"why-is-my-managed-job-pending","prompt":"My managed job 42 has been PENDING for 20 minutes on our Slurm cluster. Why? SkyPilot is already set up.","expected_output":"Agent reads `details` from `sky jobs queue -v -o json` and finds the Slurm pending reason with its partition (e.g. `Launching (pending: QOSGrpGRES; partition: h200)`), explains that category (quota vs no free capacity vs dependency vs held) and what to do about it. Does not claim the job is stuck in SkyPilot's own queue when the reason is Slurm's.","assertions":[{"text":"Agent runs `sky jobs queue -v -o json` (`-v` is required for the `details` field) and reads the job's details","type":"required_action"},{"text":"Agent reads the Slurm pending reason and its partition from `details`","type":"required_action"},{"text":"Agent explains the pending reason in plain terms (quota, capacity, dependency, or held) with a next step","type":"required_action"},{"text":"Agent does NOT poll with `sleep` + `sky jobs queue`","type":"negative"},{"text":"Agent does NOT tear down the jobs controller as a first step","type":"negative"}]},{"id":7,"name":"what-happened-to-my-job","prompt":"Managed job 17 keeps flipping between RUNNING and RECOVERING. Show me what happened.","expected_output":"Agent reads `sky jobs queue -v -o json` for `details` / `failure_reason` / `recovery_count`, then pulls `sky jobs logs 17 --controller --no-follow` for the recovery history, and points out that a repeated identical reason means a persistent cause the controller cannot fix by retrying.","assertions":[{"text":"Agent reads `details`, `failure_reason` and `recovery_count` from `sky jobs queue -v -o json`","type":"required_action"},{"text":"Agent identifies the recurring RECOVERING reason from the events","type":"required_action"},{"text":"Agent uses `--tail` or `--no-follow` when fetching logs rather than streaming the full log","type":"required_action"}]}]},"isInternal":false,"tokens":2095,"sizeBytes":8506},{"name":"advanced-patterns.md","path":"agent/skills/skypilot/references/advanced-patterns.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/advanced-patterns.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# SkyPilot Advanced Patterns Reference\n\nThis reference covers advanced SkyPilot patterns for multi-cloud strategies,\ndistributed training, managed jobs, serving, Kubernetes integration, API server\nsetup, and cost optimization.\n\n---\n\n## 1. Multi-Cloud Strategies\n\nSkyPilot can automatically select the cheapest cloud/region/zone for a\nworkload, or follow explicit priority ordering. The key mechanisms are:\n- `any_of`: try all listed resource options, optimize for cost (unordered)\n- `ordered`: try resource options in strict priority order (first match wins)\n- `infra:` field with wildcard patterns for flexible cloud/region/zone targeting\n\n### 1.1 any_of -- Cheapest Available GPU\n\nUse `any_of` when you have multiple acceptable resource configurations and\nwant SkyPilot to pick the cheapest one available across all of them.\n\nNote: If all options use the same accelerator, you can simply specify\n`accelerators:` without `any_of` — SkyPilot will automatically search\nacross all enabled clouds. Use `any_of` when mixing different GPU types:\n\n```yaml\n# cheapest-gpu.yaml — mix different GPU types, SkyPilot picks cheapest\nname: cheapest-gpu-training\n\nresources:\n any_of:\n - accelerators: H100:8\n - accelerators: A100-80GB:8\n - accelerators: A100:8\n\nrun: |\n torchrun --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE train.py\n```\n\nYou can also mix different GPU types across clouds:\n\n```yaml\n# mixed-gpus.yaml\nname: mixed-gpu-training\n\nresources:\n any_of:\n - infra: aws\n accelerators: A10g:4\n - infra: gcp\n accelerators: L4:4\n - infra: azure\n accelerators: A100:1\n - infra: lambda\n accelerators: A100:1\n\nrun: |\n python train.py\n```\n\nThe optimizer evaluates all options together, considering current prices,\navailability, and any quotas, then launches on the cheapest option.\n\n### 1.2 ordered -- Strict Priority Fallback\n\nUse `ordered` when you have a strict preference. SkyPilot tries each option\nin order and uses the first one that succeeds.\n\n```yaml\n# priority-fallback.yaml\nname: priority-training\n\nresources:\n ordered:\n - infra: aws/us-east-1\n accelerators: A100:8\n - infra: gcp/us-central1\n accelerators: A100:8\n - infra: aws/us-west-2\n accelerators: A100:4\n - infra: gcp/europe-west4\n accelerators: A100:4\n\nrun: |\n python train.py\n```\n\nThis is useful when you have reserved instances or negotiated pricing on a\nspecific cloud/region and want to use those first.\n\n### 1.3 Kubernetes + Cloud Fallback\n\nA common pattern is to try on-prem Kubernetes first, then fall back to cloud:\n\n```yaml\n# k8s-cloud-fallback.yaml\nname: k8s-with-cloud-fallback\n\nresources:\n ordered:\n # Try on-prem K8s cluster first (free/cheaper)\n - infra: k8s/my-onprem-context\n accelerators: A100:4\n # Fall back to cloud K8s\n - infra: k8s/gke-prod-cluster\n accelerators: A100:4\n # Fall back to cloud VMs\n - infra: aws\n accelerators: A100:4\n - infra: gcp\n accelerators: A100:4\n\nrun: |\n python train.py\n```\n\n### 1.4 SSH Node Pools\n\nSkyPilot can use your own bare-metal machines or VMs via SSH node pools.\nFirst, define node pools in `~/.sky/ssh_node_pools.yaml`:\n\n```yaml\n# ~/.sky/ssh_node_pools.yaml\nmy-gpu-cluster:\n hosts:\n - 10.0.1.10\n - 10.0.1.11\n - 10.0.1.12\n\nmy-inference-box:\n hosts:\n - gpu-server.internal.example.com\n```\n\nThen reference them in your task YAML:\n\n```yaml\n# ssh-nodes.yaml\nname: train-on-ssh-nodes\n\nresources:\n infra: ssh/my-gpu-cluster\n accelerators: A100:8\n\nrun: |\n python train.py\n```\n\nSSH node pools can also participate in `any_of` or `ordered` fallback:\n\n```yaml\nresources:\n ordered:\n - infra: ssh/my-gpu-cluster\n accelerators: A100:8\n - infra: aws\n accelerators: A100:8\n```\n\n### 1.5 Wildcard Patterns in infra\n\nThe `infra` field supports `*` as a wildcard for any component\n(cloud, region, zone):\n\n```yaml\n# Any cloud, specific zone\nresources:\n infra: \"*/us-east-1/us-east-1a\"\n accelerators: A100:4\n```\n\n```yaml\n# Specific cloud, any region, specific zone suffix\nresources:\n infra: \"aws/*/us-east-1a\"\n accelerators: A100:4\n```\n\n```yaml\n# Any cloud, any region (equivalent to not specifying infra)\nresources:\n infra: \"*\"\n accelerators: A100:4\n```\n\nThe infra format is `cloud[/region[/zone]]`. For Kubernetes, the format is\n`k8s/context-name` (context names may contain slashes).\n\n### 1.6 Multi-Region Strategies\n\nTarget a specific region across any cloud, or multiple specific regions:\n\n```yaml\n# Multi-region with any_of\nresources:\n any_of:\n - infra: aws/us-east-1\n accelerators: H100:8\n - infra: aws/us-west-2\n accelerators: H100:8\n - infra: gcp/us-central1\n accelerators: H100:8\n - infra: gcp/us-east4\n accelerators: H100:8\n```\n\nFor single-cloud multi-region with the same GPU:\n\n```yaml\n# Will automatically try all available regions for this cloud\nresources:\n infra: aws\n accelerators: H100:8\n```\n\nSkyPilot's optimizer automatically considers all regions within the specified\ncloud and picks the cheapest available one. You only need `any_of` or\n`ordered` when you want to cross cloud boundaries or mix GPU types.\n\n### 1.7 Accelerator Sets and Lists\n\nFor single-resource blocks, you can use accelerator sets (unordered, cost-\noptimized) or lists (ordered, priority-based):\n\n```yaml\n# Unordered set: SkyPilot picks the cheapest available\nresources:\n accelerators: {A100:1, V100:1, L4:1, T4:1}\n```\n\n```yaml\n# Ordered list: SkyPilot tries in order\nresources:\n accelerators: [A100:1, V100:1, L4:1, T4:1]\n```\n\n---\n\n## 2. Distributed Training\n\nSkyPilot supports multi-node distributed training with automatic cluster\nsetup. It provides environment variables on every node for coordination.\n\n### 2.1 SkyPilot Environment Variables for Distributed Training\n\nOn each node, SkyPilot sets:\n\n| Variable | Description |\n|---|---|\n| `SKYPILOT_NODE_IPS` | Newline-separated list of all node IPs (head node first) |\n| `SKYPILOT_NODE_RANK` | 0-indexed rank of the current node (head = 0) |\n| `SKYPILOT_NUM_NODES` | Total number of nodes in the cluster |\n| `SKYPILOT_NUM_GPUS_PER_NODE` | Number of GPUs on the current node |\n\n### 2.2 PyTorch DDP Multi-Node Setup\n\nThe standard pattern uses `torchrun` with SkyPilot's environment variables:\n\n```yaml\n# ddp-training.yaml\nname: pytorch-ddp\n\nresources:\n accelerators: A100:8\n\nnum_nodes: 4\n\nsetup: |\n pip install torch torchvision\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n\n torchrun \\\n --nnodes=$NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$HEAD_IP \\\n --master_port=12355 \\\n train_ddp.py \\\n --epochs 100 \\\n --batch-size 256\n```\n\nA minimal `train_ddp.py` would use:\n\n```python\nimport os\n\nimport torch\nimport torch.distributed as dist\n\ndist.init_process_group(backend='nccl')\nlocal_rank = int(os.environ['LOCAL_RANK'])\ntorch.cuda.set_device(local_rank)\n# ... training loop\n```\n\n### 2.3 DeepSpeed ZeRO-3 Multi-Node\n\n```yaml\n# deepspeed-zero3.yaml\nname: deepspeed-training\n\nresources:\n accelerators: A100-80GB:8\n\nnum_nodes: 2\n\nfile_mounts:\n /workspace: .\n\nsetup: |\n pip install deepspeed torch transformers accelerate\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n\n # Create hostfile for DeepSpeed\n echo \"$SKYPILOT_NODE_IPS\" | while read ip; do\n echo \"$ip slots=$SKYPILOT_NUM_GPUS_PER_NODE\"\n done > /tmp/hostfile\n\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n deepspeed \\\n --hostfile /tmp/hostfile \\\n --master_addr $HEAD_IP \\\n --master_port 29500 \\\n --num_nodes $NUM_NODES \\\n --num_gpus $SKYPILOT_NUM_GPUS_PER_NODE \\\n /workspace/train.py \\\n --deepspeed /workspace/ds_config_zero3.json \\\n --model_name_or_path meta-llama/Llama-2-7b-hf \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 4\n fi\n```\n\nExample `ds_config_zero3.json`:\n\n```json\n{\n \"bf16\": { \"enabled\": true },\n \"zero_optimization\": {\n \"stage\": 3,\n \"offload_optimizer\": { \"device\": \"none\" },\n \"offload_param\": { \"device\": \"none\" },\n \"overlap_comm\": true,\n \"contiguous_gradients\": true,\n \"reduce_bucket_size\": 5e7,\n \"stage3_prefetch_bucket_size\": 5e7,\n \"stage3_param_persistence_threshold\": 1e5\n },\n \"gradient_accumulation_steps\": 4,\n \"train_micro_batch_size_per_gpu\": 2,\n \"wall_clock_breakdown\": false\n}\n```\n\n### 2.4 Ray Train Integration\n\nSkyPilot includes a helper script to start a Ray cluster across nodes. Use it\nfor Ray Train, Ray Tune, and other Ray-based distributed workloads:\n\n```yaml\n# ray-train.yaml\nname: ray-distributed-training\n\nresources:\n accelerators: A100:8\n ports:\n - 8280 # Ray dashboard\n\nnum_nodes: 2\n\nenvs:\n HF_HUB_ENABLE_HF_TRANSFER: \"1\"\n\nsetup: |\n pip install \"ray[default,train]\" torch transformers\n\nrun: |\n head_ip=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n\n # Use the built-in SkyPilot Ray start script\n export RAY_HEAD_PORT=6385\n export RAY_DASHBOARD_PORT=8280\n export RAY_DASHBOARD_HOST=0.0.0.0\n export RAY_HEAD_IP_ADDRESS=\"$head_ip\"\n ~/sky_templates/ray/start_cluster\n\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n export RAY_ADDRESS=\"http://localhost:8280\"\n ray job submit --address=\"$RAY_ADDRESS\" \\\n -- python train_with_ray.py\n fi\n```\n\n### 2.5 FSDP Configuration\n\nPyTorch Fully Sharded Data Parallel (FSDP) with SkyPilot:\n\n```yaml\n# fsdp-training.yaml\nname: fsdp-llama\n\nresources:\n accelerators: A100-80GB:8\n\nnum_nodes: 2\n\nsetup: |\n pip install torch transformers accelerate\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n TOTAL_GPUS=$((NUM_NODES * SKYPILOT_NUM_GPUS_PER_NODE))\n\n accelerate launch \\\n --num_processes $TOTAL_GPUS \\\n --num_machines $NUM_NODES \\\n --machine_rank $SKYPILOT_NODE_RANK \\\n --main_process_ip $HEAD_IP \\\n --main_process_port 29500 \\\n --use_fsdp \\\n --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP \\\n --fsdp_sharding_strategy FULL_SHARD \\\n --fsdp_state_dict_type SHARDED_STATE_DICT \\\n --mixed_precision bf16 \\\n train_fsdp.py \\\n --model_name meta-llama/Llama-2-7b-hf \\\n --per_device_train_batch_size 2\n```\n\n### 2.6 NCCL Environment Tuning\n\nFor multi-node GPU training, NCCL configuration is critical for performance.\nSet these via `envs:` in your YAML:\n\n```yaml\n# nccl-tuned.yaml\nname: nccl-optimized-training\n\nresources:\n accelerators: H100:8\n\nnum_nodes: 4\n\nenvs:\n # Debug: set to INFO or WARN for troubleshooting\n NCCL_DEBUG: WARN\n # Network interface selection (check with `ip addr` on your nodes)\n NCCL_SOCKET_IFNAME: eth0\n # Use tree algorithm for better all-reduce on large clusters\n NCCL_ALGO: Tree\n # Avoid CUDA stream recording overhead\n TORCH_NCCL_AVOID_RECORD_STREAMS: \"1\"\n # InfiniBand settings (for clouds with IB/RDMA)\n NCCL_IB_DISABLE: \"0\"\n NCCL_IB_GID_INDEX: 3\n # Cross-node buffer sizes\n NCCL_BUFFSIZE: \"8388608\"\n # P2P transport (NVLink within node, network across nodes)\n NCCL_P2P_LEVEL: NVL\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$HEAD_IP \\\n --master_port=12355 \\\n train.py\n```\n\n### 2.7 Common Distributed Training Gotchas\n\n1. **Port conflicts**: Always use a non-default master port (not 29500)\n if running multiple training jobs on the same cluster. Or use SkyPilot's\n port allocation.\n\n2. **Head node vs worker pattern**: Only the head node (`SKYPILOT_NODE_RANK\n == 0`) should launch the training coordinator (e.g., DeepSpeed launcher,\n Ray job submit). Worker nodes run the setup + the Ray/torchrun rendezvous.\n\n3. **Hanging at initialization**: Usually caused by NCCL not finding the right\n network interface. Set `NCCL_SOCKET_IFNAME` to the correct interface.\n Use `NCCL_DEBUG=INFO` to diagnose.\n\n4. **OOM on multi-node**: Gradient synchronization buffers consume extra\n memory. Reduce per-device batch size or use gradient accumulation.\n\n5. **Run section runs on ALL nodes**: The `run:` section executes on every\n node in the cluster. Use `if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]` to gate\n head-only logic (like launching the coordinator or downloading data).\n\n6. **Setup runs on ALL nodes**: The `setup:` section also runs on all nodes.\n This is typically correct (install dependencies everywhere), but be aware\n of it when doing one-time data preparation.\n\n---\n\n## 3. Managed Jobs Production Patterns\n\nManaged jobs (`sky jobs launch`) provide automatic recovery from preemptions,\nspot instance failures, and transient errors. They run on a lightweight jobs\ncontroller that monitors and relaunches your workload.\n\n### 3.1 job_recovery Configuration\n\nThe `job_recovery` field controls how managed jobs handle failures.\n\n**Strategy options:**\n- `EAGER_NEXT_REGION` (default): On preemption, immediately try a different\n region/cloud before retrying the same region.\n- `FAILOVER`: On preemption, first retry in the same region, then fail over\n to other regions.\n\n**Full configuration as a dict:**\n\n```yaml\n# managed-job-recovery.yaml\nname: resilient-training\n\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n recover_on_exit_codes: [1, 137]\n\nrun: |\n python train.py --checkpoint-dir /ckpts\n```\n\n**Shorthand (strategy only):**\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery: FAILOVER\n```\n\n**Field reference:**\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `strategy` | string | `EAGER_NEXT_REGION` | Recovery strategy name |\n| `max_restarts_on_errors` | int | 0 | Max restarts on non-preemption errors |\n| `recover_on_exit_codes` | int or list[int] | none | Exit codes that always trigger recovery (ignores max_restarts_on_errors counter) |\n\nWhen `recover_on_exit_codes` is set, any job failure with a matching exit code\ntriggers recovery without counting against `max_restarts_on_errors`. This is\nuseful for known transient failures.\n\nTo give up on a job that cannot get resources, set the top-level\n`job.wait_for_scheduling_timeout` (not part of `job_recovery`). The job ends\n`CANCELLED` if it has not started within that long of being submitted, whether\nit is retrying cloud VM launches or waiting in a Kubernetes queue such as\nKueue. Later recoveries do not re-arm it.\n\n```yaml\njob:\n wait_for_scheduling_timeout: 2h # duration (e.g. 90s, 30m, 2h) or int seconds\nresources:\n accelerators: A100:8\n job_recovery: FAILOVER\n```\n\n### 3.2 Checkpointing Patterns\n\nFor spot instances, checkpointing to cloud storage is essential. The pattern\nis: save checkpoints periodically to a mounted bucket, and on restart, resume\nfrom the latest checkpoint.\n\n```yaml\n# checkpointed-training.yaml\nname: checkpointed-llm-training\n\nresources:\n accelerators: A100-80GB:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 5\n\nfile_mounts:\n /ckpts:\n name: my-training-checkpoints # S3/GCS/R2 bucket\n mode: MOUNT\n\nrun: |\n # Resume from latest checkpoint if it exists\n CKPT_ARG=\"\"\n LATEST=$(ls -td /ckpts/checkpoint-* 2>/dev/null | head -1)\n if [ -n \"$LATEST\" ]; then\n echo \"Resuming from checkpoint: $LATEST\"\n CKPT_ARG=\"--resume_from_checkpoint $LATEST\"\n fi\n\n python train.py \\\n --output_dir /ckpts \\\n --save_steps 500 \\\n $CKPT_ARG\n```\n\nLaunch as a managed job:\n\n```bash\nsky jobs launch checkpointed-training.yaml\n```\n\n### 3.3 Hyperparameter Sweeps as Managed Jobs\n\nLaunch multiple managed jobs for a hyperparameter sweep. Each job runs\nindependently with automatic recovery:\n\n```bash\n# Launch sweep via CLI\nfor lr in 1e-4 5e-5 1e-5; do\n for bs in 16 32 64; do\n sky jobs launch sweep.yaml \\\n --env LR=$lr \\\n --env BATCH_SIZE=$bs \\\n -n \"sweep-lr${lr}-bs${bs}\" \\\n -y\n done\ndone\n```\n\n```yaml\n# sweep.yaml\nname: hp-sweep\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 2\n\nenvs:\n LR: 1e-4\n BATCH_SIZE: 32\n\nfile_mounts:\n /results:\n name: sweep-results-bucket\n mode: MOUNT\n\nrun: |\n python train.py \\\n --learning-rate $LR \\\n --batch-size $BATCH_SIZE \\\n --output-dir \"/results/lr${LR}_bs${BATCH_SIZE}\"\n```\n\nMonitor all running jobs:\n\n```bash\nsky jobs queue -o json\n```\n\n### 3.4 Job Pools (Experimental)\n\nJob pools pre-provision clusters that multiple jobs can be submitted to,\navoiding cold-start overhead. Pools are defined with a `service:` section\ncontaining `pool:` configuration.\n\n**Create a pool:**\n\n```yaml\n# gpu-pool.yaml\nname: gpu-pool\n\nresources:\n accelerators: A100:8\n\nservice:\n pool:\n workers: 3\n```\n\n```bash\n# Apply the pool configuration\nsky jobs pool apply gpu-pool.yaml -p my-gpu-pool\n```\n\n**Submit jobs to the pool:**\n\n```bash\nsky jobs launch job.yaml --pool my-gpu-pool\n```\n\n**Pool with autoscaling:**\n\n```yaml\n# autoscaling-pool.yaml\nname: autoscaling-pool\n\nresources:\n accelerators: A100:4\n\nservice:\n pool:\n min_workers: 1\n max_workers: 10\n queue_length_threshold: 2\n upscale_delay_seconds: 60\n downscale_delay_seconds: 300\n```\n\n**Pool management commands:**\n\n```bash\n# Check pool status\nsky jobs pool status\n\n# Check a specific pool\nsky jobs pool status my-gpu-pool\n\n# Update number of workers\nsky jobs pool apply -p my-gpu-pool --workers 5\n\n# Delete a pool\nsky jobs pool down my-gpu-pool\n\n# Delete all pools\nsky jobs pool down -a\n```\n\n---\n\n## 4. SkyServe Advanced\n\nSkyServe deploys model serving endpoints with autoscaling, multi-cloud\nreplicas, rolling updates, and spot instance support.\n\n### 4.1 Autoscaling Configuration\n\n```yaml\n# autoscaling-service.yaml\nname: llm-service\n\nresources:\n accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe:\n path: /health\n initial_delay_seconds: 300\n timeout_seconds: 10\n replica_policy:\n min_replicas: 2\n max_replicas: 10\n target_qps_per_replica: 5.0\n upscale_delay_seconds: 120\n downscale_delay_seconds: 600\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-2-7b-hf \\\n --port 8080\n```\n\n**Autoscaling field reference:**\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `min_replicas` | int | 1 | Minimum number of replicas |\n| `max_replicas` | int | same as min | Maximum number of replicas |\n| `target_qps_per_replica` | float | required if max != min | Target QPS per replica for scaling |\n| `upscale_delay_seconds` | int | 300 | Seconds target must exceed before scaling up |\n| `downscale_delay_seconds` | int | 1200 | Seconds target must be below before scaling down |\n| `num_overprovision` | int | none | Extra replicas beyond what autoscaling calculates |\n\nDeploy and manage:\n\n```bash\n# Deploy the service\nsky serve up autoscaling-service.yaml -n my-llm\n\n# Check service status\nsky serve status my-llm\n\n# Get the service endpoint\nsky serve status my-llm --endpoint\n\n# Tail controller logs\nsky serve logs --controller my-llm\n```\n\n### 4.2 Rolling Updates vs Blue-Green Updates\n\n**Rolling update** (default): Gradually replaces replicas one by one. Some\nold and new replicas serve traffic simultaneously during the transition.\n\n```bash\n# Update with rolling strategy (default)\nsky serve update my-llm updated-service.yaml\n\n# Explicitly specify rolling\nsky serve update --mode rolling my-llm updated-service.yaml\n```\n\n**Blue-green update**: Provisions all new replicas first, then switches\ntraffic atomically. No mixed versions serve traffic simultaneously.\n\n```bash\n# Blue-green update\nsky serve update --mode blue_green my-llm updated-service.yaml\n```\n\nBlue-green requires enough quota to run both old and new replicas\nsimultaneously during the transition.\n\n### 4.3 Spot Replicas for Cost Savings\n\nUse spot instances for serving replicas with automatic on-demand fallback:\n\n```yaml\n# spot-serving.yaml\nname: spot-llm-service\n\nresources:\n accelerators: A100:1\n use_spot: true\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 3\n max_replicas: 8\n target_qps_per_replica: 5.0\n # Maintain 1 on-demand replica as a baseline\n base_ondemand_fallback_replicas: 1\n # Dynamically add on-demand replicas if spot is unavailable\n dynamic_ondemand_fallback: true\n\nrun: |\n python serve.py --port 8080\n```\n\n### 4.4 TLS/HTTPS Configuration\n\nEnable TLS termination at the load balancer:\n\n```yaml\n# tls-service.yaml\nname: secure-service\n\nresources:\n accelerators: L4:1\n ports:\n - 8080\n\nfile_mounts:\n /certs/server.key: ./certs/server.key\n /certs/server.crt: ./certs/server.crt\n\nservice:\n readiness_probe: /health\n replicas: 2\n tls:\n keyfile: /certs/server.key\n certfile: /certs/server.crt\n\nrun: |\n python serve.py --port 8080\n```\n\n### 4.5 Readiness Probe Configuration\n\nThe readiness probe determines when a replica is ready to receive traffic.\n\n**Simple GET probe (most common):**\n\n```yaml\nservice:\n readiness_probe: /health\n replicas: 2\n```\n\n**Full probe configuration:**\n\n```yaml\nservice:\n readiness_probe:\n path: /v1/models\n initial_delay_seconds: 600\n timeout_seconds: 30\n headers:\n Authorization: \"Bearer test-token\"\n replicas: 1\n```\n\n**POST probe (useful for LLMs that need a generation test):**\n\n```yaml\nservice:\n readiness_probe:\n path: /v1/completions\n post_data:\n model: llama-2-7b\n prompt: \"Hello\"\n max_tokens: 1\n initial_delay_seconds: 900\n timeout_seconds: 60\n replicas: 1\n```\n\n**Default values:**\n- `initial_delay_seconds`: 1200 (20 minutes) -- readiness failures during\n this period are ignored\n- `timeout_seconds`: 15\n\n### 4.6 Multi-Cloud Replicas\n\nDeploy replicas across multiple clouds for availability and cost optimization:\n\n```yaml\n# multi-cloud-serving.yaml\nname: multi-cloud-llm\n\nresources:\n any_of:\n - infra: aws\n accelerators: A100:1\n - infra: gcp\n accelerators: A100:1\n - infra: azure\n accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 3\n max_replicas: 10\n target_qps_per_replica: 5.0\n\nrun: |\n python serve.py --port 8080\n```\n\nSkyPilot distributes replicas across available clouds for redundancy and\npicks the cheapest options.\n\n### 4.7 Load Balancing Policies\n\nSkyServe supports different load balancing policies:\n\n```yaml\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 2\n max_replicas: 8\n target_qps_per_replica: 5.0\n load_balancing_policy: round_robin\n```\n\nFor heterogeneous GPU deployments, use instance-aware load balancing:\n\n```yaml\n# Instance-aware balancing across different GPU types\nresources:\n any_of:\n - accelerators: H100:1\n - accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 2\n max_replicas: 10\n target_qps_per_replica:\n H100: 10.0\n A100: 5.0\n load_balancing_policy: instance_aware_least_load\n```\n\n---\n\n## 5. Kubernetes Integration\n\nSkyPilot treats Kubernetes clusters as first-class infrastructure alongside\ncloud providers.\n\n### 5.1 Using K8s Contexts\n\nSkyPilot uses your kubeconfig contexts. Reference them with `infra: k8s/`:\n\n```yaml\n# Use a specific K8s context\nresources:\n infra: k8s/my-cluster-context\n accelerators: A100:4\n```\n\n```yaml\n# Use the default K8s context (current-context in kubeconfig)\nresources:\n infra: kubernetes\n accelerators: A100:4\n```\n\n```yaml\n# EKS context (contexts can contain special characters)\nresources:\n infra: k8s/arn:aws:eks:us-east-1:123456789012:cluster/my-cluster\n accelerators: A100:4\n```\n\n### 5.2 Pod Config Overrides\n\nCustomize the pod spec for SkyPilot pods via `~/.sky/config.yaml`. This\nallows setting tolerations, nodeSelectors, labels, and other Kubernetes-\nspecific configurations.\n\n```yaml\n# ~/.sky/config.yaml\nkubernetes:\n allowed_contexts:\n - my-production-cluster\n - my-dev-cluster\n context_configs:\n my-production-cluster:\n pod_config:\n spec:\n tolerations:\n - key: \"nvidia.com/gpu\"\n operator: \"Exists\"\n effect: \"NoSchedule\"\n nodeSelector:\n node-type: gpu\n team: ml-training\n containers:\n - name: skypilot\n resources:\n limits:\n nvidia.com/gpu: \"1\"\n metadata:\n labels:\n team: ml-platform\n environment: production\n custom_metadata:\n labels:\n app: skypilot\n managed-by: skypilot\n my-dev-cluster:\n pod_config:\n spec:\n nodeSelector:\n node-type: dev-gpu\n```\n\n### 5.3 Multi-Cluster K8s with any_of\n\nSpread workloads across multiple Kubernetes clusters:\n\n```yaml\n# multi-k8s.yaml\nname: multi-cluster-training\n\nresources:\n any_of:\n - infra: k8s/us-east-cluster\n accelerators: A100:8\n - infra: k8s/eu-west-cluster\n accelerators: A100:8\n - infra: k8s/asia-cluster\n accelerators: H100:8\n\nrun: |\n python train.py\n```\n\nCombined with cloud fallback:\n\n```yaml\nresources:\n ordered:\n - infra: k8s/onprem-cluster\n accelerators: A100:8\n - infra: k8s/gke-cluster\n accelerators: A100:8\n - infra: aws\n accelerators: A100:8\n```\n\n### 5.4 GPU Detection and Labeling\n\nSkyPilot needs GPU labels on Kubernetes nodes to schedule GPU workloads.\nUse `sky gpus label` to automatically detect and label GPUs:\n\n```bash\n# Label GPUs in the current Kubernetes context\nsky gpus label\n\n# Label GPUs in a specific context\nsky gpus label --context my-k8s-cluster\n\n# Start labeling without waiting for completion\nsky gpus label --async\n\n# Cleanup labeling resources when done\nsky gpus label --cleanup\n```\n\nCheck available GPUs across all configured infrastructure:\n\n```bash\n# List all available GPUs\nsky gpus list\n\n# Show GPUs for a specific type\nsky gpus list A100\n\n# Show GPUs available on Kubernetes\nsky gpus list --infra kubernetes\n```\n\n### 5.5 Kubernetes Configuration in ~/.sky/config.yaml\n\n```yaml\n# ~/.sky/config.yaml\nkubernetes:\n # Restrict which contexts SkyPilot can use\n allowed_contexts:\n - gke-prod\n - eks-staging\n\n # Port exposure mode\n ports: loadbalancer # or nodeport, ingress, podip\n\n # Cluster autoscaler integration\n autoscaler: gke # or karpenter, generic\n\n # Per-context configuration\n context_configs:\n gke-prod:\n provision_timeout: 600\n pod_config:\n spec:\n nodeSelector:\n cloud.google.com/gke-accelerator: nvidia-tesla-a100\n```\n\n---\n\n## 6. API Server and Team Setup\n\nThe SkyPilot API server enables team collaboration by providing a shared\ncontrol plane for launching and managing clusters, jobs, and services.\n\n### 6.1 Local API Server\n\nFor individual use or development:\n\n```bash\n# Start the local API server\nsky api start\n\n# Check server status\nsky api status\n\n# Stop the server\nsky api stop\n```\n\nThe local API server runs on port 46580 by default. It starts automatically\nwhen you run any `sky` command.\n\n### 6.2 Remote API Server with Helm\n\nDeploy a shared API server on Kubernetes for team use:\n\n```bash\n# Add the SkyPilot Helm repo\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\n\n# Build chart dependencies\nhelm dependency build ./charts/skypilot\n\n# Deploy the API server\nNAMESPACE=skypilot\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace\n```\n\nFor custom images (e.g., testing local changes):\n\n```bash\n# Build and push custom image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 \\\n -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy with custom image\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n --namespace skypilot \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\nUpgrade an existing deployment (always use `--reuse-values`):\n\n```bash\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n### 6.3 PostgreSQL Backend\n\nFor production deployments, use PostgreSQL instead of SQLite:\n\n```bash\n# Create a secret with the database connection string\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/skypilot\"\n\n# Deploy with PostgreSQL backend\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n -n skypilot --create-namespace \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\nOr set the connection string directly (less secure, not recommended for\nproduction):\n\n```bash\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n -n skypilot --create-namespace \\\n --set apiService.dbConnectionString=\"postgresql://user:pass@host:5432/db\"\n```\n\n### 6.4 Authentication\n\nConnect to a remote API server:\n\n```bash\n# OAuth2 browser login\nsky api login -e https://skypilot.example.com\n\n# Service account token login\nsky api login -e https://skypilot.example.com --token sky_abc123...\n```\n\nCheck the current connection:\n\n```bash\nsky api status\n```\n\n---\n\n## 7. Cost Optimization\n\nSkyPilot provides multiple mechanisms to minimize cloud spending while\nmaximizing GPU availability.\n\n### 7.1 Automatic Cloud Selection (Cheapest)\n\nBy default, SkyPilot's optimizer picks the cheapest cloud/region for your\nrequested resources:\n\n```yaml\n# SkyPilot automatically finds the cheapest option\nresources:\n accelerators: A100:4\n```\n\n```bash\n# Preview the optimizer's choice without launching\nsky launch --dryrun task.yaml\n```\n\nThe `--dryrun` flag shows you which cloud, region, instance type, and price\nSkyPilot would choose, without actually provisioning anything.\n\n### 7.2 Spot Instances\n\nSpot/preemptible instances offer 60-90% savings over on-demand:\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n```\n\nFor production spot workloads, combine with managed jobs for automatic\nrecovery:\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n```\n\n```bash\nsky jobs launch spot-training.yaml\n```\n\n### 7.3 Dryrun to Preview Costs\n\nAlways preview costs before launching expensive workloads:\n\n```bash\n# Show what would be launched and estimated cost\nsky launch --dryrun gpu-training.yaml\n\n# Show cost for a specific cloud\nsky launch --dryrun --infra aws gpu-training.yaml\n```\n\n### 7.4 Cost Report\n\nTrack spending across all clusters:\n\n```bash\n# Show cost report for active clusters\nsky cost-report\n\n# Show all clusters including terminated (last 30 days)\nsky cost-report --all\n\n# Show costs for specific number of days\nsky cost-report --days 7\n```\n\n### 7.5 Autostop Configuration\n\nAutomatically stop or terminate idle clusters to avoid waste. Autostop is\nconfigured in the `resources:` section of your YAML.\n\n**Simple autostop (idle minutes):**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop: 30 # Stop after 30 minutes idle\n```\n\n**Full autostop configuration:**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop:\n idle_minutes: 30\n down: true # Terminate instead of just stopping\n wait_for: jobs_and_ssh # Reset timer while jobs or SSH sessions active\n```\n\n**wait_for options:**\n\n| Value | Description |\n|---|---|\n| `jobs_and_ssh` | (default) Wait for running jobs AND open SSH sessions |\n| `jobs` | Only wait for running jobs to finish |\n| `none` | Stop immediately after idle_minutes, regardless of activity |\n\n**Disable autostop:**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop: false\n```\n\n**CLI overrides:**\n\n```bash\n# Set autostop on an existing cluster\nsky autostop my-cluster -i 60\n\n# Set autodown (terminate when idle)\nsky autostop my-cluster -i 30 --down\n\n# Cancel autostop\nsky autostop my-cluster --cancel\n```\n\n### 7.6 Autostop Hooks for Pre-Shutdown Actions\n\nRun a custom script before the cluster stops or terminates. This is useful\nfor saving state, uploading logs, or graceful shutdown procedures:\n\n```yaml\nresources:\n accelerators: A100:4\n autostop:\n idle_minutes: 30\n down: true\n hook: |\n echo \"Cluster shutting down, saving state...\"\n cp -r /tmp/logs /data/saved-logs/\n python cleanup.py\n hook_timeout: 120 # Seconds to wait for hook to complete\n```\n\nThe hook runs before the cluster is stopped or terminated. If the hook\nexceeds `hook_timeout` seconds, it is killed and the shutdown proceeds.\n\n### 7.7 Combining Cost Optimization Strategies\n\nFor maximum savings, combine multiple strategies:\n\n```yaml\n# maximum-savings.yaml\nname: cost-optimized-training\n\nresources:\n any_of:\n - infra: aws\n accelerators: A100:8\n - infra: gcp\n accelerators: A100:8\n - infra: lambda\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 5\n autostop:\n idle_minutes: 10\n down: true\n\nfile_mounts:\n /ckpts:\n name: training-checkpoints\n mode: MOUNT\n\nrun: |\n python train.py --checkpoint-dir /ckpts --resume-latest\n```\n\nLaunch as a managed job for automatic recovery:\n\n```bash\nsky jobs launch maximum-savings.yaml\n```\n\nThis configuration:\n1. Uses `any_of` to find the cheapest cloud\n2. Uses spot instances for 60-90% savings\n3. Recovers automatically from preemptions via managed jobs\n4. Auto-terminates idle clusters\n5. Checkpoints to cloud storage for seamless recovery\n","isInternal":false,"tokens":9116,"sizeBytes":33329},{"name":"cli-reference.md","path":"agent/skills/skypilot/references/cli-reference.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/cli-reference.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"\n\n\n# SkyPilot CLI Reference\n\nComplete reference for all `sky` commands and options.\n\n## Core Cluster Commands\n\n### `sky launch`\n\nLaunch a cluster or task.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--cluster`, `-c` — A cluster name. If provided, either reuse an existing cluster with that name or provision a new cluster with that name. Otherwise provision a new cluster with an autogenerated name.\n- `--dryrun` — If True, do not actually run the job.\n- `--detach-run`, `-d`, `-no-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--docker` — (Deprecated) Local docker support is deprecated. To run locally, create a local Kubernetes cluster with ``sky local up``.\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--priority` — Priority for this task. Accepts either an integer (from -1000 to 1000, or a string priority class name. Passing \"none\" clears both.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--idle-minutes-to-autostop`, `-i` — Automatically stop the cluster after this many minutes of idleness, i.e., no running or pending jobs in the cluster's job queue. Idleness gets reset depending on the ``--wait-for`` flag. Setting this ...\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes-to-autostop``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs an...\n- `--down` — Autodown the cluster: tear down the cluster after all jobs finish (successfully or abnormally). If --idle-minutes-to-autostop is also set, the cluster will be torn down after the specified idle time. ...\n- `--retry-until-up`, `-r` — Whether to retry provisioning infinitely until the cluster is up, if we fail to launch the cluster on any possible region/cloud due to unavailability errors.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--no-setup` — Skip setup phase when (re-)launching cluster.\n- `--clone-disk-from`, `--clone` — [Experimental] Clone disk from an existing cluster to launch a new one. This is useful when the new cluster needs to have the same data on the boot disk as an existing cluster.\n- `--fast` — [Experimental] If the cluster is already up and available, skip provisioning and setup steps.\n- `--resize` — Resize an existing cluster to --num-nodes. Supports both scale-up (adding workers) and scale-down (removing workers). Scale-down requires no running jobs. Requires -c to specify an existing cluster.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workspace`, `-w` — Workspace to launch into. Shorthand for `--config active_workspace=`.\n\n### `sky exec`\n\nExecute a task or command on an existing cluster.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTER` — text\n- `--cluster`, `-c` — This is the same as the positional argument, just for consistency.\n- `ENTRYPOINT` — text\n- `--detach-run`, `-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n\n### `sky status`\n\nShow clusters.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--refresh`, `-r` — Query the latest cluster statuses from the cloud provider(s).\n- `--ip` — Get the IP address of the head node of a cluster. This option will override all other options. For Kubernetes clusters, the returned IP address is the internal IP of the head pod, and may not be acces...\n- `--endpoints` — Get all exposed endpoints and corresponding URLs for acluster. This option will override all other options.\n- `--endpoint` — Get the endpoint URL for the specified port number on the cluster. This option will override all other options.\n- `--show-managed-jobs`, `--no-show-managed-jobs` — Also show recent in-progress managed jobs, if any.\n- `--show-services`, `--no-show-services` — Also show sky serve services, if any.\n- `--show-pools`, `--no-show-pools` — Also show pools, if any.\n- `CLUSTERS` — text\n- `--all-users`, `-u` — Show all clusters, including those not owned by the current user.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n### `sky stop`\n\nStop cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Stop all existing clusters.\n- `--all-users`, `-u` — Stop all existing clusters for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky start`\n\nRestart cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Start all existing clusters.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--idle-minutes-to-autostop`, `-i` — Automatically stop the cluster after this many minutes of idleness, i.e., no running or pending jobs in the cluster's job queue. Idleness gets reset depending on the ``--wait-for`` flag. Setting this ...\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes-to-autostop``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs an...\n- `--down` — Autodown the cluster: tear down the cluster after specified minutes of idle time after all jobs finish (successfully or abnormally). Requires --idle-minutes-to-autostop to be set.\n- `--retry-until-up`, `-r` — Retry provisioning infinitely until the cluster is up, if we fail to start the cluster due to unavailability errors.\n- `--force`, `-f` — Force start the cluster even if it is already UP. Useful for upgrading the SkyPilot runtime on the cluster.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky down`\n\nTear down cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Tear down all existing clusters.\n- `--all-users`, `-u` — Tear down all existing clusters for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--purge`, `-p` — (Advanced) Forcefully remove the cluster(s) from SkyPilot's cluster table, even if the actual cluster termination failed on the cloud. WARNING: This flag should only be set sparingly in certain manual...\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky autostop`\n\nSchedule an autostop or autodown for cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Autostop all existing clusters.\n- `--all-users`, `-u` — Autostop all existing clusters for all users.\n- `--idle-minutes`, `-i` — Set the idle minutes before autostopping the cluster. See the doc above for detailed semantics.\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs and SSH connec...\n- `--cancel` — Cancel any currently active auto{stop,down} setting for the cluster. No-op if there is no active setting.\n- `--down` — Use autodown (tear down the cluster; non-restartable), instead of autostop (restartable).\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky cost-report`\n\nShow estimated costs for launched clusters.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--all`, `-a` — Show all cluster information.\n- `--days` (default: `30`) — Show clusters from the last N days. Default is 30 days. If set to 0, show all clusters.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Job Queue Commands\n\n### `sky queue`\n\nShow the job queue for cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--all-users`, `-u` — Show all users' information in full.\n- `--skip-finished`, `-s` — Show only pending/running jobs' information.\n- `CLUSTERS` — text\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n### `sky logs`\n\nTail the log of a job.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--provision` — Stream the cluster provisioning logs (provision.log).\n- `--hook` — Stream a per-event lifecycle-hook log from the cluster. Omit the event name to auto-select whichever log exists.\n- `--autostop` — [DEPRECATED] Alias for `--hook stop`. The autostop event was renamed to `stop` in the lifecycle-hooks framework.\n- `--worker`, `-w` — The worker ID to stream the logs from. If not set, stream the logs of the head node.\n- `--sync-down`, `-s` — Sync down the logs of a job to the local machine. For a distributed job, a separate log file from each worker will be downloaded.\n- `--status` — If specified, do not show logs but exit with a status code for the job's status: 0 for succeeded, or 1 for all other statuses.\n- `--follow`, `--no-follow` — Follow the logs of a job. If --no-follow is specified, print the log so far and exit. [default: --follow]\n- `--tail` (default: `-1`) — Number of lines to display from the end of the log file. Default is the last 1000 lines — sensible for multi-GB logs where downloading the full file is slow. Pass --tail 0 to print the entire log.\n- `CLUSTER` — text\n- `JOB_IDS` — text\n\n### `sky cancel`\n\nCancel job(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTER` — text\n- `--all`, `-a` — Cancel all jobs from current user on the specified cluster.\n- `--all-users`, `-u` — Cancel all jobs on the specified cluster for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `JOBS` — integer\n\n## Infrastructure Commands\n\n### `sky check`\n\nCheck which clouds are available to use.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `INFRA_LIST` — text\n- `--verbose`, `-v` — Show the activated account for each cloud.\n- `--workspace`, `-w` — The workspace to check. If None, all workspaces will be checked.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Dashboard\n\n### `sky dashboard`\n\nOpens the SkyPilot dashboard.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n\n## GPU/Accelerator Commands\n\nSkyPilot GPU/Accelerator CLI.\n\n### `sky gpus label`\n\nLabel GPU nodes in a Kubernetes cluster for use with SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--context`, `-c` — Kubernetes context to use. If not specified, uses the current context from the API server.\n- `--cleanup` — Only cleanup existing GPU labeler resources.\n- `--async` — Do not wait for GPU labeling to complete.\n\n### `sky gpus list`\n\nShow supported GPU/TPU/accelerators and their prices.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ACCELERATOR_STR` — text\n- `--all`, `-a` — Show details of all GPU/TPU/accelerator offerings.\n- `--infra` — Infrastructure to query. Examples: \"aws\", \"aws/us-east-1\"\n- `--cloud` — Cloud provider to query.\n- `--region` — The region to use. If not specified, shows accelerators from all regions.\n- `--all-regions` — Show pricing and instance details for a specified accelerator across all regions and clouds.\n- `--verbose`, `-v` — Show all information in full.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Managed Jobs Commands\n\nManaged Jobs CLI (jobs with auto-recovery).\n\n### `sky jobs cancel`\n\nCancel managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--name`, `-n` — Managed job name to cancel.\n- `--pool`, `-p` — Pool name to cancel.\n- `JOB_IDS` — integer\n- `--task` — Cancel one dynamic task of the job (a job launched from inside it), by the index shown in `sky jobs queue` or by name. A declared task cannot be cancelled alone.\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--all`, `-a` — Cancel all managed jobs for the current user.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--all-users`, `-u` — Cancel all managed jobs from all users.\n\n### `sky jobs dashboard`\n\nOpens a dashboard for managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n\n### `sky jobs launch`\n\nLaunch a managed job from a YAML or a command.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--priority` — Priority for this task. Accepts either an integer (from -1000 to 1000, or a string priority class name. Passing \"none\" clears both.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--cluster`, `-c` — Alias for --name, the name of the managed job.\n- `--job-recovery` — Recovery strategy to use for managed jobs.\n- `--detach-run`, `-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--pool`, `-p` — (Experimental; optional) Pool to use for jobs submission.\n- `--num-jobs` — Number of jobs to submit.\n- `--job-group` — Attach to an existing job group, by job id or unique running job name. The job is shown under it and cancelled with it. Defaults to the surrounding job group when launched from inside one.\n- `--no-job-group` — Launch a top-level job even when running inside a job group (do not attach to it).\n- `--depends-on` — Comma-separated managed job IDs to wait for. The job starts once all of them succeed, and is cancelled if any of them ends otherwise. The launch fails if one of them has already ended without succeedi...\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workspace`, `-w` — Workspace to submit the managed job into. Shorthand for `--config active_workspace=`.\n- `--output`, `-o` — Print only the submitted job IDs, for scripts: `id` prints one per line, `json` prints {\"job_ids\": [...]}. Everything else goes to stderr. Implies --detach-run.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky jobs logs`\n\nTail or sync down the log of a managed job.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--name`, `-n` — Managed job name.\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this job; useful for debugging launching/recoveries, etc.\n- `--refresh`, `-r` — Query the latest job logs, restarting the jobs controller if stopped.\n- `--sync-down`, `-s` — Download logs for all jobs shown in the queue.\n- `--tail` (default: `-1`) — Number of lines to display from the end of the log file. Default is the last 1000 lines — sensible for multi-GB logs where downloading the full file is slow. Pass --tail 0 to print the entire log.\n- `JOB_ID` — integer\n- `TASK` — text\n\n### `sky jobs pool` (subgroup)\n\n(Experimental) Pool management commands.\n\n### `sky jobs pool apply`\n\nEither apply a config to a pool for managed jobs submission or update the number of workers in the pool. One of POOL_YAML or --workers must be provided.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `POOL_YAML` — text\n- `--pool`, `-p` — A pool name. Unique for each pool. If not provided, a unique name is autogenerated.\n- `--mode` (default: `rolling`) — Update mode. If \"rolling\", pool will be updated with rolling update. If \"blue_green\", pool will be updated with blue-green update. This option is only valid when the pool is already running.\n- `--workers` — Can be used to update the number of workers in the pool.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky jobs pool down`\n\nDelete pool(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `POOL_NAMES` — text\n- `--all`, `-a` — Delete all pools.\n- `--purge`, `-p` — Tear down pools in failed status.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky jobs pool logs`\n\nTail or sync down logs of a pool.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this pool.\n- `--sync-down`, `-s` — Sync down logs to the local machine. Can be combined with --controller or worker ID to narrow scope.\n- `--tail` — The number of lines to display from the end of the log file. Default is None, which means print all lines.\n- `POOL_NAME` — text\n- `WORKER_IDS` — integer\n\n### `sky jobs pool status`\n\nShow statuses of pools.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `POOL_NAMES` — text\n- `--all`, `-a` — Show all workers.\n\n### `sky jobs queue`\n\nShow statuses of managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--limit`, `-l` (default: `50`) — Number of jobs to show, default is 50, use \"-a/--all\" to show all jobs.\n- `--refresh`, `-r` — Query the latest statuses, restarting the jobs controller if stopped.\n- `--skip-finished` — Show only pending/running jobs' information.\n- `-s`, `--status` — Filter by status, comma-separated (e.g. -s FAILED,FAILED_SETUP). A bare -s (no value) is a deprecated alias for --skip-finished.\n- `--since` — Show only jobs submitted within this time window, relative to now (e.g. \"30m\", \"48h\", \"7d\", \"2w\"). A bare number is seconds. Mutually exclusive with --after.\n- `--after` — Show only jobs submitted at or after this absolute local time (e.g. \"2026-01-13\" or \"2026-01-13 15:30:00\"). Mutually exclusive with --since.\n- `--before` — Show only jobs submitted at or before this absolute local time (e.g. \"2026-01-13\" or \"2026-01-13 15:30:00\").\n- `--infra` — Show only jobs running on this infrastructure. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/u...\n- `--all-users`, `-u` — Show jobs from all users.\n- `--all`, `-a` — Show all jobs.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## SkyServe Commands\n\nSkyServe CLI (multi-region, multi-cloud serving).\n\n### `sky serve down`\n\nTeardown service(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_NAMES` — text\n- `--all`, `-a` — Tear down all services.\n- `--purge`, `-p` — Tear down services in failed status.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--replica-id` — Tear down a given replica\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky serve logs`\n\nTail or sync down logs of a service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this service.\n- `--load-balancer` — Show the load balancer logs of this service.\n- `--sync-down`, `-s` — Sync down logs to the local machine. Can be combined with --controller, --load-balancer, or a replica ID to narrow scope.\n- `--tail` — The number of lines to display from the end of the log file. Default is None, which means print all lines.\n- `SERVICE_NAME` — text\n- `REPLICA_IDS` — integer\n\n### `sky serve status`\n\nShow statuses of SkyServe services.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--endpoint` — Show service endpoint.\n- `SERVICE_NAMES` — text\n\n### `sky serve up`\n\nLaunch a SkyServe service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_YAML` — text\n- `--service-name`, `-n` — A service name. Unique for each service. If not provided, a unique name is autogenerated.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky serve update`\n\nUpdate a SkyServe service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_NAME` — text\n- `SERVICE_YAML` — text\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--mode` (default: `rolling`) — Update mode. If \"rolling\", SkyServe will update the service with rolling update. If \"blue_green\", SkyServe will update the service with blue-green update.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n## Storage Commands\n\nSkyPilot Storage CLI.\n\n### `sky storage delete`\n\nDelete storage objects.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--all`, `-a` — Delete all storage objects.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky storage ls`\n\nList storage objects managed by SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n\n## Volume Commands\n\nSkyPilot Volumes CLI.\n\n### `sky volumes apply`\n\nApply a volume.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--name`, `-n` — Volume name. Override the name defined in the YAML.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, or k8s/context-name.Examples: k8s, k8s/my-context, runpod/US/US-CA-2. Override the infra defined in the YAML.\n- `--type` — Volume type. Override the type defined in the YAML.\n- `--size` — Volume size. Override the size defined in the YAML.\n- `--use-existing`, `--no-use-existing` — Whether to use an existing volume. Override the use_existing defined in the YAML.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky volumes delete`\n\nDelete volumes.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--all`, `-a` — Delete all volumes.\n- `--purge`, `-p` — Forcibly delete the volume from the volumes table even if the deletion API fails.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky volumes ls`\n\nList volumes managed by SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--verbose`, `-v` — Show all information in full.\n- `--refresh`, `-r` — Refresh volume state from cloud APIs before listing. Without this flag, cached data is returned which is updated periodically by the background daemon.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## API Server Commands\n\nSkyPilot API server commands.\n\n### `sky api cancel`\n\nCancel a request running on SkyPilot API server.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `REQUEST_IDS` — text\n- `--all`, `-a` — Cancel all your requests.\n- `--all-users`, `-u` — Cancel all requests from all users. On a server with per-user auth, a non-admin cancels only their own.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky api info`\n\nShows the SkyPilot API server URL.\n\n**Options:**\n\n- `--output`, `-o` (default: `table`) — Output forma","isInternal":false,"tokens":13509,"sizeBytes":50786},{"name":"examples.md","path":"agent/skills/skypilot/references/examples.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/examples.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# SkyPilot Examples Reference\n\nCopy-paste-ready YAML examples and CLI commands for common SkyPilot workflows.\n\n---\n\n## 1. Basic Examples\n\n### Minimal Task (Just a Run Command)\n\nThe simplest possible SkyPilot task -- just a command to run.\n\n```yaml\n# minimal.yaml\nname: hello-world\n\nrun: |\n echo \"Hello from SkyPilot!\"\n hostname\n uname -a\n```\n\n```bash\n# Launch on any available cloud (cheapest option selected automatically)\nsky launch -c hello minimal.yaml\n\n# Tear down when done\nsky down hello\n```\n\n### GPU Task with Specific Accelerator\n\nRequest a specific GPU type and count.\n\n```yaml\n# gpu-task.yaml\nname: gpu-check\n\nresources:\n # Request 1x A100 GPU\n accelerators: A100:1\n\nsetup: |\n pip install torch\n\nrun: |\n nvidia-smi\n python -c \"import torch; print(f'CUDA available: {torch.cuda.is_available()}'); print(f'GPU: {torch.cuda.get_device_name(0)}')\"\n```\n\n```bash\n# Launch on any cloud that has A100\nsky launch -c gpu-test gpu-task.yaml\n\n# Or pin to a specific cloud\nsky launch -c gpu-test gpu-task.yaml --infra aws\n\n# Tear down\nsky down gpu-test\n```\n\n### Task with Setup and Workdir\n\nSync local code and install dependencies before running.\n\n```yaml\n# train-task.yaml\nname: my-training\n\n# Sync current directory to ~/sky_workdir on the remote machine\nworkdir: .\n\nresources:\n accelerators: L4:1\n\n# Setup runs once on cluster creation, cached on subsequent runs\nsetup: |\n pip install torch torchvision\n pip install -r requirements.txt\n\n# Run executes your main command\nrun: |\n python train.py --epochs 10\n```\n\n```bash\n# Launch -- setup runs first, then the run command\nsky launch -c train train-task.yaml\n\n# Re-run with different args (setup is cached, only run executes)\nsky exec train train-task.yaml\n```\n\n### Task with Environment Variables and Secrets\n\nPass configuration via envs and sensitive values via secrets.\n\n```yaml\n# env-task.yaml\nname: training-with-config\n\nresources:\n accelerators: A100:1\n\n# Environment variables -- visible in logs and dashboard\nenvs:\n MODEL_NAME: bert-base-uncased\n BATCH_SIZE: \"32\"\n LEARNING_RATE: \"1e-4\"\n\n# Secrets -- redacted in logs and dashboard\nsecrets:\n HF_TOKEN: null # Will be passed via CLI\n WANDB_API_KEY: null # Will be passed via CLI\n\nsetup: |\n pip install transformers datasets wandb\n\nrun: |\n echo \"Training model: $MODEL_NAME with batch size $BATCH_SIZE\"\n wandb login $WANDB_API_KEY\n python train.py \\\n --model $MODEL_NAME \\\n --batch-size $BATCH_SIZE \\\n --lr $LEARNING_RATE\n```\n\n```bash\n# Pass secrets and override envs from the CLI\nHF_TOKEN=hf_abc123 WANDB_API_KEY=wk_xyz789 \\\n sky launch -c train env-task.yaml \\\n --secret HF_TOKEN --secret WANDB_API_KEY \\\n --env BATCH_SIZE=64\n\n# Secrets can also be specified inline\nsky launch -c train env-task.yaml \\\n --secret HF_TOKEN=hf_abc123 \\\n --secret WANDB_API_KEY=wk_xyz789\n```\n\n### Inline Command (No YAML File)\n\nRun a quick command without writing a YAML file.\n\n```bash\n# Launch a GPU instance and run nvidia-smi\nsky launch -c quick --gpus H100 -- nvidia-smi\n\n# Run a Python script on an existing cluster\nsky exec quick -- python -c \"import torch; print(torch.cuda.device_count())\"\n\n# Launch with specific cloud and region\nsky launch -c quick --gpus A100 --infra gcp/us-central1 -- nvidia-smi\n```\n\n### Multi-Node Task\n\nLaunch a task across multiple nodes.\n\n```yaml\n# multi-node.yaml\nname: multi-node-check\n\n# Launch 2 nodes (1 head + 1 worker)\nnum_nodes: 2\n\nresources:\n accelerators: L4:1\n\nrun: |\n echo \"Node rank: $SKYPILOT_NODE_RANK\"\n echo \"Total nodes: $SKYPILOT_NUM_NODES\"\n echo \"All node IPs:\"\n echo \"$SKYPILOT_NODE_IPS\"\n echo \"GPUs per node: $SKYPILOT_NUM_GPUS_PER_NODE\"\n nvidia-smi\n```\n\n```bash\nsky launch -c multi multi-node.yaml\n```\n\n---\n\n## 2. Training Examples\n\n### Single-GPU PyTorch Training\n\nA complete single-GPU training example with local code sync.\n\n```yaml\n# single-gpu-train.yaml\nname: pytorch-single-gpu\n\nworkdir: .\n\nresources:\n accelerators: A100:1\n disk_size: 256\n\nenvs:\n NUM_EPOCHS: \"10\"\n BATCH_SIZE: \"32\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n python train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c train single-gpu-train.yaml\nsky logs train # Stream logs\nsky down train # Tear down when done\n```\n\n### Multi-GPU Single-Node Training\n\nUse all GPUs on a single node with torchrun.\n\n```yaml\n# multi-gpu-train.yaml\nname: pytorch-multi-gpu\n\nworkdir: .\n\nresources:\n # 4x A100 GPUs on a single node\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n NUM_EPOCHS: \"10\"\n BATCH_SIZE: \"64\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_port=12345 \\\n train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c multi-gpu multi-gpu-train.yaml\n```\n\n### Multi-Node Distributed Training (PyTorch DDP)\n\nDistributed training across 2 nodes with 8 GPUs each using torchrun.\n\n```yaml\n# multi-node-ddp.yaml\nname: pytorch-ddp-training\n\nworkdir: .\n\n# 2 nodes, each with 8 GPUs = 16 GPUs total\nnum_nodes: 2\n\nresources:\n accelerators: A100-80GB:8\n disk_size: 512\n\nenvs:\n NUM_EPOCHS: \"20\"\n BATCH_SIZE: \"256\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n # SKYPILOT_NODE_IPS contains all node IPs, one per line\n # The first IP is always the head node (rank 0)\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=12345 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c ddp-train multi-node-ddp.yaml\n\n# SSH into the head node to check progress\nssh ddp-train\n```\n\n### Training with Checkpointing to Cloud Storage\n\nSave checkpoints to S3/GCS for persistence across runs.\n\n```yaml\n# train-with-checkpoints.yaml\nname: training-with-checkpoints\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n CHECKPOINT_BUCKET: my-training-checkpoints\n\nfile_mounts:\n # Mount an S3 bucket for writing checkpoints\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\n # Mount a dataset bucket (read-only streaming)\n /data:\n source: s3://my-training-data/imagenet\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision boto3\n\nrun: |\n # Training script should save checkpoints to /checkpoints\n # and load dataset from /data\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n train.py \\\n --data-dir /data \\\n --checkpoint-dir /checkpoints \\\n --resume-from-checkpoint\n```\n\n```bash\n# Launch training\nsky launch -c train train-with-checkpoints.yaml\n\n# Re-launch after preemption or failure -- resumes from last checkpoint\nsky launch -c train train-with-checkpoints.yaml\n```\n\n### Training on Spot Instances with Auto-Recovery (Managed Job)\n\nRun long training on cheap spot instances with automatic recovery from preemptions.\n\n```yaml\n# spot-train.yaml\nname: spot-training-resnet\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n # Use spot instances for ~3x cost savings\n use_spot: true\n disk_size: 512\n\n # Recovery strategy when spot instance is preempted\n job_recovery:\n strategy: EAGER_NEXT_REGION # Move to next region on preemption\n max_restarts_on_errors: 3 # Retry up to 3 times on user code errors\n\nenvs:\n CHECKPOINT_BUCKET: my-spot-training-ckpts\n\nfile_mounts:\n # Persistent checkpoint storage -- survives preemptions\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision\n\nrun: |\n # IMPORTANT: Your training script must handle checkpoint resume.\n # SkyPilot recovers the cluster; your script recovers the state.\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n train.py \\\n --checkpoint-dir /checkpoints \\\n --resume-from-checkpoint\n```\n\n```bash\n# Launch as a managed job (NOT sky launch -- use sky jobs launch)\nsky jobs launch spot-train.yaml\n\n# Monitor the job\nsky jobs queue\n\n# Stream logs\nsky jobs logs \n```\n\n### Fine-Tuning a HuggingFace Model\n\nFine-tune a HuggingFace model with LoRA, using secrets for authentication.\n\n```yaml\n# finetune-hf.yaml\nname: llama3-finetune\n\nworkdir: .\n\nresources:\n accelerators: A100-80GB:4\n disk_size: 512\n disk_tier: best\n\nenvs:\n MODEL_ID: meta-llama/Meta-Llama-3.1-8B-Instruct\n DATASET: yahma/alpaca-cleaned\n CHECKPOINT_BUCKET: my-finetune-checkpoints\n\n# HuggingFace token is a secret -- redacted in logs/dashboard\nsecrets:\n HF_TOKEN: null # Pass via CLI: --secret HF_TOKEN=hf_xxx\n\nfile_mounts:\n /output:\n name: ${CHECKPOINT_BUCKET}\n mode: MOUNT\n\nsetup: |\n pip install torch==2.4.0 torchvision\n pip install transformers peft trl datasets accelerate\n pip install bitsandbytes\n\n # Download the model (needs HF_TOKEN for gated models)\n huggingface-cli download $MODEL_ID \\\n --token $HF_TOKEN \\\n --local-dir /tmp/model\n\nrun: |\n python finetune_lora.py \\\n --model-path /tmp/model \\\n --dataset $DATASET \\\n --output-dir /output/lora-adapter \\\n --num-epochs 3 \\\n --batch-size 4 \\\n --learning-rate 2e-4\n```\n\n```bash\n# Launch with HF_TOKEN secret\nHF_TOKEN=hf_abc123 sky launch -c finetune finetune-hf.yaml --secret HF_TOKEN\n```\n\n---\n\n## 3. Model Serving Examples\n\n### Basic vLLM Serving (Single GPU)\n\nServe a model with vLLM behind a SkyServe endpoint.\n\n```yaml\n# vllm-serve.yaml\nname: vllm-llama\n\nresources:\n accelerators: A100:1\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\n# SkyServe configuration\nservice:\n # Readiness check endpoint (must return 200 when ready)\n readiness_probe: /v1/models\n # Fixed number of replicas\n replicas: 1\n```\n\n```bash\n# Deploy as a service\nHF_TOKEN=hf_xxx sky serve up vllm-serve.yaml -n llama-svc --secret HF_TOKEN\n\n# Get the endpoint URL\nsky serve status llama-svc --endpoint\n\n# Test the endpoint\nENDPOINT=$(sky serve status llama-svc --endpoint)\ncurl $ENDPOINT/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n }'\n\n# Tear down\nsky serve down llama-svc\n```\n\n### vLLM with Autoscaling\n\nScale replicas automatically based on query load.\n\n```yaml\n# vllm-autoscale.yaml\nname: vllm-autoscaling\n\nresources:\n accelerators: A100:1\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe:\n path: /v1/models\n # Wait 20 minutes for model loading before checking readiness\n initial_delay_seconds: 1200\n replica_policy:\n min_replicas: 1\n max_replicas: 5\n # Scale up when each replica handles more than 5 QPS\n target_qps_per_replica: 5\n # Wait 5 minutes before adding replicas\n upscale_delay_seconds: 300\n # Wait 20 minutes before removing replicas\n downscale_delay_seconds: 1200\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up vllm-autoscale.yaml -n llama-auto --secret HF_TOKEN\n```\n\n### Serving on Spot Instances\n\nUse spot instances for serving replicas to reduce cost.\n\n```yaml\n# vllm-spot-serve.yaml\nname: vllm-spot-serving\n\nresources:\n accelerators: A100:1\n ports: 8000\n # Use spot instances for replicas -- SkyServe handles recovery\n use_spot: true\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe: /v1/models\n replica_policy:\n # Keep at least 2 replicas for high availability with spot\n min_replicas: 2\n max_replicas: 6\n target_qps_per_replica: 5\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up vllm-spot-serve.yaml -n llama-spot --secret HF_TOKEN\n```\n\n### Multi-Model Serving\n\nServe multiple models by deploying separate services.\n\n```yaml\n# serve-llama-70b.yaml\nname: llama-70b-service\n\nresources:\n # 70B model needs multi-GPU with tensor parallelism\n accelerators: A100-80GB:4\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-70B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --tensor-parallel-size 4 \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe: /v1/models\n replicas: 2\n```\n\n```bash\n# Deploy multiple model services\nHF_TOKEN=hf_xxx sky serve up vllm-serve.yaml -n llama-8b --secret HF_TOKEN\nHF_TOKEN=hf_xxx sky serve up serve-llama-70b.yaml -n llama-70b --secret HF_TOKEN\n\n# Check all services\nsky serve status\n\n# Get endpoints\nsky serve status llama-8b --endpoint\nsky serve status llama-70b --endpoint\n```\n\n### TGI (Text Generation Inference) Example\n\nServe a model using HuggingFace TGI with Docker.\n\n```yaml\n# tgi-serve.yaml\nname: tgi-service\n\nresources:\n accelerators: A100:1\n ports: 8080\n\nenvs:\n MODEL_ID: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nrun: |\n docker run --gpus all --shm-size 1g \\\n -p 8080:80 \\\n -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \\\n -v ~/data:/data \\\n ghcr.io/huggingface/text-generation-inference \\\n --model-id $MODEL_ID\n\nservice:\n readiness_probe: /health\n replicas: 2\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up tgi-serve.yaml -n tgi-llama --secret HF_TOKEN\nsky serve status tgi-llama --endpoint\n```\n\n### Service with Custom Readiness Probe (POST)\n\nUse a POST request with custom data for the readiness check.\n\n```yaml\n# custom-probe-serve.yaml\nname: custom-model-service\n\nresources:\n accelerators: A100:1\n ports: 8080\n\nsetup: |\n pip install my-model-server\n\nrun: |\n python -m my_model_server --port 8080\n\nservice:\n readiness_probe:\n path: /v1/predict\n # Use POST with custom payload for readiness checking\n post_data: '{\"text\": \"health check\", \"max_tokens\": 1}'\n # Wait 30 minutes for model loading\n initial_delay_seconds: 1800\n # Timeout for each probe request\n timeout_seconds: 30\n replica_policy:\n min_replicas: 1\n max_replicas: 3\n target_qps_per_replica: 10\n```\n\n```bash\nsky serve up custom-probe-serve.yaml -n custom-svc\n```\n\n---\n\n## 4. File Mounts and Storage Examples\n\n### Sync Local Directory (workdir)\n\nThe workdir syncs your local directory to `~/sky_workdir` on the remote cluster.\n\n```yaml\n# workdir-example.yaml\nname: local-code-sync\n\n# Sync current directory to ~/sky_workdir\nworkdir: .\n\nresources:\n accelerators: L4:1\n\nsetup: |\n pip install -r requirements.txt\n\nrun: |\n # Commands run under ~/sky_workdir\n python train.py\n```\n\n```bash\n# Any changes to local files are synced on each sky launch/exec\nsky launch -c dev workdir-example.yaml\n\n# Re-sync and re-run (setup is cached)\nsky exec dev workdir-example.yaml\n```\n\n### Mount S3 Bucket (mode: MOUNT)\n\nStream data from S3 without downloading -- read-only, low disk usage.\n\n```yaml\n# s3-mount.yaml\nname: s3-streaming\n\nresources:\n accelerators: A100:1\n\nfile_mounts:\n # MOUNT mode: read-only, streams data on access, no local copy\n /data/imagenet:\n source: s3://my-imagenet-bucket/train\n mode: MOUNT\n\nrun: |\n python train.py --data-dir /data/imagenet\n```\n\n```bash\nsky launch -c train s3-mount.yaml\n```\n\n### Mount with Cache (mode: MOUNT_CACHED)\n\nCache accessed data locally for fast random access -- ideal for model weights.\n\n```yaml\n# mount-cached.yaml\nname: cached-model-weights\n\nresources:\n accelerators: A100:1\n disk_size: 512 # Need enough disk for cached data\n\nfile_mounts:\n # MOUNT_CACHED: caches accessed files locally for fast random access\n # Good for model weights that are read repeatedly\n /models/llama:\n source: s3://my-model-weights/llama-3.1-8b\n mode: MOUNT_CACHED\n\nrun: |\n python serve.py --model-path /models/llama\n```\n\n```bash\nsky launch -c serve mount-cached.yaml\n```\n\n### Copy from Cloud Storage (mode: COPY)\n\nDownload data fully -- read-write access, full local copy.\n\n```yaml\n# copy-mount.yaml\nname: full-copy\n\nresources:\n accelerators: A100:1\n # Need enough disk for the full dataset\n disk_size: 1024\n\nfile_mounts:\n # COPY mode: full download, read-write, supports modification\n /data/dataset:\n source: s3://my-dataset-bucket/training-data\n mode: COPY\n\nrun: |\n # Can read AND write to the copied data\n python preprocess.py --input /data/dataset --output /data/dataset/processed\n python train.py --data-dir /data/dataset/processed\n```\n\n```bash\nsky launch -c train copy-mount.yaml\n```\n\n### Mount GCS Bucket\n\nMount a Google Cloud Storage bucket.\n\n```yaml\n# gcs-mount.yaml\nname: gcs-data\n\nresources:\n infra: gcp\n accelerators: A100:1\n\nfile_mounts:\n # GCS bucket mount\n /data:\n source: gs://my-gcs-bucket/datasets\n mode: MOUNT\n\n # Create a new GCS bucket for output\n /output:\n name: my-training-output-bucket\n store: gcs\n mode: MOUNT\n\nrun: |\n python train.py --data /data --output /output\n```\n\n```bash\nsky launch -c gcp-train gcs-mount.yaml\n```\n\n### Using Volumes on Kubernetes\n\nKubernetes persistent and ephemeral volumes.\n\n```yaml\n# k8s-volumes.yaml\nname: k8s-with-volumes\n\nresources:\n infra: k8s\n accelerators: A100:1\n\n# Volumes (Kubernetes only)\nvolumes:\n # Persistent volume -- data survives cluster restarts\n /mnt/data: my-persistent-volume\n\n # Ephemeral volume -- fast scratch space, lost on restart\n /mnt/scratch:\n size: 200Gi\n\nrun: |\n # /mnt/data persists across restarts\n # /mnt/scratch is fast ephemeral storage\n python train.py \\\n --data-dir /mnt/data \\\n --cache-dir /mnt/scratch\n```\n\n```bash\nsky launch -c k8s-train k8s-volumes.yaml\n```\n\n### Multiple Mounts in One Task\n\nCombine different mount types in a single task.\n\n```yaml\n# multi-mount.yaml\nname: multi-mount-training\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n OUTPUT_BUCKET: my-training-outputs\n\nfile_mounts:\n # Sync local config files\n /configs: ./configs\n\n # Stream training data from S3 (read-only)\n /data/train:\n source: s3://my-dataset/train\n mode: MOUNT\n\n # Cache model weights locally for fast loading\n /models/pretrained:\n source: s3://my-models/llama-3.1-8b\n mode: MOUNT_CACHED\n\n # Writable output bucket for checkpoints\n /output:\n name: ${OUTPUT_BUCKET}\n store: s3\n mode: MOUNT\n\n # Copy evaluation data (small, needs read-write)\n /data/eval:\n source: s3://my-dataset/eval\n mode: COPY\n\nsetup: |\n pip install torch transformers datasets\n\nrun: |\n python train.py \\\n --config /configs/train_config.yaml \\\n --data-dir /data/train \\\n --eval-dir /data/eval \\\n --pretrained-model /models/pretrained \\\n --output-dir /output\n```\n\n```bash\nsky launch -c train multi-mount.yaml\n```\n\n---\n\n## 5. Multi-Cloud and Fallback Examples\n\n### any_of: Multiple GPU Types\n\nLet SkyPilot pick the cheapest available GPU from a set.\n\n```yaml\n# any-gpu.yaml\nname: flexible-gpu-training\n\nresources:\n # SkyPilot picks the cheapest available option\n any_of:\n - accelerators: H100:1\n - accelerators: A100-80GB:1\n - accelerators: A100:1\n - accelerators: L40S:1\n\nrun: |\n nvidia-smi\n python train.py\n```\n\n```bash\nsky launch -c train any-gpu.yaml\n```\n\n### any_of: Multiple Clouds for Same GPU\n\nTry different clouds for the same GPU type.\n\n```yaml\n# any-cloud.yaml\nname: multi-cloud-a100\n\nresources:\n accelerators: A100-80GB:8\n any_of:\n - infra: aws/us-east-1\n - infra: gcp/us-central1\n - infra: azure/eastus\n\nrun: |\n nvidia-smi\n python train.py\n```\n\n```bash\nsky launch -c train any-cloud.yaml\n```\n\n### ordered: Strict Priority\n\nTry clouds/regions in a specific order -- first match wins.\n\n```yaml\n# ordered-priority.yaml\nname: priority-training\n\nresources:\n ordered:\n # Try AWS us-east-1 first (our reserved instances)\n - infra: aws/us-east-1\n accelerators: H100:8\n # Then try GCP us-central1\n - infra: gcp/us-central1\n accelerators: H100:8\n # Fall back to A100s on any cloud\n - accelerators: A100-80GB:8\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train ordered-priority.yaml\n```\n\n### K8s + Cloud Fallback\n\nTry Kubernetes first, fall back to cloud if K8s has no capacity.\n\n```yaml\n# k8s-fallback.yaml\nname: k8s-or-cloud\n\nresources:\n accelerators: A100:1\n ordered:\n # Try on-prem K8s cluster first (free)\n - infra: k8s/my-onprem-cluster\n # Fall back to AWS\n - infra: aws\n # Fall back to GCP\n - infra: gcp\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train k8s-fallback.yaml\n```\n\n### Region Preference with Wildcards\n\nUse wildcards to match multiple regions or zones.\n\n```yaml\n# wildcard-region.yaml\nname: us-region-only\n\nresources:\n accelerators: A100:8\n any_of:\n # Any zone in us-east-1\n - infra: aws/us-east-1\n # Any zone in us-west-2\n - infra: aws/us-west-2\n # Any US region on GCP\n - infra: gcp/us-*\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train wildcard-region.yaml\n```\n\n### Cost Optimization: Cheapest Across All Clouds\n\nOmit `infra` to let SkyPilot search all enabled clouds for the cheapest option.\n\n```yaml\n# cheapest.yaml\nname: cheapest-option\n\nresources:\n # No infra specified -- SkyPilot searches all enabled clouds\n accelerators: A100:1\n\nrun: |\n python train.py\n```\n\n```bash\n# Dry run to see the cheapest option without launching\nsky launch cheapest.yaml --dryrun\n\n# Launch on the cheapest cloud/region\nsky launch -c train cheapest.yaml\n\n# Check GPU prices across clouds\nsky gpus list A100\n```\n\n---\n\n## 6. Managed Job Examples\n\n### Basic Managed Job\n\nManaged jobs run on a dedicated controller that monitors and manages your job.\n\n```yaml\n# basic-job.yaml\nname: data-processing\n\nresources:\n cpus: 16+\n memory: 64+\n\nsetup: |\n pip install pandas pyarrow\n\nrun: |\n python process_data.py --output /tmp/results\n```\n\n```bash\n# Launch as a managed job (runs on a separate controller)\nsky jobs launch basic-job.yaml\n\n# Check status\nsky jobs queue\n\n# Stream logs\nsky jobs logs 1\n\n# Cancel the job\nsky jobs cancel 1\n```\n\n### Spot Job with FAILOVER Recovery\n\nAutomatically retry in the same region first, then try other regions.\n\n```yaml\n# spot-failover.yaml\nname: training-failover\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n # FAILOVER: retry same region first, then move to next region\n strategy: FAILOVER\n # Retry up to 3 times on user code errors\n max_restarts_on_errors: 3\n\nfile_mounts:\n /checkpoints:\n name: my-training-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch spot-failover.yaml\n```\n\n### Spot Job with EAGER_NEXT_REGION Recovery\n\nMove to a different region immediately on preemption -- good for spot shortages.\n\n```yaml\n# spot-eager.yaml\nname: training-eager-recovery\n\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n # EAGER_NEXT_REGION: immediately try a different region on preemption\n # Best for spot instances where preemption signals regional shortage\n strategy: EAGER_NEXT_REGION\n\nfile_mounts:\n /checkpoints:\n name: my-training-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch spot-eager.yaml\n```\n\n### Hyperparameter Sweep\n\nLaunch multiple managed jobs with different hyperparameters.\n\n```yaml\n# sweep.yaml\nname: hp-sweep\n\nresources:\n accelerators: A100:1\n\nenvs:\n LR: \"1e-4\"\n WEIGHT_DECAY: \"0.01\"\n BATCH_SIZE: \"32\"\n\nfile_mounts:\n /results:\n name: sweep-results\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers wandb\n\nrun: |\n python train.py \\\n --lr $LR \\\n --weight-decay $WEIGHT_DECAY \\\n --batch-size $BATCH_SIZE \\\n --output-dir /results/lr-${LR}_wd-${WEIGHT_DECAY}_bs-${BATCH_SIZE}\n```\n\n```bash\n# Launch multiple jobs with different hyperparameters\nfor lr in 1e-3 1e-4 1e-5; do\n for wd in 0.01 0.1; do\n sky jobs launch sweep.yaml \\\n --env LR=$lr --env WEIGHT_DECAY=$wd \\\n --name sweep-lr${lr}-wd${wd}\n done\ndone\n\n# Monitor all sweep jobs\nsky jobs queue\n```\n\n### Job with max_restarts_on_errors\n\nAutomatically restart on user code errors (non-zero exit codes).\n\n```yaml\n# auto-restart.yaml\nname: resilient-training\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n strategy: FAILOVER\n # Restart up to 5 times if the training script crashes\n max_restarts_on_errors: 5\n\nfile_mounts:\n /checkpoints:\n name: my-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch auto-restart.yaml\n```\n\n### Job with recover_on_exit_codes for NCCL Errors\n\nAlways retry on specific exit codes (such as NCCL timeouts).\n\n```yaml\n# nccl-recovery.yaml\nname: nccl-resilient-training\n\nnum_nodes: 4\n\nresources:\n accelerators: H100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n # Exit code 33 = NCCL timeout (convention in many training scripts)\n # These restarts do NOT count toward max_restarts_on_errors\n recover_on_exit_codes: [33, 34]\n\nfile_mounts:\n /checkpoints:\n name: my-distributed-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=12345 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch nccl-recovery.yaml\n```\n\n### Job Pool Example\n\nUse a job pool for batch processing with pre-provisioned workers.\n\n```yaml\n# pool.yaml\n# Pool configuration: pre-provisions workers with setup\nworkdir: .\n\nresources:\n accelerators: H100:1\n disk_size: 100\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B\n\nsetup: |\n pip install vllm transformers datasets\n # Pre-download model so all jobs can use it immediately\n huggingface-cli download $MODEL_NAME --local-dir /tmp/model\n\npool:\n workers: 3\n```\n\n```yaml\n# job.yaml\n# Job that runs on the pool\nname: batch-inference\n\nresources:\n accelerators: H100:1\n\nrun: |\n echo \"Processing partition ${SKYPILOT_JOB_RANK} of ${SKYPILOT_NUM_JOBS}\"\n python classify.py \\\n --job-rank ${SKYPILOT_JOB_RANK} \\\n --num-jobs ${SKYPILOT_NUM_JOBS} \\\n --model-path /tmp/model \\\n --output-dir /results\n```\n\n```bash\n# Create the pool with 3 workers\nsky jobs pool apply -p my-pool pool.yaml\n\n# Submit 10 jobs to the pool (distributed across 3 workers)\nsky jobs launch -p my-pool --num-jobs 10 job.yaml\n\n# Check pool status\nsky jobs pool status my-pool\n\n# Check job status\nsky jobs queue\n\n# Tear down pool when done\nsky jobs pool down my-pool\n```\n\n---\n\n## 7. Distributed Training Examples\n\n### PyTorch DDP on 4 Nodes (Complete Example)\n\nFull distributed training with torchrun and NCCL configuration.\n\n```yaml\n# ddp-4node.yaml\nname: ddp-4node-training\n\nworkdir: .\n\nnum_nodes: 4\n\nresources:\n accelerators: H100:8\n disk_size: 1024\n\nenvs:\n # NCCL configuration for optimal multi-node communication\n NCCL_DEBUG: INFO\n NCCL_SOCKET_IFNAME: eth0\n # Training hyperparameters\n GLOBAL_BATCH_SIZE: \"512\"\n NUM_EPOCHS: \"50\"\n CHECKPOINT_BUCKET: my-ddp-checkpoints\n\nfile_mounts:\n # Training data (streaming)\n /data:\n source: s3://my-training-data/preprocessed\n mode: MOUNT\n\n # Checkpoint storage (persistent, writable)\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision\n pip install wandb tensorboard\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_GPUS=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))\n\n echo \"Node rank: $SKYPILOT_NODE_RANK / $SKYPILOT_NUM_NODES\"\n echo \"Master: $MASTER_ADDR\"\n echo \"Total GPUs: $NUM_GPUS\"\n\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=29500 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train_ddp.py \\\n --data-dir /data \\\n --checkpoint-dir /checkpoints \\\n --global-batch-size $GLOBAL_BATCH_SIZE \\\n --epochs $NUM_EPOCHS \\\n --resume\n```\n\n```bash\n# Launch distributed training\nsky launch -c ddp4 ddp-4node.yaml\n\n# SSH into head node\nssh ddp4\n\n# Check worker node\nssh ddp4-worker1\n\n# Monitor logs\nsky logs ddp4\n```\n\n### DeepSpeed ZeRO-3 on 2 Nodes\n\nDistributed training with DeepSpeed, including hostfile generation.\n\n```yaml\n# deepspeed-2node.yaml\nname: deepspeed-zero3\n\nnum_nodes: 2\n\nresources:\n accelerators: A100-80GB:8\n disk_size: 1024\n image_id: docker:nvidia/cuda:12.1.1-devel-ubuntu20.04\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B\n DEEPSPEED_ENVS: \"MODEL_NAME,SKYPILOT_NODE_RANK\"\n\nsetup: |\n pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121\n pip install deepspeed==0.14.4\n pip install transformers datasets accelerate\n sudo apt-get update && sudo apt-get -y install pdsh\n\nrun: |\n # Only the head node (rank 0) launches DeepSpeed\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n # Generate hostfile for DeepSpeed\n HOSTFILE=/tmp/hostfile\n python -c \"\n import os\n n_gpus = os.environ['SKYPILOT_NUM_GPUS_PER_NODE']\n ips = os.environ['SKYPILOT_NODE_IPS'].splitlines()\n with open('$HOSTFILE', 'w') as f:\n for ip in ips:\n f.write(f'{ip} slots={n_gpus}\\n')\n \"\n echo \"Hostfile:\"\n cat $HOSTFILE\n\n # Generate .deepspeed_env for environment propagation\n python -c \"\n import os\n envs = os.environ.get('DEEPSPEED_ENVS', '').split(',')\n with open('.deepspeed_env', 'w') as f:\n for var in envs:\n f.write(f'{var}=\\\"{os.getenv(var, \\\"\\\")}\\\"\\n')\n \"\n\n deepspeed \\\n --hostfile $HOSTFILE \\\n train_deepspeed.py \\\n --model-name $MODEL_NAME \\\n --deepspeed \\\n --deepspeed-config ds_config_zero3.json\n fi\n```\n\n```bash\nsky launch -c ds2 deepspeed-2node.yaml\n```\n\n### FSDP Training Example\n\nMulti-node Fully Sharded Data Parallel training with HuggingFace Accelerate.\n\n```yaml\n# fsdp-train.yaml\nname: fsdp-training\n\nworkdir: .\n\nnum_nodes: 2\n\nresources:\n accelerators: H100:8\n disk_size: 1024\n\nenvs:\n MODEL_ID: meta-llama/Llama-3.1-8B\n CHECKPOINT_BUCKET: my-fsdp-checkpoints\n\nsecrets:\n HF_TOKEN: null\n\nfile_mounts:\n /output:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch --index-url https://download.pytorch.org/whl/cu128\n pip install transformers accelerate trl peft datasets\n pip install deepspeed # Needed for some FSDP optimizations\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_PROCESSES=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))\n\n accelerate launch \\\n --config_file configs/fsdp_config.yaml \\\n --num_machines $SKYPILOT_NUM_NODES \\\n --num_processes $NUM_PROCESSES \\\n --machine_rank $SKYPILOT_NODE_RANK \\\n --main_process_ip $MASTER_ADDR \\\n --main_process_port 29500 \\\n train_fsdp.py \\\n --model-id $MODEL_ID \\\n --output-dir /output \\\n --resume-from-checkpoint\n```\n\n```bash\nHF_TOKEN=hf_xxx sky launch -c fsdp fsdp-train.yaml --secret HF_TOKEN\n```\n\n### Ray Train Multi-Node Example\n\nDistributed training using Ray Train across multiple nodes.\n\n```yaml\n# ray-train.yaml\nname: ray-train-distributed\n\nworkdir: .\n\nnum_nodes: 2\n\nresources:\n accelerators: A100:4\n\nsetup: |\n pip install \"ray[train]\" torch torchvision\n pip install transformers datasets\n\nrun: |\n # SkyPilot sets up a Ray cluster across all nodes automatically.\n # The head node's Ray address is available at the standard port.\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n # Only run training script on the head node;\n # Ray distributes the work to all nodes\n python train_ray.py \\\n --num-workers $((SKYPILOT_NUM_NODES * SKYPILOT_NUM_GPUS_PER_NODE)) \\\n --gpus-per-worker 1\n fi\n```\n\nThe `train_ray.py` script uses `ray.train`:\n\n```python\n# train_ray.py (referenced by the YAML above)\nimport ray\nfrom ray import train\nfrom ray.train.torch import TorchTrainer\nfrom ray.train import ScalingConfig\n\ndef train_func(config):\n import torch\n # Your PyTorch training code here\n # ray.train handles distributed setup automatically\n model = ...\n optimizer = ...\n for epoch in range(config[\"epochs\"]):\n train_epoch(model, optimizer)\n # Report metrics\n train.report({\"loss\": loss, \"epoch\": epoch})\n\nscaling_config = ScalingConfig(\n num_workers=args.num_workers,\n use_gpu=True,\n resources_per_worker={\"GPU\": args.gpus_per_worker},\n)\n\ntrainer = TorchTrainer(\n train_func,\n train_loop_config={\"epochs\": 10},\n scaling_config=scaling_config,\n)\nresult = trainer.fit()\n```\n\n```bash\nsky launch -c ray-train ray-train.yaml\n```\n\n---\n\n## 8. Python SDK Examples\n\n### Basic: Create Task, Set Resources, Launch\n\n```python\nimport sky\n\n# Create a task programmatically\ntask = sky.Task(\n name='hello-gpu',\n run='nvidia-smi && python -c \"import torch; print(torch.cuda.device_count())\"',\n)\ntask.set_resources(sky.Resources(\n accelerators='A100:1',\n))\n\n# Launch returns a request ID (async by default)\nrequest_id = sky.launch(task, cluster_name='my-cluster')\n\n# Wait for the launch to complete\njob_id, handle = sky.get(request_id)\nprint(f'Job ID: {job_id}')\n```\n\n### Launch from YAML File\n\n```python\nimport sky\n\n# Load task from a YAML file\ntask = sky.Task.from_yaml('train.yaml')\n\n# Launch with options\nrequest_id = sky.launch(\n task,\n cluster_name='train-cluster',\n idle_minutes_to_autostop=30, # Auto-stop after 30 min idle\n retry_until_up=True, # Keep trying if resources unavailable\n)\nresult = sky.get(request_id)\n```\n\n### Check Status and Get Cluster Info\n\n```python\nimport sky\n\n# Get all cluster statuses\nrequest_id = sky.status()\nclusters = sky.get(request_id)\n\nfor cluster in clusters:\n print(f\"Cluster: {cluster['name']}\")\n print(f\" Status: {cluster['status']}\")\n print(f\" Resources: {cluster['resources_str']}\")\n print(f\" Autostop: {cluster['autostop']} min\")\n\n# Get status for a specific cluster\nrequest_id = sky.status(cluster_names=['my-cluster'])\nclusters = sky.get(request_id)\n\n# Get job queue for a cluster\nrequest_id = sky.queue('my-cluster')\njobs = sky.get(request_id)\nfor job in jobs:\n print(f\"Job {job['job_id']}: {job['status']}\")\n```\n\n### Batch Submission (Launch Multiple Tasks)\n\n```python\nimport sky\n\n# Launch multiple training runs with different configs\nconfigs = [\n {'lr': 1e-3, 'batch_size': 32},\n {'lr': 1e-4, 'batch_size': 64},\n {'lr': 1e-5, 'batch_size': 128},\n]\n\nrequest_ids = []\nfor i, cfg in enumerate(configs):\n task = sky.Task(\n name=f'sweep-{i}',\n setup='pip install torch transformers',\n run=f'python train.py --lr {cfg[\"lr\"]} --batch-size {cfg[\"batch_size\"]}',\n )\n task.set_resources(sky.Resources(accelerators='A100:1'))\n\n request_id = sky.launch(\n task,\n cluster_name=f'sweep-{i}',\n idle_minutes_to_autostop=10,\n down=True, # Auto-teardown after idle\n )\n request_ids.append(request_id)\n print(f'Launched sweep-{i}: {request_id}')\n\n# Wait for all to complete\nfor rid in request_ids:\n result = sky.get(rid)\n print(f'Completed: {rid}')\n```\n\n### Wait for Completion and Get Results\n\n```python\nimport sky\n\n# Launch a task\ntask = sky.Task.from_yaml('train.yaml')\nrequest_id = sky.launch(task, cluster_name='train')\n\n# stream_and_get streams logs AND waits for completion\nresult = sky.stream_and_get(request_id)\n\n# Check job status\nrequest_id = sky.job_status('train', job_ids=[1])\nstatuses = sky.get(request_id)\nprint(f'Job 1 status: {statuses[1]}')\n\n# Tail logs (blocking, streams to console)\nexit_code = sky.tail_logs('train', job_id=1, follow=True)\nprint(f'Job exit code: {exit_code}')\n```\n\n### Programmatic Managed Job Launch\n\n```python\nimport sky\n\n# Create a managed job task\ntask = sky.Task(\n name='managed-training',\n setup='pip install torch transformers',\n run='python train.py --resume',\n)\ntask.set_resources(sky.Resources(\n accelerators='A100:4',\n use_spot=True,\n job_recovery={\n 'strategy': 'EAGER_NEXT_REGION',\n 'max_restarts_on_errors': 3,\n },\n))\n\n# Launch as a managed job\nrequest_id = sky.jobs.launch(task)\njob_id = sky.get(request_id)\nprint(f'Managed job ID: {job_id}')\n\n# Check managed job queue\nrequest_id = sky.jobs.queue()\njobs = sky.get(request_id)\nfor job in jobs:\n print(f\"Job {job['job_id']}: {job['job_name']} - {job['status']}\")\n```\n\n### DAG Workflow (Task Dependencies)\n\n```python\nimport sky\n\n# Create a DAG with task dependencies\nwith sky.Dag() as dag:\n # Step 1: Data preprocessing\n preprocess = sky.Task(\n name='preprocess',\n setup='pip install pandas pyarrow',\n run='python preprocess.py --output /tmp/data',\n )\n preprocess.set_resources(sky.Resources(cpus='8+', memory='32+'))\n\n # Step 2: Training (depends on preprocessing)\n train = sky.Task(\n name='train',\n setup='pip install torch transformers',\n run='python train.py --data /tmp/data',\n )\n train.set_resources(sky.Resources(accelerators='A100:4'))\n\n # Define dependency: preprocess must finish before train\n preprocess >> train\n\n# Launch the DAG (experimental)\nrequest_id = sky.launch(dag, cluster_name='pipeline')\nsky.get(request_id)\n```\n\n### Autostop Management\n\n```python\nimport sky\n\n# Set autostop: stop after 30 minutes of idleness\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=30,\n)\nsky.get(request_id)\n\n# Set autodown: tear down (not just stop) after 60 minutes idle\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=60,\n down=True,\n)\nsky.get(request_id)\n\n# Cancel autostop\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=-1, # Negative value cancels autostop\n)\nsky.get(request_id)\n\n# Stop and start a cluster\nsky.get(sky.stop('my-cluster'))\nsky.get(sky.start('my-cluster'))\n\n# Tear down a cluster\nsky.get(sky.down('my-cluster'))\n```\n\n### List GPUs Programmatically\n\n```python\nimport sky\n\n# List all available accelerators\nrequest_id = sky.list_accelerators(\n gpus_only=True,\n name_filter='A100',\n)\naccelerators = sky.get(request_id)\n\nfor acc_name, offerings in accelerators.items():\n print(f'\\n{acc_name}:')\n for offering in offerings:\n print(f' {offering}')\n\n# List accelerator counts (how many GPUs per node)\nrequest_id = sky.list_accelerator_counts(\n gpus_only=True,\n name_filter='H100',\n)\ncounts = sky.get(request_id)\nfor acc_name, available_counts in counts.items():\n print(f'{acc_name}: available in counts {available_counts}')\n\n# Check which clouds are enabled\nrequest_id = sky.enabled_clouds()\nclouds = sky.get(request_id)\nprint(f'Enabled clouds: {clouds}')\n```\n\n---\n\n## Quick Reference: SkyPilot Environment Variables\n\nThese variables are automatically set by SkyPilot on every node and are available\nin `setup` and `run` commands:\n\n| Variable | Description | Example Value |\n|----------|-------------|---------------|\n| `SKYPILOT_NODE_RANK` | Rank of the current node (0 = head) | `0` |\n| `SKYPILOT_NODE_IPS` | Newline-separated IPs of all nodes | `10.0.0.1\\n10.0.0.2` |\n| `SKYPILOT_NUM_NODES` | Total number of nodes in the cluster | `2` |\n| `SKYPILOT_NUM_GPUS_PER_NODE` | Number of GPUs on this node | `8` |\n| `SKYPILOT_TASK_ID` | Unique task identifier | `sky-2024-01-15-abc123` |\n| `SKYPILOT_CLUSTER_INFO` | JSON with cluster metadata | `{\"cloud\": \"aws\", ...}` |\n| `SKYPILOT_JOB_RANK` | Job rank within a pool (pool jobs only) | `3` |\n| `SKYPILOT_NUM_JOBS` | Total number of jobs in batch (pool jobs only) | `10` |\n\n## Quick Reference: Common CLI Patterns\n\n```bash\n# Launch and auto-stop after 30 minutes idle\nsky launch -c mycluster task.yaml -i 30\n\n# Launch with auto-teardown (not just stop) after idle\nsky launch -c mycluster task.yaml -i 30 --down\n\n# Launch detached (don't stream logs)\nsky launch -c mycluster task.yaml -d\n\n# Retry until resources are available\nsky launch -c mycluster task.yaml -r\n\n# Override GPU from CLI\nsky launch -c mycluster task.yaml --gpus H100:4\n\n# Override cloud from CLI\nsky launch -c mycluster task.yaml --infra aws/us-east-1\n\n# Execute on existing cluster (skip setup)\nsky exec mycluster another-task.yaml\n\n# Execute inline command on existing cluster\nsky exec mycluster -- python eval.py --checkpoint /tmp/best.pt\n\n# Check GPU pricing\nsky gpus list H100\nsky gpus list A100 --all-regions\nsky gpus list --infra aws\n```\n","isInternal":false,"tokens":11673,"sizeBytes":41154},{"name":"job-investigation.md","path":"agent/skills/skypilot/references/job-investigation.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/job-investigation.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Investigating a Managed Job\n\nHow to answer \"what is my job doing and why\" with as few commands as\npossible. Every command here supports `-o json`; prefer it over parsing\ntable output.\n\n## Which command answers which question\n\n| Question | Command | Look at |\n|----------|---------|---------|\n| What state is job N in, and why? | `sky jobs queue -v -o json` | `status`, `details`, `failure_reason`, `schedule_state`, `recovery_count` |\n| Why is job N still provisioning / pending on the cluster? | `sky jobs queue -v -o json` | `details`, when it starts with `Launching (` |\n| How long has job N been at this? | `sky jobs queue -v -o json` | `submitted_at`, and the duration columns in the table |\n| What did job N print recently? | `sky jobs logs N --tail 100 --no-follow` | stdout/stderr of the task |\n| What did the controller do (provision, recover, retry)? | `sky jobs logs N --controller --no-follow` | controller log |\n| Wait until job N finishes | `sky jobs logs N` (blocks) | exit when the job ends |\n\nDo not poll with `sleep` + `sky jobs queue`. Stream logs to block, or\nfetch `--tail` once after the fact.\n\n`details` is the current explanation, not a history. For the sequence of\ntransitions a job went through — and, on Slurm, how long the allocation\nwaited and on what — the timeline lives behind the `jobs.events` API and is\nsurfaced by the enterprise client.\n\n## Reading `details`\n\n`details` in `sky jobs queue -v -o json` is the one-line explanation of the\ncurrent state. Common shapes:\n\n| `details` | Meaning | Next step |\n|-----------|---------|-----------|\n| `Waiting for other jobs to launch` | Controller is at its concurrent-launch limit | Nothing to fix; the duration columns say how long it has waited |\n| `Waiting for higher priority jobs to launch` | A higher-priority managed job is ahead | Raise `priority` in the task YAML if this job matters more |\n| `In backoff, waiting for resources` | Provisioning failed and is retrying | `sky jobs logs N --controller --no-follow` shows the last failure |\n| `Recovering: ` | Cluster was lost or the task crashed (e.g. OOMKilled) | Fix the cause; recovery is automatic |\n| `Failure: ` | Terminal failure | Read the reason; `sky jobs logs N --no-follow` for the task output |\n| `Cancellation requested by ...` | Someone cancelled it | The requester and request ID are in the text |\n| `Launching (pending: ; partition:

)` | Job is STARTING and Slurm has not allocated nodes yet | See \"Slurm pending reasons\" below |\n| `Launching (nodes allocated; Slurm job on )` | The queue wait is over; the job is still bootstrapping | Nothing to fix. That id is what `sacct -j ` takes if you have login-node access |\n\n## Slurm pending reasons\n\nWhile a managed job's Slurm allocation is queued, `details` carries the\n`squeue` reason and the partition it is queued in — the partition matters\nbecause the same reason means different things in different partitions, and\nby the time anyone looks the allocation may be gone. The common codes:\n\n| Reason | Category | What to do |\n|--------|----------|------------|\n| `Resources`, `Priority`, `ReqNodeNotAvail` | No free capacity in the partition | Wait, or resubmit to another partition via `config.slurm` |\n| `QOSGrpGRES`, `QOSMax*`, `AssocGrp*` | A QoS / account quota is full | Wait for the group's jobs to finish, or use a different QoS/partition |\n| `Dependency`, `DependencyNeverSatisfied` | Waiting on another Slurm job; `NeverSatisfied` will pend forever | Check the dependency job; cancel and resubmit if it failed |\n| `JobHeldUser`, `JobHeldAdmin`, `launch failed requeued held` | Job is held | Ask the admin to `scontrol release`, or cancel and relaunch |\n| `BeginTime` | Scheduled start time not reached | Nothing to fix |\n\nThe reason is recorded each time it changes, so a job that moved from\n`Resources` to `Priority` and back leaves all three behind it — `details`\nshows only the latest.\n\n## Recognising a stuck job\n\n- Repeated `Recovering: ` means a persistent problem (bad node,\n OOM, quota). The controller keeps retrying and will not fix it; read\n `sky jobs logs N --controller --no-follow`.\n- A long gap between STARTING and RUNNING is provisioning time. On Slurm\n that is queue time, and `details` says what the queue is waiting on.\n- `failure_reason` distinguishes the user's program failing from an\n infrastructure failure; do not report the latter as a code bug.\n","isInternal":false,"tokens":1109,"sizeBytes":4454},{"name":"migrating-from-slurm.md","path":"agent/skills/skypilot/references/migrating-from-slurm.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/migrating-from-slurm.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# Migrating from Slurm Reference\n\nThis reference is for converting existing Slurm workloads into SkyPilot task\nYAMLs, and for answering \"how do I do X, which I used to do with Slurm?\".\n\nTwo distinct situations, which need different answers:\n\n- **SkyPilot on the user's existing Slurm cluster.** SkyPilot submits to it\n through the login node via `sbatch`. The cluster is unchanged. Most users\n start here — do not assume they want to leave Slurm.\n- **SkyPilot on Kubernetes or a cloud.** The workload moves off Slurm. The\n YAML is the same; the constraints are different (see section 8).\n\nAsk which one applies if it is not obvious. The conversion is nearly identical;\nthe difference shows up in lifecycle (section 7) and what is unsupported\n(section 8).\n\n---\n\n## 1. Conversion Procedure\n\nFollow this order. Do not skip step 5.\n\n1. **Read the whole `sbatch` script**, not just the `#SBATCH` block. The\n `srun` invocations and `module load` lines carry as much information as the\n directives.\n2. **Map the `#SBATCH` directives** to `resources` / `num_nodes` (section 2).\n3. **Map the body**: `module load` → `setup`, environment activation → `setup`,\n the actual work → `run` (section 6).\n4. **Translate `srun`** — this is the step most likely to be done wrong\n (section 5).\n5. **Validate**: `sky launch --dryrun `. This resolves resources and\n runs the optimizer without provisioning anything. Note there is **no**\n `--dryrun` on `sky jobs launch`, so dry-run against `sky launch` even when\n the final command will be `sky jobs launch`.\n6. **Tell the user which submit command to use**: `sky jobs launch` for batch\n work (an `sbatch` script), `sky launch` for an interactive allocation (an\n `salloc` session).\n\n### 1.1 Choosing the submit command\n\n| The user's script is | Use | Why |\n|---|---|---|\n| `sbatch` batch job | `sky jobs launch` | Allocation released when the job ends, and the job is recovered if the node fails |\n| `salloc` / `srun --pty` session | `sky launch` + `ssh ` | Long-lived allocation to work in |\n| `sbatch --array` | `sky jobs launch --num-jobs N` | Section 7.1 |\n| A chain with `--dependency` | One managed pipeline | Section 7.2 |\n\n---\n\n## 2. `#SBATCH` Directive Mapping\n\n| `#SBATCH` directive | SkyPilot YAML | Notes |\n|---|---|---|\n| `--nodes=2` | `num_nodes: 2` | Task-level field, **not** under `resources` |\n| `--gpus-per-node=8`, `--gres=gpu:h100:8` | `resources.accelerators: H100:8` | Emitted as `--gres=gpu::`. The name must match the cluster's GRES config |\n| `--cpus-per-task=32` | `resources.cpus: 32+` | `+` means \"at least\". Emitted as `--cpus-per-task` |\n| `--mem=256G` | `resources.memory: 256+` | In GB. Emitted as `--mem` |\n| `--partition=gpu` | `resources.infra: slurm//gpu` | Omit to let the optimizer choose a partition |\n| `--time=24:00:00` | `config.slurm.sbatch_options.time: \"24:00:00\"` | Defaults to the partition's `MaxTime` (or `DefaultTime` when `MaxTime` is `UNLIMITED`) if unset. **Not** `autostop` — see section 8 |\n| `--job-name=train` | `name: train` | |\n| `--output=`, `--error=` | *(nothing)* | SkyPilot manages job logs; see section 7.4 |\n| `--account`, `--qos`, `--reservation`, `--exclusive`, `--constraint`, ... | `config.slurm.sbatch_options.` | Section 3 |\n\n### 2.1 Worked example\n\nInput:\n\n```bash\n#!/bin/bash\n#SBATCH --job-name=train\n#SBATCH --nodes=2\n#SBATCH --gpus-per-node=8\n#SBATCH --cpus-per-task=32\n#SBATCH --mem=256G\n#SBATCH --partition=gpu\n#SBATCH --time=24:00:00\n#SBATCH --account=ai-research\n\nmodule load cuda/12.1\nsource ~/venv/bin/activate\n\nsrun python train.py --epochs 100\n```\n\nOutput:\n\n```yaml\n# train.yaml\nname: train\n\nnum_nodes: 2\n\nresources:\n infra: slurm/mycluster/gpu\n accelerators: H100:8\n cpus: 32+\n memory: 256+\n\nconfig:\n slurm:\n sbatch_options:\n time: \"24:00:00\"\n account: ai-research\n\nsetup: |\n pip install -r requirements.txt\n\nrun: |\n python train.py --epochs 100\n```\n\nSubmit with `sky jobs launch -n train train.yaml`.\n\nNote what happened to the `srun`: it disappeared. `run` already executes on\nevery node of the allocation, so wrapping the command in `srun` again would\nnest a job step inside SkyPilot's own. See section 5.\n\n---\n\n## 3. Directives SkyPilot Does Not Model\n\nAnything without a `resources` equivalent goes through\n`slurm.sbatch_options`, which is emitted verbatim as `#SBATCH` lines. Keys are\n`sbatch` long option names; underscores become hyphens; `true` emits a bare\nflag; `null` / `false` omit the line.\n\n```yaml\nconfig:\n slurm:\n sbatch_options:\n account: ai-research\n qos: high\n time: \"24:00:00\"\n exclusive: true\n constraint: ib\n```\n\nThe same block works in `~/.sky/config.yaml` under a top-level `slurm:` key to\napply fleet-wide, and can be scoped per cluster or per partition:\n`slurm.cluster_configs..sbatch_options` and\n`slurm.cluster_configs..partition_configs..sbatch_options`.\n\n**Do not put these in `sbatch_options`** — SkyPilot manages them and will drop\nthem with a warning:\n\n```\njob-name output error nodes wait-all-nodes\nno-requeue requeue cpus-per-task mem gres partition\n```\n\nIf a user asks for one of those, set it through `resources` / `name` instead.\n\n---\n\n## 4. Environment Variable Mapping\n\n| Slurm | SkyPilot | Notes |\n|---|---|---|\n| `$SLURM_JOB_NODELIST` | `$SKYPILOT_NODE_IPS` | Newline-separated IPs, **not** a Slurm hostlist expression. `$(echo \"$SKYPILOT_NODE_IPS\" \\| head -n1)` is the head node |\n| `$SLURM_NNODES` | `$SKYPILOT_NUM_NODES` | |\n| `$SLURM_NODEID`, `$SLURM_PROCID` | `$SKYPILOT_NODE_RANK` | `0` to `num_nodes-1` |\n| `$SLURM_GPUS_PER_NODE` | `$SKYPILOT_NUM_GPUS_PER_NODE` | |\n| `$SLURM_JOB_ID` | `$SKYPILOT_TASK_ID` | |\n| `$SLURM_ARRAY_TASK_ID` | `$SKYPILOT_JOB_RANK` | Only set under `--num-jobs`; section 7.1 |\n| `$SLURM_ARRAY_TASK_COUNT` | `$SKYPILOT_NUM_JOBS` | |\n\n**On Slurm specifically**, the *job-scoped* `SLURM_*` variables of the\nunderlying allocation are still present inside `run` — `SLURM_JOB_ID`,\n`SLURM_JOB_NODELIST`, `SLURM_GPUS_ON_NODE`, etc. Scripts that read them keep\nworking. The *step-scoped* variables from SkyPilot's own job step\n(`SLURM_CPUS_PER_TASK`, `SLURM_CPU_BIND*`, `SLURM_STEP_*`, `SLURM_PROCID`, ...)\nare deliberately unset before the user script runs, so a nested `srun` targets\nthe full allocation instead of inheriting SkyPilot's step shape.\n\nPrefer the `SKYPILOT_*` variables in generated YAML — they are what makes the\nsame task portable to Kubernetes and clouds. Use the `SLURM_*` ones only when\ntranslating a script that must keep working unchanged.\n\n---\n\n## 5. Translating `srun`\n\nThree cases. Pick by what the `srun` was *for*.\n\n### 5.1 `srun ` launching one process per node\n\nDrop the `srun`. `run` already runs on every node.\n\n```bash\n# Slurm\nsrun python preprocess.py\n```\n\n```yaml\nrun: |\n python preprocess.py\n```\n\n### 5.2 `srun` launching distributed training ranks\n\nUse the rank variables and the framework's own launcher. This is the portable\nform and should be the default for training workloads.\n\n```yaml\nnum_nodes: 2\n\nresources:\n accelerators: H100:8\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$MASTER_ADDR \\\n --master_port=8008 \\\n train.py\n```\n\n### 5.3 `srun` launching an MPI/PMIx program\n\nSome binaries must be launched by Slurm's own PMI (NCCL tests, anything\nMPI-linked, `--mpi=pmix`). Keep `srun`, but:\n\n- **Gate it on rank 0**, or every node will launch the whole job.\n- **Pass `--overlap`**, so the step can share the allocation with SkyPilot's\n own step.\n\n```yaml\nnum_nodes: 2\n\nresources:\n infra: slurm\n accelerators: H100:8\n\nrun: |\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n srun --overlap --mpi=pmix \\\n --ntasks-per-node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n ./my_mpi_program\n fi\n```\n\nThis form is Slurm-only by construction — it will not run on Kubernetes or a\ncloud. Say so when generating it.\n\n---\n\n## 6. Replacing the Module System\n\n`module load` reads a cluster-wide tree shared by all jobs. SkyPilot gives\neach job its own environment. In increasing order of isolation:\n\n```yaml\n# 1. setup commands — runs once per cluster, before the job\nsetup: |\n pip install -r requirements.txt\n```\n\n```yaml\n# 2. a container image — the closest thing to a reproducible module set.\n# On Slurm this needs the Pyxis SPANK plugin on the cluster.\nresources:\n image_id: docker:pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime\n```\n\n```yaml\n# 3. conda on the shared filesystem, if the user wants to keep their layout\nsetup: |\n conda create -n myenv python=3.10 -y\n conda activate myenv\n conda install pytorch pytorch-cuda=12.1 -c pytorch -c nvidia -y\n\nrun: |\n conda activate myenv\n python train.py\n```\n\n`module load` itself still works inside `setup` / `run` on a Slurm cluster,\nas long as the task is not running in a container. Keeping it is a legitimate\nfirst migration step — it just does not port to Kubernetes or clouds.\n\n---\n\n## 7. Patterns Without a One-to-One Field\n\n### 7.1 Job arrays → `--num-jobs`\n\n`sbatch --array=0-99` becomes:\n\n```bash\nsky jobs launch --num-jobs 100 -y -d task.yaml\n```\n\nEach job gets `$SKYPILOT_JOB_RANK` (the index) and `$SKYPILOT_NUM_JOBS`:\n\n```yaml\nrun: |\n python train.py --shard $SKYPILOT_JOB_RANK --num-shards $SKYPILOT_NUM_JOBS\n```\n\nEach job provisions its own allocation. To reuse workers across submissions\ninstead — closer to how a long-lived Slurm allocation behaves, and much\ncheaper when `setup` is expensive — submit to a pool (note `pool.yaml` is a\npool spec: it needs a `pool:` section, not a bare task YAML):\n\n```bash\nsky jobs pool apply -p mypool pool.yaml\nsky jobs launch -p mypool --num-jobs 100 task.yaml\n```\n\nFor sweeps over *named* parameters rather than an index, loop with `--env`:\n\n```bash\nfor lr in 0.001 0.01 0.1; do\n sky jobs launch --env LR=$lr -y -d task.yaml\ndone\n```\n\n### 7.2 Job dependencies → a managed pipeline\n\n`sbatch --dependency=afterok:` becomes a sequence of tasks in one\nmanaged job, separated by `---`. Each task can request different resources.\n\n```yaml\nname: train-then-eval\n\n---\n\nname: train\nresources:\n accelerators: H100:8\nrun: python train.py\n\n---\n\nname: eval\nresources:\n accelerators: L4:1\nrun: python eval.py\n```\n\nFor tasks that should run *in parallel* rather than in sequence, use a job\ngroup instead of a pipeline. A job group is the same multi-document layout,\nbut the header must set `execution: parallel` — without it the file is a\n*serial* pipeline:\n\n```yaml\nname: train-and-eval\nexecution: parallel\n\n---\n\nname: train\nresources:\n accelerators: H100:8\nrun: python train.py\n\n---\n\nname: eval\nresources:\n accelerators: L4:1\nrun: python eval.py\n```\n\n### 7.3 Login node workflow\n\nThere is no login node in the SkyPilot workflow. Submit from the user's laptop\nor CI; SkyPilot makes the SSH hop to the login node itself. Sync code with\n`workdir` (a local path or a git URL) rather than editing on NFS. For\ninteractive work, `ssh ` lands on the *allocated compute node*, not a\nshared login node.\n\n### 7.4 Job output files\n\nSlurm writes `slurm-.out`. SkyPilot streams logs instead:\n\n```bash\nsky jobs logs # managed job\nsky logs # latest job on a cluster\nsky logs 2 # a specific job\n```\n\nDo not translate `--output` / `--error` into anything. If the script writes\nits own result files, point them at the shared filesystem or a bucket.\n\n---\n\n## 8. Constraints When Running on Slurm\n\nDo not generate YAML that uses these on a Slurm target — they will fail or be\nrejected:\n\n| Feature | Status on Slurm | What to do instead |\n|---|---|---|\n| `autostop` / `sky stop` | Unsupported | `sky down` explicitly; bound wall-clock with `sbatch_options.time` |\n| `ports:` / `sky serve` | Unsupported | Run serving on Kubernetes or a cloud |\n| `use_spot: true` | Unsupported | Omit; there is no spot tier |\n| Jobs/serve controllers | Not hosted on Slurm | With a remote API server, consolidation mode runs the controller inside the API server, so `sky jobs launch` works against a Slurm-only fleet. With a local API server the controller needs Kubernetes or a cloud |\n| `image_id: docker:...` | Needs Pyxis on the cluster | Fall back to `setup` commands |\n| Bucket mounting (`mode: MOUNT`) | Needs FUSE on compute nodes | Use the shared filesystem, or `mode: COPY` |\n\nBecause autostop does not exist on Slurm, an idle `sky launch` cluster holds\nits allocation until `sky down`. Prefer `sky jobs launch` for anything\nbatch-like so the allocation is released automatically.\n\n### 8.1 If the target is Kubernetes instead\n\nThe assumption that breaks first is shared storage: Kubernetes does not mount\nhome directories. A Slurm script that reads `~/data` or writes `~/checkpoints`\nneeds one of:\n\n- an NFS mount via `config.kubernetes.pod_config` (if the nodes can reach the\n existing NFS server),\n- a `ReadWriteMany` PVC through SkyPilot volumes,\n- a bucket under `file_mounts` with `mode: MOUNT`,\n- `workdir` for code only.\n\nPartitions map to Kubernetes contexts (`infra: kubernetes/`), and\nSlurm's fairshare/QoS has no built-in equivalent — that is Kueue or priority\nclasses.\n\n---\n\n## 9. Validation Checklist\n\nBefore handing a converted YAML to the user:\n\n- [ ] `sky launch --dryrun ` resolves without error.\n- [ ] `num_nodes` is at task level, not under `resources`.\n- [ ] No `autostop` / `ports` / `use_spot` if the target is Slurm.\n- [ ] Every `srun` is accounted for by section 5 — dropped, replaced with\n rank variables, or kept with `--overlap` and gated on rank 0.\n- [ ] GPU name matches the cluster's GRES names (check `sky gpus list --infra\n slurm`).\n- [ ] Protected `sbatch` options are not in `sbatch_options` (section 3).\n- [ ] The user is told whether to run `sky launch` or `sky jobs launch`.\n","isInternal":false,"tokens":4003,"sizeBytes":13979},{"name":"python-sdk.md","path":"agent/skills/skypilot/references/python-sdk.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/python-sdk.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"\n\n\n# SkyPilot Python SDK Reference\n\nAll SDK functions return a request ID (future). Use `sky.get(request_id)` to await the result.\n\n```python\nimport sky\n\nrequest_id = sky.launch(sky.Task.from_yaml(\"task.yaml\"))\nresult = sky.get(request_id)\n```\n\n## Cluster Operations\n\n### `sky.launch`\n\n```python\nsky.launch(task: Union['sky.Task', 'sky.Dag'], cluster_name: Optional[str] = None, retry_until_up: bool = False, idle_minutes_to_autostop: Optional[int] = None, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, dryrun: bool = False, down: bool = False, backend: Optional['backends.Backend'] = None, optimize_target: common.OptimizeTarget = common.OptimizeTarget.COST, no_setup: bool = False, clone_disk_from: Optional[str] = None, fast: bool = False, resize: bool = False, _need_confirmation: bool = False, _is_launched_by_jobs_controller: bool = False, _is_launched_by_sky_serve_controller: bool = False, _file_mounts_blob_id: Optional[str] = None, _extra_launch_context: Optional[Dict[str, Any]] = None, _include_credentials: bool = False) -> server_common.RequestId[Tuple[Optional[int], Optional['backends.ResourceHandle']]]\n```\n\nLaunches a cluster or task.\n\nThe task's setup and run commands are executed under the task's workdir\n(when specified, it is synced to remote cluster). The task undergoes job\nqueue scheduling on the cluster.\n\nCurrently, the first argument must be a sky.Task, or (EXPERIMENTAL advanced\nusage) a sky.Dag. In the latter case, currently it must contain a single\ntask; support for pipelines/general DAGs are in experimental branches.\n\n**Example:**\n ```python\n\n import sky\n task = sky.Task(run='echo hello SkyPilot')\n task.set_resources(\n sky.Resources(infra='aws', accelerators='V100:4'))\n sky.launch(task, cluster_name='my-cluster')\n\n\n**Args:**\n task: sky.Task, or sky.Dag (experimental; 1-task only) to launch.\n cluster_name: name of the cluster to create/reuse. If None,\n auto-generate a name.\n retry_until_up: whether to retry launching the cluster until it is\n up.\n idle_minutes_to_autostop: automatically stop the cluster after this\n many minute of idleness, i.e., no running or pending jobs in the\n cluster's job queue. Idleness gets reset whenever setting-up/\n running/pending jobs are found in the job queue. Setting this\n flag is equivalent to running\n ``sky.launch(...)`` and then\n ``sky.autostop(idle_minutes=)``. If set, the autostop\n config specified in the task' resources will be overridden by\n this parameter.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes_to_autostop``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after\n ``idle_minutes_to_autostop``.\n dryrun: if True, do not actually launch the cluster.\n down: Tear down the cluster after all jobs finish (successfully or\n abnormally). If --idle-minutes-to-autostop is also set, the\n cluster will be torn down after the specified idle time.\n Note that if errors occur during provisioning/data syncing/setting\n up, the cluster will not be torn down for debugging purposes. If\n set, the autostop config specified in the task' resources will be\n overridden by this parameter.\n backend: backend to use. If None, use the default backend\n (CloudVMRayBackend).\n optimize_target: target to optimize for. Choices: OptimizeTarget.COST,\n OptimizeTarget.TIME.\n no_setup: if True, do not re-run setup commands.\n clone_disk_from: [Experimental] if set, clone the disk from the\n specified cluster. This is useful to migrate the cluster to a\n different availability zone or region.\n fast: [Experimental] If the cluster is already up and available,\n skip provisioning and setup steps.\n resize: if True, resize the existing cluster to the ``num_nodes``\n specified in the task. Supports both scaling up (adding workers)\n and scaling down (removing workers). Scale-down requires no\n running jobs on the cluster. If True, requires ``cluster_name``\n to be set.\n _need_confirmation: (Internal only) If True, show the confirmation\n prompt.\n\n**Returns:**\n The request ID of the launch request.\n\n**Request Returns:**\n job_id (Optional[int]): the job ID of the submitted job. None if the\n backend is not ``CloudVmRayBackend``, or no job is submitted to the\n cluster.\n handle (Optional[backends.ResourceHandle]): the handle to the cluster.\n None if dryrun.\n\n**Request Raises:**\n exceptions.ClusterOwnerIdentityMismatchError: if the cluster is owned\n by another user.\n exceptions.InvalidClusterNameError: if the cluster name is invalid.\n exceptions.ResourcesMismatchError: if the requested resources\n do not match the existing cluster.\n exceptions.NotSupportedError: if required features are not supported\n by the backend/cloud/cluster.\n exceptions.ResourcesUnavailableError: if the requested resources\n cannot be satisfied. The failover_history of the exception will be set\n as:\n\n 1. Empty: iff the first-ever sky.optimize() fails to find a feasible\n resource; no pre-check or actual launch is attempted.\n\n 2. Non-empty: iff at least 1 exception from either our pre-checks\n (e.g., cluster name invalid) or a region/zone throwing resource\n unavailability.\n\n exceptions.CommandError: any ssh command error.\n exceptions.NoCloudAccessError: if all clouds are disabled.\n\nOther exceptions may be raised depending on the backend.\n\n### `sky.exec`\n\n```python\nsky.exec(task: Union['sky.Task', 'sky.Dag'], cluster_name: Optional[str] = None, dryrun: bool = False, down: bool = False, backend: Optional['backends.Backend'] = None) -> server_common.RequestId[Tuple[Optional[int], Optional['backends.ResourceHandle']]]\n```\n\nExecutes a task on an existing cluster.\n\n**This function performs two actions:**\n\n(1) workdir syncing, if the task has a workdir specified;\n(2) executing the task's ``run`` commands.\n\nAll other steps (provisioning, setup commands, file mounts syncing) are\nskipped. If any of those specifications changed in the task, this function\nwill not reflect those changes. To ensure a cluster's setup is up to date,\nuse ``sky.launch()`` instead.\n\n**Execution and scheduling behavior:**\n\n- The task will undergo job queue scheduling, respecting any specified\n resource requirement. It can be executed on any node of the cluster with\n enough resources.\n- The task is run under the workdir (if specified).\n- The task is run non-interactively (without a pseudo-terminal or\n pty), so interactive commands such as ``htop`` do not work.\n Use ``ssh my_cluster`` instead.\n\n**Args:**\n task: sky.Task, or sky.Dag (experimental; 1-task only) containing the\n task to execute.\n cluster_name: name of an existing cluster to execute the task.\n dryrun: if True, do not actually execute the task.\n down: Tear down the cluster after all jobs finish (successfully or\n abnormally). If --idle-minutes-to-autostop is also set, the\n cluster will be torn down after the specified idle time.\n Note that if errors occur during provisioning/data syncing/setting\n up, the cluster will not be torn down for debugging purposes.\n backend: backend to use. If None, use the default backend\n (CloudVMRayBackend).\n\n**Returns:**\n The request ID of the exec request.\n\n\n**Request Returns:**\n job_id (Optional[int]): the job ID of the submitted job. None if the\n backend is not CloudVmRayBackend, or no job is submitted to\n the cluster.\n handle (Optional[backends.ResourceHandle]): the handle to the cluster.\n None if dryrun.\n\n**Request Raises:**\n ValueError: if the specified cluster is not in UP status.\n sky.exceptions.ClusterDoesNotExist: if the specified cluster does not\n exist.\n sky.exceptions.NotSupportedError: if the specified cluster is a\n controller that does not support this operation.\n\n### `sky.stop`\n\n```python\nsky.stop(cluster_name: str, purge: bool = False, graceful: bool = False, graceful_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nStops a cluster.\n\nData on attached disks is not lost when a cluster is stopped. Billing for\nthe instances will stop, while the disks will still be charged. Those\ndisks will be reattached when restarting the cluster.\n\nCurrently, spot instance clusters cannot be stopped (except for GCP, which\ndoes allow disk contents to be preserved when stopping spot VMs).\n\n**Args:**\n cluster_name: name of the cluster to stop.\n purge: (Advanced) Forcefully mark the cluster as stopped in SkyPilot's\n cluster table, even if the actual cluster stop operation failed on\n the cloud. WARNING: This flag should only be set sparingly in\n certain manual troubleshooting scenarios; with it set, it is the\n user's responsibility to ensure there are no leaked instances and\n related resources.\n\n**Returns:**\n The request ID of the stop request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n RuntimeError: failed to stop the cluster.\n sky.exceptions.NotSupportedError: if the specified cluster is a spot\n cluster, or a TPU VM Pod cluster, or the managed jobs controller.\n\n### `sky.start`\n\n```python\nsky.start(cluster_name: str, idle_minutes_to_autostop: Optional[int] = None, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, retry_until_up: bool = False, down: bool = False, force: bool = False) -> server_common.RequestId['backends.CloudVmRayResourceHandle']\n```\n\nRestart a cluster.\n\nIf a cluster is previously stopped (status is STOPPED) or failed in\nprovisioning/runtime installation (status is INIT), this function will\nattempt to start the cluster. In the latter case, provisioning and runtime\ninstallation will be retried.\n\nAuto-failover provisioning is not used when restarting a stopped\ncluster. It will be started on the same cloud, region, and zone that were\nchosen before.\n\nIf a cluster is already in the UP status, this function has no effect.\n\n**Args:**\n cluster_name: name of the cluster to start.\n idle_minutes_to_autostop: automatically stop the cluster after this\n many minute of idleness, i.e., no running or pending jobs in the\n cluster's job queue. Idleness gets reset whenever setting-up/\n running/pending jobs are found in the job queue. Setting this\n flag is equivalent to running ``sky.launch()`` and then\n ``sky.autostop(idle_minutes=)``. If not set, the\n cluster will not be autostopped.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes_to_autostop``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after\n ``idle_minutes_to_autostop``.\n retry_until_up: whether to retry launching the cluster until it is\n up.\n down: Autodown the cluster: tear down the cluster after specified\n minutes of idle time after all jobs finish (successfully or\n abnormally). Requires ``idle_minutes_to_autostop`` to be set.\n force: whether to force start the cluster even if it is already up.\n Useful for upgrading SkyPilot runtime.\n\n**Returns:**\n The request ID of the start request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: argument values are invalid: (1) if ``down`` is set to True\n but ``idle_minutes_to_autostop`` is None; (2) if the specified\n cluster is the managed jobs controller, and either\n ``idle_minutes_to_autostop`` is not None or ``down`` is True (omit\n them to use the default autostop settings).\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n sky.exceptions.NotSupportedError: if the cluster to restart was\n launched using a non-default backend that does not support this\n operation.\n sky.exceptions.ClusterOwnerIdentitiesMismatchError: if the cluster to\n restart was launched by a different user.\n\n### `sky.down`\n\n```python\nsky.down(cluster_name: str, purge: bool = False, graceful: bool = False, graceful_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nTears down a cluster.\n\nTearing down a cluster will delete all associated resources (all billing\nstops), and any data on the attached disks will be lost. Accelerators\n(e.g., TPUs) that are part of the cluster will be deleted too.\n\n**Args:**\n cluster_name: name of the cluster to down.\n purge: (Advanced) Forcefully remove the cluster from SkyPilot's cluster\n table, even if the actual cluster termination failed on the cloud.\n WARNING: This flag should only be set sparingly in certain manual\n troubleshooting scenarios; with it set, it is the user's\n responsibility to ensure there are no leaked instances and related\n resources.\n graceful: Cancel the user's task but block until MOUNT_CACHED data is\n fully uploaded. This helps with preserving user data integrity.\n graceful_timeout: If not None, sets a timeout for the graceful option\n above (in seconds).\n\n**Returns:**\n The request ID of the down request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n RuntimeError: failed to tear down the cluster.\n sky.exceptions.NotSupportedError: the specified cluster is the managed\n jobs controller.\n\n### `sky.status`\n\n```python\nsky.status(cluster_names: Optional[List[str]] = None, refresh: common.StatusRefreshMode = common.StatusRefreshMode.NONE, all_users: bool = False, *, _include_credentials: bool = False, _summary_response: bool = False) -> server_common.RequestId[List[responses.StatusResponse]]\n```\n\nGets cluster statuses.\n\nIf cluster_names is given, return those clusters. Otherwise, return all\nclusters.\n\n**Each cluster can have one of the following statuses:**\n\n- ``INIT``: The cluster may be live or down. It can happen in the following\n cases:\n\n - Ongoing provisioning or runtime setup. (A ``sky.launch()`` has started\n but has not completed.)\n - Or, the cluster is in an abnormal state, e.g., some cluster nodes are\n down, or the SkyPilot runtime is unhealthy. (To recover the cluster,\n try ``sky launch`` again on it.)\n\n- ``UP``: Provisioning and runtime setup have succeeded and the cluster is\n live. (The most recent ``sky.launch()`` has completed successfully.)\n\n- ``STOPPED``: The cluster is stopped and the storage is persisted. Use\n ``sky.start()`` to restart the cluster.\n\n**Autostop column:**\n\n- The autostop column indicates how long the cluster will be autostopped\n after minutes of idling (no jobs running). If ``to_down`` is True, the\n cluster will be autodowned, rather than autostopped.\n\nGetting up-to-date cluster statuses:\n\n- In normal cases where clusters are entirely managed by SkyPilot (i.e., no\n manual operations in cloud consoles) and no autostopping is used, the\n table returned by this command will accurately reflect the cluster\n statuses.\n\n- In cases where the clusters are changed outside of SkyPilot (e.g., manual\n operations in cloud consoles; unmanaged spot clusters getting preempted)\n or for autostop-enabled clusters, use ``refresh=True`` to query the\n latest cluster statuses from the cloud providers.\n\n**Args:**\n cluster_names: a list of cluster names to query. If not\n provided, all clusters will be queried.\n refresh: whether to query the latest cluster statuses from the cloud\n provider(s).\n all_users: whether to include all users' clusters. By default, only\n the current user's clusters are included.\n _include_credentials: (internal) whether to include cluster ssh\n credentials in the response (default: False).\n\n**Returns:**\n The request ID of the status request.\n\n**Request Returns:**\n cluster_records (List[Dict[str, Any]]): A list of dicts, with each dict\n containing the information of a cluster. If a cluster is found to be\n terminated or not found, it will be omitted from the returned list.\n\n ```python\n\n {\n 'name': (str) cluster name,\n 'launched_at': (int) timestamp of last launch on this cluster,\n 'handle': (ResourceHandle) an internal handle to the cluster,\n 'last_use': (str) the last command/entrypoint that affected this\n cluster,\n 'status': (sky.ClusterStatus) cluster status,\n 'autostop': (int) idle time before autostop,\n 'to_down': (bool) whether autodown is used instead of autostop,\n 'metadata': (dict) metadata of the cluster,\n 'user_hash': (str) user hash of the cluster owner,\n 'user_name': (str) user name of the cluster owner,\n 'resources_str': (str) the resource string representation of the\n cluster,\n }\n\n### `sky.autostop`\n\n```python\nsky.autostop(cluster_name: str, idle_minutes: int, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, down: bool = False, hook: Optional[str] = None, hook_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nSchedules an autostop/autodown for a cluster.\n\nAutostop/autodown will automatically stop or teardown a cluster when it\nbecomes idle for a specified duration. Idleness means there are no\nin-progress (pending/running) jobs in a cluster's job queue.\n\nIdleness time of a cluster is reset to zero, whenever:\n\n- A job is submitted (``sky.launch()`` or ``sky.exec()``).\n\n- The cluster has restarted.\n\n- An autostop is set when there is no active setting. (Namely, either\n there's never any autostop setting set, or the previous autostop setting\n was canceled.) This is useful for restarting the autostop timer.\n\nExample: say a cluster without any autostop set has been idle for 1 hour,\nthen an autostop of 30 minutes is set. The cluster will not be immediately\nautostopped. Instead, the idleness timer only starts counting after the\nautostop setting was set.\n\nWhen multiple autostop settings are specified for the same cluster, the\nlast setting takes precedence.\n\n**Args:**\n cluster_name: name of the cluster.\n idle_minutes: the number of minutes of idleness (no pending/running\n jobs) after which the cluster will be stopped automatically. Setting\n to a negative number cancels any autostop/autodown setting.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after ``idle_minutes``.\n down: if true, use autodown (tear down the cluster; non-restartable),\n rather than autostop (restartable).\n hook: optional script to execute on the remote cluster before autostop.\n The script runs before the cluster is stopped or torn down. If the\n hook fails, autostop will still proceed but a warning will be\n logged.\n hook_timeout: timeout in seconds for hook execution. If None, uses\n DEFAULT_HOOK_TIMEOUT_SECONDS (3600 = 1 hour). The hook will\n be terminated if it exceeds this timeout.\n\n**Returns:**\n The request ID of the autostop request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: if arguments are invalid.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend or the cluster is TPU VM Pod.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.cost_report`\n\n```python\nsky.cost_report(days: Optional[int] = None) -> server_common.RequestId[List[Dict[str, Any]]]\n```\n\nGets all cluster cost reports, including those that have been downed.\n\nThe estimated cost column indicates price for the cluster based on the type\nof resources being used and the duration of use up until the call to\nstatus. This means if the cluster is UP, successive calls to report will\nshow increasing price. The estimated cost is calculated based on the local\ncache of the cluster status, and may not be accurate for the cluster with\nautostop/use_spot set or terminated/stopped on the cloud console.\n\n**Args:**\n days: The number of days to get the cost report for. If not provided,\n the default is 30 days.\n\n**Returns:**\n The request ID of the cost report request.\n\n**Request Returns:**\n cluster_cost_records (List[Dict[str, Any]]): A list of dicts, with each\n dict containing the cost information of a cluster.\n\n ```python\n\n {\n 'name': (str) cluster name,\n 'launched_at': (int) timestamp of last launch on this cluster,\n 'duration': (int) total seconds that cluster was up and running,\n 'last_use': (str) the last command/entrypoint that affected this\n 'num_nodes': (int) number of nodes launched for cluster,\n 'resources': (resources.Resources) type of resource launched,\n 'cluster_hash': (str) unique hash identifying cluster,\n 'usage_intervals': (List[Tuple[int, int]]) cluster usage times,\n 'total_cost': (float) cost given resources and usage intervals,\n }\n\n### `sky.endpoints`\n\n```python\nsky.endpoints(cluster: str, port: Optional[Union[int, str]] = None) -> server_common.RequestId[Dict[int, str]]\n```\n\nGets the endpoint for a given cluster and port number (endpoint).\n\n**Example:**\n ```python\n\n import sky\n request_id = sky.endpoints('test-cluster')\n sky.get(request_id)\n\n\n**Args:**\n cluster: The name of the cluster.\n port: The port number to get the endpoint for. If None, endpoints\n for all ports are returned.\n\n**Returns:**\n The request ID of the endpoints request.\n\n**Request Returns:**\n A dictionary of port numbers to endpoints.\n If port is None, the dictionary contains all\n ports:endpoints exposed on the cluster.\n\n**Request Raises:**\n ValueError: if the cluster is not UP or the endpoint is not exposed.\n RuntimeError: if the cluster has no ports to be exposed or no endpoints\n are exposed yet.\n\n## Job Operations\n\n### `sky.queue`\n\n```python\nsky.queue(cluster_name: str, skip_finished: bool = False, all_users: bool = False) -> server_common.RequestId[List[responses.ClusterJobRecord]]\n```\n\nGets the job queue of a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n skip_finished: if True, skip finished jobs.\n all_users: if True, return jobs from all users.\n\n\n**Returns:**\n The request ID of the queue request.\n\n**Request Returns:**\n job_records (List[responses.ClusterJobRecord]): A list of job records\n for each job in the queue.\n\n ```python\n\n [\n {\n 'job_id': (int) job id,\n 'job_name': (str) job name,\n 'username': (str) username,\n 'user_hash': (str) user hash,\n 'submitted_at': (int) timestamp of submitted,\n 'start_at': (int) timestamp of started,\n 'end_at': (int) timestamp of ended,\n 'resources': (str) resources,\n 'status': (job_lib.JobStatus) job status,\n 'log_path': (str) log path,\n }\n ]\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n ``CloudVmRayBackend``.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n sky.exceptions.CommandError: if failed to get the job queue with ssh.\n\n### `sky.job_status`\n\n```python\nsky.job_status(cluster_name: str, job_ids: Optional[List[int]] = None) -> server_common.RequestId[Dict[Optional[int], Optional['job_lib.JobStatus']]]\n```\n\nGets the status of jobs on a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n job_ids: job ids. If None, get the status of the last job.\n\n**Returns:**\n The request ID of the job status request.\n\n**Request Returns:**\n job_statuses (Dict[Optional[int], Optional[job_lib.JobStatus]]): A\n mapping of job_id to job statuses. The status will be None if the\n job does not exist. If job_ids is None and there is no job on the\n cluster, it will return {None: None}.\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n ``CloudVmRayBackend``.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.cluster_job_spec`\n\n```python\nsky.cluster_job_spec(cluster_name: str, job_id: int) -> server_common.RequestId[Optional[Dict[str, Any]]]\n```\n\nGets the task YAML and entrypoint a cluster job was submitted with.\n\n**Args:**\n cluster_name: name of the cluster.\n job_id: id of the job on the cluster.\n\n**Returns:**\n The request ID of the cluster job spec request.\n\n**Request Returns:**\n spec (Optional[Dict[str, Any]]): None if the cluster does not exist or\n the job's spec was not recorded (e.g. it was submitted by an older\n API server version), otherwise:\n\n ```python\n\n {\n 'task_yaml': (str) the task YAML, with secrets redacted,\n 'entrypoint': (str) the command that submitted the job,\n }\n\n### `sky.cancel`\n\n```python\nsky.cancel(cluster_name: str, all: bool = False, all_users: bool = False, job_ids: Optional[List[int]] = None, _try_cancel_if_cluster_is_init: bool = False) -> server_common.RequestId[None]\n```\n\nCancels jobs on a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n all: if True, cancel all jobs.\n all_users: if True, cancel all jobs from all users.\n job_ids: a list of job IDs to cancel.\n _try_cancel_if_cluster_is_init: (bool) whether to try cancelling the job\n even if the cluster is not UP, but the head node is still alive.\n This is used by the jobs controller to cancel the job when the\n worker node is preempted in the spot cluster.\n\n**Returns:**\n The request ID of the cancel request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: if arguments are invalid.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the specified cluster is a\n controller that does not support this operation.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.tail_logs`\n\n```python\nsky.tail_logs(cluster_name: str, job_id: Optional[int], follow: bool, tail: int = 0, output_stream: Optional['io.TextIOBase'] = None, *, preload_content: bool = True) -> Union[int, Iterator[Optional[str]]]\n```\n\nTails the logs of a job.\n\n**Args:**\n cluster_name: name of the cluster.\n job_id: job id.\n follow: if True, follow the logs. Otherwise, return the logs\n immediately.\n tail: if > 0, tail the last N lines of the logs.\n output_stream: the stream to write the logs to. If None, print to the\n console. Cannot be used with preload_content=False.\n preload_content: if False, returns an Iterator[str | None] containing\n the logs without the function blocking on the retrieval of entire\n log. Iterator returns None when the log has been completely\n streamed. Default True. Cannot be used with output_stream.\n\n**Returns:**\n If preload_content is True:\n Exit code based on success or failure of the job. 0 if success,\n 100 if the job failed. See exceptions.JobExitCode for possible exit\n codes.\n If preload_content is False:\n Iterator[str | None] containing the logs without the function\n blocking on the retrieval of entire log. Iterator returns None\n when the log has been completely streamed.\n\n**Request Raises:**\n ValueError: if arguments are invalid or the cluster is not supported.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.tail_provision_logs`\n\n```python\nsky.tail_provision_logs(cluster_name: str, worker: Optional[int] = None, follow: bool = True, tail: int = 0, output_stream: Optional['io.TextIOBase'] = None) -> int\n```\n\nTails the provisioning logs (provision.log) for a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n worker: worker id in multi-node cluster.\n If None, stream the logs of the head node.\n follow: follow the logs.\n tail: lines from end to tail.\n output_stream: optional stream to write logs.\n**Returns:**\n Exit code 0 on streaming success; raises on HTTP error.\n\n### `sky.tail_hook_logs`\n\n```python\nsky.tail_hook_logs(cluster_name: str, event: Optional[str] = None, follow: bool = True, tail: int = 0) -> int\n```\n\nTails a per-event lifecycle-hook log on the cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n event: one of ``stop``, ``preemption``, ``down``. When None,\n auto-selects whichever log exists on the cluster.\n follow: whether to follow the logs.\n tail: number of lines to display from the end of the log file.\n\n**Returns:**\n Exit code 0 on streaming success; non-zero on failure.\n\n### `sky.download_logs`\n\n```python\nsky.download_logs(cluster_name: str, job_ids: Optional[List[str]]) -> Dict[str, str]\n```\n\nDownloads the logs of jobs.\n\n**Args:**\n cluster_name: (str) name of the cluster.\n job_ids: (List[str]) job ids.\n\n**Returns:**\n The request ID of the download_logs request.\n\n**Request Returns:**\n job_log_paths (Dict[str, str]): a mapping of job_id to local log path.\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n## Infrastructure & Configuration\n\n### `sky.check`\n\n```python\nsky.check(infra_list: Optional[Tuple[str, ...]], verbose: bool, workspace: Optional[str] = None) -> server_common.RequestId[Dict[str, Dict[str, List[str]]]]\n```\n\nChecks the credentials to enable clouds.\n\n**Args:**\n infra: The infra to check.\n verbose: Whether to show verbose output.\n workspace: The workspace to check. If None, all workspaces will be\n checked.\n\n**Returns:**\n The request ID of the check request.\n\n**Request Returns:**\n Dict mapping workspace name to a dict of cloud name to list of\n enabled capability strings (e.g. 'compute', 'storage').\n\n### `sky.enabled_clouds`\n\n```python\nsky.enabled_clouds(workspace: Optional[str] = None, expand: bool = False) -> server_common.RequestId[List[str]]\n```\n\nGets the enabled clouds.\n\n**Args:**\n workspace: The workspace to get the enabled clouds for. If None, the\n active workspace will be used.\n expand: Whether to expand Kubernetes and SSH to list of resource pools.\n\n**Returns:**\n The request ID of the enabled clouds request.\n\n**Request Returns:**\n A list of enabled clouds in string format.\n\n### `sky.list_accelerators`\n\n```python\nsky.list_accelerators(gpus_only: bool = True, name_filter: Optional[str] = None, region_filter: Optional[str] = None, quantity_filter: Optional[int] = None, clouds: Optional[Union[List[str], str]] = None, all_regions: bool = False, require_price: bool = True, case_sensitive: bool = True) -> server_common.RequestId[Dict[str, List['catalog.common.InstanceTypeInfo']]]\n```\n\nLists the names of all accelerators offered by Sky.\n\nThis will include all accelerators offered by Sky, including those\nthat may not be available in the user's account.\n\n**Args:**\n gpus_only: Whether to only list GPU accelerators.\n name_filter: The name filter.\n region_filter: The region filter.\n quantity_filter: The quantity filter.\n clouds: The clouds to list.\n all_regions: Whether to list all regions.\n require_price: Whether to require price.\n case_sensitive: Whether to case sensitive.\n\n**Returns:**\n The request ID of the list accelerator counts request.\n\n**Request Returns:**\n acc_to_instance_type_dict (Dict[str, List[InstanceTypeInfo]]): A\n dictionary of canonical accelerator names mapped to a list of\n instance type offerings. See usage in cli.py.\n\n### `sky.kubernetes_node_info`\n\n```python\nsky.kubernetes_node_info(context: Optional[str] = None) -> server_common.RequestId['models.KubernetesNodesInfo']\n```\n\nGets the resource information for all the nodes in the cluster.\n\nCurrently only GPU resources are supported. The function returns the total\nnumber of GPUs available on the node and the number of free GPUs on the\nnode.\n\nIf the user does not have sufficient permissions to list pods in all\nnamespaces, the function will return free GPUs as -1.\n\n**Args:**\n context: The Kubernetes context. If None, the default context is used.\n\n**Returns:**\n The request ID of the Kubernetes node info request.\n\n**Request Returns:**\n KubernetesNodesInfo: A model that contains the node info map and other\n information.\n\n### `sky.realtime_kubernetes_gpu_availability`\n\n```python\nsky.realtime_kubernetes_gpu_availability(context: Optional[str] = None, name_filter: Optional[str] = None, quantity_filter: Optional[int] = None, is_ssh: Optional[bool] = None) -> server_common.RequestId[List[Tuple[str, List['models.RealtimeGpuAvailability']]]]\n```\n\nGets the real-time Kubernetes GPU availability.\n\n**Returns:**\n The request ID of the real-time Kubernetes GPU availability request.\n\n### `sky.optimize`\n\n```python\nsky.optimize(dag: 'sky.Dag', minimize: common.OptimizeTarget = common.OptimizeTarget.COST, admin_policy_request_options: Optional[admin_policy.RequestOptions] = None) -> server_common.RequestId['sky.Dag']\n```\n\nFinds the best execution plan for the given DAG.\n\n**Args:**\n dag: the DAG to optimize.\n minimize: whether to minimize cost or time.\n admin_policy_request_options: Request options used for admin policy\n validation. This is only required when a admin policy is in use,\n see: https://docs.skypilot.co/en/latest/cloud-setup/policy.html\n\n**Returns:**\n The request ID of the optimize request.\n\n**Request Returns:**\n optimized_dag (str): The optimized DAG in YAML format.\n\n**Request Raises:**\n exceptions.ResourcesUnavailableError: if no resources are available\n for a task.\n exceptions.NoCloudAccessError: if no public clouds are enabled.\n\n### `sky.validate`\n\n```python\nsky.validate(dag: 'sky.Dag', workdir_only: bool = False, admin_policy_request_options: Optional[admin_policy.RequestOptions] = None, _skip_local_paths: bool = False) -> None\n```\n\nValidates the tasks.\n\nThe file paths (workdir and file_mounts) are validated on the client side\nwhile the rest (e.g. resource) are validated on server side.\n\nRaises exceptions if the DAG is invalid.\n\n**Args:**\n dag: the DAG to validate.\n workdir_only: whether to only validate the workdir. This is used for\n `exec` as it does not need other files/folders in file_mounts.\n admin_policy_request_options: Request options used for admin policy\n validation. This is only required when a admin policy is in use,\n see: https://docs.skypilot.co/en/latest/cloud-setup/policy.html\n _skip_local_paths: Do not check the workdir and file mount sources on\n this machine. For callers whose sources are API server paths\n inside an upload blob; the server resolves the blob and checks\n the paths when it runs the request.\n\n### `sky.reload_config`\n\n```python\nsky.reload_config() -> None\n```\n\nReloads the client-side config.\n\n## Storage & Volumes\n\n### `sky.storage_ls`\n\n```python\nsky.storage_ls() -> server_common.RequestId[List[responses.StorageRecord]]\n```\n\nGets the storages.\n\n**Returns:**\n The request ID of the storage list request.\n\n**Request Returns:**\n storage_records (List[responses.StorageRecord]):\n A list of storage records.\n\n### `sky.storage_delete`\n\n```python\nsky.storage_delete(name: str) -> server_common.RequestId[None]\n```\n\nDeletes a storage.\n\n**Args:**\n name: The name of the storage to delete.\n\n**Returns:**\n The request ID of the storage delete request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: If the storage does not exist.\n\n## API Server\n\n### `sky.api_start`\n\n```python\nsky.api_start(*, deploy: bool = False, host: str = '127.0.0.1', foreground: bool = False, metrics: bool = False, metrics_port: Optional[int] = None, enable_basic_auth: bool = False, port: Optional[int] = None) -> None\n```\n\nStarts the API server.\n\nIt checks the existence of the API server and starts it if it does not\nexist.\n\n**Args:**\n deploy: Whether to deploy the API server, i.e. fully utilize the\n resources of the machine.\n host: The host to bind the API server to. Under deploy, the server\n always binds a wildcard address for remote access: ``::`` when an\n IPv6 host is given, otherwise ``0.0.0.0``. Without deploy the host\n is used as-is.\n foreground: Whether to run the API server in the foreground (run in\n the current process).\n metrics: Whether to export metrics of the API server.\n metrics_port: The port to export metrics of the API server.\n enable_basic_auth: Whether to enable basic authentication\n in the API server.\n port: The port to bind the API server to. Defaults to the\n SKYPILOT_API_SERVER_LOCAL_PORT environment variable, or 46580.\n Other client commands only find a server on a non-default port\n if the same environment variable is exported.\n**Returns:**\n None\n\n### `sky.api_stop`\n\n```python\nsky.api_stop() -> None\n```\n\nStops the API server.\n\nIt will do nothing if the API server is remotely hosted.\n\n**Returns:**\n None\n\n### `sky.api_status`\n\n```python\nsky.api_status(request_ids: Optional[List[Union[server_common.RequestId[T], str]]] = None, all_status: bool = False, limit: Optional[int] = None, fields: Optional[List[str]] = None, cluster_name: Optional[str] = None) -> List[payloads.RequestPayload]\n```\n\nLists all requests.\n\n**Args:**\n request_ids: The prefixes of the request IDs of the requests to query.\n If None, all requests are queried.\n all_status: Whether to list all finished requests as well. This argument\n is ignored if request_ids is not None.\n limit: The number of requests to show. If None, show all requests.\n fields: The fields to get. If None, get all fields except\n return_value and error. This argument is ignored if request_ids\n is not None.\n cluster_name: Filter requests by cluster name.\n If None, show all requests.\n\n**Returns:**\n A list of request payloads.\n\n### `sky.api_info`\n\n```python\nsky.api_info() -> responses.APIHealthResponse\n```\n\nGets the server's status, commit and version.\n\n**Returns:**\n A dictionary containing the server's status, commit and version.\n\n ```python\n\n {\n 'status': 'healthy',\n 'api_version': '1',\n 'commit': 'abc1234567890',\n 'version': '1.0.0',\n 'version_on_disk': '1.0.0',\n 'user': {\n 'name': 'test@example.com',\n 'id': '12345abcd',\n },\n }\n\n Note that user may be None if we are not using an auth proxy.\n\n### `sky.api_cancel`\n\n```python\nsky.api_cancel(request_ids: Optional[Union[server_common.RequestId[T], List[server_common.RequestId[T]], str, List[str]]] = None, all_users: bool = False, silent: bool = False) -> server_common.RequestId[List[str]]\n```\n\nAborts a request or all requests.\n\nWith neither request_ids nor all_users, aborts all of the caller's own\nrequests.\n\n**Args:**\n request_ids: The request ID(s) to abort. Can be a single string or a\n list of strings. On a server with per-user auth, an admin can\n abort any request by id and other users only their own; on a\n server without it, anyone can abort any request by id.\n all_users: Whether to abort all requests from all users. On a server\n with per-user auth, a non-admin aborts only their own. Cannot be\n combined with request_ids.\n silent: Whether to suppress the output.\n\n**Returns:**\n The request ID of the abort request itself.\n\n**Request Returns:**\n A list of request IDs that were cancelled.\n\n**Raises:**\n ValueError: If both request_ids and all_users are given.\n\n### `sky.api_server_logs`\n\n```python\nsky.api_server_logs(follow: bool = True, tail: Optional[int] = None) -> None\n```\n\nStreams the API server logs.\n\n**Args:**\n follow: Whether to follow the logs.\n tail: the number of lines to show from the end of the logs.\n If None, show all logs.\n\n**Returns:**\n None\n\n### `sky.api_login`\n\n```python\nsky.api_login(endpoint: Optional[str] = None, relogin: bool = False, service_account_token: Optional[str] = None, no_browser: bool = False) -> None\n```\n\nLogs into a SkyPilot API server.\n\nThis sets the endpoint globally, i.e., all SkyPilot CLI and SDK calls will\nuse this endpoint.\n\nTo temporarily override the endpoint, use the environment variable\n`SKYPILOT_API_SERVER_ENDPOINT` instead.\n\n**Args:**\n endpoint: The endpoint of the SkyPilot API server, e.g.,\n http://1.2.3.4:46580 or https://skypilot.mydomain.com.\n relogin: Whether to force relogin with OAuth2 when enabled.\n service_account_token: Service account token for authentication.\n no_browser: If True, do not attempt to open a browser locally; print\n the auth URL and let the user open it themselves. Skips the\n localhost-callback flow, which requires a local browser.\n\n**Returns:**\n None\n\n### `sky.api_logout`\n\n```python\nsky.api_logout() -> None\n```\n\nLogout of the API server.\n\nClears all cookies and settings stored in ~/.sky/config.yaml\n\n## Utilities\n\n### `sky.dashboard`\n\n```python\nsky.dashboard(starting_page: Optional[str] = None) -> None\n```\n\nStarts the dashboard for SkyPilot.\n\n### `sky.get`\n\n```python\nsky.get(request_id: server_common.RequestId[T]) -> T\n```\n\nWaits for and gets the result of a request.\n\nThis function will not check the server health since /api/get is typically\nnot the first API call in an SDK session and checking the server health\nmay cause GET /api/get being sent to a restarted API server.\n\n**Args:**\n request_id: The request ID of the request to get. May be a full request\n ID or a prefix.\n\n**Returns:**\n The ``Request Returns`` of the specified request. See the documentation\n of the specific requests above for more details.\n\n**Raises:**\n Exception: It raises the same exceptions as the specific requests,\n see ``Request Raises`` in the documentation of the specific requests\n above.\n\n### `sky.stream_and_get`\n\n```python\nsky.stream_and_get(request_id: Optional[server_common.RequestId[T]] = None, log_path: Optional[str] = None, tail: Optional[int] = None, follow: bool = True, output_stream: Optional['io.TextIOBase'] = None, relay_rich_status: bool = False) -> Optional[T]\n```\n\nStreams the logs of a request or a log file and gets the final result.\n\nThis will block until the request is finished. The request id can be a\nprefix of the full request id.\n\n**Args:**\n request_id: The request ID of the request to stream. May be a full\n request ID or a prefix.\n If None, the latest request submitted to the API server is streamed.\n Using None request_id is not recommended in multi-user environments.\n log_path: The path to the log file to stream.\n tail: The number of lines to show from the end of the logs.\n If None, show all logs.\n follow: Whether to follow the logs.\n output_stream: The output stream to write to. If None, print to the\n console.\n relay_rich_status: If True, forward encoded rich-status control payloads\n verbatim to the output instead of rendering a local spinner. Used by\n the managed jobs controller to preserve provisioning spinner codes\n in its per-job log. See\n :func:`sky.utils.rich_utils.decode_rich_status`.\n\n**Returns:**\n The ``Request Returns`` of the specified request. See the documentation\n of the specific requests above for more details.\n If follow is False, will always return None. See note on\n stream_response.\n\n**Raises:**\n Exception: It raises the same exceptions as the specific requests,\n see ``Request Raises`` in the documentation of the specific requests\n above.\n\n### `sky.workspaces`\n\n```python\nsky.workspaces() -> server_common.RequestId[Dict[str, Any]]\n```\n\nGets the workspaces.\n\n### `sky.set_preferred_workspace`\n\n```python\nsky.set_preferred_workspace(preferred: Optional[str]) -> Dict[str, Any]\n```\n\nSets (or clears with None) the user's preferred workspace.\n\n**Args:**\n preferred: workspace name to set as default, or None to clear.\n\n**Returns:**\n ``{'preferred': }`` echoing what was set. Callers that\n need the resolved workspace + accessible list should follow up\n with :func:`get_user_workspace`. Raises if the server rejects\n the change (workspace does not exist, or user lacks permission\n to it).\n\n### `sky.get_user_workspace`\n\n```python\nsky.get_user_workspace(requested: Optional[str] = None) -> Dict[str, Any]\n```\n\nReturns workspace state for the calling user.\n\nMirrors the launch-path precedence — if the caller has an explicit\n``active_workspace``, the server returns that with ``source='explicit'``;\notherwise the resolver runs (preferred / default-fallback /\nsingle-membership).\n\n**Args:**\n requested: explicit active workspace to ask about. ``None`` (the\n default) — the SDK reads your locally-configured\n ``active_workspace`` (the value `skypilot_config` merges\n from ``~/.sky/config.yaml`` + ``./.sky.yaml`` + any\n ``--config active_workspace=X`` override) and forwards it\n on the wire as ``?requested=``. Pass a non-None value to\n query the resolver as if ``active_workspace`` were that\n value, without changing your local config — useful for\n previewing \"what would land if I switched to X\".\n\n**Returns:**\n ``{workspace, source, note, preferred, accessible}``.\n\n * ``workspace``: the workspace the launch path would pick. Can\n be ``None`` when the resolver couldn't pick (no access /\n ambiguous / explicit ``requested`` rejected by RBAC); the\n reason is then in ``note``.\n * ``source``: one of ``WORKSPACE_SOURCE_*`` on success, ``None``\n when ``workspace`` is ``None``.\n * ``note``: optional message — drift on success\n (``preferred 'team-x' not accessible``) or the resolver error\n when ``workspace`` is ``None``.\n * ``preferred``: the persisted preferred workspace (``None`` if\n unset).\n * ``accessible``: sorted list of workspaces the user can launch\n into.\n\n## Other Functions\n\n### `sky.create_debug_dump`\n\n```python\nsky.create_debug_dump(request_ids: Optional[List[str]] = None, cluster_names: Optional[List[str]] = None, managed_job_ids: Optional[List[int]] = None, recent_minutes: Optional[float] = None) -> server_common.RequestId[str]\n```\n\nCreate a debug dump for troubleshooting.\n\nAt least one of ``request_ids``, ``cluster_names``, ``managed_job_ids``,\nor ``recent_minutes`` must be provided.\n\n**Args:**\n request_ids: List of request IDs or prefixes to include in the\n dump. Prefixes are resolved to all matching request IDs on\n the server.\n cluster_names: List of cluster names to include in the dump.\n managed_job_ids: List of managed job IDs to include in the dump.\n recent_minutes: If specified, include all resources active within\n this many minutes.\n\n**Returns:**\n The request ID of the debug dump creation request.\n\n**Request Returns:**\n Path to the created zip file on the server.\n\n### `sky.download_debug_dump`\n\n```python\nsky.do","isInternal":false,"tokens":11558,"sizeBytes":50008},{"name":"troubleshooting.md","path":"agent/skills/skypilot/references/troubleshooting.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/troubleshooting.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `agent/skills/skypilot/references/troubleshooting.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/troubleshooting.md)","tokens":300,"sizeBytes":223},{"name":"yaml-spec.md","path":"agent/skills/skypilot/references/yaml-spec.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/yaml-spec.md","title":"References Skill","category":"anthropic-skill","format":"markdown","content":"# References Skill\n\nPath: `agent/skills/skypilot/references/yaml-spec.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/yaml-spec.md)","tokens":300,"sizeBytes":211},{"name":"SKILL.md","path":"agent/skills/skypilot/SKILL.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/SKILL.md","title":"skypilot","category":"anthropic-skill","format":"markdown","content":"---\nname: skypilot\ndescription: \"Use when launching cloud VMs, Kubernetes pods, or Slurm jobs for GPU/TPU/CPU workloads, training or fine-tuning models on cloud GPUs, deploying inference servers (vllm, TGI, etc.) with autoscaling, writing or debugging SkyPilot task YAML files, using spot/preemptible instances for cost savings, comparing GPU prices across clouds, managing compute across 25+ clouds, Kubernetes, Slurm, and on-prem clusters with failover between them, troubleshooting resource availability or SkyPilot errors, optimizing cost and GPU availability, or migrating an existing Slurm workload (converting sbatch scripts, salloc sessions, job arrays or srun invocations into SkyPilot task YAMLs, and mapping Slurm commands, directives and SLURM_* environment variables to their SkyPilot equivalents).\"\n---\n\n# SkyPilot Skill\n\nSkyPilot is a unified framework to run AI workloads on any cloud, Slurm or Kubernetes. It provides a single interface to launch clusters, run jobs, and serve models across 25+ clouds (AWS, GCP, Azure, Coreweave, Nebius, Lambda, Together AI, RunPod, and more), Kubernetes clusters, and Slurm clusters.\n\n## When to Use SkyPilot\n\n**Use SkyPilot when you need to:**\n- Manage compute resources on any cloud, Slurm, or Kubernetes cluster\n- Launch CPU/GPU/TPU (GB300, GB200, B200, H200, H100, etc.) on any cloud, Kubernetes or Slurm\n- Run training, fine-tuning, or batch inference jobs\n- Serve models with autoscaling and multi-cloud replicas (SkyServe)\n- Run long-running jobs with automatic lifecycle management and recovery (managed jobs)\n- Find the cheapest or most available GPU across clouds\n\n**Don't use SkyPilot for:**\n- Local-only workloads (use Docker/conda directly)\n\n## Capabilities: When to Use What\n\nSkyPilot has three core abstractions. Use the right one for each stage of your workflow:\n\n**1. SkyPilot Clusters** (`sky launch` / `sky exec`) — Interactive development and debugging\n- Use during initial development, debugging, and experimentation\n- Launch a cluster, SSH in or connect VSCode/Cursor (`code --remote ssh-remote+CLUSTER`), iterate quickly\n- Cluster stays up until you stop/down it or autostop triggers\n- Best for: prototyping, debugging, short experiments\n\n**2. Managed Jobs** (`sky jobs launch`) — Long-running training and batch jobs\n- Use when submitting long-running jobs that should run unattended\n- Manages the full lifecycle: provisioning, execution, recovery, and teardown\n- Automatically recovers from spot preemptions, quota limits, and transient failures\n- Works across clouds, Kubernetes, and Slurm (handles preemptions and quota)\n- Best for: training runs, fine-tuning, hyperparameter sweeps, batch inference\n\n**3. SkyServe** (`sky serve up`) — Production model serving\n- Use when serving models at scale with autoscaling\n- Start with `sky launch` + open port to test your serving setup, then use `sky serve up` to scale\n- Provides load balancing, autoscaling, and multi-cloud replicas\n- Best for: model serving endpoints, API services\n\n## Before You Start (Agent Bootstrap)\n\nBootstrap to confirm SkyPilot is installed, connected to an API server, and has cloud credentials. Once confirmed, skip straight to the user's task.\n\n**Step 1: Check installation and API server connectivity**\n\n```bash\nsky api info\n```\n| Output contains | Meaning | Next action |\n|-----------------|---------|-------------|\n| Server version and status | Server is running and connected | **Bootstrap done.** Skip to user's task. |\n| `No SkyPilot API server is connected` | No server connected | Go to \"Start or connect a server\" below. |\n| `Could not connect to SkyPilot API server` | Remote server unreachable or auth expired | Tell the user and suggest `sky api login --relogin -e ` to reconnect. |\n| `command not found: sky` | SkyPilot not installed | Go to \"Install SkyPilot\" below. |\n\n**Install SkyPilot** (only if `sky` command not found):\n```bash\npip install \"skypilot[aws,gcp,kubernetes]\" # Pick clouds the user needs\n```\nAsk the user which clouds they need if unclear, then re-run `sky api info`.\n\n**Start or connect a server** (only if \"not running\"):\n\nAsk the user:\n> Do you have an existing SkyPilot API server to connect to, or should I start one locally?\n\n- **Connect to existing server:** `sky api login -e ` — get the URL from the user.\n- **Start locally:** `sky api start`\n\nAfter either path, re-run `sky api info` to confirm the server is reachable.\n\n**Step 2: Check cloud credentials** (only for fresh setups — skip if the server was already running)\n```bash\nsky check -o json\n```\nThis shows which clouds are enabled or disabled. If the user's target cloud is not enabled, guide them through credential setup (see [Troubleshooting](references/troubleshooting.md#1-installation-and-credentials)).\n\n## Essential Commands\n\nUse `-o json` with status/query commands to get structured JSON output instead of tables.\n\n**Clusters** — interactive development and debugging:\n\n| Command | Description |\n|---------|-------------|\n| `sky launch -c NAME task.yaml` | Launch a cluster or run a task |\n| `sky exec NAME task.yaml` | Run task on existing cluster (skips provisioning); syncs workdir each time |\n| `sky exec NAME task.yaml -d` | Same, but detach immediately (don't stream logs) |\n| `sky status -o json` | Show all clusters |\n| `sky logs NAME` | Stream job logs from a cluster |\n| `sky logs NAME --no-follow` | Print existing logs and exit immediately |\n| `sky logs NAME --tail 50` | Print last 50 lines of logs and exit |\n| `sky logs NAME --status` | Exit with code 0=succeeded, 100=failed, 101=not finished, 102=not found, 103=cancelled |\n| `sky queue NAME -o json` | List jobs on a cluster with status (structured JSON) |\n| `sky stop NAME` / `sky start NAME` | Stop/restart to save costs (preserves disk) |\n| `sky down NAME` | Tear down a cluster completely |\n| `sky gpus list -o json` | List available GPU types across clouds |\n\n**Managed Jobs** — long-running unattended workloads:\n\n| Command | Description |\n|---------|-------------|\n| `sky jobs launch task.yaml` | Launch a managed job (auto lifecycle + recovery) |\n| `sky jobs queue -o json` | Show all managed jobs and their status |\n| `sky jobs logs JOB_ID` | Stream logs from a managed job |\n| `sky jobs cancel JOB_ID` | Cancel a managed job |\n\n**SkyServe** — model serving with autoscaling:\n\n| Command | Description |\n|---------|-------------|\n| `sky serve up serve.yaml -n NAME` | Start a model serving service |\n| `sky serve status NAME` | Show service status and endpoint URL |\n| `sky serve update NAME new.yaml` | Update a running service (rolling) |\n| `sky serve down NAME` | Tear down a service |\n\nFor complete CLI reference, see [CLI Reference](references/cli-reference.md).\n\n## Quick Start\n\n```bash\n# Launch a GPU cluster\nsky launch -c mycluster --gpus H100 -- nvidia-smi\n\n# Run a task from YAML\nsky launch -c mycluster task.yaml\n\n# SSH into cluster\nssh mycluster\n\n# Connect VSCode or Cursor to the cluster for interactive development\ncode --remote ssh-remote+mycluster /home/user/sky_workdir\n# or: cursor --remote ssh-remote+mycluster /home/user/sky_workdir\n\n# Tear down\nsky down mycluster\n```\n\n## Task YAML Structure\n\nThe task YAML is SkyPilot's primary interface. All fields are optional.\n\n```yaml\n# task.yaml\nname: my-training-job\n\n# Local directory to sync to remote ~/sky_workdir\nworkdir: .\n\n# Number of nodes (for distributed training)\nnum_nodes: 1\n\nresources:\n # GPU/TPU accelerators (SkyPilot auto-selects the cheapest cloud/region)\n accelerators: H200:8\n # Optional: pin to a specific cloud/region/infra\n # infra: aws # or aws/us-east-1, k8s, ssh/my-pool\n # If infra is left out, SkyPilot automatically fails over across all\n # enabled clouds/regions to find the cheapest available option.\n # Use spot instances for cost savings\n use_spot: false\n # Disk size in GB\n disk_size: 256\n # Open ports for serving\n ports: 8080\n\n# Environment variables (accessible in file_mounts, setup, and run)\nenvs:\n MODEL_NAME: my-model\n BATCH_SIZE: 32\n\n# Setup: runs once on cluster creation, cached on reuse\nsetup: |\n pip install torch transformers\n\n# Run: the main command\nrun: |\n python train.py --model $MODEL_NAME --batch-size $BATCH_SIZE\n```\n\nFor complete YAML schema including file mounts, environment variables set by SkyPilot, and advanced fields, see [YAML Specification](references/yaml-spec.md).\n\n## GPU and Cloud Selection\n\n**IMPORTANT: Let SkyPilot choose the cloud and region.** Do NOT manually pick a cloud/region/instance by parsing `sky gpus list` output. SkyPilot's optimizer automatically selects the cheapest available option across all enabled clouds. Only specify `infra:` when the user explicitly requests a specific cloud or region.\n\n**Default behavior (recommended):** Just specify the GPU type. SkyPilot finds the cheapest cloud/region automatically:\n\n```yaml\nresources:\n accelerators: H200:8 # SkyPilot picks the cheapest cloud/region with H200:8\n```\n\nIf the user doesn't specify a GPU type, ask them what GPU they need (or what model/workload they're running so you can recommend one). Do NOT run `sky gpus list` and pick for them — present options and let the user decide, or use `any_of` to let SkyPilot maximize availability:\n\n```yaml\n# Let SkyPilot choose from multiple acceptable GPU types (cheapest wins)\nresources:\n any_of:\n - accelerators: H100:8\n - accelerators: A100-80GB:8\n - accelerators: A100:8\n```\n\nUse `ordered` only when the user has a strict preference:\n\n```yaml\n# Try H100 first on AWS, fall back to GCP, then A100\nresources:\n ordered:\n - infra: aws/us-east-1\n accelerators: H100:8\n - infra: gcp/us-central1\n accelerators: H100:8\n - infra: aws/us-west-2\n accelerators: A100-80GB:8\n```\n\nOnly set `infra:` when the user explicitly says something like \"use AWS\" or \"run on GCP us-central1\":\n\n```yaml\nresources:\n infra: aws # User asked for AWS specifically\n accelerators: H100:8\n```\n\n## Cluster Lifecycle\n\n```bash\n# Launch and run a task\nsky launch -c mycluster task.yaml\n\n# Launch with autostop at launch time (preferred: saves cost, no follow-up command needed)\nsky launch -c mycluster task.yaml -i 30 # stop after 30 min idle\nsky launch -c mycluster task.yaml -i 30 --down # tear down after 30 min idle\n\n# Override or pass environment variables via CLI\nsky launch -c mycluster task.yaml --env MODEL_NAME=llama3 --env BATCH_SIZE=64\n\n# Re-run a different task on the same cluster (fast, skips provisioning)\nsky exec mycluster another_task.yaml\n\n# Run an inline command\nsky exec mycluster -- python train.py --epochs 10\n\n# Set autostop after launch (use if you forgot to set -i at launch time)\nsky autostop mycluster -i 30 # stop after 30 min idle, preserving disk (can restart with sky start)\nsky autostop mycluster -i 30 --down # tear down after 30 min idle (disk is deleted, cannot restart)\n\n# Stop to save costs, restart later\nsky stop mycluster\nsky start mycluster\n\n# Tear down completely\nsky down mycluster\n```\n\n## Workdir Sync Behavior\n\n`workdir:` is synced to `~/sky_workdir` on the remote via `rsync` before every `sky exec`. **rsync is additive — deleted local files are NOT removed from the remote.** This can cause experiments to run against stale build artifacts or old configs.\n\nTo ensure a clean slate, SSH and wipe before `sky exec`:\n```bash\nssh mycluster \"rm -rf ~/sky_workdir\"\nsky exec mycluster task.yaml\n```\n\nOr clean inside `run:` if only specific artifacts need removal:\n```yaml\nrun: |\n find ~/sky_workdir/build -name '*.o' -delete 2>/dev/null || true\n cd ~/sky_workdir && make\n```\n\n## Managed Jobs\n\nUse `sky jobs launch` for long-running jobs that should run unattended. SkyPilot manages the full lifecycle — provisioning, execution, recovery from preemptions/quota/failures, and teardown:\n\n```yaml\n# managed-job.yaml\nname: training-job\n\nresources:\n accelerators: A100:8\n\nrun: |\n python train.py --resume-from-checkpoint\n```\n\n```bash\n# Launch as managed job\nsky jobs launch managed-job.yaml\n\n# Check status\nsky jobs queue -o json\n\n# Stream logs\nsky jobs logs \n\n# Cancel\nsky jobs cancel \n```\n\n**Checkpoint pattern**: Your training script should save checkpoints to persistent storage (cloud bucket or volume) and resume from the latest checkpoint on restart. SkyPilot handles the cluster recovery; your script handles the state recovery.\n\n## SkyServe: Model Serving\n\n```yaml\n# serve.yaml\nresources:\n accelerators: A100:1\n ports: 8080\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-3.1-8B-Instruct \\\n --port 8080\n\nservice:\n readiness_probe: /v1/models\n replica_policy:\n min_replicas: 1\n max_replicas: 3\n target_qps_per_replica: 5\n```\n\n```bash\n# Start service\nsky serve up serve.yaml -n my-llm\n\n# Check status / get endpoint\nsky serve status my-llm\nsky serve status my-llm --endpoint\n\n# Update (rolling)\nsky serve update my-llm new-serve.yaml\n\n# Tear down\nsky serve down my-llm\n```\n\n## Common Workflows\n\n### Fine-Tuning Workflow\n1. Write task YAML with `setup` (install deps) and `run` (training command)\n2. Use `file_mounts` or `workdir` to sync code\n3. `sky launch -c train task.yaml` to launch\n4. `sky logs train` to monitor\n5. `sky exec train -- python eval.py` to evaluate on same cluster\n6. `sky down train` when done\n\n### Hyperparameter Sweep\n1. Create parameterized YAML with `envs`\n2. Launch multiple managed jobs:\n ```bash\n for lr in 1e-4 1e-5 1e-6; do\n sky jobs launch sweep.yaml --env LR=$lr --name sweep-lr-$lr\n done\n ```\n3. Monitor with `sky jobs queue -o json`\n\n### Model Serving Deployment\n1. Write serve YAML with `service:` section\n2. `sky serve up serve.yaml -n my-service`\n3. Get endpoint: `sky serve status my-service --endpoint`\n4. Update model: `sky serve update my-service updated.yaml`\n\n### Parallel Experiment Submission\n\nUse `sky exec -d` to submit jobs to multiple VMs without blocking, then collect results:\n\n```bash\n# Submit all experiments (detached, returns after job is queued)\nfor i in 1 2 3 4; do\n sky exec exp-vm-0$i task.yaml --env LR=1e-$i -d\ndone\n\n# Get the latest job ID from a cluster\njob_id=$(sky queue exp-vm-01 -o json \\\n | python3 -c \"import sys, json; jobs = json.load(sys.stdin).get('exp-vm-01', []); print(max(j['job_id'] for j in jobs) if jobs else '')\")\n\n# Wait for a specific job and fetch last 50 lines\nsky logs exp-vm-01 $job_id --status && sky logs exp-vm-01 $job_id --tail 50\n\n# Check all jobs across a cluster at once\nsky queue exp-vm-01 -o json\n```\n\n### Converting a Slurm Workload\n\nWhen the user has an existing `sbatch` script, `salloc` workflow, or asks how a\nSlurm command maps to SkyPilot, **read\n[Migrating from Slurm](references/migrating-from-slurm.md) before writing any\nYAML.** The mapping has non-obvious parts that are easy to get wrong:\n\n- `--time` does **not** map to `autostop` — autostop is unsupported on Slurm.\n Use `config.slurm.sbatch_options.time`.\n- A bare `srun ` should usually be **dropped**, not translated: `run`\n already executes on every node. Only MPI/PMIx launchers keep `srun`, and\n then they need `--overlap` and a rank-0 guard.\n- `--account` / `--qos` / `--exclusive` and other unmodeled directives go\n through `config.slurm.sbatch_options`, which passes them to `sbatch`\n verbatim.\n- `sbatch --array` maps to `sky jobs launch --num-jobs` with\n `$SKYPILOT_JOB_RANK`, not to a shell loop.\n\nValidate the result with `sky launch --dryrun ` — note there is no\n`--dryrun` on `sky jobs launch`, so dry-run via `sky launch` even when the\nfinal command will be `sky jobs launch`.\n\nDo not assume the user wants to leave Slurm. SkyPilot submits to an existing\nSlurm cluster through its login node; migrating *to Kubernetes* is a separate\nquestion, and the constraints differ.\n\n## Agent Feedback Loop\n\nWhen using SkyPilot programmatically, follow this loop:\n\n1. **Validate**: `sky launch --dryrun task.yaml` (check resource availability/cost)\n2. **Launch**: `sky launch -c mycluster task.yaml`\n3. **Monitor**: `sky status -o json` and `sky queue mycluster -o json`\n4. **Wait for completion**: `sky logs mycluster ` (streams logs so you can observe progress and react to stalls; blocks until job finishes; get JOB_ID from `sky queue mycluster -o json`). For long-running jobs where you don't need intermediate output, use `sky logs mycluster --status` instead (blocks silently, exits 0 on success).\n5. **Inspect output**: `sky logs mycluster --no-follow` or `sky logs mycluster --tail 100`\n6. **Debug**: `ssh mycluster` (interactive)\n7. **Iterate**: `sky exec mycluster updated_task.yaml` (run on existing cluster)\n8. **Cleanup**: `sky down mycluster`\n\n> **Never poll with `sleep` + `sky queue`** — use `sky logs CLUSTER JOB_ID` to stream logs and block until done. Use `--status` if you only need the exit code, or `--tail N` to fetch recent output after completion.\n\n## Investigating a Managed Job\n\nWhen the user asks \"what is my job doing\" or \"why is it pending/failing\", run\nthese in order and stop as soon as the question is answered:\n\n1. `sky jobs queue -v -o json` — `status`, `details`, `failure_reason` (`-v` is what adds `details`). For a job that has not started, `details` is the answer: it says what the cluster is waiting on, e.g. a Slurm `QOSGrpGRES` or `Dependency` reason with the partition it is queued in\n2. `sky jobs logs JOB_ID --tail 100 --no-follow` — recent task output; `--controller` for provisioning/recovery logs\n\nSee [Job Investigation](references/job-investigation.md) for how to read `details` and the Slurm pending reasons.\n\n## Common Agent Mistakes\n\n| Mistake | Why it's wrong | Do this instead |\n|---------|---------------|-----------------|\n| Manually picking cloud/region from `sky gpus list` output | SkyPilot optimizer does this automatically and better | Just set `accelerators:` and let SkyPilot choose |\n| Using `sky launch` for long-running unattended jobs | No recovery if preempted or interrupted | Use `sky jobs launch` for unattended work |\n| Forgetting `sky down` or autostop after work is done | Wastes money on idle clusters | Always clean up, or use `-i --down` at launch |\n| Hardcoding `infra: aws` without user asking | Limits availability and increases cost | Only set `infra:` when user explicitly requests a cloud |\n| Not using `envs:` for configurable values | Hard to reuse or override from CLI | Use `envs:` in YAML + `--env KEY=VAL` for parameterization |\n| Running `sky launch` without `-c ` | Creates randomly-named cluster, hard to reference | Always name clusters with `-c` |\n| Parsing table output from status commands | Table formatting is for humans, fragile to parse | Use `-o json` for structured output |\n| Using deprecated `cloud:`/`region:`/`zone:` fields | Deprecated in favor of `infra:` | Use `infra: aws/us-east-1` instead |\n| Polling job status with `sleep` + `sky queue` | Wastes tokens, introduces timing bugs, fragile | Use `sky logs CLUSTER JOB_ID --status` to block until done |\n| Assuming workdir sync removes remote files | rsync is additive; old remote files persist across `sky exec` calls | SSH and manually clean `~/sky_workdir`, or clean in `run:` script |\n| Not using `--tail` when only last output matters | Streaming full logs wastes tokens for long jobs | Use `sky logs CLUSTER JOB_ID --tail 50` for last N lines |\n\n## Common Issues Quick Reference\n\n| Issue | Solution |\n|-------|----------|\n| GPU not available | Use `any_of` for fallback, or try different regions/clouds |\n| Setup takes too long | SkyPilot caches setup; use `sky exec` to skip it on reruns |\n| Task fails silently | Check `sky logs ` or `ssh ` to debug |\n| Cluster stuck in INIT | `sky down ` and relaunch |\n| Preemption/quota | Use `sky jobs launch` for automatic recovery and lifecycle management |\n| Port not accessible | Ensure `ports:` is set in resources and security groups allow traffic |\n| File sync slow | Use cloud bucket mounts instead of `workdir` for large datasets |\n| Credentials error | Run `sky check -o json` and inspect which clouds are disabled |\n\n## References\n\nFor detailed reference documentation:\n\n- [CLI Reference](references/cli-reference.md) — All commands and flags\n- [YAML Specification](references/yaml-spec.md) — Complete task YAML schema, file mounts, environment variables\n- [Python SDK](references/python-sdk.md) — Programmatic API and SDK usage\n- [Advanced Patterns](references/advanced-patterns.md) — Multi-cloud, distributed training, production patterns\n- [Migrating from Slurm](references/migrating-from-slurm.md) — Converting `sbatch` scripts to task YAMLs, Slurm command/env-var mapping, Slurm-specific constraints\n- [Job Investigation](references/job-investigation.md) — Which command answers \"what is my job doing and why\"; reading `details`, events, Slurm pending reasons\n- [Troubleshooting](references/troubleshooting.md) — Error diagnosis and solutions\n- [Examples](references/examples.md) — Copy-paste task YAML examples\n","frontmatter":{"name":"skypilot","description":"Use when launching cloud VMs, Kubernetes pods, or Slurm jobs for GPU/TPU/CPU workloads, training or fine-tuning models on cloud GPUs, deploying inference servers (vllm, TGI, etc.) with autoscaling, writing or debugging SkyPilot task YAML files, using spot/preemptible instances for cost savings, comparing GPU prices across clouds, managing compute across 25+ clouds, Kubernetes, Slurm, and on-prem clusters with failover between them, troubleshooting resource availability or SkyPilot errors, optimizing cost and GPU availability, or migrating an existing Slurm workload (converting sbatch scripts, salloc sessions, job arrays or srun invocations into SkyPilot task YAMLs, and mapping Slurm commands, directives and SLURM_* environment variables to their SkyPilot equivalents)."},"isInternal":false,"tokens":5345,"sizeBytes":21036},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md - SkyPilot Development Guide\n\nThis document provides guidance for AI assistants working with the SkyPilot codebase.\n\n## Project Overview\n\nSkyPilot is a system to run, manage, and scale AI workloads on any AI infrastructure. It provides a unified interface across 25+ cloud providers (AWS, GCP, Azure, Kubernetes, Slurm, and many others), enabling users to launch compute resources, run jobs, and serve models without vendor lock-in.\n\n**Key capabilities:**\n- **Clusters**: Launch and manage compute clusters across clouds and Kubernetes\n- **Managed jobs**: Run jobs with automatic recovery from preemptions and failures\n- **Multi-cloud and multi-Kubernetes**: Unified interface across 25+ clouds and multiple K8s clusters\n- Cost optimization and GPU availability maximization\n\n## Repository Structure\n\n```\nskypilot/\n├── sky/ # Main source code\n│ ├── __init__.py # Package exports and version\n│ ├── core.py # Core orchestration logic\n│ ├── task.py # Task definition and YAML parsing\n│ ├── resources.py # Cloud resource specifications\n│ ├── optimizer.py # Resource allocation optimizer\n│ ├── execution.py # Task execution pipeline\n│ ├── exceptions.py # Custom exceptions\n│ ├── skypilot_config.py # Configuration management\n│ ├── clouds/ # Cloud provider abstractions (25+ providers)\n│ ├── backends/ # Execution backends (Ray-based VM, Docker)\n│ ├── provision/ # Cloud resource provisioning\n│ ├── jobs/ # Managed job lifecycle\n│ ├── serve/ # Model serving (SkyServe)\n│ ├── skylet/ # On-cluster execution agent\n│ ├── client/ # SDK and CLI\n│ ├── server/ # API server and dashboard backend\n│ ├── dashboard/ # Web UI (Next.js)\n│ ├── utils/ # 50+ utility modules\n│ ├── catalog/ # Cloud pricing and instance catalogs\n│ ├── data/ # Storage and data handling\n│ ├── adaptors/ # Cloud-specific metadata adapters\n│ └── schemas/ # Protobuf definitions and API schemas\n├── tests/ # Test suite\n│ ├── unit_tests/ # Unit tests with subdirectories per module\n│ ├── smoke_tests/ # Quick validation tests\n│ ├── integration_tests/ # End-to-end tests\n│ ├── kubernetes/ # K8s-specific tests\n│ └── conftest.py # Pytest fixtures\n├── examples/ # 50+ usage examples\n├── llm/ # 45+ LLM training/serving examples\n├── docs/ # Sphinx documentation\n├── charts/ # Helm charts for K8s deployment\n└── format.sh # Code formatting script\n```\n\n## Development Setup\n\n### Environment Setup\n\n```bash\n# Create virtual environment with uv (Python 3.8-3.11 supported)\n# --seed is required to ensure pip is installed (needed for building wheels)\nuv venv --seed --python 3.11\nsource .venv/bin/activate\n\n# Install in editable mode with all cloud support\nuv pip install -e \".[all]\"\n# Or specific clouds only:\n# uv pip install -e \".[aws,gcp,kubernetes]\"\n\n# Install development dependencies\nuv pip install -r requirements-dev.txt\n\n# Optional: Install pre-commit hooks\nuv pip install pre-commit\npre-commit install\n```\n\n### Environment Variables\n\n```bash\nexport SKYPILOT_DEV=1 # Enable development mode\nexport SKYPILOT_DEBUG=1 # Enable debug logging\n```\n\n## Code Formatting and Linting\n\n**Always run `format.sh` before committing:**\n\n```bash\nbash format.sh # Format changed files (vs origin/master)\nbash format.sh --all # Format entire codebase\nbash format.sh --files path/to/file.py # Format specific files\n```\n\nThe script runs:\n1. **YAPF** - Google style for all Python code\n2. **isort** - Import sorting (Google profile)\n3. **mypy** - Type checking\n4. **pylint** - Linting with custom rules\n\n### Tool Versions (must match exactly)\n\nFrom `requirements-dev.txt`:\n- yapf==0.32.0\n- pylint==2.14.5\n- mypy==1.19.1\n- isort==5.12.0\n- pylint-quotes==0.2.3\n\n### Excluded from Formatting\n\n- `sky/schemas/generated/` - Auto-generated protobuf files\n- `build/` - Build artifacts\n\n## Testing\n\n### Running Tests\n\n```bash\n# Unit tests (fast, no cloud resources)\npytest tests/unit_tests/\n\n# Specific test file\npytest tests/unit_tests/test_resources.py\n```\n\n### CI Tests via PR Comments\n\nTrigger CI tests on pull requests using comments:\n- `/quicktest-core` - Run quick core tests\n- `/smoke-test` - Run smoke tests (launches cloud clusters)\n- `/smoke-test --kubernetes --postgres` - Test with PostgreSQL backend on Kubernetes\n- `/smoke-test --kubernetes --remote-server --postgres` - Test remote API server with PostgreSQL\n\n### Test Configuration\n\nFrom `pyproject.toml`:\n- Uses pytest-xdist with 16 parallel workers\n- Environment: `SKYPILOT_DEBUG=1`, `SKYPILOT_DEV=1`\n- Buildkite integration for CI\n\n### Checking Buildkite CI Status\n\nTo check CI test results for a PR:\n\n1. **Get Buildkite URL from GitHub**: Check the PR's commit status checks to find the Buildkite build URL (may take a moment to appear after triggering)\n2. **Fetch build logs via Buildkite API**: Use the Buildkite API to retrieve detailed logs and test results\n\n```bash\n# Example: Get build details (requires BUILDKITE_TOKEN)\ncurl -H \"Authorization: Bearer $BUILDKITE_TOKEN\" \\\n \"https://api.buildkite.com/v2/organizations/skypilot/pipelines/skypilot/builds/\"\n```\n\n## Code Style Guidelines\n\n### General Principles\n\n- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)\n- Use f-strings over `.format()` for readability\n- Use `class MyClass:` not `class MyClass(object):`\n- Use `abc` module for abstract classes\n- Use Python typing with `if typing.TYPE_CHECKING:` for type-only imports\n- Always place imports at the top of the file, not inside function definitions. If this causes a circular dependency, try to resolve it (e.g., restructure modules or use `if typing.TYPE_CHECKING:`). Only as a last resort, place the import inside the function with a comment explaining why. Import performance is not a valid reason for in-function imports; use `LazyImport` instead (see below).\n\n### TODOs and FIXMEs\n\nAlways include author attribution:\n```python\n# TODO(username): Description of what needs to be done\n# FIXME(username): Description of what needs fixing\n```\n\n### Exceptions vs Assertions\n\n- Use exceptions for errors (better error messages)\n- Use `assert` only for debugging/proof-checking\n\n### Lazy Imports\n\nFor modules with significant import time (>100ms) that are imported during `import sky`, use `LazyImport` from `sky/adaptors/common.py`:\n\n```python\nfrom sky.adaptors.common import LazyImport\nheavy_module = LazyImport('heavy_module')\n```\n\nMeasure import time with:\n```bash\npython -X importtime -c \"import sky\" 2> import.log\ntuna import.log # pip install tuna\n```\n\n## Architecture Patterns\n\n### Cloud Provider Abstraction\n\nAll cloud providers inherit from `sky.clouds.Cloud` (in `sky/clouds/cloud.py`). Cloud objects are **lightweight and stateless**. Key design principles:\n\n- Methods should be inexpensive to call\n- Don't store heavy state in cloud objects\n- Cache cloud-specific queries in `sky/clouds/utils/` modules\n- Each cloud implements feature flags via `CloudImplementationFeatures`\n\nAdding a new cloud provider:\n1. Create `sky/clouds/.py` inheriting from `Cloud`\n2. Implement required abstract methods\n3. Add provisioning logic in `sky/provision//`\n4. Register in `sky/clouds/__init__.py`\n\n### Client-Server Architecture\n\nSkyPilot uses a client-server model with API versioning:\n\n- Client (SDK/CLI) in `sky/client/`\n- Server in `sky/server/`\n- API version in `sky/server/constants.py`\n\n**Backward Compatibility Rules (from v0.10.0+):**\n- Changes must be backward compatible\n- Bump `API_VERSION` when introducing API changes\n- Use `@versions.minimal_api_version(N)` decorator for new SDK methods\n- Handle version differences with `versions.get_remote_api_version()`\n\n### Protobuf Regeneration\n\nWhen modifying `.proto` files in `sky/schemas/proto/`:\n\n```bash\npython -m grpc_tools.protoc \\\n --proto_path=sky/schemas/generated=sky/schemas/proto \\\n --python_out=. \\\n --grpc_python_out=. \\\n --pyi_out=. \\\n sky/schemas/proto/*.proto\n```\n\n### Dependency Management\n\nDependencies are defined in `sky/setup_files/dependencies.py`:\n\n- **Core dependencies**: Listed in `install_requires`\n- **Cloud-specific**: Defined in `extras_require` (e.g., `aws`, `gcp`, `kubernetes`)\n- **Development**: Listed in `requirements-dev.txt`\n\nWhen updating dependencies:\n\n1. Check version constraints carefully - some packages have breaking changes\n2. Consider Python version compatibility (3.8-3.11)\n3. Test with both minimum and latest allowed versions\n4. Document version constraints with comments explaining why\n\n## Key Modules Reference\n\n| Module | Purpose |\n|--------|---------|\n| `sky/core.py` | Core orchestration (launch, exec, stop, down) |\n| `sky/task.py` | Task YAML parsing and specification |\n| `sky/resources.py` | Resource requirements (GPUs, memory, disk) |\n| `sky/optimizer.py` | Cloud selection and cost optimization |\n| `sky/client/sdk.py` | Python SDK implementation |\n| `sky/client/cli/` | CLI commands |\n| `sky/backends/cloud_vm_ray_backend.py` | Main execution backend for clusters |\n| `sky/provision/provisioner.py` | Resource provisioning |\n| `sky/jobs/` | Managed jobs with recovery and scheduling |\n\n## Pull Request Guidelines\n\n1. **Branch from master**, create descriptive branch name\n2. **Run `format.sh`** before committing\n3. **Add tests** for core system changes\n4. **Run smoke tests** for significant changes\n5. **Include `Tested:` section** in PR description with test plan\n6. **Delete branch** after merging\n\n### PR Description Format\n\nPRs should include:\n- **Summary**: Brief description of changes (1-3 bullet points)\n- **Test plan**: How the changes were tested (commands run, manual verification steps)\n\n**Important**: Always generate a manual test plan describing how to verify the changes work correctly. Include specific commands, expected outputs, or UI verification steps. Whenever possible, add unit tests and smoke tests for the changes.\n\n### Commit Message Format\n\nUse the `[Area] Description` format:\n- `[Core]` for core system changes\n- `[CLI]` for CLI changes\n- `[API]` for API server changes\n- `[Docs]` for documentation\n- `[AWS]`, `[GCP]`, `[Azure]`, `[Kubernetes]` for cloud-specific changes\n- `[Jobs]` for managed jobs\n- `[Dashboard]` for web UI changes\n- `[Serve]` for model serving (SkyServe)\n- `[Test]` for testing changes\n- `[CI]` for CI/CD changes\n\nExamples:\n- `[Core] Fix cluster status refresh logic`\n- `[AWS] Add support for new instance types`\n- `[Docs] Update installation guide`\n\n## API Server Testing\n\n### Local API Server (Recommended for Development)\n\n```bash\n# Always restart API server after code changes to pick up changes\nsky api stop\nsky api start\n\n# Verify server is running\nsky api status\n```\n\n### Dashboard Development\n\n**For local API server development**, rebuild the dashboard before restarting:\n\n```bash\n# Install dependencies (first time or after package.json changes)\nnpm --prefix sky/dashboard install\n\n# Rebuild the dashboard\nnpm --prefix sky/dashboard run build\n\n# Then restart the API server\nsky api stop\nsky api start\n```\n\n**For remote API server (Docker/Kubernetes)**, the Dockerfile automatically builds the dashboard - no manual build needed before `docker build`.\n\nThe dashboard is a Next.js application. For development with hot reloading:\n\n```bash\n# Run dashboard in development mode (separate from API server)\ncd sky/dashboard\nnpm run dev\n```\n\n### Mocking Remote API Server Locally\n\nTo test remote API server behavior locally:\n\n```bash\n# Start local API server (runs on port 46580 by default)\nsky api stop\nsky api start\n\n# Forward to a different port to simulate remote server\nsocat TCP-LISTEN:46590,fork TCP:127.0.0.1:46580 &\n\n# Connect to the forwarded port as if it were a remote server\nsky api login -e http://127.0.0.1:46590\n```\n\n### Remote API Server (Kubernetes Deployment)\n\nFor testing with a remote API server on Kubernetes:\n\n```bash\n# Build local changes\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\nhelm dependency build ./charts/skypilot\n\n# Build Docker image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy (NEW installation)\nNAMESPACE=skypilot\nRELEASE_NAME=skypilot\nhelm upgrade --install $RELEASE_NAME ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n#### Upgrading Existing Deployments\n\n**CRITICAL:** Always use `--reuse-values` to preserve database/credential config:\n\n```bash\n# Upgrade existing deployment (keeps PostgreSQL, auth, etc.)\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n\n# Check current values / rollback if needed\nhelm get values skypilot -n skypilot\nhelm rollback skypilot -n skypilot\n```\n\n#### PostgreSQL Backend\n\n```bash\n# Create connection secret\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/db\"\n\n# Deploy with PostgreSQL\nhelm upgrade --install skypilot ./charts/skypilot -n skypilot \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\n## Critical Code Paths (Handle with Care)\n\nThe following modules contain complex, stateful logic that requires careful review when modifying:\n\n**Managed Jobs Recovery:**\n- `sky/jobs/controller.py` - Job lifecycle management, state transitions, async coordination\n- `sky/jobs/recovery_strategy.py` - Preemption recovery, retry logic for managed jobs\n- `sky/backends/cloud_vm_ray_backend.py` - Execution backend with complex state handling\n\nThese modules handle edge cases like preemption during job submission, controller failures mid-recovery, and race conditions between concurrent operations. Changes can have subtle effects on job reliability.\n\n**API Server Performance & Robustness:**\n- `sky/server/` - API server with memory efficiency, low latency requirements\n- `sky/backends/backend_utils.py` - Cluster status caching, network resilience, SSH handling\n\nThese modules are optimized for performance. Be cautious about adding blocking calls, memory-heavy operations, or changing caching behavior.\n\n**CLI/SDK Interface Design:**\n- `sky/client/cli/` - Command-line interface\n- `sky/client/sdk.py` - Python SDK\n\nBe cautious about adding new interfaces or changing existing UX significantly. Keep the interface clean and minimal for both CLI and SDK.\n\n**Backward Compatibility:**\n- Test with old server + new client, and old client + new server\n- Ensure existing clusters and jobs continue working after server upgrades\n- `/quicktest-core` tests some backward compatibility scenarios\n- See `docs/source/developers/CONTRIBUTING.md` for API versioning guidelines\n\n## Common Pitfalls\n\n1. **Always restart API server after code changes** - Run `sky api stop; sky api start` to pick up changes\n2. **Don't modify `sky/schemas/generated/`** - These are auto-generated\n3. **Match formatter versions exactly** - Version mismatches cause CI failures\n4. **Consider import time** - Heavy imports slow down CLI responsiveness\n5. **API versioning** - Always maintain backward compatibility\n\n## Useful Commands\n\n```bash\n# Profile CLI performance\nuv pip install py-spy\npy-spy record -t -o sky.svg -- python -m sky.cli status\n\n# Check cloud credentials\nsky check\n\n# View cluster status\nsky status\n\n# Launch a cluster\nsky launch \n\n# Execute a command on the cluster\nsky exec -- bash\n\n# SSH into the cluster\nssh \n\n# Launch a managed job\nsky jobs launch \n```\n\n## Documentation\n\n- **User docs**: https://docs.skypilot.co/\n- **Source**: `docs/source/` (Sphinx)\n- **Build docs**: See `.readthedocs.yml`\n\n## Additional Resources\n\n- **Full contributing guide**: `docs/source/developers/CONTRIBUTING.md`\n- **User docs**: https://docs.skypilot.co/\n- **GitHub Issues**: https://github.com/skypilot-org/skypilot/issues\n- **Discussions**: https://github.com/skypilot-org/skypilot/discussions\n- **Slack**: http://slack.skypilot.co\n","isInternal":false,"tokens":3889,"sizeBytes":16618},{"name":"marketplace.json","path":".claude-plugin/marketplace.json","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/.claude-plugin/marketplace.json","title":"skypilot","category":"marketplace","format":"json","content":"{\n \"name\": \"skypilot\",\n \"owner\": {\n \"name\": \"SkyPilot Team\"\n },\n \"metadata\": {\n \"description\": \"Official SkyPilot skills for AI coding agents\",\n \"version\": \"1.0.0\"\n },\n \"plugins\": [\n {\n \"name\": \"skypilot\",\n \"source\": \"./agent\",\n \"description\": \"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds\",\n \"version\": \"1.0.0\",\n \"author\": {\n \"name\": \"SkyPilot Team\"\n },\n \"homepage\": \"https://docs.skypilot.co/\",\n \"repository\": \"https://github.com/skypilot-org/skypilot\",\n \"license\": \"Apache-2.0\",\n \"keywords\": [\"AI Infrastructure\", \"Multi-Cloud\", \"GPU\", \"Kubernetes\", \"Slurm\", \"Training\", \"Serving\", \"Cost Optimization\", \"SkyPilot\"]\n }\n ]\n}\n","frontmatter":{"name":"skypilot","owner":{"name":"SkyPilot Team"},"metadata":{"description":"Official SkyPilot skills for AI coding agents","version":"1.0.0"},"plugins":[{"name":"skypilot","source":"./agent","description":"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds","version":"1.0.0","author":{"name":"SkyPilot Team"},"homepage":"https://docs.skypilot.co/","repository":"https://github.com/skypilot-org/skypilot","license":"Apache-2.0","keywords":["AI Infrastructure","Multi-Cloud","GPU","Kubernetes","Slurm","Training","Serving","Cost Optimization","SkyPilot"]}]},"isInternal":false,"tokens":228,"sizeBytes":756},{"name":"plugin.json","path":"agent/.claude-plugin/plugin.json","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/.claude-plugin/plugin.json","title":"skypilot","category":"plugin-manifest","format":"json","content":"{\n \"name\": \"skypilot\",\n \"description\": \"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds\",\n \"version\": \"1.0.0\",\n \"author\": {\n \"name\": \"SkyPilot Team\"\n },\n \"homepage\": \"https://docs.skypilot.co/\",\n \"repository\": \"https://github.com/skypilot-org/skypilot\",\n \"license\": \"Apache-2.0\",\n \"keywords\": [\"AI Infrastructure\", \"Multi-Cloud\", \"GPU\", \"Kubernetes\", \"Slurm\", \"Training\", \"Serving\", \"Cost Optimization\", \"SkyPilot\"]\n}\n","frontmatter":{"name":"skypilot","description":"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds","version":"1.0.0","author":{"name":"SkyPilot Team"},"homepage":"https://docs.skypilot.co/","repository":"https://github.com/skypilot-org/skypilot","license":"Apache-2.0","keywords":["AI Infrastructure","Multi-Cloud","GPU","Kubernetes","Slurm","Training","Serving","Cost Optimization","SkyPilot"]},"isInternal":false,"tokens":151,"sizeBytes":478},{"name":"autonomous-code-optimization.md","path":"docs/source/examples/agents/autonomous-code-optimization.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autonomous-code-optimization.md","title":"Subagent: autonomous-code-optimization","category":"subagent-persona","format":"markdown","content":"# Subagent: autonomous-code-optimization\n\nPath: `docs/source/examples/agents/autonomous-code-optimization.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autonomous-code-optimization.md)","tokens":300,"sizeBytes":261},{"name":"autoresearch.md","path":"docs/source/examples/agents/autoresearch.md","rawUrl":"https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autoresearch.md","title":"Subagent: autoresearch","category":"subagent-persona","format":"markdown","content":"# Subagent: autoresearch\n\nPath: `docs/source/examples/agents/autoresearch.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autoresearch.md)","tokens":300,"sizeBytes":213}],"systemPromptSnippet":"\n\n# CLAUDE.md - SkyPilot Development Guide\n\nThis document provides guidance for AI assistants working with the SkyPilot codebase.\n\n## Project Overview\n\nSkyPilot is a system to run, manage, and scale AI workloads on any AI infrastructure. It provides a unified interface across 25+ cloud providers (AWS, GCP, Azure, Kubernetes, Slurm, and many others), enabling users to launch compute resources, run jobs, and serve models without vendor lock-in.\n\n**Key capabilities:**\n- **Clusters**: Launch and manage compute clusters across clouds and Kubernetes\n- **Managed jobs**: Run jobs with automatic recovery from preemptions and failures\n- **Multi-cloud and multi-Kubernetes**: Unified interface across 25+ clouds and multiple K8s clusters\n- Cost optimization and GPU availability maximization\n\n## Repository Structure\n\n```\nskypilot/\n├── sky/ # Main source code\n│ ├── __init__.py # Package exports and version\n│ ├── core.py # Core orchestration logic\n│ ├── task.py # Task definition and YAML parsing\n│ ├── resources.py # Cloud resource specifications\n│ ├── optimizer.py # Resource allocation optimizer\n│ ├── execution.py # Task execution pipeline\n│ ├── exceptions.py # Custom exceptions\n│ ├── skypilot_config.py # Configuration management\n│ ├── clouds/ # Cloud provider abstractions (25+ providers)\n│ ├── backends/ # Execution backends (Ray-based VM, Docker)\n│ ├── provision/ # Cloud resource provisioning\n│ ├── jobs/ # Managed job lifecycle\n│ ├── serve/ # Model serving (SkyServe)\n│ ├── skylet/ # On-cluster execution agent\n│ ├── client/ # SDK and CLI\n│ ├── server/ # API server and dashboard backend\n│ ├── dashboard/ # Web UI (Next.js)\n│ ├── utils/ # 50+ utility modules\n│ ├── catalog/ # Cloud pricing and instance catalogs\n│ ├── data/ # Storage and data handling\n│ ├── adaptors/ # Cloud-specific metadata adapters\n│ └── schemas/ # Protobuf definitions and API schemas\n├── tests/ # Test suite\n│ ├── unit_tests/ # Unit tests with subdirectories per module\n│ ├── smoke_tests/ # Quick validation tests\n│ ├── integration_tests/ # End-to-end tests\n│ ├── kubernetes/ # K8s-specific tests\n│ └── conftest.py # Pytest fixtures\n├── examples/ # 50+ usage examples\n├── llm/ # 45+ LLM training/serving examples\n├── docs/ # Sphinx documentation\n├── charts/ # Helm charts for K8s deployment\n└── format.sh # Code formatting script\n```\n\n## Development Setup\n\n### Environment Setup\n\n```bash\n# Create virtual environment with uv (Python 3.8-3.11 supported)\n# --seed is required to ensure pip is installed (needed for building wheels)\nuv venv --seed --python 3.11\nsource .venv/bin/activate\n\n# Install in editable mode with all cloud support\nuv pip install -e \".[all]\"\n# Or specific clouds only:\n# uv pip install -e \".[aws,gcp,kubernetes]\"\n\n# Install development dependencies\nuv pip install -r requirements-dev.txt\n\n# Optional: Install pre-commit hooks\nuv pip install pre-commit\npre-commit install\n```\n\n### Environment Variables\n\n```bash\nexport SKYPILOT_DEV=1 # Enable development mode\nexport SKYPILOT_DEBUG=1 # Enable debug logging\n```\n\n## Code Formatting and Linting\n\n**Always run `format.sh` before committing:**\n\n```bash\nbash format.sh # Format changed files (vs origin/master)\nbash format.sh --all # Format entire codebase\nbash format.sh --files path/to/file.py # Format specific files\n```\n\nThe script runs:\n1. **YAPF** - Google style for all Python code\n2. **isort** - Import sorting (Google profile)\n3. **mypy** - Type checking\n4. **pylint** - Linting with custom rules\n\n### Tool Versions (must match exactly)\n\nFrom `requirements-dev.txt`:\n- yapf==0.32.0\n- pylint==2.14.5\n- mypy==1.19.1\n- isort==5.12.0\n- pylint-quotes==0.2.3\n\n### Excluded from Formatting\n\n- `sky/schemas/generated/` - Auto-generated protobuf files\n- `build/` - Build artifacts\n\n## Testing\n\n### Running Tests\n\n```bash\n# Unit tests (fast, no cloud resources)\npytest tests/unit_tests/\n\n# Specific test file\npytest tests/unit_tests/test_resources.py\n```\n\n### CI Tests via PR Comments\n\nTrigger CI tests on pull requests using comments:\n- `/quicktest-core` - Run quick core tests\n- `/smoke-test` - Run smoke tests (launches cloud clusters)\n- `/smoke-test --kubernetes --postgres` - Test with PostgreSQL backend on Kubernetes\n- `/smoke-test --kubernetes --remote-server --postgres` - Test remote API server with PostgreSQL\n\n### Test Configuration\n\nFrom `pyproject.toml`:\n- Uses pytest-xdist with 16 parallel workers\n- Environment: `SKYPILOT_DEBUG=1`, `SKYPILOT_DEV=1`\n- Buildkite integration for CI\n\n### Checking Buildkite CI Status\n\nTo check CI test results for a PR:\n\n1. **Get Buildkite URL from GitHub**: Check the PR's commit status checks to find the Buildkite build URL (may take a moment to appear after triggering)\n2. **Fetch build logs via Buildkite API**: Use the Buildkite API to retrieve detailed logs and test results\n\n```bash\n# Example: Get build details (requires BUILDKITE_TOKEN)\ncurl -H \"Authorization: Bearer $BUILDKITE_TOKEN\" \\\n \"https://api.buildkite.com/v2/organizations/skypilot/pipelines/skypilot/builds/\"\n```\n\n## Code Style Guidelines\n\n### General Principles\n\n- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)\n- Use f-strings over `.format()` for readability\n- Use `class MyClass:` not `class MyClass(object):`\n- Use `abc` module for abstract classes\n- Use Python typing with `if typing.TYPE_CHECKING:` for type-only imports\n- Always place imports at the top of the file, not inside function definitions. If this causes a circular dependency, try to resolve it (e.g., restructure modules or use `if typing.TYPE_CHECKING:`). Only as a last resort, place the import inside the function with a comment explaining why. Import performance is not a valid reason for in-function imports; use `LazyImport` instead (see below).\n\n### TODOs and FIXMEs\n\nAlways include author attribution:\n```python\n# TODO(username): Description of what needs to be done\n# FIXME(username): Description of what needs fixing\n```\n\n### Exceptions vs Assertions\n\n- Use exceptions for errors (better error messages)\n- Use `assert` only for debugging/proof-checking\n\n### Lazy Imports\n\nFor modules with significant import time (>100ms) that are imported during `import sky`, use `LazyImport` from `sky/adaptors/common.py`:\n\n```python\nfrom sky.adaptors.common import LazyImport\nheavy_module = LazyImport('heavy_module')\n```\n\nMeasure import time with:\n```bash\npython -X importtime -c \"import sky\" 2> import.log\ntuna import.log # pip install tuna\n```\n\n## Architecture Patterns\n\n### Cloud Provider Abstraction\n\nAll cloud providers inherit from `sky.clouds.Cloud` (in `sky/clouds/cloud.py`). Cloud objects are **lightweight and stateless**. Key design principles:\n\n- Methods should be inexpensive to call\n- Don't store heavy state in cloud objects\n- Cache cloud-specific queries in `sky/clouds/utils/` modules\n- Each cloud implements feature flags via `CloudImplementationFeatures`\n\nAdding a new cloud provider:\n1. Create `sky/clouds/.py` inheriting from `Cloud`\n2. Implement required abstract methods\n3. Add provisioning logic in `sky/provision//`\n4. Register in `sky/clouds/__init__.py`\n\n### Client-Server Architecture\n\nSkyPilot uses a client-server model with API versioning:\n\n- Client (SDK/CLI) in `sky/client/`\n- Server in `sky/server/`\n- API version in `sky/server/constants.py`\n\n**Backward Compatibility Rules (from v0.10.0+):**\n- Changes must be backward compatible\n- Bump `API_VERSION` when introducing API changes\n- Use `@versions.minimal_api_version(N)` decorator for new SDK methods\n- Handle version differences with `versions.get_remote_api_version()`\n\n### Protobuf Regeneration\n\nWhen modifying `.proto` files in `sky/schemas/proto/`:\n\n```bash\npython -m grpc_tools.protoc \\\n --proto_path=sky/schemas/generated=sky/schemas/proto \\\n --python_out=. \\\n --grpc_python_out=. \\\n --pyi_out=. \\\n sky/schemas/proto/*.proto\n```\n\n### Dependency Management\n\nDependencies are defined in `sky/setup_files/dependencies.py`:\n\n- **Core dependencies**: Listed in `install_requires`\n- **Cloud-specific**: Defined in `extras_require` (e.g., `aws`, `gcp`, `kubernetes`)\n- **Development**: Listed in `requirements-dev.txt`\n\nWhen updating dependencies:\n\n1. Check version constraints carefully - some packages have breaking changes\n2. Consider Python version compatibility (3.8-3.11)\n3. Test with both minimum and latest allowed versions\n4. Document version constraints with comments explaining why\n\n## Key Modules Reference\n\n| Module | Purpose |\n|--------|---------|\n| `sky/core.py` | Core orchestration (launch, exec, stop, down) |\n| `sky/task.py` | Task YAML parsing and specification |\n| `sky/resources.py` | Resource requirements (GPUs, memory, disk) |\n| `sky/optimizer.py` | Cloud selection and cost optimization |\n| `sky/client/sdk.py` | Python SDK implementation |\n| `sky/client/cli/` | CLI commands |\n| `sky/backends/cloud_vm_ray_backend.py` | Main execution backend for clusters |\n| `sky/provision/provisioner.py` | Resource provisioning |\n| `sky/jobs/` | Managed jobs with recovery and scheduling |\n\n## Pull Request Guidelines\n\n1. **Branch from master**, create descriptive branch name\n2. **Run `format.sh`** before committing\n3. **Add tests** for core system changes\n4. **Run smoke tests** for significant changes\n5. **Include `Tested:` section** in PR description with test plan\n6. **Delete branch** after merging\n\n### PR Description Format\n\nPRs should include:\n- **Summary**: Brief description of changes (1-3 bullet points)\n- **Test plan**: How the changes were tested (commands run, manual verification steps)\n\n**Important**: Always generate a manual test plan describing how to verify the changes work correctly. Include specific commands, expected outputs, or UI verification steps. Whenever possible, add unit tests and smoke tests for the changes.\n\n### Commit Message Format\n\nUse the `[Area] Description` format:\n- `[Core]` for core system changes\n- `[CLI]` for CLI changes\n- `[API]` for API server changes\n- `[Docs]` for documentation\n- `[AWS]`, `[GCP]`, `[Azure]`, `[Kubernetes]` for cloud-specific changes\n- `[Jobs]` for managed jobs\n- `[Dashboard]` for web UI changes\n- `[Serve]` for model serving (SkyServe)\n- `[Test]` for testing changes\n- `[CI]` for CI/CD changes\n\nExamples:\n- `[Core] Fix cluster status refresh logic`\n- `[AWS] Add support for new instance types`\n- `[Docs] Update installation guide`\n\n## API Server Testing\n\n### Local API Server (Recommended for Development)\n\n```bash\n# Always restart API server after code changes to pick up changes\nsky api stop\nsky api start\n\n# Verify server is running\nsky api status\n```\n\n### Dashboard Development\n\n**For local API server development**, rebuild the dashboard before restarting:\n\n```bash\n# Install dependencies (first time or after package.json changes)\nnpm --prefix sky/dashboard install\n\n# Rebuild the dashboard\nnpm --prefix sky/dashboard run build\n\n# Then restart the API server\nsky api stop\nsky api start\n```\n\n**For remote API server (Docker/Kubernetes)**, the Dockerfile automatically builds the dashboard - no manual build needed before `docker build`.\n\nThe dashboard is a Next.js application. For development with hot reloading:\n\n```bash\n# Run dashboard in development mode (separate from API server)\ncd sky/dashboard\nnpm run dev\n```\n\n### Mocking Remote API Server Locally\n\nTo test remote API server behavior locally:\n\n```bash\n# Start local API server (runs on port 46580 by default)\nsky api stop\nsky api start\n\n# Forward to a different port to simulate remote server\nsocat TCP-LISTEN:46590,fork TCP:127.0.0.1:46580 &\n\n# Connect to the forwarded port as if it were a remote server\nsky api login -e http://127.0.0.1:46590\n```\n\n### Remote API Server (Kubernetes Deployment)\n\nFor testing with a remote API server on Kubernetes:\n\n```bash\n# Build local changes\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\nhelm dependency build ./charts/skypilot\n\n# Build Docker image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy (NEW installation)\nNAMESPACE=skypilot\nRELEASE_NAME=skypilot\nhelm upgrade --install $RELEASE_NAME ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n#### Upgrading Existing Deployments\n\n**CRITICAL:** Always use `--reuse-values` to preserve database/credential config:\n\n```bash\n# Upgrade existing deployment (keeps PostgreSQL, auth, etc.)\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n\n# Check current values / rollback if needed\nhelm get values skypilot -n skypilot\nhelm rollback skypilot -n skypilot\n```\n\n#### PostgreSQL Backend\n\n```bash\n# Create connection secret\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/db\"\n\n# Deploy with PostgreSQL\nhelm upgrade --install skypilot ./charts/skypilot -n skypilot \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\n## Critical Code Paths (Handle with Care)\n\nThe following modules contain complex, stateful logic that requires careful review when modifying:\n\n**Managed Jobs Recovery:**\n- `sky/jobs/controller.py` - Job lifecycle management, state transitions, async coordination\n- `sky/jobs/recovery_strategy.py` - Preemption recovery, retry logic for managed jobs\n- `sky/backends/cloud_vm_ray_backend.py` - Execution backend with complex state handling\n\nThese modules handle edge cases like preemption during job submission, controller failures mid-recovery, and race conditions between concurrent operations. Changes can have subtle effects on job reliability.\n\n**API Server Performance & Robustness:**\n- `sky/server/` - API server with memory efficiency, low latency requirements\n- `sky/backends/backend_utils.py` - Cluster status caching, network resilience, SSH handling\n\nThese modules are optimized for performance. Be cautious about adding blocking calls, memory-heavy operations, or changing caching behavior.\n\n**CLI/SDK Interface Design:**\n- `sky/client/cli/` - Command-line interface\n- `sky/client/sdk.py` - Python SDK\n\nBe cautious about adding new interfaces or changing existing UX significantly. Keep the interface clean and minimal for both CLI and SDK.\n\n**Backward Compatibility:**\n- Test with old server + new client, and old client + new server\n- Ensure existing clusters and jobs continue working after server upgrades\n- `/quicktest-core` tests some backward compatibility scenarios\n- See `docs/source/developers/CONTRIBUTING.md` for API versioning guidelines\n\n## Common Pitfalls\n\n1. **Always restart API server after code changes** - Run `sky api stop; sky api start` to pick up changes\n2. **Don't modify `sky/schemas/generated/`** - These are auto-generated\n3. **Match formatter versions exactly** - Version mismatches cause CI failures\n4. **Consider import time** - Heavy imports slow down CLI responsiveness\n5. **API versioning** - Always maintain backward compatibility\n\n## Useful Commands\n\n```bash\n# Profile CLI performance\nuv pip install py-spy\npy-spy record -t -o sky.svg -- python -m sky.cli status\n\n# Check cloud credentials\nsky check\n\n# View cluster status\nsky status\n\n# Launch a cluster\nsky launch \n\n# Execute a command on the cluster\nsky exec -- bash\n\n# SSH into the cluster\nssh \n\n# Launch a managed job\nsky jobs launch \n```\n\n## Documentation\n\n- **User docs**: https://docs.skypilot.co/\n- **Source**: `docs/source/` (Sphinx)\n- **Build docs**: See `.readthedocs.yml`\n\n## Additional Resources\n\n- **Full contributing guide**: `docs/source/developers/CONTRIBUTING.md`\n- **User docs**: https://docs.skypilot.co/\n- **GitHub Issues**: https://github.com/skypilot-org/skypilot/issues\n- **Discussions**: https://github.com/skypilot-org/skypilot/discussions\n- **Slack**: http://slack.skypilot.co\n\n\n{\n \"skill_name\": \"skypilot\",\n \"description\": \"Evaluates the 'Before You Start (Agent Bootstrap)', 'Converting a Slurm Workload' and 'Investigating a Managed Job' section behavior\",\n \"evals\": [\n {\n \"id\": 1,\n \"name\": \"happy-path-server-running\",\n \"prompt\": \"I want to launch an 8xH100 cluster for training a LLaMA model. My SkyPilot is already set up.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, sees the server is running (table output with request data), skips rest of bootstrap (no `sky check`, no install, no `sky api start`), and proceeds directly to help write the task YAML or run `sky launch`.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` (or `sky api status -l 1`) as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT run `sky check` when the server is already running\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent does NOT run `sky api start` or `sky api login` when the server is already running\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent proceeds to help with the user's actual task (launching H100 cluster) without unnecessary bootstrap steps\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 2,\n \"name\": \"server-not-running-local-setup\",\n \"prompt\": \"I just installed SkyPilot and want to try launching a GPU instance on AWS. I don't have a remote API server.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, detects 'SkyPilot API server is not running.', asks about local vs remote, user indicates local, agent runs `sky api start`, re-checks with `sky api status -l 1`, then runs `sky check` for credentials since this is a fresh setup.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent asks the user whether they have a remote server or want to start locally\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent runs `sky api start` after user confirms local setup\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent re-runs `sky api status` after starting the server to confirm it's reachable\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent runs `sky check` since this is a fresh setup\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 3,\n \"name\": \"server-not-running-remote-connect\",\n \"prompt\": \"I need to connect to our team's SkyPilot API server at https://skypilot.internal.company.com and launch a fine-tuning job.\",\n \"expected_output\": \"Agent runs `sky api status -l 1`, detects server not running, recognizes user already provided a remote URL, runs `sky api login -e https://skypilot.internal.company.com` without asking unnecessary questions, re-checks status, then proceeds to help with the fine-tuning job.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky api status` as the first SkyPilot command\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent uses `sky api login -e https://skypilot.internal.company.com` (not `sky api start`) since the user provided a remote endpoint\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT ask the user for the server URL again since they already provided it\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent re-runs `sky api status` after connecting to confirm it's reachable\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 4,\n \"name\": \"slurm-sbatch-time-not-autostop\",\n \"prompt\": \"Convert this to SkyPilot for our Slurm cluster:\\n#!/bin/bash\\n#SBATCH --nodes=2\\n#SBATCH --gpus-per-node=8\\n#SBATCH --time=24:00:00\\n#SBATCH --account=ai-research\\nsrun python train.py\",\n \"expected_output\": \"Agent produces a task YAML with num_nodes: 2, resources.accelerators: H100:8 (or asks for the GRES name), and puts both --time and --account under config.slurm.sbatch_options. The bare `srun` is dropped because `run` already executes on every node. Agent recommends `sky jobs launch` for this batch script.\",\n \"assertions\": [\n {\n \"text\": \"Agent maps --time to config.slurm.sbatch_options.time\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT map --time to autostop, which is unsupported on Slurm\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent maps --account to config.slurm.sbatch_options.account\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent puts num_nodes at task level, not under resources\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT wrap the run command in `srun` again\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent recommends `sky jobs launch` (not `sky launch`) for this batch script\",\n \"type\": \"required_action\"\n }\n ]\n },\n {\n \"id\": 5,\n \"name\": \"slurm-job-array-to-num-jobs\",\n \"prompt\": \"On Slurm I run `sbatch --array=0-99 shard.sh` and the script reads $SLURM_ARRAY_TASK_ID to pick its shard. What's the SkyPilot equivalent?\",\n \"expected_output\": \"Agent answers `sky jobs launch --num-jobs 100` and maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK (with $SKYPILOT_NUM_JOBS for the count). May also mention pools for reusing workers across submissions.\",\n \"assertions\": [\n {\n \"text\": \"Agent uses `sky jobs launch --num-jobs 100`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent maps $SLURM_ARRAY_TASK_ID to $SKYPILOT_JOB_RANK\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT propose a shell for-loop as the primary answer\",\n \"type\": \"negative\"\n }\n ]\n },\n {\n \"id\": 6,\n \"name\": \"why-is-my-managed-job-pending\",\n \"prompt\": \"My managed job 42 has been PENDING for 20 minutes on our Slurm cluster. Why? SkyPilot is already set up.\",\n \"expected_output\": \"Agent reads `details` from `sky jobs queue -v -o json` and finds the Slurm pending reason with its partition (e.g. `Launching (pending: QOSGrpGRES; partition: h200)`), explains that category (quota vs no free capacity vs dependency vs held) and what to do about it. Does not claim the job is stuck in SkyPilot's own queue when the reason is Slurm's.\",\n \"assertions\": [\n {\n \"text\": \"Agent runs `sky jobs queue -v -o json` (`-v` is required for the `details` field) and reads the job's details\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent reads the Slurm pending reason and its partition from `details`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent explains the pending reason in plain terms (quota, capacity, dependency, or held) with a next step\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent does NOT poll with `sleep` + `sky jobs queue`\",\n \"type\": \"negative\"\n },\n {\n \"text\": \"Agent does NOT tear down the jobs controller as a first step\",\n \"type\": \"negative\"\n }\n ]\n },\n {\n \"id\": 7,\n \"name\": \"what-happened-to-my-job\",\n \"prompt\": \"Managed job 17 keeps flipping between RUNNING and RECOVERING. Show me what happened.\",\n \"expected_output\": \"Agent reads `sky jobs queue -v -o json` for `details` / `failure_reason` / `recovery_count`, then pulls `sky jobs logs 17 --controller --no-follow` for the recovery history, and points out that a repeated identical reason means a persistent cause the controller cannot fix by retrying.\",\n \"assertions\": [\n {\n \"text\": \"Agent reads `details`, `failure_reason` and `recovery_count` from `sky jobs queue -v -o json`\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent identifies the recurring RECOVERING reason from the events\",\n \"type\": \"required_action\"\n },\n {\n \"text\": \"Agent uses `--tail` or `--no-follow` when fetching logs rather than streaming the full log\",\n \"type\": \"required_action\"\n }\n ]\n }\n ]\n}\n\n\n# SkyPilot Advanced Patterns Reference\n\nThis reference covers advanced SkyPilot patterns for multi-cloud strategies,\ndistributed training, managed jobs, serving, Kubernetes integration, API server\nsetup, and cost optimization.\n\n---\n\n## 1. Multi-Cloud Strategies\n\nSkyPilot can automatically select the cheapest cloud/region/zone for a\nworkload, or follow explicit priority ordering. The key mechanisms are:\n- `any_of`: try all listed resource options, optimize for cost (unordered)\n- `ordered`: try resource options in strict priority order (first match wins)\n- `infra:` field with wildcard patterns for flexible cloud/region/zone targeting\n\n### 1.1 any_of -- Cheapest Available GPU\n\nUse `any_of` when you have multiple acceptable resource configurations and\nwant SkyPilot to pick the cheapest one available across all of them.\n\nNote: If all options use the same accelerator, you can simply specify\n`accelerators:` without `any_of` — SkyPilot will automatically search\nacross all enabled clouds. Use `any_of` when mixing different GPU types:\n\n```yaml\n# cheapest-gpu.yaml — mix different GPU types, SkyPilot picks cheapest\nname: cheapest-gpu-training\n\nresources:\n any_of:\n - accelerators: H100:8\n - accelerators: A100-80GB:8\n - accelerators: A100:8\n\nrun: |\n torchrun --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE train.py\n```\n\nYou can also mix different GPU types across clouds:\n\n```yaml\n# mixed-gpus.yaml\nname: mixed-gpu-training\n\nresources:\n any_of:\n - infra: aws\n accelerators: A10g:4\n - infra: gcp\n accelerators: L4:4\n - infra: azure\n accelerators: A100:1\n - infra: lambda\n accelerators: A100:1\n\nrun: |\n python train.py\n```\n\nThe optimizer evaluates all options together, considering current prices,\navailability, and any quotas, then launches on the cheapest option.\n\n### 1.2 ordered -- Strict Priority Fallback\n\nUse `ordered` when you have a strict preference. SkyPilot tries each option\nin order and uses the first one that succeeds.\n\n```yaml\n# priority-fallback.yaml\nname: priority-training\n\nresources:\n ordered:\n - infra: aws/us-east-1\n accelerators: A100:8\n - infra: gcp/us-central1\n accelerators: A100:8\n - infra: aws/us-west-2\n accelerators: A100:4\n - infra: gcp/europe-west4\n accelerators: A100:4\n\nrun: |\n python train.py\n```\n\nThis is useful when you have reserved instances or negotiated pricing on a\nspecific cloud/region and want to use those first.\n\n### 1.3 Kubernetes + Cloud Fallback\n\nA common pattern is to try on-prem Kubernetes first, then fall back to cloud:\n\n```yaml\n# k8s-cloud-fallback.yaml\nname: k8s-with-cloud-fallback\n\nresources:\n ordered:\n # Try on-prem K8s cluster first (free/cheaper)\n - infra: k8s/my-onprem-context\n accelerators: A100:4\n # Fall back to cloud K8s\n - infra: k8s/gke-prod-cluster\n accelerators: A100:4\n # Fall back to cloud VMs\n - infra: aws\n accelerators: A100:4\n - infra: gcp\n accelerators: A100:4\n\nrun: |\n python train.py\n```\n\n### 1.4 SSH Node Pools\n\nSkyPilot can use your own bare-metal machines or VMs via SSH node pools.\nFirst, define node pools in `~/.sky/ssh_node_pools.yaml`:\n\n```yaml\n# ~/.sky/ssh_node_pools.yaml\nmy-gpu-cluster:\n hosts:\n - 10.0.1.10\n - 10.0.1.11\n - 10.0.1.12\n\nmy-inference-box:\n hosts:\n - gpu-server.internal.example.com\n```\n\nThen reference them in your task YAML:\n\n```yaml\n# ssh-nodes.yaml\nname: train-on-ssh-nodes\n\nresources:\n infra: ssh/my-gpu-cluster\n accelerators: A100:8\n\nrun: |\n python train.py\n```\n\nSSH node pools can also participate in `any_of` or `ordered` fallback:\n\n```yaml\nresources:\n ordered:\n - infra: ssh/my-gpu-cluster\n accelerators: A100:8\n - infra: aws\n accelerators: A100:8\n```\n\n### 1.5 Wildcard Patterns in infra\n\nThe `infra` field supports `*` as a wildcard for any component\n(cloud, region, zone):\n\n```yaml\n# Any cloud, specific zone\nresources:\n infra: \"*/us-east-1/us-east-1a\"\n accelerators: A100:4\n```\n\n```yaml\n# Specific cloud, any region, specific zone suffix\nresources:\n infra: \"aws/*/us-east-1a\"\n accelerators: A100:4\n```\n\n```yaml\n# Any cloud, any region (equivalent to not specifying infra)\nresources:\n infra: \"*\"\n accelerators: A100:4\n```\n\nThe infra format is `cloud[/region[/zone]]`. For Kubernetes, the format is\n`k8s/context-name` (context names may contain slashes).\n\n### 1.6 Multi-Region Strategies\n\nTarget a specific region across any cloud, or multiple specific regions:\n\n```yaml\n# Multi-region with any_of\nresources:\n any_of:\n - infra: aws/us-east-1\n accelerators: H100:8\n - infra: aws/us-west-2\n accelerators: H100:8\n - infra: gcp/us-central1\n accelerators: H100:8\n - infra: gcp/us-east4\n accelerators: H100:8\n```\n\nFor single-cloud multi-region with the same GPU:\n\n```yaml\n# Will automatically try all available regions for this cloud\nresources:\n infra: aws\n accelerators: H100:8\n```\n\nSkyPilot's optimizer automatically considers all regions within the specified\ncloud and picks the cheapest available one. You only need `any_of` or\n`ordered` when you want to cross cloud boundaries or mix GPU types.\n\n### 1.7 Accelerator Sets and Lists\n\nFor single-resource blocks, you can use accelerator sets (unordered, cost-\noptimized) or lists (ordered, priority-based):\n\n```yaml\n# Unordered set: SkyPilot picks the cheapest available\nresources:\n accelerators: {A100:1, V100:1, L4:1, T4:1}\n```\n\n```yaml\n# Ordered list: SkyPilot tries in order\nresources:\n accelerators: [A100:1, V100:1, L4:1, T4:1]\n```\n\n---\n\n## 2. Distributed Training\n\nSkyPilot supports multi-node distributed training with automatic cluster\nsetup. It provides environment variables on every node for coordination.\n\n### 2.1 SkyPilot Environment Variables for Distributed Training\n\nOn each node, SkyPilot sets:\n\n| Variable | Description |\n|---|---|\n| `SKYPILOT_NODE_IPS` | Newline-separated list of all node IPs (head node first) |\n| `SKYPILOT_NODE_RANK` | 0-indexed rank of the current node (head = 0) |\n| `SKYPILOT_NUM_NODES` | Total number of nodes in the cluster |\n| `SKYPILOT_NUM_GPUS_PER_NODE` | Number of GPUs on the current node |\n\n### 2.2 PyTorch DDP Multi-Node Setup\n\nThe standard pattern uses `torchrun` with SkyPilot's environment variables:\n\n```yaml\n# ddp-training.yaml\nname: pytorch-ddp\n\nresources:\n accelerators: A100:8\n\nnum_nodes: 4\n\nsetup: |\n pip install torch torchvision\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n\n torchrun \\\n --nnodes=$NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$HEAD_IP \\\n --master_port=12355 \\\n train_ddp.py \\\n --epochs 100 \\\n --batch-size 256\n```\n\nA minimal `train_ddp.py` would use:\n\n```python\nimport os\n\nimport torch\nimport torch.distributed as dist\n\ndist.init_process_group(backend='nccl')\nlocal_rank = int(os.environ['LOCAL_RANK'])\ntorch.cuda.set_device(local_rank)\n# ... training loop\n```\n\n### 2.3 DeepSpeed ZeRO-3 Multi-Node\n\n```yaml\n# deepspeed-zero3.yaml\nname: deepspeed-training\n\nresources:\n accelerators: A100-80GB:8\n\nnum_nodes: 2\n\nfile_mounts:\n /workspace: .\n\nsetup: |\n pip install deepspeed torch transformers accelerate\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n\n # Create hostfile for DeepSpeed\n echo \"$SKYPILOT_NODE_IPS\" | while read ip; do\n echo \"$ip slots=$SKYPILOT_NUM_GPUS_PER_NODE\"\n done > /tmp/hostfile\n\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n deepspeed \\\n --hostfile /tmp/hostfile \\\n --master_addr $HEAD_IP \\\n --master_port 29500 \\\n --num_nodes $NUM_NODES \\\n --num_gpus $SKYPILOT_NUM_GPUS_PER_NODE \\\n /workspace/train.py \\\n --deepspeed /workspace/ds_config_zero3.json \\\n --model_name_or_path meta-llama/Llama-2-7b-hf \\\n --per_device_train_batch_size 2 \\\n --gradient_accumulation_steps 4\n fi\n```\n\nExample `ds_config_zero3.json`:\n\n```json\n{\n \"bf16\": { \"enabled\": true },\n \"zero_optimization\": {\n \"stage\": 3,\n \"offload_optimizer\": { \"device\": \"none\" },\n \"offload_param\": { \"device\": \"none\" },\n \"overlap_comm\": true,\n \"contiguous_gradients\": true,\n \"reduce_bucket_size\": 5e7,\n \"stage3_prefetch_bucket_size\": 5e7,\n \"stage3_param_persistence_threshold\": 1e5\n },\n \"gradient_accumulation_steps\": 4,\n \"train_micro_batch_size_per_gpu\": 2,\n \"wall_clock_breakdown\": false\n}\n```\n\n### 2.4 Ray Train Integration\n\nSkyPilot includes a helper script to start a Ray cluster across nodes. Use it\nfor Ray Train, Ray Tune, and other Ray-based distributed workloads:\n\n```yaml\n# ray-train.yaml\nname: ray-distributed-training\n\nresources:\n accelerators: A100:8\n ports:\n - 8280 # Ray dashboard\n\nnum_nodes: 2\n\nenvs:\n HF_HUB_ENABLE_HF_TRANSFER: \"1\"\n\nsetup: |\n pip install \"ray[default,train]\" torch transformers\n\nrun: |\n head_ip=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n\n # Use the built-in SkyPilot Ray start script\n export RAY_HEAD_PORT=6385\n export RAY_DASHBOARD_PORT=8280\n export RAY_DASHBOARD_HOST=0.0.0.0\n export RAY_HEAD_IP_ADDRESS=\"$head_ip\"\n ~/sky_templates/ray/start_cluster\n\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n export RAY_ADDRESS=\"http://localhost:8280\"\n ray job submit --address=\"$RAY_ADDRESS\" \\\n -- python train_with_ray.py\n fi\n```\n\n### 2.5 FSDP Configuration\n\nPyTorch Fully Sharded Data Parallel (FSDP) with SkyPilot:\n\n```yaml\n# fsdp-training.yaml\nname: fsdp-llama\n\nresources:\n accelerators: A100-80GB:8\n\nnum_nodes: 2\n\nsetup: |\n pip install torch transformers accelerate\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_NODES=$(echo \"$SKYPILOT_NODE_IPS\" | wc -l)\n TOTAL_GPUS=$((NUM_NODES * SKYPILOT_NUM_GPUS_PER_NODE))\n\n accelerate launch \\\n --num_processes $TOTAL_GPUS \\\n --num_machines $NUM_NODES \\\n --machine_rank $SKYPILOT_NODE_RANK \\\n --main_process_ip $HEAD_IP \\\n --main_process_port 29500 \\\n --use_fsdp \\\n --fsdp_auto_wrap_policy TRANSFORMER_BASED_WRAP \\\n --fsdp_sharding_strategy FULL_SHARD \\\n --fsdp_state_dict_type SHARDED_STATE_DICT \\\n --mixed_precision bf16 \\\n train_fsdp.py \\\n --model_name meta-llama/Llama-2-7b-hf \\\n --per_device_train_batch_size 2\n```\n\n### 2.6 NCCL Environment Tuning\n\nFor multi-node GPU training, NCCL configuration is critical for performance.\nSet these via `envs:` in your YAML:\n\n```yaml\n# nccl-tuned.yaml\nname: nccl-optimized-training\n\nresources:\n accelerators: H100:8\n\nnum_nodes: 4\n\nenvs:\n # Debug: set to INFO or WARN for troubleshooting\n NCCL_DEBUG: WARN\n # Network interface selection (check with `ip addr` on your nodes)\n NCCL_SOCKET_IFNAME: eth0\n # Use tree algorithm for better all-reduce on large clusters\n NCCL_ALGO: Tree\n # Avoid CUDA stream recording overhead\n TORCH_NCCL_AVOID_RECORD_STREAMS: \"1\"\n # InfiniBand settings (for clouds with IB/RDMA)\n NCCL_IB_DISABLE: \"0\"\n NCCL_IB_GID_INDEX: 3\n # Cross-node buffer sizes\n NCCL_BUFFSIZE: \"8388608\"\n # P2P transport (NVLink within node, network across nodes)\n NCCL_P2P_LEVEL: NVL\n\nrun: |\n HEAD_IP=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$HEAD_IP \\\n --master_port=12355 \\\n train.py\n```\n\n### 2.7 Common Distributed Training Gotchas\n\n1. **Port conflicts**: Always use a non-default master port (not 29500)\n if running multiple training jobs on the same cluster. Or use SkyPilot's\n port allocation.\n\n2. **Head node vs worker pattern**: Only the head node (`SKYPILOT_NODE_RANK\n == 0`) should launch the training coordinator (e.g., DeepSpeed launcher,\n Ray job submit). Worker nodes run the setup + the Ray/torchrun rendezvous.\n\n3. **Hanging at initialization**: Usually caused by NCCL not finding the right\n network interface. Set `NCCL_SOCKET_IFNAME` to the correct interface.\n Use `NCCL_DEBUG=INFO` to diagnose.\n\n4. **OOM on multi-node**: Gradient synchronization buffers consume extra\n memory. Reduce per-device batch size or use gradient accumulation.\n\n5. **Run section runs on ALL nodes**: The `run:` section executes on every\n node in the cluster. Use `if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]` to gate\n head-only logic (like launching the coordinator or downloading data).\n\n6. **Setup runs on ALL nodes**: The `setup:` section also runs on all nodes.\n This is typically correct (install dependencies everywhere), but be aware\n of it when doing one-time data preparation.\n\n---\n\n## 3. Managed Jobs Production Patterns\n\nManaged jobs (`sky jobs launch`) provide automatic recovery from preemptions,\nspot instance failures, and transient errors. They run on a lightweight jobs\ncontroller that monitors and relaunches your workload.\n\n### 3.1 job_recovery Configuration\n\nThe `job_recovery` field controls how managed jobs handle failures.\n\n**Strategy options:**\n- `EAGER_NEXT_REGION` (default): On preemption, immediately try a different\n region/cloud before retrying the same region.\n- `FAILOVER`: On preemption, first retry in the same region, then fail over\n to other regions.\n\n**Full configuration as a dict:**\n\n```yaml\n# managed-job-recovery.yaml\nname: resilient-training\n\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n recover_on_exit_codes: [1, 137]\n\nrun: |\n python train.py --checkpoint-dir /ckpts\n```\n\n**Shorthand (strategy only):**\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery: FAILOVER\n```\n\n**Field reference:**\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `strategy` | string | `EAGER_NEXT_REGION` | Recovery strategy name |\n| `max_restarts_on_errors` | int | 0 | Max restarts on non-preemption errors |\n| `recover_on_exit_codes` | int or list[int] | none | Exit codes that always trigger recovery (ignores max_restarts_on_errors counter) |\n\nWhen `recover_on_exit_codes` is set, any job failure with a matching exit code\ntriggers recovery without counting against `max_restarts_on_errors`. This is\nuseful for known transient failures.\n\nTo give up on a job that cannot get resources, set the top-level\n`job.wait_for_scheduling_timeout` (not part of `job_recovery`). The job ends\n`CANCELLED` if it has not started within that long of being submitted, whether\nit is retrying cloud VM launches or waiting in a Kubernetes queue such as\nKueue. Later recoveries do not re-arm it.\n\n```yaml\njob:\n wait_for_scheduling_timeout: 2h # duration (e.g. 90s, 30m, 2h) or int seconds\nresources:\n accelerators: A100:8\n job_recovery: FAILOVER\n```\n\n### 3.2 Checkpointing Patterns\n\nFor spot instances, checkpointing to cloud storage is essential. The pattern\nis: save checkpoints periodically to a mounted bucket, and on restart, resume\nfrom the latest checkpoint.\n\n```yaml\n# checkpointed-training.yaml\nname: checkpointed-llm-training\n\nresources:\n accelerators: A100-80GB:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 5\n\nfile_mounts:\n /ckpts:\n name: my-training-checkpoints # S3/GCS/R2 bucket\n mode: MOUNT\n\nrun: |\n # Resume from latest checkpoint if it exists\n CKPT_ARG=\"\"\n LATEST=$(ls -td /ckpts/checkpoint-* 2>/dev/null | head -1)\n if [ -n \"$LATEST\" ]; then\n echo \"Resuming from checkpoint: $LATEST\"\n CKPT_ARG=\"--resume_from_checkpoint $LATEST\"\n fi\n\n python train.py \\\n --output_dir /ckpts \\\n --save_steps 500 \\\n $CKPT_ARG\n```\n\nLaunch as a managed job:\n\n```bash\nsky jobs launch checkpointed-training.yaml\n```\n\n### 3.3 Hyperparameter Sweeps as Managed Jobs\n\nLaunch multiple managed jobs for a hyperparameter sweep. Each job runs\nindependently with automatic recovery:\n\n```bash\n# Launch sweep via CLI\nfor lr in 1e-4 5e-5 1e-5; do\n for bs in 16 32 64; do\n sky jobs launch sweep.yaml \\\n --env LR=$lr \\\n --env BATCH_SIZE=$bs \\\n -n \"sweep-lr${lr}-bs${bs}\" \\\n -y\n done\ndone\n```\n\n```yaml\n# sweep.yaml\nname: hp-sweep\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 2\n\nenvs:\n LR: 1e-4\n BATCH_SIZE: 32\n\nfile_mounts:\n /results:\n name: sweep-results-bucket\n mode: MOUNT\n\nrun: |\n python train.py \\\n --learning-rate $LR \\\n --batch-size $BATCH_SIZE \\\n --output-dir \"/results/lr${LR}_bs${BATCH_SIZE}\"\n```\n\nMonitor all running jobs:\n\n```bash\nsky jobs queue -o json\n```\n\n### 3.4 Job Pools (Experimental)\n\nJob pools pre-provision clusters that multiple jobs can be submitted to,\navoiding cold-start overhead. Pools are defined with a `service:` section\ncontaining `pool:` configuration.\n\n**Create a pool:**\n\n```yaml\n# gpu-pool.yaml\nname: gpu-pool\n\nresources:\n accelerators: A100:8\n\nservice:\n pool:\n workers: 3\n```\n\n```bash\n# Apply the pool configuration\nsky jobs pool apply gpu-pool.yaml -p my-gpu-pool\n```\n\n**Submit jobs to the pool:**\n\n```bash\nsky jobs launch job.yaml --pool my-gpu-pool\n```\n\n**Pool with autoscaling:**\n\n```yaml\n# autoscaling-pool.yaml\nname: autoscaling-pool\n\nresources:\n accelerators: A100:4\n\nservice:\n pool:\n min_workers: 1\n max_workers: 10\n queue_length_threshold: 2\n upscale_delay_seconds: 60\n downscale_delay_seconds: 300\n```\n\n**Pool management commands:**\n\n```bash\n# Check pool status\nsky jobs pool status\n\n# Check a specific pool\nsky jobs pool status my-gpu-pool\n\n# Update number of workers\nsky jobs pool apply -p my-gpu-pool --workers 5\n\n# Delete a pool\nsky jobs pool down my-gpu-pool\n\n# Delete all pools\nsky jobs pool down -a\n```\n\n---\n\n## 4. SkyServe Advanced\n\nSkyServe deploys model serving endpoints with autoscaling, multi-cloud\nreplicas, rolling updates, and spot instance support.\n\n### 4.1 Autoscaling Configuration\n\n```yaml\n# autoscaling-service.yaml\nname: llm-service\n\nresources:\n accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe:\n path: /health\n initial_delay_seconds: 300\n timeout_seconds: 10\n replica_policy:\n min_replicas: 2\n max_replicas: 10\n target_qps_per_replica: 5.0\n upscale_delay_seconds: 120\n downscale_delay_seconds: 600\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-2-7b-hf \\\n --port 8080\n```\n\n**Autoscaling field reference:**\n\n| Field | Type | Default | Description |\n|---|---|---|---|\n| `min_replicas` | int | 1 | Minimum number of replicas |\n| `max_replicas` | int | same as min | Maximum number of replicas |\n| `target_qps_per_replica` | float | required if max != min | Target QPS per replica for scaling |\n| `upscale_delay_seconds` | int | 300 | Seconds target must exceed before scaling up |\n| `downscale_delay_seconds` | int | 1200 | Seconds target must be below before scaling down |\n| `num_overprovision` | int | none | Extra replicas beyond what autoscaling calculates |\n\nDeploy and manage:\n\n```bash\n# Deploy the service\nsky serve up autoscaling-service.yaml -n my-llm\n\n# Check service status\nsky serve status my-llm\n\n# Get the service endpoint\nsky serve status my-llm --endpoint\n\n# Tail controller logs\nsky serve logs --controller my-llm\n```\n\n### 4.2 Rolling Updates vs Blue-Green Updates\n\n**Rolling update** (default): Gradually replaces replicas one by one. Some\nold and new replicas serve traffic simultaneously during the transition.\n\n```bash\n# Update with rolling strategy (default)\nsky serve update my-llm updated-service.yaml\n\n# Explicitly specify rolling\nsky serve update --mode rolling my-llm updated-service.yaml\n```\n\n**Blue-green update**: Provisions all new replicas first, then switches\ntraffic atomically. No mixed versions serve traffic simultaneously.\n\n```bash\n# Blue-green update\nsky serve update --mode blue_green my-llm updated-service.yaml\n```\n\nBlue-green requires enough quota to run both old and new replicas\nsimultaneously during the transition.\n\n### 4.3 Spot Replicas for Cost Savings\n\nUse spot instances for serving replicas with automatic on-demand fallback:\n\n```yaml\n# spot-serving.yaml\nname: spot-llm-service\n\nresources:\n accelerators: A100:1\n use_spot: true\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 3\n max_replicas: 8\n target_qps_per_replica: 5.0\n # Maintain 1 on-demand replica as a baseline\n base_ondemand_fallback_replicas: 1\n # Dynamically add on-demand replicas if spot is unavailable\n dynamic_ondemand_fallback: true\n\nrun: |\n python serve.py --port 8080\n```\n\n### 4.4 TLS/HTTPS Configuration\n\nEnable TLS termination at the load balancer:\n\n```yaml\n# tls-service.yaml\nname: secure-service\n\nresources:\n accelerators: L4:1\n ports:\n - 8080\n\nfile_mounts:\n /certs/server.key: ./certs/server.key\n /certs/server.crt: ./certs/server.crt\n\nservice:\n readiness_probe: /health\n replicas: 2\n tls:\n keyfile: /certs/server.key\n certfile: /certs/server.crt\n\nrun: |\n python serve.py --port 8080\n```\n\n### 4.5 Readiness Probe Configuration\n\nThe readiness probe determines when a replica is ready to receive traffic.\n\n**Simple GET probe (most common):**\n\n```yaml\nservice:\n readiness_probe: /health\n replicas: 2\n```\n\n**Full probe configuration:**\n\n```yaml\nservice:\n readiness_probe:\n path: /v1/models\n initial_delay_seconds: 600\n timeout_seconds: 30\n headers:\n Authorization: \"Bearer test-token\"\n replicas: 1\n```\n\n**POST probe (useful for LLMs that need a generation test):**\n\n```yaml\nservice:\n readiness_probe:\n path: /v1/completions\n post_data:\n model: llama-2-7b\n prompt: \"Hello\"\n max_tokens: 1\n initial_delay_seconds: 900\n timeout_seconds: 60\n replicas: 1\n```\n\n**Default values:**\n- `initial_delay_seconds`: 1200 (20 minutes) -- readiness failures during\n this period are ignored\n- `timeout_seconds`: 15\n\n### 4.6 Multi-Cloud Replicas\n\nDeploy replicas across multiple clouds for availability and cost optimization:\n\n```yaml\n# multi-cloud-serving.yaml\nname: multi-cloud-llm\n\nresources:\n any_of:\n - infra: aws\n accelerators: A100:1\n - infra: gcp\n accelerators: A100:1\n - infra: azure\n accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 3\n max_replicas: 10\n target_qps_per_replica: 5.0\n\nrun: |\n python serve.py --port 8080\n```\n\nSkyPilot distributes replicas across available clouds for redundancy and\npicks the cheapest options.\n\n### 4.7 Load Balancing Policies\n\nSkyServe supports different load balancing policies:\n\n```yaml\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 2\n max_replicas: 8\n target_qps_per_replica: 5.0\n load_balancing_policy: round_robin\n```\n\nFor heterogeneous GPU deployments, use instance-aware load balancing:\n\n```yaml\n# Instance-aware balancing across different GPU types\nresources:\n any_of:\n - accelerators: H100:1\n - accelerators: A100:1\n ports:\n - 8080\n\nservice:\n readiness_probe: /health\n replica_policy:\n min_replicas: 2\n max_replicas: 10\n target_qps_per_replica:\n H100: 10.0\n A100: 5.0\n load_balancing_policy: instance_aware_least_load\n```\n\n---\n\n## 5. Kubernetes Integration\n\nSkyPilot treats Kubernetes clusters as first-class infrastructure alongside\ncloud providers.\n\n### 5.1 Using K8s Contexts\n\nSkyPilot uses your kubeconfig contexts. Reference them with `infra: k8s/`:\n\n```yaml\n# Use a specific K8s context\nresources:\n infra: k8s/my-cluster-context\n accelerators: A100:4\n```\n\n```yaml\n# Use the default K8s context (current-context in kubeconfig)\nresources:\n infra: kubernetes\n accelerators: A100:4\n```\n\n```yaml\n# EKS context (contexts can contain special characters)\nresources:\n infra: k8s/arn:aws:eks:us-east-1:123456789012:cluster/my-cluster\n accelerators: A100:4\n```\n\n### 5.2 Pod Config Overrides\n\nCustomize the pod spec for SkyPilot pods via `~/.sky/config.yaml`. This\nallows setting tolerations, nodeSelectors, labels, and other Kubernetes-\nspecific configurations.\n\n```yaml\n# ~/.sky/config.yaml\nkubernetes:\n allowed_contexts:\n - my-production-cluster\n - my-dev-cluster\n context_configs:\n my-production-cluster:\n pod_config:\n spec:\n tolerations:\n - key: \"nvidia.com/gpu\"\n operator: \"Exists\"\n effect: \"NoSchedule\"\n nodeSelector:\n node-type: gpu\n team: ml-training\n containers:\n - name: skypilot\n resources:\n limits:\n nvidia.com/gpu: \"1\"\n metadata:\n labels:\n team: ml-platform\n environment: production\n custom_metadata:\n labels:\n app: skypilot\n managed-by: skypilot\n my-dev-cluster:\n pod_config:\n spec:\n nodeSelector:\n node-type: dev-gpu\n```\n\n### 5.3 Multi-Cluster K8s with any_of\n\nSpread workloads across multiple Kubernetes clusters:\n\n```yaml\n# multi-k8s.yaml\nname: multi-cluster-training\n\nresources:\n any_of:\n - infra: k8s/us-east-cluster\n accelerators: A100:8\n - infra: k8s/eu-west-cluster\n accelerators: A100:8\n - infra: k8s/asia-cluster\n accelerators: H100:8\n\nrun: |\n python train.py\n```\n\nCombined with cloud fallback:\n\n```yaml\nresources:\n ordered:\n - infra: k8s/onprem-cluster\n accelerators: A100:8\n - infra: k8s/gke-cluster\n accelerators: A100:8\n - infra: aws\n accelerators: A100:8\n```\n\n### 5.4 GPU Detection and Labeling\n\nSkyPilot needs GPU labels on Kubernetes nodes to schedule GPU workloads.\nUse `sky gpus label` to automatically detect and label GPUs:\n\n```bash\n# Label GPUs in the current Kubernetes context\nsky gpus label\n\n# Label GPUs in a specific context\nsky gpus label --context my-k8s-cluster\n\n# Start labeling without waiting for completion\nsky gpus label --async\n\n# Cleanup labeling resources when done\nsky gpus label --cleanup\n```\n\nCheck available GPUs across all configured infrastructure:\n\n```bash\n# List all available GPUs\nsky gpus list\n\n# Show GPUs for a specific type\nsky gpus list A100\n\n# Show GPUs available on Kubernetes\nsky gpus list --infra kubernetes\n```\n\n### 5.5 Kubernetes Configuration in ~/.sky/config.yaml\n\n```yaml\n# ~/.sky/config.yaml\nkubernetes:\n # Restrict which contexts SkyPilot can use\n allowed_contexts:\n - gke-prod\n - eks-staging\n\n # Port exposure mode\n ports: loadbalancer # or nodeport, ingress, podip\n\n # Cluster autoscaler integration\n autoscaler: gke # or karpenter, generic\n\n # Per-context configuration\n context_configs:\n gke-prod:\n provision_timeout: 600\n pod_config:\n spec:\n nodeSelector:\n cloud.google.com/gke-accelerator: nvidia-tesla-a100\n```\n\n---\n\n## 6. API Server and Team Setup\n\nThe SkyPilot API server enables team collaboration by providing a shared\ncontrol plane for launching and managing clusters, jobs, and services.\n\n### 6.1 Local API Server\n\nFor individual use or development:\n\n```bash\n# Start the local API server\nsky api start\n\n# Check server status\nsky api status\n\n# Stop the server\nsky api stop\n```\n\nThe local API server runs on port 46580 by default. It starts automatically\nwhen you run any `sky` command.\n\n### 6.2 Remote API Server with Helm\n\nDeploy a shared API server on Kubernetes for team use:\n\n```bash\n# Add the SkyPilot Helm repo\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\n\n# Build chart dependencies\nhelm dependency build ./charts/skypilot\n\n# Deploy the API server\nNAMESPACE=skypilot\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace\n```\n\nFor custom images (e.g., testing local changes):\n\n```bash\n# Build and push custom image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 \\\n -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy with custom image\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n --namespace skypilot \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\nUpgrade an existing deployment (always use `--reuse-values`):\n\n```bash\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n### 6.3 PostgreSQL Backend\n\nFor production deployments, use PostgreSQL instead of SQLite:\n\n```bash\n# Create a secret with the database connection string\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/skypilot\"\n\n# Deploy with PostgreSQL backend\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n -n skypilot --create-namespace \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\nOr set the connection string directly (less secure, not recommended for\nproduction):\n\n```bash\nhelm upgrade --install skypilot ./charts/skypilot --devel \\\n -n skypilot --create-namespace \\\n --set apiService.dbConnectionString=\"postgresql://user:pass@host:5432/db\"\n```\n\n### 6.4 Authentication\n\nConnect to a remote API server:\n\n```bash\n# OAuth2 browser login\nsky api login -e https://skypilot.example.com\n\n# Service account token login\nsky api login -e https://skypilot.example.com --token sky_abc123...\n```\n\nCheck the current connection:\n\n```bash\nsky api status\n```\n\n---\n\n## 7. Cost Optimization\n\nSkyPilot provides multiple mechanisms to minimize cloud spending while\nmaximizing GPU availability.\n\n### 7.1 Automatic Cloud Selection (Cheapest)\n\nBy default, SkyPilot's optimizer picks the cheapest cloud/region for your\nrequested resources:\n\n```yaml\n# SkyPilot automatically finds the cheapest option\nresources:\n accelerators: A100:4\n```\n\n```bash\n# Preview the optimizer's choice without launching\nsky launch --dryrun task.yaml\n```\n\nThe `--dryrun` flag shows you which cloud, region, instance type, and price\nSkyPilot would choose, without actually provisioning anything.\n\n### 7.2 Spot Instances\n\nSpot/preemptible instances offer 60-90% savings over on-demand:\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n```\n\nFor production spot workloads, combine with managed jobs for automatic\nrecovery:\n\n```yaml\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n```\n\n```bash\nsky jobs launch spot-training.yaml\n```\n\n### 7.3 Dryrun to Preview Costs\n\nAlways preview costs before launching expensive workloads:\n\n```bash\n# Show what would be launched and estimated cost\nsky launch --dryrun gpu-training.yaml\n\n# Show cost for a specific cloud\nsky launch --dryrun --infra aws gpu-training.yaml\n```\n\n### 7.4 Cost Report\n\nTrack spending across all clusters:\n\n```bash\n# Show cost report for active clusters\nsky cost-report\n\n# Show all clusters including terminated (last 30 days)\nsky cost-report --all\n\n# Show costs for specific number of days\nsky cost-report --days 7\n```\n\n### 7.5 Autostop Configuration\n\nAutomatically stop or terminate idle clusters to avoid waste. Autostop is\nconfigured in the `resources:` section of your YAML.\n\n**Simple autostop (idle minutes):**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop: 30 # Stop after 30 minutes idle\n```\n\n**Full autostop configuration:**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop:\n idle_minutes: 30\n down: true # Terminate instead of just stopping\n wait_for: jobs_and_ssh # Reset timer while jobs or SSH sessions active\n```\n\n**wait_for options:**\n\n| Value | Description |\n|---|---|\n| `jobs_and_ssh` | (default) Wait for running jobs AND open SSH sessions |\n| `jobs` | Only wait for running jobs to finish |\n| `none` | Stop immediately after idle_minutes, regardless of activity |\n\n**Disable autostop:**\n\n```yaml\nresources:\n accelerators: A100:4\n autostop: false\n```\n\n**CLI overrides:**\n\n```bash\n# Set autostop on an existing cluster\nsky autostop my-cluster -i 60\n\n# Set autodown (terminate when idle)\nsky autostop my-cluster -i 30 --down\n\n# Cancel autostop\nsky autostop my-cluster --cancel\n```\n\n### 7.6 Autostop Hooks for Pre-Shutdown Actions\n\nRun a custom script before the cluster stops or terminates. This is useful\nfor saving state, uploading logs, or graceful shutdown procedures:\n\n```yaml\nresources:\n accelerators: A100:4\n autostop:\n idle_minutes: 30\n down: true\n hook: |\n echo \"Cluster shutting down, saving state...\"\n cp -r /tmp/logs /data/saved-logs/\n python cleanup.py\n hook_timeout: 120 # Seconds to wait for hook to complete\n```\n\nThe hook runs before the cluster is stopped or terminated. If the hook\nexceeds `hook_timeout` seconds, it is killed and the shutdown proceeds.\n\n### 7.7 Combining Cost Optimization Strategies\n\nFor maximum savings, combine multiple strategies:\n\n```yaml\n# maximum-savings.yaml\nname: cost-optimized-training\n\nresources:\n any_of:\n - infra: aws\n accelerators: A100:8\n - infra: gcp\n accelerators: A100:8\n - infra: lambda\n accelerators: A100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 5\n autostop:\n idle_minutes: 10\n down: true\n\nfile_mounts:\n /ckpts:\n name: training-checkpoints\n mode: MOUNT\n\nrun: |\n python train.py --checkpoint-dir /ckpts --resume-latest\n```\n\nLaunch as a managed job for automatic recovery:\n\n```bash\nsky jobs launch maximum-savings.yaml\n```\n\nThis configuration:\n1. Uses `any_of` to find the cheapest cloud\n2. Uses spot instances for 60-90% savings\n3. Recovers automatically from preemptions via managed jobs\n4. Auto-terminates idle clusters\n5. Checkpoints to cloud storage for seamless recovery\n\n\n\n\n\n# SkyPilot CLI Reference\n\nComplete reference for all `sky` commands and options.\n\n## Core Cluster Commands\n\n### `sky launch`\n\nLaunch a cluster or task.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--cluster`, `-c` — A cluster name. If provided, either reuse an existing cluster with that name or provision a new cluster with that name. Otherwise provision a new cluster with an autogenerated name.\n- `--dryrun` — If True, do not actually run the job.\n- `--detach-run`, `-d`, `-no-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--docker` — (Deprecated) Local docker support is deprecated. To run locally, create a local Kubernetes cluster with ``sky local up``.\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--priority` — Priority for this task. Accepts either an integer (from -1000 to 1000, or a string priority class name. Passing \"none\" clears both.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--idle-minutes-to-autostop`, `-i` — Automatically stop the cluster after this many minutes of idleness, i.e., no running or pending jobs in the cluster's job queue. Idleness gets reset depending on the ``--wait-for`` flag. Setting this ...\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes-to-autostop``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs an...\n- `--down` — Autodown the cluster: tear down the cluster after all jobs finish (successfully or abnormally). If --idle-minutes-to-autostop is also set, the cluster will be torn down after the specified idle time. ...\n- `--retry-until-up`, `-r` — Whether to retry provisioning infinitely until the cluster is up, if we fail to launch the cluster on any possible region/cloud due to unavailability errors.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--no-setup` — Skip setup phase when (re-)launching cluster.\n- `--clone-disk-from`, `--clone` — [Experimental] Clone disk from an existing cluster to launch a new one. This is useful when the new cluster needs to have the same data on the boot disk as an existing cluster.\n- `--fast` — [Experimental] If the cluster is already up and available, skip provisioning and setup steps.\n- `--resize` — Resize an existing cluster to --num-nodes. Supports both scale-up (adding workers) and scale-down (removing workers). Scale-down requires no running jobs. Requires -c to specify an existing cluster.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workspace`, `-w` — Workspace to launch into. Shorthand for `--config active_workspace=`.\n\n### `sky exec`\n\nExecute a task or command on an existing cluster.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTER` — text\n- `--cluster`, `-c` — This is the same as the positional argument, just for consistency.\n- `ENTRYPOINT` — text\n- `--detach-run`, `-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n\n### `sky status`\n\nShow clusters.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--refresh`, `-r` — Query the latest cluster statuses from the cloud provider(s).\n- `--ip` — Get the IP address of the head node of a cluster. This option will override all other options. For Kubernetes clusters, the returned IP address is the internal IP of the head pod, and may not be acces...\n- `--endpoints` — Get all exposed endpoints and corresponding URLs for acluster. This option will override all other options.\n- `--endpoint` — Get the endpoint URL for the specified port number on the cluster. This option will override all other options.\n- `--show-managed-jobs`, `--no-show-managed-jobs` — Also show recent in-progress managed jobs, if any.\n- `--show-services`, `--no-show-services` — Also show sky serve services, if any.\n- `--show-pools`, `--no-show-pools` — Also show pools, if any.\n- `CLUSTERS` — text\n- `--all-users`, `-u` — Show all clusters, including those not owned by the current user.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n### `sky stop`\n\nStop cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Stop all existing clusters.\n- `--all-users`, `-u` — Stop all existing clusters for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky start`\n\nRestart cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Start all existing clusters.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--idle-minutes-to-autostop`, `-i` — Automatically stop the cluster after this many minutes of idleness, i.e., no running or pending jobs in the cluster's job queue. Idleness gets reset depending on the ``--wait-for`` flag. Setting this ...\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes-to-autostop``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs an...\n- `--down` — Autodown the cluster: tear down the cluster after specified minutes of idle time after all jobs finish (successfully or abnormally). Requires --idle-minutes-to-autostop to be set.\n- `--retry-until-up`, `-r` — Retry provisioning infinitely until the cluster is up, if we fail to start the cluster due to unavailability errors.\n- `--force`, `-f` — Force start the cluster even if it is already UP. Useful for upgrading the SkyPilot runtime on the cluster.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky down`\n\nTear down cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Tear down all existing clusters.\n- `--all-users`, `-u` — Tear down all existing clusters for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--purge`, `-p` — (Advanced) Forcefully remove the cluster(s) from SkyPilot's cluster table, even if the actual cluster termination failed on the cloud. WARNING: This flag should only be set sparingly in certain manual...\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky autostop`\n\nSchedule an autostop or autodown for cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTERS` — text\n- `--all`, `-a` — Autostop all existing clusters.\n- `--all-users`, `-u` — Autostop all existing clusters for all users.\n- `--idle-minutes`, `-i` — Set the idle minutes before autostopping the cluster. See the doc above for detailed semantics.\n- `--wait-for` — Determines the condition for resetting the idleness timer. This option works in conjunction with ``--idle-minutes``. Options: \b 1. ``jobs_and_ssh`` (default): Wait for in-progress jobs and SSH connec...\n- `--cancel` — Cancel any currently active auto{stop,down} setting for the cluster. No-op if there is no active setting.\n- `--down` — Use autodown (tear down the cluster; non-restartable), instead of autostop (restartable).\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky cost-report`\n\nShow estimated costs for launched clusters.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--all`, `-a` — Show all cluster information.\n- `--days` (default: `30`) — Show clusters from the last N days. Default is 30 days. If set to 0, show all clusters.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Job Queue Commands\n\n### `sky queue`\n\nShow the job queue for cluster(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--all-users`, `-u` — Show all users' information in full.\n- `--skip-finished`, `-s` — Show only pending/running jobs' information.\n- `CLUSTERS` — text\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n### `sky logs`\n\nTail the log of a job.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--provision` — Stream the cluster provisioning logs (provision.log).\n- `--hook` — Stream a per-event lifecycle-hook log from the cluster. Omit the event name to auto-select whichever log exists.\n- `--autostop` — [DEPRECATED] Alias for `--hook stop`. The autostop event was renamed to `stop` in the lifecycle-hooks framework.\n- `--worker`, `-w` — The worker ID to stream the logs from. If not set, stream the logs of the head node.\n- `--sync-down`, `-s` — Sync down the logs of a job to the local machine. For a distributed job, a separate log file from each worker will be downloaded.\n- `--status` — If specified, do not show logs but exit with a status code for the job's status: 0 for succeeded, or 1 for all other statuses.\n- `--follow`, `--no-follow` — Follow the logs of a job. If --no-follow is specified, print the log so far and exit. [default: --follow]\n- `--tail` (default: `-1`) — Number of lines to display from the end of the log file. Default is the last 1000 lines — sensible for multi-GB logs where downloading the full file is slow. Pass --tail 0 to print the entire log.\n- `CLUSTER` — text\n- `JOB_IDS` — text\n\n### `sky cancel`\n\nCancel job(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `CLUSTER` — text\n- `--all`, `-a` — Cancel all jobs from current user on the specified cluster.\n- `--all-users`, `-u` — Cancel all jobs on the specified cluster for all users.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `JOBS` — integer\n\n## Infrastructure Commands\n\n### `sky check`\n\nCheck which clouds are available to use.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `INFRA_LIST` — text\n- `--verbose`, `-v` — Show the activated account for each cloud.\n- `--workspace`, `-w` — The workspace to check. If None, all workspaces will be checked.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Dashboard\n\n### `sky dashboard`\n\nOpens the SkyPilot dashboard.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n\n## GPU/Accelerator Commands\n\nSkyPilot GPU/Accelerator CLI.\n\n### `sky gpus label`\n\nLabel GPU nodes in a Kubernetes cluster for use with SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--context`, `-c` — Kubernetes context to use. If not specified, uses the current context from the API server.\n- `--cleanup` — Only cleanup existing GPU labeler resources.\n- `--async` — Do not wait for GPU labeling to complete.\n\n### `sky gpus list`\n\nShow supported GPU/TPU/accelerators and their prices.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ACCELERATOR_STR` — text\n- `--all`, `-a` — Show details of all GPU/TPU/accelerator offerings.\n- `--infra` — Infrastructure to query. Examples: \"aws\", \"aws/us-east-1\"\n- `--cloud` — Cloud provider to query.\n- `--region` — The region to use. If not specified, shows accelerators from all regions.\n- `--all-regions` — Show pricing and instance details for a specified accelerator across all regions and clouds.\n- `--verbose`, `-v` — Show all information in full.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## Managed Jobs Commands\n\nManaged Jobs CLI (jobs with auto-recovery).\n\n### `sky jobs cancel`\n\nCancel managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--name`, `-n` — Managed job name to cancel.\n- `--pool`, `-p` — Pool name to cancel.\n- `JOB_IDS` — integer\n- `--task` — Cancel one dynamic task of the job (a job launched from inside it), by the index shown in `sky jobs queue` or by name. A declared task cannot be cancelled alone.\n- `--graceful` — Wait for MOUNT_CACHED uploads to complete before stopping/terminating. Will cancel current jobs first.\n- `--graceful-timeout` — Timeout in seconds for `--graceful` flag. When not set, will wait for MOUNT_CACHED uploads until they are finished.\n- `--all`, `-a` — Cancel all managed jobs for the current user.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--all-users`, `-u` — Cancel all managed jobs from all users.\n\n### `sky jobs dashboard`\n\nOpens a dashboard for managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n\n### `sky jobs launch`\n\nLaunch a managed job from a YAML or a command.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--name`, `-n` — Task name. Overrides the \"name\" config in the YAML if both are supplied.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--priority` — Priority for this task. Accepts either an integer (from -1000 to 1000, or a string priority class name. Passing \"none\" clears both.\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--cluster`, `-c` — Alias for --name, the name of the managed job.\n- `--job-recovery` — Recovery strategy to use for managed jobs.\n- `--detach-run`, `-d` — If True, as soon as a job is submitted, return from this call and do not stream execution logs.\n- `--pool`, `-p` — (Experimental; optional) Pool to use for jobs submission.\n- `--num-jobs` — Number of jobs to submit.\n- `--job-group` — Attach to an existing job group, by job id or unique running job name. The job is shown under it and cancelled with it. Defaults to the surrounding job group when launched from inside one.\n- `--no-job-group` — Launch a top-level job even when running inside a job group (do not attach to it).\n- `--depends-on` — Comma-separated managed job IDs to wait for. The job starts once all of them succeed, and is cancelled if any of them ends otherwise. The launch fails if one of them has already ended without succeedi...\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workspace`, `-w` — Workspace to submit the managed job into. Shorthand for `--config active_workspace=`.\n- `--output`, `-o` — Print only the submitted job IDs, for scripts: `id` prints one per line, `json` prints {\"job_ids\": [...]}. Everything else goes to stderr. Implies --detach-run.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky jobs logs`\n\nTail or sync down the log of a managed job.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--name`, `-n` — Managed job name.\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this job; useful for debugging launching/recoveries, etc.\n- `--refresh`, `-r` — Query the latest job logs, restarting the jobs controller if stopped.\n- `--sync-down`, `-s` — Download logs for all jobs shown in the queue.\n- `--tail` (default: `-1`) — Number of lines to display from the end of the log file. Default is the last 1000 lines — sensible for multi-GB logs where downloading the full file is slow. Pass --tail 0 to print the entire log.\n- `JOB_ID` — integer\n- `TASK` — text\n\n### `sky jobs pool` (subgroup)\n\n(Experimental) Pool management commands.\n\n### `sky jobs pool apply`\n\nEither apply a config to a pool for managed jobs submission or update the number of workers in the pool. One of POOL_YAML or --workers must be provided.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `POOL_YAML` — text\n- `--pool`, `-p` — A pool name. Unique for each pool. If not provided, a unique name is autogenerated.\n- `--mode` (default: `rolling`) — Update mode. If \"rolling\", pool will be updated with rolling update. If \"blue_green\", pool will be updated with blue-green update. This option is only valid when the pool is already running.\n- `--workers` — Can be used to update the number of workers in the pool.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky jobs pool down`\n\nDelete pool(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `POOL_NAMES` — text\n- `--all`, `-a` — Delete all pools.\n- `--purge`, `-p` — Tear down pools in failed status.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky jobs pool logs`\n\nTail or sync down logs of a pool.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this pool.\n- `--sync-down`, `-s` — Sync down logs to the local machine. Can be combined with --controller or worker ID to narrow scope.\n- `--tail` — The number of lines to display from the end of the log file. Default is None, which means print all lines.\n- `POOL_NAME` — text\n- `WORKER_IDS` — integer\n\n### `sky jobs pool status`\n\nShow statuses of pools.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `POOL_NAMES` — text\n- `--all`, `-a` — Show all workers.\n\n### `sky jobs queue`\n\nShow statuses of managed jobs.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--limit`, `-l` (default: `50`) — Number of jobs to show, default is 50, use \"-a/--all\" to show all jobs.\n- `--refresh`, `-r` — Query the latest statuses, restarting the jobs controller if stopped.\n- `--skip-finished` — Show only pending/running jobs' information.\n- `-s`, `--status` — Filter by status, comma-separated (e.g. -s FAILED,FAILED_SETUP). A bare -s (no value) is a deprecated alias for --skip-finished.\n- `--since` — Show only jobs submitted within this time window, relative to now (e.g. \"30m\", \"48h\", \"7d\", \"2w\"). A bare number is seconds. Mutually exclusive with --after.\n- `--after` — Show only jobs submitted at or after this absolute local time (e.g. \"2026-01-13\" or \"2026-01-13 15:30:00\"). Mutually exclusive with --since.\n- `--before` — Show only jobs submitted at or before this absolute local time (e.g. \"2026-01-13\" or \"2026-01-13 15:30:00\").\n- `--infra` — Show only jobs running on this infrastructure. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/u...\n- `--all-users`, `-u` — Show jobs from all users.\n- `--all`, `-a` — Show all jobs.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## SkyServe Commands\n\nSkyServe CLI (multi-region, multi-cloud serving).\n\n### `sky serve down`\n\nTeardown service(s).\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_NAMES` — text\n- `--all`, `-a` — Tear down all services.\n- `--purge`, `-p` — Tear down services in failed status.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--replica-id` — Tear down a given replica\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky serve logs`\n\nTail or sync down logs of a service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--follow`, `--no-follow` — Follow the logs of the job. [default: --follow] If --no-follow is specified, print the log so far and exit.\n- `--controller` — Show the controller logs of this service.\n- `--load-balancer` — Show the load balancer logs of this service.\n- `--sync-down`, `-s` — Sync down logs to the local machine. Can be combined with --controller, --load-balancer, or a replica ID to narrow scope.\n- `--tail` — The number of lines to display from the end of the log file. Default is None, which means print all lines.\n- `SERVICE_NAME` — text\n- `REPLICA_IDS` — integer\n\n### `sky serve status`\n\nShow statuses of SkyServe services.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n- `--endpoint` — Show service endpoint.\n- `SERVICE_NAMES` — text\n\n### `sky serve up`\n\nLaunch a SkyServe service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_YAML` — text\n- `--service-name`, `-n` — A service name. Unique for each service. If not provided, a unique name is autogenerated.\n- `--git-url` — Git repository URL.\n- `--git-ref` — Git reference (branch, tag, or commit hash) to use.\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky serve update`\n\nUpdate a SkyServe service.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `SERVICE_NAME` — text\n- `SERVICE_YAML` — text\n- `--workdir` — If specified, sync this dir to the remote working directory, where the task will be invoked. Overrides the \"workdir\" config in the YAML if both are supplied.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, k8s/context-name, or ssh/node-pool-name. Examples: aws, aws/us-east-1, aws/us-east-1/us-east-1a, aws/\\*/us-east-1a, k8s/my-contex...\n- `--cloud` — The cloud to use. If specified, overrides the \"resources.cloud\" config. Passing \"none\" resets the config.\n- `--region` — The region to use. If specified, overrides the \"resources.region\" config. Passing \"none\" resets the config.\n- `--zone` — The zone to use. If specified, overrides the \"resources.zone\" config. Passing \"none\" resets the config.\n- `--num-nodes` — Number of nodes to execute the task on. Overrides the \"num_nodes\" config in the YAML if both are supplied.\n- `--cpus` — Number of vCPUs each instance must have (e.g., ``--cpus=4`` (exactly 4) or ``--cpus=4+`` (at least 4)). This is used to automatically select the instance type.\n- `--memory` — Amount of memory each instance must have in GB (e.g., ``--memory=16`` (exactly 16GB), ``--memory=16+`` (at least 16GB))\n- `--disk-size` — OS disk size in GBs.\n- `--disk-tier` — OS disk tier. Could be one of low, medium, high, ultra, best, none. If best is specified, use the best possible disk tier. If none is specified, enforce to use default value and override the option in...\n- `--network-tier` — Network tier. Could be one of standard, best. If best is specified, use the best network tier available on the specified instance. Default: standard\n- `--use-spot`, `--no-use-spot` — Whether to request spot instances. If specified, overrides the \"resources.use_spot\" config.\n- `--image-id` — Custom image id for launching the instances. Passing \"none\" resets the config.\n- `--env-file` — Path to a dotenv file with environment variables to set on the remote node. If any values from ``--env-file`` conflict with values set by ``--env``, the ``--env`` value will be preferred. Values fro...\n- `--env` — Environment variable to set on the remote node. It can be specified multiple times. Examples: \b 1. ``--env MY_ENV=1``: set ``$MY_ENV`` on the cluster to be 1. 2. ``--env MY_ENV2=$HOME``: set ``$MY_E...\n- `--secret-file` — Path to a dotenv file with secret variables to set on the remote node. If any values from ``--secret-file`` conflict with values set by ``--secret``, the ``--secret`` value will be preferred.\n- `--secret` — Secret variable to set on the remote node. These variables will be redacted in logs and YAML outputs for security. It can be specified multiple times. Examples: \b 1. ``--secret API_KEY=secret123``: s...\n- `--local-disk` — Local (instance) storage requirement. Format: [mode:]size[+] or mode. Mode is \"nvme\" (default) or \"ssd\". Size is total GB across all disks. If only mode specified, size defaults to 100+. If only size ...\n- `--gpus` — Type and number of GPUs to use. Example values: \"V100:8\", \"V100\" (short for a count of 1), or \"V100:0.5\" (fractional counts are supported by the scheduling framework). If a new cluster is being launch...\n- `--instance-type`, `-t` — The instance type to use. If specified, overrides the \"resources.instance_type\" config. Passing \"none\" resets the config.\n- `--ports` — Ports to open on the cluster. Accepts a single port (``--ports 8080``), a range (``--ports 8000-8010``), or a comma-separated list (``--ports 8080,9090``). Repeat the flag to specify multiple values (...\n- `--async`, `--no-async` — Run the command asynchronously.\n- `--mode` (default: `rolling`) — Update mode. If \"rolling\", SkyServe will update the service with rolling update. If \"blue_green\", SkyServe will update the service with blue-green update.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n## Storage Commands\n\nSkyPilot Storage CLI.\n\n### `sky storage delete`\n\nDelete storage objects.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--all`, `-a` — Delete all storage objects.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky storage ls`\n\nList storage objects managed by SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `--verbose`, `-v` — Show all information in full.\n\n## Volume Commands\n\nSkyPilot Volumes CLI.\n\n### `sky volumes apply`\n\nApply a volume.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `ENTRYPOINT` — text\n- `--name`, `-n` — Volume name. Override the name defined in the YAML.\n- `--infra` — Infrastructure to use. Format: cloud, cloud/region, cloud/region/zone, or k8s/context-name.Examples: k8s, k8s/my-context, runpod/US/US-CA-2. Override the infra defined in the YAML.\n- `--type` — Volume type. Override the type defined in the YAML.\n- `--size` — Volume size. Override the size defined in the YAML.\n- `--use-existing`, `--no-use-existing` — Whether to use an existing volume. Override the use_existing defined in the YAML.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky volumes delete`\n\nDelete volumes.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--all`, `-a` — Delete all volumes.\n- `--purge`, `-p` — Forcibly delete the volume from the volumes table even if the deletion API fails.\n- `--yes`, `-y` — Skip confirmation prompt.\n- `--async`, `--no-async` — Run the command asynchronously.\n\n### `sky volumes ls`\n\nList volumes managed by SkyPilot.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `NAMES` — text\n- `--verbose`, `-v` — Show all information in full.\n- `--refresh`, `-r` — Refresh volume state from cloud APIs before listing. Without this flag, cached data is returned which is updated periodically by the background daemon.\n- `--output`, `-o` (default: `table`) — Output format. Choices: table, json. Default: table.\n\n## API Server Commands\n\nSkyPilot API server commands.\n\n### `sky api cancel`\n\nCancel a request running on SkyPilot API server.\n\n**Options:**\n\n- `--config` — Path to a config file or a single key-value pair. To add multiple key-value pairs add multiple flags (e.g. --config nested.key1=val1 --config nested.key2=val2).\n- `REQUEST_IDS` — text\n- `--all`, `-a` — Cancel all your requests.\n- `--all-users`, `-u` — Cancel all requests from all users. On a server with per-user auth, a non-admin cancels only their own.\n- `--yes`, `-y` — Skip confirmation prompt.\n\n### `sky api info`\n\nShows the SkyPilot API server URL.\n\n**Options:**\n\n- `--output`, `-o` (default: `table`) — Output forma\n\n\n# SkyPilot Examples Reference\n\nCopy-paste-ready YAML examples and CLI commands for common SkyPilot workflows.\n\n---\n\n## 1. Basic Examples\n\n### Minimal Task (Just a Run Command)\n\nThe simplest possible SkyPilot task -- just a command to run.\n\n```yaml\n# minimal.yaml\nname: hello-world\n\nrun: |\n echo \"Hello from SkyPilot!\"\n hostname\n uname -a\n```\n\n```bash\n# Launch on any available cloud (cheapest option selected automatically)\nsky launch -c hello minimal.yaml\n\n# Tear down when done\nsky down hello\n```\n\n### GPU Task with Specific Accelerator\n\nRequest a specific GPU type and count.\n\n```yaml\n# gpu-task.yaml\nname: gpu-check\n\nresources:\n # Request 1x A100 GPU\n accelerators: A100:1\n\nsetup: |\n pip install torch\n\nrun: |\n nvidia-smi\n python -c \"import torch; print(f'CUDA available: {torch.cuda.is_available()}'); print(f'GPU: {torch.cuda.get_device_name(0)}')\"\n```\n\n```bash\n# Launch on any cloud that has A100\nsky launch -c gpu-test gpu-task.yaml\n\n# Or pin to a specific cloud\nsky launch -c gpu-test gpu-task.yaml --infra aws\n\n# Tear down\nsky down gpu-test\n```\n\n### Task with Setup and Workdir\n\nSync local code and install dependencies before running.\n\n```yaml\n# train-task.yaml\nname: my-training\n\n# Sync current directory to ~/sky_workdir on the remote machine\nworkdir: .\n\nresources:\n accelerators: L4:1\n\n# Setup runs once on cluster creation, cached on subsequent runs\nsetup: |\n pip install torch torchvision\n pip install -r requirements.txt\n\n# Run executes your main command\nrun: |\n python train.py --epochs 10\n```\n\n```bash\n# Launch -- setup runs first, then the run command\nsky launch -c train train-task.yaml\n\n# Re-run with different args (setup is cached, only run executes)\nsky exec train train-task.yaml\n```\n\n### Task with Environment Variables and Secrets\n\nPass configuration via envs and sensitive values via secrets.\n\n```yaml\n# env-task.yaml\nname: training-with-config\n\nresources:\n accelerators: A100:1\n\n# Environment variables -- visible in logs and dashboard\nenvs:\n MODEL_NAME: bert-base-uncased\n BATCH_SIZE: \"32\"\n LEARNING_RATE: \"1e-4\"\n\n# Secrets -- redacted in logs and dashboard\nsecrets:\n HF_TOKEN: null # Will be passed via CLI\n WANDB_API_KEY: null # Will be passed via CLI\n\nsetup: |\n pip install transformers datasets wandb\n\nrun: |\n echo \"Training model: $MODEL_NAME with batch size $BATCH_SIZE\"\n wandb login $WANDB_API_KEY\n python train.py \\\n --model $MODEL_NAME \\\n --batch-size $BATCH_SIZE \\\n --lr $LEARNING_RATE\n```\n\n```bash\n# Pass secrets and override envs from the CLI\nHF_TOKEN=hf_abc123 WANDB_API_KEY=wk_xyz789 \\\n sky launch -c train env-task.yaml \\\n --secret HF_TOKEN --secret WANDB_API_KEY \\\n --env BATCH_SIZE=64\n\n# Secrets can also be specified inline\nsky launch -c train env-task.yaml \\\n --secret HF_TOKEN=hf_abc123 \\\n --secret WANDB_API_KEY=wk_xyz789\n```\n\n### Inline Command (No YAML File)\n\nRun a quick command without writing a YAML file.\n\n```bash\n# Launch a GPU instance and run nvidia-smi\nsky launch -c quick --gpus H100 -- nvidia-smi\n\n# Run a Python script on an existing cluster\nsky exec quick -- python -c \"import torch; print(torch.cuda.device_count())\"\n\n# Launch with specific cloud and region\nsky launch -c quick --gpus A100 --infra gcp/us-central1 -- nvidia-smi\n```\n\n### Multi-Node Task\n\nLaunch a task across multiple nodes.\n\n```yaml\n# multi-node.yaml\nname: multi-node-check\n\n# Launch 2 nodes (1 head + 1 worker)\nnum_nodes: 2\n\nresources:\n accelerators: L4:1\n\nrun: |\n echo \"Node rank: $SKYPILOT_NODE_RANK\"\n echo \"Total nodes: $SKYPILOT_NUM_NODES\"\n echo \"All node IPs:\"\n echo \"$SKYPILOT_NODE_IPS\"\n echo \"GPUs per node: $SKYPILOT_NUM_GPUS_PER_NODE\"\n nvidia-smi\n```\n\n```bash\nsky launch -c multi multi-node.yaml\n```\n\n---\n\n## 2. Training Examples\n\n### Single-GPU PyTorch Training\n\nA complete single-GPU training example with local code sync.\n\n```yaml\n# single-gpu-train.yaml\nname: pytorch-single-gpu\n\nworkdir: .\n\nresources:\n accelerators: A100:1\n disk_size: 256\n\nenvs:\n NUM_EPOCHS: \"10\"\n BATCH_SIZE: \"32\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n python train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c train single-gpu-train.yaml\nsky logs train # Stream logs\nsky down train # Tear down when done\n```\n\n### Multi-GPU Single-Node Training\n\nUse all GPUs on a single node with torchrun.\n\n```yaml\n# multi-gpu-train.yaml\nname: pytorch-multi-gpu\n\nworkdir: .\n\nresources:\n # 4x A100 GPUs on a single node\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n NUM_EPOCHS: \"10\"\n BATCH_SIZE: \"64\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_port=12345 \\\n train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c multi-gpu multi-gpu-train.yaml\n```\n\n### Multi-Node Distributed Training (PyTorch DDP)\n\nDistributed training across 2 nodes with 8 GPUs each using torchrun.\n\n```yaml\n# multi-node-ddp.yaml\nname: pytorch-ddp-training\n\nworkdir: .\n\n# 2 nodes, each with 8 GPUs = 16 GPUs total\nnum_nodes: 2\n\nresources:\n accelerators: A100-80GB:8\n disk_size: 512\n\nenvs:\n NUM_EPOCHS: \"20\"\n BATCH_SIZE: \"256\"\n\nsetup: |\n pip install torch torchvision\n pip install wandb tqdm\n\nrun: |\n # SKYPILOT_NODE_IPS contains all node IPs, one per line\n # The first IP is always the head node (rank 0)\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=12345 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train.py \\\n --epochs $NUM_EPOCHS \\\n --batch-size $BATCH_SIZE \\\n --output-dir /tmp/checkpoints\n```\n\n```bash\nsky launch -c ddp-train multi-node-ddp.yaml\n\n# SSH into the head node to check progress\nssh ddp-train\n```\n\n### Training with Checkpointing to Cloud Storage\n\nSave checkpoints to S3/GCS for persistence across runs.\n\n```yaml\n# train-with-checkpoints.yaml\nname: training-with-checkpoints\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n CHECKPOINT_BUCKET: my-training-checkpoints\n\nfile_mounts:\n # Mount an S3 bucket for writing checkpoints\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\n # Mount a dataset bucket (read-only streaming)\n /data:\n source: s3://my-training-data/imagenet\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision boto3\n\nrun: |\n # Training script should save checkpoints to /checkpoints\n # and load dataset from /data\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n train.py \\\n --data-dir /data \\\n --checkpoint-dir /checkpoints \\\n --resume-from-checkpoint\n```\n\n```bash\n# Launch training\nsky launch -c train train-with-checkpoints.yaml\n\n# Re-launch after preemption or failure -- resumes from last checkpoint\nsky launch -c train train-with-checkpoints.yaml\n```\n\n### Training on Spot Instances with Auto-Recovery (Managed Job)\n\nRun long training on cheap spot instances with automatic recovery from preemptions.\n\n```yaml\n# spot-train.yaml\nname: spot-training-resnet\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n # Use spot instances for ~3x cost savings\n use_spot: true\n disk_size: 512\n\n # Recovery strategy when spot instance is preempted\n job_recovery:\n strategy: EAGER_NEXT_REGION # Move to next region on preemption\n max_restarts_on_errors: 3 # Retry up to 3 times on user code errors\n\nenvs:\n CHECKPOINT_BUCKET: my-spot-training-ckpts\n\nfile_mounts:\n # Persistent checkpoint storage -- survives preemptions\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision\n\nrun: |\n # IMPORTANT: Your training script must handle checkpoint resume.\n # SkyPilot recovers the cluster; your script recovers the state.\n torchrun \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n train.py \\\n --checkpoint-dir /checkpoints \\\n --resume-from-checkpoint\n```\n\n```bash\n# Launch as a managed job (NOT sky launch -- use sky jobs launch)\nsky jobs launch spot-train.yaml\n\n# Monitor the job\nsky jobs queue\n\n# Stream logs\nsky jobs logs \n```\n\n### Fine-Tuning a HuggingFace Model\n\nFine-tune a HuggingFace model with LoRA, using secrets for authentication.\n\n```yaml\n# finetune-hf.yaml\nname: llama3-finetune\n\nworkdir: .\n\nresources:\n accelerators: A100-80GB:4\n disk_size: 512\n disk_tier: best\n\nenvs:\n MODEL_ID: meta-llama/Meta-Llama-3.1-8B-Instruct\n DATASET: yahma/alpaca-cleaned\n CHECKPOINT_BUCKET: my-finetune-checkpoints\n\n# HuggingFace token is a secret -- redacted in logs/dashboard\nsecrets:\n HF_TOKEN: null # Pass via CLI: --secret HF_TOKEN=hf_xxx\n\nfile_mounts:\n /output:\n name: ${CHECKPOINT_BUCKET}\n mode: MOUNT\n\nsetup: |\n pip install torch==2.4.0 torchvision\n pip install transformers peft trl datasets accelerate\n pip install bitsandbytes\n\n # Download the model (needs HF_TOKEN for gated models)\n huggingface-cli download $MODEL_ID \\\n --token $HF_TOKEN \\\n --local-dir /tmp/model\n\nrun: |\n python finetune_lora.py \\\n --model-path /tmp/model \\\n --dataset $DATASET \\\n --output-dir /output/lora-adapter \\\n --num-epochs 3 \\\n --batch-size 4 \\\n --learning-rate 2e-4\n```\n\n```bash\n# Launch with HF_TOKEN secret\nHF_TOKEN=hf_abc123 sky launch -c finetune finetune-hf.yaml --secret HF_TOKEN\n```\n\n---\n\n## 3. Model Serving Examples\n\n### Basic vLLM Serving (Single GPU)\n\nServe a model with vLLM behind a SkyServe endpoint.\n\n```yaml\n# vllm-serve.yaml\nname: vllm-llama\n\nresources:\n accelerators: A100:1\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\n# SkyServe configuration\nservice:\n # Readiness check endpoint (must return 200 when ready)\n readiness_probe: /v1/models\n # Fixed number of replicas\n replicas: 1\n```\n\n```bash\n# Deploy as a service\nHF_TOKEN=hf_xxx sky serve up vllm-serve.yaml -n llama-svc --secret HF_TOKEN\n\n# Get the endpoint URL\nsky serve status llama-svc --endpoint\n\n# Test the endpoint\nENDPOINT=$(sky serve status llama-svc --endpoint)\ncurl $ENDPOINT/v1/chat/completions \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"model\": \"meta-llama/Llama-3.1-8B-Instruct\",\n \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]\n }'\n\n# Tear down\nsky serve down llama-svc\n```\n\n### vLLM with Autoscaling\n\nScale replicas automatically based on query load.\n\n```yaml\n# vllm-autoscale.yaml\nname: vllm-autoscaling\n\nresources:\n accelerators: A100:1\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe:\n path: /v1/models\n # Wait 20 minutes for model loading before checking readiness\n initial_delay_seconds: 1200\n replica_policy:\n min_replicas: 1\n max_replicas: 5\n # Scale up when each replica handles more than 5 QPS\n target_qps_per_replica: 5\n # Wait 5 minutes before adding replicas\n upscale_delay_seconds: 300\n # Wait 20 minutes before removing replicas\n downscale_delay_seconds: 1200\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up vllm-autoscale.yaml -n llama-auto --secret HF_TOKEN\n```\n\n### Serving on Spot Instances\n\nUse spot instances for serving replicas to reduce cost.\n\n```yaml\n# vllm-spot-serve.yaml\nname: vllm-spot-serving\n\nresources:\n accelerators: A100:1\n ports: 8000\n # Use spot instances for replicas -- SkyServe handles recovery\n use_spot: true\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe: /v1/models\n replica_policy:\n # Keep at least 2 replicas for high availability with spot\n min_replicas: 2\n max_replicas: 6\n target_qps_per_replica: 5\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up vllm-spot-serve.yaml -n llama-spot --secret HF_TOKEN\n```\n\n### Multi-Model Serving\n\nServe multiple models by deploying separate services.\n\n```yaml\n# serve-llama-70b.yaml\nname: llama-70b-service\n\nresources:\n # 70B model needs multi-GPU with tensor parallelism\n accelerators: A100-80GB:4\n ports: 8000\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-70B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nsetup: |\n pip install vllm transformers\n huggingface-cli login --token $HF_TOKEN\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model $MODEL_NAME \\\n --tensor-parallel-size 4 \\\n --host 0.0.0.0 \\\n --port 8000\n\nservice:\n readiness_probe: /v1/models\n replicas: 2\n```\n\n```bash\n# Deploy multiple model services\nHF_TOKEN=hf_xxx sky serve up vllm-serve.yaml -n llama-8b --secret HF_TOKEN\nHF_TOKEN=hf_xxx sky serve up serve-llama-70b.yaml -n llama-70b --secret HF_TOKEN\n\n# Check all services\nsky serve status\n\n# Get endpoints\nsky serve status llama-8b --endpoint\nsky serve status llama-70b --endpoint\n```\n\n### TGI (Text Generation Inference) Example\n\nServe a model using HuggingFace TGI with Docker.\n\n```yaml\n# tgi-serve.yaml\nname: tgi-service\n\nresources:\n accelerators: A100:1\n ports: 8080\n\nenvs:\n MODEL_ID: meta-llama/Llama-3.1-8B-Instruct\n\nsecrets:\n HF_TOKEN: null\n\nrun: |\n docker run --gpus all --shm-size 1g \\\n -p 8080:80 \\\n -e HUGGING_FACE_HUB_TOKEN=$HF_TOKEN \\\n -v ~/data:/data \\\n ghcr.io/huggingface/text-generation-inference \\\n --model-id $MODEL_ID\n\nservice:\n readiness_probe: /health\n replicas: 2\n```\n\n```bash\nHF_TOKEN=hf_xxx sky serve up tgi-serve.yaml -n tgi-llama --secret HF_TOKEN\nsky serve status tgi-llama --endpoint\n```\n\n### Service with Custom Readiness Probe (POST)\n\nUse a POST request with custom data for the readiness check.\n\n```yaml\n# custom-probe-serve.yaml\nname: custom-model-service\n\nresources:\n accelerators: A100:1\n ports: 8080\n\nsetup: |\n pip install my-model-server\n\nrun: |\n python -m my_model_server --port 8080\n\nservice:\n readiness_probe:\n path: /v1/predict\n # Use POST with custom payload for readiness checking\n post_data: '{\"text\": \"health check\", \"max_tokens\": 1}'\n # Wait 30 minutes for model loading\n initial_delay_seconds: 1800\n # Timeout for each probe request\n timeout_seconds: 30\n replica_policy:\n min_replicas: 1\n max_replicas: 3\n target_qps_per_replica: 10\n```\n\n```bash\nsky serve up custom-probe-serve.yaml -n custom-svc\n```\n\n---\n\n## 4. File Mounts and Storage Examples\n\n### Sync Local Directory (workdir)\n\nThe workdir syncs your local directory to `~/sky_workdir` on the remote cluster.\n\n```yaml\n# workdir-example.yaml\nname: local-code-sync\n\n# Sync current directory to ~/sky_workdir\nworkdir: .\n\nresources:\n accelerators: L4:1\n\nsetup: |\n pip install -r requirements.txt\n\nrun: |\n # Commands run under ~/sky_workdir\n python train.py\n```\n\n```bash\n# Any changes to local files are synced on each sky launch/exec\nsky launch -c dev workdir-example.yaml\n\n# Re-sync and re-run (setup is cached)\nsky exec dev workdir-example.yaml\n```\n\n### Mount S3 Bucket (mode: MOUNT)\n\nStream data from S3 without downloading -- read-only, low disk usage.\n\n```yaml\n# s3-mount.yaml\nname: s3-streaming\n\nresources:\n accelerators: A100:1\n\nfile_mounts:\n # MOUNT mode: read-only, streams data on access, no local copy\n /data/imagenet:\n source: s3://my-imagenet-bucket/train\n mode: MOUNT\n\nrun: |\n python train.py --data-dir /data/imagenet\n```\n\n```bash\nsky launch -c train s3-mount.yaml\n```\n\n### Mount with Cache (mode: MOUNT_CACHED)\n\nCache accessed data locally for fast random access -- ideal for model weights.\n\n```yaml\n# mount-cached.yaml\nname: cached-model-weights\n\nresources:\n accelerators: A100:1\n disk_size: 512 # Need enough disk for cached data\n\nfile_mounts:\n # MOUNT_CACHED: caches accessed files locally for fast random access\n # Good for model weights that are read repeatedly\n /models/llama:\n source: s3://my-model-weights/llama-3.1-8b\n mode: MOUNT_CACHED\n\nrun: |\n python serve.py --model-path /models/llama\n```\n\n```bash\nsky launch -c serve mount-cached.yaml\n```\n\n### Copy from Cloud Storage (mode: COPY)\n\nDownload data fully -- read-write access, full local copy.\n\n```yaml\n# copy-mount.yaml\nname: full-copy\n\nresources:\n accelerators: A100:1\n # Need enough disk for the full dataset\n disk_size: 1024\n\nfile_mounts:\n # COPY mode: full download, read-write, supports modification\n /data/dataset:\n source: s3://my-dataset-bucket/training-data\n mode: COPY\n\nrun: |\n # Can read AND write to the copied data\n python preprocess.py --input /data/dataset --output /data/dataset/processed\n python train.py --data-dir /data/dataset/processed\n```\n\n```bash\nsky launch -c train copy-mount.yaml\n```\n\n### Mount GCS Bucket\n\nMount a Google Cloud Storage bucket.\n\n```yaml\n# gcs-mount.yaml\nname: gcs-data\n\nresources:\n infra: gcp\n accelerators: A100:1\n\nfile_mounts:\n # GCS bucket mount\n /data:\n source: gs://my-gcs-bucket/datasets\n mode: MOUNT\n\n # Create a new GCS bucket for output\n /output:\n name: my-training-output-bucket\n store: gcs\n mode: MOUNT\n\nrun: |\n python train.py --data /data --output /output\n```\n\n```bash\nsky launch -c gcp-train gcs-mount.yaml\n```\n\n### Using Volumes on Kubernetes\n\nKubernetes persistent and ephemeral volumes.\n\n```yaml\n# k8s-volumes.yaml\nname: k8s-with-volumes\n\nresources:\n infra: k8s\n accelerators: A100:1\n\n# Volumes (Kubernetes only)\nvolumes:\n # Persistent volume -- data survives cluster restarts\n /mnt/data: my-persistent-volume\n\n # Ephemeral volume -- fast scratch space, lost on restart\n /mnt/scratch:\n size: 200Gi\n\nrun: |\n # /mnt/data persists across restarts\n # /mnt/scratch is fast ephemeral storage\n python train.py \\\n --data-dir /mnt/data \\\n --cache-dir /mnt/scratch\n```\n\n```bash\nsky launch -c k8s-train k8s-volumes.yaml\n```\n\n### Multiple Mounts in One Task\n\nCombine different mount types in a single task.\n\n```yaml\n# multi-mount.yaml\nname: multi-mount-training\n\nworkdir: .\n\nresources:\n accelerators: A100:4\n disk_size: 512\n\nenvs:\n OUTPUT_BUCKET: my-training-outputs\n\nfile_mounts:\n # Sync local config files\n /configs: ./configs\n\n # Stream training data from S3 (read-only)\n /data/train:\n source: s3://my-dataset/train\n mode: MOUNT\n\n # Cache model weights locally for fast loading\n /models/pretrained:\n source: s3://my-models/llama-3.1-8b\n mode: MOUNT_CACHED\n\n # Writable output bucket for checkpoints\n /output:\n name: ${OUTPUT_BUCKET}\n store: s3\n mode: MOUNT\n\n # Copy evaluation data (small, needs read-write)\n /data/eval:\n source: s3://my-dataset/eval\n mode: COPY\n\nsetup: |\n pip install torch transformers datasets\n\nrun: |\n python train.py \\\n --config /configs/train_config.yaml \\\n --data-dir /data/train \\\n --eval-dir /data/eval \\\n --pretrained-model /models/pretrained \\\n --output-dir /output\n```\n\n```bash\nsky launch -c train multi-mount.yaml\n```\n\n---\n\n## 5. Multi-Cloud and Fallback Examples\n\n### any_of: Multiple GPU Types\n\nLet SkyPilot pick the cheapest available GPU from a set.\n\n```yaml\n# any-gpu.yaml\nname: flexible-gpu-training\n\nresources:\n # SkyPilot picks the cheapest available option\n any_of:\n - accelerators: H100:1\n - accelerators: A100-80GB:1\n - accelerators: A100:1\n - accelerators: L40S:1\n\nrun: |\n nvidia-smi\n python train.py\n```\n\n```bash\nsky launch -c train any-gpu.yaml\n```\n\n### any_of: Multiple Clouds for Same GPU\n\nTry different clouds for the same GPU type.\n\n```yaml\n# any-cloud.yaml\nname: multi-cloud-a100\n\nresources:\n accelerators: A100-80GB:8\n any_of:\n - infra: aws/us-east-1\n - infra: gcp/us-central1\n - infra: azure/eastus\n\nrun: |\n nvidia-smi\n python train.py\n```\n\n```bash\nsky launch -c train any-cloud.yaml\n```\n\n### ordered: Strict Priority\n\nTry clouds/regions in a specific order -- first match wins.\n\n```yaml\n# ordered-priority.yaml\nname: priority-training\n\nresources:\n ordered:\n # Try AWS us-east-1 first (our reserved instances)\n - infra: aws/us-east-1\n accelerators: H100:8\n # Then try GCP us-central1\n - infra: gcp/us-central1\n accelerators: H100:8\n # Fall back to A100s on any cloud\n - accelerators: A100-80GB:8\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train ordered-priority.yaml\n```\n\n### K8s + Cloud Fallback\n\nTry Kubernetes first, fall back to cloud if K8s has no capacity.\n\n```yaml\n# k8s-fallback.yaml\nname: k8s-or-cloud\n\nresources:\n accelerators: A100:1\n ordered:\n # Try on-prem K8s cluster first (free)\n - infra: k8s/my-onprem-cluster\n # Fall back to AWS\n - infra: aws\n # Fall back to GCP\n - infra: gcp\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train k8s-fallback.yaml\n```\n\n### Region Preference with Wildcards\n\nUse wildcards to match multiple regions or zones.\n\n```yaml\n# wildcard-region.yaml\nname: us-region-only\n\nresources:\n accelerators: A100:8\n any_of:\n # Any zone in us-east-1\n - infra: aws/us-east-1\n # Any zone in us-west-2\n - infra: aws/us-west-2\n # Any US region on GCP\n - infra: gcp/us-*\n\nrun: |\n python train.py\n```\n\n```bash\nsky launch -c train wildcard-region.yaml\n```\n\n### Cost Optimization: Cheapest Across All Clouds\n\nOmit `infra` to let SkyPilot search all enabled clouds for the cheapest option.\n\n```yaml\n# cheapest.yaml\nname: cheapest-option\n\nresources:\n # No infra specified -- SkyPilot searches all enabled clouds\n accelerators: A100:1\n\nrun: |\n python train.py\n```\n\n```bash\n# Dry run to see the cheapest option without launching\nsky launch cheapest.yaml --dryrun\n\n# Launch on the cheapest cloud/region\nsky launch -c train cheapest.yaml\n\n# Check GPU prices across clouds\nsky gpus list A100\n```\n\n---\n\n## 6. Managed Job Examples\n\n### Basic Managed Job\n\nManaged jobs run on a dedicated controller that monitors and manages your job.\n\n```yaml\n# basic-job.yaml\nname: data-processing\n\nresources:\n cpus: 16+\n memory: 64+\n\nsetup: |\n pip install pandas pyarrow\n\nrun: |\n python process_data.py --output /tmp/results\n```\n\n```bash\n# Launch as a managed job (runs on a separate controller)\nsky jobs launch basic-job.yaml\n\n# Check status\nsky jobs queue\n\n# Stream logs\nsky jobs logs 1\n\n# Cancel the job\nsky jobs cancel 1\n```\n\n### Spot Job with FAILOVER Recovery\n\nAutomatically retry in the same region first, then try other regions.\n\n```yaml\n# spot-failover.yaml\nname: training-failover\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n # FAILOVER: retry same region first, then move to next region\n strategy: FAILOVER\n # Retry up to 3 times on user code errors\n max_restarts_on_errors: 3\n\nfile_mounts:\n /checkpoints:\n name: my-training-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch spot-failover.yaml\n```\n\n### Spot Job with EAGER_NEXT_REGION Recovery\n\nMove to a different region immediately on preemption -- good for spot shortages.\n\n```yaml\n# spot-eager.yaml\nname: training-eager-recovery\n\nresources:\n accelerators: A100:8\n use_spot: true\n job_recovery:\n # EAGER_NEXT_REGION: immediately try a different region on preemption\n # Best for spot instances where preemption signals regional shortage\n strategy: EAGER_NEXT_REGION\n\nfile_mounts:\n /checkpoints:\n name: my-training-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch spot-eager.yaml\n```\n\n### Hyperparameter Sweep\n\nLaunch multiple managed jobs with different hyperparameters.\n\n```yaml\n# sweep.yaml\nname: hp-sweep\n\nresources:\n accelerators: A100:1\n\nenvs:\n LR: \"1e-4\"\n WEIGHT_DECAY: \"0.01\"\n BATCH_SIZE: \"32\"\n\nfile_mounts:\n /results:\n name: sweep-results\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers wandb\n\nrun: |\n python train.py \\\n --lr $LR \\\n --weight-decay $WEIGHT_DECAY \\\n --batch-size $BATCH_SIZE \\\n --output-dir /results/lr-${LR}_wd-${WEIGHT_DECAY}_bs-${BATCH_SIZE}\n```\n\n```bash\n# Launch multiple jobs with different hyperparameters\nfor lr in 1e-3 1e-4 1e-5; do\n for wd in 0.01 0.1; do\n sky jobs launch sweep.yaml \\\n --env LR=$lr --env WEIGHT_DECAY=$wd \\\n --name sweep-lr${lr}-wd${wd}\n done\ndone\n\n# Monitor all sweep jobs\nsky jobs queue\n```\n\n### Job with max_restarts_on_errors\n\nAutomatically restart on user code errors (non-zero exit codes).\n\n```yaml\n# auto-restart.yaml\nname: resilient-training\n\nresources:\n accelerators: A100:4\n use_spot: true\n job_recovery:\n strategy: FAILOVER\n # Restart up to 5 times if the training script crashes\n max_restarts_on_errors: 5\n\nfile_mounts:\n /checkpoints:\n name: my-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n python train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch auto-restart.yaml\n```\n\n### Job with recover_on_exit_codes for NCCL Errors\n\nAlways retry on specific exit codes (such as NCCL timeouts).\n\n```yaml\n# nccl-recovery.yaml\nname: nccl-resilient-training\n\nnum_nodes: 4\n\nresources:\n accelerators: H100:8\n use_spot: true\n job_recovery:\n strategy: EAGER_NEXT_REGION\n max_restarts_on_errors: 3\n # Exit code 33 = NCCL timeout (convention in many training scripts)\n # These restarts do NOT count toward max_restarts_on_errors\n recover_on_exit_codes: [33, 34]\n\nfile_mounts:\n /checkpoints:\n name: my-distributed-ckpts\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch transformers\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=12345 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train.py --checkpoint-dir /checkpoints --resume\n```\n\n```bash\nsky jobs launch nccl-recovery.yaml\n```\n\n### Job Pool Example\n\nUse a job pool for batch processing with pre-provisioned workers.\n\n```yaml\n# pool.yaml\n# Pool configuration: pre-provisions workers with setup\nworkdir: .\n\nresources:\n accelerators: H100:1\n disk_size: 100\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B\n\nsetup: |\n pip install vllm transformers datasets\n # Pre-download model so all jobs can use it immediately\n huggingface-cli download $MODEL_NAME --local-dir /tmp/model\n\npool:\n workers: 3\n```\n\n```yaml\n# job.yaml\n# Job that runs on the pool\nname: batch-inference\n\nresources:\n accelerators: H100:1\n\nrun: |\n echo \"Processing partition ${SKYPILOT_JOB_RANK} of ${SKYPILOT_NUM_JOBS}\"\n python classify.py \\\n --job-rank ${SKYPILOT_JOB_RANK} \\\n --num-jobs ${SKYPILOT_NUM_JOBS} \\\n --model-path /tmp/model \\\n --output-dir /results\n```\n\n```bash\n# Create the pool with 3 workers\nsky jobs pool apply -p my-pool pool.yaml\n\n# Submit 10 jobs to the pool (distributed across 3 workers)\nsky jobs launch -p my-pool --num-jobs 10 job.yaml\n\n# Check pool status\nsky jobs pool status my-pool\n\n# Check job status\nsky jobs queue\n\n# Tear down pool when done\nsky jobs pool down my-pool\n```\n\n---\n\n## 7. Distributed Training Examples\n\n### PyTorch DDP on 4 Nodes (Complete Example)\n\nFull distributed training with torchrun and NCCL configuration.\n\n```yaml\n# ddp-4node.yaml\nname: ddp-4node-training\n\nworkdir: .\n\nnum_nodes: 4\n\nresources:\n accelerators: H100:8\n disk_size: 1024\n\nenvs:\n # NCCL configuration for optimal multi-node communication\n NCCL_DEBUG: INFO\n NCCL_SOCKET_IFNAME: eth0\n # Training hyperparameters\n GLOBAL_BATCH_SIZE: \"512\"\n NUM_EPOCHS: \"50\"\n CHECKPOINT_BUCKET: my-ddp-checkpoints\n\nfile_mounts:\n # Training data (streaming)\n /data:\n source: s3://my-training-data/preprocessed\n mode: MOUNT\n\n # Checkpoint storage (persistent, writable)\n /checkpoints:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch torchvision\n pip install wandb tensorboard\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_GPUS=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))\n\n echo \"Node rank: $SKYPILOT_NODE_RANK / $SKYPILOT_NUM_NODES\"\n echo \"Master: $MASTER_ADDR\"\n echo \"Total GPUs: $NUM_GPUS\"\n\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --master_addr=$MASTER_ADDR \\\n --master_port=29500 \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n train_ddp.py \\\n --data-dir /data \\\n --checkpoint-dir /checkpoints \\\n --global-batch-size $GLOBAL_BATCH_SIZE \\\n --epochs $NUM_EPOCHS \\\n --resume\n```\n\n```bash\n# Launch distributed training\nsky launch -c ddp4 ddp-4node.yaml\n\n# SSH into head node\nssh ddp4\n\n# Check worker node\nssh ddp4-worker1\n\n# Monitor logs\nsky logs ddp4\n```\n\n### DeepSpeed ZeRO-3 on 2 Nodes\n\nDistributed training with DeepSpeed, including hostfile generation.\n\n```yaml\n# deepspeed-2node.yaml\nname: deepspeed-zero3\n\nnum_nodes: 2\n\nresources:\n accelerators: A100-80GB:8\n disk_size: 1024\n image_id: docker:nvidia/cuda:12.1.1-devel-ubuntu20.04\n\nenvs:\n MODEL_NAME: meta-llama/Llama-3.1-8B\n DEEPSPEED_ENVS: \"MODEL_NAME,SKYPILOT_NODE_RANK\"\n\nsetup: |\n pip install torch==2.4.0 --index-url https://download.pytorch.org/whl/cu121\n pip install deepspeed==0.14.4\n pip install transformers datasets accelerate\n sudo apt-get update && sudo apt-get -y install pdsh\n\nrun: |\n # Only the head node (rank 0) launches DeepSpeed\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n # Generate hostfile for DeepSpeed\n HOSTFILE=/tmp/hostfile\n python -c \"\n import os\n n_gpus = os.environ['SKYPILOT_NUM_GPUS_PER_NODE']\n ips = os.environ['SKYPILOT_NODE_IPS'].splitlines()\n with open('$HOSTFILE', 'w') as f:\n for ip in ips:\n f.write(f'{ip} slots={n_gpus}\\n')\n \"\n echo \"Hostfile:\"\n cat $HOSTFILE\n\n # Generate .deepspeed_env for environment propagation\n python -c \"\n import os\n envs = os.environ.get('DEEPSPEED_ENVS', '').split(',')\n with open('.deepspeed_env', 'w') as f:\n for var in envs:\n f.write(f'{var}=\\\"{os.getenv(var, \\\"\\\")}\\\"\\n')\n \"\n\n deepspeed \\\n --hostfile $HOSTFILE \\\n train_deepspeed.py \\\n --model-name $MODEL_NAME \\\n --deepspeed \\\n --deepspeed-config ds_config_zero3.json\n fi\n```\n\n```bash\nsky launch -c ds2 deepspeed-2node.yaml\n```\n\n### FSDP Training Example\n\nMulti-node Fully Sharded Data Parallel training with HuggingFace Accelerate.\n\n```yaml\n# fsdp-train.yaml\nname: fsdp-training\n\nworkdir: .\n\nnum_nodes: 2\n\nresources:\n accelerators: H100:8\n disk_size: 1024\n\nenvs:\n MODEL_ID: meta-llama/Llama-3.1-8B\n CHECKPOINT_BUCKET: my-fsdp-checkpoints\n\nsecrets:\n HF_TOKEN: null\n\nfile_mounts:\n /output:\n name: ${CHECKPOINT_BUCKET}\n store: s3\n mode: MOUNT\n\nsetup: |\n pip install torch --index-url https://download.pytorch.org/whl/cu128\n pip install transformers accelerate trl peft datasets\n pip install deepspeed # Needed for some FSDP optimizations\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n NUM_PROCESSES=$(($SKYPILOT_NUM_GPUS_PER_NODE * $SKYPILOT_NUM_NODES))\n\n accelerate launch \\\n --config_file configs/fsdp_config.yaml \\\n --num_machines $SKYPILOT_NUM_NODES \\\n --num_processes $NUM_PROCESSES \\\n --machine_rank $SKYPILOT_NODE_RANK \\\n --main_process_ip $MASTER_ADDR \\\n --main_process_port 29500 \\\n train_fsdp.py \\\n --model-id $MODEL_ID \\\n --output-dir /output \\\n --resume-from-checkpoint\n```\n\n```bash\nHF_TOKEN=hf_xxx sky launch -c fsdp fsdp-train.yaml --secret HF_TOKEN\n```\n\n### Ray Train Multi-Node Example\n\nDistributed training using Ray Train across multiple nodes.\n\n```yaml\n# ray-train.yaml\nname: ray-train-distributed\n\nworkdir: .\n\nnum_nodes: 2\n\nresources:\n accelerators: A100:4\n\nsetup: |\n pip install \"ray[train]\" torch torchvision\n pip install transformers datasets\n\nrun: |\n # SkyPilot sets up a Ray cluster across all nodes automatically.\n # The head node's Ray address is available at the standard port.\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n # Only run training script on the head node;\n # Ray distributes the work to all nodes\n python train_ray.py \\\n --num-workers $((SKYPILOT_NUM_NODES * SKYPILOT_NUM_GPUS_PER_NODE)) \\\n --gpus-per-worker 1\n fi\n```\n\nThe `train_ray.py` script uses `ray.train`:\n\n```python\n# train_ray.py (referenced by the YAML above)\nimport ray\nfrom ray import train\nfrom ray.train.torch import TorchTrainer\nfrom ray.train import ScalingConfig\n\ndef train_func(config):\n import torch\n # Your PyTorch training code here\n # ray.train handles distributed setup automatically\n model = ...\n optimizer = ...\n for epoch in range(config[\"epochs\"]):\n train_epoch(model, optimizer)\n # Report metrics\n train.report({\"loss\": loss, \"epoch\": epoch})\n\nscaling_config = ScalingConfig(\n num_workers=args.num_workers,\n use_gpu=True,\n resources_per_worker={\"GPU\": args.gpus_per_worker},\n)\n\ntrainer = TorchTrainer(\n train_func,\n train_loop_config={\"epochs\": 10},\n scaling_config=scaling_config,\n)\nresult = trainer.fit()\n```\n\n```bash\nsky launch -c ray-train ray-train.yaml\n```\n\n---\n\n## 8. Python SDK Examples\n\n### Basic: Create Task, Set Resources, Launch\n\n```python\nimport sky\n\n# Create a task programmatically\ntask = sky.Task(\n name='hello-gpu',\n run='nvidia-smi && python -c \"import torch; print(torch.cuda.device_count())\"',\n)\ntask.set_resources(sky.Resources(\n accelerators='A100:1',\n))\n\n# Launch returns a request ID (async by default)\nrequest_id = sky.launch(task, cluster_name='my-cluster')\n\n# Wait for the launch to complete\njob_id, handle = sky.get(request_id)\nprint(f'Job ID: {job_id}')\n```\n\n### Launch from YAML File\n\n```python\nimport sky\n\n# Load task from a YAML file\ntask = sky.Task.from_yaml('train.yaml')\n\n# Launch with options\nrequest_id = sky.launch(\n task,\n cluster_name='train-cluster',\n idle_minutes_to_autostop=30, # Auto-stop after 30 min idle\n retry_until_up=True, # Keep trying if resources unavailable\n)\nresult = sky.get(request_id)\n```\n\n### Check Status and Get Cluster Info\n\n```python\nimport sky\n\n# Get all cluster statuses\nrequest_id = sky.status()\nclusters = sky.get(request_id)\n\nfor cluster in clusters:\n print(f\"Cluster: {cluster['name']}\")\n print(f\" Status: {cluster['status']}\")\n print(f\" Resources: {cluster['resources_str']}\")\n print(f\" Autostop: {cluster['autostop']} min\")\n\n# Get status for a specific cluster\nrequest_id = sky.status(cluster_names=['my-cluster'])\nclusters = sky.get(request_id)\n\n# Get job queue for a cluster\nrequest_id = sky.queue('my-cluster')\njobs = sky.get(request_id)\nfor job in jobs:\n print(f\"Job {job['job_id']}: {job['status']}\")\n```\n\n### Batch Submission (Launch Multiple Tasks)\n\n```python\nimport sky\n\n# Launch multiple training runs with different configs\nconfigs = [\n {'lr': 1e-3, 'batch_size': 32},\n {'lr': 1e-4, 'batch_size': 64},\n {'lr': 1e-5, 'batch_size': 128},\n]\n\nrequest_ids = []\nfor i, cfg in enumerate(configs):\n task = sky.Task(\n name=f'sweep-{i}',\n setup='pip install torch transformers',\n run=f'python train.py --lr {cfg[\"lr\"]} --batch-size {cfg[\"batch_size\"]}',\n )\n task.set_resources(sky.Resources(accelerators='A100:1'))\n\n request_id = sky.launch(\n task,\n cluster_name=f'sweep-{i}',\n idle_minutes_to_autostop=10,\n down=True, # Auto-teardown after idle\n )\n request_ids.append(request_id)\n print(f'Launched sweep-{i}: {request_id}')\n\n# Wait for all to complete\nfor rid in request_ids:\n result = sky.get(rid)\n print(f'Completed: {rid}')\n```\n\n### Wait for Completion and Get Results\n\n```python\nimport sky\n\n# Launch a task\ntask = sky.Task.from_yaml('train.yaml')\nrequest_id = sky.launch(task, cluster_name='train')\n\n# stream_and_get streams logs AND waits for completion\nresult = sky.stream_and_get(request_id)\n\n# Check job status\nrequest_id = sky.job_status('train', job_ids=[1])\nstatuses = sky.get(request_id)\nprint(f'Job 1 status: {statuses[1]}')\n\n# Tail logs (blocking, streams to console)\nexit_code = sky.tail_logs('train', job_id=1, follow=True)\nprint(f'Job exit code: {exit_code}')\n```\n\n### Programmatic Managed Job Launch\n\n```python\nimport sky\n\n# Create a managed job task\ntask = sky.Task(\n name='managed-training',\n setup='pip install torch transformers',\n run='python train.py --resume',\n)\ntask.set_resources(sky.Resources(\n accelerators='A100:4',\n use_spot=True,\n job_recovery={\n 'strategy': 'EAGER_NEXT_REGION',\n 'max_restarts_on_errors': 3,\n },\n))\n\n# Launch as a managed job\nrequest_id = sky.jobs.launch(task)\njob_id = sky.get(request_id)\nprint(f'Managed job ID: {job_id}')\n\n# Check managed job queue\nrequest_id = sky.jobs.queue()\njobs = sky.get(request_id)\nfor job in jobs:\n print(f\"Job {job['job_id']}: {job['job_name']} - {job['status']}\")\n```\n\n### DAG Workflow (Task Dependencies)\n\n```python\nimport sky\n\n# Create a DAG with task dependencies\nwith sky.Dag() as dag:\n # Step 1: Data preprocessing\n preprocess = sky.Task(\n name='preprocess',\n setup='pip install pandas pyarrow',\n run='python preprocess.py --output /tmp/data',\n )\n preprocess.set_resources(sky.Resources(cpus='8+', memory='32+'))\n\n # Step 2: Training (depends on preprocessing)\n train = sky.Task(\n name='train',\n setup='pip install torch transformers',\n run='python train.py --data /tmp/data',\n )\n train.set_resources(sky.Resources(accelerators='A100:4'))\n\n # Define dependency: preprocess must finish before train\n preprocess >> train\n\n# Launch the DAG (experimental)\nrequest_id = sky.launch(dag, cluster_name='pipeline')\nsky.get(request_id)\n```\n\n### Autostop Management\n\n```python\nimport sky\n\n# Set autostop: stop after 30 minutes of idleness\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=30,\n)\nsky.get(request_id)\n\n# Set autodown: tear down (not just stop) after 60 minutes idle\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=60,\n down=True,\n)\nsky.get(request_id)\n\n# Cancel autostop\nrequest_id = sky.autostop(\n cluster_name='my-cluster',\n idle_minutes=-1, # Negative value cancels autostop\n)\nsky.get(request_id)\n\n# Stop and start a cluster\nsky.get(sky.stop('my-cluster'))\nsky.get(sky.start('my-cluster'))\n\n# Tear down a cluster\nsky.get(sky.down('my-cluster'))\n```\n\n### List GPUs Programmatically\n\n```python\nimport sky\n\n# List all available accelerators\nrequest_id = sky.list_accelerators(\n gpus_only=True,\n name_filter='A100',\n)\naccelerators = sky.get(request_id)\n\nfor acc_name, offerings in accelerators.items():\n print(f'\\n{acc_name}:')\n for offering in offerings:\n print(f' {offering}')\n\n# List accelerator counts (how many GPUs per node)\nrequest_id = sky.list_accelerator_counts(\n gpus_only=True,\n name_filter='H100',\n)\ncounts = sky.get(request_id)\nfor acc_name, available_counts in counts.items():\n print(f'{acc_name}: available in counts {available_counts}')\n\n# Check which clouds are enabled\nrequest_id = sky.enabled_clouds()\nclouds = sky.get(request_id)\nprint(f'Enabled clouds: {clouds}')\n```\n\n---\n\n## Quick Reference: SkyPilot Environment Variables\n\nThese variables are automatically set by SkyPilot on every node and are available\nin `setup` and `run` commands:\n\n| Variable | Description | Example Value |\n|----------|-------------|---------------|\n| `SKYPILOT_NODE_RANK` | Rank of the current node (0 = head) | `0` |\n| `SKYPILOT_NODE_IPS` | Newline-separated IPs of all nodes | `10.0.0.1\\n10.0.0.2` |\n| `SKYPILOT_NUM_NODES` | Total number of nodes in the cluster | `2` |\n| `SKYPILOT_NUM_GPUS_PER_NODE` | Number of GPUs on this node | `8` |\n| `SKYPILOT_TASK_ID` | Unique task identifier | `sky-2024-01-15-abc123` |\n| `SKYPILOT_CLUSTER_INFO` | JSON with cluster metadata | `{\"cloud\": \"aws\", ...}` |\n| `SKYPILOT_JOB_RANK` | Job rank within a pool (pool jobs only) | `3` |\n| `SKYPILOT_NUM_JOBS` | Total number of jobs in batch (pool jobs only) | `10` |\n\n## Quick Reference: Common CLI Patterns\n\n```bash\n# Launch and auto-stop after 30 minutes idle\nsky launch -c mycluster task.yaml -i 30\n\n# Launch with auto-teardown (not just stop) after idle\nsky launch -c mycluster task.yaml -i 30 --down\n\n# Launch detached (don't stream logs)\nsky launch -c mycluster task.yaml -d\n\n# Retry until resources are available\nsky launch -c mycluster task.yaml -r\n\n# Override GPU from CLI\nsky launch -c mycluster task.yaml --gpus H100:4\n\n# Override cloud from CLI\nsky launch -c mycluster task.yaml --infra aws/us-east-1\n\n# Execute on existing cluster (skip setup)\nsky exec mycluster another-task.yaml\n\n# Execute inline command on existing cluster\nsky exec mycluster -- python eval.py --checkpoint /tmp/best.pt\n\n# Check GPU pricing\nsky gpus list H100\nsky gpus list A100 --all-regions\nsky gpus list --infra aws\n```\n\n\n# Investigating a Managed Job\n\nHow to answer \"what is my job doing and why\" with as few commands as\npossible. Every command here supports `-o json`; prefer it over parsing\ntable output.\n\n## Which command answers which question\n\n| Question | Command | Look at |\n|----------|---------|---------|\n| What state is job N in, and why? | `sky jobs queue -v -o json` | `status`, `details`, `failure_reason`, `schedule_state`, `recovery_count` |\n| Why is job N still provisioning / pending on the cluster? | `sky jobs queue -v -o json` | `details`, when it starts with `Launching (` |\n| How long has job N been at this? | `sky jobs queue -v -o json` | `submitted_at`, and the duration columns in the table |\n| What did job N print recently? | `sky jobs logs N --tail 100 --no-follow` | stdout/stderr of the task |\n| What did the controller do (provision, recover, retry)? | `sky jobs logs N --controller --no-follow` | controller log |\n| Wait until job N finishes | `sky jobs logs N` (blocks) | exit when the job ends |\n\nDo not poll with `sleep` + `sky jobs queue`. Stream logs to block, or\nfetch `--tail` once after the fact.\n\n`details` is the current explanation, not a history. For the sequence of\ntransitions a job went through — and, on Slurm, how long the allocation\nwaited and on what — the timeline lives behind the `jobs.events` API and is\nsurfaced by the enterprise client.\n\n## Reading `details`\n\n`details` in `sky jobs queue -v -o json` is the one-line explanation of the\ncurrent state. Common shapes:\n\n| `details` | Meaning | Next step |\n|-----------|---------|-----------|\n| `Waiting for other jobs to launch` | Controller is at its concurrent-launch limit | Nothing to fix; the duration columns say how long it has waited |\n| `Waiting for higher priority jobs to launch` | A higher-priority managed job is ahead | Raise `priority` in the task YAML if this job matters more |\n| `In backoff, waiting for resources` | Provisioning failed and is retrying | `sky jobs logs N --controller --no-follow` shows the last failure |\n| `Recovering: ` | Cluster was lost or the task crashed (e.g. OOMKilled) | Fix the cause; recovery is automatic |\n| `Failure: ` | Terminal failure | Read the reason; `sky jobs logs N --no-follow` for the task output |\n| `Cancellation requested by ...` | Someone cancelled it | The requester and request ID are in the text |\n| `Launching (pending: ; partition:

)` | Job is STARTING and Slurm has not allocated nodes yet | See \"Slurm pending reasons\" below |\n| `Launching (nodes allocated; Slurm job on )` | The queue wait is over; the job is still bootstrapping | Nothing to fix. That id is what `sacct -j ` takes if you have login-node access |\n\n## Slurm pending reasons\n\nWhile a managed job's Slurm allocation is queued, `details` carries the\n`squeue` reason and the partition it is queued in — the partition matters\nbecause the same reason means different things in different partitions, and\nby the time anyone looks the allocation may be gone. The common codes:\n\n| Reason | Category | What to do |\n|--------|----------|------------|\n| `Resources`, `Priority`, `ReqNodeNotAvail` | No free capacity in the partition | Wait, or resubmit to another partition via `config.slurm` |\n| `QOSGrpGRES`, `QOSMax*`, `AssocGrp*` | A QoS / account quota is full | Wait for the group's jobs to finish, or use a different QoS/partition |\n| `Dependency`, `DependencyNeverSatisfied` | Waiting on another Slurm job; `NeverSatisfied` will pend forever | Check the dependency job; cancel and resubmit if it failed |\n| `JobHeldUser`, `JobHeldAdmin`, `launch failed requeued held` | Job is held | Ask the admin to `scontrol release`, or cancel and relaunch |\n| `BeginTime` | Scheduled start time not reached | Nothing to fix |\n\nThe reason is recorded each time it changes, so a job that moved from\n`Resources` to `Priority` and back leaves all three behind it — `details`\nshows only the latest.\n\n## Recognising a stuck job\n\n- Repeated `Recovering: ` means a persistent problem (bad node,\n OOM, quota). The controller keeps retrying and will not fix it; read\n `sky jobs logs N --controller --no-follow`.\n- A long gap between STARTING and RUNNING is provisioning time. On Slurm\n that is queue time, and `details` says what the queue is waiting on.\n- `failure_reason` distinguishes the user's program failing from an\n infrastructure failure; do not report the latter as a code bug.\n\n\n# Migrating from Slurm Reference\n\nThis reference is for converting existing Slurm workloads into SkyPilot task\nYAMLs, and for answering \"how do I do X, which I used to do with Slurm?\".\n\nTwo distinct situations, which need different answers:\n\n- **SkyPilot on the user's existing Slurm cluster.** SkyPilot submits to it\n through the login node via `sbatch`. The cluster is unchanged. Most users\n start here — do not assume they want to leave Slurm.\n- **SkyPilot on Kubernetes or a cloud.** The workload moves off Slurm. The\n YAML is the same; the constraints are different (see section 8).\n\nAsk which one applies if it is not obvious. The conversion is nearly identical;\nthe difference shows up in lifecycle (section 7) and what is unsupported\n(section 8).\n\n---\n\n## 1. Conversion Procedure\n\nFollow this order. Do not skip step 5.\n\n1. **Read the whole `sbatch` script**, not just the `#SBATCH` block. The\n `srun` invocations and `module load` lines carry as much information as the\n directives.\n2. **Map the `#SBATCH` directives** to `resources` / `num_nodes` (section 2).\n3. **Map the body**: `module load` → `setup`, environment activation → `setup`,\n the actual work → `run` (section 6).\n4. **Translate `srun`** — this is the step most likely to be done wrong\n (section 5).\n5. **Validate**: `sky launch --dryrun `. This resolves resources and\n runs the optimizer without provisioning anything. Note there is **no**\n `--dryrun` on `sky jobs launch`, so dry-run against `sky launch` even when\n the final command will be `sky jobs launch`.\n6. **Tell the user which submit command to use**: `sky jobs launch` for batch\n work (an `sbatch` script), `sky launch` for an interactive allocation (an\n `salloc` session).\n\n### 1.1 Choosing the submit command\n\n| The user's script is | Use | Why |\n|---|---|---|\n| `sbatch` batch job | `sky jobs launch` | Allocation released when the job ends, and the job is recovered if the node fails |\n| `salloc` / `srun --pty` session | `sky launch` + `ssh ` | Long-lived allocation to work in |\n| `sbatch --array` | `sky jobs launch --num-jobs N` | Section 7.1 |\n| A chain with `--dependency` | One managed pipeline | Section 7.2 |\n\n---\n\n## 2. `#SBATCH` Directive Mapping\n\n| `#SBATCH` directive | SkyPilot YAML | Notes |\n|---|---|---|\n| `--nodes=2` | `num_nodes: 2` | Task-level field, **not** under `resources` |\n| `--gpus-per-node=8`, `--gres=gpu:h100:8` | `resources.accelerators: H100:8` | Emitted as `--gres=gpu::`. The name must match the cluster's GRES config |\n| `--cpus-per-task=32` | `resources.cpus: 32+` | `+` means \"at least\". Emitted as `--cpus-per-task` |\n| `--mem=256G` | `resources.memory: 256+` | In GB. Emitted as `--mem` |\n| `--partition=gpu` | `resources.infra: slurm//gpu` | Omit to let the optimizer choose a partition |\n| `--time=24:00:00` | `config.slurm.sbatch_options.time: \"24:00:00\"` | Defaults to the partition's `MaxTime` (or `DefaultTime` when `MaxTime` is `UNLIMITED`) if unset. **Not** `autostop` — see section 8 |\n| `--job-name=train` | `name: train` | |\n| `--output=`, `--error=` | *(nothing)* | SkyPilot manages job logs; see section 7.4 |\n| `--account`, `--qos`, `--reservation`, `--exclusive`, `--constraint`, ... | `config.slurm.sbatch_options.` | Section 3 |\n\n### 2.1 Worked example\n\nInput:\n\n```bash\n#!/bin/bash\n#SBATCH --job-name=train\n#SBATCH --nodes=2\n#SBATCH --gpus-per-node=8\n#SBATCH --cpus-per-task=32\n#SBATCH --mem=256G\n#SBATCH --partition=gpu\n#SBATCH --time=24:00:00\n#SBATCH --account=ai-research\n\nmodule load cuda/12.1\nsource ~/venv/bin/activate\n\nsrun python train.py --epochs 100\n```\n\nOutput:\n\n```yaml\n# train.yaml\nname: train\n\nnum_nodes: 2\n\nresources:\n infra: slurm/mycluster/gpu\n accelerators: H100:8\n cpus: 32+\n memory: 256+\n\nconfig:\n slurm:\n sbatch_options:\n time: \"24:00:00\"\n account: ai-research\n\nsetup: |\n pip install -r requirements.txt\n\nrun: |\n python train.py --epochs 100\n```\n\nSubmit with `sky jobs launch -n train train.yaml`.\n\nNote what happened to the `srun`: it disappeared. `run` already executes on\nevery node of the allocation, so wrapping the command in `srun` again would\nnest a job step inside SkyPilot's own. See section 5.\n\n---\n\n## 3. Directives SkyPilot Does Not Model\n\nAnything without a `resources` equivalent goes through\n`slurm.sbatch_options`, which is emitted verbatim as `#SBATCH` lines. Keys are\n`sbatch` long option names; underscores become hyphens; `true` emits a bare\nflag; `null` / `false` omit the line.\n\n```yaml\nconfig:\n slurm:\n sbatch_options:\n account: ai-research\n qos: high\n time: \"24:00:00\"\n exclusive: true\n constraint: ib\n```\n\nThe same block works in `~/.sky/config.yaml` under a top-level `slurm:` key to\napply fleet-wide, and can be scoped per cluster or per partition:\n`slurm.cluster_configs..sbatch_options` and\n`slurm.cluster_configs..partition_configs..sbatch_options`.\n\n**Do not put these in `sbatch_options`** — SkyPilot manages them and will drop\nthem with a warning:\n\n```\njob-name output error nodes wait-all-nodes\nno-requeue requeue cpus-per-task mem gres partition\n```\n\nIf a user asks for one of those, set it through `resources` / `name` instead.\n\n---\n\n## 4. Environment Variable Mapping\n\n| Slurm | SkyPilot | Notes |\n|---|---|---|\n| `$SLURM_JOB_NODELIST` | `$SKYPILOT_NODE_IPS` | Newline-separated IPs, **not** a Slurm hostlist expression. `$(echo \"$SKYPILOT_NODE_IPS\" \\| head -n1)` is the head node |\n| `$SLURM_NNODES` | `$SKYPILOT_NUM_NODES` | |\n| `$SLURM_NODEID`, `$SLURM_PROCID` | `$SKYPILOT_NODE_RANK` | `0` to `num_nodes-1` |\n| `$SLURM_GPUS_PER_NODE` | `$SKYPILOT_NUM_GPUS_PER_NODE` | |\n| `$SLURM_JOB_ID` | `$SKYPILOT_TASK_ID` | |\n| `$SLURM_ARRAY_TASK_ID` | `$SKYPILOT_JOB_RANK` | Only set under `--num-jobs`; section 7.1 |\n| `$SLURM_ARRAY_TASK_COUNT` | `$SKYPILOT_NUM_JOBS` | |\n\n**On Slurm specifically**, the *job-scoped* `SLURM_*` variables of the\nunderlying allocation are still present inside `run` — `SLURM_JOB_ID`,\n`SLURM_JOB_NODELIST`, `SLURM_GPUS_ON_NODE`, etc. Scripts that read them keep\nworking. The *step-scoped* variables from SkyPilot's own job step\n(`SLURM_CPUS_PER_TASK`, `SLURM_CPU_BIND*`, `SLURM_STEP_*`, `SLURM_PROCID`, ...)\nare deliberately unset before the user script runs, so a nested `srun` targets\nthe full allocation instead of inheriting SkyPilot's step shape.\n\nPrefer the `SKYPILOT_*` variables in generated YAML — they are what makes the\nsame task portable to Kubernetes and clouds. Use the `SLURM_*` ones only when\ntranslating a script that must keep working unchanged.\n\n---\n\n## 5. Translating `srun`\n\nThree cases. Pick by what the `srun` was *for*.\n\n### 5.1 `srun ` launching one process per node\n\nDrop the `srun`. `run` already runs on every node.\n\n```bash\n# Slurm\nsrun python preprocess.py\n```\n\n```yaml\nrun: |\n python preprocess.py\n```\n\n### 5.2 `srun` launching distributed training ranks\n\nUse the rank variables and the framework's own launcher. This is the portable\nform and should be the default for training workloads.\n\n```yaml\nnum_nodes: 2\n\nresources:\n accelerators: H100:8\n\nrun: |\n MASTER_ADDR=$(echo \"$SKYPILOT_NODE_IPS\" | head -n1)\n torchrun \\\n --nnodes=$SKYPILOT_NUM_NODES \\\n --nproc_per_node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n --node_rank=$SKYPILOT_NODE_RANK \\\n --master_addr=$MASTER_ADDR \\\n --master_port=8008 \\\n train.py\n```\n\n### 5.3 `srun` launching an MPI/PMIx program\n\nSome binaries must be launched by Slurm's own PMI (NCCL tests, anything\nMPI-linked, `--mpi=pmix`). Keep `srun`, but:\n\n- **Gate it on rank 0**, or every node will launch the whole job.\n- **Pass `--overlap`**, so the step can share the allocation with SkyPilot's\n own step.\n\n```yaml\nnum_nodes: 2\n\nresources:\n infra: slurm\n accelerators: H100:8\n\nrun: |\n if [ \"$SKYPILOT_NODE_RANK\" == \"0\" ]; then\n srun --overlap --mpi=pmix \\\n --ntasks-per-node=$SKYPILOT_NUM_GPUS_PER_NODE \\\n ./my_mpi_program\n fi\n```\n\nThis form is Slurm-only by construction — it will not run on Kubernetes or a\ncloud. Say so when generating it.\n\n---\n\n## 6. Replacing the Module System\n\n`module load` reads a cluster-wide tree shared by all jobs. SkyPilot gives\neach job its own environment. In increasing order of isolation:\n\n```yaml\n# 1. setup commands — runs once per cluster, before the job\nsetup: |\n pip install -r requirements.txt\n```\n\n```yaml\n# 2. a container image — the closest thing to a reproducible module set.\n# On Slurm this needs the Pyxis SPANK plugin on the cluster.\nresources:\n image_id: docker:pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime\n```\n\n```yaml\n# 3. conda on the shared filesystem, if the user wants to keep their layout\nsetup: |\n conda create -n myenv python=3.10 -y\n conda activate myenv\n conda install pytorch pytorch-cuda=12.1 -c pytorch -c nvidia -y\n\nrun: |\n conda activate myenv\n python train.py\n```\n\n`module load` itself still works inside `setup` / `run` on a Slurm cluster,\nas long as the task is not running in a container. Keeping it is a legitimate\nfirst migration step — it just does not port to Kubernetes or clouds.\n\n---\n\n## 7. Patterns Without a One-to-One Field\n\n### 7.1 Job arrays → `--num-jobs`\n\n`sbatch --array=0-99` becomes:\n\n```bash\nsky jobs launch --num-jobs 100 -y -d task.yaml\n```\n\nEach job gets `$SKYPILOT_JOB_RANK` (the index) and `$SKYPILOT_NUM_JOBS`:\n\n```yaml\nrun: |\n python train.py --shard $SKYPILOT_JOB_RANK --num-shards $SKYPILOT_NUM_JOBS\n```\n\nEach job provisions its own allocation. To reuse workers across submissions\ninstead — closer to how a long-lived Slurm allocation behaves, and much\ncheaper when `setup` is expensive — submit to a pool (note `pool.yaml` is a\npool spec: it needs a `pool:` section, not a bare task YAML):\n\n```bash\nsky jobs pool apply -p mypool pool.yaml\nsky jobs launch -p mypool --num-jobs 100 task.yaml\n```\n\nFor sweeps over *named* parameters rather than an index, loop with `--env`:\n\n```bash\nfor lr in 0.001 0.01 0.1; do\n sky jobs launch --env LR=$lr -y -d task.yaml\ndone\n```\n\n### 7.2 Job dependencies → a managed pipeline\n\n`sbatch --dependency=afterok:` becomes a sequence of tasks in one\nmanaged job, separated by `---`. Each task can request different resources.\n\n```yaml\nname: train-then-eval\n\n---\n\nname: train\nresources:\n accelerators: H100:8\nrun: python train.py\n\n---\n\nname: eval\nresources:\n accelerators: L4:1\nrun: python eval.py\n```\n\nFor tasks that should run *in parallel* rather than in sequence, use a job\ngroup instead of a pipeline. A job group is the same multi-document layout,\nbut the header must set `execution: parallel` — without it the file is a\n*serial* pipeline:\n\n```yaml\nname: train-and-eval\nexecution: parallel\n\n---\n\nname: train\nresources:\n accelerators: H100:8\nrun: python train.py\n\n---\n\nname: eval\nresources:\n accelerators: L4:1\nrun: python eval.py\n```\n\n### 7.3 Login node workflow\n\nThere is no login node in the SkyPilot workflow. Submit from the user's laptop\nor CI; SkyPilot makes the SSH hop to the login node itself. Sync code with\n`workdir` (a local path or a git URL) rather than editing on NFS. For\ninteractive work, `ssh ` lands on the *allocated compute node*, not a\nshared login node.\n\n### 7.4 Job output files\n\nSlurm writes `slurm-.out`. SkyPilot streams logs instead:\n\n```bash\nsky jobs logs # managed job\nsky logs # latest job on a cluster\nsky logs 2 # a specific job\n```\n\nDo not translate `--output` / `--error` into anything. If the script writes\nits own result files, point them at the shared filesystem or a bucket.\n\n---\n\n## 8. Constraints When Running on Slurm\n\nDo not generate YAML that uses these on a Slurm target — they will fail or be\nrejected:\n\n| Feature | Status on Slurm | What to do instead |\n|---|---|---|\n| `autostop` / `sky stop` | Unsupported | `sky down` explicitly; bound wall-clock with `sbatch_options.time` |\n| `ports:` / `sky serve` | Unsupported | Run serving on Kubernetes or a cloud |\n| `use_spot: true` | Unsupported | Omit; there is no spot tier |\n| Jobs/serve controllers | Not hosted on Slurm | With a remote API server, consolidation mode runs the controller inside the API server, so `sky jobs launch` works against a Slurm-only fleet. With a local API server the controller needs Kubernetes or a cloud |\n| `image_id: docker:...` | Needs Pyxis on the cluster | Fall back to `setup` commands |\n| Bucket mounting (`mode: MOUNT`) | Needs FUSE on compute nodes | Use the shared filesystem, or `mode: COPY` |\n\nBecause autostop does not exist on Slurm, an idle `sky launch` cluster holds\nits allocation until `sky down`. Prefer `sky jobs launch` for anything\nbatch-like so the allocation is released automatically.\n\n### 8.1 If the target is Kubernetes instead\n\nThe assumption that breaks first is shared storage: Kubernetes does not mount\nhome directories. A Slurm script that reads `~/data` or writes `~/checkpoints`\nneeds one of:\n\n- an NFS mount via `config.kubernetes.pod_config` (if the nodes can reach the\n existing NFS server),\n- a `ReadWriteMany` PVC through SkyPilot volumes,\n- a bucket under `file_mounts` with `mode: MOUNT`,\n- `workdir` for code only.\n\nPartitions map to Kubernetes contexts (`infra: kubernetes/`), and\nSlurm's fairshare/QoS has no built-in equivalent — that is Kueue or priority\nclasses.\n\n---\n\n## 9. Validation Checklist\n\nBefore handing a converted YAML to the user:\n\n- [ ] `sky launch --dryrun ` resolves without error.\n- [ ] `num_nodes` is at task level, not under `resources`.\n- [ ] No `autostop` / `ports` / `use_spot` if the target is Slurm.\n- [ ] Every `srun` is accounted for by section 5 — dropped, replaced with\n rank variables, or kept with `--overlap` and gated on rank 0.\n- [ ] GPU name matches the cluster's GRES names (check `sky gpus list --infra\n slurm`).\n- [ ] Protected `sbatch` options are not in `sbatch_options` (section 3).\n- [ ] The user is told whether to run `sky launch` or `sky jobs launch`.\n\n\n\n\n\n# SkyPilot Python SDK Reference\n\nAll SDK functions return a request ID (future). Use `sky.get(request_id)` to await the result.\n\n```python\nimport sky\n\nrequest_id = sky.launch(sky.Task.from_yaml(\"task.yaml\"))\nresult = sky.get(request_id)\n```\n\n## Cluster Operations\n\n### `sky.launch`\n\n```python\nsky.launch(task: Union['sky.Task', 'sky.Dag'], cluster_name: Optional[str] = None, retry_until_up: bool = False, idle_minutes_to_autostop: Optional[int] = None, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, dryrun: bool = False, down: bool = False, backend: Optional['backends.Backend'] = None, optimize_target: common.OptimizeTarget = common.OptimizeTarget.COST, no_setup: bool = False, clone_disk_from: Optional[str] = None, fast: bool = False, resize: bool = False, _need_confirmation: bool = False, _is_launched_by_jobs_controller: bool = False, _is_launched_by_sky_serve_controller: bool = False, _file_mounts_blob_id: Optional[str] = None, _extra_launch_context: Optional[Dict[str, Any]] = None, _include_credentials: bool = False) -> server_common.RequestId[Tuple[Optional[int], Optional['backends.ResourceHandle']]]\n```\n\nLaunches a cluster or task.\n\nThe task's setup and run commands are executed under the task's workdir\n(when specified, it is synced to remote cluster). The task undergoes job\nqueue scheduling on the cluster.\n\nCurrently, the first argument must be a sky.Task, or (EXPERIMENTAL advanced\nusage) a sky.Dag. In the latter case, currently it must contain a single\ntask; support for pipelines/general DAGs are in experimental branches.\n\n**Example:**\n ```python\n\n import sky\n task = sky.Task(run='echo hello SkyPilot')\n task.set_resources(\n sky.Resources(infra='aws', accelerators='V100:4'))\n sky.launch(task, cluster_name='my-cluster')\n\n\n**Args:**\n task: sky.Task, or sky.Dag (experimental; 1-task only) to launch.\n cluster_name: name of the cluster to create/reuse. If None,\n auto-generate a name.\n retry_until_up: whether to retry launching the cluster until it is\n up.\n idle_minutes_to_autostop: automatically stop the cluster after this\n many minute of idleness, i.e., no running or pending jobs in the\n cluster's job queue. Idleness gets reset whenever setting-up/\n running/pending jobs are found in the job queue. Setting this\n flag is equivalent to running\n ``sky.launch(...)`` and then\n ``sky.autostop(idle_minutes=)``. If set, the autostop\n config specified in the task' resources will be overridden by\n this parameter.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes_to_autostop``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after\n ``idle_minutes_to_autostop``.\n dryrun: if True, do not actually launch the cluster.\n down: Tear down the cluster after all jobs finish (successfully or\n abnormally). If --idle-minutes-to-autostop is also set, the\n cluster will be torn down after the specified idle time.\n Note that if errors occur during provisioning/data syncing/setting\n up, the cluster will not be torn down for debugging purposes. If\n set, the autostop config specified in the task' resources will be\n overridden by this parameter.\n backend: backend to use. If None, use the default backend\n (CloudVMRayBackend).\n optimize_target: target to optimize for. Choices: OptimizeTarget.COST,\n OptimizeTarget.TIME.\n no_setup: if True, do not re-run setup commands.\n clone_disk_from: [Experimental] if set, clone the disk from the\n specified cluster. This is useful to migrate the cluster to a\n different availability zone or region.\n fast: [Experimental] If the cluster is already up and available,\n skip provisioning and setup steps.\n resize: if True, resize the existing cluster to the ``num_nodes``\n specified in the task. Supports both scaling up (adding workers)\n and scaling down (removing workers). Scale-down requires no\n running jobs on the cluster. If True, requires ``cluster_name``\n to be set.\n _need_confirmation: (Internal only) If True, show the confirmation\n prompt.\n\n**Returns:**\n The request ID of the launch request.\n\n**Request Returns:**\n job_id (Optional[int]): the job ID of the submitted job. None if the\n backend is not ``CloudVmRayBackend``, or no job is submitted to the\n cluster.\n handle (Optional[backends.ResourceHandle]): the handle to the cluster.\n None if dryrun.\n\n**Request Raises:**\n exceptions.ClusterOwnerIdentityMismatchError: if the cluster is owned\n by another user.\n exceptions.InvalidClusterNameError: if the cluster name is invalid.\n exceptions.ResourcesMismatchError: if the requested resources\n do not match the existing cluster.\n exceptions.NotSupportedError: if required features are not supported\n by the backend/cloud/cluster.\n exceptions.ResourcesUnavailableError: if the requested resources\n cannot be satisfied. The failover_history of the exception will be set\n as:\n\n 1. Empty: iff the first-ever sky.optimize() fails to find a feasible\n resource; no pre-check or actual launch is attempted.\n\n 2. Non-empty: iff at least 1 exception from either our pre-checks\n (e.g., cluster name invalid) or a region/zone throwing resource\n unavailability.\n\n exceptions.CommandError: any ssh command error.\n exceptions.NoCloudAccessError: if all clouds are disabled.\n\nOther exceptions may be raised depending on the backend.\n\n### `sky.exec`\n\n```python\nsky.exec(task: Union['sky.Task', 'sky.Dag'], cluster_name: Optional[str] = None, dryrun: bool = False, down: bool = False, backend: Optional['backends.Backend'] = None) -> server_common.RequestId[Tuple[Optional[int], Optional['backends.ResourceHandle']]]\n```\n\nExecutes a task on an existing cluster.\n\n**This function performs two actions:**\n\n(1) workdir syncing, if the task has a workdir specified;\n(2) executing the task's ``run`` commands.\n\nAll other steps (provisioning, setup commands, file mounts syncing) are\nskipped. If any of those specifications changed in the task, this function\nwill not reflect those changes. To ensure a cluster's setup is up to date,\nuse ``sky.launch()`` instead.\n\n**Execution and scheduling behavior:**\n\n- The task will undergo job queue scheduling, respecting any specified\n resource requirement. It can be executed on any node of the cluster with\n enough resources.\n- The task is run under the workdir (if specified).\n- The task is run non-interactively (without a pseudo-terminal or\n pty), so interactive commands such as ``htop`` do not work.\n Use ``ssh my_cluster`` instead.\n\n**Args:**\n task: sky.Task, or sky.Dag (experimental; 1-task only) containing the\n task to execute.\n cluster_name: name of an existing cluster to execute the task.\n dryrun: if True, do not actually execute the task.\n down: Tear down the cluster after all jobs finish (successfully or\n abnormally). If --idle-minutes-to-autostop is also set, the\n cluster will be torn down after the specified idle time.\n Note that if errors occur during provisioning/data syncing/setting\n up, the cluster will not be torn down for debugging purposes.\n backend: backend to use. If None, use the default backend\n (CloudVMRayBackend).\n\n**Returns:**\n The request ID of the exec request.\n\n\n**Request Returns:**\n job_id (Optional[int]): the job ID of the submitted job. None if the\n backend is not CloudVmRayBackend, or no job is submitted to\n the cluster.\n handle (Optional[backends.ResourceHandle]): the handle to the cluster.\n None if dryrun.\n\n**Request Raises:**\n ValueError: if the specified cluster is not in UP status.\n sky.exceptions.ClusterDoesNotExist: if the specified cluster does not\n exist.\n sky.exceptions.NotSupportedError: if the specified cluster is a\n controller that does not support this operation.\n\n### `sky.stop`\n\n```python\nsky.stop(cluster_name: str, purge: bool = False, graceful: bool = False, graceful_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nStops a cluster.\n\nData on attached disks is not lost when a cluster is stopped. Billing for\nthe instances will stop, while the disks will still be charged. Those\ndisks will be reattached when restarting the cluster.\n\nCurrently, spot instance clusters cannot be stopped (except for GCP, which\ndoes allow disk contents to be preserved when stopping spot VMs).\n\n**Args:**\n cluster_name: name of the cluster to stop.\n purge: (Advanced) Forcefully mark the cluster as stopped in SkyPilot's\n cluster table, even if the actual cluster stop operation failed on\n the cloud. WARNING: This flag should only be set sparingly in\n certain manual troubleshooting scenarios; with it set, it is the\n user's responsibility to ensure there are no leaked instances and\n related resources.\n\n**Returns:**\n The request ID of the stop request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n RuntimeError: failed to stop the cluster.\n sky.exceptions.NotSupportedError: if the specified cluster is a spot\n cluster, or a TPU VM Pod cluster, or the managed jobs controller.\n\n### `sky.start`\n\n```python\nsky.start(cluster_name: str, idle_minutes_to_autostop: Optional[int] = None, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, retry_until_up: bool = False, down: bool = False, force: bool = False) -> server_common.RequestId['backends.CloudVmRayResourceHandle']\n```\n\nRestart a cluster.\n\nIf a cluster is previously stopped (status is STOPPED) or failed in\nprovisioning/runtime installation (status is INIT), this function will\nattempt to start the cluster. In the latter case, provisioning and runtime\ninstallation will be retried.\n\nAuto-failover provisioning is not used when restarting a stopped\ncluster. It will be started on the same cloud, region, and zone that were\nchosen before.\n\nIf a cluster is already in the UP status, this function has no effect.\n\n**Args:**\n cluster_name: name of the cluster to start.\n idle_minutes_to_autostop: automatically stop the cluster after this\n many minute of idleness, i.e., no running or pending jobs in the\n cluster's job queue. Idleness gets reset whenever setting-up/\n running/pending jobs are found in the job queue. Setting this\n flag is equivalent to running ``sky.launch()`` and then\n ``sky.autostop(idle_minutes=)``. If not set, the\n cluster will not be autostopped.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes_to_autostop``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after\n ``idle_minutes_to_autostop``.\n retry_until_up: whether to retry launching the cluster until it is\n up.\n down: Autodown the cluster: tear down the cluster after specified\n minutes of idle time after all jobs finish (successfully or\n abnormally). Requires ``idle_minutes_to_autostop`` to be set.\n force: whether to force start the cluster even if it is already up.\n Useful for upgrading SkyPilot runtime.\n\n**Returns:**\n The request ID of the start request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: argument values are invalid: (1) if ``down`` is set to True\n but ``idle_minutes_to_autostop`` is None; (2) if the specified\n cluster is the managed jobs controller, and either\n ``idle_minutes_to_autostop`` is not None or ``down`` is True (omit\n them to use the default autostop settings).\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n sky.exceptions.NotSupportedError: if the cluster to restart was\n launched using a non-default backend that does not support this\n operation.\n sky.exceptions.ClusterOwnerIdentitiesMismatchError: if the cluster to\n restart was launched by a different user.\n\n### `sky.down`\n\n```python\nsky.down(cluster_name: str, purge: bool = False, graceful: bool = False, graceful_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nTears down a cluster.\n\nTearing down a cluster will delete all associated resources (all billing\nstops), and any data on the attached disks will be lost. Accelerators\n(e.g., TPUs) that are part of the cluster will be deleted too.\n\n**Args:**\n cluster_name: name of the cluster to down.\n purge: (Advanced) Forcefully remove the cluster from SkyPilot's cluster\n table, even if the actual cluster termination failed on the cloud.\n WARNING: This flag should only be set sparingly in certain manual\n troubleshooting scenarios; with it set, it is the user's\n responsibility to ensure there are no leaked instances and related\n resources.\n graceful: Cancel the user's task but block until MOUNT_CACHED data is\n fully uploaded. This helps with preserving user data integrity.\n graceful_timeout: If not None, sets a timeout for the graceful option\n above (in seconds).\n\n**Returns:**\n The request ID of the down request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: the specified cluster does not\n exist.\n RuntimeError: failed to tear down the cluster.\n sky.exceptions.NotSupportedError: the specified cluster is the managed\n jobs controller.\n\n### `sky.status`\n\n```python\nsky.status(cluster_names: Optional[List[str]] = None, refresh: common.StatusRefreshMode = common.StatusRefreshMode.NONE, all_users: bool = False, *, _include_credentials: bool = False, _summary_response: bool = False) -> server_common.RequestId[List[responses.StatusResponse]]\n```\n\nGets cluster statuses.\n\nIf cluster_names is given, return those clusters. Otherwise, return all\nclusters.\n\n**Each cluster can have one of the following statuses:**\n\n- ``INIT``: The cluster may be live or down. It can happen in the following\n cases:\n\n - Ongoing provisioning or runtime setup. (A ``sky.launch()`` has started\n but has not completed.)\n - Or, the cluster is in an abnormal state, e.g., some cluster nodes are\n down, or the SkyPilot runtime is unhealthy. (To recover the cluster,\n try ``sky launch`` again on it.)\n\n- ``UP``: Provisioning and runtime setup have succeeded and the cluster is\n live. (The most recent ``sky.launch()`` has completed successfully.)\n\n- ``STOPPED``: The cluster is stopped and the storage is persisted. Use\n ``sky.start()`` to restart the cluster.\n\n**Autostop column:**\n\n- The autostop column indicates how long the cluster will be autostopped\n after minutes of idling (no jobs running). If ``to_down`` is True, the\n cluster will be autodowned, rather than autostopped.\n\nGetting up-to-date cluster statuses:\n\n- In normal cases where clusters are entirely managed by SkyPilot (i.e., no\n manual operations in cloud consoles) and no autostopping is used, the\n table returned by this command will accurately reflect the cluster\n statuses.\n\n- In cases where the clusters are changed outside of SkyPilot (e.g., manual\n operations in cloud consoles; unmanaged spot clusters getting preempted)\n or for autostop-enabled clusters, use ``refresh=True`` to query the\n latest cluster statuses from the cloud providers.\n\n**Args:**\n cluster_names: a list of cluster names to query. If not\n provided, all clusters will be queried.\n refresh: whether to query the latest cluster statuses from the cloud\n provider(s).\n all_users: whether to include all users' clusters. By default, only\n the current user's clusters are included.\n _include_credentials: (internal) whether to include cluster ssh\n credentials in the response (default: False).\n\n**Returns:**\n The request ID of the status request.\n\n**Request Returns:**\n cluster_records (List[Dict[str, Any]]): A list of dicts, with each dict\n containing the information of a cluster. If a cluster is found to be\n terminated or not found, it will be omitted from the returned list.\n\n ```python\n\n {\n 'name': (str) cluster name,\n 'launched_at': (int) timestamp of last launch on this cluster,\n 'handle': (ResourceHandle) an internal handle to the cluster,\n 'last_use': (str) the last command/entrypoint that affected this\n cluster,\n 'status': (sky.ClusterStatus) cluster status,\n 'autostop': (int) idle time before autostop,\n 'to_down': (bool) whether autodown is used instead of autostop,\n 'metadata': (dict) metadata of the cluster,\n 'user_hash': (str) user hash of the cluster owner,\n 'user_name': (str) user name of the cluster owner,\n 'resources_str': (str) the resource string representation of the\n cluster,\n }\n\n### `sky.autostop`\n\n```python\nsky.autostop(cluster_name: str, idle_minutes: int, wait_for: Optional[autostop_lib.AutostopWaitFor] = None, down: bool = False, hook: Optional[str] = None, hook_timeout: Optional[int] = None) -> server_common.RequestId[None]\n```\n\nSchedules an autostop/autodown for a cluster.\n\nAutostop/autodown will automatically stop or teardown a cluster when it\nbecomes idle for a specified duration. Idleness means there are no\nin-progress (pending/running) jobs in a cluster's job queue.\n\nIdleness time of a cluster is reset to zero, whenever:\n\n- A job is submitted (``sky.launch()`` or ``sky.exec()``).\n\n- The cluster has restarted.\n\n- An autostop is set when there is no active setting. (Namely, either\n there's never any autostop setting set, or the previous autostop setting\n was canceled.) This is useful for restarting the autostop timer.\n\nExample: say a cluster without any autostop set has been idle for 1 hour,\nthen an autostop of 30 minutes is set. The cluster will not be immediately\nautostopped. Instead, the idleness timer only starts counting after the\nautostop setting was set.\n\nWhen multiple autostop settings are specified for the same cluster, the\nlast setting takes precedence.\n\n**Args:**\n cluster_name: name of the cluster.\n idle_minutes: the number of minutes of idleness (no pending/running\n jobs) after which the cluster will be stopped automatically. Setting\n to a negative number cancels any autostop/autodown setting.\n wait_for: determines the condition for resetting the idleness timer.\n This option works in conjunction with ``idle_minutes``.\n Choices:\n\n 1. \"jobs_and_ssh\" (default) - Wait for in-progress jobs and SSH\n connections to finish.\n 2. \"jobs\" - Only wait for in-progress jobs.\n 3. \"none\" - Wait for nothing; autostop right after ``idle_minutes``.\n down: if true, use autodown (tear down the cluster; non-restartable),\n rather than autostop (restartable).\n hook: optional script to execute on the remote cluster before autostop.\n The script runs before the cluster is stopped or torn down. If the\n hook fails, autostop will still proceed but a warning will be\n logged.\n hook_timeout: timeout in seconds for hook execution. If None, uses\n DEFAULT_HOOK_TIMEOUT_SECONDS (3600 = 1 hour). The hook will\n be terminated if it exceeds this timeout.\n\n**Returns:**\n The request ID of the autostop request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: if arguments are invalid.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend or the cluster is TPU VM Pod.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.cost_report`\n\n```python\nsky.cost_report(days: Optional[int] = None) -> server_common.RequestId[List[Dict[str, Any]]]\n```\n\nGets all cluster cost reports, including those that have been downed.\n\nThe estimated cost column indicates price for the cluster based on the type\nof resources being used and the duration of use up until the call to\nstatus. This means if the cluster is UP, successive calls to report will\nshow increasing price. The estimated cost is calculated based on the local\ncache of the cluster status, and may not be accurate for the cluster with\nautostop/use_spot set or terminated/stopped on the cloud console.\n\n**Args:**\n days: The number of days to get the cost report for. If not provided,\n the default is 30 days.\n\n**Returns:**\n The request ID of the cost report request.\n\n**Request Returns:**\n cluster_cost_records (List[Dict[str, Any]]): A list of dicts, with each\n dict containing the cost information of a cluster.\n\n ```python\n\n {\n 'name': (str) cluster name,\n 'launched_at': (int) timestamp of last launch on this cluster,\n 'duration': (int) total seconds that cluster was up and running,\n 'last_use': (str) the last command/entrypoint that affected this\n 'num_nodes': (int) number of nodes launched for cluster,\n 'resources': (resources.Resources) type of resource launched,\n 'cluster_hash': (str) unique hash identifying cluster,\n 'usage_intervals': (List[Tuple[int, int]]) cluster usage times,\n 'total_cost': (float) cost given resources and usage intervals,\n }\n\n### `sky.endpoints`\n\n```python\nsky.endpoints(cluster: str, port: Optional[Union[int, str]] = None) -> server_common.RequestId[Dict[int, str]]\n```\n\nGets the endpoint for a given cluster and port number (endpoint).\n\n**Example:**\n ```python\n\n import sky\n request_id = sky.endpoints('test-cluster')\n sky.get(request_id)\n\n\n**Args:**\n cluster: The name of the cluster.\n port: The port number to get the endpoint for. If None, endpoints\n for all ports are returned.\n\n**Returns:**\n The request ID of the endpoints request.\n\n**Request Returns:**\n A dictionary of port numbers to endpoints.\n If port is None, the dictionary contains all\n ports:endpoints exposed on the cluster.\n\n**Request Raises:**\n ValueError: if the cluster is not UP or the endpoint is not exposed.\n RuntimeError: if the cluster has no ports to be exposed or no endpoints\n are exposed yet.\n\n## Job Operations\n\n### `sky.queue`\n\n```python\nsky.queue(cluster_name: str, skip_finished: bool = False, all_users: bool = False) -> server_common.RequestId[List[responses.ClusterJobRecord]]\n```\n\nGets the job queue of a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n skip_finished: if True, skip finished jobs.\n all_users: if True, return jobs from all users.\n\n\n**Returns:**\n The request ID of the queue request.\n\n**Request Returns:**\n job_records (List[responses.ClusterJobRecord]): A list of job records\n for each job in the queue.\n\n ```python\n\n [\n {\n 'job_id': (int) job id,\n 'job_name': (str) job name,\n 'username': (str) username,\n 'user_hash': (str) user hash,\n 'submitted_at': (int) timestamp of submitted,\n 'start_at': (int) timestamp of started,\n 'end_at': (int) timestamp of ended,\n 'resources': (str) resources,\n 'status': (job_lib.JobStatus) job status,\n 'log_path': (str) log path,\n }\n ]\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n ``CloudVmRayBackend``.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n sky.exceptions.CommandError: if failed to get the job queue with ssh.\n\n### `sky.job_status`\n\n```python\nsky.job_status(cluster_name: str, job_ids: Optional[List[int]] = None) -> server_common.RequestId[Dict[Optional[int], Optional['job_lib.JobStatus']]]\n```\n\nGets the status of jobs on a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n job_ids: job ids. If None, get the status of the last job.\n\n**Returns:**\n The request ID of the job status request.\n\n**Request Returns:**\n job_statuses (Dict[Optional[int], Optional[job_lib.JobStatus]]): A\n mapping of job_id to job statuses. The status will be None if the\n job does not exist. If job_ids is None and there is no job on the\n cluster, it will return {None: None}.\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n ``CloudVmRayBackend``.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.cluster_job_spec`\n\n```python\nsky.cluster_job_spec(cluster_name: str, job_id: int) -> server_common.RequestId[Optional[Dict[str, Any]]]\n```\n\nGets the task YAML and entrypoint a cluster job was submitted with.\n\n**Args:**\n cluster_name: name of the cluster.\n job_id: id of the job on the cluster.\n\n**Returns:**\n The request ID of the cluster job spec request.\n\n**Request Returns:**\n spec (Optional[Dict[str, Any]]): None if the cluster does not exist or\n the job's spec was not recorded (e.g. it was submitted by an older\n API server version), otherwise:\n\n ```python\n\n {\n 'task_yaml': (str) the task YAML, with secrets redacted,\n 'entrypoint': (str) the command that submitted the job,\n }\n\n### `sky.cancel`\n\n```python\nsky.cancel(cluster_name: str, all: bool = False, all_users: bool = False, job_ids: Optional[List[int]] = None, _try_cancel_if_cluster_is_init: bool = False) -> server_common.RequestId[None]\n```\n\nCancels jobs on a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n all: if True, cancel all jobs.\n all_users: if True, cancel all jobs from all users.\n job_ids: a list of job IDs to cancel.\n _try_cancel_if_cluster_is_init: (bool) whether to try cancelling the job\n even if the cluster is not UP, but the head node is still alive.\n This is used by the jobs controller to cancel the job when the\n worker node is preempted in the spot cluster.\n\n**Returns:**\n The request ID of the cancel request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: if arguments are invalid.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the specified cluster is a\n controller that does not support this operation.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.tail_logs`\n\n```python\nsky.tail_logs(cluster_name: str, job_id: Optional[int], follow: bool, tail: int = 0, output_stream: Optional['io.TextIOBase'] = None, *, preload_content: bool = True) -> Union[int, Iterator[Optional[str]]]\n```\n\nTails the logs of a job.\n\n**Args:**\n cluster_name: name of the cluster.\n job_id: job id.\n follow: if True, follow the logs. Otherwise, return the logs\n immediately.\n tail: if > 0, tail the last N lines of the logs.\n output_stream: the stream to write the logs to. If None, print to the\n console. Cannot be used with preload_content=False.\n preload_content: if False, returns an Iterator[str | None] containing\n the logs without the function blocking on the retrieval of entire\n log. Iterator returns None when the log has been completely\n streamed. Default True. Cannot be used with output_stream.\n\n**Returns:**\n If preload_content is True:\n Exit code based on success or failure of the job. 0 if success,\n 100 if the job failed. See exceptions.JobExitCode for possible exit\n codes.\n If preload_content is False:\n Iterator[str | None] containing the logs without the function\n blocking on the retrieval of entire log. Iterator returns None\n when the log has been completely streamed.\n\n**Request Raises:**\n ValueError: if arguments are invalid or the cluster is not supported.\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n### `sky.tail_provision_logs`\n\n```python\nsky.tail_provision_logs(cluster_name: str, worker: Optional[int] = None, follow: bool = True, tail: int = 0, output_stream: Optional['io.TextIOBase'] = None) -> int\n```\n\nTails the provisioning logs (provision.log) for a cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n worker: worker id in multi-node cluster.\n If None, stream the logs of the head node.\n follow: follow the logs.\n tail: lines from end to tail.\n output_stream: optional stream to write logs.\n**Returns:**\n Exit code 0 on streaming success; raises on HTTP error.\n\n### `sky.tail_hook_logs`\n\n```python\nsky.tail_hook_logs(cluster_name: str, event: Optional[str] = None, follow: bool = True, tail: int = 0) -> int\n```\n\nTails a per-event lifecycle-hook log on the cluster.\n\n**Args:**\n cluster_name: name of the cluster.\n event: one of ``stop``, ``preemption``, ``down``. When None,\n auto-selects whichever log exists on the cluster.\n follow: whether to follow the logs.\n tail: number of lines to display from the end of the log file.\n\n**Returns:**\n Exit code 0 on streaming success; non-zero on failure.\n\n### `sky.download_logs`\n\n```python\nsky.download_logs(cluster_name: str, job_ids: Optional[List[str]]) -> Dict[str, str]\n```\n\nDownloads the logs of jobs.\n\n**Args:**\n cluster_name: (str) name of the cluster.\n job_ids: (List[str]) job ids.\n\n**Returns:**\n The request ID of the download_logs request.\n\n**Request Returns:**\n job_log_paths (Dict[str, str]): a mapping of job_id to local log path.\n\n**Request Raises:**\n sky.exceptions.ClusterDoesNotExist: if the cluster does not exist.\n sky.exceptions.ClusterNotUpError: if the cluster is not UP.\n sky.exceptions.NotSupportedError: if the cluster is not based on\n CloudVmRayBackend.\n sky.exceptions.ClusterOwnerIdentityMismatchError: if the current user is\n not the same as the user who created the cluster.\n sky.exceptions.CloudUserIdentityError: if we fail to get the current\n user identity.\n\n## Infrastructure & Configuration\n\n### `sky.check`\n\n```python\nsky.check(infra_list: Optional[Tuple[str, ...]], verbose: bool, workspace: Optional[str] = None) -> server_common.RequestId[Dict[str, Dict[str, List[str]]]]\n```\n\nChecks the credentials to enable clouds.\n\n**Args:**\n infra: The infra to check.\n verbose: Whether to show verbose output.\n workspace: The workspace to check. If None, all workspaces will be\n checked.\n\n**Returns:**\n The request ID of the check request.\n\n**Request Returns:**\n Dict mapping workspace name to a dict of cloud name to list of\n enabled capability strings (e.g. 'compute', 'storage').\n\n### `sky.enabled_clouds`\n\n```python\nsky.enabled_clouds(workspace: Optional[str] = None, expand: bool = False) -> server_common.RequestId[List[str]]\n```\n\nGets the enabled clouds.\n\n**Args:**\n workspace: The workspace to get the enabled clouds for. If None, the\n active workspace will be used.\n expand: Whether to expand Kubernetes and SSH to list of resource pools.\n\n**Returns:**\n The request ID of the enabled clouds request.\n\n**Request Returns:**\n A list of enabled clouds in string format.\n\n### `sky.list_accelerators`\n\n```python\nsky.list_accelerators(gpus_only: bool = True, name_filter: Optional[str] = None, region_filter: Optional[str] = None, quantity_filter: Optional[int] = None, clouds: Optional[Union[List[str], str]] = None, all_regions: bool = False, require_price: bool = True, case_sensitive: bool = True) -> server_common.RequestId[Dict[str, List['catalog.common.InstanceTypeInfo']]]\n```\n\nLists the names of all accelerators offered by Sky.\n\nThis will include all accelerators offered by Sky, including those\nthat may not be available in the user's account.\n\n**Args:**\n gpus_only: Whether to only list GPU accelerators.\n name_filter: The name filter.\n region_filter: The region filter.\n quantity_filter: The quantity filter.\n clouds: The clouds to list.\n all_regions: Whether to list all regions.\n require_price: Whether to require price.\n case_sensitive: Whether to case sensitive.\n\n**Returns:**\n The request ID of the list accelerator counts request.\n\n**Request Returns:**\n acc_to_instance_type_dict (Dict[str, List[InstanceTypeInfo]]): A\n dictionary of canonical accelerator names mapped to a list of\n instance type offerings. See usage in cli.py.\n\n### `sky.kubernetes_node_info`\n\n```python\nsky.kubernetes_node_info(context: Optional[str] = None) -> server_common.RequestId['models.KubernetesNodesInfo']\n```\n\nGets the resource information for all the nodes in the cluster.\n\nCurrently only GPU resources are supported. The function returns the total\nnumber of GPUs available on the node and the number of free GPUs on the\nnode.\n\nIf the user does not have sufficient permissions to list pods in all\nnamespaces, the function will return free GPUs as -1.\n\n**Args:**\n context: The Kubernetes context. If None, the default context is used.\n\n**Returns:**\n The request ID of the Kubernetes node info request.\n\n**Request Returns:**\n KubernetesNodesInfo: A model that contains the node info map and other\n information.\n\n### `sky.realtime_kubernetes_gpu_availability`\n\n```python\nsky.realtime_kubernetes_gpu_availability(context: Optional[str] = None, name_filter: Optional[str] = None, quantity_filter: Optional[int] = None, is_ssh: Optional[bool] = None) -> server_common.RequestId[List[Tuple[str, List['models.RealtimeGpuAvailability']]]]\n```\n\nGets the real-time Kubernetes GPU availability.\n\n**Returns:**\n The request ID of the real-time Kubernetes GPU availability request.\n\n### `sky.optimize`\n\n```python\nsky.optimize(dag: 'sky.Dag', minimize: common.OptimizeTarget = common.OptimizeTarget.COST, admin_policy_request_options: Optional[admin_policy.RequestOptions] = None) -> server_common.RequestId['sky.Dag']\n```\n\nFinds the best execution plan for the given DAG.\n\n**Args:**\n dag: the DAG to optimize.\n minimize: whether to minimize cost or time.\n admin_policy_request_options: Request options used for admin policy\n validation. This is only required when a admin policy is in use,\n see: https://docs.skypilot.co/en/latest/cloud-setup/policy.html\n\n**Returns:**\n The request ID of the optimize request.\n\n**Request Returns:**\n optimized_dag (str): The optimized DAG in YAML format.\n\n**Request Raises:**\n exceptions.ResourcesUnavailableError: if no resources are available\n for a task.\n exceptions.NoCloudAccessError: if no public clouds are enabled.\n\n### `sky.validate`\n\n```python\nsky.validate(dag: 'sky.Dag', workdir_only: bool = False, admin_policy_request_options: Optional[admin_policy.RequestOptions] = None, _skip_local_paths: bool = False) -> None\n```\n\nValidates the tasks.\n\nThe file paths (workdir and file_mounts) are validated on the client side\nwhile the rest (e.g. resource) are validated on server side.\n\nRaises exceptions if the DAG is invalid.\n\n**Args:**\n dag: the DAG to validate.\n workdir_only: whether to only validate the workdir. This is used for\n `exec` as it does not need other files/folders in file_mounts.\n admin_policy_request_options: Request options used for admin policy\n validation. This is only required when a admin policy is in use,\n see: https://docs.skypilot.co/en/latest/cloud-setup/policy.html\n _skip_local_paths: Do not check the workdir and file mount sources on\n this machine. For callers whose sources are API server paths\n inside an upload blob; the server resolves the blob and checks\n the paths when it runs the request.\n\n### `sky.reload_config`\n\n```python\nsky.reload_config() -> None\n```\n\nReloads the client-side config.\n\n## Storage & Volumes\n\n### `sky.storage_ls`\n\n```python\nsky.storage_ls() -> server_common.RequestId[List[responses.StorageRecord]]\n```\n\nGets the storages.\n\n**Returns:**\n The request ID of the storage list request.\n\n**Request Returns:**\n storage_records (List[responses.StorageRecord]):\n A list of storage records.\n\n### `sky.storage_delete`\n\n```python\nsky.storage_delete(name: str) -> server_common.RequestId[None]\n```\n\nDeletes a storage.\n\n**Args:**\n name: The name of the storage to delete.\n\n**Returns:**\n The request ID of the storage delete request.\n\n**Request Returns:**\n None\n\n**Request Raises:**\n ValueError: If the storage does not exist.\n\n## API Server\n\n### `sky.api_start`\n\n```python\nsky.api_start(*, deploy: bool = False, host: str = '127.0.0.1', foreground: bool = False, metrics: bool = False, metrics_port: Optional[int] = None, enable_basic_auth: bool = False, port: Optional[int] = None) -> None\n```\n\nStarts the API server.\n\nIt checks the existence of the API server and starts it if it does not\nexist.\n\n**Args:**\n deploy: Whether to deploy the API server, i.e. fully utilize the\n resources of the machine.\n host: The host to bind the API server to. Under deploy, the server\n always binds a wildcard address for remote access: ``::`` when an\n IPv6 host is given, otherwise ``0.0.0.0``. Without deploy the host\n is used as-is.\n foreground: Whether to run the API server in the foreground (run in\n the current process).\n metrics: Whether to export metrics of the API server.\n metrics_port: The port to export metrics of the API server.\n enable_basic_auth: Whether to enable basic authentication\n in the API server.\n port: The port to bind the API server to. Defaults to the\n SKYPILOT_API_SERVER_LOCAL_PORT environment variable, or 46580.\n Other client commands only find a server on a non-default port\n if the same environment variable is exported.\n**Returns:**\n None\n\n### `sky.api_stop`\n\n```python\nsky.api_stop() -> None\n```\n\nStops the API server.\n\nIt will do nothing if the API server is remotely hosted.\n\n**Returns:**\n None\n\n### `sky.api_status`\n\n```python\nsky.api_status(request_ids: Optional[List[Union[server_common.RequestId[T], str]]] = None, all_status: bool = False, limit: Optional[int] = None, fields: Optional[List[str]] = None, cluster_name: Optional[str] = None) -> List[payloads.RequestPayload]\n```\n\nLists all requests.\n\n**Args:**\n request_ids: The prefixes of the request IDs of the requests to query.\n If None, all requests are queried.\n all_status: Whether to list all finished requests as well. This argument\n is ignored if request_ids is not None.\n limit: The number of requests to show. If None, show all requests.\n fields: The fields to get. If None, get all fields except\n return_value and error. This argument is ignored if request_ids\n is not None.\n cluster_name: Filter requests by cluster name.\n If None, show all requests.\n\n**Returns:**\n A list of request payloads.\n\n### `sky.api_info`\n\n```python\nsky.api_info() -> responses.APIHealthResponse\n```\n\nGets the server's status, commit and version.\n\n**Returns:**\n A dictionary containing the server's status, commit and version.\n\n ```python\n\n {\n 'status': 'healthy',\n 'api_version': '1',\n 'commit': 'abc1234567890',\n 'version': '1.0.0',\n 'version_on_disk': '1.0.0',\n 'user': {\n 'name': 'test@example.com',\n 'id': '12345abcd',\n },\n }\n\n Note that user may be None if we are not using an auth proxy.\n\n### `sky.api_cancel`\n\n```python\nsky.api_cancel(request_ids: Optional[Union[server_common.RequestId[T], List[server_common.RequestId[T]], str, List[str]]] = None, all_users: bool = False, silent: bool = False) -> server_common.RequestId[List[str]]\n```\n\nAborts a request or all requests.\n\nWith neither request_ids nor all_users, aborts all of the caller's own\nrequests.\n\n**Args:**\n request_ids: The request ID(s) to abort. Can be a single string or a\n list of strings. On a server with per-user auth, an admin can\n abort any request by id and other users only their own; on a\n server without it, anyone can abort any request by id.\n all_users: Whether to abort all requests from all users. On a server\n with per-user auth, a non-admin aborts only their own. Cannot be\n combined with request_ids.\n silent: Whether to suppress the output.\n\n**Returns:**\n The request ID of the abort request itself.\n\n**Request Returns:**\n A list of request IDs that were cancelled.\n\n**Raises:**\n ValueError: If both request_ids and all_users are given.\n\n### `sky.api_server_logs`\n\n```python\nsky.api_server_logs(follow: bool = True, tail: Optional[int] = None) -> None\n```\n\nStreams the API server logs.\n\n**Args:**\n follow: Whether to follow the logs.\n tail: the number of lines to show from the end of the logs.\n If None, show all logs.\n\n**Returns:**\n None\n\n### `sky.api_login`\n\n```python\nsky.api_login(endpoint: Optional[str] = None, relogin: bool = False, service_account_token: Optional[str] = None, no_browser: bool = False) -> None\n```\n\nLogs into a SkyPilot API server.\n\nThis sets the endpoint globally, i.e., all SkyPilot CLI and SDK calls will\nuse this endpoint.\n\nTo temporarily override the endpoint, use the environment variable\n`SKYPILOT_API_SERVER_ENDPOINT` instead.\n\n**Args:**\n endpoint: The endpoint of the SkyPilot API server, e.g.,\n http://1.2.3.4:46580 or https://skypilot.mydomain.com.\n relogin: Whether to force relogin with OAuth2 when enabled.\n service_account_token: Service account token for authentication.\n no_browser: If True, do not attempt to open a browser locally; print\n the auth URL and let the user open it themselves. Skips the\n localhost-callback flow, which requires a local browser.\n\n**Returns:**\n None\n\n### `sky.api_logout`\n\n```python\nsky.api_logout() -> None\n```\n\nLogout of the API server.\n\nClears all cookies and settings stored in ~/.sky/config.yaml\n\n## Utilities\n\n### `sky.dashboard`\n\n```python\nsky.dashboard(starting_page: Optional[str] = None) -> None\n```\n\nStarts the dashboard for SkyPilot.\n\n### `sky.get`\n\n```python\nsky.get(request_id: server_common.RequestId[T]) -> T\n```\n\nWaits for and gets the result of a request.\n\nThis function will not check the server health since /api/get is typically\nnot the first API call in an SDK session and checking the server health\nmay cause GET /api/get being sent to a restarted API server.\n\n**Args:**\n request_id: The request ID of the request to get. May be a full request\n ID or a prefix.\n\n**Returns:**\n The ``Request Returns`` of the specified request. See the documentation\n of the specific requests above for more details.\n\n**Raises:**\n Exception: It raises the same exceptions as the specific requests,\n see ``Request Raises`` in the documentation of the specific requests\n above.\n\n### `sky.stream_and_get`\n\n```python\nsky.stream_and_get(request_id: Optional[server_common.RequestId[T]] = None, log_path: Optional[str] = None, tail: Optional[int] = None, follow: bool = True, output_stream: Optional['io.TextIOBase'] = None, relay_rich_status: bool = False) -> Optional[T]\n```\n\nStreams the logs of a request or a log file and gets the final result.\n\nThis will block until the request is finished. The request id can be a\nprefix of the full request id.\n\n**Args:**\n request_id: The request ID of the request to stream. May be a full\n request ID or a prefix.\n If None, the latest request submitted to the API server is streamed.\n Using None request_id is not recommended in multi-user environments.\n log_path: The path to the log file to stream.\n tail: The number of lines to show from the end of the logs.\n If None, show all logs.\n follow: Whether to follow the logs.\n output_stream: The output stream to write to. If None, print to the\n console.\n relay_rich_status: If True, forward encoded rich-status control payloads\n verbatim to the output instead of rendering a local spinner. Used by\n the managed jobs controller to preserve provisioning spinner codes\n in its per-job log. See\n :func:`sky.utils.rich_utils.decode_rich_status`.\n\n**Returns:**\n The ``Request Returns`` of the specified request. See the documentation\n of the specific requests above for more details.\n If follow is False, will always return None. See note on\n stream_response.\n\n**Raises:**\n Exception: It raises the same exceptions as the specific requests,\n see ``Request Raises`` in the documentation of the specific requests\n above.\n\n### `sky.workspaces`\n\n```python\nsky.workspaces() -> server_common.RequestId[Dict[str, Any]]\n```\n\nGets the workspaces.\n\n### `sky.set_preferred_workspace`\n\n```python\nsky.set_preferred_workspace(preferred: Optional[str]) -> Dict[str, Any]\n```\n\nSets (or clears with None) the user's preferred workspace.\n\n**Args:**\n preferred: workspace name to set as default, or None to clear.\n\n**Returns:**\n ``{'preferred': }`` echoing what was set. Callers that\n need the resolved workspace + accessible list should follow up\n with :func:`get_user_workspace`. Raises if the server rejects\n the change (workspace does not exist, or user lacks permission\n to it).\n\n### `sky.get_user_workspace`\n\n```python\nsky.get_user_workspace(requested: Optional[str] = None) -> Dict[str, Any]\n```\n\nReturns workspace state for the calling user.\n\nMirrors the launch-path precedence — if the caller has an explicit\n``active_workspace``, the server returns that with ``source='explicit'``;\notherwise the resolver runs (preferred / default-fallback /\nsingle-membership).\n\n**Args:**\n requested: explicit active workspace to ask about. ``None`` (the\n default) — the SDK reads your locally-configured\n ``active_workspace`` (the value `skypilot_config` merges\n from ``~/.sky/config.yaml`` + ``./.sky.yaml`` + any\n ``--config active_workspace=X`` override) and forwards it\n on the wire as ``?requested=``. Pass a non-None value to\n query the resolver as if ``active_workspace`` were that\n value, without changing your local config — useful for\n previewing \"what would land if I switched to X\".\n\n**Returns:**\n ``{workspace, source, note, preferred, accessible}``.\n\n * ``workspace``: the workspace the launch path would pick. Can\n be ``None`` when the resolver couldn't pick (no access /\n ambiguous / explicit ``requested`` rejected by RBAC); the\n reason is then in ``note``.\n * ``source``: one of ``WORKSPACE_SOURCE_*`` on success, ``None``\n when ``workspace`` is ``None``.\n * ``note``: optional message — drift on success\n (``preferred 'team-x' not accessible``) or the resolver error\n when ``workspace`` is ``None``.\n * ``preferred``: the persisted preferred workspace (``None`` if\n unset).\n * ``accessible``: sorted list of workspaces the user can launch\n into.\n\n## Other Functions\n\n### `sky.create_debug_dump`\n\n```python\nsky.create_debug_dump(request_ids: Optional[List[str]] = None, cluster_names: Optional[List[str]] = None, managed_job_ids: Optional[List[int]] = None, recent_minutes: Optional[float] = None) -> server_common.RequestId[str]\n```\n\nCreate a debug dump for troubleshooting.\n\nAt least one of ``request_ids``, ``cluster_names``, ``managed_job_ids``,\nor ``recent_minutes`` must be provided.\n\n**Args:**\n request_ids: List of request IDs or prefixes to include in the\n dump. Prefixes are resolved to all matching request IDs on\n the server.\n cluster_names: List of cluster names to include in the dump.\n managed_job_ids: List of managed job IDs to include in the dump.\n recent_minutes: If specified, include all resources active within\n this many minutes.\n\n**Returns:**\n The request ID of the debug dump creation request.\n\n**Request Returns:**\n Path to the created zip file on the server.\n\n### `sky.download_debug_dump`\n\n```python\nsky.do\n\n\n# References Skill\n\nPath: `agent/skills/skypilot/references/troubleshooting.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/troubleshooting.md)\n\n\n# References Skill\n\nPath: `agent/skills/skypilot/references/yaml-spec.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/agent/skills/skypilot/references/yaml-spec.md)\n\n\n---\nname: skypilot\ndescription: \"Use when launching cloud VMs, Kubernetes pods, or Slurm jobs for GPU/TPU/CPU workloads, training or fine-tuning models on cloud GPUs, deploying inference servers (vllm, TGI, etc.) with autoscaling, writing or debugging SkyPilot task YAML files, using spot/preemptible instances for cost savings, comparing GPU prices across clouds, managing compute across 25+ clouds, Kubernetes, Slurm, and on-prem clusters with failover between them, troubleshooting resource availability or SkyPilot errors, optimizing cost and GPU availability, or migrating an existing Slurm workload (converting sbatch scripts, salloc sessions, job arrays or srun invocations into SkyPilot task YAMLs, and mapping Slurm commands, directives and SLURM_* environment variables to their SkyPilot equivalents).\"\n---\n\n# SkyPilot Skill\n\nSkyPilot is a unified framework to run AI workloads on any cloud, Slurm or Kubernetes. It provides a single interface to launch clusters, run jobs, and serve models across 25+ clouds (AWS, GCP, Azure, Coreweave, Nebius, Lambda, Together AI, RunPod, and more), Kubernetes clusters, and Slurm clusters.\n\n## When to Use SkyPilot\n\n**Use SkyPilot when you need to:**\n- Manage compute resources on any cloud, Slurm, or Kubernetes cluster\n- Launch CPU/GPU/TPU (GB300, GB200, B200, H200, H100, etc.) on any cloud, Kubernetes or Slurm\n- Run training, fine-tuning, or batch inference jobs\n- Serve models with autoscaling and multi-cloud replicas (SkyServe)\n- Run long-running jobs with automatic lifecycle management and recovery (managed jobs)\n- Find the cheapest or most available GPU across clouds\n\n**Don't use SkyPilot for:**\n- Local-only workloads (use Docker/conda directly)\n\n## Capabilities: When to Use What\n\nSkyPilot has three core abstractions. Use the right one for each stage of your workflow:\n\n**1. SkyPilot Clusters** (`sky launch` / `sky exec`) — Interactive development and debugging\n- Use during initial development, debugging, and experimentation\n- Launch a cluster, SSH in or connect VSCode/Cursor (`code --remote ssh-remote+CLUSTER`), iterate quickly\n- Cluster stays up until you stop/down it or autostop triggers\n- Best for: prototyping, debugging, short experiments\n\n**2. Managed Jobs** (`sky jobs launch`) — Long-running training and batch jobs\n- Use when submitting long-running jobs that should run unattended\n- Manages the full lifecycle: provisioning, execution, recovery, and teardown\n- Automatically recovers from spot preemptions, quota limits, and transient failures\n- Works across clouds, Kubernetes, and Slurm (handles preemptions and quota)\n- Best for: training runs, fine-tuning, hyperparameter sweeps, batch inference\n\n**3. SkyServe** (`sky serve up`) — Production model serving\n- Use when serving models at scale with autoscaling\n- Start with `sky launch` + open port to test your serving setup, then use `sky serve up` to scale\n- Provides load balancing, autoscaling, and multi-cloud replicas\n- Best for: model serving endpoints, API services\n\n## Before You Start (Agent Bootstrap)\n\nBootstrap to confirm SkyPilot is installed, connected to an API server, and has cloud credentials. Once confirmed, skip straight to the user's task.\n\n**Step 1: Check installation and API server connectivity**\n\n```bash\nsky api info\n```\n| Output contains | Meaning | Next action |\n|-----------------|---------|-------------|\n| Server version and status | Server is running and connected | **Bootstrap done.** Skip to user's task. |\n| `No SkyPilot API server is connected` | No server connected | Go to \"Start or connect a server\" below. |\n| `Could not connect to SkyPilot API server` | Remote server unreachable or auth expired | Tell the user and suggest `sky api login --relogin -e ` to reconnect. |\n| `command not found: sky` | SkyPilot not installed | Go to \"Install SkyPilot\" below. |\n\n**Install SkyPilot** (only if `sky` command not found):\n```bash\npip install \"skypilot[aws,gcp,kubernetes]\" # Pick clouds the user needs\n```\nAsk the user which clouds they need if unclear, then re-run `sky api info`.\n\n**Start or connect a server** (only if \"not running\"):\n\nAsk the user:\n> Do you have an existing SkyPilot API server to connect to, or should I start one locally?\n\n- **Connect to existing server:** `sky api login -e ` — get the URL from the user.\n- **Start locally:** `sky api start`\n\nAfter either path, re-run `sky api info` to confirm the server is reachable.\n\n**Step 2: Check cloud credentials** (only for fresh setups — skip if the server was already running)\n```bash\nsky check -o json\n```\nThis shows which clouds are enabled or disabled. If the user's target cloud is not enabled, guide them through credential setup (see [Troubleshooting](references/troubleshooting.md#1-installation-and-credentials)).\n\n## Essential Commands\n\nUse `-o json` with status/query commands to get structured JSON output instead of tables.\n\n**Clusters** — interactive development and debugging:\n\n| Command | Description |\n|---------|-------------|\n| `sky launch -c NAME task.yaml` | Launch a cluster or run a task |\n| `sky exec NAME task.yaml` | Run task on existing cluster (skips provisioning); syncs workdir each time |\n| `sky exec NAME task.yaml -d` | Same, but detach immediately (don't stream logs) |\n| `sky status -o json` | Show all clusters |\n| `sky logs NAME` | Stream job logs from a cluster |\n| `sky logs NAME --no-follow` | Print existing logs and exit immediately |\n| `sky logs NAME --tail 50` | Print last 50 lines of logs and exit |\n| `sky logs NAME --status` | Exit with code 0=succeeded, 100=failed, 101=not finished, 102=not found, 103=cancelled |\n| `sky queue NAME -o json` | List jobs on a cluster with status (structured JSON) |\n| `sky stop NAME` / `sky start NAME` | Stop/restart to save costs (preserves disk) |\n| `sky down NAME` | Tear down a cluster completely |\n| `sky gpus list -o json` | List available GPU types across clouds |\n\n**Managed Jobs** — long-running unattended workloads:\n\n| Command | Description |\n|---------|-------------|\n| `sky jobs launch task.yaml` | Launch a managed job (auto lifecycle + recovery) |\n| `sky jobs queue -o json` | Show all managed jobs and their status |\n| `sky jobs logs JOB_ID` | Stream logs from a managed job |\n| `sky jobs cancel JOB_ID` | Cancel a managed job |\n\n**SkyServe** — model serving with autoscaling:\n\n| Command | Description |\n|---------|-------------|\n| `sky serve up serve.yaml -n NAME` | Start a model serving service |\n| `sky serve status NAME` | Show service status and endpoint URL |\n| `sky serve update NAME new.yaml` | Update a running service (rolling) |\n| `sky serve down NAME` | Tear down a service |\n\nFor complete CLI reference, see [CLI Reference](references/cli-reference.md).\n\n## Quick Start\n\n```bash\n# Launch a GPU cluster\nsky launch -c mycluster --gpus H100 -- nvidia-smi\n\n# Run a task from YAML\nsky launch -c mycluster task.yaml\n\n# SSH into cluster\nssh mycluster\n\n# Connect VSCode or Cursor to the cluster for interactive development\ncode --remote ssh-remote+mycluster /home/user/sky_workdir\n# or: cursor --remote ssh-remote+mycluster /home/user/sky_workdir\n\n# Tear down\nsky down mycluster\n```\n\n## Task YAML Structure\n\nThe task YAML is SkyPilot's primary interface. All fields are optional.\n\n```yaml\n# task.yaml\nname: my-training-job\n\n# Local directory to sync to remote ~/sky_workdir\nworkdir: .\n\n# Number of nodes (for distributed training)\nnum_nodes: 1\n\nresources:\n # GPU/TPU accelerators (SkyPilot auto-selects the cheapest cloud/region)\n accelerators: H200:8\n # Optional: pin to a specific cloud/region/infra\n # infra: aws # or aws/us-east-1, k8s, ssh/my-pool\n # If infra is left out, SkyPilot automatically fails over across all\n # enabled clouds/regions to find the cheapest available option.\n # Use spot instances for cost savings\n use_spot: false\n # Disk size in GB\n disk_size: 256\n # Open ports for serving\n ports: 8080\n\n# Environment variables (accessible in file_mounts, setup, and run)\nenvs:\n MODEL_NAME: my-model\n BATCH_SIZE: 32\n\n# Setup: runs once on cluster creation, cached on reuse\nsetup: |\n pip install torch transformers\n\n# Run: the main command\nrun: |\n python train.py --model $MODEL_NAME --batch-size $BATCH_SIZE\n```\n\nFor complete YAML schema including file mounts, environment variables set by SkyPilot, and advanced fields, see [YAML Specification](references/yaml-spec.md).\n\n## GPU and Cloud Selection\n\n**IMPORTANT: Let SkyPilot choose the cloud and region.** Do NOT manually pick a cloud/region/instance by parsing `sky gpus list` output. SkyPilot's optimizer automatically selects the cheapest available option across all enabled clouds. Only specify `infra:` when the user explicitly requests a specific cloud or region.\n\n**Default behavior (recommended):** Just specify the GPU type. SkyPilot finds the cheapest cloud/region automatically:\n\n```yaml\nresources:\n accelerators: H200:8 # SkyPilot picks the cheapest cloud/region with H200:8\n```\n\nIf the user doesn't specify a GPU type, ask them what GPU they need (or what model/workload they're running so you can recommend one). Do NOT run `sky gpus list` and pick for them — present options and let the user decide, or use `any_of` to let SkyPilot maximize availability:\n\n```yaml\n# Let SkyPilot choose from multiple acceptable GPU types (cheapest wins)\nresources:\n any_of:\n - accelerators: H100:8\n - accelerators: A100-80GB:8\n - accelerators: A100:8\n```\n\nUse `ordered` only when the user has a strict preference:\n\n```yaml\n# Try H100 first on AWS, fall back to GCP, then A100\nresources:\n ordered:\n - infra: aws/us-east-1\n accelerators: H100:8\n - infra: gcp/us-central1\n accelerators: H100:8\n - infra: aws/us-west-2\n accelerators: A100-80GB:8\n```\n\nOnly set `infra:` when the user explicitly says something like \"use AWS\" or \"run on GCP us-central1\":\n\n```yaml\nresources:\n infra: aws # User asked for AWS specifically\n accelerators: H100:8\n```\n\n## Cluster Lifecycle\n\n```bash\n# Launch and run a task\nsky launch -c mycluster task.yaml\n\n# Launch with autostop at launch time (preferred: saves cost, no follow-up command needed)\nsky launch -c mycluster task.yaml -i 30 # stop after 30 min idle\nsky launch -c mycluster task.yaml -i 30 --down # tear down after 30 min idle\n\n# Override or pass environment variables via CLI\nsky launch -c mycluster task.yaml --env MODEL_NAME=llama3 --env BATCH_SIZE=64\n\n# Re-run a different task on the same cluster (fast, skips provisioning)\nsky exec mycluster another_task.yaml\n\n# Run an inline command\nsky exec mycluster -- python train.py --epochs 10\n\n# Set autostop after launch (use if you forgot to set -i at launch time)\nsky autostop mycluster -i 30 # stop after 30 min idle, preserving disk (can restart with sky start)\nsky autostop mycluster -i 30 --down # tear down after 30 min idle (disk is deleted, cannot restart)\n\n# Stop to save costs, restart later\nsky stop mycluster\nsky start mycluster\n\n# Tear down completely\nsky down mycluster\n```\n\n## Workdir Sync Behavior\n\n`workdir:` is synced to `~/sky_workdir` on the remote via `rsync` before every `sky exec`. **rsync is additive — deleted local files are NOT removed from the remote.** This can cause experiments to run against stale build artifacts or old configs.\n\nTo ensure a clean slate, SSH and wipe before `sky exec`:\n```bash\nssh mycluster \"rm -rf ~/sky_workdir\"\nsky exec mycluster task.yaml\n```\n\nOr clean inside `run:` if only specific artifacts need removal:\n```yaml\nrun: |\n find ~/sky_workdir/build -name '*.o' -delete 2>/dev/null || true\n cd ~/sky_workdir && make\n```\n\n## Managed Jobs\n\nUse `sky jobs launch` for long-running jobs that should run unattended. SkyPilot manages the full lifecycle — provisioning, execution, recovery from preemptions/quota/failures, and teardown:\n\n```yaml\n# managed-job.yaml\nname: training-job\n\nresources:\n accelerators: A100:8\n\nrun: |\n python train.py --resume-from-checkpoint\n```\n\n```bash\n# Launch as managed job\nsky jobs launch managed-job.yaml\n\n# Check status\nsky jobs queue -o json\n\n# Stream logs\nsky jobs logs \n\n# Cancel\nsky jobs cancel \n```\n\n**Checkpoint pattern**: Your training script should save checkpoints to persistent storage (cloud bucket or volume) and resume from the latest checkpoint on restart. SkyPilot handles the cluster recovery; your script handles the state recovery.\n\n## SkyServe: Model Serving\n\n```yaml\n# serve.yaml\nresources:\n accelerators: A100:1\n ports: 8080\n\nrun: |\n python -m vllm.entrypoints.openai.api_server \\\n --model meta-llama/Llama-3.1-8B-Instruct \\\n --port 8080\n\nservice:\n readiness_probe: /v1/models\n replica_policy:\n min_replicas: 1\n max_replicas: 3\n target_qps_per_replica: 5\n```\n\n```bash\n# Start service\nsky serve up serve.yaml -n my-llm\n\n# Check status / get endpoint\nsky serve status my-llm\nsky serve status my-llm --endpoint\n\n# Update (rolling)\nsky serve update my-llm new-serve.yaml\n\n# Tear down\nsky serve down my-llm\n```\n\n## Common Workflows\n\n### Fine-Tuning Workflow\n1. Write task YAML with `setup` (install deps) and `run` (training command)\n2. Use `file_mounts` or `workdir` to sync code\n3. `sky launch -c train task.yaml` to launch\n4. `sky logs train` to monitor\n5. `sky exec train -- python eval.py` to evaluate on same cluster\n6. `sky down train` when done\n\n### Hyperparameter Sweep\n1. Create parameterized YAML with `envs`\n2. Launch multiple managed jobs:\n ```bash\n for lr in 1e-4 1e-5 1e-6; do\n sky jobs launch sweep.yaml --env LR=$lr --name sweep-lr-$lr\n done\n ```\n3. Monitor with `sky jobs queue -o json`\n\n### Model Serving Deployment\n1. Write serve YAML with `service:` section\n2. `sky serve up serve.yaml -n my-service`\n3. Get endpoint: `sky serve status my-service --endpoint`\n4. Update model: `sky serve update my-service updated.yaml`\n\n### Parallel Experiment Submission\n\nUse `sky exec -d` to submit jobs to multiple VMs without blocking, then collect results:\n\n```bash\n# Submit all experiments (detached, returns after job is queued)\nfor i in 1 2 3 4; do\n sky exec exp-vm-0$i task.yaml --env LR=1e-$i -d\ndone\n\n# Get the latest job ID from a cluster\njob_id=$(sky queue exp-vm-01 -o json \\\n | python3 -c \"import sys, json; jobs = json.load(sys.stdin).get('exp-vm-01', []); print(max(j['job_id'] for j in jobs) if jobs else '')\")\n\n# Wait for a specific job and fetch last 50 lines\nsky logs exp-vm-01 $job_id --status && sky logs exp-vm-01 $job_id --tail 50\n\n# Check all jobs across a cluster at once\nsky queue exp-vm-01 -o json\n```\n\n### Converting a Slurm Workload\n\nWhen the user has an existing `sbatch` script, `salloc` workflow, or asks how a\nSlurm command maps to SkyPilot, **read\n[Migrating from Slurm](references/migrating-from-slurm.md) before writing any\nYAML.** The mapping has non-obvious parts that are easy to get wrong:\n\n- `--time` does **not** map to `autostop` — autostop is unsupported on Slurm.\n Use `config.slurm.sbatch_options.time`.\n- A bare `srun ` should usually be **dropped**, not translated: `run`\n already executes on every node. Only MPI/PMIx launchers keep `srun`, and\n then they need `--overlap` and a rank-0 guard.\n- `--account` / `--qos` / `--exclusive` and other unmodeled directives go\n through `config.slurm.sbatch_options`, which passes them to `sbatch`\n verbatim.\n- `sbatch --array` maps to `sky jobs launch --num-jobs` with\n `$SKYPILOT_JOB_RANK`, not to a shell loop.\n\nValidate the result with `sky launch --dryrun ` — note there is no\n`--dryrun` on `sky jobs launch`, so dry-run via `sky launch` even when the\nfinal command will be `sky jobs launch`.\n\nDo not assume the user wants to leave Slurm. SkyPilot submits to an existing\nSlurm cluster through its login node; migrating *to Kubernetes* is a separate\nquestion, and the constraints differ.\n\n## Agent Feedback Loop\n\nWhen using SkyPilot programmatically, follow this loop:\n\n1. **Validate**: `sky launch --dryrun task.yaml` (check resource availability/cost)\n2. **Launch**: `sky launch -c mycluster task.yaml`\n3. **Monitor**: `sky status -o json` and `sky queue mycluster -o json`\n4. **Wait for completion**: `sky logs mycluster ` (streams logs so you can observe progress and react to stalls; blocks until job finishes; get JOB_ID from `sky queue mycluster -o json`). For long-running jobs where you don't need intermediate output, use `sky logs mycluster --status` instead (blocks silently, exits 0 on success).\n5. **Inspect output**: `sky logs mycluster --no-follow` or `sky logs mycluster --tail 100`\n6. **Debug**: `ssh mycluster` (interactive)\n7. **Iterate**: `sky exec mycluster updated_task.yaml` (run on existing cluster)\n8. **Cleanup**: `sky down mycluster`\n\n> **Never poll with `sleep` + `sky queue`** — use `sky logs CLUSTER JOB_ID` to stream logs and block until done. Use `--status` if you only need the exit code, or `--tail N` to fetch recent output after completion.\n\n## Investigating a Managed Job\n\nWhen the user asks \"what is my job doing\" or \"why is it pending/failing\", run\nthese in order and stop as soon as the question is answered:\n\n1. `sky jobs queue -v -o json` — `status`, `details`, `failure_reason` (`-v` is what adds `details`). For a job that has not started, `details` is the answer: it says what the cluster is waiting on, e.g. a Slurm `QOSGrpGRES` or `Dependency` reason with the partition it is queued in\n2. `sky jobs logs JOB_ID --tail 100 --no-follow` — recent task output; `--controller` for provisioning/recovery logs\n\nSee [Job Investigation](references/job-investigation.md) for how to read `details` and the Slurm pending reasons.\n\n## Common Agent Mistakes\n\n| Mistake | Why it's wrong | Do this instead |\n|---------|---------------|-----------------|\n| Manually picking cloud/region from `sky gpus list` output | SkyPilot optimizer does this automatically and better | Just set `accelerators:` and let SkyPilot choose |\n| Using `sky launch` for long-running unattended jobs | No recovery if preempted or interrupted | Use `sky jobs launch` for unattended work |\n| Forgetting `sky down` or autostop after work is done | Wastes money on idle clusters | Always clean up, or use `-i --down` at launch |\n| Hardcoding `infra: aws` without user asking | Limits availability and increases cost | Only set `infra:` when user explicitly requests a cloud |\n| Not using `envs:` for configurable values | Hard to reuse or override from CLI | Use `envs:` in YAML + `--env KEY=VAL` for parameterization |\n| Running `sky launch` without `-c ` | Creates randomly-named cluster, hard to reference | Always name clusters with `-c` |\n| Parsing table output from status commands | Table formatting is for humans, fragile to parse | Use `-o json` for structured output |\n| Using deprecated `cloud:`/`region:`/`zone:` fields | Deprecated in favor of `infra:` | Use `infra: aws/us-east-1` instead |\n| Polling job status with `sleep` + `sky queue` | Wastes tokens, introduces timing bugs, fragile | Use `sky logs CLUSTER JOB_ID --status` to block until done |\n| Assuming workdir sync removes remote files | rsync is additive; old remote files persist across `sky exec` calls | SSH and manually clean `~/sky_workdir`, or clean in `run:` script |\n| Not using `--tail` when only last output matters | Streaming full logs wastes tokens for long jobs | Use `sky logs CLUSTER JOB_ID --tail 50` for last N lines |\n\n## Common Issues Quick Reference\n\n| Issue | Solution |\n|-------|----------|\n| GPU not available | Use `any_of` for fallback, or try different regions/clouds |\n| Setup takes too long | SkyPilot caches setup; use `sky exec` to skip it on reruns |\n| Task fails silently | Check `sky logs ` or `ssh ` to debug |\n| Cluster stuck in INIT | `sky down ` and relaunch |\n| Preemption/quota | Use `sky jobs launch` for automatic recovery and lifecycle management |\n| Port not accessible | Ensure `ports:` is set in resources and security groups allow traffic |\n| File sync slow | Use cloud bucket mounts instead of `workdir` for large datasets |\n| Credentials error | Run `sky check -o json` and inspect which clouds are disabled |\n\n## References\n\nFor detailed reference documentation:\n\n- [CLI Reference](references/cli-reference.md) — All commands and flags\n- [YAML Specification](references/yaml-spec.md) — Complete task YAML schema, file mounts, environment variables\n- [Python SDK](references/python-sdk.md) — Programmatic API and SDK usage\n- [Advanced Patterns](references/advanced-patterns.md) — Multi-cloud, distributed training, production patterns\n- [Migrating from Slurm](references/migrating-from-slurm.md) — Converting `sbatch` scripts to task YAMLs, Slurm command/env-var mapping, Slurm-specific constraints\n- [Job Investigation](references/job-investigation.md) — Which command answers \"what is my job doing and why\"; reading `details`, events, Slurm pending reasons\n- [Troubleshooting](references/troubleshooting.md) — Error diagnosis and solutions\n- [Examples](references/examples.md) — Copy-paste task YAML examples\n\n\n# CLAUDE.md - SkyPilot Development Guide\n\nThis document provides guidance for AI assistants working with the SkyPilot codebase.\n\n## Project Overview\n\nSkyPilot is a system to run, manage, and scale AI workloads on any AI infrastructure. It provides a unified interface across 25+ cloud providers (AWS, GCP, Azure, Kubernetes, Slurm, and many others), enabling users to launch compute resources, run jobs, and serve models without vendor lock-in.\n\n**Key capabilities:**\n- **Clusters**: Launch and manage compute clusters across clouds and Kubernetes\n- **Managed jobs**: Run jobs with automatic recovery from preemptions and failures\n- **Multi-cloud and multi-Kubernetes**: Unified interface across 25+ clouds and multiple K8s clusters\n- Cost optimization and GPU availability maximization\n\n## Repository Structure\n\n```\nskypilot/\n├── sky/ # Main source code\n│ ├── __init__.py # Package exports and version\n│ ├── core.py # Core orchestration logic\n│ ├── task.py # Task definition and YAML parsing\n│ ├── resources.py # Cloud resource specifications\n│ ├── optimizer.py # Resource allocation optimizer\n│ ├── execution.py # Task execution pipeline\n│ ├── exceptions.py # Custom exceptions\n│ ├── skypilot_config.py # Configuration management\n│ ├── clouds/ # Cloud provider abstractions (25+ providers)\n│ ├── backends/ # Execution backends (Ray-based VM, Docker)\n│ ├── provision/ # Cloud resource provisioning\n│ ├── jobs/ # Managed job lifecycle\n│ ├── serve/ # Model serving (SkyServe)\n│ ├── skylet/ # On-cluster execution agent\n│ ├── client/ # SDK and CLI\n│ ├── server/ # API server and dashboard backend\n│ ├── dashboard/ # Web UI (Next.js)\n│ ├── utils/ # 50+ utility modules\n│ ├── catalog/ # Cloud pricing and instance catalogs\n│ ├── data/ # Storage and data handling\n│ ├── adaptors/ # Cloud-specific metadata adapters\n│ └── schemas/ # Protobuf definitions and API schemas\n├── tests/ # Test suite\n│ ├── unit_tests/ # Unit tests with subdirectories per module\n│ ├── smoke_tests/ # Quick validation tests\n│ ├── integration_tests/ # End-to-end tests\n│ ├── kubernetes/ # K8s-specific tests\n│ └── conftest.py # Pytest fixtures\n├── examples/ # 50+ usage examples\n├── llm/ # 45+ LLM training/serving examples\n├── docs/ # Sphinx documentation\n├── charts/ # Helm charts for K8s deployment\n└── format.sh # Code formatting script\n```\n\n## Development Setup\n\n### Environment Setup\n\n```bash\n# Create virtual environment with uv (Python 3.8-3.11 supported)\n# --seed is required to ensure pip is installed (needed for building wheels)\nuv venv --seed --python 3.11\nsource .venv/bin/activate\n\n# Install in editable mode with all cloud support\nuv pip install -e \".[all]\"\n# Or specific clouds only:\n# uv pip install -e \".[aws,gcp,kubernetes]\"\n\n# Install development dependencies\nuv pip install -r requirements-dev.txt\n\n# Optional: Install pre-commit hooks\nuv pip install pre-commit\npre-commit install\n```\n\n### Environment Variables\n\n```bash\nexport SKYPILOT_DEV=1 # Enable development mode\nexport SKYPILOT_DEBUG=1 # Enable debug logging\n```\n\n## Code Formatting and Linting\n\n**Always run `format.sh` before committing:**\n\n```bash\nbash format.sh # Format changed files (vs origin/master)\nbash format.sh --all # Format entire codebase\nbash format.sh --files path/to/file.py # Format specific files\n```\n\nThe script runs:\n1. **YAPF** - Google style for all Python code\n2. **isort** - Import sorting (Google profile)\n3. **mypy** - Type checking\n4. **pylint** - Linting with custom rules\n\n### Tool Versions (must match exactly)\n\nFrom `requirements-dev.txt`:\n- yapf==0.32.0\n- pylint==2.14.5\n- mypy==1.19.1\n- isort==5.12.0\n- pylint-quotes==0.2.3\n\n### Excluded from Formatting\n\n- `sky/schemas/generated/` - Auto-generated protobuf files\n- `build/` - Build artifacts\n\n## Testing\n\n### Running Tests\n\n```bash\n# Unit tests (fast, no cloud resources)\npytest tests/unit_tests/\n\n# Specific test file\npytest tests/unit_tests/test_resources.py\n```\n\n### CI Tests via PR Comments\n\nTrigger CI tests on pull requests using comments:\n- `/quicktest-core` - Run quick core tests\n- `/smoke-test` - Run smoke tests (launches cloud clusters)\n- `/smoke-test --kubernetes --postgres` - Test with PostgreSQL backend on Kubernetes\n- `/smoke-test --kubernetes --remote-server --postgres` - Test remote API server with PostgreSQL\n\n### Test Configuration\n\nFrom `pyproject.toml`:\n- Uses pytest-xdist with 16 parallel workers\n- Environment: `SKYPILOT_DEBUG=1`, `SKYPILOT_DEV=1`\n- Buildkite integration for CI\n\n### Checking Buildkite CI Status\n\nTo check CI test results for a PR:\n\n1. **Get Buildkite URL from GitHub**: Check the PR's commit status checks to find the Buildkite build URL (may take a moment to appear after triggering)\n2. **Fetch build logs via Buildkite API**: Use the Buildkite API to retrieve detailed logs and test results\n\n```bash\n# Example: Get build details (requires BUILDKITE_TOKEN)\ncurl -H \"Authorization: Bearer $BUILDKITE_TOKEN\" \\\n \"https://api.buildkite.com/v2/organizations/skypilot/pipelines/skypilot/builds/\"\n```\n\n## Code Style Guidelines\n\n### General Principles\n\n- Follow [Google Python Style Guide](https://google.github.io/styleguide/pyguide.html)\n- Use f-strings over `.format()` for readability\n- Use `class MyClass:` not `class MyClass(object):`\n- Use `abc` module for abstract classes\n- Use Python typing with `if typing.TYPE_CHECKING:` for type-only imports\n- Always place imports at the top of the file, not inside function definitions. If this causes a circular dependency, try to resolve it (e.g., restructure modules or use `if typing.TYPE_CHECKING:`). Only as a last resort, place the import inside the function with a comment explaining why. Import performance is not a valid reason for in-function imports; use `LazyImport` instead (see below).\n\n### TODOs and FIXMEs\n\nAlways include author attribution:\n```python\n# TODO(username): Description of what needs to be done\n# FIXME(username): Description of what needs fixing\n```\n\n### Exceptions vs Assertions\n\n- Use exceptions for errors (better error messages)\n- Use `assert` only for debugging/proof-checking\n\n### Lazy Imports\n\nFor modules with significant import time (>100ms) that are imported during `import sky`, use `LazyImport` from `sky/adaptors/common.py`:\n\n```python\nfrom sky.adaptors.common import LazyImport\nheavy_module = LazyImport('heavy_module')\n```\n\nMeasure import time with:\n```bash\npython -X importtime -c \"import sky\" 2> import.log\ntuna import.log # pip install tuna\n```\n\n## Architecture Patterns\n\n### Cloud Provider Abstraction\n\nAll cloud providers inherit from `sky.clouds.Cloud` (in `sky/clouds/cloud.py`). Cloud objects are **lightweight and stateless**. Key design principles:\n\n- Methods should be inexpensive to call\n- Don't store heavy state in cloud objects\n- Cache cloud-specific queries in `sky/clouds/utils/` modules\n- Each cloud implements feature flags via `CloudImplementationFeatures`\n\nAdding a new cloud provider:\n1. Create `sky/clouds/.py` inheriting from `Cloud`\n2. Implement required abstract methods\n3. Add provisioning logic in `sky/provision//`\n4. Register in `sky/clouds/__init__.py`\n\n### Client-Server Architecture\n\nSkyPilot uses a client-server model with API versioning:\n\n- Client (SDK/CLI) in `sky/client/`\n- Server in `sky/server/`\n- API version in `sky/server/constants.py`\n\n**Backward Compatibility Rules (from v0.10.0+):**\n- Changes must be backward compatible\n- Bump `API_VERSION` when introducing API changes\n- Use `@versions.minimal_api_version(N)` decorator for new SDK methods\n- Handle version differences with `versions.get_remote_api_version()`\n\n### Protobuf Regeneration\n\nWhen modifying `.proto` files in `sky/schemas/proto/`:\n\n```bash\npython -m grpc_tools.protoc \\\n --proto_path=sky/schemas/generated=sky/schemas/proto \\\n --python_out=. \\\n --grpc_python_out=. \\\n --pyi_out=. \\\n sky/schemas/proto/*.proto\n```\n\n### Dependency Management\n\nDependencies are defined in `sky/setup_files/dependencies.py`:\n\n- **Core dependencies**: Listed in `install_requires`\n- **Cloud-specific**: Defined in `extras_require` (e.g., `aws`, `gcp`, `kubernetes`)\n- **Development**: Listed in `requirements-dev.txt`\n\nWhen updating dependencies:\n\n1. Check version constraints carefully - some packages have breaking changes\n2. Consider Python version compatibility (3.8-3.11)\n3. Test with both minimum and latest allowed versions\n4. Document version constraints with comments explaining why\n\n## Key Modules Reference\n\n| Module | Purpose |\n|--------|---------|\n| `sky/core.py` | Core orchestration (launch, exec, stop, down) |\n| `sky/task.py` | Task YAML parsing and specification |\n| `sky/resources.py` | Resource requirements (GPUs, memory, disk) |\n| `sky/optimizer.py` | Cloud selection and cost optimization |\n| `sky/client/sdk.py` | Python SDK implementation |\n| `sky/client/cli/` | CLI commands |\n| `sky/backends/cloud_vm_ray_backend.py` | Main execution backend for clusters |\n| `sky/provision/provisioner.py` | Resource provisioning |\n| `sky/jobs/` | Managed jobs with recovery and scheduling |\n\n## Pull Request Guidelines\n\n1. **Branch from master**, create descriptive branch name\n2. **Run `format.sh`** before committing\n3. **Add tests** for core system changes\n4. **Run smoke tests** for significant changes\n5. **Include `Tested:` section** in PR description with test plan\n6. **Delete branch** after merging\n\n### PR Description Format\n\nPRs should include:\n- **Summary**: Brief description of changes (1-3 bullet points)\n- **Test plan**: How the changes were tested (commands run, manual verification steps)\n\n**Important**: Always generate a manual test plan describing how to verify the changes work correctly. Include specific commands, expected outputs, or UI verification steps. Whenever possible, add unit tests and smoke tests for the changes.\n\n### Commit Message Format\n\nUse the `[Area] Description` format:\n- `[Core]` for core system changes\n- `[CLI]` for CLI changes\n- `[API]` for API server changes\n- `[Docs]` for documentation\n- `[AWS]`, `[GCP]`, `[Azure]`, `[Kubernetes]` for cloud-specific changes\n- `[Jobs]` for managed jobs\n- `[Dashboard]` for web UI changes\n- `[Serve]` for model serving (SkyServe)\n- `[Test]` for testing changes\n- `[CI]` for CI/CD changes\n\nExamples:\n- `[Core] Fix cluster status refresh logic`\n- `[AWS] Add support for new instance types`\n- `[Docs] Update installation guide`\n\n## API Server Testing\n\n### Local API Server (Recommended for Development)\n\n```bash\n# Always restart API server after code changes to pick up changes\nsky api stop\nsky api start\n\n# Verify server is running\nsky api status\n```\n\n### Dashboard Development\n\n**For local API server development**, rebuild the dashboard before restarting:\n\n```bash\n# Install dependencies (first time or after package.json changes)\nnpm --prefix sky/dashboard install\n\n# Rebuild the dashboard\nnpm --prefix sky/dashboard run build\n\n# Then restart the API server\nsky api stop\nsky api start\n```\n\n**For remote API server (Docker/Kubernetes)**, the Dockerfile automatically builds the dashboard - no manual build needed before `docker build`.\n\nThe dashboard is a Next.js application. For development with hot reloading:\n\n```bash\n# Run dashboard in development mode (separate from API server)\ncd sky/dashboard\nnpm run dev\n```\n\n### Mocking Remote API Server Locally\n\nTo test remote API server behavior locally:\n\n```bash\n# Start local API server (runs on port 46580 by default)\nsky api stop\nsky api start\n\n# Forward to a different port to simulate remote server\nsocat TCP-LISTEN:46590,fork TCP:127.0.0.1:46580 &\n\n# Connect to the forwarded port as if it were a remote server\nsky api login -e http://127.0.0.1:46590\n```\n\n### Remote API Server (Kubernetes Deployment)\n\nFor testing with a remote API server on Kubernetes:\n\n```bash\n# Build local changes\nhelm repo add skypilot https://helm.skypilot.co\nhelm repo update\nhelm dependency build ./charts/skypilot\n\n# Build Docker image\nDOCKER_IMAGE=my-repo/skypilot:v1\ndocker buildx build --push --platform linux/amd64 -t $DOCKER_IMAGE -f Dockerfile .\n\n# Deploy (NEW installation)\nNAMESPACE=skypilot\nRELEASE_NAME=skypilot\nhelm upgrade --install $RELEASE_NAME ./charts/skypilot --devel \\\n --namespace $NAMESPACE \\\n --create-namespace \\\n --set apiService.image=$DOCKER_IMAGE\n```\n\n#### Upgrading Existing Deployments\n\n**CRITICAL:** Always use `--reuse-values` to preserve database/credential config:\n\n```bash\n# Upgrade existing deployment (keeps PostgreSQL, auth, etc.)\nhelm upgrade skypilot ./charts/skypilot -n skypilot --reuse-values \\\n --set apiService.image=$DOCKER_IMAGE\n\n# Check current values / rollback if needed\nhelm get values skypilot -n skypilot\nhelm rollback skypilot -n skypilot\n```\n\n#### PostgreSQL Backend\n\n```bash\n# Create connection secret\nkubectl create secret generic db-uri -n skypilot \\\n --from-literal=uri=\"postgresql://user:pass@host:5432/db\"\n\n# Deploy with PostgreSQL\nhelm upgrade --install skypilot ./charts/skypilot -n skypilot \\\n --set apiService.dbConnectionSecretName=db-uri \\\n --set storage.enabled=false\n```\n\n## Critical Code Paths (Handle with Care)\n\nThe following modules contain complex, stateful logic that requires careful review when modifying:\n\n**Managed Jobs Recovery:**\n- `sky/jobs/controller.py` - Job lifecycle management, state transitions, async coordination\n- `sky/jobs/recovery_strategy.py` - Preemption recovery, retry logic for managed jobs\n- `sky/backends/cloud_vm_ray_backend.py` - Execution backend with complex state handling\n\nThese modules handle edge cases like preemption during job submission, controller failures mid-recovery, and race conditions between concurrent operations. Changes can have subtle effects on job reliability.\n\n**API Server Performance & Robustness:**\n- `sky/server/` - API server with memory efficiency, low latency requirements\n- `sky/backends/backend_utils.py` - Cluster status caching, network resilience, SSH handling\n\nThese modules are optimized for performance. Be cautious about adding blocking calls, memory-heavy operations, or changing caching behavior.\n\n**CLI/SDK Interface Design:**\n- `sky/client/cli/` - Command-line interface\n- `sky/client/sdk.py` - Python SDK\n\nBe cautious about adding new interfaces or changing existing UX significantly. Keep the interface clean and minimal for both CLI and SDK.\n\n**Backward Compatibility:**\n- Test with old server + new client, and old client + new server\n- Ensure existing clusters and jobs continue working after server upgrades\n- `/quicktest-core` tests some backward compatibility scenarios\n- See `docs/source/developers/CONTRIBUTING.md` for API versioning guidelines\n\n## Common Pitfalls\n\n1. **Always restart API server after code changes** - Run `sky api stop; sky api start` to pick up changes\n2. **Don't modify `sky/schemas/generated/`** - These are auto-generated\n3. **Match formatter versions exactly** - Version mismatches cause CI failures\n4. **Consider import time** - Heavy imports slow down CLI responsiveness\n5. **API versioning** - Always maintain backward compatibility\n\n## Useful Commands\n\n```bash\n# Profile CLI performance\nuv pip install py-spy\npy-spy record -t -o sky.svg -- python -m sky.cli status\n\n# Check cloud credentials\nsky check\n\n# View cluster status\nsky status\n\n# Launch a cluster\nsky launch \n\n# Execute a command on the cluster\nsky exec -- bash\n\n# SSH into the cluster\nssh \n\n# Launch a managed job\nsky jobs launch \n```\n\n## Documentation\n\n- **User docs**: https://docs.skypilot.co/\n- **Source**: `docs/source/` (Sphinx)\n- **Build docs**: See `.readthedocs.yml`\n\n## Additional Resources\n\n- **Full contributing guide**: `docs/source/developers/CONTRIBUTING.md`\n- **User docs**: https://docs.skypilot.co/\n- **GitHub Issues**: https://github.com/skypilot-org/skypilot/issues\n- **Discussions**: https://github.com/skypilot-org/skypilot/discussions\n- **Slack**: http://slack.skypilot.co\n\n\n{\n \"name\": \"skypilot\",\n \"owner\": {\n \"name\": \"SkyPilot Team\"\n },\n \"metadata\": {\n \"description\": \"Official SkyPilot skills for AI coding agents\",\n \"version\": \"1.0.0\"\n },\n \"plugins\": [\n {\n \"name\": \"skypilot\",\n \"source\": \"./agent\",\n \"description\": \"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds\",\n \"version\": \"1.0.0\",\n \"author\": {\n \"name\": \"SkyPilot Team\"\n },\n \"homepage\": \"https://docs.skypilot.co/\",\n \"repository\": \"https://github.com/skypilot-org/skypilot\",\n \"license\": \"Apache-2.0\",\n \"keywords\": [\"AI Infrastructure\", \"Multi-Cloud\", \"GPU\", \"Kubernetes\", \"Slurm\", \"Training\", \"Serving\", \"Cost Optimization\", \"SkyPilot\"]\n }\n ]\n}\n\n\n{\n \"name\": \"skypilot\",\n \"description\": \"SkyPilot agent skill for launching cloud VMs, Kubernetes pods, and Slurm jobs across 25+ clouds\",\n \"version\": \"1.0.0\",\n \"author\": {\n \"name\": \"SkyPilot Team\"\n },\n \"homepage\": \"https://docs.skypilot.co/\",\n \"repository\": \"https://github.com/skypilot-org/skypilot\",\n \"license\": \"Apache-2.0\",\n \"keywords\": [\"AI Infrastructure\", \"Multi-Cloud\", \"GPU\", \"Kubernetes\", \"Slurm\", \"Training\", \"Serving\", \"Cost Optimization\", \"SkyPilot\"]\n}\n\n\n# Subagent: autonomous-code-optimization\n\nPath: `docs/source/examples/agents/autonomous-code-optimization.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autonomous-code-optimization.md)\n\n\n# Subagent: autoresearch\n\nPath: `docs/source/examples/agents/autoresearch.md`\n\n[View Raw Content on GitHub](https://raw.githubusercontent.com/skypilot-org/skypilot/HEAD/docs/source/examples/agents/autoresearch.md)\n"}