{"owner":"e2b-dev","repo":"infra","hasSkills":true,"totalSkillsCount":13,"totalTokensCount":21991,"categories":["root-instruction","claude-rule","project-spec","plugin-manifest"],"hasMcp":false,"mcpConfig":null,"found":["AGENTS.md","CLAUDE.md","docs/ARCHITECTURE.md","packages/api/internal/sandbox/reservations/redis/README.md","packages/envd/README.md","packages/envd/internal/services/legacy/readme.md","packages/nomad-nodepool-apm/README.md","packages/orchestrator/README.md","packages/orchestrator/cmd/simulate-gcs-traffic/readme.md","packages/shared/pkg/apiid/README.md","packages/shared/pkg/featureflags/README.md","packages/shared/pkg/id/README.md","packages/shared/pkg/sandboxtypes/README.md"],"skills":{"AGENTS.md":"# AGENTS.md\n\nInstructions for AI agents working in this repository.\n\nAlso read [CLAUDE.md](CLAUDE.md) for development commands and package\nconventions if it is not already in context. It applies to all agents.\n\n## Understanding the repository\n\nRead [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) before working on or reviewing code here. It describes what each service does, how services interact (sandbox creation, traffic routing, pause/resume, template builds), the data stores, and the deployment topology — it is the fastest way to build correct context about this codebase.\n\nKeep that document updated: if a code change alters anything it describes (service responsibilities, ports, protocols, data stores, flows, deployment topology), update `docs/ARCHITECTURE.md` as part of the same change. When reviewing a PR that changes such details without updating the document, flag it.\n\n## Review guidelines (pull requests)\n\nOutput style rules (strict, override any defaults):\n\n- Only report concrete bugs, regressions, or correctness issues. Do not summarize the PR, the diff, or what changed.\n- One short paragraph per finding. No preamble, no closing remark.\n- No headers, no bullet lists, no tables, no diagrams.\n- No emojis. No severity tags (\"bug\", \"nit\", \"suggestion\", \"enhancement\"). No labels.\n- No branding or footer lines.\n- If there are no real issues to flag, post no review at all.\n\n## Scope\n\n- Skip style/nit comments — `golangci-lint` covers those.\n- Skip test-coverage comments.\n- Focus on: race conditions, nil-deref, error handling, auth/authz, request routing, resource leaks, SQL/migration correctness, and gRPC/proto compatibility.\n","CLAUDE.md":"# CLAUDE.md\n\nThis development guide applies to Codex, Claude Code, and OpenCode.\nRepository and review rules are shared through:\n\n@AGENTS.md\n\n## Project Overview\n\nE2B Infrastructure is the backend infrastructure powering E2B (e2b.dev), an open-source cloud platform for AI code interpreting. It provides sandboxed execution environments using Firecracker microVMs.\n\n**Start with [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** — it explains what each service does, how services interact (with diagrams of the core flows: sandbox creation, traffic routing, pause/resume, template builds), and the deployment topology. Reading it first is the fastest way to understand this repository.\n\n**Keep docs/ARCHITECTURE.md updated**: if your change alters anything it describes (service responsibilities, ports, protocols, data stores, flows, deployment topology), update the document in the same change.\n\n## Common Development Commands\n\n### Setup & Environment\n```bash\n# Switch between environments (prod, staging, dev)\nmake switch-env ENV=staging\n\n# Setup local development stack (PostgreSQL, Redis, ClickHouse, monitoring)\nmake local-infra\n```\n\n### Building & Testing\n```bash\n# Run all unit tests across packages\nmake test\n\n# Run integration tests\nmake test-integration\n\n# Build specific package\nmake build/api\nmake build/orchestrator\n\n# Generate code (proto, SQL, OpenAPI)\nmake generate\n\n# Format and lint code\nmake fmt\nmake lint\n\n# Regenerate mocks\nmake generate-mocks\n\n# Tidy go dependencies\nmake tidy\n```\n\n### Running Services Locally\n```bash\n# From packages/api/\nmake run-local # Run API server on :3000\nmake dev # Run with air (hot reload)\n\n# From packages/orchestrator/\nmake run-local # Run orchestrator\nmake run-debug # Run with race detector\n```\n\n### Package-Specific Commands\n```bash\n# API: Generate OpenAPI code\ncd packages/api && make generate\n\n# Orchestrator: Generate proto + OpenAPI\ncd packages/orchestrator && make generate\n\n# DB: Run migrations\nmake migrate\n\n# Run single test\ncd packages/ && go test -v -run TestName ./path/to/package\n```\n\n### Publishing Artifacts\n```bash\n# Build and upload all service images to your GCP project\nmake build-and-upload\n\n# Build specific service\nmake build-and-upload/api\nmake build-and-upload/orchestrator\n```\n\n## Architecture Overview\n\n### Service Communication Flow\n```\nClient → Client-Proxy → API (REST) ⟷ PostgreSQL\n ↓ ⟷ Redis\n Orchestrator ⟷ ClickHouse\n ↓ (gRPC)\n Firecracker VMs\n ↓\n Envd (in-VM daemon)\n```\n\n### Core Services\n\n**API (`packages/api/`)** - REST API using Gin framework\n- Entry point: `main.go`\n- Core logic: `internal/handlers/store.go` (APIStore)\n- Authentication: API keys and OIDC auth provider JWTs\n- OpenAPI code generation: `internal/api/*.gen.go`\n- Port: 80\n\n**Dashboard API (`packages/dashboard-api/`)** - REST backend for the web dashboard, not the SDK\n- Entry point: `main.go`\n- Spec: `spec/openapi-dashboard.yml`; template tags, `/v1/management`\n- Talks to Postgres and ClickHouse; never to orchestrators (see `docs/ARCHITECTURE.md`)\n- Port: 3010\n\n**Orchestrator (`packages/orchestrator/`)** - Firecracker microVM orchestration\n- Entry point: `main.go`\n- VM management: `pkg/sandbox/`\n- Firecracker integration: `pkg/sandbox/fc/`\n- Networking: `pkg/sandbox/network/`\n- Storage: `pkg/sandbox/nbd/` (Network Block Device)\n- Template caching: `pkg/sandbox/template/`\n- Template building: `pkg/template/`\n- gRPC server: `pkg/server/`\n- Utilities: `cmd/clean-nfs-cache/`, `cmd/inspect-build/`, `cmd/dummy-orchestrator/`\n\n**Envd (`packages/envd/`)** - In-VM daemon using Connect RPC\n- Runs inside each Firecracker VM\n- Process management API: `packages/envd/spec/process/process.proto`\n- Filesystem API: `packages/envd/spec/filesystem/filesystem.proto`\n- Port: 49983\n- **`pkg/version.go` is release-please's**: it is set from the Conventional Commits merged since the last envd release, so do not bump it by hand. A behavioral change ships under a `feat` or `fix` type so a version exists for the orchestrator to gate on; comment or docs-only changes use `docs`, which the changelog hides, and cut no release (`chore` and `refactor` are visible and cut one)\n\n**Client Proxy (`packages/client-proxy/`)** - Edge routing layer\n- Service discovery via `packages/shared/pkg/servicediscovery`\n- Request routing to orchestrators\n- Redis-backed state management\n\n**Shared (`packages/shared/`)** - Common utilities\n- Proto definitions: `pkg/grpc/orchestrator/`, `pkg/grpc/envd/`\n- Telemetry: `pkg/telemetry/` (OpenTelemetry)\n- Logging: `pkg/logger/` (Zap + OTEL)\n- Database: `pkg/db/` (ent ORM)\n- Models: `pkg/models/`\n- Storage: `pkg/storage/` (GCS/S3 clients)\n- Feature flags: `pkg/featureflags/` (LaunchDarkly)\n\n**Database (`packages/db/`)** - PostgreSQL layer\n- Migrations: `migrations/*.sql` (goose)\n- Queries: `queries/*.sql` (sqlc)\n- Generated code: `queries/` (plus `pkg/auth/queries/`, `pkg/dashboard/queries/`)\n\n### Key Technologies\n\n- **go 1.26.10** with workspaces (`go.work`)\n- **Firecracker** for microVM virtualization\n- **PostgreSQL** for primary data (sqlc for queries)\n- **ClickHouse** for analytics\n- **Redis** for caching and state\n- **OpenTelemetry** for observability (Grafana stack: Loki, Tempo, Mimir)\n- **gRPC/Connect RPC** for service communication\n- **Gin** (API), **chi** (Envd) for HTTP\n\n### Code Generation\n\nThe codebase uses several code generators:\n\n1. **Protocol Buffers** (`packages/orchestrator/generate.Dockerfile`)\n - Generates: `packages/shared/pkg/grpc/*/`\n - Run: `make generate/orchestrator`\n\n2. **OpenAPI** (`oapi-codegen`)\n - Spec: `spec/openapi.yml`\n - Generates: API handlers, types, specs\n - Run: `make generate/api`\n\n3. **SQL** (`sqlc`)\n - Queries: `packages/db/queries/*.sql`\n - Generates: Type-safe DB code\n - Run: `make generate/db`\n\n4. **Mocks** (`mockery`)\n - Config: `.mockery.yaml`\n - Run: `make generate-mocks`\n\n### Testing Patterns\n\n- **Unit tests**: Use `testify/assert` and `testify/require`\n- **Database tests**: `packages/db/pkg/testutils.SetupDatabase` gives each test its own database on the PostgreSQL server `PG_TEST_DATABASE_URL` names (see [DEV-LOCAL.md](DEV-LOCAL.md#unit-tests-that-need-postgresql))\n- **Integration tests**: `tests/integration/` with shared test utilities\n- **Mocking**: Generated mocks in `mocks/` directories\n- **Race detection**: Tests run with `-race` flag\n\nExample test invocation:\n```bash\n# Single package\ngo test -race -v ./internal/handlers\n\n# Specific test\ngo test -race -v -run TestCreateSandbox ./internal/handlers\n```\n\n## Important Development Notes\n\n### Working with Proto/gRPC\n- Proto files: `packages/envd/spec/process/`, `packages/envd/spec/filesystem/`, orchestrator protos at `packages/orchestrator/*.proto`\n- Shared protos: `packages/shared/pkg/grpc/`\n- After editing proto files, run `make generate/orchestrator` and `make generate/shared`\n\n### Database Migrations\n- Migrations: `packages/db/migrations/`\n- Create: `cd packages/db && make create-migration NAME=your-migration-name` — this generates the file with a correct `YYYYMMDDHHMMSS` timestamp. Do NOT hand-create migration files; placeholder timestamps like `120000`/`000000` cause same-day version collisions and are rejected by the out-of-order-migrations CI check.\n- Apply: `make migrate` (requires POSTGRES_CONNECTION_STRING)\n- Code generation: `make generate/db` (regenerates sqlc code)\n\n### Environment Variables\n- Local development defaults: `packages//.env.local`; see [DEV-LOCAL.md](DEV-LOCAL.md).\n- Self-hosting configuration: [embed/compose/.env](embed/compose/.env); see [embed/README.md](embed/README.md).\n- Custom environment configs: `.env.`, selected with `make switch-env ENV=`.\n\n### Firecracker & VM Management\n- Orchestrator requires **sudo** to run (Firecracker needs root)\n- VM networking uses `iptables` and Linux `netlink`\n- Storage uses NBD (Network Block Device)\n- Templates cached in GCS bucket (configurable via TEMPLATE_BUCKET_NAME)\n- Kernel/Firecracker versions: `packages/fc-versions/`\n\n### Observability\n- All services export OpenTelemetry traces/metrics/logs\n- Local stack includes Grafana + Loki + Tempo + Mimir\n- Telemetry setup: `packages/shared/pkg/telemetry/`\n- Logger: `packages/shared/pkg/logger/` (Zap with OTEL)\n- Profiling: API exposes pprof on `/debug/pprof/` (see `packages/api/Makefile` profiler target)\n\n### CI/CD Workflows\n- `.github/workflows/pull-request.yml` - PR orchestrator (lint, OpenAPI, unit, arm64, integration)\n- `.github/workflows/pr-tests.yml` - Unit test shards\n- `.github/workflows/integration_tests.yml` - Integration test suite\n\n## Architecture Patterns\n\n1. **Service Isolation**: Each service runs in containers with defined gRPC/HTTP interfaces\n2. **Shared Libraries**: Cross-cutting concerns (logging, telemetry, DB) in `packages/shared`\n3. **Event-Driven**: ClickHouse + Redis pub/sub for async operations\n4. **Caching Strategy**: Redis for templates, auth tokens, performance optimization\n5. **Feature Flags**: LaunchDarkly for gradual rollouts\n6. **Graceful Shutdown**: Services handle SIGTERM with context cancellation\n7. **Health Checks**: gRPC health protocol + HTTP health endpoints\n\n## Debugging\n\n### Logs\n- Local: Docker logs in `make local-infra`\n- Production: Grafana Loki\n","docs/ARCHITECTURE.md":"# E2B Infrastructure Architecture\n\nThis document explains what this repository implements, what each service does, and how the\nservices interact. It is the fastest way to build a mental model of the codebase before diving\ninto the code.\n\n> **Keep this document updated.** If a code change alters anything described here (service\n> responsibilities, ports, protocols, data stores, flows, deployment topology), update this file\n> in the same PR.\n\n## What this repository implements\n\nE2B provides **sandboxes**: isolated Linux VMs that start almost instantly (they resume pre-booted\nsnapshots instead of cold-booting), run arbitrary code (typically\ngenerated by AI agents), and can be paused, snapshotted, and resumed. This repo contains the whole\nbackend: the control-plane REST API, the data-plane VM orchestration built on\n**Firecracker microVMs**, the in-VM agent, the edge routing layer, and template building.\n\nTwo ideas drive the design:\n\n1. **A sandbox is a resumed snapshot.** Templates are pre-booted VM snapshots (memory + disk +\n VM state) stored in object storage. \"Creating\" a sandbox means restoring a snapshot, which is\n why startup is fast. Memory pages are loaded lazily on page-fault (userfaultfd) and the root\n filesystem is a copy-on-write overlay, so only touched data is ever fetched. A template can\n also be filesystem-only (rootfs, no memfile), taken by a filesystem-only pause or checkpoint;\n sandboxes created from it cold-boot from disk instead of resuming.\n2. **Control plane and data plane are separate.** The API decides *where* a sandbox runs and\n tracks *that* it runs (Postgres/Redis); the orchestrator on each node owns *how* it runs\n (Firecracker, networking, storage). Sandbox traffic never passes through the API.\n\n## System overview\n\n```mermaid\nflowchart TB\n subgraph clients[\"Clients\"]\n SDK[\"SDK / CLI\"]\n Browser[\"Browser / HTTP clients\"]\n end\n\n LB[\"Load balancer
api.* | *.domain wildcard\"]\n\n VC[\"volume-content API (belt)
api.<domain>\"]\n\n subgraph controlplane[\"Control plane (API node pool)\"]\n API[\"API
REST :80, gRPC :5009/:5109\"]\n DashAPI[\"dashboard-api :3010\"]\n CP[\"client-proxy
:3002\"]\n end\n\n subgraph datastores[\"State\"]\n PG[(\"PostgreSQL
teams, templates, builds, snapshots\")]\n RD[(\"Redis
running sandboxes, routing catalog, caches\")]\n CH[(\"ClickHouse
metrics, events, optional logs\")]\n OS[(\"Object storage GCS/S3
template + snapshot artifacts\")]\n end\n\n subgraph clientnode[\"Sandbox nodes (one orchestrator per node)\"]\n ORCH[\"orchestrator
gRPC :5008, proxy :5007\"]\n subgraph vm[\"Firecracker microVM (per sandbox)\"]\n ENVD[\"envd :49983\"]\n USERPROC[\"user processes\"]\n end\n end\n\n subgraph buildnode[\"Build nodes\"]\n TM[\"template-manager
(orchestrator binary, gRPC :5008)\"]\n end\n\n SDK -->|REST| LB --> API\n Browser -->|\"port-sandboxid.domain\"| LB --> CP\n API -.->|\"mint content token + domain\"| SDK\n SDK -->|\"volume content (token-authed)
api.<BYOC or default domain>\"| VC\n API -->|\"gRPC Create/Delete/Pause\"| ORCH\n API -->|\"gRPC TemplateCreate\"| TM\n CP -->|\"lookup sandbox → node\"| RD\n CP -->|\"forward :5007\"| ORCH\n CP -.->|\"gRPC auto-resume\"| API\n ORCH --> ENVD\n ENVD --> USERPROC\n API --> PG & RD & CH\n DashAPI --> PG & CH\n ORCH --> OS & CH\n TM --> OS\n```\n\n## Services\n\n| Service | Package | Runs on | Purpose |\n|---|---|---|---|\n| API | `packages/api` | API nodes | Public REST API; sandbox lifecycle, placement, auth, quotas |\n| Orchestrator | `packages/orchestrator` | every sandbox node | Runs Firecracker VMs; sandbox create/pause/resume/kill |\n| Template manager | `packages/orchestrator` (role) | build nodes | Builds templates from Docker images |\n| Client proxy | `packages/client-proxy` | API nodes | Edge router: sandbox URL → correct node |\n| Envd | `packages/envd` | inside every VM | In-VM agent: process/filesystem API for SDKs |\n| Dashboard API | `packages/dashboard-api` | API nodes | Backend for the web dashboard and workspace control plane (sandbox records, admin, `/v1/management`) |\n\nSupporting packages: `packages/shared` (protos, telemetry, storage clients, feature flags),\n`packages/auth` (authentication library), `packages/db` (Postgres migrations + sqlc queries),\n`packages/clickhouse` (ClickHouse schema + clients), `packages/otel-collector` (collector config),\n`packages/nomad-nodepool-apm` (autoscaler plugin), `packages/local-dev` (local stack).\n\n### API (`packages/api`)\n\nThe control-plane entry point (Gin, OpenAPI-generated from `spec/openapi.yml`, port 80).\n\n- **Resources**: sandboxes (create/list/kill/pause/resume/connect/timeout/metrics/logs),\n templates and builds (including the web dashboard's build listings under `/unstable/builds`),\n teams, volumes, API keys, secrets, sandbox events and webhooks, admin operations.\n- **Default templates**: `env_defaults` names the platform's default templates. Admin routes\n under `/unstable/admin/templates` add, describe and remove default templates and aliases\n without a namespace, and clear the alias cache after each alias change so name resolution\n sees it at once.\n- **Spec document**: `GET /openapi.json` serves, without authentication, the spec the request\n validator runs. An `ETag` over the document answers a matching `If-None-Match` with 304. It is\n registered before the validator, which rejects undeclared paths.\n- **Auth** (via `packages/auth`): team API keys (`X-API-Key`, `e2b_` prefix), auth-provider JWTs\n (OIDC), and either an admin token or a service JWT verified from the configured admin JWKS.\n Backed by an auth DB (Postgres) with a Redis team cache.\n- **API-group rate limits**: operations declare a group through the `x-api-group` OpenAPI\n extension. After authentication succeeds, its callback wrapper reads the validator's\n already-matched operation and stores the API group directly in Gin's context. The rate limiter\n reads that group and maps it to a field in the authenticated team's limits. Grouped operations\n require team-identifying authentication; public operations do not populate an API group.\n The `list` group uses `api_team_rps_list`; the `delete` group uses `api_team_rps_delete` only\n for sandbox and template deletion. Other DELETE operations are ungrouped.\n Delete rates default to zero until configured.\n A group's rate is one of the team's limits and resolves with the rest of them in the statement\n that loads the team, which authentication caches; it has no source of its own.\n Propagated project limits carry both rates, and each accepted revision replaces\n them. A team with no `project_limits` row uses the deployment's `default_limits`; zero\n disables group limiting. The migration that added `default_limits` gave every existing team a\n row holding its limits at the time, so a change to the defaults reaches only teams created since.\n A positive rate sets both requests per second and burst capacity. Redis enforces a separate\n budget for each team, HTTP method, and route template across API replicas: groups select\n rates, but routes do not share a budget. Exhausted budgets return HTTP 429 with rate-limit\n and `Retry-After` headers. Ungrouped operations and unconfigured rates bypass group limiting;\n Redis failures are logged and counted while requests proceed.\n V1 route limits from `rate-limit-config` are always enforced first. For requests that pass V1,\n LaunchDarkly's `rate-limit-v2-mode` is the master switch for all V2 checks, and\n `rate-limit-delete-mode` can further restrict delete checks. Both default to `disabled`;\n missing, invalid, or unknown modes are treated as disabled. Deletes skip V2 if either flag is\n disabled and enforce only when both are enabled. If both are active and either is `shadow`,\n deletes run in shadow mode. Lists follow the global mode.\n The groups use the same limiter logic: `shadow` observes V2 decisions while\n preserving V1 headers, and `enabled` also enforces V2. Ungrouped operations, unconfigured V2\n rates, and V2 Redis errors retain V1 enforcement. A rejection reports the rejecting limiter's\n headers; when both checks allow a request in enabled mode, V2 supplies the rate-limit headers.\n- **Sandbox list snapshots**: a list request reads the team's whole sandbox index from Redis\n (its running and pausing records) and pages in memory, so with the LaunchDarkly\n `sandbox-list-max-lag-ms` flag set for a team, each API replica shares one read between\n that team's concurrent list requests and keeps it for the flag's value, counted from the\n start of the read; a request that joins a read already in flight gets that read's result\n however long it takes. A lowered flag applies to a live snapshot; a raised one from the\n next read. Within that lag a sandbox created, killed, paused or resumed can be missing from\n the list or shown in its previous state on the replica holding the snapshot, and one\n resumed inside it can appear on two pages of a walk served by different replicas. The\n default, zero, gives every request its own read with the same store commands as before.\n Only the two list endpoints use the snapshot; the evictor, reconcile, template delete and\n team teardown keep reading the store directly.\n- **Workload identity**: sandbox create accepts an optional `iam.tokens` map of caller-named\n workload-token definitions (each an exact `audience` and `tokenType`). A non-empty, validated\n map enables workload identity, whose identity the orchestrator derives from the sandbox's\n already-authoritative team/sandbox/execution/template IDs; the definitions are passed to the\n orchestrator in `SandboxConfig.iam`. The API mints no credential and delivers nothing into the\n sandbox; file-based delivery is rejected at admission. Definitions are persisted in the\n running-sandbox (Redis) and paused-snapshot (Postgres) state so they survive pause/resume and\n orchestrator re-sync; a fork starts a new workload and does not inherit them.\n- **Placement**: keeps a live map of orchestrator nodes (discovered via Nomad, Kubernetes, or a\n static list). Chooses a node per sandbox with a **best-of-K** algorithm\n (`internal/orchestrator/placement/`): sample K ready nodes, score by the\n higher of CPU load and hugepage-pool load, pick the lowest; retry on\n exhausted nodes. A node that refuses a create (`ResourceExhausted`) stays\n eligible, and each refusal waits a full-jitter backoff that doubles up to a\n cap, so a placement that finds every node busy does not re-send creates as\n fast as they are refused. The hugepage-pool load is measured against the node's\n base pool (`nr_hugepages`, allocated at boot), so a node that has borrowed surplus\n pages scores above 1 instead of reading as just full; a pool that is all overcommit\n is measured against its ceiling, and an orchestrator that does not report the base\n pool against `HugePages_Total` as before. A node that reports no pool scores 0.5. Hugepage scoring\n is on by default (`BEST_OF_K_HUGEPAGE_MEMORY`); set it false to rank on\n CPU alone. K, overcommit ratio, and alpha are tunable live via feature\n flags. With `placement-hugepage-limit-percent` above 0, a hugepage sandbox\n is also refused by any node whose committed hugepages would pass that share\n of the node's pool ceiling (base pool plus overcommit allowance, reported by\n the orchestrator); within an API replica the check and the in-flight record\n are one step per node, and when the compatible nodes are all full placement\n answers with a capacity error, not \"no compatible node\". With the\n orchestrator's `create-memory-refusal` flag on (`CREATE_MEMORY_REFUSAL` for a\n deployment without LaunchDarkly), a create whose guest memory the pool\n cannot back while a sandbox on that node is still tearing down is refused\n with `ResourceExhausted`, so placement retries the node once the pages are\n back; with nothing tearing down it fails at once and says why. Off, it fails\n as an internal error and the node is excluded.\n- **State**: writes sandbox records to Redis (source of truth for *running* sandboxes). It does\n not write routing: client-proxy reads the orchestrator-written `sandbox:routing:{id}` (see\n \"Sandbox routing records\"). Persistent entities (templates, builds, snapshots, teams) live in\n Postgres.\n- **Secrets**: `/secrets` is the only public surface for secret management (create, list, get,\n update, delete). The API authenticates the caller with the customer alternatives above, converts\n the authenticated team UUID to the project UUID the backend knows, checks the `customer-secrets`\n feature flag, and forwards a metadata-only request over a unary gRPC contract\n (`e2b.secretsstore.management.v1`) to the secrets store backend named by\n `SECRETS_STORE_BACKEND_GRPC_ADDRESS`. No caller credential, header or client-supplied tenant\n crosses that hop, and no response - here or in a log, a span or an error - carries a secret\n value. What a caller stores is a runtime marker, not a resolved secret; orchestrator-ee resolves\n a marker to a value at sandbox egress, never here. Without a configured address, or with the flag\n off, the routes stay registered and answer 403. Responses are `Cache-Control: no-store`, request bodies are capped at 512 KiB, and values\n at 64 KiB. A create carries the cap on live secrets the backend enforces: `max_secrets` is one\n of the team's limits and resolves with the rest of them in the statement that loads the team (its\n `project_limits` value, else the deployment's `default_limits`).\n- **Sandbox events and webhooks**: `/events/sandboxes` reads a team's sandbox lifecycle events,\n and `/events/webhooks` manages the webhook subscriptions those events are delivered to, along\n with each subscription's delivery attempts and delivery statistics. The API stores none of\n this: it authenticates the caller, resolves the team's cluster, and forwards the request over\n two unary gRPC contracts (`e2b.webhooks.events.v1` and `e2b.webhooks.management.v1`) to the\n backend named by `WEBHOOKS_BACKEND_GRPC_ADDRESS`. The team and cluster come from the\n authenticated context, never from the request, and a team with no cluster selects the local\n one. Without a configured address the routes stay registered and answer 503, so the surface\n does not depend on how a deployment is wired. A backend failure the caller cannot act on is\n answered with the endpoint's own message; the outcomes that keep their identity are\n not-found, a rejected request, the webhook-count limit, and a timeout. A create or update\n body carries the webhook's signing secret, so a request rejected before the handler has its\n body kept out of the response, the access log and the span. Reads are capped at 100\n deliveries per page.\n- **Rig passthrough**: admin endpoints under `/clusters/{clusterID}/rigs` manage a cluster's\n orchestrator node pools (\"rigs\"). They list rigs, change rig capacity, list and terminate\n instances, and read recent scaling errors. Every handler delegates to the cluster's edge service\n (`/v1/rigs/...`) through the per-cluster HTTP client that the cluster sync keeps fresh. The API\n holds no cloud credentials and makes no scaling decisions. The caller authenticates with an\n admin credential and never holds the cluster's edge secret. Edge status codes pass through unchanged\n (400, 404, 409, 501). An edge 401 becomes a 500: it means the API is misconfigured against the\n cluster, not that the caller's token is invalid. A cluster with no rig management configured\n returns an empty list. The local cluster has no edge deployment and answers 501 for every rig\n operation.\n- **Extra listeners**: internal gRPC :5009 and edge gRPC :5109 expose `ResumeSandbox` so\n client-proxy can wake paused sandboxes on incoming traffic.\n- **Outbox** (`internal/outbox`): a River client on the shared database's `river` schema\n works jobs that need the orchestrator connections. Its job arguments live in\n `packages/db/pkg/outbox`. The one kind today, `teardown_team_resources`, stops a deleted\n team's workloads; the API's own project deletion enqueues it. A\n guard runs first on every attempt: a team that exists and is not blocked fails the attempt,\n and the job retries rather than kill a live team's sandboxes; a missing team row does not\n stop it. Then\n `kill_sandboxes` kills the team's running sandboxes and succeeds only once the sandbox store\n holds none of them in any state, so a sandbox still pausing or being killed makes the job\n retry. A sandbox that is not running cannot be paused, so no new snapshot of the team\n appears afterwards. Killing through the orchestrator is what removes the sandboxes' store\n records, so the job deletes no Redis state itself, and every attempt is safe to run again.\n Builds are not cancelled: they end on their own build timeout. An attempt is bounded at\n 30 minutes; retries back off from 30 seconds to a 15-minute cap, so the 100 attempts last\n about a day before River discards the job and keeps it. Volumes, templates, snapshots and\n the team row are left alone. Steps report\n `api.outbox.steps.finished` and `api.outbox.step.duration`; finished jobs report\n `outbox.jobs.finished`, and the backlog reports `outbox.jobs` and `outbox.oldest_*_age`.\n `OUTBOX_MAX_WORKERS` (default 10) and `OUTBOX_BACKLOG_INTERVAL` (default 30s) configure it;\n shutdown stops it after the sandbox-work drain, before the database and Redis clients close.\n- Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to\n Loki, with a LaunchDarkly-gated ClickHouse read path (`logs-read-config`) for local-cluster logs\n during the log storage migration. `LOKI_URL` is optional: without it the api has no Loki client\n and those reads fail until the flag routes them to ClickHouse. `LOGS_READ_CONFIG=true|false` sets\n the value the flag falls back to when LaunchDarkly has none (no LaunchDarkly, or the flag not\n defined there); a LaunchDarkly value wins. LaunchDarkly feature flags also gate placement\n parameters, rate limits, and rollouts.\n\n#### Project management writes (`/unstable/management`)\n\n`PUT /unstable/management/projects/{projectID}/settings` applies a revisioned settings\nprojection from the owning management service. This endpoint belongs to the API\nand accepts only service JWTs through `AdminJWTAuth`. It locks the team and checks\nthat it has not been deleted before applying the projection. The shared `ProjectSettings` type\ncontains `SandboxSettings`, with the optional ingress default under\n`sandboxSettings.defaultAllowPublicTraffic`. Storage uses `projection.project_settings`,\nwith `revision` and nullable `sandbox_default_allow_public_traffic` in the same row.\nThe receiver applies both in one guarded upsert. Equal or older revisions cannot\noverwrite it. Clearing stores NULL at a newer revision, retaining the ordering fence.\nCache invalidation follows commit and is retried on duplicate\ndelivery after a cache failure. There is no rollout flag check on accepted delivery.\nSandbox creation resolves an absent default to true and honors a saved default\nindependently of rollout eligibility. New sandboxes can use an older value while\ndelivery or cache refresh is pending.\n\nThe owning management service also drives the project writes under\n`/unstable/management/projects/{projectID}`. Like the settings route, each accepts only service\nJWTs through `AdminJWTAuth`, and each is idempotent, because the caller is level-triggered and\nretries.\n\n- `PUT /unstable/management/projects/{projectID}` creates or reconciles a project and answers 201\n or 200 to say which. A project is a `public.teams` row created from a caller-supplied UUID, with\n no tier. An optional cluster ID is stored on the project; omitting it preserves the current\n assignment. A changed slug renames the project and, in the same transaction, gives each of its\n namespaced template aliases under the new slug as well, keeping the old ones and leaving an\n alias the new slug already holds as it is. It answers 409 when another project already holds the slug, when the\n request assigns a cluster whose deletion is recorded, or when a concurrent request created the\n same project first.\n- `PUT .../members/{userID}` applies the desired presence for one user, gated by a monotonic\n per-project/user revision in `projection.project_members`; a duplicate or older revision\n succeeds without changing target state. A present projection carries the user's OIDC\n issuer/subject identities. Every projected user has at least one, and an identity already owned\n by a different user returns 409. A revocation removes only that user's `users_teams` row;\n projected users and identities are retained. Auth caches member authorization, so each accepted\n command evicts that user's authorization for the project after commit.\n- `PUT .../limits` syncs the project's effective limits into `project_limits`, which the team\n lookups read in preference to `default_limits`. It is gated by a monotonic per-project revision\n in `projection.project_limits`: the caller raises it whenever the limits it resolved change, and\n a delivery at or below the recorded revision is dropped and still answers 204. The ledger and the\n values advance in one transaction, so a revision is never recorded without the values it\n admitted.\n- `PUT .../block` applies a revision-fenced block state: the `projection.project_blocks` revision\n and `teams.is_blocked` / `blocked_reason` commit together, and an older or duplicate revision\n leaves stored state unchanged. Every accepted delivery, a dropped one included, evicts the team's\n auth cache; a failed eviction answers an error so the retry finishes it after commit. The API\n reads this state when admitting billable work. Apart from project deletion, which blocks the\n team for good, this is the only route that writes it.\n- `DELETE /unstable/management/projects/{projectID}` soft-deletes the project, described below.\n\nThe revision fences are the target's for the same reason: the caller can only fence what it\nsends, so two deliveries in flight arrive in whichever order the network gives them, and the older\none has to be refused where it lands. The project, membership, limits and block writes live in\n`internal/management` with their post-commit cache evictions rather than in the handlers.\n\nA limits or block delivery may carry `decided_at`, when the caller decided that revision; the\nledger row stores it beside the revision. Each delivery the ledger accepts records the seconds\nsince then in `dashboard-api.management.apply_lag` with `projection` `project_limits` or\n`project_blocks`; one without `decided_at`\ncounts in `dashboard-api.management.origin_unknown` instead, never as zero lag. Dropped, refused\nand failed deliveries record nothing. The names keep the dashboard-api prefix from when\ndashboard-api served these writes, so the alert and dashboard reading them carry on unchanged.\n\nProject deletion soft-deletes any project, including a member's default project; whether a project\nmay be deleted is the caller's decision. `envs`, `snapshots` and `volumes` reference `teams` with\n`ON DELETE NO ACTION`, so the team row stays. One transaction:\n\n- locks the row (404 if it is missing) and sets `teams.deleted_at`;\n- blocks the team (a CHECK keeps every deleted team blocked) and clears its cluster assignment;\n- scrubs its name and email to `DELETED` and its slug to `deleted-`, releasing the\n old slug for reuse;\n- deletes its memberships and API keys;\n- soft-deletes its templates, snapshot templates and paused-sandbox snapshots the way a template\n deletion does, releasing their aliases and build tracking;\n- enqueues `teardown_team_resources` with `InsertTx` on the outbox's own River client, so the\n teardown runs exactly when the deletion commits.\n\nAfter commit it evicts the deleted keys' auth cache entries best-effort; the team's other cached\nentries refresh or expire on their own. Repeating the call answers 204 and runs every step again,\nkeeping only the first deletion's time. That undoes a block reason, rename or cluster assignment\nwritten by a delivery that passed its deleted-project check just before the first call committed;\ndeletes memberships, keys and envs created since (a sandbox that paused before the teardown killed\nit leaves a snapshot); and enqueues the teardown again, which is a no-op while one is still open.\n\nAfterwards the API's deleted-project middleware, running after authentication, answers 404 to every\nnon-GET route under `/unstable/management/` that names `{projectID}`: the upsert, member, limits,\nblock and settings writes. The DELETE is exempt so a retry still reaches it. dashboard-api guards its own\nmanagement and admin writes the same way. Shared auth refuses a membership or API key still left\nfor the project with 403, and the API leaves the project out of a user's team list. Volumes, builds\nand stored artifacts are not reclaimed.\n\n#### Team bans (`/unstable/admin/teams`)\n\n`PUT /unstable/admin/teams/{teamID}/ban` sets `teams.is_banned`, evicts the team's auth cache so\nits API keys stop authenticating, then kills its running sandboxes and cancels its builds through\nthe same orchestrator and template-manager calls as `POST /admin/teams/{teamID}/sandboxes/kill`\nand `.../builds/cancel`. The ban commits first: a kill or cancel step that fails is reported in\nthe 200 response and does not undo it. The handler detaches the request from the caller and gives\nit one 25-second budget at the same moment, so a caller that disconnects does not stop it and\nnothing it runs (deletion checks, ban writes, cache evictions, kills, cancellations) blocks past\nthe budget. A ban write or eviction that runs out of it fails like any other (500, or `failed` in\nthe bulk route). Each team's kill and cancel run side by side; a step the budget cuts short is\nreported as failed with what it counted so far. A build delete already under way keeps its own 30\nseconds, so a request ends within about 55 seconds. A failed cache eviction answers 500, so the\ncaller's retry repeats the ban and its containment; banning a banned team contains it again. `DELETE .../ban` clears the flag and evicts the cache.\n`POST /unstable/admin/teams/bans` takes up to 100 teams. It stores every ban first, five teams at\na time, then contains the banned teams, five at a time, all under the one budget, and sorts each team\ninto banned, not found or failed; one team's failure does not stop the others. A deleted project\nanswers 404 to the single-team routes and counts as not found in the bulk route. The logic lives\nin `internal/team`.\n\n### Orchestrator (`packages/orchestrator`)\n\nA single Go binary running on every sandbox node (as root). `ORCHESTRATOR_SERVICES` selects its\nroles: `orchestrator` (run sandboxes) and/or `template-manager` (build templates). Code lives\nunder `pkg/`, almost all Linux-only.\n\ngRPC services on :5008 (`pkg/server/`, `pkg/service/`, `pkg/template/server/`, `pkg/volumes/`):\n\n- **SandboxService** — `Create`, `Update`, `List`, `Delete`, `Pause`, `Checkpoint`.\n `Update` callbacks are serialized per sandbox. A successful transition out of the live\n map closes update admission and joins any admitted callback before stop/checkpoint\n work or stopping notifications begin; `Close` repeats that check for crash cleanup.\n An update using an inactive sandbox returns gRPC `NotFound`, which the API maps to\n HTTP 404 for timeout and network updates. Teardown waits for admitted callbacks;\n it cannot preempt one that does not return.\n- **TemplateService** — `TemplateCreate`, `TemplateBuildStatus`, `TemplateBuildDelete` (template-manager role only).\n- **InfoService** — node identity, roles, capacity, health status (used by API node discovery).\n- **ChunkService / VolumeService** — peer-to-peer template chunk serving; persistent volumes.\n\nProcess shutdown, triggered by `SIGINT`, `SIGTERM`, `SIGUSR1`, or a service failure, moves\nthe node to `ShuttingDown` before draining builds and sandboxes. Like `Draining`, this excludes new\nplacement while existing work stays reachable, and `/health` returns HTTP 200 with status\n`draining`. Unlike reversible `Draining`, `ShuttingDown` is terminal for the process;\nstatus overrides cannot enter or leave it. `FORCE_STOP` skips waiting for sandboxes to exit.\nThe sandbox drain ends only when no sandbox is live and the node's outstanding work (below) is\nzero, so a create or a resume-fresh checkpoint in flight, whose sandbox is briefly not live, holds\nit until its sandbox goes live and then ends.\n\nAfter sandbox drain and snapshot uploads complete, sandbox proxy cleanup attempts graceful HTTP\nserver shutdown with a separate deadline, then forces server closure if needed. Successful forced\nclosure recovers graceful deadline expiry; other server errors still propagate. Pprof stays\navailable through service teardown; its final shutdown also has a deadline and forced-close fallback.\n\n`InfoService.ServiceInfo` reports `outstanding_work` for sandbox, template-builder, and\nmixed-role nodes. It counts overlapping work holds, not distinct sandboxes or builds, and includes\ntracked background persistence and cleanup. The producer reports zero when idle. The API caches\nthis report and exposes it as required top-level `outstandingWork` in admin node list and detail\nresponses, where zero also covers a node that has not reported. This observational count does not\nby itself authorize node deletion.\n\n`InfoService.ServiceInfo.max_sandboxes` reports the cached node limit for every service role.\nAdmin `/nodes` and `/nodes/{nodeID}` return required `maxSandboxes`, including zero. Admission,\nmetrics, and node info share the value refreshed from node-scoped flags every 30 seconds.\n\nKey mechanisms (all under `pkg/sandbox/`):\n\n- **Firecracker** (`fc/`): each sandbox is one Firecracker process in its own cgroup and network\n namespace. The FC HTTP API (unix socket) configures machine, drives, network, and snapshots.\n Guest metadata (sandbox ID, envd access token hash) is passed via MMDS.\n A sandbox may use fewer CPUs than its VM has (`SandboxConfig.max_vcpus` above `vcpu`): the VM\n is created with `max_vcpus` vCPUs, a boot puts `maxcpus=` on the kernel command line, and\n the `fc_vcpu` threads are held to `vcpu` CPUs by a `cpu.max` quota in a threaded `vcpu` child of\n the sandbox cgroup, so the VMM's own threads stay outside it. The quota is in place before the\n guest runs an instruction: a boot limits the whole cgroup until the vCPU threads exist and then\n moves the limit onto them; a resume confines the paused threads before the VM resumes, falls\n back to the whole cgroup, and fails if neither holds. The snapshot's metadata records the VM's\n vCPU count (`vcpu_count`, stamped at build and on every pause), so a resume limits below it\n without trusting the request or discovering capacity from cgroup threads. A legacy snapshot\n carries its stored original vCPU count as `max_vcpus`; if unavailable, it may only resume at its\n unchanged `vcpu`.\n `/init` carries the count as `cpuCount`\n so envd onlines or offlines guest CPUs to match and reports the outcome on `X-Envd-Cpus` and\n `/metrics`.\n- **Lazy memory / UFFD** (`uffd/`): on resume, Firecracker restores the VM without loading\n memory; a userfaultfd handler serves page faults directly from the template's memfile, so only\n touched pages are read. An optional prefetcher warms known-hot pages.\n- **Copy-on-write rootfs** (`rootfs/`, `nbd/`, `block/`): the template rootfs stays read-only;\n writes go to a per-sandbox COW cache exposed to Firecracker as an NBD block device served by\n an in-process userspace NBD server. On pause, the dirty blocks are exported as a diff.\n- **Template cache** (`template/`): templates are fetched lazily from object storage and cached\n on local disk (and optionally on a shared NFS chunk cache, or fetched peer-to-peer from other\n nodes before upload completes). Every reader of a cached template holds a pin on it — a\n running sandbox, a build, a peer stream, an ancestor lookup — and eviction never closes a\n pinned template. A snapshot's own upload holds a pin while it runs and records on the entry\n whether the snapshot landed in storage; a layer that did not land is never released, so a\n failed upload's local copy stays until its TTL expires. When a pause, or a\n checkpoint that resumes a fresh sandbox, publishes a new layer, the layer its sandbox ran\n from is marked superseded if this node inserted it for an earlier pause, because no resume\n can ask for it again. With `snapshot-cache-release-superseded` and\n `snapshot-cache-ancestor-storage-fallback` both on, a superseded layer is released once no\n pin holds it, its devices have resolved and its upload has landed, instead of staying until\n its TTL expires. A descendant's upload then heals that ancestor's build data from storage,\n and writes the same header bytes it would have written with the ancestor still cached.\n The sandbox and template caches start empty after a restart: the template cache empties the\n build-data directory, and startup reclaim empties the sandbox, template and template-build\n scratch directories.\n- **Networking** (`network/`): each sandbox gets a slot — a network namespace with a veth pair\n and a tap device, unique host-side IP (from a /16), NAT, and per-slot nftables egress firewall\n (with SNI/Host-inspecting TCP firewall for domain allow/deny lists). Slots are pooled and\n reused; slot indexes are allocated locally against the node's netns state (leftover\n namespaces from a previous run are torn down by startup reclaim).\n- **Sandbox proxy** (:5007, `pkg/proxy/`): reverse-proxies incoming traffic from client-proxy to\n the sandbox's slot IP and requested port over HTTP or configured HTTPS, enforcing per-sandbox\n traffic access tokens through the shared ingress policy. HTTPS backends may use self-signed certificates.\n It refuses a full-router sandbox with 503 `sandbox_feature_unsupported` before dialing the\n slot.\n Each admitted lifecycle owns its proxy handlers and backend sockets, including pending dials\n and upgrades. Stopping seals admission and interrupts connections; network release waits for\n that work to finish. Restarting envd resets the owner while allowing the live lifecycle to\n admit a replacement. Retired streams have one second to write their terminal response before\n a stalled frontend is interrupted.\n Shared proxy shutdown waits for H2C and upgraded connections to close and includes them in\n forced connection cleanup. Failed H2C handshakes release their hijacked connections when the\n handshake handler returns.\n- Writes sandbox lifecycle **events** and cgroup **host stats** to ClickHouse; exports metrics via\n OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy\n collector remains the fallback primary destination, and configured shadow destinations can mirror\n writes during collector/storage migrations without changing sandbox behavior.\n\nDetached sandbox-event publication holds node work until the publisher returns. This covers\ndelivery attempts, not end-to-end delivery: the ClickHouse target enqueues into an in-memory\nbatcher, whose flushing remains part of shutdown.\n\n### Envd (`packages/envd`)\n\nThe agent inside every VM (started by systemd very early in boot), port 49983, chi + Connect RPC.\n\n- **Process service** (`spec/process/process.proto`): start/list/connect to processes, stream\n stdout/stderr, stdin, signals, PTYs — this is what SDKs use to \"run code\". A start is refused\n with `unavailable` while the workload cgroups are frozen for a pause, and the freeze waits for\n in-flight starts, so a pause does not capture a child stopped before exec in a frozen cgroup. A\n start into a cgroup the guest froze itself is refused the same way. Starts carrying the system\n tag are unaffected: they stay in envd's own cgroup, which no freeze covers.\n- **Filesystem service** (`spec/filesystem/filesystem.proto`): stat/list/make/move/remove/watch.\n- **REST**: `/health`, `/metrics`, `/envs`, `/files` upload/download, `/init` (orchestrator pushes\n env vars, access token, metadata and the online CPU count after boot/resume), `/upgrade` (live self-upgrade, below),\n freeze/thaw hooks used during pause.\n- **Public vs. control-plane routes**: the control routes (`/init`, `/collapse`, `/upgrade`, and the\n freeze/thaw hooks) are marked `x-internal: true` in `spec/envd.yaml`, and `/upgrade` — which the spec does not\n describe — is listed alongside them in the orchestrator's `pkg/sandbox/envd`. The orchestrator\n reaches them over the host network at the sandbox slot IP; the sandbox proxy refuses them with a\n 404, for every method, so they are not reachable through a sandbox URL. Adding a control route\n therefore means marking it in the spec (`go generate` carries the marker into the proxy's\n rejection list) — otherwise it ships reachable from the internet.\n- **Guest trust** (`internal/host/cacerts.go`): `/init` supplies the desired CA bundle.\n envd atomically records its PEM and owned DER identities in `/run/e2b/ca/state.json`,\n then publishes a complete replacement of the active bundle with the requested roots\n before returning. Ordinary installs copy the existing bundle without decoding it;\n recovery first repairs interrupted writes. All supplied identities are owned,\n including identical preexisting copies. A background loop resumes saved intent at startup\n and retries cleanup every second. It atomically removes retired roots from the active\n bundle before rootfs work, so source-file failures cannot delay that cleanup.\n It persists each requested certificate as\n `e2b-.crt` in `/usr/local/share/ca-certificates`, removes retired managed files,\n and retains retirement identities until both bundle and source cleanup succeed.\n Superseded cleanup cannot publish an older bundle. Legacy-file removal checks the latest\n desired roots under the init lock, preserving hash links to a still-requested certificate.\n State survives envd process restart,\n not guest reboot. Retired roots may be trusted until cleanup succeeds, and an independent\n trust-store rebuild can restore a copy from a customer source file. Applications that\n cached their trust store must reload it.\n- **Auth**: `X-Access-Token` header checked against a token delivered via Firecracker MMDS;\n signed URLs for file endpoints. `/init` is exempt (it is what *delivers* the token), which is the\n main reason the proxy refuses it outright.\n- **Live upgrade** (`internal/services/process/upgrade.go`): an authenticated `POST /upgrade` lets\n the orchestrator swap envd inside a *running* sandbox at resume. It streams the new binary in the\n request body and envd `syscall.Exec`s into it **with the same PID**, carrying the workload's\n stdio/PTY fds, process table, recently-retained exit codes and filesystem watchers forward via a\n tmpfs handover blob. The workload cgroups stay frozen until the post-upgrade `/init` restores the\n access token (so no re-adopted process runs unauthenticated), and the handover outcome\n (procs/watchers re-adopted, plus any failures) rides back on that `/init`'s `X-Envd-Handover`\n header for fleet visibility.\n- Scans guest ports and forwards them so any port a user process opens becomes reachable through\n sandbox URLs. **`pkg/version.go` carries the released version**, maintained by release-please from\n the Conventional Commits merged since the last envd release; do not edit it by hand. The API and\n the orchestrator gate features on the envd version recorded in each template build — the\n orchestrator's held `/init` is the one gate that reads the version of the envd expected to\n answer instead (see sandbox creation) — so a behavioral change ships under a `feat`/`fix`\n commit type that cuts a version it can be gated on.\n\n### Client proxy (`packages/client-proxy`)\n\nThe stateless edge for all sandbox traffic (port 3002; health on 3003). Terminates\n`https://-.` requests (host parsing in `packages/shared/pkg/proxy/host.go`),\nlooks the sandbox up in the Redis routing record to find the owning node, and reverse-proxies to\nthat node's orchestrator proxy on :5007 by default. `ORCHESTRATOR_PROXY_PORT` selects a\ndifferent downstream port when the node proxy listens elsewhere. If the sandbox is not in the record (paused), it calls\nthe API's `ResumeSandbox` gRPC — paused sandboxes wake transparently on traffic.\nThe record is the orchestrator-owned `sandbox:routing:{id}` (see \"Sandbox routing records\"\nbelow).\n\nThe record on a hit, or the `ResumeSandbox` answer on a miss, is the sandbox's route, placement\nincluded. A miss uses the resume answer directly and does not read the record again. A legacy\n(no placement) or egress-router route goes to `orchestrator_ip` on the orchestrator proxy port,\nnever to the placement's ingress endpoint. A full-router route goes to the Router's ingress\nendpoint saved in its placement, never to an orchestrator, over the same pooled, kept-alive\nconnections as the orchestrator hop: the proxies never resend a request themselves (only its\ndial is retried), and net/http's resend of a bodyless idempotent request whose reused connection\nclosed before any response is accepted on both hops. A full-router route without that endpoint,\nan unknown mode or a missing `orchestrator_ip` answers the not-found page before anything is\ndialed.\n\nConcurrent requests that miss the record for the same sandbox, port and credentials share one\n**resume flight** per client-proxy process: the first request's call runs detached from its\nconnection under a 90 s ceiling and later requests join it, so one resume RPC serves them all\nand a client that disconnects mid-resume does not fail the others. A call whose requests have\nall left runs to its end, so a client that comes back finds the sandbox awake, but no further\nattempt starts for nobody, and a process keeps at most 64 such callerless calls; beyond that a\ndeparture cancels the proxy's call. That bound and that cancellation apply to the proxy's flights\nonly: the API's resume entry point lets a create already under way run on past the proxy's\ncancellation, so the node finishes the create and the API registers the sandbox inside the\nreservation's 90 s window (the create may run until ten seconds before the pending reservation can\nbe reaped and the registration until five seconds before, so the run-on never lets a fresh\nreservation start a second resume of the same snapshot), and the API's shutdown drain waits for\nthat work, bounded at 95 s. The run-on is granted when the resume is admitted, and an API already\ndraining admits none: a resume that arrives then keeps its caller's cut, so a create the proxy\ncancels mid-drain is cut with it and the reservation lapses. The sandbox otherwise lives its\nordinary timeout or auto-pause, billed, with nobody waiting for it. With\n`client-proxy-resume-parking` on (default off), a request whose flight got a still-transitioning\nor `Unavailable` answer parks instead of failing: the flight retries\nwhile a parked request waits, under a budget (`client-proxy-resume-parking-budget-ms`, 15 s from\nthe first retryable failure) that gates only whether another retry starts: every attempt, a late\nretry included, runs under its full 90 s ceiling while a caller waits on it, so a parked request\ncan wait up to the budget plus one attempt. The proxy cuts a retry at the budget's end only once\nevery caller has left, and that cut ends the proxy's wait, not the create on the node.\nA process keeps at most\n`client-proxy-resume-parking-lot-size` (512) parked requests, shedding the rest. A resume the\ncontrol plane cannot carry out right now (no capacity, no node, the API unreachable, the budget\nspent) answers **503** with `Retry-After: 2` and no error code; a request shed because the lot was\nfull answers the same **503** without the header and with `proxy_parking_lot_full` as its\n`error_code`, so a client, or one behind an ingress that strips the header, still tells the two\napart. The SDKs do not act on the header either way: their status-based retry\nwraps only the control-plane API client, and sandbox traffic through the proxy is retried only on\na connection failure, never on a 503. The omission is for hand-rolled retry loops and browsers\nthat honour the header, which would otherwise come back in one wave while the lot is still full,\nand for a client that does the SDKs' jittered backoff on its own; a placement no retry helps, which\nthe API names in an `ErrorInfo` detail on the RPC status, is not retried and answers without the\nheader, with the status the API carried: **503** for a feature no orchestrator in the cluster\nsupports, **500** (`sandbox_create_refused`) for a sandbox the node refuses to create, such as a\nstored build whose vCPU count is outside the node's bounds; a sandbox still pausing\nanswers **409** with the header on a user port and the same page as **503** on the envd port (the\nSDK reads an envd 409 as \"already exists\"); a request whose client left before it was routed\nanswers 408.\n\n### Dashboard API (`packages/dashboard-api`)\n\nA separate REST service (port 3010, spec `spec/openapi-dashboard.yml`) consumed by the web\ndashboard and the workspace control plane, not the SDK: the killed-sandbox record, admin routes\n(clusters, team cluster assignment, user-profile resolution) and the `/v1/management`\noperations. Template reads are served by the API, not here. Its admin routes accept either a\nshared admin token or a short-lived service JWT verified against the configured admin JWKS. The\nsandbox record route accepts dashboard user auth only. Its workspace-agnostic `/v1/management` operations are defined in the\nsame dashboard OpenAPI contract and registered on the existing router. Their `AdminJWTAuth`\nOpenAPI security scheme accepts only short-lived service JWTs verified against the workspace-api\n`/.well-known/jwks.json` endpoint, with accepted signing methods derived from each JWK's required\n`alg` metadata. Issuers and audiences are configured through the JSON `ADMIN_AUTH_PROVIDER_CONFIG` value —\nthe same config shape as `AUTH_PROVIDER_CONFIG`. Talks to Postgres and ClickHouse; never talks to\norchestrators. The web dashboard itself is a separate repository (github.com/e2b-dev/dashboard)\npublished as an image; E2B Embed runs it on port 3001 beside dashboard-api.\n\nAn issuer normally configures at least one accepted audience, binding a JWT to its intended target.\nAn issuer may omit `audiences` only with no `audienceMatchPolicy`; this deliberately disables\naudience matching while issuer, signature, and temporal-claim verification remain required.\n\nThe `/v1/management` operations are the cluster's half of a contract the workspace residency owns:\nthe cluster lifecycle below. Project upsert, membership, limits, block state and deletion are\nserved by the API under `/unstable/management/projects/{projectID}`, and team bans under\n`/unstable/admin/teams` (see API above).\n\nThe management surface owns a replay-safe cluster lifecycle. A caller registers a stable\ncluster UUID, assigns it only to the named project, detaches that exact assignment before provider\ncleanup, and deletes the cluster only after no project references it. Replaying the same\nregistration or assignment succeeds. Registering the cluster again with the same name and auth\norganization may change its endpoint, TLS setting, sandbox proxy domain and token, for example when\nthe cluster moves to a new domain; the API rebuilds that cluster's clients on its next sync and gives\nsandboxes the new domain. A changed name or auth organization, a different assignment, an inexact\ndetach, or deletion of a referenced cluster returns a conflict. The\nassignment does not check the project's tier; the caller decides which projects may use a cluster.\nA cluster first registered through this surface starts without deletion protection; one created any\nother way starts protected, a later registration never changes it, and deleting a protected cluster\nor checking its destroy readiness returns a conflict.\nA caller that deletes a cluster's projects before the cluster first lists them with\n`GET /v1/management/clusters/{clusterID}/projects`, which returns every project whose team is\nassigned to the cluster however it was assigned, leaving out deleted teams, and records the deletion with\n`PUT /v1/management/clusters/{clusterID}/deletion` before deleting any of them. That request\nlocks the cluster row, returns a conflict while deletion protection is on, and sets\n`clusters.deletion_requested_at`; it succeeds for an absent cluster and on repeat. From then on a\ncheck constraint keeps deletion protection off, and a trigger on `teams` refuses to newly assign a\nteam to the cluster (`cluster_assignment_cluster_deleting` from the assignment route, a conflict\nfrom the API's project upsert carrying the cluster ID). The trigger takes a key-share lock on the cluster row\nbefore it checks, so an assignment either commits before the deletion is recorded, where the\ncaller's next listing finds it, or is refused. Deleting each project clears its team's cluster,\nthe cluster delete clears it from any team deleted earlier, and then succeeds.\nThe shared error response supports an optional `error_code`, defined by the dashboard OpenAPI\n`ErrorCode` schema. Registration and assignment rejections currently populate it; other errors\nmay omit it. Callers use that stable code rather than the human-readable message, and treat\nuncoded or unknown reasons as generic errors.\n\nA project deletion keeps the team row, so dashboard-api middleware, running after\nauthentication, answers 404 to every write under `/v1/management/` or `/admin/` that names a\ndeleted project by `projectID` or `teamID` (\n\n","packages/api/internal/sandbox/reservations/redis/README.md":"# Redis Reservation Storage\n\nThis package coordinates sandbox creation reservations across API instances.\n\n## Keys\n\n- Storage index: `sandbox:storage:{teamID}:index`\n- Pending zset: `sandbox:storage:{teamID}:reservations:pending`\n- Result key: `sandbox:storage:{teamID}:reservations:{sandboxID}:result`\n- PubSub routing key: `sandbox:storage:{teamID}:reservations:{sandboxID}:notify`\n\n## Flow\n\n`Reserve` runs a Lua script that atomically removes stale pending entries, checks whether the sandbox already exists or has already pending start, enforces the team limit using `SCARD(storage index) + ZCARD(pending zset)`, deletes any stale result key, and adds the sandbox ID to the pending zset.\n\nWhen creation completes, it removes the sandbox from the pending zset, writes a TTL result key containing either the sandbox or the creation error, and publishes the routing key.\n\nA waiter subscribes to the routing key, probes the result key immediately, then waits for PubSub notifications or the 1 second fallback ticker. PubSub is best-effort; the fallback ticker is required for correctness.\n\n`Release` is called when the sandbox is removed from storage (`Store.Remove`). It removes the sandbox from the pending zset, deletes the result key, and publishes the routing key.\n","packages/envd/README.md":"# envd\n\nDaemon that runs inside a sandbox that allows interacting with the sandbox via calls from the SDK.\n\n## Development\n\n### Versioning\n\nThe envd version in `pkg/version.go` is maintained by release-please from the Conventional Commits merged since the last envd release; do not edit it by hand. A change that affects behavior (code, dependencies) ships under a `feat` or `fix` commit type, so that a version the orchestrator can gate on is cut. Pure comment or documentation changes use `docs`, a type the changelog hides, and cut no release; `chore` and `refactor` are visible in it and do cut one.\n\n### Running locally\n\nRun the following command to (re)build the envd daemon and start a Docker container with envd running inside:\n\n```bash\nmake start-docker\n```\n\nYou can use E2B SDKs with env var `E2B_DEBUG=true` or with a debug parameter set to `true` when creating or connecting to a sandbox, to connect to the envd started with this command.\n\n### Generating API server stubs\n\nRun the following command to install the necessary dependencies for generating server stubs:\n\n```bash\nmake init-generate\n```\n\n#### Generating\n\nAfter changing the API specs in `./spec/` run the following command to generate the server stubs:\n\n```bash\nmake generate\n```\n\n### Debugging\n\n- \n- \n\nRun `make run-debug` and then connect to the port `2345` with a debugger or use the VSCode run/debug and run the \"Debug envd\" to build the envd, Docker, and start the debugging.\n","packages/envd/internal/services/legacy/readme.md":"The python SDK had a bug: new fields introduced to the API would throw exceptions when processed. In order to work around this, we convert all responses to a list of structs that were valid when the python SDK still had the bug.\n\nWe copy the proto files to this folder in order to keep track of what was available when this SDK was supported. These should never be modified.\n\nThis entire package can be removed once we no longer have clients that send a user agent of `connect-python`.\n\n","packages/nomad-nodepool-apm/README.md":"# Nomad Node Pool Autoscaler Plugins\n\nCustom plugins for the [Nomad Autoscaler](https://github.com/hashicorp/nomad-autoscaler) that keep a service-job allocation count aligned with the number of nodes in a Nomad node pool.\n\n## Purpose\n\nThe package builds two plugins because Nomad Autoscaler supports only one plugin type per external binary:\n\n- `nomad-nodepool-apm` reports the number of ready nodes in a pool.\n- `nomad-deployment-aware-target` fails an active deployment before applying the requested task-group count.\n\nTogether they let service jobs replicate system-job placement while retaining rolling updates. The deployment-aware target is used by the `template-manager` and `orchestrator-ee` service jobs. When the task-group count must change, scaling intentionally abandons a conflicting in-progress rollout (Nomad rejects scaling while a deployment is active); Nomad records that deployment as `failed`, rather than `cancelled`. The rollout spawned by the scale itself is left to run, and no deployment is touched when the count already matches. The target requires `auto_revert = false` on the scaled group (otherwise failing a deployment would restore an older job version and ping-pong); it refuses to scale groups with `auto_revert = true` and returns an error on every attempt.\n\n## Usage\n\n### Building\n\n```bash\nmake build\n```\n\n### Uploading to GCS\n\n```bash\nGCP_PROJECT_ID=your-project-id make build-and-upload\n```\n\n### Configuration\n\nIn your Nomad Autoscaler configuration:\n\n```hcl\napm \"nomad-nodepool-apm\" {\n driver = \"nomad-nodepool-apm\"\n config = {\n nomad_address = \"http://localhost:4646\" # Optional, uses NOMAD_ADDR env var\n nomad_token = \"your-token\" # Optional, uses NOMAD_TOKEN env var\n nomad_region = \"global\" # Optional\n }\n}\n\ntarget \"nomad-deployment-aware-target\" {\n driver = \"nomad-deployment-aware-target\"\n config = {\n nomad_address = \"http://localhost:4646\"\n nomad_token = \"your-token\"\n nomad_region = \"global\"\n }\n}\n```\n\n### In Job Scaling Policy\n\n```hcl\nscaling {\n enabled = true\n min = 1\n max = 50\n\n policy {\n evaluation_interval = \"30s\"\n cooldown = \"2m\"\n\n target \"nomad-deployment-aware-target\" {}\n\n check \"match_node_count\" {\n source = \"nomad-nodepool-apm\"\n query = \"orchestrator\" # Node pool name\n\n strategy \"pass-through\" {} # Use node count directly as desired count\n }\n }\n}\n```\n\n## How It Works\n\n1. The plugin queries the Nomad API to list nodes filtered by the specified node pool\n2. It counts only nodes that are:\n - Status: `ready`\n3. Returns the count as a metric for the autoscaler to use\n4. With the `pass-through` strategy, this count becomes the desired number of allocations\n5. The target serializes scaling per namespaced job and is a no-op when the durable count already matches\n6. When the count must change it fails a conflicting active deployment, rereads the job, and scales with the current job modify index\n7. It verifies the final durable count; the rollout spawned by the scale proceeds normally\n8. Concurrent job or deployment changes are retried from fresh state with a bounded attempt count\n\nDry-run actions do not fail deployments or write task-group counts.\n\n## Query Format\n\nThe `query` parameter should be the name of the node pool you want to count nodes from.\n\nExample: `query = \"orchestrator\"` counts all ready nodes in the `orchestrator` node pool.\n\n## Configuration Options\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `nomad_address` | Nomad server address | `NOMAD_ADDR` env var or `http://127.0.0.1:4646` |\n| `nomad_token` | Nomad ACL token | `NOMAD_TOKEN` env var |\n| `nomad_region` | Nomad region | Default region |\n| `nomad_namespace` | Nomad namespace | Default namespace |\n","packages/orchestrator/README.md":"# Orchestrator\n\n## Commands\n\n**Prerequisite:** Enable NBD module first:\n\n```bash\nmodprobe nbd nbds_max=4096\n\ncat </etc/udev/rules.d/97-nbd-device.rules\n# Disable inotify watching of change events for NBD devices\nACTION==\"add|change\", KERNEL==\"nbd*\", OPTIONS:=\"nowatch\"\nEOH\nudevadm control --reload-rules\nudevadm trigger\n```\n\n**Prerequisite:** Allocate enough 2MB HugeTLB pages (hugepages) for testing:\n\n```bash\necho 1024 | sudo tee /proc/sys/vm/nr_hugepages # for 2GB hugepages, adjust as needed\n```\n\n### Create Build\n\nCreate a new build.\n\n```bash\nsudo go run ./cmd/create-build -to-build -storage .local-build\n```\n\nFlags:\n\n- `-to-build ` - Output build ID (UUID, required)\n- `-from-build ` - Base build ID for incremental builds\n- `-template ` - Template ID (default: `local-template`)\n- `-storage ` - Local path or `gs://bucket` (enables local mode with auto-download of kernel/FC)\n- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)\n- `-kernel ` - Kernel version (default: `vmlinux-6.1.158`)\n- `-firecracker ` - Firecracker version (default: `v1.14.1_458ca91`)\n- `-vcpu ` - vCPUs (default: `1`)\n- `-memory ` - Memory in MB (default: `512`)\n- `-disk ` - Disk in MB (default: `1000`)\n- `-hugepages` - Use 2MB huge pages (default: `true`, set `false` for 4KB pages)\n- `-start-cmd ` - Start command\n- `-ready-cmd ` - Ready check command\n\n> You can use the `$(uuidgen)` command to generate a random UUID.\n>\n> If you are using Mise, you can run the command as `sudo $(which go) run ./cmd/create-build -to-build -storage .local-build`.\n\n### Resume Build\n\nResume a sandbox from a build.\n\n```bash\n# Local storage\nsudo go run ./cmd/resume-build -from-build -storage .local-build -iterations 10\n\n# Remote GCS storage (pass credentials to sudo)\nsudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \\\n go run ./cmd/resume-build -from-build -storage gs://bucket -iterations 10\n\n# Pause mode: resume, run command, then snapshot\nsudo go run ./cmd/resume-build -from-build -to-build \\\n -storage .local-build -cmd-pause \"pip install numpy\"\n\n# Pause mode: start command through envd, then wait for SIGUSR1 before snapshot\nsudo go run ./cmd/resume-build -from-build -to-build \\\n -storage .local-build -cmd-signal-pause \"python3 /home/user/workspace/job.py\"\n```\n\nFlags:\n\n- `-from-build ` - Build ID (UUID) to resume from (required)\n- `-to-build ` - Output build ID (UUID) for pause snapshot (auto-generated if not specified)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)\n- `-iterations ` - Number of iterations, 0 = interactive (default: `0`)\n- `-cold` - Clear cache between iterations (cold start each time)\n- `-no-prefetch` - Disable memory prefetching\n- `-v` - Verbose logging\n- `-pause` - Start and immediately pause (create snapshot)\n- `-signal-pause ` - Wait for signal before pause (e.g., `SIGTERM`, `SIGUSR1`)\n- `-cmd-pause ` - Execute command in sandbox, then pause on success\n- `-cmd-signal-pause ` - Execute command in sandbox, then wait for `SIGUSR1` before pause\n- `-envd-version ` - The guest's envd version (default: a placeholder — see the caveat below)\n\n**envd version caveat:**\n\n`resume-build` has no database and the snapshot metadata does not record the build's envd\nversion (only the kernel and Firecracker ones), so unless `-envd-version` is given the tool\nreports a fixed placeholder (`1.0.0`). Every `envd.version` field in its output and logs is\nthen that placeholder, **not** the guest's version — don't reason from it. To read the real\none, mount the build's rootfs (`mount-build-rootfs`) and run its `/usr/bin/envd -version`,\nor read `X-Envd-Version`, which a running envd reports on `/init`.\n\nThe placeholder sits above every envd version gate, so the gated capabilities (`/freeze`,\n`/fsfreeze`, `/collapse`, the KVM clock, the upgrade `from_version`) stay on. Against an\nolder guest that is a path production would not take, so pass `-envd-version` with the\nguest's real version when the behavior under test depends on one of those gates. The tool\nprints which of the two it is using when it loads the build.\n\n**Pause mode example:**\n\n```bash\n# Chain builds with explicit build IDs\nsudo go run ./cmd/resume-build -from-build $BUILD1 -to-build $BUILD2 \\\n -storage .local-build -cmd-pause \"apt install curl\"\n\nsudo go run ./cmd/resume-build -from-build $BUILD2 -to-build $BUILD3 \\\n -storage .local-build -cmd-pause \"pip install requests\"\n\n# Start a long-running command through envd, then trigger pause from the host\nsudo go run ./cmd/resume-build -from-build $BUILD3 -to-build $BUILD4 \\\n -storage .local-build -cmd-signal-pause \"python3 /home/user/workspace/job.py\"\n# In another shell:\nps -ef | grep 'cmd/resume-build' | grep -v grep\nsudo kill -SIGUSR1 \n```\n\n### Copy Build\n\nCopy a build between storage locations (local or GCS).\n\n```bash\n# Local to local\ngo run ./cmd/copy-build -build -from .local-build -to /other/path\n\n# Local to GCS\ngo run ./cmd/copy-build -build -from .local-build -to gs://bucket\n\n# GCS to GCS\ngo run ./cmd/copy-build -build -from gs://bucket1 -to gs://bucket2\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required)\n- `-from ` - Source: local path or `gs://bucket`\n- `-to ` - Destination: local path or `gs://bucket`\n- `-team ` - Team UUID (if set, prints SQL to populate DB on stdout)\n- `-envd-version ` - Envd version (required if team provided) — must match the version present in the template\n- `-vcpu ` - vCPUs (default: `2`)\n- `-memory ` - Memory in MB (default: `1024`)\n- `-disk ` - Disk in MB (default: `1024`)\n- `-tag ` - Build assignment tag (default: `default`)\n\n**Print DB seed SQL:**\n\nWhen `-team` is set, the command reads `metadata.json` from the destination to get `kernel_version` and `firecracker_version`, generates a new env ID, and prints SQL statements (`BEGIN`/`COMMIT` wrapped) to stdout. Pipe directly to `psql` to populate the database:\n\n```bash\ngo run ./cmd/copy-build -build -from .local-build -to gs://bucket \\\n -team -envd-version 0.2.11 -vcpu 2 -memory 1024 -disk 1024 \\\n | psql $POSTGRES_CONNECTION_STRING\n```\n\n### Mount Build Rootfs\n\n```bash\n# Local storage\nsudo go run ./cmd/mount-build-rootfs -build -storage .local-build -mount /mnt/rootfs\n\n# Remote GCS storage (pass credentials to sudo)\nsudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \\\n go run ./cmd/mount-build-rootfs -build -storage gs://bucket -mount /mnt/rootfs\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required unless `-empty`)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-mount ` - Mount path\n- `-verify` - Verify rootfs integrity (requires `-mount`)\n- `-log` - Enable logging\n- `-empty` - Create an empty rootfs instead of loading a build\n- `-size ` - Size of empty rootfs (default: 1GB)\n- `-block-size ` - Block size (default: 4096)\n\n### Inspect Build\n\nInspect build artifacts (headers and data blocks).\n\n```bash\n# Inspect memfile header (default)\ngo run ./cmd/inspect-build -build -storage .local-build\n\n# Inspect rootfs header\ngo run ./cmd/inspect-build -build -storage .local-build -rootfs\n\n# Inspect header + data blocks\ngo run ./cmd/inspect-build -build -storage .local-build -data\n\n# Inspect first 100 data blocks\ngo run ./cmd/inspect-build -build -storage .local-build -data -end 100\n\n# Remote GCS storage\ngo run ./cmd/inspect-build -build -storage gs://bucket -rootfs -data\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-memfile` - Inspect memfile artifact (default)\n- `-rootfs` - Inspect rootfs artifact\n- `-data` - Also inspect data blocks (not just header)\n- `-start ` - Start block index (default: `0`, only with `-data`)\n- `-end ` - End block index (default: all, only with `-data`)\n\n### Show Build Diff\n\nShow the diff between two builds and preview merge result.\n\n```bash\n# Memfile (default)\ngo run ./cmd/show-build-diff -from-build -to-build -storage .local-build\n\n# Rootfs\ngo run ./cmd/show-build-diff -from-build -to-build -storage .local-build -rootfs\n\n# With visualization\ngo run ./cmd/show-build-diff -from-build -to-build -storage gs://bucket -visualize\n```\n\nFlags:\n\n- `-from-build ` - Base build ID (required)\n- `-to-build ` - Diff build ID (required)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-memfile` - Inspect memfile artifact (default)\n- `-rootfs` - Inspect rootfs artifact\n- `-visualize` - Visualize the headers\n\n---\n\n## Architecture (ARM64) Support\n\nThe orchestrator supports both `amd64` (x86_64) and `arm64` (aarch64) architectures. Architecture is detected automatically via `runtime.GOARCH` at compile time.\n\n### Architecture naming convention\n\nThis project uses **Go/Docker/Debian naming** (`amd64`/`arm64`) for architecture directories in binary paths and GCS buckets:\n\n| Convention | x86_64 name | ARM64 name | Used by |\n|------------|-------------|------------|---------|\n| **Go/Docker/Debian** | `amd64` | `arm64` | This repo, Docker, `dpkg --print-architecture` |\n| Linux/GNU | `x86_64` | `aarch64` | `uname -m`, kernel Makefiles |\n\nBinary paths follow the `{version}/{arch}/` layout:\n\n```\n# Firecracker (GCS bucket or FIRECRACKER_VERSIONS_DIR)\nfc-versions/v1.12.1_210cbac/amd64/firecracker\nfc-versions/v1.12.1_210cbac/arm64/firecracker\n\n# Kernels (GCS bucket or HOST_KERNELS_DIR)\nkernels/vmlinux-6.1.158/amd64/vmlinux.bin\nkernels/vmlinux-6.1.158/arm64/vmlinux.bin\n```\n\n> **Note:** The [fc-kernels](https://github.com/e2b-dev/fc-kernels) repo currently uses `x86_64` instead of `amd64` for its directory names. This will be aligned in a follow-up change.\n\n### ARM64-specific behavior\n\n- **SMT** is disabled (ARM processors don't support simultaneous multi-threading)\n- **CPU detection** uses fallback values since `gopsutil` doesn't populate Family/Model on ARM64\n- **OCI platform** is set to the target architecture instead of hardcoded `amd64`\n- **Busybox binaries** are fetched per architecture by `scripts/fetch-busybox.sh` and selected at runtime from `///busybox`\n- **Guest kernel command line** drops the x86-only `i8042.*` and `clocksource=kvm-clock` parameters on arm64; the console stays `ttyS0` (Firecracker exposes a 16550-compatible UART on both architectures)\n\n### Cross-architecture deployment\n\n`TARGET_ARCH` is a **runtime** environment variable that overrides the architecture used for path resolution and OCI image pulls. When unset, defaults to the host architecture (`runtime.GOARCH`).\n\n```bash\n# Run orchestrator targeting amd64 paths from an arm64 host\nTARGET_ARCH=amd64 ./bin/orchestrator\n\n# Or in .env file (read at runtime)\necho \"TARGET_ARCH=amd64\" >> .env.local\n```\n\n`TARGET_ARCH` affects:\n- Firecracker and kernel binary path resolution (`{version}/{arch}/...`)\n- OCI image platform for container pulls\n\nIt does **not** affect:\n- Makefile compilation (see `BUILD_ARCH` below)\n- Hardware-dependent runtime behavior (SMT detection, CPU info) which always uses the actual host architecture\n\n### Building for ARM64\n\n`BUILD_ARCH` defaults to `amd64`. To build ARM64 binaries:\n\n```bash\n# Single command\nBUILD_ARCH=arm64 make build-local\n\n# Or set in .env.local for persistent override\necho \"BUILD_ARCH=arm64\" >> .env.local\n```\n\nThis applies to all services: orchestrator, envd, client-proxy.\n\n---\n\n## Environment Variables\n\nAutomatically set in local mode. Set before running to override:\n\n- `TARGET_ARCH` - Target architecture override (`amd64` or `arm64`; default: host architecture)\n- `HOST_ENVD_PATH` - Envd binary path (default: `../envd/bin/envd`)\n- `HOST_KERNELS_DIR` - Kernel versions dir (local: `{storage}/kernels`, prod: `/fc-kernels`)\n- `FIRECRACKER_VERSIONS_DIR` - Firecracker versions dir (local: `{storage}/fc-versions`, prod: `/fc-versions`)\n- `ORCHESTRATOR_BASE_PATH` - Base orchestrator data (local: `{storage}/orchestrator`, prod: `/orchestrator`)\n- `SANDBOX_DIR` - Sandbox working dir (default: `/fc-vm`)\n\n## Limitations\n\n- Custom template builds support base images from the declared distro families: **Debian/Ubuntu** (apt), the **RHEL family** — Fedora, CentOS Stream, Rocky, Alma — (dnf/microdnf/yum), **Arch** (pacman), and **Alpine** (apk, OpenRC). The distro is resolved from the image's `/etc/os-release` `ID` — never by probing for package managers. Images without an os-release identity (distroless, scratch) or from an undeclared family are rejected fast, with a build-log error naming the reason and the supported families (see `pkg/template/build/phases/base/distro`). Minimal/restricted-repo images may fail provisioning if their repos don't carry the required packages; the failure is surfaced in the build log.\n- Sandboxes always boot the kernel E2B supplies, never one from the base image: `/lib/modules` is empty, no kernel modules can be loaded, and SELinux is disabled on the kernel command line (`selinux=0`), so a base image that ships an SELinux policy boots without it. `kernel-devel` from a distro's repos resolves against a kernel that isn't running. **RHEL** (`rhel`, incl. UBI), **Oracle Linux** (`ol`) and **Amazon Linux** (`amzn`) are therefore not accepted even though they are RPM/dnf images — they are chosen for kABI, signed kernel modules and UEK, which this cannot honour. The community rebuilds above are supported because they are chosen for the userland.\n","packages/orchestrator/cmd/simulate-gcs-traffic/readme.md":"To run inside of a host in GCS, trust this fancy one-liner:\n\n host=@ && bucket= && go build -o simulator . && ssh $host rm ./simulator && scp ./simulator $host:~/simulator && ssh $host -- ./simulator -csv-path gcs.csv -test-duration=5s $bucket && ssh $host cat gcs.csv\n","packages/shared/pkg/apiid/README.md":"# Resource IDs at HTTP boundaries\n\n`apiid.ProjectID` and `apiid.WorkspaceID` accept a legacy UUID or the matching\npublic ID. Both inputs produce the same struct:\n\n```go\nprojectID.UUID // uuid.UUID\nprojectID.PublicID // canonical prj_ string\n```\n\nThe existing `pkg/id` codecs remain unchanged. These structs are HTTP input\nvalues; database and domain functions can continue taking `uuid.UUID`.\n\n## Adding or changing an API ID\n\nUpdate [`id`](../id/README.md#adding-or-changing-a-resource-id) and this package\nin the same PR whenever adding a resource kind to either package. Follow the\nlinked checklist: `id` owns the codec, and `apiid` provides the matching input\nstruct and binding support. Do not add an API-only encoding or a new codec\nwithout its API type. Keep existing kinds' encoding and validation changes,\ntests, and documentation synchronized across both packages.\n\n## OpenAPI and generated Gin handlers\n\nUse the standard oapi-codegen type override on a string schema:\n\n```yaml\nprojectID:\n name: projectID\n in: path\n required: true\n schema:\n type: string\n description: Project UUID or prj_ public ID.\n x-go-type: apiid.ProjectID\n x-go-type-import:\n name: apiid\n path: github.com/e2b-dev/infra/packages/shared/pkg/apiid\n```\n\nRegenerate the server and dependent clients using the API's generation command.\nNo middleware, registry, or generator fork is needed:\n\n```go\nfunc (s *Server) GetProject(c *gin.Context, projectID api.ProjectID) {\n project, err := s.projects.Get(c.Request.Context(), projectID.UUID)\n // Handle err and build the response as usual.\n}\n```\n\nUse `apiid.WorkspaceID` for workspace parameters. Apply the same schema override\nto query or header parameters and to JSON body properties or array items:\n\n```yaml\nworkspace_id:\n type: string\n description: Workspace UUID or wrk_ public ID.\n x-go-type: apiid.WorkspaceID\n x-go-type-import:\n name: apiid\n path: github.com/e2b-dev/infra/packages/shared/pkg/apiid\n```\n\nAfter `c.ShouldBindJSON(&body)`, use `body.WorkspaceId.UUID` and\n`body.WorkspaceId.PublicID`. Keep response UUID fields unchanged unless a\nresponse-contract change is intended. Do not keep `format: uuid` on inputs\nthat also accept public IDs.\n\n## Handwritten Gin binding\n\n```go\nvar params struct {\n ProjectID apiid.ProjectID `uri:\"projectID\" binding:\"required\"`\n}\nif err := c.ShouldBindUri(¶ms); err != nil {\n c.AbortWithStatus(http.StatusBadRequest)\n return\n}\nprojectUUID := params.ProjectID.UUID\n```\n\n`ShouldBindQuery` and `ShouldBindHeader` work with `form` and `header` tags.\n`UnmarshalText` supports generated path binding; `Bind` supports oapi-codegen's\nexploded query binding; `UnmarshalParam` supports Gin's own binders. Tests cover\neach path because these binders do not all use the same interface.\n\n## Construction and encoding\n\nUse `apiid.NewProjectID(u)` or `apiid.NewWorkspaceID(u)` when constructing an ID\nin Go. Decoders and constructors populate both fields. Malformed input and\nwrong-kind public IDs fail without modifying the receiver. JSON accepts only\nstrings; encoding emits a public ID string, not a JSON object.\n\nThe fields are exported for direct reads. Treat decoded values as immutable;\nGo cannot stop a caller from manually making them inconsistent. Encoding derives\nthe public string from `UUID`, rather than trusting a manually edited `PublicID`.\nA zero-valued struct denotes an unset input; requiredness belongs in the request\nschema or Gin validation. Use a pointer for optional inputs when absence matters.\n\nOther APIs adopt the types by updating their schemas and regeneration, not by\ninstalling middleware. This package does not change their contracts automatically.\n","packages/shared/pkg/featureflags/README.md":"# LaunchDarkly client authentication\n\nThe client selects authentication in this order:\n\n1. A non-empty `LAUNCH_DARKLY_API_KEY` selects SDK-key authentication and ignores\n the OAuth variables.\n2. Without an SDK key, OAuth variables select OAuth client-credentials authentication.\n Incomplete OAuth configuration fails startup.\n3. Without either, the client uses the offline flag store.\n\nFor direct LaunchDarkly access, set `LAUNCH_DARKLY_API_KEY`. An optional\n`LAUNCH_DARKLY_BASE_URL` redirects streaming, polling, events, and diagnostics\nto a Relay proxy.\n\nFor an OAuth gateway, leave `LAUNCH_DARKLY_API_KEY` unset and configure:\n\n| Variable | Value |\n| --- | --- |\n| `LAUNCH_DARKLY_BASE_URL` | HTTPS gateway URL |\n| `LAUNCH_DARKLY_OAUTH_TOKEN_URL` | HTTPS OAuth token endpoint |\n| `LAUNCH_DARKLY_OAUTH_CLIENT_ID` | Deployment client ID |\n| `LAUNCH_DARKLY_OAUTH_CLIENT_SECRET` | Deployment client secret |\n| `LAUNCH_DARKLY_OAUTH_AUDIENCE` | Optional audience parameter for the token request |\n\nUse the existing deployment OAuth credentials and audience when the gateway\naccepts that identity. The token endpoint must support the `client_credentials`\ngrant with credentials in the request body.\n\nBoth client constructors enable live flag updates without an SDK key in OAuth\nmode.\nOAuth URLs must not contain embedded credentials, query parameters, or fragments.\n\nThe client caches tokens and refreshes them before expiry. Every new HTTP\nrequest uses the current bearer token, including stream reconnections, events,\nand diagnostics. The gateway must bound stream duration because an open stream\ndoes not reauthenticate when its token expires.\n\nToken requests have a five-second timeout. OAuth requests do not follow\nredirects. Authentication failures do not switch between SDK-key and OAuth modes.\n\nThe client mints its first token at startup. If the token endpoint answers with\na non-5xx response, construction fails, including with `WithStartOnInitTimeout`.\nIf the token endpoint is unreachable or returns 5xx, the SDK retries and the\nexisting initialization timeout policy applies.\n\nA gateway `401` or `403` before the gateway has accepted any request stops the\nSDK, as a rejected SDK key does. After the first accepted request, the client\ndrops its cached token on `401` or `403` and the SDK retries with a fresh one.\nThe SDK's own backoff bounds the retry and mint rate.\n","packages/shared/pkg/id/README.md":"# Typed public IDs\n\nThis package converts resource UUIDs to and from human-readable public IDs:\n\n```text\nprj_7mfb0gpp6c8dsaasre0asc7n3s\nwrk_2n1t201rmv87aae5j4csam8000\nsec_01kyjhkfvse96rg4d2qzd9enft\n```\n\nEvery public ID has a stable resource prefix, an underscore, and a\n26-character UUID suffix. E2B's public representation is always lowercase.\n\n## UUID ownership\n\nResource owners remain responsible for generating and storing UUIDs. UUIDs\nmay, for example, be generated by application code or a database. This package\ndoes not use `typeid.New` or `typeid.MustNew`.\n\n[`github.com/sumup/typeid`](https://github.com/sumup/typeid) is used only to\nconvert an existing UUID to its prefixed public representation and parse that\nrepresentation back into the UUID-backed resource type. PostgreSQL and other\npersistence layers continue to store the underlying UUID, not the prefixed\nstring.\n\n```go\nprojectUUID := uuid.New()\nprojectID := id.ProjectID(projectUUID)\n\npublicID := projectID.String()\nparsedID, err := id.ParseProjectID(publicID)\nif err != nil {\n\treturn err\n}\n\nprojectUUID = uuid.UUID(parsedID)\n```\n\nProject and workspace UUIDs are normally UUIDv4 and use `typeid.Random`.\nSecret UUIDs are UUIDv7 and use `typeid.Sortable`. SumUp renders Random IDs in\nuppercase, so the E2B wrapper normalizes them to the lowercase public contract.\n\n## Team IDs\n\nTeams are projects: a team UUID is also its project UUID.\n`ConvertTeamIDToProjectID` renders a team UUID as its public `prj_` ID, and\n`ParseTeamID` accepts either spelling wherever an API takes a team ID.\n\n## Adding or changing a resource ID\n\nKeep this package and [`apiid`](../apiid/README.md) in sync in the same PR.\nAdding a resource kind here requires its matching API input type in `apiid`;\nadding a type in `apiid` requires a matching codec here. Changes to an existing\nkind's prefix, encoding, or validation must also update the affected API type\nand tests. Encoding rules belong here; `apiid` must reuse them, not duplicate them.\n\nTo add another public resource ID:\n\n1. Choose a short, stable, lowercase prefix that keeps the full public ID at\n the expected fixed width.\n2. Add a private prefix type implementing `Prefix()`.\n3. Select `typeid.Random` for a UUIDv4 resource or `typeid.Sortable` for a\n UUIDv7 resource.\n4. Add a distinct UUID-backed public Go type with `String` and parse methods.\n5. Keep UUID generation in the resource-owning application or database.\n6. Test the exact prefix, lowercase output, fixed length, invalid-prefix\n rejection, and UUID round trip.\n7. Add the matching `apiid` struct with `UUID` and `PublicID`, its constructor,\n and its text, JSON, generated-query, and Gin binding methods.\n8. Test that UUID and public inputs produce identical API structs, wrong kinds\n are rejected, and encoding agrees with this package's codec. Update both\n READMEs and regenerate affected API contracts and clients.\n\nDo not add JSON, text, SQL, or UUID-generation behavior implicitly. Those\ninterfaces should be introduced only when a concrete caller needs them.\n","packages/shared/pkg/sandboxtypes/README.md":"# Shared sandbox types\n\nThis package owns `RuntimeMetadata`, `SandboxType`, and `EgressClass`, including\nidentity logging and the empty-type fallback to a regular sandbox. It contains\nno worker resources, network initialization, or VM lifecycle management.\n\nCallers import these types directly. `SandboxType.EgressClass()` selects the\ntraffic class; DSCP configuration and socket handling belong to worker networking.\nThe resource-owning `Sandbox` stays in the orchestrator package. Runtime team\nmetadata is attribution, not trusted authorization data.\n"},"files":{"AGENTS.md":"# AGENTS.md\n\nInstructions for AI agents working in this repository.\n\nAlso read [CLAUDE.md](CLAUDE.md) for development commands and package\nconventions if it is not already in context. It applies to all agents.\n\n## Understanding the repository\n\nRead [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) before working on or reviewing code here. It describes what each service does, how services interact (sandbox creation, traffic routing, pause/resume, template builds), the data stores, and the deployment topology — it is the fastest way to build correct context about this codebase.\n\nKeep that document updated: if a code change alters anything it describes (service responsibilities, ports, protocols, data stores, flows, deployment topology), update `docs/ARCHITECTURE.md` as part of the same change. When reviewing a PR that changes such details without updating the document, flag it.\n\n## Review guidelines (pull requests)\n\nOutput style rules (strict, override any defaults):\n\n- Only report concrete bugs, regressions, or correctness issues. Do not summarize the PR, the diff, or what changed.\n- One short paragraph per finding. No preamble, no closing remark.\n- No headers, no bullet lists, no tables, no diagrams.\n- No emojis. No severity tags (\"bug\", \"nit\", \"suggestion\", \"enhancement\"). No labels.\n- No branding or footer lines.\n- If there are no real issues to flag, post no review at all.\n\n## Scope\n\n- Skip style/nit comments — `golangci-lint` covers those.\n- Skip test-coverage comments.\n- Focus on: race conditions, nil-deref, error handling, auth/authz, request routing, resource leaks, SQL/migration correctness, and gRPC/proto compatibility.\n","CLAUDE.md":"# CLAUDE.md\n\nThis development guide applies to Codex, Claude Code, and OpenCode.\nRepository and review rules are shared through:\n\n@AGENTS.md\n\n## Project Overview\n\nE2B Infrastructure is the backend infrastructure powering E2B (e2b.dev), an open-source cloud platform for AI code interpreting. It provides sandboxed execution environments using Firecracker microVMs.\n\n**Start with [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** — it explains what each service does, how services interact (with diagrams of the core flows: sandbox creation, traffic routing, pause/resume, template builds), and the deployment topology. Reading it first is the fastest way to understand this repository.\n\n**Keep docs/ARCHITECTURE.md updated**: if your change alters anything it describes (service responsibilities, ports, protocols, data stores, flows, deployment topology), update the document in the same change.\n\n## Common Development Commands\n\n### Setup & Environment\n```bash\n# Switch between environments (prod, staging, dev)\nmake switch-env ENV=staging\n\n# Setup local development stack (PostgreSQL, Redis, ClickHouse, monitoring)\nmake local-infra\n```\n\n### Building & Testing\n```bash\n# Run all unit tests across packages\nmake test\n\n# Run integration tests\nmake test-integration\n\n# Build specific package\nmake build/api\nmake build/orchestrator\n\n# Generate code (proto, SQL, OpenAPI)\nmake generate\n\n# Format and lint code\nmake fmt\nmake lint\n\n# Regenerate mocks\nmake generate-mocks\n\n# Tidy go dependencies\nmake tidy\n```\n\n### Running Services Locally\n```bash\n# From packages/api/\nmake run-local # Run API server on :3000\nmake dev # Run with air (hot reload)\n\n# From packages/orchestrator/\nmake run-local # Run orchestrator\nmake run-debug # Run with race detector\n```\n\n### Package-Specific Commands\n```bash\n# API: Generate OpenAPI code\ncd packages/api && make generate\n\n# Orchestrator: Generate proto + OpenAPI\ncd packages/orchestrator && make generate\n\n# DB: Run migrations\nmake migrate\n\n# Run single test\ncd packages/ && go test -v -run TestName ./path/to/package\n```\n\n### Publishing Artifacts\n```bash\n# Build and upload all service images to your GCP project\nmake build-and-upload\n\n# Build specific service\nmake build-and-upload/api\nmake build-and-upload/orchestrator\n```\n\n## Architecture Overview\n\n### Service Communication Flow\n```\nClient → Client-Proxy → API (REST) ⟷ PostgreSQL\n ↓ ⟷ Redis\n Orchestrator ⟷ ClickHouse\n ↓ (gRPC)\n Firecracker VMs\n ↓\n Envd (in-VM daemon)\n```\n\n### Core Services\n\n**API (`packages/api/`)** - REST API using Gin framework\n- Entry point: `main.go`\n- Core logic: `internal/handlers/store.go` (APIStore)\n- Authentication: API keys and OIDC auth provider JWTs\n- OpenAPI code generation: `internal/api/*.gen.go`\n- Port: 80\n\n**Dashboard API (`packages/dashboard-api/`)** - REST backend for the web dashboard, not the SDK\n- Entry point: `main.go`\n- Spec: `spec/openapi-dashboard.yml`; template tags, `/v1/management`\n- Talks to Postgres and ClickHouse; never to orchestrators (see `docs/ARCHITECTURE.md`)\n- Port: 3010\n\n**Orchestrator (`packages/orchestrator/`)** - Firecracker microVM orchestration\n- Entry point: `main.go`\n- VM management: `pkg/sandbox/`\n- Firecracker integration: `pkg/sandbox/fc/`\n- Networking: `pkg/sandbox/network/`\n- Storage: `pkg/sandbox/nbd/` (Network Block Device)\n- Template caching: `pkg/sandbox/template/`\n- Template building: `pkg/template/`\n- gRPC server: `pkg/server/`\n- Utilities: `cmd/clean-nfs-cache/`, `cmd/inspect-build/`, `cmd/dummy-orchestrator/`\n\n**Envd (`packages/envd/`)** - In-VM daemon using Connect RPC\n- Runs inside each Firecracker VM\n- Process management API: `packages/envd/spec/process/process.proto`\n- Filesystem API: `packages/envd/spec/filesystem/filesystem.proto`\n- Port: 49983\n- **`pkg/version.go` is release-please's**: it is set from the Conventional Commits merged since the last envd release, so do not bump it by hand. A behavioral change ships under a `feat` or `fix` type so a version exists for the orchestrator to gate on; comment or docs-only changes use `docs`, which the changelog hides, and cut no release (`chore` and `refactor` are visible and cut one)\n\n**Client Proxy (`packages/client-proxy/`)** - Edge routing layer\n- Service discovery via `packages/shared/pkg/servicediscovery`\n- Request routing to orchestrators\n- Redis-backed state management\n\n**Shared (`packages/shared/`)** - Common utilities\n- Proto definitions: `pkg/grpc/orchestrator/`, `pkg/grpc/envd/`\n- Telemetry: `pkg/telemetry/` (OpenTelemetry)\n- Logging: `pkg/logger/` (Zap + OTEL)\n- Database: `pkg/db/` (ent ORM)\n- Models: `pkg/models/`\n- Storage: `pkg/storage/` (GCS/S3 clients)\n- Feature flags: `pkg/featureflags/` (LaunchDarkly)\n\n**Database (`packages/db/`)** - PostgreSQL layer\n- Migrations: `migrations/*.sql` (goose)\n- Queries: `queries/*.sql` (sqlc)\n- Generated code: `queries/` (plus `pkg/auth/queries/`, `pkg/dashboard/queries/`)\n\n### Key Technologies\n\n- **go 1.26.10** with workspaces (`go.work`)\n- **Firecracker** for microVM virtualization\n- **PostgreSQL** for primary data (sqlc for queries)\n- **ClickHouse** for analytics\n- **Redis** for caching and state\n- **OpenTelemetry** for observability (Grafana stack: Loki, Tempo, Mimir)\n- **gRPC/Connect RPC** for service communication\n- **Gin** (API), **chi** (Envd) for HTTP\n\n### Code Generation\n\nThe codebase uses several code generators:\n\n1. **Protocol Buffers** (`packages/orchestrator/generate.Dockerfile`)\n - Generates: `packages/shared/pkg/grpc/*/`\n - Run: `make generate/orchestrator`\n\n2. **OpenAPI** (`oapi-codegen`)\n - Spec: `spec/openapi.yml`\n - Generates: API handlers, types, specs\n - Run: `make generate/api`\n\n3. **SQL** (`sqlc`)\n - Queries: `packages/db/queries/*.sql`\n - Generates: Type-safe DB code\n - Run: `make generate/db`\n\n4. **Mocks** (`mockery`)\n - Config: `.mockery.yaml`\n - Run: `make generate-mocks`\n\n### Testing Patterns\n\n- **Unit tests**: Use `testify/assert` and `testify/require`\n- **Database tests**: `packages/db/pkg/testutils.SetupDatabase` gives each test its own database on the PostgreSQL server `PG_TEST_DATABASE_URL` names (see [DEV-LOCAL.md](DEV-LOCAL.md#unit-tests-that-need-postgresql))\n- **Integration tests**: `tests/integration/` with shared test utilities\n- **Mocking**: Generated mocks in `mocks/` directories\n- **Race detection**: Tests run with `-race` flag\n\nExample test invocation:\n```bash\n# Single package\ngo test -race -v ./internal/handlers\n\n# Specific test\ngo test -race -v -run TestCreateSandbox ./internal/handlers\n```\n\n## Important Development Notes\n\n### Working with Proto/gRPC\n- Proto files: `packages/envd/spec/process/`, `packages/envd/spec/filesystem/`, orchestrator protos at `packages/orchestrator/*.proto`\n- Shared protos: `packages/shared/pkg/grpc/`\n- After editing proto files, run `make generate/orchestrator` and `make generate/shared`\n\n### Database Migrations\n- Migrations: `packages/db/migrations/`\n- Create: `cd packages/db && make create-migration NAME=your-migration-name` — this generates the file with a correct `YYYYMMDDHHMMSS` timestamp. Do NOT hand-create migration files; placeholder timestamps like `120000`/`000000` cause same-day version collisions and are rejected by the out-of-order-migrations CI check.\n- Apply: `make migrate` (requires POSTGRES_CONNECTION_STRING)\n- Code generation: `make generate/db` (regenerates sqlc code)\n\n### Environment Variables\n- Local development defaults: `packages//.env.local`; see [DEV-LOCAL.md](DEV-LOCAL.md).\n- Self-hosting configuration: [embed/compose/.env](embed/compose/.env); see [embed/README.md](embed/README.md).\n- Custom environment configs: `.env.`, selected with `make switch-env ENV=`.\n\n### Firecracker & VM Management\n- Orchestrator requires **sudo** to run (Firecracker needs root)\n- VM networking uses `iptables` and Linux `netlink`\n- Storage uses NBD (Network Block Device)\n- Templates cached in GCS bucket (configurable via TEMPLATE_BUCKET_NAME)\n- Kernel/Firecracker versions: `packages/fc-versions/`\n\n### Observability\n- All services export OpenTelemetry traces/metrics/logs\n- Local stack includes Grafana + Loki + Tempo + Mimir\n- Telemetry setup: `packages/shared/pkg/telemetry/`\n- Logger: `packages/shared/pkg/logger/` (Zap with OTEL)\n- Profiling: API exposes pprof on `/debug/pprof/` (see `packages/api/Makefile` profiler target)\n\n### CI/CD Workflows\n- `.github/workflows/pull-request.yml` - PR orchestrator (lint, OpenAPI, unit, arm64, integration)\n- `.github/workflows/pr-tests.yml` - Unit test shards\n- `.github/workflows/integration_tests.yml` - Integration test suite\n\n## Architecture Patterns\n\n1. **Service Isolation**: Each service runs in containers with defined gRPC/HTTP interfaces\n2. **Shared Libraries**: Cross-cutting concerns (logging, telemetry, DB) in `packages/shared`\n3. **Event-Driven**: ClickHouse + Redis pub/sub for async operations\n4. **Caching Strategy**: Redis for templates, auth tokens, performance optimization\n5. **Feature Flags**: LaunchDarkly for gradual rollouts\n6. **Graceful Shutdown**: Services handle SIGTERM with context cancellation\n7. **Health Checks**: gRPC health protocol + HTTP health endpoints\n\n## Debugging\n\n### Logs\n- Local: Docker logs in `make local-infra`\n- Production: Grafana Loki\n","docs/ARCHITECTURE.md":"# E2B Infrastructure Architecture\n\nThis document explains what this repository implements, what each service does, and how the\nservices interact. It is the fastest way to build a mental model of the codebase before diving\ninto the code.\n\n> **Keep this document updated.** If a code change alters anything described here (service\n> responsibilities, ports, protocols, data stores, flows, deployment topology), update this file\n> in the same PR.\n\n## What this repository implements\n\nE2B provides **sandboxes**: isolated Linux VMs that start almost instantly (they resume pre-booted\nsnapshots instead of cold-booting), run arbitrary code (typically\ngenerated by AI agents), and can be paused, snapshotted, and resumed. This repo contains the whole\nbackend: the control-plane REST API, the data-plane VM orchestration built on\n**Firecracker microVMs**, the in-VM agent, the edge routing layer, and template building.\n\nTwo ideas drive the design:\n\n1. **A sandbox is a resumed snapshot.** Templates are pre-booted VM snapshots (memory + disk +\n VM state) stored in object storage. \"Creating\" a sandbox means restoring a snapshot, which is\n why startup is fast. Memory pages are loaded lazily on page-fault (userfaultfd) and the root\n filesystem is a copy-on-write overlay, so only touched data is ever fetched. A template can\n also be filesystem-only (rootfs, no memfile), taken by a filesystem-only pause or checkpoint;\n sandboxes created from it cold-boot from disk instead of resuming.\n2. **Control plane and data plane are separate.** The API decides *where* a sandbox runs and\n tracks *that* it runs (Postgres/Redis); the orchestrator on each node owns *how* it runs\n (Firecracker, networking, storage). Sandbox traffic never passes through the API.\n\n## System overview\n\n```mermaid\nflowchart TB\n subgraph clients[\"Clients\"]\n SDK[\"SDK / CLI\"]\n Browser[\"Browser / HTTP clients\"]\n end\n\n LB[\"Load balancer
api.* | *.domain wildcard\"]\n\n VC[\"volume-content API (belt)
api.<domain>\"]\n\n subgraph controlplane[\"Control plane (API node pool)\"]\n API[\"API
REST :80, gRPC :5009/:5109\"]\n DashAPI[\"dashboard-api :3010\"]\n CP[\"client-proxy
:3002\"]\n end\n\n subgraph datastores[\"State\"]\n PG[(\"PostgreSQL
teams, templates, builds, snapshots\")]\n RD[(\"Redis
running sandboxes, routing catalog, caches\")]\n CH[(\"ClickHouse
metrics, events, optional logs\")]\n OS[(\"Object storage GCS/S3
template + snapshot artifacts\")]\n end\n\n subgraph clientnode[\"Sandbox nodes (one orchestrator per node)\"]\n ORCH[\"orchestrator
gRPC :5008, proxy :5007\"]\n subgraph vm[\"Firecracker microVM (per sandbox)\"]\n ENVD[\"envd :49983\"]\n USERPROC[\"user processes\"]\n end\n end\n\n subgraph buildnode[\"Build nodes\"]\n TM[\"template-manager
(orchestrator binary, gRPC :5008)\"]\n end\n\n SDK -->|REST| LB --> API\n Browser -->|\"port-sandboxid.domain\"| LB --> CP\n API -.->|\"mint content token + domain\"| SDK\n SDK -->|\"volume content (token-authed)
api.<BYOC or default domain>\"| VC\n API -->|\"gRPC Create/Delete/Pause\"| ORCH\n API -->|\"gRPC TemplateCreate\"| TM\n CP -->|\"lookup sandbox → node\"| RD\n CP -->|\"forward :5007\"| ORCH\n CP -.->|\"gRPC auto-resume\"| API\n ORCH --> ENVD\n ENVD --> USERPROC\n API --> PG & RD & CH\n DashAPI --> PG & CH\n ORCH --> OS & CH\n TM --> OS\n```\n\n## Services\n\n| Service | Package | Runs on | Purpose |\n|---|---|---|---|\n| API | `packages/api` | API nodes | Public REST API; sandbox lifecycle, placement, auth, quotas |\n| Orchestrator | `packages/orchestrator` | every sandbox node | Runs Firecracker VMs; sandbox create/pause/resume/kill |\n| Template manager | `packages/orchestrator` (role) | build nodes | Builds templates from Docker images |\n| Client proxy | `packages/client-proxy` | API nodes | Edge router: sandbox URL → correct node |\n| Envd | `packages/envd` | inside every VM | In-VM agent: process/filesystem API for SDKs |\n| Dashboard API | `packages/dashboard-api` | API nodes | Backend for the web dashboard and workspace control plane (sandbox records, admin, `/v1/management`) |\n\nSupporting packages: `packages/shared` (protos, telemetry, storage clients, feature flags),\n`packages/auth` (authentication library), `packages/db` (Postgres migrations + sqlc queries),\n`packages/clickhouse` (ClickHouse schema + clients), `packages/otel-collector` (collector config),\n`packages/nomad-nodepool-apm` (autoscaler plugin), `packages/local-dev` (local stack).\n\n### API (`packages/api`)\n\nThe control-plane entry point (Gin, OpenAPI-generated from `spec/openapi.yml`, port 80).\n\n- **Resources**: sandboxes (create/list/kill/pause/resume/connect/timeout/metrics/logs),\n templates and builds (including the web dashboard's build listings under `/unstable/builds`),\n teams, volumes, API keys, secrets, sandbox events and webhooks, admin operations.\n- **Default templates**: `env_defaults` names the platform's default templates. Admin routes\n under `/unstable/admin/templates` add, describe and remove default templates and aliases\n without a namespace, and clear the alias cache after each alias change so name resolution\n sees it at once.\n- **Spec document**: `GET /openapi.json` serves, without authentication, the spec the request\n validator runs. An `ETag` over the document answers a matching `If-None-Match` with 304. It is\n registered before the validator, which rejects undeclared paths.\n- **Auth** (via `packages/auth`): team API keys (`X-API-Key`, `e2b_` prefix), auth-provider JWTs\n (OIDC), and either an admin token or a service JWT verified from the configured admin JWKS.\n Backed by an auth DB (Postgres) with a Redis team cache.\n- **API-group rate limits**: operations declare a group through the `x-api-group` OpenAPI\n extension. After authentication succeeds, its callback wrapper reads the validator's\n already-matched operation and stores the API group directly in Gin's context. The rate limiter\n reads that group and maps it to a field in the authenticated team's limits. Grouped operations\n require team-identifying authentication; public operations do not populate an API group.\n The `list` group uses `api_team_rps_list`; the `delete` group uses `api_team_rps_delete` only\n for sandbox and template deletion. Other DELETE operations are ungrouped.\n Delete rates default to zero until configured.\n A group's rate is one of the team's limits and resolves with the rest of them in the statement\n that loads the team, which authentication caches; it has no source of its own.\n Propagated project limits carry both rates, and each accepted revision replaces\n them. A team with no `project_limits` row uses the deployment's `default_limits`; zero\n disables group limiting. The migration that added `default_limits` gave every existing team a\n row holding its limits at the time, so a change to the defaults reaches only teams created since.\n A positive rate sets both requests per second and burst capacity. Redis enforces a separate\n budget for each team, HTTP method, and route template across API replicas: groups select\n rates, but routes do not share a budget. Exhausted budgets return HTTP 429 with rate-limit\n and `Retry-After` headers. Ungrouped operations and unconfigured rates bypass group limiting;\n Redis failures are logged and counted while requests proceed.\n V1 route limits from `rate-limit-config` are always enforced first. For requests that pass V1,\n LaunchDarkly's `rate-limit-v2-mode` is the master switch for all V2 checks, and\n `rate-limit-delete-mode` can further restrict delete checks. Both default to `disabled`;\n missing, invalid, or unknown modes are treated as disabled. Deletes skip V2 if either flag is\n disabled and enforce only when both are enabled. If both are active and either is `shadow`,\n deletes run in shadow mode. Lists follow the global mode.\n The groups use the same limiter logic: `shadow` observes V2 decisions while\n preserving V1 headers, and `enabled` also enforces V2. Ungrouped operations, unconfigured V2\n rates, and V2 Redis errors retain V1 enforcement. A rejection reports the rejecting limiter's\n headers; when both checks allow a request in enabled mode, V2 supplies the rate-limit headers.\n- **Sandbox list snapshots**: a list request reads the team's whole sandbox index from Redis\n (its running and pausing records) and pages in memory, so with the LaunchDarkly\n `sandbox-list-max-lag-ms` flag set for a team, each API replica shares one read between\n that team's concurrent list requests and keeps it for the flag's value, counted from the\n start of the read; a request that joins a read already in flight gets that read's result\n however long it takes. A lowered flag applies to a live snapshot; a raised one from the\n next read. Within that lag a sandbox created, killed, paused or resumed can be missing from\n the list or shown in its previous state on the replica holding the snapshot, and one\n resumed inside it can appear on two pages of a walk served by different replicas. The\n default, zero, gives every request its own read with the same store commands as before.\n Only the two list endpoints use the snapshot; the evictor, reconcile, template delete and\n team teardown keep reading the store directly.\n- **Workload identity**: sandbox create accepts an optional `iam.tokens` map of caller-named\n workload-token definitions (each an exact `audience` and `tokenType`). A non-empty, validated\n map enables workload identity, whose identity the orchestrator derives from the sandbox's\n already-authoritative team/sandbox/execution/template IDs; the definitions are passed to the\n orchestrator in `SandboxConfig.iam`. The API mints no credential and delivers nothing into the\n sandbox; file-based delivery is rejected at admission. Definitions are persisted in the\n running-sandbox (Redis) and paused-snapshot (Postgres) state so they survive pause/resume and\n orchestrator re-sync; a fork starts a new workload and does not inherit them.\n- **Placement**: keeps a live map of orchestrator nodes (discovered via Nomad, Kubernetes, or a\n static list). Chooses a node per sandbox with a **best-of-K** algorithm\n (`internal/orchestrator/placement/`): sample K ready nodes, score by the\n higher of CPU load and hugepage-pool load, pick the lowest; retry on\n exhausted nodes. A node that refuses a create (`ResourceExhausted`) stays\n eligible, and each refusal waits a full-jitter backoff that doubles up to a\n cap, so a placement that finds every node busy does not re-send creates as\n fast as they are refused. The hugepage-pool load is measured against the node's\n base pool (`nr_hugepages`, allocated at boot), so a node that has borrowed surplus\n pages scores above 1 instead of reading as just full; a pool that is all overcommit\n is measured against its ceiling, and an orchestrator that does not report the base\n pool against `HugePages_Total` as before. A node that reports no pool scores 0.5. Hugepage scoring\n is on by default (`BEST_OF_K_HUGEPAGE_MEMORY`); set it false to rank on\n CPU alone. K, overcommit ratio, and alpha are tunable live via feature\n flags. With `placement-hugepage-limit-percent` above 0, a hugepage sandbox\n is also refused by any node whose committed hugepages would pass that share\n of the node's pool ceiling (base pool plus overcommit allowance, reported by\n the orchestrator); within an API replica the check and the in-flight record\n are one step per node, and when the compatible nodes are all full placement\n answers with a capacity error, not \"no compatible node\". With the\n orchestrator's `create-memory-refusal` flag on (`CREATE_MEMORY_REFUSAL` for a\n deployment without LaunchDarkly), a create whose guest memory the pool\n cannot back while a sandbox on that node is still tearing down is refused\n with `ResourceExhausted`, so placement retries the node once the pages are\n back; with nothing tearing down it fails at once and says why. Off, it fails\n as an internal error and the node is excluded.\n- **State**: writes sandbox records to Redis (source of truth for *running* sandboxes). It does\n not write routing: client-proxy reads the orchestrator-written `sandbox:routing:{id}` (see\n \"Sandbox routing records\"). Persistent entities (templates, builds, snapshots, teams) live in\n Postgres.\n- **Secrets**: `/secrets` is the only public surface for secret management (create, list, get,\n update, delete). The API authenticates the caller with the customer alternatives above, converts\n the authenticated team UUID to the project UUID the backend knows, checks the `customer-secrets`\n feature flag, and forwards a metadata-only request over a unary gRPC contract\n (`e2b.secretsstore.management.v1`) to the secrets store backend named by\n `SECRETS_STORE_BACKEND_GRPC_ADDRESS`. No caller credential, header or client-supplied tenant\n crosses that hop, and no response - here or in a log, a span or an error - carries a secret\n value. What a caller stores is a runtime marker, not a resolved secret; orchestrator-ee resolves\n a marker to a value at sandbox egress, never here. Without a configured address, or with the flag\n off, the routes stay registered and answer 403. Responses are `Cache-Control: no-store`, request bodies are capped at 512 KiB, and values\n at 64 KiB. A create carries the cap on live secrets the backend enforces: `max_secrets` is one\n of the team's limits and resolves with the rest of them in the statement that loads the team (its\n `project_limits` value, else the deployment's `default_limits`).\n- **Sandbox events and webhooks**: `/events/sandboxes` reads a team's sandbox lifecycle events,\n and `/events/webhooks` manages the webhook subscriptions those events are delivered to, along\n with each subscription's delivery attempts and delivery statistics. The API stores none of\n this: it authenticates the caller, resolves the team's cluster, and forwards the request over\n two unary gRPC contracts (`e2b.webhooks.events.v1` and `e2b.webhooks.management.v1`) to the\n backend named by `WEBHOOKS_BACKEND_GRPC_ADDRESS`. The team and cluster come from the\n authenticated context, never from the request, and a team with no cluster selects the local\n one. Without a configured address the routes stay registered and answer 503, so the surface\n does not depend on how a deployment is wired. A backend failure the caller cannot act on is\n answered with the endpoint's own message; the outcomes that keep their identity are\n not-found, a rejected request, the webhook-count limit, and a timeout. A create or update\n body carries the webhook's signing secret, so a request rejected before the handler has its\n body kept out of the response, the access log and the span. Reads are capped at 100\n deliveries per page.\n- **Rig passthrough**: admin endpoints under `/clusters/{clusterID}/rigs` manage a cluster's\n orchestrator node pools (\"rigs\"). They list rigs, change rig capacity, list and terminate\n instances, and read recent scaling errors. Every handler delegates to the cluster's edge service\n (`/v1/rigs/...`) through the per-cluster HTTP client that the cluster sync keeps fresh. The API\n holds no cloud credentials and makes no scaling decisions. The caller authenticates with an\n admin credential and never holds the cluster's edge secret. Edge status codes pass through unchanged\n (400, 404, 409, 501). An edge 401 becomes a 500: it means the API is misconfigured against the\n cluster, not that the caller's token is invalid. A cluster with no rig management configured\n returns an empty list. The local cluster has no edge deployment and answers 501 for every rig\n operation.\n- **Extra listeners**: internal gRPC :5009 and edge gRPC :5109 expose `ResumeSandbox` so\n client-proxy can wake paused sandboxes on incoming traffic.\n- **Outbox** (`internal/outbox`): a River client on the shared database's `river` schema\n works jobs that need the orchestrator connections. Its job arguments live in\n `packages/db/pkg/outbox`. The one kind today, `teardown_team_resources`, stops a deleted\n team's workloads; the API's own project deletion enqueues it. A\n guard runs first on every attempt: a team that exists and is not blocked fails the attempt,\n and the job retries rather than kill a live team's sandboxes; a missing team row does not\n stop it. Then\n `kill_sandboxes` kills the team's running sandboxes and succeeds only once the sandbox store\n holds none of them in any state, so a sandbox still pausing or being killed makes the job\n retry. A sandbox that is not running cannot be paused, so no new snapshot of the team\n appears afterwards. Killing through the orchestrator is what removes the sandboxes' store\n records, so the job deletes no Redis state itself, and every attempt is safe to run again.\n Builds are not cancelled: they end on their own build timeout. An attempt is bounded at\n 30 minutes; retries back off from 30 seconds to a 15-minute cap, so the 100 attempts last\n about a day before River discards the job and keeps it. Volumes, templates, snapshots and\n the team row are left alone. Steps report\n `api.outbox.steps.finished` and `api.outbox.step.duration`; finished jobs report\n `outbox.jobs.finished`, and the backlog reports `outbox.jobs` and `outbox.oldest_*_age`.\n `OUTBOX_MAX_WORKERS` (default 10) and `OUTBOX_BACKLOG_INTERVAL` (default 30s) configure it;\n shutdown stops it after the sandbox-work drain, before the database and Redis clients close.\n- Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to\n Loki, with a LaunchDarkly-gated ClickHouse read path (`logs-read-config`) for local-cluster logs\n during the log storage migration. `LOKI_URL` is optional: without it the api has no Loki client\n and those reads fail until the flag routes them to ClickHouse. `LOGS_READ_CONFIG=true|false` sets\n the value the flag falls back to when LaunchDarkly has none (no LaunchDarkly, or the flag not\n defined there); a LaunchDarkly value wins. LaunchDarkly feature flags also gate placement\n parameters, rate limits, and rollouts.\n\n#### Project management writes (`/unstable/management`)\n\n`PUT /unstable/management/projects/{projectID}/settings` applies a revisioned settings\nprojection from the owning management service. This endpoint belongs to the API\nand accepts only service JWTs through `AdminJWTAuth`. It locks the team and checks\nthat it has not been deleted before applying the projection. The shared `ProjectSettings` type\ncontains `SandboxSettings`, with the optional ingress default under\n`sandboxSettings.defaultAllowPublicTraffic`. Storage uses `projection.project_settings`,\nwith `revision` and nullable `sandbox_default_allow_public_traffic` in the same row.\nThe receiver applies both in one guarded upsert. Equal or older revisions cannot\noverwrite it. Clearing stores NULL at a newer revision, retaining the ordering fence.\nCache invalidation follows commit and is retried on duplicate\ndelivery after a cache failure. There is no rollout flag check on accepted delivery.\nSandbox creation resolves an absent default to true and honors a saved default\nindependently of rollout eligibility. New sandboxes can use an older value while\ndelivery or cache refresh is pending.\n\nThe owning management service also drives the project writes under\n`/unstable/management/projects/{projectID}`. Like the settings route, each accepts only service\nJWTs through `AdminJWTAuth`, and each is idempotent, because the caller is level-triggered and\nretries.\n\n- `PUT /unstable/management/projects/{projectID}` creates or reconciles a project and answers 201\n or 200 to say which. A project is a `public.teams` row created from a caller-supplied UUID, with\n no tier. An optional cluster ID is stored on the project; omitting it preserves the current\n assignment. A changed slug renames the project and, in the same transaction, gives each of its\n namespaced template aliases under the new slug as well, keeping the old ones and leaving an\n alias the new slug already holds as it is. It answers 409 when another project already holds the slug, when the\n request assigns a cluster whose deletion is recorded, or when a concurrent request created the\n same project first.\n- `PUT .../members/{userID}` applies the desired presence for one user, gated by a monotonic\n per-project/user revision in `projection.project_members`; a duplicate or older revision\n succeeds without changing target state. A present projection carries the user's OIDC\n issuer/subject identities. Every projected user has at least one, and an identity already owned\n by a different user returns 409. A revocation removes only that user's `users_teams` row;\n projected users and identities are retained. Auth caches member authorization, so each accepted\n command evicts that user's authorization for the project after commit.\n- `PUT .../limits` syncs the project's effective limits into `project_limits`, which the team\n lookups read in preference to `default_limits`. It is gated by a monotonic per-project revision\n in `projection.project_limits`: the caller raises it whenever the limits it resolved change, and\n a delivery at or below the recorded revision is dropped and still answers 204. The ledger and the\n values advance in one transaction, so a revision is never recorded without the values it\n admitted.\n- `PUT .../block` applies a revision-fenced block state: the `projection.project_blocks` revision\n and `teams.is_blocked` / `blocked_reason` commit together, and an older or duplicate revision\n leaves stored state unchanged. Every accepted delivery, a dropped one included, evicts the team's\n auth cache; a failed eviction answers an error so the retry finishes it after commit. The API\n reads this state when admitting billable work. Apart from project deletion, which blocks the\n team for good, this is the only route that writes it.\n- `DELETE /unstable/management/projects/{projectID}` soft-deletes the project, described below.\n\nThe revision fences are the target's for the same reason: the caller can only fence what it\nsends, so two deliveries in flight arrive in whichever order the network gives them, and the older\none has to be refused where it lands. The project, membership, limits and block writes live in\n`internal/management` with their post-commit cache evictions rather than in the handlers.\n\nA limits or block delivery may carry `decided_at`, when the caller decided that revision; the\nledger row stores it beside the revision. Each delivery the ledger accepts records the seconds\nsince then in `dashboard-api.management.apply_lag` with `projection` `project_limits` or\n`project_blocks`; one without `decided_at`\ncounts in `dashboard-api.management.origin_unknown` instead, never as zero lag. Dropped, refused\nand failed deliveries record nothing. The names keep the dashboard-api prefix from when\ndashboard-api served these writes, so the alert and dashboard reading them carry on unchanged.\n\nProject deletion soft-deletes any project, including a member's default project; whether a project\nmay be deleted is the caller's decision. `envs`, `snapshots` and `volumes` reference `teams` with\n`ON DELETE NO ACTION`, so the team row stays. One transaction:\n\n- locks the row (404 if it is missing) and sets `teams.deleted_at`;\n- blocks the team (a CHECK keeps every deleted team blocked) and clears its cluster assignment;\n- scrubs its name and email to `DELETED` and its slug to `deleted-`, releasing the\n old slug for reuse;\n- deletes its memberships and API keys;\n- soft-deletes its templates, snapshot templates and paused-sandbox snapshots the way a template\n deletion does, releasing their aliases and build tracking;\n- enqueues `teardown_team_resources` with `InsertTx` on the outbox's own River client, so the\n teardown runs exactly when the deletion commits.\n\nAfter commit it evicts the deleted keys' auth cache entries best-effort; the team's other cached\nentries refresh or expire on their own. Repeating the call answers 204 and runs every step again,\nkeeping only the first deletion's time. That undoes a block reason, rename or cluster assignment\nwritten by a delivery that passed its deleted-project check just before the first call committed;\ndeletes memberships, keys and envs created since (a sandbox that paused before the teardown killed\nit leaves a snapshot); and enqueues the teardown again, which is a no-op while one is still open.\n\nAfterwards the API's deleted-project middleware, running after authentication, answers 404 to every\nnon-GET route under `/unstable/management/` that names `{projectID}`: the upsert, member, limits,\nblock and settings writes. The DELETE is exempt so a retry still reaches it. dashboard-api guards its own\nmanagement and admin writes the same way. Shared auth refuses a membership or API key still left\nfor the project with 403, and the API leaves the project out of a user's team list. Volumes, builds\nand stored artifacts are not reclaimed.\n\n#### Team bans (`/unstable/admin/teams`)\n\n`PUT /unstable/admin/teams/{teamID}/ban` sets `teams.is_banned`, evicts the team's auth cache so\nits API keys stop authenticating, then kills its running sandboxes and cancels its builds through\nthe same orchestrator and template-manager calls as `POST /admin/teams/{teamID}/sandboxes/kill`\nand `.../builds/cancel`. The ban commits first: a kill or cancel step that fails is reported in\nthe 200 response and does not undo it. The handler detaches the request from the caller and gives\nit one 25-second budget at the same moment, so a caller that disconnects does not stop it and\nnothing it runs (deletion checks, ban writes, cache evictions, kills, cancellations) blocks past\nthe budget. A ban write or eviction that runs out of it fails like any other (500, or `failed` in\nthe bulk route). Each team's kill and cancel run side by side; a step the budget cuts short is\nreported as failed with what it counted so far. A build delete already under way keeps its own 30\nseconds, so a request ends within about 55 seconds. A failed cache eviction answers 500, so the\ncaller's retry repeats the ban and its containment; banning a banned team contains it again. `DELETE .../ban` clears the flag and evicts the cache.\n`POST /unstable/admin/teams/bans` takes up to 100 teams. It stores every ban first, five teams at\na time, then contains the banned teams, five at a time, all under the one budget, and sorts each team\ninto banned, not found or failed; one team's failure does not stop the others. A deleted project\nanswers 404 to the single-team routes and counts as not found in the bulk route. The logic lives\nin `internal/team`.\n\n### Orchestrator (`packages/orchestrator`)\n\nA single Go binary running on every sandbox node (as root). `ORCHESTRATOR_SERVICES` selects its\nroles: `orchestrator` (run sandboxes) and/or `template-manager` (build templates). Code lives\nunder `pkg/`, almost all Linux-only.\n\ngRPC services on :5008 (`pkg/server/`, `pkg/service/`, `pkg/template/server/`, `pkg/volumes/`):\n\n- **SandboxService** — `Create`, `Update`, `List`, `Delete`, `Pause`, `Checkpoint`.\n `Update` callbacks are serialized per sandbox. A successful transition out of the live\n map closes update admission and joins any admitted callback before stop/checkpoint\n work or stopping notifications begin; `Close` repeats that check for crash cleanup.\n An update using an inactive sandbox returns gRPC `NotFound`, which the API maps to\n HTTP 404 for timeout and network updates. Teardown waits for admitted callbacks;\n it cannot preempt one that does not return.\n- **TemplateService** — `TemplateCreate`, `TemplateBuildStatus`, `TemplateBuildDelete` (template-manager role only).\n- **InfoService** — node identity, roles, capacity, health status (used by API node discovery).\n- **ChunkService / VolumeService** — peer-to-peer template chunk serving; persistent volumes.\n\nProcess shutdown, triggered by `SIGINT`, `SIGTERM`, `SIGUSR1`, or a service failure, moves\nthe node to `ShuttingDown` before draining builds and sandboxes. Like `Draining`, this excludes new\nplacement while existing work stays reachable, and `/health` returns HTTP 200 with status\n`draining`. Unlike reversible `Draining`, `ShuttingDown` is terminal for the process;\nstatus overrides cannot enter or leave it. `FORCE_STOP` skips waiting for sandboxes to exit.\nThe sandbox drain ends only when no sandbox is live and the node's outstanding work (below) is\nzero, so a create or a resume-fresh checkpoint in flight, whose sandbox is briefly not live, holds\nit until its sandbox goes live and then ends.\n\nAfter sandbox drain and snapshot uploads complete, sandbox proxy cleanup attempts graceful HTTP\nserver shutdown with a separate deadline, then forces server closure if needed. Successful forced\nclosure recovers graceful deadline expiry; other server errors still propagate. Pprof stays\navailable through service teardown; its final shutdown also has a deadline and forced-close fallback.\n\n`InfoService.ServiceInfo` reports `outstanding_work` for sandbox, template-builder, and\nmixed-role nodes. It counts overlapping work holds, not distinct sandboxes or builds, and includes\ntracked background persistence and cleanup. The producer reports zero when idle. The API caches\nthis report and exposes it as required top-level `outstandingWork` in admin node list and detail\nresponses, where zero also covers a node that has not reported. This observational count does not\nby itself authorize node deletion.\n\n`InfoService.ServiceInfo.max_sandboxes` reports the cached node limit for every service role.\nAdmin `/nodes` and `/nodes/{nodeID}` return required `maxSandboxes`, including zero. Admission,\nmetrics, and node info share the value refreshed from node-scoped flags every 30 seconds.\n\nKey mechanisms (all under `pkg/sandbox/`):\n\n- **Firecracker** (`fc/`): each sandbox is one Firecracker process in its own cgroup and network\n namespace. The FC HTTP API (unix socket) configures machine, drives, network, and snapshots.\n Guest metadata (sandbox ID, envd access token hash) is passed via MMDS.\n A sandbox may use fewer CPUs than its VM has (`SandboxConfig.max_vcpus` above `vcpu`): the VM\n is created with `max_vcpus` vCPUs, a boot puts `maxcpus=` on the kernel command line, and\n the `fc_vcpu` threads are held to `vcpu` CPUs by a `cpu.max` quota in a threaded `vcpu` child of\n the sandbox cgroup, so the VMM's own threads stay outside it. The quota is in place before the\n guest runs an instruction: a boot limits the whole cgroup until the vCPU threads exist and then\n moves the limit onto them; a resume confines the paused threads before the VM resumes, falls\n back to the whole cgroup, and fails if neither holds. The snapshot's metadata records the VM's\n vCPU count (`vcpu_count`, stamped at build and on every pause), so a resume limits below it\n without trusting the request or discovering capacity from cgroup threads. A legacy snapshot\n carries its stored original vCPU count as `max_vcpus`; if unavailable, it may only resume at its\n unchanged `vcpu`.\n `/init` carries the count as `cpuCount`\n so envd onlines or offlines guest CPUs to match and reports the outcome on `X-Envd-Cpus` and\n `/metrics`.\n- **Lazy memory / UFFD** (`uffd/`): on resume, Firecracker restores the VM without loading\n memory; a userfaultfd handler serves page faults directly from the template's memfile, so only\n touched pages are read. An optional prefetcher warms known-hot pages.\n- **Copy-on-write rootfs** (`rootfs/`, `nbd/`, `block/`): the template rootfs stays read-only;\n writes go to a per-sandbox COW cache exposed to Firecracker as an NBD block device served by\n an in-process userspace NBD server. On pause, the dirty blocks are exported as a diff.\n- **Template cache** (`template/`): templates are fetched lazily from object storage and cached\n on local disk (and optionally on a shared NFS chunk cache, or fetched peer-to-peer from other\n nodes before upload completes). Every reader of a cached template holds a pin on it — a\n running sandbox, a build, a peer stream, an ancestor lookup — and eviction never closes a\n pinned template. A snapshot's own upload holds a pin while it runs and records on the entry\n whether the snapshot landed in storage; a layer that did not land is never released, so a\n failed upload's local copy stays until its TTL expires. When a pause, or a\n checkpoint that resumes a fresh sandbox, publishes a new layer, the layer its sandbox ran\n from is marked superseded if this node inserted it for an earlier pause, because no resume\n can ask for it again. With `snapshot-cache-release-superseded` and\n `snapshot-cache-ancestor-storage-fallback` both on, a superseded layer is released once no\n pin holds it, its devices have resolved and its upload has landed, instead of staying until\n its TTL expires. A descendant's upload then heals that ancestor's build data from storage,\n and writes the same header bytes it would have written with the ancestor still cached.\n The sandbox and template caches start empty after a restart: the template cache empties the\n build-data directory, and startup reclaim empties the sandbox, template and template-build\n scratch directories.\n- **Networking** (`network/`): each sandbox gets a slot — a network namespace with a veth pair\n and a tap device, unique host-side IP (from a /16), NAT, and per-slot nftables egress firewall\n (with SNI/Host-inspecting TCP firewall for domain allow/deny lists). Slots are pooled and\n reused; slot indexes are allocated locally against the node's netns state (leftover\n namespaces from a previous run are torn down by startup reclaim).\n- **Sandbox proxy** (:5007, `pkg/proxy/`): reverse-proxies incoming traffic from client-proxy to\n the sandbox's slot IP and requested port over HTTP or configured HTTPS, enforcing per-sandbox\n traffic access tokens through the shared ingress policy. HTTPS backends may use self-signed certificates.\n It refuses a full-router sandbox with 503 `sandbox_feature_unsupported` before dialing the\n slot.\n Each admitted lifecycle owns its proxy handlers and backend sockets, including pending dials\n and upgrades. Stopping seals admission and interrupts connections; network release waits for\n that work to finish. Restarting envd resets the owner while allowing the live lifecycle to\n admit a replacement. Retired streams have one second to write their terminal response before\n a stalled frontend is interrupted.\n Shared proxy shutdown waits for H2C and upgraded connections to close and includes them in\n forced connection cleanup. Failed H2C handshakes release their hijacked connections when the\n handshake handler returns.\n- Writes sandbox lifecycle **events** and cgroup **host stats** to ClickHouse; exports metrics via\n OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy\n collector remains the fallback primary destination, and configured shadow destinations can mirror\n writes during collector/storage migrations without changing sandbox behavior.\n\nDetached sandbox-event publication holds node work until the publisher returns. This covers\ndelivery attempts, not end-to-end delivery: the ClickHouse target enqueues into an in-memory\nbatcher, whose flushing remains part of shutdown.\n\n### Envd (`packages/envd`)\n\nThe agent inside every VM (started by systemd very early in boot), port 49983, chi + Connect RPC.\n\n- **Process service** (`spec/process/process.proto`): start/list/connect to processes, stream\n stdout/stderr, stdin, signals, PTYs — this is what SDKs use to \"run code\". A start is refused\n with `unavailable` while the workload cgroups are frozen for a pause, and the freeze waits for\n in-flight starts, so a pause does not capture a child stopped before exec in a frozen cgroup. A\n start into a cgroup the guest froze itself is refused the same way. Starts carrying the system\n tag are unaffected: they stay in envd's own cgroup, which no freeze covers.\n- **Filesystem service** (`spec/filesystem/filesystem.proto`): stat/list/make/move/remove/watch.\n- **REST**: `/health`, `/metrics`, `/envs`, `/files` upload/download, `/init` (orchestrator pushes\n env vars, access token, metadata and the online CPU count after boot/resume), `/upgrade` (live self-upgrade, below),\n freeze/thaw hooks used during pause.\n- **Public vs. control-plane routes**: the control routes (`/init`, `/collapse`, `/upgrade`, and the\n freeze/thaw hooks) are marked `x-internal: true` in `spec/envd.yaml`, and `/upgrade` — which the spec does not\n describe — is listed alongside them in the orchestrator's `pkg/sandbox/envd`. The orchestrator\n reaches them over the host network at the sandbox slot IP; the sandbox proxy refuses them with a\n 404, for every method, so they are not reachable through a sandbox URL. Adding a control route\n therefore means marking it in the spec (`go generate` carries the marker into the proxy's\n rejection list) — otherwise it ships reachable from the internet.\n- **Guest trust** (`internal/host/cacerts.go`): `/init` supplies the desired CA bundle.\n envd atomically records its PEM and owned DER identities in `/run/e2b/ca/state.json`,\n then publishes a complete replacement of the active bundle with the requested roots\n before returning. Ordinary installs copy the existing bundle without decoding it;\n recovery first repairs interrupted writes. All supplied identities are owned,\n including identical preexisting copies. A background loop resumes saved intent at startup\n and retries cleanup every second. It atomically removes retired roots from the active\n bundle before rootfs work, so source-file failures cannot delay that cleanup.\n It persists each requested certificate as\n `e2b-.crt` in `/usr/local/share/ca-certificates`, removes retired managed files,\n and retains retirement identities until both bundle and source cleanup succeed.\n Superseded cleanup cannot publish an older bundle. Legacy-file removal checks the latest\n desired roots under the init lock, preserving hash links to a still-requested certificate.\n State survives envd process restart,\n not guest reboot. Retired roots may be trusted until cleanup succeeds, and an independent\n trust-store rebuild can restore a copy from a customer source file. Applications that\n cached their trust store must reload it.\n- **Auth**: `X-Access-Token` header checked against a token delivered via Firecracker MMDS;\n signed URLs for file endpoints. `/init` is exempt (it is what *delivers* the token), which is the\n main reason the proxy refuses it outright.\n- **Live upgrade** (`internal/services/process/upgrade.go`): an authenticated `POST /upgrade` lets\n the orchestrator swap envd inside a *running* sandbox at resume. It streams the new binary in the\n request body and envd `syscall.Exec`s into it **with the same PID**, carrying the workload's\n stdio/PTY fds, process table, recently-retained exit codes and filesystem watchers forward via a\n tmpfs handover blob. The workload cgroups stay frozen until the post-upgrade `/init` restores the\n access token (so no re-adopted process runs unauthenticated), and the handover outcome\n (procs/watchers re-adopted, plus any failures) rides back on that `/init`'s `X-Envd-Handover`\n header for fleet visibility.\n- Scans guest ports and forwards them so any port a user process opens becomes reachable through\n sandbox URLs. **`pkg/version.go` carries the released version**, maintained by release-please from\n the Conventional Commits merged since the last envd release; do not edit it by hand. The API and\n the orchestrator gate features on the envd version recorded in each template build — the\n orchestrator's held `/init` is the one gate that reads the version of the envd expected to\n answer instead (see sandbox creation) — so a behavioral change ships under a `feat`/`fix`\n commit type that cuts a version it can be gated on.\n\n### Client proxy (`packages/client-proxy`)\n\nThe stateless edge for all sandbox traffic (port 3002; health on 3003). Terminates\n`https://-.` requests (host parsing in `packages/shared/pkg/proxy/host.go`),\nlooks the sandbox up in the Redis routing record to find the owning node, and reverse-proxies to\nthat node's orchestrator proxy on :5007 by default. `ORCHESTRATOR_PROXY_PORT` selects a\ndifferent downstream port when the node proxy listens elsewhere. If the sandbox is not in the record (paused), it calls\nthe API's `ResumeSandbox` gRPC — paused sandboxes wake transparently on traffic.\nThe record is the orchestrator-owned `sandbox:routing:{id}` (see \"Sandbox routing records\"\nbelow).\n\nThe record on a hit, or the `ResumeSandbox` answer on a miss, is the sandbox's route, placement\nincluded. A miss uses the resume answer directly and does not read the record again. A legacy\n(no placement) or egress-router route goes to `orchestrator_ip` on the orchestrator proxy port,\nnever to the placement's ingress endpoint. A full-router route goes to the Router's ingress\nendpoint saved in its placement, never to an orchestrator, over the same pooled, kept-alive\nconnections as the orchestrator hop: the proxies never resend a request themselves (only its\ndial is retried), and net/http's resend of a bodyless idempotent request whose reused connection\nclosed before any response is accepted on both hops. A full-router route without that endpoint,\nan unknown mode or a missing `orchestrator_ip` answers the not-found page before anything is\ndialed.\n\nConcurrent requests that miss the record for the same sandbox, port and credentials share one\n**resume flight** per client-proxy process: the first request's call runs detached from its\nconnection under a 90 s ceiling and later requests join it, so one resume RPC serves them all\nand a client that disconnects mid-resume does not fail the others. A call whose requests have\nall left runs to its end, so a client that comes back finds the sandbox awake, but no further\nattempt starts for nobody, and a process keeps at most 64 such callerless calls; beyond that a\ndeparture cancels the proxy's call. That bound and that cancellation apply to the proxy's flights\nonly: the API's resume entry point lets a create already under way run on past the proxy's\ncancellation, so the node finishes the create and the API registers the sandbox inside the\nreservation's 90 s window (the create may run until ten seconds before the pending reservation can\nbe reaped and the registration until five seconds before, so the run-on never lets a fresh\nreservation start a second resume of the same snapshot), and the API's shutdown drain waits for\nthat work, bounded at 95 s. The run-on is granted when the resume is admitted, and an API already\ndraining admits none: a resume that arrives then keeps its caller's cut, so a create the proxy\ncancels mid-drain is cut with it and the reservation lapses. The sandbox otherwise lives its\nordinary timeout or auto-pause, billed, with nobody waiting for it. With\n`client-proxy-resume-parking` on (default off), a request whose flight got a still-transitioning\nor `Unavailable` answer parks instead of failing: the flight retries\nwhile a parked request waits, under a budget (`client-proxy-resume-parking-budget-ms`, 15 s from\nthe first retryable failure) that gates only whether another retry starts: every attempt, a late\nretry included, runs under its full 90 s ceiling while a caller waits on it, so a parked request\ncan wait up to the budget plus one attempt. The proxy cuts a retry at the budget's end only once\nevery caller has left, and that cut ends the proxy's wait, not the create on the node.\nA process keeps at most\n`client-proxy-resume-parking-lot-size` (512) parked requests, shedding the rest. A resume the\ncontrol plane cannot carry out right now (no capacity, no node, the API unreachable, the budget\nspent) answers **503** with `Retry-After: 2` and no error code; a request shed because the lot was\nfull answers the same **503** without the header and with `proxy_parking_lot_full` as its\n`error_code`, so a client, or one behind an ingress that strips the header, still tells the two\napart. The SDKs do not act on the header either way: their status-based retry\nwraps only the control-plane API client, and sandbox traffic through the proxy is retried only on\na connection failure, never on a 503. The omission is for hand-rolled retry loops and browsers\nthat honour the header, which would otherwise come back in one wave while the lot is still full,\nand for a client that does the SDKs' jittered backoff on its own; a placement no retry helps, which\nthe API names in an `ErrorInfo` detail on the RPC status, is not retried and answers without the\nheader, with the status the API carried: **503** for a feature no orchestrator in the cluster\nsupports, **500** (`sandbox_create_refused`) for a sandbox the node refuses to create, such as a\nstored build whose vCPU count is outside the node's bounds; a sandbox still pausing\nanswers **409** with the header on a user port and the same page as **503** on the envd port (the\nSDK reads an envd 409 as \"already exists\"); a request whose client left before it was routed\nanswers 408.\n\n### Dashboard API (`packages/dashboard-api`)\n\nA separate REST service (port 3010, spec `spec/openapi-dashboard.yml`) consumed by the web\ndashboard and the workspace control plane, not the SDK: the killed-sandbox record, admin routes\n(clusters, team cluster assignment, user-profile resolution) and the `/v1/management`\noperations. Template reads are served by the API, not here. Its admin routes accept either a\nshared admin token or a short-lived service JWT verified against the configured admin JWKS. The\nsandbox record route accepts dashboard user auth only. Its workspace-agnostic `/v1/management` operations are defined in the\nsame dashboard OpenAPI contract and registered on the existing router. Their `AdminJWTAuth`\nOpenAPI security scheme accepts only short-lived service JWTs verified against the workspace-api\n`/.well-known/jwks.json` endpoint, with accepted signing methods derived from each JWK's required\n`alg` metadata. Issuers and audiences are configured through the JSON `ADMIN_AUTH_PROVIDER_CONFIG` value —\nthe same config shape as `AUTH_PROVIDER_CONFIG`. Talks to Postgres and ClickHouse; never talks to\norchestrators. The web dashboard itself is a separate repository (github.com/e2b-dev/dashboard)\npublished as an image; E2B Embed runs it on port 3001 beside dashboard-api.\n\nAn issuer normally configures at least one accepted audience, binding a JWT to its intended target.\nAn issuer may omit `audiences` only with no `audienceMatchPolicy`; this deliberately disables\naudience matching while issuer, signature, and temporal-claim verification remain required.\n\nThe `/v1/management` operations are the cluster's half of a contract the workspace residency owns:\nthe cluster lifecycle below. Project upsert, membership, limits, block state and deletion are\nserved by the API under `/unstable/management/projects/{projectID}`, and team bans under\n`/unstable/admin/teams` (see API above).\n\nThe management surface owns a replay-safe cluster lifecycle. A caller registers a stable\ncluster UUID, assigns it only to the named project, detaches that exact assignment before provider\ncleanup, and deletes the cluster only after no project references it. Replaying the same\nregistration or assignment succeeds. Registering the cluster again with the same name and auth\norganization may change its endpoint, TLS setting, sandbox proxy domain and token, for example when\nthe cluster moves to a new domain; the API rebuilds that cluster's clients on its next sync and gives\nsandboxes the new domain. A changed name or auth organization, a different assignment, an inexact\ndetach, or deletion of a referenced cluster returns a conflict. The\nassignment does not check the project's tier; the caller decides which projects may use a cluster.\nA cluster first registered through this surface starts without deletion protection; one created any\nother way starts protected, a later registration never changes it, and deleting a protected cluster\nor checking its destroy readiness returns a conflict.\nA caller that deletes a cluster's projects before the cluster first lists them with\n`GET /v1/management/clusters/{clusterID}/projects`, which returns every project whose team is\nassigned to the cluster however it was assigned, leaving out deleted teams, and records the deletion with\n`PUT /v1/management/clusters/{clusterID}/deletion` before deleting any of them. That request\nlocks the cluster row, returns a conflict while deletion protection is on, and sets\n`clusters.deletion_requested_at`; it succeeds for an absent cluster and on repeat. From then on a\ncheck constraint keeps deletion protection off, and a trigger on `teams` refuses to newly assign a\nteam to the cluster (`cluster_assignment_cluster_deleting` from the assignment route, a conflict\nfrom the API's project upsert carrying the cluster ID). The trigger takes a key-share lock on the cluster row\nbefore it checks, so an assignment either commits before the deletion is recorded, where the\ncaller's next listing finds it, or is refused. Deleting each project clears its team's cluster,\nthe cluster delete clears it from any team deleted earlier, and then succeeds.\nThe shared error response supports an optional `error_code`, defined by the dashboard OpenAPI\n`ErrorCode` schema. Registration and assignment rejections currently populate it; other errors\nmay omit it. Callers use that stable code rather than the human-readable message, and treat\nuncoded or unknown reasons as generic errors.\n\nA project deletion keeps the team row, so dashboard-api middleware, running after\nauthentication, answers 404 to every write under `/v1/management/` or `/admin/` that names a\ndeleted project by `projectID` or `teamID` (\n\n","packages/api/internal/sandbox/reservations/redis/README.md":"# Redis Reservation Storage\n\nThis package coordinates sandbox creation reservations across API instances.\n\n## Keys\n\n- Storage index: `sandbox:storage:{teamID}:index`\n- Pending zset: `sandbox:storage:{teamID}:reservations:pending`\n- Result key: `sandbox:storage:{teamID}:reservations:{sandboxID}:result`\n- PubSub routing key: `sandbox:storage:{teamID}:reservations:{sandboxID}:notify`\n\n## Flow\n\n`Reserve` runs a Lua script that atomically removes stale pending entries, checks whether the sandbox already exists or has already pending start, enforces the team limit using `SCARD(storage index) + ZCARD(pending zset)`, deletes any stale result key, and adds the sandbox ID to the pending zset.\n\nWhen creation completes, it removes the sandbox from the pending zset, writes a TTL result key containing either the sandbox or the creation error, and publishes the routing key.\n\nA waiter subscribes to the routing key, probes the result key immediately, then waits for PubSub notifications or the 1 second fallback ticker. PubSub is best-effort; the fallback ticker is required for correctness.\n\n`Release` is called when the sandbox is removed from storage (`Store.Remove`). It removes the sandbox from the pending zset, deletes the result key, and publishes the routing key.\n","packages/envd/README.md":"# envd\n\nDaemon that runs inside a sandbox that allows interacting with the sandbox via calls from the SDK.\n\n## Development\n\n### Versioning\n\nThe envd version in `pkg/version.go` is maintained by release-please from the Conventional Commits merged since the last envd release; do not edit it by hand. A change that affects behavior (code, dependencies) ships under a `feat` or `fix` commit type, so that a version the orchestrator can gate on is cut. Pure comment or documentation changes use `docs`, a type the changelog hides, and cut no release; `chore` and `refactor` are visible in it and do cut one.\n\n### Running locally\n\nRun the following command to (re)build the envd daemon and start a Docker container with envd running inside:\n\n```bash\nmake start-docker\n```\n\nYou can use E2B SDKs with env var `E2B_DEBUG=true` or with a debug parameter set to `true` when creating or connecting to a sandbox, to connect to the envd started with this command.\n\n### Generating API server stubs\n\nRun the following command to install the necessary dependencies for generating server stubs:\n\n```bash\nmake init-generate\n```\n\n#### Generating\n\nAfter changing the API specs in `./spec/` run the following command to generate the server stubs:\n\n```bash\nmake generate\n```\n\n### Debugging\n\n- \n- \n\nRun `make run-debug` and then connect to the port `2345` with a debugger or use the VSCode run/debug and run the \"Debug envd\" to build the envd, Docker, and start the debugging.\n","packages/envd/internal/services/legacy/readme.md":"The python SDK had a bug: new fields introduced to the API would throw exceptions when processed. In order to work around this, we convert all responses to a list of structs that were valid when the python SDK still had the bug.\n\nWe copy the proto files to this folder in order to keep track of what was available when this SDK was supported. These should never be modified.\n\nThis entire package can be removed once we no longer have clients that send a user agent of `connect-python`.\n\n","packages/nomad-nodepool-apm/README.md":"# Nomad Node Pool Autoscaler Plugins\n\nCustom plugins for the [Nomad Autoscaler](https://github.com/hashicorp/nomad-autoscaler) that keep a service-job allocation count aligned with the number of nodes in a Nomad node pool.\n\n## Purpose\n\nThe package builds two plugins because Nomad Autoscaler supports only one plugin type per external binary:\n\n- `nomad-nodepool-apm` reports the number of ready nodes in a pool.\n- `nomad-deployment-aware-target` fails an active deployment before applying the requested task-group count.\n\nTogether they let service jobs replicate system-job placement while retaining rolling updates. The deployment-aware target is used by the `template-manager` and `orchestrator-ee` service jobs. When the task-group count must change, scaling intentionally abandons a conflicting in-progress rollout (Nomad rejects scaling while a deployment is active); Nomad records that deployment as `failed`, rather than `cancelled`. The rollout spawned by the scale itself is left to run, and no deployment is touched when the count already matches. The target requires `auto_revert = false` on the scaled group (otherwise failing a deployment would restore an older job version and ping-pong); it refuses to scale groups with `auto_revert = true` and returns an error on every attempt.\n\n## Usage\n\n### Building\n\n```bash\nmake build\n```\n\n### Uploading to GCS\n\n```bash\nGCP_PROJECT_ID=your-project-id make build-and-upload\n```\n\n### Configuration\n\nIn your Nomad Autoscaler configuration:\n\n```hcl\napm \"nomad-nodepool-apm\" {\n driver = \"nomad-nodepool-apm\"\n config = {\n nomad_address = \"http://localhost:4646\" # Optional, uses NOMAD_ADDR env var\n nomad_token = \"your-token\" # Optional, uses NOMAD_TOKEN env var\n nomad_region = \"global\" # Optional\n }\n}\n\ntarget \"nomad-deployment-aware-target\" {\n driver = \"nomad-deployment-aware-target\"\n config = {\n nomad_address = \"http://localhost:4646\"\n nomad_token = \"your-token\"\n nomad_region = \"global\"\n }\n}\n```\n\n### In Job Scaling Policy\n\n```hcl\nscaling {\n enabled = true\n min = 1\n max = 50\n\n policy {\n evaluation_interval = \"30s\"\n cooldown = \"2m\"\n\n target \"nomad-deployment-aware-target\" {}\n\n check \"match_node_count\" {\n source = \"nomad-nodepool-apm\"\n query = \"orchestrator\" # Node pool name\n\n strategy \"pass-through\" {} # Use node count directly as desired count\n }\n }\n}\n```\n\n## How It Works\n\n1. The plugin queries the Nomad API to list nodes filtered by the specified node pool\n2. It counts only nodes that are:\n - Status: `ready`\n3. Returns the count as a metric for the autoscaler to use\n4. With the `pass-through` strategy, this count becomes the desired number of allocations\n5. The target serializes scaling per namespaced job and is a no-op when the durable count already matches\n6. When the count must change it fails a conflicting active deployment, rereads the job, and scales with the current job modify index\n7. It verifies the final durable count; the rollout spawned by the scale proceeds normally\n8. Concurrent job or deployment changes are retried from fresh state with a bounded attempt count\n\nDry-run actions do not fail deployments or write task-group counts.\n\n## Query Format\n\nThe `query` parameter should be the name of the node pool you want to count nodes from.\n\nExample: `query = \"orchestrator\"` counts all ready nodes in the `orchestrator` node pool.\n\n## Configuration Options\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `nomad_address` | Nomad server address | `NOMAD_ADDR` env var or `http://127.0.0.1:4646` |\n| `nomad_token` | Nomad ACL token | `NOMAD_TOKEN` env var |\n| `nomad_region` | Nomad region | Default region |\n| `nomad_namespace` | Nomad namespace | Default namespace |\n","packages/orchestrator/README.md":"# Orchestrator\n\n## Commands\n\n**Prerequisite:** Enable NBD module first:\n\n```bash\nmodprobe nbd nbds_max=4096\n\ncat </etc/udev/rules.d/97-nbd-device.rules\n# Disable inotify watching of change events for NBD devices\nACTION==\"add|change\", KERNEL==\"nbd*\", OPTIONS:=\"nowatch\"\nEOH\nudevadm control --reload-rules\nudevadm trigger\n```\n\n**Prerequisite:** Allocate enough 2MB HugeTLB pages (hugepages) for testing:\n\n```bash\necho 1024 | sudo tee /proc/sys/vm/nr_hugepages # for 2GB hugepages, adjust as needed\n```\n\n### Create Build\n\nCreate a new build.\n\n```bash\nsudo go run ./cmd/create-build -to-build -storage .local-build\n```\n\nFlags:\n\n- `-to-build ` - Output build ID (UUID, required)\n- `-from-build ` - Base build ID for incremental builds\n- `-template ` - Template ID (default: `local-template`)\n- `-storage ` - Local path or `gs://bucket` (enables local mode with auto-download of kernel/FC)\n- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)\n- `-kernel ` - Kernel version (default: `vmlinux-6.1.158`)\n- `-firecracker ` - Firecracker version (default: `v1.14.1_458ca91`)\n- `-vcpu ` - vCPUs (default: `1`)\n- `-memory ` - Memory in MB (default: `512`)\n- `-disk ` - Disk in MB (default: `1000`)\n- `-hugepages` - Use 2MB huge pages (default: `true`, set `false` for 4KB pages)\n- `-start-cmd ` - Start command\n- `-ready-cmd ` - Ready check command\n\n> You can use the `$(uuidgen)` command to generate a random UUID.\n>\n> If you are using Mise, you can run the command as `sudo $(which go) run ./cmd/create-build -to-build -storage .local-build`.\n\n### Resume Build\n\nResume a sandbox from a build.\n\n```bash\n# Local storage\nsudo go run ./cmd/resume-build -from-build -storage .local-build -iterations 10\n\n# Remote GCS storage (pass credentials to sudo)\nsudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \\\n go run ./cmd/resume-build -from-build -storage gs://bucket -iterations 10\n\n# Pause mode: resume, run command, then snapshot\nsudo go run ./cmd/resume-build -from-build -to-build \\\n -storage .local-build -cmd-pause \"pip install numpy\"\n\n# Pause mode: start command through envd, then wait for SIGUSR1 before snapshot\nsudo go run ./cmd/resume-build -from-build -to-build \\\n -storage .local-build -cmd-signal-pause \"python3 /home/user/workspace/job.py\"\n```\n\nFlags:\n\n- `-from-build ` - Build ID (UUID) to resume from (required)\n- `-to-build ` - Output build ID (UUID) for pause snapshot (auto-generated if not specified)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)\n- `-iterations ` - Number of iterations, 0 = interactive (default: `0`)\n- `-cold` - Clear cache between iterations (cold start each time)\n- `-no-prefetch` - Disable memory prefetching\n- `-v` - Verbose logging\n- `-pause` - Start and immediately pause (create snapshot)\n- `-signal-pause ` - Wait for signal before pause (e.g., `SIGTERM`, `SIGUSR1`)\n- `-cmd-pause ` - Execute command in sandbox, then pause on success\n- `-cmd-signal-pause ` - Execute command in sandbox, then wait for `SIGUSR1` before pause\n- `-envd-version ` - The guest's envd version (default: a placeholder — see the caveat below)\n\n**envd version caveat:**\n\n`resume-build` has no database and the snapshot metadata does not record the build's envd\nversion (only the kernel and Firecracker ones), so unless `-envd-version` is given the tool\nreports a fixed placeholder (`1.0.0`). Every `envd.version` field in its output and logs is\nthen that placeholder, **not** the guest's version — don't reason from it. To read the real\none, mount the build's rootfs (`mount-build-rootfs`) and run its `/usr/bin/envd -version`,\nor read `X-Envd-Version`, which a running envd reports on `/init`.\n\nThe placeholder sits above every envd version gate, so the gated capabilities (`/freeze`,\n`/fsfreeze`, `/collapse`, the KVM clock, the upgrade `from_version`) stay on. Against an\nolder guest that is a path production would not take, so pass `-envd-version` with the\nguest's real version when the behavior under test depends on one of those gates. The tool\nprints which of the two it is using when it loads the build.\n\n**Pause mode example:**\n\n```bash\n# Chain builds with explicit build IDs\nsudo go run ./cmd/resume-build -from-build $BUILD1 -to-build $BUILD2 \\\n -storage .local-build -cmd-pause \"apt install curl\"\n\nsudo go run ./cmd/resume-build -from-build $BUILD2 -to-build $BUILD3 \\\n -storage .local-build -cmd-pause \"pip install requests\"\n\n# Start a long-running command through envd, then trigger pause from the host\nsudo go run ./cmd/resume-build -from-build $BUILD3 -to-build $BUILD4 \\\n -storage .local-build -cmd-signal-pause \"python3 /home/user/workspace/job.py\"\n# In another shell:\nps -ef | grep 'cmd/resume-build' | grep -v grep\nsudo kill -SIGUSR1 \n```\n\n### Copy Build\n\nCopy a build between storage locations (local or GCS).\n\n```bash\n# Local to local\ngo run ./cmd/copy-build -build -from .local-build -to /other/path\n\n# Local to GCS\ngo run ./cmd/copy-build -build -from .local-build -to gs://bucket\n\n# GCS to GCS\ngo run ./cmd/copy-build -build -from gs://bucket1 -to gs://bucket2\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required)\n- `-from ` - Source: local path or `gs://bucket`\n- `-to ` - Destination: local path or `gs://bucket`\n- `-team ` - Team UUID (if set, prints SQL to populate DB on stdout)\n- `-envd-version ` - Envd version (required if team provided) — must match the version present in the template\n- `-vcpu ` - vCPUs (default: `2`)\n- `-memory ` - Memory in MB (default: `1024`)\n- `-disk ` - Disk in MB (default: `1024`)\n- `-tag ` - Build assignment tag (default: `default`)\n\n**Print DB seed SQL:**\n\nWhen `-team` is set, the command reads `metadata.json` from the destination to get `kernel_version` and `firecracker_version`, generates a new env ID, and prints SQL statements (`BEGIN`/`COMMIT` wrapped) to stdout. Pipe directly to `psql` to populate the database:\n\n```bash\ngo run ./cmd/copy-build -build -from .local-build -to gs://bucket \\\n -team -envd-version 0.2.11 -vcpu 2 -memory 1024 -disk 1024 \\\n | psql $POSTGRES_CONNECTION_STRING\n```\n\n### Mount Build Rootfs\n\n```bash\n# Local storage\nsudo go run ./cmd/mount-build-rootfs -build -storage .local-build -mount /mnt/rootfs\n\n# Remote GCS storage (pass credentials to sudo)\nsudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \\\n go run ./cmd/mount-build-rootfs -build -storage gs://bucket -mount /mnt/rootfs\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required unless `-empty`)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-mount ` - Mount path\n- `-verify` - Verify rootfs integrity (requires `-mount`)\n- `-log` - Enable logging\n- `-empty` - Create an empty rootfs instead of loading a build\n- `-size ` - Size of empty rootfs (default: 1GB)\n- `-block-size ` - Block size (default: 4096)\n\n### Inspect Build\n\nInspect build artifacts (headers and data blocks).\n\n```bash\n# Inspect memfile header (default)\ngo run ./cmd/inspect-build -build -storage .local-build\n\n# Inspect rootfs header\ngo run ./cmd/inspect-build -build -storage .local-build -rootfs\n\n# Inspect header + data blocks\ngo run ./cmd/inspect-build -build -storage .local-build -data\n\n# Inspect first 100 data blocks\ngo run ./cmd/inspect-build -build -storage .local-build -data -end 100\n\n# Remote GCS storage\ngo run ./cmd/inspect-build -build -storage gs://bucket -rootfs -data\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-memfile` - Inspect memfile artifact (default)\n- `-rootfs` - Inspect rootfs artifact\n- `-data` - Also inspect data blocks (not just header)\n- `-start ` - Start block index (default: `0`, only with `-data`)\n- `-end ` - End block index (default: all, only with `-data`)\n\n### Show Build Diff\n\nShow the diff between two builds and preview merge result.\n\n```bash\n# Memfile (default)\ngo run ./cmd/show-build-diff -from-build -to-build -storage .local-build\n\n# Rootfs\ngo run ./cmd/show-build-diff -from-build -to-build -storage .local-build -rootfs\n\n# With visualization\ngo run ./cmd/show-build-diff -from-build -to-build -storage gs://bucket -visualize\n```\n\nFlags:\n\n- `-from-build ` - Base build ID (required)\n- `-to-build ` - Diff build ID (required)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-memfile` - Inspect memfile artifact (default)\n- `-rootfs` - Inspect rootfs artifact\n- `-visualize` - Visualize the headers\n\n---\n\n## Architecture (ARM64) Support\n\nThe orchestrator supports both `amd64` (x86_64) and `arm64` (aarch64) architectures. Architecture is detected automatically via `runtime.GOARCH` at compile time.\n\n### Architecture naming convention\n\nThis project uses **Go/Docker/Debian naming** (`amd64`/`arm64`) for architecture directories in binary paths and GCS buckets:\n\n| Convention | x86_64 name | ARM64 name | Used by |\n|------------|-------------|------------|---------|\n| **Go/Docker/Debian** | `amd64` | `arm64` | This repo, Docker, `dpkg --print-architecture` |\n| Linux/GNU | `x86_64` | `aarch64` | `uname -m`, kernel Makefiles |\n\nBinary paths follow the `{version}/{arch}/` layout:\n\n```\n# Firecracker (GCS bucket or FIRECRACKER_VERSIONS_DIR)\nfc-versions/v1.12.1_210cbac/amd64/firecracker\nfc-versions/v1.12.1_210cbac/arm64/firecracker\n\n# Kernels (GCS bucket or HOST_KERNELS_DIR)\nkernels/vmlinux-6.1.158/amd64/vmlinux.bin\nkernels/vmlinux-6.1.158/arm64/vmlinux.bin\n```\n\n> **Note:** The [fc-kernels](https://github.com/e2b-dev/fc-kernels) repo currently uses `x86_64` instead of `amd64` for its directory names. This will be aligned in a follow-up change.\n\n### ARM64-specific behavior\n\n- **SMT** is disabled (ARM processors don't support simultaneous multi-threading)\n- **CPU detection** uses fallback values since `gopsutil` doesn't populate Family/Model on ARM64\n- **OCI platform** is set to the target architecture instead of hardcoded `amd64`\n- **Busybox binaries** are fetched per architecture by `scripts/fetch-busybox.sh` and selected at runtime from `///busybox`\n- **Guest kernel command line** drops the x86-only `i8042.*` and `clocksource=kvm-clock` parameters on arm64; the console stays `ttyS0` (Firecracker exposes a 16550-compatible UART on both architectures)\n\n### Cross-architecture deployment\n\n`TARGET_ARCH` is a **runtime** environment variable that overrides the architecture used for path resolution and OCI image pulls. When unset, defaults to the host architecture (`runtime.GOARCH`).\n\n```bash\n# Run orchestrator targeting amd64 paths from an arm64 host\nTARGET_ARCH=amd64 ./bin/orchestrator\n\n# Or in .env file (read at runtime)\necho \"TARGET_ARCH=amd64\" >> .env.local\n```\n\n`TARGET_ARCH` affects:\n- Firecracker and kernel binary path resolution (`{version}/{arch}/...`)\n- OCI image platform for container pulls\n\nIt does **not** affect:\n- Makefile compilation (see `BUILD_ARCH` below)\n- Hardware-dependent runtime behavior (SMT detection, CPU info) which always uses the actual host architecture\n\n### Building for ARM64\n\n`BUILD_ARCH` defaults to `amd64`. To build ARM64 binaries:\n\n```bash\n# Single command\nBUILD_ARCH=arm64 make build-local\n\n# Or set in .env.local for persistent override\necho \"BUILD_ARCH=arm64\" >> .env.local\n```\n\nThis applies to all services: orchestrator, envd, client-proxy.\n\n---\n\n## Environment Variables\n\nAutomatically set in local mode. Set before running to override:\n\n- `TARGET_ARCH` - Target architecture override (`amd64` or `arm64`; default: host architecture)\n- `HOST_ENVD_PATH` - Envd binary path (default: `../envd/bin/envd`)\n- `HOST_KERNELS_DIR` - Kernel versions dir (local: `{storage}/kernels`, prod: `/fc-kernels`)\n- `FIRECRACKER_VERSIONS_DIR` - Firecracker versions dir (local: `{storage}/fc-versions`, prod: `/fc-versions`)\n- `ORCHESTRATOR_BASE_PATH` - Base orchestrator data (local: `{storage}/orchestrator`, prod: `/orchestrator`)\n- `SANDBOX_DIR` - Sandbox working dir (default: `/fc-vm`)\n\n## Limitations\n\n- Custom template builds support base images from the declared distro families: **Debian/Ubuntu** (apt), the **RHEL family** — Fedora, CentOS Stream, Rocky, Alma — (dnf/microdnf/yum), **Arch** (pacman), and **Alpine** (apk, OpenRC). The distro is resolved from the image's `/etc/os-release` `ID` — never by probing for package managers. Images without an os-release identity (distroless, scratch) or from an undeclared family are rejected fast, with a build-log error naming the reason and the supported families (see `pkg/template/build/phases/base/distro`). Minimal/restricted-repo images may fail provisioning if their repos don't carry the required packages; the failure is surfaced in the build log.\n- Sandboxes always boot the kernel E2B supplies, never one from the base image: `/lib/modules` is empty, no kernel modules can be loaded, and SELinux is disabled on the kernel command line (`selinux=0`), so a base image that ships an SELinux policy boots without it. `kernel-devel` from a distro's repos resolves against a kernel that isn't running. **RHEL** (`rhel`, incl. UBI), **Oracle Linux** (`ol`) and **Amazon Linux** (`amzn`) are therefore not accepted even though they are RPM/dnf images — they are chosen for kABI, signed kernel modules and UEK, which this cannot honour. The community rebuilds above are supported because they are chosen for the userland.\n","packages/orchestrator/cmd/simulate-gcs-traffic/readme.md":"To run inside of a host in GCS, trust this fancy one-liner:\n\n host=@ && bucket= && go build -o simulator . && ssh $host rm ./simulator && scp ./simulator $host:~/simulator && ssh $host -- ./simulator -csv-path gcs.csv -test-duration=5s $bucket && ssh $host cat gcs.csv\n","packages/shared/pkg/apiid/README.md":"# Resource IDs at HTTP boundaries\n\n`apiid.ProjectID` and `apiid.WorkspaceID` accept a legacy UUID or the matching\npublic ID. Both inputs produce the same struct:\n\n```go\nprojectID.UUID // uuid.UUID\nprojectID.PublicID // canonical prj_ string\n```\n\nThe existing `pkg/id` codecs remain unchanged. These structs are HTTP input\nvalues; database and domain functions can continue taking `uuid.UUID`.\n\n## Adding or changing an API ID\n\nUpdate [`id`](../id/README.md#adding-or-changing-a-resource-id) and this package\nin the same PR whenever adding a resource kind to either package. Follow the\nlinked checklist: `id` owns the codec, and `apiid` provides the matching input\nstruct and binding support. Do not add an API-only encoding or a new codec\nwithout its API type. Keep existing kinds' encoding and validation changes,\ntests, and documentation synchronized across both packages.\n\n## OpenAPI and generated Gin handlers\n\nUse the standard oapi-codegen type override on a string schema:\n\n```yaml\nprojectID:\n name: projectID\n in: path\n required: true\n schema:\n type: string\n description: Project UUID or prj_ public ID.\n x-go-type: apiid.ProjectID\n x-go-type-import:\n name: apiid\n path: github.com/e2b-dev/infra/packages/shared/pkg/apiid\n```\n\nRegenerate the server and dependent clients using the API's generation command.\nNo middleware, registry, or generator fork is needed:\n\n```go\nfunc (s *Server) GetProject(c *gin.Context, projectID api.ProjectID) {\n project, err := s.projects.Get(c.Request.Context(), projectID.UUID)\n // Handle err and build the response as usual.\n}\n```\n\nUse `apiid.WorkspaceID` for workspace parameters. Apply the same schema override\nto query or header parameters and to JSON body properties or array items:\n\n```yaml\nworkspace_id:\n type: string\n description: Workspace UUID or wrk_ public ID.\n x-go-type: apiid.WorkspaceID\n x-go-type-import:\n name: apiid\n path: github.com/e2b-dev/infra/packages/shared/pkg/apiid\n```\n\nAfter `c.ShouldBindJSON(&body)`, use `body.WorkspaceId.UUID` and\n`body.WorkspaceId.PublicID`. Keep response UUID fields unchanged unless a\nresponse-contract change is intended. Do not keep `format: uuid` on inputs\nthat also accept public IDs.\n\n## Handwritten Gin binding\n\n```go\nvar params struct {\n ProjectID apiid.ProjectID `uri:\"projectID\" binding:\"required\"`\n}\nif err := c.ShouldBindUri(¶ms); err != nil {\n c.AbortWithStatus(http.StatusBadRequest)\n return\n}\nprojectUUID := params.ProjectID.UUID\n```\n\n`ShouldBindQuery` and `ShouldBindHeader` work with `form` and `header` tags.\n`UnmarshalText` supports generated path binding; `Bind` supports oapi-codegen's\nexploded query binding; `UnmarshalParam` supports Gin's own binders. Tests cover\neach path because these binders do not all use the same interface.\n\n## Construction and encoding\n\nUse `apiid.NewProjectID(u)` or `apiid.NewWorkspaceID(u)` when constructing an ID\nin Go. Decoders and constructors populate both fields. Malformed input and\nwrong-kind public IDs fail without modifying the receiver. JSON accepts only\nstrings; encoding emits a public ID string, not a JSON object.\n\nThe fields are exported for direct reads. Treat decoded values as immutable;\nGo cannot stop a caller from manually making them inconsistent. Encoding derives\nthe public string from `UUID`, rather than trusting a manually edited `PublicID`.\nA zero-valued struct denotes an unset input; requiredness belongs in the request\nschema or Gin validation. Use a pointer for optional inputs when absence matters.\n\nOther APIs adopt the types by updating their schemas and regeneration, not by\ninstalling middleware. This package does not change their contracts automatically.\n","packages/shared/pkg/featureflags/README.md":"# LaunchDarkly client authentication\n\nThe client selects authentication in this order:\n\n1. A non-empty `LAUNCH_DARKLY_API_KEY` selects SDK-key authentication and ignores\n the OAuth variables.\n2. Without an SDK key, OAuth variables select OAuth client-credentials authentication.\n Incomplete OAuth configuration fails startup.\n3. Without either, the client uses the offline flag store.\n\nFor direct LaunchDarkly access, set `LAUNCH_DARKLY_API_KEY`. An optional\n`LAUNCH_DARKLY_BASE_URL` redirects streaming, polling, events, and diagnostics\nto a Relay proxy.\n\nFor an OAuth gateway, leave `LAUNCH_DARKLY_API_KEY` unset and configure:\n\n| Variable | Value |\n| --- | --- |\n| `LAUNCH_DARKLY_BASE_URL` | HTTPS gateway URL |\n| `LAUNCH_DARKLY_OAUTH_TOKEN_URL` | HTTPS OAuth token endpoint |\n| `LAUNCH_DARKLY_OAUTH_CLIENT_ID` | Deployment client ID |\n| `LAUNCH_DARKLY_OAUTH_CLIENT_SECRET` | Deployment client secret |\n| `LAUNCH_DARKLY_OAUTH_AUDIENCE` | Optional audience parameter for the token request |\n\nUse the existing deployment OAuth credentials and audience when the gateway\naccepts that identity. The token endpoint must support the `client_credentials`\ngrant with credentials in the request body.\n\nBoth client constructors enable live flag updates without an SDK key in OAuth\nmode.\nOAuth URLs must not contain embedded credentials, query parameters, or fragments.\n\nThe client caches tokens and refreshes them before expiry. Every new HTTP\nrequest uses the current bearer token, including stream reconnections, events,\nand diagnostics. The gateway must bound stream duration because an open stream\ndoes not reauthenticate when its token expires.\n\nToken requests have a five-second timeout. OAuth requests do not follow\nredirects. Authentication failures do not switch between SDK-key and OAuth modes.\n\nThe client mints its first token at startup. If the token endpoint answers with\na non-5xx response, construction fails, including with `WithStartOnInitTimeout`.\nIf the token endpoint is unreachable or returns 5xx, the SDK retries and the\nexisting initialization timeout policy applies.\n\nA gateway `401` or `403` before the gateway has accepted any request stops the\nSDK, as a rejected SDK key does. After the first accepted request, the client\ndrops its cached token on `401` or `403` and the SDK retries with a fresh one.\nThe SDK's own backoff bounds the retry and mint rate.\n","packages/shared/pkg/id/README.md":"# Typed public IDs\n\nThis package converts resource UUIDs to and from human-readable public IDs:\n\n```text\nprj_7mfb0gpp6c8dsaasre0asc7n3s\nwrk_2n1t201rmv87aae5j4csam8000\nsec_01kyjhkfvse96rg4d2qzd9enft\n```\n\nEvery public ID has a stable resource prefix, an underscore, and a\n26-character UUID suffix. E2B's public representation is always lowercase.\n\n## UUID ownership\n\nResource owners remain responsible for generating and storing UUIDs. UUIDs\nmay, for example, be generated by application code or a database. This package\ndoes not use `typeid.New` or `typeid.MustNew`.\n\n[`github.com/sumup/typeid`](https://github.com/sumup/typeid) is used only to\nconvert an existing UUID to its prefixed public representation and parse that\nrepresentation back into the UUID-backed resource type. PostgreSQL and other\npersistence layers continue to store the underlying UUID, not the prefixed\nstring.\n\n```go\nprojectUUID := uuid.New()\nprojectID := id.ProjectID(projectUUID)\n\npublicID := projectID.String()\nparsedID, err := id.ParseProjectID(publicID)\nif err != nil {\n\treturn err\n}\n\nprojectUUID = uuid.UUID(parsedID)\n```\n\nProject and workspace UUIDs are normally UUIDv4 and use `typeid.Random`.\nSecret UUIDs are UUIDv7 and use `typeid.Sortable`. SumUp renders Random IDs in\nuppercase, so the E2B wrapper normalizes them to the lowercase public contract.\n\n## Team IDs\n\nTeams are projects: a team UUID is also its project UUID.\n`ConvertTeamIDToProjectID` renders a team UUID as its public `prj_` ID, and\n`ParseTeamID` accepts either spelling wherever an API takes a team ID.\n\n## Adding or changing a resource ID\n\nKeep this package and [`apiid`](../apiid/README.md) in sync in the same PR.\nAdding a resource kind here requires its matching API input type in `apiid`;\nadding a type in `apiid` requires a matching codec here. Changes to an existing\nkind's prefix, encoding, or validation must also update the affected API type\nand tests. Encoding rules belong here; `apiid` must reuse them, not duplicate them.\n\nTo add another public resource ID:\n\n1. Choose a short, stable, lowercase prefix that keeps the full public ID at\n the expected fixed width.\n2. Add a private prefix type implementing `Prefix()`.\n3. Select `typeid.Random` for a UUIDv4 resource or `typeid.Sortable` for a\n UUIDv7 resource.\n4. Add a distinct UUID-backed public Go type with `String` and parse methods.\n5. Keep UUID generation in the resource-owning application or database.\n6. Test the exact prefix, lowercase output, fixed length, invalid-prefix\n rejection, and UUID round trip.\n7. Add the matching `apiid` struct with `UUID` and `PublicID`, its constructor,\n and its text, JSON, generated-query, and Gin binding methods.\n8. Test that UUID and public inputs produce identical API structs, wrong kinds\n are rejected, and encoding agrees with this package's codec. Update both\n READMEs and regenerate affected API contracts and clients.\n\nDo not add JSON, text, SQL, or UUID-generation behavior implicitly. Those\ninterfaces should be introduced only when a concrete caller needs them.\n","packages/shared/pkg/sandboxtypes/README.md":"# Shared sandbox types\n\nThis package owns `RuntimeMetadata`, `SandboxType`, and `EgressClass`, including\nidentity logging and the empty-type fallback to a regular sandbox. It contains\nno worker resources, network initialization, or VM lifecycle management.\n\nCallers import these types directly. `SandboxType.EgressClass()` selects the\ntraffic class; DSCP configuration and socket handling belong to worker networking.\nThe resource-owning `Sandbox` stays in the orchestrator package. Runtime team\nmetadata is attribution, not trusted authorization data.\n"},"items":[{"name":"AGENTS.md","path":"AGENTS.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/AGENTS.md","title":"AI Agent Protocol & Instructions","category":"root-instruction","format":"markdown","content":"# AGENTS.md\n\nInstructions for AI agents working in this repository.\n\nAlso read [CLAUDE.md](CLAUDE.md) for development commands and package\nconventions if it is not already in context. It applies to all agents.\n\n## Understanding the repository\n\nRead [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) before working on or reviewing code here. It describes what each service does, how services interact (sandbox creation, traffic routing, pause/resume, template builds), the data stores, and the deployment topology — it is the fastest way to build correct context about this codebase.\n\nKeep that document updated: if a code change alters anything it describes (service responsibilities, ports, protocols, data stores, flows, deployment topology), update `docs/ARCHITECTURE.md` as part of the same change. When reviewing a PR that changes such details without updating the document, flag it.\n\n## Review guidelines (pull requests)\n\nOutput style rules (strict, override any defaults):\n\n- Only report concrete bugs, regressions, or correctness issues. Do not summarize the PR, the diff, or what changed.\n- One short paragraph per finding. No preamble, no closing remark.\n- No headers, no bullet lists, no tables, no diagrams.\n- No emojis. No severity tags (\"bug\", \"nit\", \"suggestion\", \"enhancement\"). No labels.\n- No branding or footer lines.\n- If there are no real issues to flag, post no review at all.\n\n## Scope\n\n- Skip style/nit comments — `golangci-lint` covers those.\n- Skip test-coverage comments.\n- Focus on: race conditions, nil-deref, error handling, auth/authz, request routing, resource leaks, SQL/migration correctness, and gRPC/proto compatibility.\n","isInternal":false,"tokens":369,"sizeBytes":1656},{"name":"CLAUDE.md","path":"CLAUDE.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/CLAUDE.md","title":"Claude Agent Guidelines & System Prompt","category":"claude-rule","format":"markdown","content":"# CLAUDE.md\n\nThis development guide applies to Codex, Claude Code, and OpenCode.\nRepository and review rules are shared through:\n\n@AGENTS.md\n\n## Project Overview\n\nE2B Infrastructure is the backend infrastructure powering E2B (e2b.dev), an open-source cloud platform for AI code interpreting. It provides sandboxed execution environments using Firecracker microVMs.\n\n**Start with [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** — it explains what each service does, how services interact (with diagrams of the core flows: sandbox creation, traffic routing, pause/resume, template builds), and the deployment topology. Reading it first is the fastest way to understand this repository.\n\n**Keep docs/ARCHITECTURE.md updated**: if your change alters anything it describes (service responsibilities, ports, protocols, data stores, flows, deployment topology), update the document in the same change.\n\n## Common Development Commands\n\n### Setup & Environment\n```bash\n# Switch between environments (prod, staging, dev)\nmake switch-env ENV=staging\n\n# Setup local development stack (PostgreSQL, Redis, ClickHouse, monitoring)\nmake local-infra\n```\n\n### Building & Testing\n```bash\n# Run all unit tests across packages\nmake test\n\n# Run integration tests\nmake test-integration\n\n# Build specific package\nmake build/api\nmake build/orchestrator\n\n# Generate code (proto, SQL, OpenAPI)\nmake generate\n\n# Format and lint code\nmake fmt\nmake lint\n\n# Regenerate mocks\nmake generate-mocks\n\n# Tidy go dependencies\nmake tidy\n```\n\n### Running Services Locally\n```bash\n# From packages/api/\nmake run-local # Run API server on :3000\nmake dev # Run with air (hot reload)\n\n# From packages/orchestrator/\nmake run-local # Run orchestrator\nmake run-debug # Run with race detector\n```\n\n### Package-Specific Commands\n```bash\n# API: Generate OpenAPI code\ncd packages/api && make generate\n\n# Orchestrator: Generate proto + OpenAPI\ncd packages/orchestrator && make generate\n\n# DB: Run migrations\nmake migrate\n\n# Run single test\ncd packages/ && go test -v -run TestName ./path/to/package\n```\n\n### Publishing Artifacts\n```bash\n# Build and upload all service images to your GCP project\nmake build-and-upload\n\n# Build specific service\nmake build-and-upload/api\nmake build-and-upload/orchestrator\n```\n\n## Architecture Overview\n\n### Service Communication Flow\n```\nClient → Client-Proxy → API (REST) ⟷ PostgreSQL\n ↓ ⟷ Redis\n Orchestrator ⟷ ClickHouse\n ↓ (gRPC)\n Firecracker VMs\n ↓\n Envd (in-VM daemon)\n```\n\n### Core Services\n\n**API (`packages/api/`)** - REST API using Gin framework\n- Entry point: `main.go`\n- Core logic: `internal/handlers/store.go` (APIStore)\n- Authentication: API keys and OIDC auth provider JWTs\n- OpenAPI code generation: `internal/api/*.gen.go`\n- Port: 80\n\n**Dashboard API (`packages/dashboard-api/`)** - REST backend for the web dashboard, not the SDK\n- Entry point: `main.go`\n- Spec: `spec/openapi-dashboard.yml`; template tags, `/v1/management`\n- Talks to Postgres and ClickHouse; never to orchestrators (see `docs/ARCHITECTURE.md`)\n- Port: 3010\n\n**Orchestrator (`packages/orchestrator/`)** - Firecracker microVM orchestration\n- Entry point: `main.go`\n- VM management: `pkg/sandbox/`\n- Firecracker integration: `pkg/sandbox/fc/`\n- Networking: `pkg/sandbox/network/`\n- Storage: `pkg/sandbox/nbd/` (Network Block Device)\n- Template caching: `pkg/sandbox/template/`\n- Template building: `pkg/template/`\n- gRPC server: `pkg/server/`\n- Utilities: `cmd/clean-nfs-cache/`, `cmd/inspect-build/`, `cmd/dummy-orchestrator/`\n\n**Envd (`packages/envd/`)** - In-VM daemon using Connect RPC\n- Runs inside each Firecracker VM\n- Process management API: `packages/envd/spec/process/process.proto`\n- Filesystem API: `packages/envd/spec/filesystem/filesystem.proto`\n- Port: 49983\n- **`pkg/version.go` is release-please's**: it is set from the Conventional Commits merged since the last envd release, so do not bump it by hand. A behavioral change ships under a `feat` or `fix` type so a version exists for the orchestrator to gate on; comment or docs-only changes use `docs`, which the changelog hides, and cut no release (`chore` and `refactor` are visible and cut one)\n\n**Client Proxy (`packages/client-proxy/`)** - Edge routing layer\n- Service discovery via `packages/shared/pkg/servicediscovery`\n- Request routing to orchestrators\n- Redis-backed state management\n\n**Shared (`packages/shared/`)** - Common utilities\n- Proto definitions: `pkg/grpc/orchestrator/`, `pkg/grpc/envd/`\n- Telemetry: `pkg/telemetry/` (OpenTelemetry)\n- Logging: `pkg/logger/` (Zap + OTEL)\n- Database: `pkg/db/` (ent ORM)\n- Models: `pkg/models/`\n- Storage: `pkg/storage/` (GCS/S3 clients)\n- Feature flags: `pkg/featureflags/` (LaunchDarkly)\n\n**Database (`packages/db/`)** - PostgreSQL layer\n- Migrations: `migrations/*.sql` (goose)\n- Queries: `queries/*.sql` (sqlc)\n- Generated code: `queries/` (plus `pkg/auth/queries/`, `pkg/dashboard/queries/`)\n\n### Key Technologies\n\n- **go 1.26.10** with workspaces (`go.work`)\n- **Firecracker** for microVM virtualization\n- **PostgreSQL** for primary data (sqlc for queries)\n- **ClickHouse** for analytics\n- **Redis** for caching and state\n- **OpenTelemetry** for observability (Grafana stack: Loki, Tempo, Mimir)\n- **gRPC/Connect RPC** for service communication\n- **Gin** (API), **chi** (Envd) for HTTP\n\n### Code Generation\n\nThe codebase uses several code generators:\n\n1. **Protocol Buffers** (`packages/orchestrator/generate.Dockerfile`)\n - Generates: `packages/shared/pkg/grpc/*/`\n - Run: `make generate/orchestrator`\n\n2. **OpenAPI** (`oapi-codegen`)\n - Spec: `spec/openapi.yml`\n - Generates: API handlers, types, specs\n - Run: `make generate/api`\n\n3. **SQL** (`sqlc`)\n - Queries: `packages/db/queries/*.sql`\n - Generates: Type-safe DB code\n - Run: `make generate/db`\n\n4. **Mocks** (`mockery`)\n - Config: `.mockery.yaml`\n - Run: `make generate-mocks`\n\n### Testing Patterns\n\n- **Unit tests**: Use `testify/assert` and `testify/require`\n- **Database tests**: `packages/db/pkg/testutils.SetupDatabase` gives each test its own database on the PostgreSQL server `PG_TEST_DATABASE_URL` names (see [DEV-LOCAL.md](DEV-LOCAL.md#unit-tests-that-need-postgresql))\n- **Integration tests**: `tests/integration/` with shared test utilities\n- **Mocking**: Generated mocks in `mocks/` directories\n- **Race detection**: Tests run with `-race` flag\n\nExample test invocation:\n```bash\n# Single package\ngo test -race -v ./internal/handlers\n\n# Specific test\ngo test -race -v -run TestCreateSandbox ./internal/handlers\n```\n\n## Important Development Notes\n\n### Working with Proto/gRPC\n- Proto files: `packages/envd/spec/process/`, `packages/envd/spec/filesystem/`, orchestrator protos at `packages/orchestrator/*.proto`\n- Shared protos: `packages/shared/pkg/grpc/`\n- After editing proto files, run `make generate/orchestrator` and `make generate/shared`\n\n### Database Migrations\n- Migrations: `packages/db/migrations/`\n- Create: `cd packages/db && make create-migration NAME=your-migration-name` — this generates the file with a correct `YYYYMMDDHHMMSS` timestamp. Do NOT hand-create migration files; placeholder timestamps like `120000`/`000000` cause same-day version collisions and are rejected by the out-of-order-migrations CI check.\n- Apply: `make migrate` (requires POSTGRES_CONNECTION_STRING)\n- Code generation: `make generate/db` (regenerates sqlc code)\n\n### Environment Variables\n- Local development defaults: `packages//.env.local`; see [DEV-LOCAL.md](DEV-LOCAL.md).\n- Self-hosting configuration: [embed/compose/.env](embed/compose/.env); see [embed/README.md](embed/README.md).\n- Custom environment configs: `.env.`, selected with `make switch-env ENV=`.\n\n### Firecracker & VM Management\n- Orchestrator requires **sudo** to run (Firecracker needs root)\n- VM networking uses `iptables` and Linux `netlink`\n- Storage uses NBD (Network Block Device)\n- Templates cached in GCS bucket (configurable via TEMPLATE_BUCKET_NAME)\n- Kernel/Firecracker versions: `packages/fc-versions/`\n\n### Observability\n- All services export OpenTelemetry traces/metrics/logs\n- Local stack includes Grafana + Loki + Tempo + Mimir\n- Telemetry setup: `packages/shared/pkg/telemetry/`\n- Logger: `packages/shared/pkg/logger/` (Zap with OTEL)\n- Profiling: API exposes pprof on `/debug/pprof/` (see `packages/api/Makefile` profiler target)\n\n### CI/CD Workflows\n- `.github/workflows/pull-request.yml` - PR orchestrator (lint, OpenAPI, unit, arm64, integration)\n- `.github/workflows/pr-tests.yml` - Unit test shards\n- `.github/workflows/integration_tests.yml` - Integration test suite\n\n## Architecture Patterns\n\n1. **Service Isolation**: Each service runs in containers with defined gRPC/HTTP interfaces\n2. **Shared Libraries**: Cross-cutting concerns (logging, telemetry, DB) in `packages/shared`\n3. **Event-Driven**: ClickHouse + Redis pub/sub for async operations\n4. **Caching Strategy**: Redis for templates, auth tokens, performance optimization\n5. **Feature Flags**: LaunchDarkly for gradual rollouts\n6. **Graceful Shutdown**: Services handle SIGTERM with context cancellation\n7. **Health Checks**: gRPC health protocol + HTTP health endpoints\n\n## Debugging\n\n### Logs\n- Local: Docker logs in `make local-infra`\n- Production: Grafana Loki\n","isInternal":false,"tokens":2388,"sizeBytes":9434},{"name":"ARCHITECTURE.md","path":"docs/ARCHITECTURE.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/docs/ARCHITECTURE.md","title":"docs - System Architecture & Module Boundaries","category":"project-spec","format":"markdown","content":"# E2B Infrastructure Architecture\n\nThis document explains what this repository implements, what each service does, and how the\nservices interact. It is the fastest way to build a mental model of the codebase before diving\ninto the code.\n\n> **Keep this document updated.** If a code change alters anything described here (service\n> responsibilities, ports, protocols, data stores, flows, deployment topology), update this file\n> in the same PR.\n\n## What this repository implements\n\nE2B provides **sandboxes**: isolated Linux VMs that start almost instantly (they resume pre-booted\nsnapshots instead of cold-booting), run arbitrary code (typically\ngenerated by AI agents), and can be paused, snapshotted, and resumed. This repo contains the whole\nbackend: the control-plane REST API, the data-plane VM orchestration built on\n**Firecracker microVMs**, the in-VM agent, the edge routing layer, and template building.\n\nTwo ideas drive the design:\n\n1. **A sandbox is a resumed snapshot.** Templates are pre-booted VM snapshots (memory + disk +\n VM state) stored in object storage. \"Creating\" a sandbox means restoring a snapshot, which is\n why startup is fast. Memory pages are loaded lazily on page-fault (userfaultfd) and the root\n filesystem is a copy-on-write overlay, so only touched data is ever fetched. A template can\n also be filesystem-only (rootfs, no memfile), taken by a filesystem-only pause or checkpoint;\n sandboxes created from it cold-boot from disk instead of resuming.\n2. **Control plane and data plane are separate.** The API decides *where* a sandbox runs and\n tracks *that* it runs (Postgres/Redis); the orchestrator on each node owns *how* it runs\n (Firecracker, networking, storage). Sandbox traffic never passes through the API.\n\n## System overview\n\n```mermaid\nflowchart TB\n subgraph clients[\"Clients\"]\n SDK[\"SDK / CLI\"]\n Browser[\"Browser / HTTP clients\"]\n end\n\n LB[\"Load balancer
api.* | *.domain wildcard\"]\n\n VC[\"volume-content API (belt)
api.<domain>\"]\n\n subgraph controlplane[\"Control plane (API node pool)\"]\n API[\"API
REST :80, gRPC :5009/:5109\"]\n DashAPI[\"dashboard-api :3010\"]\n CP[\"client-proxy
:3002\"]\n end\n\n subgraph datastores[\"State\"]\n PG[(\"PostgreSQL
teams, templates, builds, snapshots\")]\n RD[(\"Redis
running sandboxes, routing catalog, caches\")]\n CH[(\"ClickHouse
metrics, events, optional logs\")]\n OS[(\"Object storage GCS/S3
template + snapshot artifacts\")]\n end\n\n subgraph clientnode[\"Sandbox nodes (one orchestrator per node)\"]\n ORCH[\"orchestrator
gRPC :5008, proxy :5007\"]\n subgraph vm[\"Firecracker microVM (per sandbox)\"]\n ENVD[\"envd :49983\"]\n USERPROC[\"user processes\"]\n end\n end\n\n subgraph buildnode[\"Build nodes\"]\n TM[\"template-manager
(orchestrator binary, gRPC :5008)\"]\n end\n\n SDK -->|REST| LB --> API\n Browser -->|\"port-sandboxid.domain\"| LB --> CP\n API -.->|\"mint content token + domain\"| SDK\n SDK -->|\"volume content (token-authed)
api.<BYOC or default domain>\"| VC\n API -->|\"gRPC Create/Delete/Pause\"| ORCH\n API -->|\"gRPC TemplateCreate\"| TM\n CP -->|\"lookup sandbox → node\"| RD\n CP -->|\"forward :5007\"| ORCH\n CP -.->|\"gRPC auto-resume\"| API\n ORCH --> ENVD\n ENVD --> USERPROC\n API --> PG & RD & CH\n DashAPI --> PG & CH\n ORCH --> OS & CH\n TM --> OS\n```\n\n## Services\n\n| Service | Package | Runs on | Purpose |\n|---|---|---|---|\n| API | `packages/api` | API nodes | Public REST API; sandbox lifecycle, placement, auth, quotas |\n| Orchestrator | `packages/orchestrator` | every sandbox node | Runs Firecracker VMs; sandbox create/pause/resume/kill |\n| Template manager | `packages/orchestrator` (role) | build nodes | Builds templates from Docker images |\n| Client proxy | `packages/client-proxy` | API nodes | Edge router: sandbox URL → correct node |\n| Envd | `packages/envd` | inside every VM | In-VM agent: process/filesystem API for SDKs |\n| Dashboard API | `packages/dashboard-api` | API nodes | Backend for the web dashboard and workspace control plane (sandbox records, admin, `/v1/management`) |\n\nSupporting packages: `packages/shared` (protos, telemetry, storage clients, feature flags),\n`packages/auth` (authentication library), `packages/db` (Postgres migrations + sqlc queries),\n`packages/clickhouse` (ClickHouse schema + clients), `packages/otel-collector` (collector config),\n`packages/nomad-nodepool-apm` (autoscaler plugin), `packages/local-dev` (local stack).\n\n### API (`packages/api`)\n\nThe control-plane entry point (Gin, OpenAPI-generated from `spec/openapi.yml`, port 80).\n\n- **Resources**: sandboxes (create/list/kill/pause/resume/connect/timeout/metrics/logs),\n templates and builds (including the web dashboard's build listings under `/unstable/builds`),\n teams, volumes, API keys, secrets, sandbox events and webhooks, admin operations.\n- **Default templates**: `env_defaults` names the platform's default templates. Admin routes\n under `/unstable/admin/templates` add, describe and remove default templates and aliases\n without a namespace, and clear the alias cache after each alias change so name resolution\n sees it at once.\n- **Spec document**: `GET /openapi.json` serves, without authentication, the spec the request\n validator runs. An `ETag` over the document answers a matching `If-None-Match` with 304. It is\n registered before the validator, which rejects undeclared paths.\n- **Auth** (via `packages/auth`): team API keys (`X-API-Key`, `e2b_` prefix), auth-provider JWTs\n (OIDC), and either an admin token or a service JWT verified from the configured admin JWKS.\n Backed by an auth DB (Postgres) with a Redis team cache.\n- **API-group rate limits**: operations declare a group through the `x-api-group` OpenAPI\n extension. After authentication succeeds, its callback wrapper reads the validator's\n already-matched operation and stores the API group directly in Gin's context. The rate limiter\n reads that group and maps it to a field in the authenticated team's limits. Grouped operations\n require team-identifying authentication; public operations do not populate an API group.\n The `list` group uses `api_team_rps_list`; the `delete` group uses `api_team_rps_delete` only\n for sandbox and template deletion. Other DELETE operations are ungrouped.\n Delete rates default to zero until configured.\n A group's rate is one of the team's limits and resolves with the rest of them in the statement\n that loads the team, which authentication caches; it has no source of its own.\n Propagated project limits carry both rates, and each accepted revision replaces\n them. A team with no `project_limits` row uses the deployment's `default_limits`; zero\n disables group limiting. The migration that added `default_limits` gave every existing team a\n row holding its limits at the time, so a change to the defaults reaches only teams created since.\n A positive rate sets both requests per second and burst capacity. Redis enforces a separate\n budget for each team, HTTP method, and route template across API replicas: groups select\n rates, but routes do not share a budget. Exhausted budgets return HTTP 429 with rate-limit\n and `Retry-After` headers. Ungrouped operations and unconfigured rates bypass group limiting;\n Redis failures are logged and counted while requests proceed.\n V1 route limits from `rate-limit-config` are always enforced first. For requests that pass V1,\n LaunchDarkly's `rate-limit-v2-mode` is the master switch for all V2 checks, and\n `rate-limit-delete-mode` can further restrict delete checks. Both default to `disabled`;\n missing, invalid, or unknown modes are treated as disabled. Deletes skip V2 if either flag is\n disabled and enforce only when both are enabled. If both are active and either is `shadow`,\n deletes run in shadow mode. Lists follow the global mode.\n The groups use the same limiter logic: `shadow` observes V2 decisions while\n preserving V1 headers, and `enabled` also enforces V2. Ungrouped operations, unconfigured V2\n rates, and V2 Redis errors retain V1 enforcement. A rejection reports the rejecting limiter's\n headers; when both checks allow a request in enabled mode, V2 supplies the rate-limit headers.\n- **Sandbox list snapshots**: a list request reads the team's whole sandbox index from Redis\n (its running and pausing records) and pages in memory, so with the LaunchDarkly\n `sandbox-list-max-lag-ms` flag set for a team, each API replica shares one read between\n that team's concurrent list requests and keeps it for the flag's value, counted from the\n start of the read; a request that joins a read already in flight gets that read's result\n however long it takes. A lowered flag applies to a live snapshot; a raised one from the\n next read. Within that lag a sandbox created, killed, paused or resumed can be missing from\n the list or shown in its previous state on the replica holding the snapshot, and one\n resumed inside it can appear on two pages of a walk served by different replicas. The\n default, zero, gives every request its own read with the same store commands as before.\n Only the two list endpoints use the snapshot; the evictor, reconcile, template delete and\n team teardown keep reading the store directly.\n- **Workload identity**: sandbox create accepts an optional `iam.tokens` map of caller-named\n workload-token definitions (each an exact `audience` and `tokenType`). A non-empty, validated\n map enables workload identity, whose identity the orchestrator derives from the sandbox's\n already-authoritative team/sandbox/execution/template IDs; the definitions are passed to the\n orchestrator in `SandboxConfig.iam`. The API mints no credential and delivers nothing into the\n sandbox; file-based delivery is rejected at admission. Definitions are persisted in the\n running-sandbox (Redis) and paused-snapshot (Postgres) state so they survive pause/resume and\n orchestrator re-sync; a fork starts a new workload and does not inherit them.\n- **Placement**: keeps a live map of orchestrator nodes (discovered via Nomad, Kubernetes, or a\n static list). Chooses a node per sandbox with a **best-of-K** algorithm\n (`internal/orchestrator/placement/`): sample K ready nodes, score by the\n higher of CPU load and hugepage-pool load, pick the lowest; retry on\n exhausted nodes. A node that refuses a create (`ResourceExhausted`) stays\n eligible, and each refusal waits a full-jitter backoff that doubles up to a\n cap, so a placement that finds every node busy does not re-send creates as\n fast as they are refused. The hugepage-pool load is measured against the node's\n base pool (`nr_hugepages`, allocated at boot), so a node that has borrowed surplus\n pages scores above 1 instead of reading as just full; a pool that is all overcommit\n is measured against its ceiling, and an orchestrator that does not report the base\n pool against `HugePages_Total` as before. A node that reports no pool scores 0.5. Hugepage scoring\n is on by default (`BEST_OF_K_HUGEPAGE_MEMORY`); set it false to rank on\n CPU alone. K, overcommit ratio, and alpha are tunable live via feature\n flags. With `placement-hugepage-limit-percent` above 0, a hugepage sandbox\n is also refused by any node whose committed hugepages would pass that share\n of the node's pool ceiling (base pool plus overcommit allowance, reported by\n the orchestrator); within an API replica the check and the in-flight record\n are one step per node, and when the compatible nodes are all full placement\n answers with a capacity error, not \"no compatible node\". With the\n orchestrator's `create-memory-refusal` flag on (`CREATE_MEMORY_REFUSAL` for a\n deployment without LaunchDarkly), a create whose guest memory the pool\n cannot back while a sandbox on that node is still tearing down is refused\n with `ResourceExhausted`, so placement retries the node once the pages are\n back; with nothing tearing down it fails at once and says why. Off, it fails\n as an internal error and the node is excluded.\n- **State**: writes sandbox records to Redis (source of truth for *running* sandboxes). It does\n not write routing: client-proxy reads the orchestrator-written `sandbox:routing:{id}` (see\n \"Sandbox routing records\"). Persistent entities (templates, builds, snapshots, teams) live in\n Postgres.\n- **Secrets**: `/secrets` is the only public surface for secret management (create, list, get,\n update, delete). The API authenticates the caller with the customer alternatives above, converts\n the authenticated team UUID to the project UUID the backend knows, checks the `customer-secrets`\n feature flag, and forwards a metadata-only request over a unary gRPC contract\n (`e2b.secretsstore.management.v1`) to the secrets store backend named by\n `SECRETS_STORE_BACKEND_GRPC_ADDRESS`. No caller credential, header or client-supplied tenant\n crosses that hop, and no response - here or in a log, a span or an error - carries a secret\n value. What a caller stores is a runtime marker, not a resolved secret; orchestrator-ee resolves\n a marker to a value at sandbox egress, never here. Without a configured address, or with the flag\n off, the routes stay registered and answer 403. Responses are `Cache-Control: no-store`, request bodies are capped at 512 KiB, and values\n at 64 KiB. A create carries the cap on live secrets the backend enforces: `max_secrets` is one\n of the team's limits and resolves with the rest of them in the statement that loads the team (its\n `project_limits` value, else the deployment's `default_limits`).\n- **Sandbox events and webhooks**: `/events/sandboxes` reads a team's sandbox lifecycle events,\n and `/events/webhooks` manages the webhook subscriptions those events are delivered to, along\n with each subscription's delivery attempts and delivery statistics. The API stores none of\n this: it authenticates the caller, resolves the team's cluster, and forwards the request over\n two unary gRPC contracts (`e2b.webhooks.events.v1` and `e2b.webhooks.management.v1`) to the\n backend named by `WEBHOOKS_BACKEND_GRPC_ADDRESS`. The team and cluster come from the\n authenticated context, never from the request, and a team with no cluster selects the local\n one. Without a configured address the routes stay registered and answer 503, so the surface\n does not depend on how a deployment is wired. A backend failure the caller cannot act on is\n answered with the endpoint's own message; the outcomes that keep their identity are\n not-found, a rejected request, the webhook-count limit, and a timeout. A create or update\n body carries the webhook's signing secret, so a request rejected before the handler has its\n body kept out of the response, the access log and the span. Reads are capped at 100\n deliveries per page.\n- **Rig passthrough**: admin endpoints under `/clusters/{clusterID}/rigs` manage a cluster's\n orchestrator node pools (\"rigs\"). They list rigs, change rig capacity, list and terminate\n instances, and read recent scaling errors. Every handler delegates to the cluster's edge service\n (`/v1/rigs/...`) through the per-cluster HTTP client that the cluster sync keeps fresh. The API\n holds no cloud credentials and makes no scaling decisions. The caller authenticates with an\n admin credential and never holds the cluster's edge secret. Edge status codes pass through unchanged\n (400, 404, 409, 501). An edge 401 becomes a 500: it means the API is misconfigured against the\n cluster, not that the caller's token is invalid. A cluster with no rig management configured\n returns an empty list. The local cluster has no edge deployment and answers 501 for every rig\n operation.\n- **Extra listeners**: internal gRPC :5009 and edge gRPC :5109 expose `ResumeSandbox` so\n client-proxy can wake paused sandboxes on incoming traffic.\n- **Outbox** (`internal/outbox`): a River client on the shared database's `river` schema\n works jobs that need the orchestrator connections. Its job arguments live in\n `packages/db/pkg/outbox`. The one kind today, `teardown_team_resources`, stops a deleted\n team's workloads; the API's own project deletion enqueues it. A\n guard runs first on every attempt: a team that exists and is not blocked fails the attempt,\n and the job retries rather than kill a live team's sandboxes; a missing team row does not\n stop it. Then\n `kill_sandboxes` kills the team's running sandboxes and succeeds only once the sandbox store\n holds none of them in any state, so a sandbox still pausing or being killed makes the job\n retry. A sandbox that is not running cannot be paused, so no new snapshot of the team\n appears afterwards. Killing through the orchestrator is what removes the sandboxes' store\n records, so the job deletes no Redis state itself, and every attempt is safe to run again.\n Builds are not cancelled: they end on their own build timeout. An attempt is bounded at\n 30 minutes; retries back off from 30 seconds to a 15-minute cap, so the 100 attempts last\n about a day before River discards the job and keeps it. Volumes, templates, snapshots and\n the team row are left alone. Steps report\n `api.outbox.steps.finished` and `api.outbox.step.duration`; finished jobs report\n `outbox.jobs.finished`, and the backlog reports `outbox.jobs` and `outbox.oldest_*_age`.\n `OUTBOX_MAX_WORKERS` (default 10) and `OUTBOX_BACKLOG_INTERVAL` (default 30s) configure it;\n shutdown stops it after the sandbox-work drain, before the database and Redis clients close.\n- Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to\n Loki, with a LaunchDarkly-gated ClickHouse read path (`logs-read-config`) for local-cluster logs\n during the log storage migration. `LOKI_URL` is optional: without it the api has no Loki client\n and those reads fail until the flag routes them to ClickHouse. `LOGS_READ_CONFIG=true|false` sets\n the value the flag falls back to when LaunchDarkly has none (no LaunchDarkly, or the flag not\n defined there); a LaunchDarkly value wins. LaunchDarkly feature flags also gate placement\n parameters, rate limits, and rollouts.\n\n#### Project management writes (`/unstable/management`)\n\n`PUT /unstable/management/projects/{projectID}/settings` applies a revisioned settings\nprojection from the owning management service. This endpoint belongs to the API\nand accepts only service JWTs through `AdminJWTAuth`. It locks the team and checks\nthat it has not been deleted before applying the projection. The shared `ProjectSettings` type\ncontains `SandboxSettings`, with the optional ingress default under\n`sandboxSettings.defaultAllowPublicTraffic`. Storage uses `projection.project_settings`,\nwith `revision` and nullable `sandbox_default_allow_public_traffic` in the same row.\nThe receiver applies both in one guarded upsert. Equal or older revisions cannot\noverwrite it. Clearing stores NULL at a newer revision, retaining the ordering fence.\nCache invalidation follows commit and is retried on duplicate\ndelivery after a cache failure. There is no rollout flag check on accepted delivery.\nSandbox creation resolves an absent default to true and honors a saved default\nindependently of rollout eligibility. New sandboxes can use an older value while\ndelivery or cache refresh is pending.\n\nThe owning management service also drives the project writes under\n`/unstable/management/projects/{projectID}`. Like the settings route, each accepts only service\nJWTs through `AdminJWTAuth`, and each is idempotent, because the caller is level-triggered and\nretries.\n\n- `PUT /unstable/management/projects/{projectID}` creates or reconciles a project and answers 201\n or 200 to say which. A project is a `public.teams` row created from a caller-supplied UUID, with\n no tier. An optional cluster ID is stored on the project; omitting it preserves the current\n assignment. A changed slug renames the project and, in the same transaction, gives each of its\n namespaced template aliases under the new slug as well, keeping the old ones and leaving an\n alias the new slug already holds as it is. It answers 409 when another project already holds the slug, when the\n request assigns a cluster whose deletion is recorded, or when a concurrent request created the\n same project first.\n- `PUT .../members/{userID}` applies the desired presence for one user, gated by a monotonic\n per-project/user revision in `projection.project_members`; a duplicate or older revision\n succeeds without changing target state. A present projection carries the user's OIDC\n issuer/subject identities. Every projected user has at least one, and an identity already owned\n by a different user returns 409. A revocation removes only that user's `users_teams` row;\n projected users and identities are retained. Auth caches member authorization, so each accepted\n command evicts that user's authorization for the project after commit.\n- `PUT .../limits` syncs the project's effective limits into `project_limits`, which the team\n lookups read in preference to `default_limits`. It is gated by a monotonic per-project revision\n in `projection.project_limits`: the caller raises it whenever the limits it resolved change, and\n a delivery at or below the recorded revision is dropped and still answers 204. The ledger and the\n values advance in one transaction, so a revision is never recorded without the values it\n admitted.\n- `PUT .../block` applies a revision-fenced block state: the `projection.project_blocks` revision\n and `teams.is_blocked` / `blocked_reason` commit together, and an older or duplicate revision\n leaves stored state unchanged. Every accepted delivery, a dropped one included, evicts the team's\n auth cache; a failed eviction answers an error so the retry finishes it after commit. The API\n reads this state when admitting billable work. Apart from project deletion, which blocks the\n team for good, this is the only route that writes it.\n- `DELETE /unstable/management/projects/{projectID}` soft-deletes the project, described below.\n\nThe revision fences are the target's for the same reason: the caller can only fence what it\nsends, so two deliveries in flight arrive in whichever order the network gives them, and the older\none has to be refused where it lands. The project, membership, limits and block writes live in\n`internal/management` with their post-commit cache evictions rather than in the handlers.\n\nA limits or block delivery may carry `decided_at`, when the caller decided that revision; the\nledger row stores it beside the revision. Each delivery the ledger accepts records the seconds\nsince then in `dashboard-api.management.apply_lag` with `projection` `project_limits` or\n`project_blocks`; one without `decided_at`\ncounts in `dashboard-api.management.origin_unknown` instead, never as zero lag. Dropped, refused\nand failed deliveries record nothing. The names keep the dashboard-api prefix from when\ndashboard-api served these writes, so the alert and dashboard reading them carry on unchanged.\n\nProject deletion soft-deletes any project, including a member's default project; whether a project\nmay be deleted is the caller's decision. `envs`, `snapshots` and `volumes` reference `teams` with\n`ON DELETE NO ACTION`, so the team row stays. One transaction:\n\n- locks the row (404 if it is missing) and sets `teams.deleted_at`;\n- blocks the team (a CHECK keeps every deleted team blocked) and clears its cluster assignment;\n- scrubs its name and email to `DELETED` and its slug to `deleted-`, releasing the\n old slug for reuse;\n- deletes its memberships and API keys;\n- soft-deletes its templates, snapshot templates and paused-sandbox snapshots the way a template\n deletion does, releasing their aliases and build tracking;\n- enqueues `teardown_team_resources` with `InsertTx` on the outbox's own River client, so the\n teardown runs exactly when the deletion commits.\n\nAfter commit it evicts the deleted keys' auth cache entries best-effort; the team's other cached\nentries refresh or expire on their own. Repeating the call answers 204 and runs every step again,\nkeeping only the first deletion's time. That undoes a block reason, rename or cluster assignment\nwritten by a delivery that passed its deleted-project check just before the first call committed;\ndeletes memberships, keys and envs created since (a sandbox that paused before the teardown killed\nit leaves a snapshot); and enqueues the teardown again, which is a no-op while one is still open.\n\nAfterwards the API's deleted-project middleware, running after authentication, answers 404 to every\nnon-GET route under `/unstable/management/` that names `{projectID}`: the upsert, member, limits,\nblock and settings writes. The DELETE is exempt so a retry still reaches it. dashboard-api guards its own\nmanagement and admin writes the same way. Shared auth refuses a membership or API key still left\nfor the project with 403, and the API leaves the project out of a user's team list. Volumes, builds\nand stored artifacts are not reclaimed.\n\n#### Team bans (`/unstable/admin/teams`)\n\n`PUT /unstable/admin/teams/{teamID}/ban` sets `teams.is_banned`, evicts the team's auth cache so\nits API keys stop authenticating, then kills its running sandboxes and cancels its builds through\nthe same orchestrator and template-manager calls as `POST /admin/teams/{teamID}/sandboxes/kill`\nand `.../builds/cancel`. The ban commits first: a kill or cancel step that fails is reported in\nthe 200 response and does not undo it. The handler detaches the request from the caller and gives\nit one 25-second budget at the same moment, so a caller that disconnects does not stop it and\nnothing it runs (deletion checks, ban writes, cache evictions, kills, cancellations) blocks past\nthe budget. A ban write or eviction that runs out of it fails like any other (500, or `failed` in\nthe bulk route). Each team's kill and cancel run side by side; a step the budget cuts short is\nreported as failed with what it counted so far. A build delete already under way keeps its own 30\nseconds, so a request ends within about 55 seconds. A failed cache eviction answers 500, so the\ncaller's retry repeats the ban and its containment; banning a banned team contains it again. `DELETE .../ban` clears the flag and evicts the cache.\n`POST /unstable/admin/teams/bans` takes up to 100 teams. It stores every ban first, five teams at\na time, then contains the banned teams, five at a time, all under the one budget, and sorts each team\ninto banned, not found or failed; one team's failure does not stop the others. A deleted project\nanswers 404 to the single-team routes and counts as not found in the bulk route. The logic lives\nin `internal/team`.\n\n### Orchestrator (`packages/orchestrator`)\n\nA single Go binary running on every sandbox node (as root). `ORCHESTRATOR_SERVICES` selects its\nroles: `orchestrator` (run sandboxes) and/or `template-manager` (build templates). Code lives\nunder `pkg/`, almost all Linux-only.\n\ngRPC services on :5008 (`pkg/server/`, `pkg/service/`, `pkg/template/server/`, `pkg/volumes/`):\n\n- **SandboxService** — `Create`, `Update`, `List`, `Delete`, `Pause`, `Checkpoint`.\n `Update` callbacks are serialized per sandbox. A successful transition out of the live\n map closes update admission and joins any admitted callback before stop/checkpoint\n work or stopping notifications begin; `Close` repeats that check for crash cleanup.\n An update using an inactive sandbox returns gRPC `NotFound`, which the API maps to\n HTTP 404 for timeout and network updates. Teardown waits for admitted callbacks;\n it cannot preempt one that does not return.\n- **TemplateService** — `TemplateCreate`, `TemplateBuildStatus`, `TemplateBuildDelete` (template-manager role only).\n- **InfoService** — node identity, roles, capacity, health status (used by API node discovery).\n- **ChunkService / VolumeService** — peer-to-peer template chunk serving; persistent volumes.\n\nProcess shutdown, triggered by `SIGINT`, `SIGTERM`, `SIGUSR1`, or a service failure, moves\nthe node to `ShuttingDown` before draining builds and sandboxes. Like `Draining`, this excludes new\nplacement while existing work stays reachable, and `/health` returns HTTP 200 with status\n`draining`. Unlike reversible `Draining`, `ShuttingDown` is terminal for the process;\nstatus overrides cannot enter or leave it. `FORCE_STOP` skips waiting for sandboxes to exit.\nThe sandbox drain ends only when no sandbox is live and the node's outstanding work (below) is\nzero, so a create or a resume-fresh checkpoint in flight, whose sandbox is briefly not live, holds\nit until its sandbox goes live and then ends.\n\nAfter sandbox drain and snapshot uploads complete, sandbox proxy cleanup attempts graceful HTTP\nserver shutdown with a separate deadline, then forces server closure if needed. Successful forced\nclosure recovers graceful deadline expiry; other server errors still propagate. Pprof stays\navailable through service teardown; its final shutdown also has a deadline and forced-close fallback.\n\n`InfoService.ServiceInfo` reports `outstanding_work` for sandbox, template-builder, and\nmixed-role nodes. It counts overlapping work holds, not distinct sandboxes or builds, and includes\ntracked background persistence and cleanup. The producer reports zero when idle. The API caches\nthis report and exposes it as required top-level `outstandingWork` in admin node list and detail\nresponses, where zero also covers a node that has not reported. This observational count does not\nby itself authorize node deletion.\n\n`InfoService.ServiceInfo.max_sandboxes` reports the cached node limit for every service role.\nAdmin `/nodes` and `/nodes/{nodeID}` return required `maxSandboxes`, including zero. Admission,\nmetrics, and node info share the value refreshed from node-scoped flags every 30 seconds.\n\nKey mechanisms (all under `pkg/sandbox/`):\n\n- **Firecracker** (`fc/`): each sandbox is one Firecracker process in its own cgroup and network\n namespace. The FC HTTP API (unix socket) configures machine, drives, network, and snapshots.\n Guest metadata (sandbox ID, envd access token hash) is passed via MMDS.\n A sandbox may use fewer CPUs than its VM has (`SandboxConfig.max_vcpus` above `vcpu`): the VM\n is created with `max_vcpus` vCPUs, a boot puts `maxcpus=` on the kernel command line, and\n the `fc_vcpu` threads are held to `vcpu` CPUs by a `cpu.max` quota in a threaded `vcpu` child of\n the sandbox cgroup, so the VMM's own threads stay outside it. The quota is in place before the\n guest runs an instruction: a boot limits the whole cgroup until the vCPU threads exist and then\n moves the limit onto them; a resume confines the paused threads before the VM resumes, falls\n back to the whole cgroup, and fails if neither holds. The snapshot's metadata records the VM's\n vCPU count (`vcpu_count`, stamped at build and on every pause), so a resume limits below it\n without trusting the request or discovering capacity from cgroup threads. A legacy snapshot\n carries its stored original vCPU count as `max_vcpus`; if unavailable, it may only resume at its\n unchanged `vcpu`.\n `/init` carries the count as `cpuCount`\n so envd onlines or offlines guest CPUs to match and reports the outcome on `X-Envd-Cpus` and\n `/metrics`.\n- **Lazy memory / UFFD** (`uffd/`): on resume, Firecracker restores the VM without loading\n memory; a userfaultfd handler serves page faults directly from the template's memfile, so only\n touched pages are read. An optional prefetcher warms known-hot pages.\n- **Copy-on-write rootfs** (`rootfs/`, `nbd/`, `block/`): the template rootfs stays read-only;\n writes go to a per-sandbox COW cache exposed to Firecracker as an NBD block device served by\n an in-process userspace NBD server. On pause, the dirty blocks are exported as a diff.\n- **Template cache** (`template/`): templates are fetched lazily from object storage and cached\n on local disk (and optionally on a shared NFS chunk cache, or fetched peer-to-peer from other\n nodes before upload completes). Every reader of a cached template holds a pin on it — a\n running sandbox, a build, a peer stream, an ancestor lookup — and eviction never closes a\n pinned template. A snapshot's own upload holds a pin while it runs and records on the entry\n whether the snapshot landed in storage; a layer that did not land is never released, so a\n failed upload's local copy stays until its TTL expires. When a pause, or a\n checkpoint that resumes a fresh sandbox, publishes a new layer, the layer its sandbox ran\n from is marked superseded if this node inserted it for an earlier pause, because no resume\n can ask for it again. With `snapshot-cache-release-superseded` and\n `snapshot-cache-ancestor-storage-fallback` both on, a superseded layer is released once no\n pin holds it, its devices have resolved and its upload has landed, instead of staying until\n its TTL expires. A descendant's upload then heals that ancestor's build data from storage,\n and writes the same header bytes it would have written with the ancestor still cached.\n The sandbox and template caches start empty after a restart: the template cache empties the\n build-data directory, and startup reclaim empties the sandbox, template and template-build\n scratch directories.\n- **Networking** (`network/`): each sandbox gets a slot — a network namespace with a veth pair\n and a tap device, unique host-side IP (from a /16), NAT, and per-slot nftables egress firewall\n (with SNI/Host-inspecting TCP firewall for domain allow/deny lists). Slots are pooled and\n reused; slot indexes are allocated locally against the node's netns state (leftover\n namespaces from a previous run are torn down by startup reclaim).\n- **Sandbox proxy** (:5007, `pkg/proxy/`): reverse-proxies incoming traffic from client-proxy to\n the sandbox's slot IP and requested port over HTTP or configured HTTPS, enforcing per-sandbox\n traffic access tokens through the shared ingress policy. HTTPS backends may use self-signed certificates.\n It refuses a full-router sandbox with 503 `sandbox_feature_unsupported` before dialing the\n slot.\n Each admitted lifecycle owns its proxy handlers and backend sockets, including pending dials\n and upgrades. Stopping seals admission and interrupts connections; network release waits for\n that work to finish. Restarting envd resets the owner while allowing the live lifecycle to\n admit a replacement. Retired streams have one second to write their terminal response before\n a stalled frontend is interrupted.\n Shared proxy shutdown waits for H2C and upgraded connections to close and includes them in\n forced connection cleanup. Failed H2C handshakes release their hijacked connections when the\n handshake handler returns.\n- Writes sandbox lifecycle **events** and cgroup **host stats** to ClickHouse; exports metrics via\n OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy\n collector remains the fallback primary destination, and configured shadow destinations can mirror\n writes during collector/storage migrations without changing sandbox behavior.\n\nDetached sandbox-event publication holds node work until the publisher returns. This covers\ndelivery attempts, not end-to-end delivery: the ClickHouse target enqueues into an in-memory\nbatcher, whose flushing remains part of shutdown.\n\n### Envd (`packages/envd`)\n\nThe agent inside every VM (started by systemd very early in boot), port 49983, chi + Connect RPC.\n\n- **Process service** (`spec/process/process.proto`): start/list/connect to processes, stream\n stdout/stderr, stdin, signals, PTYs — this is what SDKs use to \"run code\". A start is refused\n with `unavailable` while the workload cgroups are frozen for a pause, and the freeze waits for\n in-flight starts, so a pause does not capture a child stopped before exec in a frozen cgroup. A\n start into a cgroup the guest froze itself is refused the same way. Starts carrying the system\n tag are unaffected: they stay in envd's own cgroup, which no freeze covers.\n- **Filesystem service** (`spec/filesystem/filesystem.proto`): stat/list/make/move/remove/watch.\n- **REST**: `/health`, `/metrics`, `/envs`, `/files` upload/download, `/init` (orchestrator pushes\n env vars, access token, metadata and the online CPU count after boot/resume), `/upgrade` (live self-upgrade, below),\n freeze/thaw hooks used during pause.\n- **Public vs. control-plane routes**: the control routes (`/init`, `/collapse`, `/upgrade`, and the\n freeze/thaw hooks) are marked `x-internal: true` in `spec/envd.yaml`, and `/upgrade` — which the spec does not\n describe — is listed alongside them in the orchestrator's `pkg/sandbox/envd`. The orchestrator\n reaches them over the host network at the sandbox slot IP; the sandbox proxy refuses them with a\n 404, for every method, so they are not reachable through a sandbox URL. Adding a control route\n therefore means marking it in the spec (`go generate` carries the marker into the proxy's\n rejection list) — otherwise it ships reachable from the internet.\n- **Guest trust** (`internal/host/cacerts.go`): `/init` supplies the desired CA bundle.\n envd atomically records its PEM and owned DER identities in `/run/e2b/ca/state.json`,\n then publishes a complete replacement of the active bundle with the requested roots\n before returning. Ordinary installs copy the existing bundle without decoding it;\n recovery first repairs interrupted writes. All supplied identities are owned,\n including identical preexisting copies. A background loop resumes saved intent at startup\n and retries cleanup every second. It atomically removes retired roots from the active\n bundle before rootfs work, so source-file failures cannot delay that cleanup.\n It persists each requested certificate as\n `e2b-.crt` in `/usr/local/share/ca-certificates`, removes retired managed files,\n and retains retirement identities until both bundle and source cleanup succeed.\n Superseded cleanup cannot publish an older bundle. Legacy-file removal checks the latest\n desired roots under the init lock, preserving hash links to a still-requested certificate.\n State survives envd process restart,\n not guest reboot. Retired roots may be trusted until cleanup succeeds, and an independent\n trust-store rebuild can restore a copy from a customer source file. Applications that\n cached their trust store must reload it.\n- **Auth**: `X-Access-Token` header checked against a token delivered via Firecracker MMDS;\n signed URLs for file endpoints. `/init` is exempt (it is what *delivers* the token), which is the\n main reason the proxy refuses it outright.\n- **Live upgrade** (`internal/services/process/upgrade.go`): an authenticated `POST /upgrade` lets\n the orchestrator swap envd inside a *running* sandbox at resume. It streams the new binary in the\n request body and envd `syscall.Exec`s into it **with the same PID**, carrying the workload's\n stdio/PTY fds, process table, recently-retained exit codes and filesystem watchers forward via a\n tmpfs handover blob. The workload cgroups stay frozen until the post-upgrade `/init` restores the\n access token (so no re-adopted process runs unauthenticated), and the handover outcome\n (procs/watchers re-adopted, plus any failures) rides back on that `/init`'s `X-Envd-Handover`\n header for fleet visibility.\n- Scans guest ports and forwards them so any port a user process opens becomes reachable through\n sandbox URLs. **`pkg/version.go` carries the released version**, maintained by release-please from\n the Conventional Commits merged since the last envd release; do not edit it by hand. The API and\n the orchestrator gate features on the envd version recorded in each template build — the\n orchestrator's held `/init` is the one gate that reads the version of the envd expected to\n answer instead (see sandbox creation) — so a behavioral change ships under a `feat`/`fix`\n commit type that cuts a version it can be gated on.\n\n### Client proxy (`packages/client-proxy`)\n\nThe stateless edge for all sandbox traffic (port 3002; health on 3003). Terminates\n`https://-.` requests (host parsing in `packages/shared/pkg/proxy/host.go`),\nlooks the sandbox up in the Redis routing record to find the owning node, and reverse-proxies to\nthat node's orchestrator proxy on :5007 by default. `ORCHESTRATOR_PROXY_PORT` selects a\ndifferent downstream port when the node proxy listens elsewhere. If the sandbox is not in the record (paused), it calls\nthe API's `ResumeSandbox` gRPC — paused sandboxes wake transparently on traffic.\nThe record is the orchestrator-owned `sandbox:routing:{id}` (see \"Sandbox routing records\"\nbelow).\n\nThe record on a hit, or the `ResumeSandbox` answer on a miss, is the sandbox's route, placement\nincluded. A miss uses the resume answer directly and does not read the record again. A legacy\n(no placement) or egress-router route goes to `orchestrator_ip` on the orchestrator proxy port,\nnever to the placement's ingress endpoint. A full-router route goes to the Router's ingress\nendpoint saved in its placement, never to an orchestrator, over the same pooled, kept-alive\nconnections as the orchestrator hop: the proxies never resend a request themselves (only its\ndial is retried), and net/http's resend of a bodyless idempotent request whose reused connection\nclosed before any response is accepted on both hops. A full-router route without that endpoint,\nan unknown mode or a missing `orchestrator_ip` answers the not-found page before anything is\ndialed.\n\nConcurrent requests that miss the record for the same sandbox, port and credentials share one\n**resume flight** per client-proxy process: the first request's call runs detached from its\nconnection under a 90 s ceiling and later requests join it, so one resume RPC serves them all\nand a client that disconnects mid-resume does not fail the others. A call whose requests have\nall left runs to its end, so a client that comes back finds the sandbox awake, but no further\nattempt starts for nobody, and a process keeps at most 64 such callerless calls; beyond that a\ndeparture cancels the proxy's call. That bound and that cancellation apply to the proxy's flights\nonly: the API's resume entry point lets a create already under way run on past the proxy's\ncancellation, so the node finishes the create and the API registers the sandbox inside the\nreservation's 90 s window (the create may run until ten seconds before the pending reservation can\nbe reaped and the registration until five seconds before, so the run-on never lets a fresh\nreservation start a second resume of the same snapshot), and the API's shutdown drain waits for\nthat work, bounded at 95 s. The run-on is granted when the resume is admitted, and an API already\ndraining admits none: a resume that arrives then keeps its caller's cut, so a create the proxy\ncancels mid-drain is cut with it and the reservation lapses. The sandbox otherwise lives its\nordinary timeout or auto-pause, billed, with nobody waiting for it. With\n`client-proxy-resume-parking` on (default off), a request whose flight got a still-transitioning\nor `Unavailable` answer parks instead of failing: the flight retries\nwhile a parked request waits, under a budget (`client-proxy-resume-parking-budget-ms`, 15 s from\nthe first retryable failure) that gates only whether another retry starts: every attempt, a late\nretry included, runs under its full 90 s ceiling while a caller waits on it, so a parked request\ncan wait up to the budget plus one attempt. The proxy cuts a retry at the budget's end only once\nevery caller has left, and that cut ends the proxy's wait, not the create on the node.\nA process keeps at most\n`client-proxy-resume-parking-lot-size` (512) parked requests, shedding the rest. A resume the\ncontrol plane cannot carry out right now (no capacity, no node, the API unreachable, the budget\nspent) answers **503** with `Retry-After: 2` and no error code; a request shed because the lot was\nfull answers the same **503** without the header and with `proxy_parking_lot_full` as its\n`error_code`, so a client, or one behind an ingress that strips the header, still tells the two\napart. The SDKs do not act on the header either way: their status-based retry\nwraps only the control-plane API client, and sandbox traffic through the proxy is retried only on\na connection failure, never on a 503. The omission is for hand-rolled retry loops and browsers\nthat honour the header, which would otherwise come back in one wave while the lot is still full,\nand for a client that does the SDKs' jittered backoff on its own; a placement no retry helps, which\nthe API names in an `ErrorInfo` detail on the RPC status, is not retried and answers without the\nheader, with the status the API carried: **503** for a feature no orchestrator in the cluster\nsupports, **500** (`sandbox_create_refused`) for a sandbox the node refuses to create, such as a\nstored build whose vCPU count is outside the node's bounds; a sandbox still pausing\nanswers **409** with the header on a user port and the same page as **503** on the envd port (the\nSDK reads an envd 409 as \"already exists\"); a request whose client left before it was routed\nanswers 408.\n\n### Dashboard API (`packages/dashboard-api`)\n\nA separate REST service (port 3010, spec `spec/openapi-dashboard.yml`) consumed by the web\ndashboard and the workspace control plane, not the SDK: the killed-sandbox record, admin routes\n(clusters, team cluster assignment, user-profile resolution) and the `/v1/management`\noperations. Template reads are served by the API, not here. Its admin routes accept either a\nshared admin token or a short-lived service JWT verified against the configured admin JWKS. The\nsandbox record route accepts dashboard user auth only. Its workspace-agnostic `/v1/management` operations are defined in the\nsame dashboard OpenAPI contract and registered on the existing router. Their `AdminJWTAuth`\nOpenAPI security scheme accepts only short-lived service JWTs verified against the workspace-api\n`/.well-known/jwks.json` endpoint, with accepted signing methods derived from each JWK's required\n`alg` metadata. Issuers and audiences are configured through the JSON `ADMIN_AUTH_PROVIDER_CONFIG` value —\nthe same config shape as `AUTH_PROVIDER_CONFIG`. Talks to Postgres and ClickHouse; never talks to\norchestrators. The web dashboard itself is a separate repository (github.com/e2b-dev/dashboard)\npublished as an image; E2B Embed runs it on port 3001 beside dashboard-api.\n\nAn issuer normally configures at least one accepted audience, binding a JWT to its intended target.\nAn issuer may omit `audiences` only with no `audienceMatchPolicy`; this deliberately disables\naudience matching while issuer, signature, and temporal-claim verification remain required.\n\nThe `/v1/management` operations are the cluster's half of a contract the workspace residency owns:\nthe cluster lifecycle below. Project upsert, membership, limits, block state and deletion are\nserved by the API under `/unstable/management/projects/{projectID}`, and team bans under\n`/unstable/admin/teams` (see API above).\n\nThe management surface owns a replay-safe cluster lifecycle. A caller registers a stable\ncluster UUID, assigns it only to the named project, detaches that exact assignment before provider\ncleanup, and deletes the cluster only after no project references it. Replaying the same\nregistration or assignment succeeds. Registering the cluster again with the same name and auth\norganization may change its endpoint, TLS setting, sandbox proxy domain and token, for example when\nthe cluster moves to a new domain; the API rebuilds that cluster's clients on its next sync and gives\nsandboxes the new domain. A changed name or auth organization, a different assignment, an inexact\ndetach, or deletion of a referenced cluster returns a conflict. The\nassignment does not check the project's tier; the caller decides which projects may use a cluster.\nA cluster first registered through this surface starts without deletion protection; one created any\nother way starts protected, a later registration never changes it, and deleting a protected cluster\nor checking its destroy readiness returns a conflict.\nA caller that deletes a cluster's projects before the cluster first lists them with\n`GET /v1/management/clusters/{clusterID}/projects`, which returns every project whose team is\nassigned to the cluster however it was assigned, leaving out deleted teams, and records the deletion with\n`PUT /v1/management/clusters/{clusterID}/deletion` before deleting any of them. That request\nlocks the cluster row, returns a conflict while deletion protection is on, and sets\n`clusters.deletion_requested_at`; it succeeds for an absent cluster and on repeat. From then on a\ncheck constraint keeps deletion protection off, and a trigger on `teams` refuses to newly assign a\nteam to the cluster (`cluster_assignment_cluster_deleting` from the assignment route, a conflict\nfrom the API's project upsert carrying the cluster ID). The trigger takes a key-share lock on the cluster row\nbefore it checks, so an assignment either commits before the deletion is recorded, where the\ncaller's next listing finds it, or is refused. Deleting each project clears its team's cluster,\nthe cluster delete clears it from any team deleted earlier, and then succeeds.\nThe shared error response supports an optional `error_code`, defined by the dashboard OpenAPI\n`ErrorCode` schema. Registration and assignment rejections currently populate it; other errors\nmay omit it. Callers use that stable code rather than the human-readable message, and treat\nuncoded or unknown reasons as generic errors.\n\nA project deletion keeps the team row, so dashboard-api middleware, running after\nauthentication, answers 404 to every write under `/v1/management/` or `/admin/` that names a\ndeleted project by `projectID` or `teamID` (\n\n","isInternal":false,"tokens":11464,"sizeBytes":50064},{"name":"README.md","path":"packages/api/internal/sandbox/reservations/redis/README.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/api/internal/sandbox/reservations/redis/README.md","title":"redis Documentation","category":"plugin-manifest","format":"markdown","content":"# Redis Reservation Storage\n\nThis package coordinates sandbox creation reservations across API instances.\n\n## Keys\n\n- Storage index: `sandbox:storage:{teamID}:index`\n- Pending zset: `sandbox:storage:{teamID}:reservations:pending`\n- Result key: `sandbox:storage:{teamID}:reservations:{sandboxID}:result`\n- PubSub routing key: `sandbox:storage:{teamID}:reservations:{sandboxID}:notify`\n\n## Flow\n\n`Reserve` runs a Lua script that atomically removes stale pending entries, checks whether the sandbox already exists or has already pending start, enforces the team limit using `SCARD(storage index) + ZCARD(pending zset)`, deletes any stale result key, and adds the sandbox ID to the pending zset.\n\nWhen creation completes, it removes the sandbox from the pending zset, writes a TTL result key containing either the sandbox or the creation error, and publishes the routing key.\n\nA waiter subscribes to the routing key, probes the result key immediately, then waits for PubSub notifications or the 1 second fallback ticker. PubSub is best-effort; the fallback ticker is required for correctness.\n\n`Release` is called when the sandbox is removed from storage (`Store.Remove`). It removes the sandbox from the pending zset, deletes the result key, and publishes the routing key.\n","isInternal":false,"tokens":276,"sizeBytes":1270},{"name":"readme.md","path":"packages/envd/internal/services/legacy/readme.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/envd/internal/services/legacy/readme.md","title":"legacy Documentation","category":"plugin-manifest","format":"markdown","content":"The python SDK had a bug: new fields introduced to the API would throw exceptions when processed. In order to work around this, we convert all responses to a list of structs that were valid when the python SDK still had the bug.\n\nWe copy the proto files to this folder in order to keep track of what was available when this SDK was supported. These should never be modified.\n\nThis entire package can be removed once we no longer have clients that send a user agent of `connect-python`.\n\n","isInternal":false,"tokens":98,"sizeBytes":487},{"name":"README.md","path":"packages/envd/README.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/envd/README.md","title":"envd Documentation","category":"plugin-manifest","format":"markdown","content":"# envd\n\nDaemon that runs inside a sandbox that allows interacting with the sandbox via calls from the SDK.\n\n## Development\n\n### Versioning\n\nThe envd version in `pkg/version.go` is maintained by release-please from the Conventional Commits merged since the last envd release; do not edit it by hand. A change that affects behavior (code, dependencies) ships under a `feat` or `fix` commit type, so that a version the orchestrator can gate on is cut. Pure comment or documentation changes use `docs`, a type the changelog hides, and cut no release; `chore` and `refactor` are visible in it and do cut one.\n\n### Running locally\n\nRun the following command to (re)build the envd daemon and start a Docker container with envd running inside:\n\n```bash\nmake start-docker\n```\n\nYou can use E2B SDKs with env var `E2B_DEBUG=true` or with a debug parameter set to `true` when creating or connecting to a sandbox, to connect to the envd started with this command.\n\n### Generating API server stubs\n\nRun the following command to install the necessary dependencies for generating server stubs:\n\n```bash\nmake init-generate\n```\n\n#### Generating\n\nAfter changing the API specs in `./spec/` run the following command to generate the server stubs:\n\n```bash\nmake generate\n```\n\n### Debugging\n\n- \n- \n\nRun `make run-debug` and then connect to the port `2345` with a debugger or use the VSCode run/debug and run the \"Debug envd\" to build the envd, Docker, and start the debugging.\n","isInternal":false,"tokens":381,"sizeBytes":1589},{"name":"README.md","path":"packages/nomad-nodepool-apm/README.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/nomad-nodepool-apm/README.md","title":"nomad-nodepool-apm Documentation","category":"plugin-manifest","format":"markdown","content":"# Nomad Node Pool Autoscaler Plugins\n\nCustom plugins for the [Nomad Autoscaler](https://github.com/hashicorp/nomad-autoscaler) that keep a service-job allocation count aligned with the number of nodes in a Nomad node pool.\n\n## Purpose\n\nThe package builds two plugins because Nomad Autoscaler supports only one plugin type per external binary:\n\n- `nomad-nodepool-apm` reports the number of ready nodes in a pool.\n- `nomad-deployment-aware-target` fails an active deployment before applying the requested task-group count.\n\nTogether they let service jobs replicate system-job placement while retaining rolling updates. The deployment-aware target is used by the `template-manager` and `orchestrator-ee` service jobs. When the task-group count must change, scaling intentionally abandons a conflicting in-progress rollout (Nomad rejects scaling while a deployment is active); Nomad records that deployment as `failed`, rather than `cancelled`. The rollout spawned by the scale itself is left to run, and no deployment is touched when the count already matches. The target requires `auto_revert = false` on the scaled group (otherwise failing a deployment would restore an older job version and ping-pong); it refuses to scale groups with `auto_revert = true` and returns an error on every attempt.\n\n## Usage\n\n### Building\n\n```bash\nmake build\n```\n\n### Uploading to GCS\n\n```bash\nGCP_PROJECT_ID=your-project-id make build-and-upload\n```\n\n### Configuration\n\nIn your Nomad Autoscaler configuration:\n\n```hcl\napm \"nomad-nodepool-apm\" {\n driver = \"nomad-nodepool-apm\"\n config = {\n nomad_address = \"http://localhost:4646\" # Optional, uses NOMAD_ADDR env var\n nomad_token = \"your-token\" # Optional, uses NOMAD_TOKEN env var\n nomad_region = \"global\" # Optional\n }\n}\n\ntarget \"nomad-deployment-aware-target\" {\n driver = \"nomad-deployment-aware-target\"\n config = {\n nomad_address = \"http://localhost:4646\"\n nomad_token = \"your-token\"\n nomad_region = \"global\"\n }\n}\n```\n\n### In Job Scaling Policy\n\n```hcl\nscaling {\n enabled = true\n min = 1\n max = 50\n\n policy {\n evaluation_interval = \"30s\"\n cooldown = \"2m\"\n\n target \"nomad-deployment-aware-target\" {}\n\n check \"match_node_count\" {\n source = \"nomad-nodepool-apm\"\n query = \"orchestrator\" # Node pool name\n\n strategy \"pass-through\" {} # Use node count directly as desired count\n }\n }\n}\n```\n\n## How It Works\n\n1. The plugin queries the Nomad API to list nodes filtered by the specified node pool\n2. It counts only nodes that are:\n - Status: `ready`\n3. Returns the count as a metric for the autoscaler to use\n4. With the `pass-through` strategy, this count becomes the desired number of allocations\n5. The target serializes scaling per namespaced job and is a no-op when the durable count already matches\n6. When the count must change it fails a conflicting active deployment, rereads the job, and scales with the current job modify index\n7. It verifies the final durable count; the rollout spawned by the scale proceeds normally\n8. Concurrent job or deployment changes are retried from fresh state with a bounded attempt count\n\nDry-run actions do not fail deployments or write task-group counts.\n\n## Query Format\n\nThe `query` parameter should be the name of the node pool you want to count nodes from.\n\nExample: `query = \"orchestrator\"` counts all ready nodes in the `orchestrator` node pool.\n\n## Configuration Options\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `nomad_address` | Nomad server address | `NOMAD_ADDR` env var or `http://127.0.0.1:4646` |\n| `nomad_token` | Nomad ACL token | `NOMAD_TOKEN` env var |\n| `nomad_region` | Nomad region | Default region |\n| `nomad_namespace` | Nomad namespace | Default namespace |\n","isInternal":false,"tokens":918,"sizeBytes":3798},{"name":"readme.md","path":"packages/orchestrator/cmd/simulate-gcs-traffic/readme.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/orchestrator/cmd/simulate-gcs-traffic/readme.md","title":"simulate-gcs-traffic Documentation","category":"plugin-manifest","format":"markdown","content":"To run inside of a host in GCS, trust this fancy one-liner:\n\n host=@ && bucket= && go build -o simulator . && ssh $host rm ./simulator && scp ./simulator $host:~/simulator && ssh $host -- ./simulator -csv-path gcs.csv -test-duration=5s $bucket && ssh $host cat gcs.csv\n","isInternal":false,"tokens":88,"sizeBytes":298},{"name":"README.md","path":"packages/orchestrator/README.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/orchestrator/README.md","title":"orchestrator Documentation","category":"plugin-manifest","format":"markdown","content":"# Orchestrator\n\n## Commands\n\n**Prerequisite:** Enable NBD module first:\n\n```bash\nmodprobe nbd nbds_max=4096\n\ncat </etc/udev/rules.d/97-nbd-device.rules\n# Disable inotify watching of change events for NBD devices\nACTION==\"add|change\", KERNEL==\"nbd*\", OPTIONS:=\"nowatch\"\nEOH\nudevadm control --reload-rules\nudevadm trigger\n```\n\n**Prerequisite:** Allocate enough 2MB HugeTLB pages (hugepages) for testing:\n\n```bash\necho 1024 | sudo tee /proc/sys/vm/nr_hugepages # for 2GB hugepages, adjust as needed\n```\n\n### Create Build\n\nCreate a new build.\n\n```bash\nsudo go run ./cmd/create-build -to-build -storage .local-build\n```\n\nFlags:\n\n- `-to-build ` - Output build ID (UUID, required)\n- `-from-build ` - Base build ID for incremental builds\n- `-template ` - Template ID (default: `local-template`)\n- `-storage ` - Local path or `gs://bucket` (enables local mode with auto-download of kernel/FC)\n- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)\n- `-kernel ` - Kernel version (default: `vmlinux-6.1.158`)\n- `-firecracker ` - Firecracker version (default: `v1.14.1_458ca91`)\n- `-vcpu ` - vCPUs (default: `1`)\n- `-memory ` - Memory in MB (default: `512`)\n- `-disk ` - Disk in MB (default: `1000`)\n- `-hugepages` - Use 2MB huge pages (default: `true`, set `false` for 4KB pages)\n- `-start-cmd ` - Start command\n- `-ready-cmd ` - Ready check command\n\n> You can use the `$(uuidgen)` command to generate a random UUID.\n>\n> If you are using Mise, you can run the command as `sudo $(which go) run ./cmd/create-build -to-build -storage .local-build`.\n\n### Resume Build\n\nResume a sandbox from a build.\n\n```bash\n# Local storage\nsudo go run ./cmd/resume-build -from-build -storage .local-build -iterations 10\n\n# Remote GCS storage (pass credentials to sudo)\nsudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \\\n go run ./cmd/resume-build -from-build -storage gs://bucket -iterations 10\n\n# Pause mode: resume, run command, then snapshot\nsudo go run ./cmd/resume-build -from-build -to-build \\\n -storage .local-build -cmd-pause \"pip install numpy\"\n\n# Pause mode: start command through envd, then wait for SIGUSR1 before snapshot\nsudo go run ./cmd/resume-build -from-build -to-build \\\n -storage .local-build -cmd-signal-pause \"python3 /home/user/workspace/job.py\"\n```\n\nFlags:\n\n- `-from-build ` - Build ID (UUID) to resume from (required)\n- `-to-build ` - Output build ID (UUID) for pause snapshot (auto-generated if not specified)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)\n- `-iterations ` - Number of iterations, 0 = interactive (default: `0`)\n- `-cold` - Clear cache between iterations (cold start each time)\n- `-no-prefetch` - Disable memory prefetching\n- `-v` - Verbose logging\n- `-pause` - Start and immediately pause (create snapshot)\n- `-signal-pause ` - Wait for signal before pause (e.g., `SIGTERM`, `SIGUSR1`)\n- `-cmd-pause ` - Execute command in sandbox, then pause on success\n- `-cmd-signal-pause ` - Execute command in sandbox, then wait for `SIGUSR1` before pause\n- `-envd-version ` - The guest's envd version (default: a placeholder — see the caveat below)\n\n**envd version caveat:**\n\n`resume-build` has no database and the snapshot metadata does not record the build's envd\nversion (only the kernel and Firecracker ones), so unless `-envd-version` is given the tool\nreports a fixed placeholder (`1.0.0`). Every `envd.version` field in its output and logs is\nthen that placeholder, **not** the guest's version — don't reason from it. To read the real\none, mount the build's rootfs (`mount-build-rootfs`) and run its `/usr/bin/envd -version`,\nor read `X-Envd-Version`, which a running envd reports on `/init`.\n\nThe placeholder sits above every envd version gate, so the gated capabilities (`/freeze`,\n`/fsfreeze`, `/collapse`, the KVM clock, the upgrade `from_version`) stay on. Against an\nolder guest that is a path production would not take, so pass `-envd-version` with the\nguest's real version when the behavior under test depends on one of those gates. The tool\nprints which of the two it is using when it loads the build.\n\n**Pause mode example:**\n\n```bash\n# Chain builds with explicit build IDs\nsudo go run ./cmd/resume-build -from-build $BUILD1 -to-build $BUILD2 \\\n -storage .local-build -cmd-pause \"apt install curl\"\n\nsudo go run ./cmd/resume-build -from-build $BUILD2 -to-build $BUILD3 \\\n -storage .local-build -cmd-pause \"pip install requests\"\n\n# Start a long-running command through envd, then trigger pause from the host\nsudo go run ./cmd/resume-build -from-build $BUILD3 -to-build $BUILD4 \\\n -storage .local-build -cmd-signal-pause \"python3 /home/user/workspace/job.py\"\n# In another shell:\nps -ef | grep 'cmd/resume-build' | grep -v grep\nsudo kill -SIGUSR1 \n```\n\n### Copy Build\n\nCopy a build between storage locations (local or GCS).\n\n```bash\n# Local to local\ngo run ./cmd/copy-build -build -from .local-build -to /other/path\n\n# Local to GCS\ngo run ./cmd/copy-build -build -from .local-build -to gs://bucket\n\n# GCS to GCS\ngo run ./cmd/copy-build -build -from gs://bucket1 -to gs://bucket2\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required)\n- `-from ` - Source: local path or `gs://bucket`\n- `-to ` - Destination: local path or `gs://bucket`\n- `-team ` - Team UUID (if set, prints SQL to populate DB on stdout)\n- `-envd-version ` - Envd version (required if team provided) — must match the version present in the template\n- `-vcpu ` - vCPUs (default: `2`)\n- `-memory ` - Memory in MB (default: `1024`)\n- `-disk ` - Disk in MB (default: `1024`)\n- `-tag ` - Build assignment tag (default: `default`)\n\n**Print DB seed SQL:**\n\nWhen `-team` is set, the command reads `metadata.json` from the destination to get `kernel_version` and `firecracker_version`, generates a new env ID, and prints SQL statements (`BEGIN`/`COMMIT` wrapped) to stdout. Pipe directly to `psql` to populate the database:\n\n```bash\ngo run ./cmd/copy-build -build -from .local-build -to gs://bucket \\\n -team -envd-version 0.2.11 -vcpu 2 -memory 1024 -disk 1024 \\\n | psql $POSTGRES_CONNECTION_STRING\n```\n\n### Mount Build Rootfs\n\n```bash\n# Local storage\nsudo go run ./cmd/mount-build-rootfs -build -storage .local-build -mount /mnt/rootfs\n\n# Remote GCS storage (pass credentials to sudo)\nsudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \\\n go run ./cmd/mount-build-rootfs -build -storage gs://bucket -mount /mnt/rootfs\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required unless `-empty`)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-mount ` - Mount path\n- `-verify` - Verify rootfs integrity (requires `-mount`)\n- `-log` - Enable logging\n- `-empty` - Create an empty rootfs instead of loading a build\n- `-size ` - Size of empty rootfs (default: 1GB)\n- `-block-size ` - Block size (default: 4096)\n\n### Inspect Build\n\nInspect build artifacts (headers and data blocks).\n\n```bash\n# Inspect memfile header (default)\ngo run ./cmd/inspect-build -build -storage .local-build\n\n# Inspect rootfs header\ngo run ./cmd/inspect-build -build -storage .local-build -rootfs\n\n# Inspect header + data blocks\ngo run ./cmd/inspect-build -build -storage .local-build -data\n\n# Inspect first 100 data blocks\ngo run ./cmd/inspect-build -build -storage .local-build -data -end 100\n\n# Remote GCS storage\ngo run ./cmd/inspect-build -build -storage gs://bucket -rootfs -data\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-memfile` - Inspect memfile artifact (default)\n- `-rootfs` - Inspect rootfs artifact\n- `-data` - Also inspect data blocks (not just header)\n- `-start ` - Start block index (default: `0`, only with `-data`)\n- `-end ` - End block index (default: all, only with `-data`)\n\n### Show Build Diff\n\nShow the diff between two builds and preview merge result.\n\n```bash\n# Memfile (default)\ngo run ./cmd/show-build-diff -from-build -to-build -storage .local-build\n\n# Rootfs\ngo run ./cmd/show-build-diff -from-build -to-build -storage .local-build -rootfs\n\n# With visualization\ngo run ./cmd/show-build-diff -from-build -to-build -storage gs://bucket -visualize\n```\n\nFlags:\n\n- `-from-build ` - Base build ID (required)\n- `-to-build ` - Diff build ID (required)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-memfile` - Inspect memfile artifact (default)\n- `-rootfs` - Inspect rootfs artifact\n- `-visualize` - Visualize the headers\n\n---\n\n## Architecture (ARM64) Support\n\nThe orchestrator supports both `amd64` (x86_64) and `arm64` (aarch64) architectures. Architecture is detected automatically via `runtime.GOARCH` at compile time.\n\n### Architecture naming convention\n\nThis project uses **Go/Docker/Debian naming** (`amd64`/`arm64`) for architecture directories in binary paths and GCS buckets:\n\n| Convention | x86_64 name | ARM64 name | Used by |\n|------------|-------------|------------|---------|\n| **Go/Docker/Debian** | `amd64` | `arm64` | This repo, Docker, `dpkg --print-architecture` |\n| Linux/GNU | `x86_64` | `aarch64` | `uname -m`, kernel Makefiles |\n\nBinary paths follow the `{version}/{arch}/` layout:\n\n```\n# Firecracker (GCS bucket or FIRECRACKER_VERSIONS_DIR)\nfc-versions/v1.12.1_210cbac/amd64/firecracker\nfc-versions/v1.12.1_210cbac/arm64/firecracker\n\n# Kernels (GCS bucket or HOST_KERNELS_DIR)\nkernels/vmlinux-6.1.158/amd64/vmlinux.bin\nkernels/vmlinux-6.1.158/arm64/vmlinux.bin\n```\n\n> **Note:** The [fc-kernels](https://github.com/e2b-dev/fc-kernels) repo currently uses `x86_64` instead of `amd64` for its directory names. This will be aligned in a follow-up change.\n\n### ARM64-specific behavior\n\n- **SMT** is disabled (ARM processors don't support simultaneous multi-threading)\n- **CPU detection** uses fallback values since `gopsutil` doesn't populate Family/Model on ARM64\n- **OCI platform** is set to the target architecture instead of hardcoded `amd64`\n- **Busybox binaries** are fetched per architecture by `scripts/fetch-busybox.sh` and selected at runtime from `///busybox`\n- **Guest kernel command line** drops the x86-only `i8042.*` and `clocksource=kvm-clock` parameters on arm64; the console stays `ttyS0` (Firecracker exposes a 16550-compatible UART on both architectures)\n\n### Cross-architecture deployment\n\n`TARGET_ARCH` is a **runtime** environment variable that overrides the architecture used for path resolution and OCI image pulls. When unset, defaults to the host architecture (`runtime.GOARCH`).\n\n```bash\n# Run orchestrator targeting amd64 paths from an arm64 host\nTARGET_ARCH=amd64 ./bin/orchestrator\n\n# Or in .env file (read at runtime)\necho \"TARGET_ARCH=amd64\" >> .env.local\n```\n\n`TARGET_ARCH` affects:\n- Firecracker and kernel binary path resolution (`{version}/{arch}/...`)\n- OCI image platform for container pulls\n\nIt does **not** affect:\n- Makefile compilation (see `BUILD_ARCH` below)\n- Hardware-dependent runtime behavior (SMT detection, CPU info) which always uses the actual host architecture\n\n### Building for ARM64\n\n`BUILD_ARCH` defaults to `amd64`. To build ARM64 binaries:\n\n```bash\n# Single command\nBUILD_ARCH=arm64 make build-local\n\n# Or set in .env.local for persistent override\necho \"BUILD_ARCH=arm64\" >> .env.local\n```\n\nThis applies to all services: orchestrator, envd, client-proxy.\n\n---\n\n## Environment Variables\n\nAutomatically set in local mode. Set before running to override:\n\n- `TARGET_ARCH` - Target architecture override (`amd64` or `arm64`; default: host architecture)\n- `HOST_ENVD_PATH` - Envd binary path (default: `../envd/bin/envd`)\n- `HOST_KERNELS_DIR` - Kernel versions dir (local: `{storage}/kernels`, prod: `/fc-kernels`)\n- `FIRECRACKER_VERSIONS_DIR` - Firecracker versions dir (local: `{storage}/fc-versions`, prod: `/fc-versions`)\n- `ORCHESTRATOR_BASE_PATH` - Base orchestrator data (local: `{storage}/orchestrator`, prod: `/orchestrator`)\n- `SANDBOX_DIR` - Sandbox working dir (default: `/fc-vm`)\n\n## Limitations\n\n- Custom template builds support base images from the declared distro families: **Debian/Ubuntu** (apt), the **RHEL family** — Fedora, CentOS Stream, Rocky, Alma — (dnf/microdnf/yum), **Arch** (pacman), and **Alpine** (apk, OpenRC). The distro is resolved from the image's `/etc/os-release` `ID` — never by probing for package managers. Images without an os-release identity (distroless, scratch) or from an undeclared family are rejected fast, with a build-log error naming the reason and the supported families (see `pkg/template/build/phases/base/distro`). Minimal/restricted-repo images may fail provisioning if their repos don't carry the required packages; the failure is surfaced in the build log.\n- Sandboxes always boot the kernel E2B supplies, never one from the base image: `/lib/modules` is empty, no kernel modules can be loaded, and SELinux is disabled on the kernel command line (`selinux=0`), so a base image that ships an SELinux policy boots without it. `kernel-devel` from a distro's repos resolves against a kernel that isn't running. **RHEL** (`rhel`, incl. UBI), **Oracle Linux** (`ol`) and **Amazon Linux** (`amzn`) are therefore not accepted even though they are RPM/dnf images — they are chosen for kABI, signed kernel modules and UEK, which this cannot honour. The community rebuilds above are supported because they are chosen for the userland.\n","isInternal":false,"tokens":3789,"sizeBytes":13887},{"name":"README.md","path":"packages/shared/pkg/apiid/README.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/shared/pkg/apiid/README.md","title":"apiid Documentation","category":"plugin-manifest","format":"markdown","content":"# Resource IDs at HTTP boundaries\n\n`apiid.ProjectID` and `apiid.WorkspaceID` accept a legacy UUID or the matching\npublic ID. Both inputs produce the same struct:\n\n```go\nprojectID.UUID // uuid.UUID\nprojectID.PublicID // canonical prj_ string\n```\n\nThe existing `pkg/id` codecs remain unchanged. These structs are HTTP input\nvalues; database and domain functions can continue taking `uuid.UUID`.\n\n## Adding or changing an API ID\n\nUpdate [`id`](../id/README.md#adding-or-changing-a-resource-id) and this package\nin the same PR whenever adding a resource kind to either package. Follow the\nlinked checklist: `id` owns the codec, and `apiid` provides the matching input\nstruct and binding support. Do not add an API-only encoding or a new codec\nwithout its API type. Keep existing kinds' encoding and validation changes,\ntests, and documentation synchronized across both packages.\n\n## OpenAPI and generated Gin handlers\n\nUse the standard oapi-codegen type override on a string schema:\n\n```yaml\nprojectID:\n name: projectID\n in: path\n required: true\n schema:\n type: string\n description: Project UUID or prj_ public ID.\n x-go-type: apiid.ProjectID\n x-go-type-import:\n name: apiid\n path: github.com/e2b-dev/infra/packages/shared/pkg/apiid\n```\n\nRegenerate the server and dependent clients using the API's generation command.\nNo middleware, registry, or generator fork is needed:\n\n```go\nfunc (s *Server) GetProject(c *gin.Context, projectID api.ProjectID) {\n project, err := s.projects.Get(c.Request.Context(), projectID.UUID)\n // Handle err and build the response as usual.\n}\n```\n\nUse `apiid.WorkspaceID` for workspace parameters. Apply the same schema override\nto query or header parameters and to JSON body properties or array items:\n\n```yaml\nworkspace_id:\n type: string\n description: Workspace UUID or wrk_ public ID.\n x-go-type: apiid.WorkspaceID\n x-go-type-import:\n name: apiid\n path: github.com/e2b-dev/infra/packages/shared/pkg/apiid\n```\n\nAfter `c.ShouldBindJSON(&body)`, use `body.WorkspaceId.UUID` and\n`body.WorkspaceId.PublicID`. Keep response UUID fields unchanged unless a\nresponse-contract change is intended. Do not keep `format: uuid` on inputs\nthat also accept public IDs.\n\n## Handwritten Gin binding\n\n```go\nvar params struct {\n ProjectID apiid.ProjectID `uri:\"projectID\" binding:\"required\"`\n}\nif err := c.ShouldBindUri(¶ms); err != nil {\n c.AbortWithStatus(http.StatusBadRequest)\n return\n}\nprojectUUID := params.ProjectID.UUID\n```\n\n`ShouldBindQuery` and `ShouldBindHeader` work with `form` and `header` tags.\n`UnmarshalText` supports generated path binding; `Bind` supports oapi-codegen's\nexploded query binding; `UnmarshalParam` supports Gin's own binders. Tests cover\neach path because these binders do not all use the same interface.\n\n## Construction and encoding\n\nUse `apiid.NewProjectID(u)` or `apiid.NewWorkspaceID(u)` when constructing an ID\nin Go. Decoders and constructors populate both fields. Malformed input and\nwrong-kind public IDs fail without modifying the receiver. JSON accepts only\nstrings; encoding emits a public ID string, not a JSON object.\n\nThe fields are exported for direct reads. Treat decoded values as immutable;\nGo cannot stop a caller from manually making them inconsistent. Encoding derives\nthe public string from `UUID`, rather than trusting a manually edited `PublicID`.\nA zero-valued struct denotes an unset input; requiredness belongs in the request\nschema or Gin validation. Use a pointer for optional inputs when absence matters.\n\nOther APIs adopt the types by updating their schemas and regeneration, not by\ninstalling middleware. This package does not change their contracts automatically.\n","isInternal":false,"tokens":858,"sizeBytes":3695},{"name":"README.md","path":"packages/shared/pkg/featureflags/README.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/shared/pkg/featureflags/README.md","title":"featureflags Documentation","category":"plugin-manifest","format":"markdown","content":"# LaunchDarkly client authentication\n\nThe client selects authentication in this order:\n\n1. A non-empty `LAUNCH_DARKLY_API_KEY` selects SDK-key authentication and ignores\n the OAuth variables.\n2. Without an SDK key, OAuth variables select OAuth client-credentials authentication.\n Incomplete OAuth configuration fails startup.\n3. Without either, the client uses the offline flag store.\n\nFor direct LaunchDarkly access, set `LAUNCH_DARKLY_API_KEY`. An optional\n`LAUNCH_DARKLY_BASE_URL` redirects streaming, polling, events, and diagnostics\nto a Relay proxy.\n\nFor an OAuth gateway, leave `LAUNCH_DARKLY_API_KEY` unset and configure:\n\n| Variable | Value |\n| --- | --- |\n| `LAUNCH_DARKLY_BASE_URL` | HTTPS gateway URL |\n| `LAUNCH_DARKLY_OAUTH_TOKEN_URL` | HTTPS OAuth token endpoint |\n| `LAUNCH_DARKLY_OAUTH_CLIENT_ID` | Deployment client ID |\n| `LAUNCH_DARKLY_OAUTH_CLIENT_SECRET` | Deployment client secret |\n| `LAUNCH_DARKLY_OAUTH_AUDIENCE` | Optional audience parameter for the token request |\n\nUse the existing deployment OAuth credentials and audience when the gateway\naccepts that identity. The token endpoint must support the `client_credentials`\ngrant with credentials in the request body.\n\nBoth client constructors enable live flag updates without an SDK key in OAuth\nmode.\nOAuth URLs must not contain embedded credentials, query parameters, or fragments.\n\nThe client caches tokens and refreshes them before expiry. Every new HTTP\nrequest uses the current bearer token, including stream reconnections, events,\nand diagnostics. The gateway must bound stream duration because an open stream\ndoes not reauthenticate when its token expires.\n\nToken requests have a five-second timeout. OAuth requests do not follow\nredirects. Authentication failures do not switch between SDK-key and OAuth modes.\n\nThe client mints its first token at startup. If the token endpoint answers with\na non-5xx response, construction fails, including with `WithStartOnInitTimeout`.\nIf the token endpoint is unreachable or returns 5xx, the SDK retries and the\nexisting initialization timeout policy applies.\n\nA gateway `401` or `403` before the gateway has accepted any request stops the\nSDK, as a rejected SDK key does. After the first accepted request, the client\ndrops its cached token on `401` or `403` and the SDK retries with a fresh one.\nThe SDK's own backoff bounds the retry and mint rate.\n","isInternal":false,"tokens":515,"sizeBytes":2379},{"name":"README.md","path":"packages/shared/pkg/id/README.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/shared/pkg/id/README.md","title":"id Documentation","category":"plugin-manifest","format":"markdown","content":"# Typed public IDs\n\nThis package converts resource UUIDs to and from human-readable public IDs:\n\n```text\nprj_7mfb0gpp6c8dsaasre0asc7n3s\nwrk_2n1t201rmv87aae5j4csam8000\nsec_01kyjhkfvse96rg4d2qzd9enft\n```\n\nEvery public ID has a stable resource prefix, an underscore, and a\n26-character UUID suffix. E2B's public representation is always lowercase.\n\n## UUID ownership\n\nResource owners remain responsible for generating and storing UUIDs. UUIDs\nmay, for example, be generated by application code or a database. This package\ndoes not use `typeid.New` or `typeid.MustNew`.\n\n[`github.com/sumup/typeid`](https://github.com/sumup/typeid) is used only to\nconvert an existing UUID to its prefixed public representation and parse that\nrepresentation back into the UUID-backed resource type. PostgreSQL and other\npersistence layers continue to store the underlying UUID, not the prefixed\nstring.\n\n```go\nprojectUUID := uuid.New()\nprojectID := id.ProjectID(projectUUID)\n\npublicID := projectID.String()\nparsedID, err := id.ParseProjectID(publicID)\nif err != nil {\n\treturn err\n}\n\nprojectUUID = uuid.UUID(parsedID)\n```\n\nProject and workspace UUIDs are normally UUIDv4 and use `typeid.Random`.\nSecret UUIDs are UUIDv7 and use `typeid.Sortable`. SumUp renders Random IDs in\nuppercase, so the E2B wrapper normalizes them to the lowercase public contract.\n\n## Team IDs\n\nTeams are projects: a team UUID is also its project UUID.\n`ConvertTeamIDToProjectID` renders a team UUID as its public `prj_` ID, and\n`ParseTeamID` accepts either spelling wherever an API takes a team ID.\n\n## Adding or changing a resource ID\n\nKeep this package and [`apiid`](../apiid/README.md) in sync in the same PR.\nAdding a resource kind here requires its matching API input type in `apiid`;\nadding a type in `apiid` requires a matching codec here. Changes to an existing\nkind's prefix, encoding, or validation must also update the affected API type\nand tests. Encoding rules belong here; `apiid` must reuse them, not duplicate them.\n\nTo add another public resource ID:\n\n1. Choose a short, stable, lowercase prefix that keeps the full public ID at\n the expected fixed width.\n2. Add a private prefix type implementing `Prefix()`.\n3. Select `typeid.Random` for a UUIDv4 resource or `typeid.Sortable` for a\n UUIDv7 resource.\n4. Add a distinct UUID-backed public Go type with `String` and parse methods.\n5. Keep UUID generation in the resource-owning application or database.\n6. Test the exact prefix, lowercase output, fixed length, invalid-prefix\n rejection, and UUID round trip.\n7. Add the matching `apiid` struct with `UUID` and `PublicID`, its constructor,\n and its text, JSON, generated-query, and Gin binding methods.\n8. Test that UUID and public inputs produce identical API structs, wrong kinds\n are rejected, and encoding agrees with this package's codec. Update both\n READMEs and regenerate affected API contracts and clients.\n\nDo not add JSON, text, SQL, or UUID-generation behavior implicitly. Those\ninterfaces should be introduced only when a concrete caller needs them.\n","isInternal":false,"tokens":735,"sizeBytes":3044},{"name":"README.md","path":"packages/shared/pkg/sandboxtypes/README.md","rawUrl":"https://raw.githubusercontent.com/e2b-dev/infra/HEAD/packages/shared/pkg/sandboxtypes/README.md","title":"sandboxtypes Documentation","category":"plugin-manifest","format":"markdown","content":"# Shared sandbox types\n\nThis package owns `RuntimeMetadata`, `SandboxType`, and `EgressClass`, including\nidentity logging and the empty-type fallback to a regular sandbox. It contains\nno worker resources, network initialization, or VM lifecycle management.\n\nCallers import these types directly. `SandboxType.EgressClass()` selects the\ntraffic class; DSCP configuration and socket handling belong to worker networking.\nThe resource-owning `Sandbox` stays in the orchestrator package. Runtime team\nmetadata is attribution, not trusted authorization data.\n","isInternal":false,"tokens":112,"sizeBytes":553}],"systemPromptSnippet":"\n\n# AGENTS.md\n\nInstructions for AI agents working in this repository.\n\nAlso read [CLAUDE.md](CLAUDE.md) for development commands and package\nconventions if it is not already in context. It applies to all agents.\n\n## Understanding the repository\n\nRead [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) before working on or reviewing code here. It describes what each service does, how services interact (sandbox creation, traffic routing, pause/resume, template builds), the data stores, and the deployment topology — it is the fastest way to build correct context about this codebase.\n\nKeep that document updated: if a code change alters anything it describes (service responsibilities, ports, protocols, data stores, flows, deployment topology), update `docs/ARCHITECTURE.md` as part of the same change. When reviewing a PR that changes such details without updating the document, flag it.\n\n## Review guidelines (pull requests)\n\nOutput style rules (strict, override any defaults):\n\n- Only report concrete bugs, regressions, or correctness issues. Do not summarize the PR, the diff, or what changed.\n- One short paragraph per finding. No preamble, no closing remark.\n- No headers, no bullet lists, no tables, no diagrams.\n- No emojis. No severity tags (\"bug\", \"nit\", \"suggestion\", \"enhancement\"). No labels.\n- No branding or footer lines.\n- If there are no real issues to flag, post no review at all.\n\n## Scope\n\n- Skip style/nit comments — `golangci-lint` covers those.\n- Skip test-coverage comments.\n- Focus on: race conditions, nil-deref, error handling, auth/authz, request routing, resource leaks, SQL/migration correctness, and gRPC/proto compatibility.\n\n\n# CLAUDE.md\n\nThis development guide applies to Codex, Claude Code, and OpenCode.\nRepository and review rules are shared through:\n\n@AGENTS.md\n\n## Project Overview\n\nE2B Infrastructure is the backend infrastructure powering E2B (e2b.dev), an open-source cloud platform for AI code interpreting. It provides sandboxed execution environments using Firecracker microVMs.\n\n**Start with [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md)** — it explains what each service does, how services interact (with diagrams of the core flows: sandbox creation, traffic routing, pause/resume, template builds), and the deployment topology. Reading it first is the fastest way to understand this repository.\n\n**Keep docs/ARCHITECTURE.md updated**: if your change alters anything it describes (service responsibilities, ports, protocols, data stores, flows, deployment topology), update the document in the same change.\n\n## Common Development Commands\n\n### Setup & Environment\n```bash\n# Switch between environments (prod, staging, dev)\nmake switch-env ENV=staging\n\n# Setup local development stack (PostgreSQL, Redis, ClickHouse, monitoring)\nmake local-infra\n```\n\n### Building & Testing\n```bash\n# Run all unit tests across packages\nmake test\n\n# Run integration tests\nmake test-integration\n\n# Build specific package\nmake build/api\nmake build/orchestrator\n\n# Generate code (proto, SQL, OpenAPI)\nmake generate\n\n# Format and lint code\nmake fmt\nmake lint\n\n# Regenerate mocks\nmake generate-mocks\n\n# Tidy go dependencies\nmake tidy\n```\n\n### Running Services Locally\n```bash\n# From packages/api/\nmake run-local # Run API server on :3000\nmake dev # Run with air (hot reload)\n\n# From packages/orchestrator/\nmake run-local # Run orchestrator\nmake run-debug # Run with race detector\n```\n\n### Package-Specific Commands\n```bash\n# API: Generate OpenAPI code\ncd packages/api && make generate\n\n# Orchestrator: Generate proto + OpenAPI\ncd packages/orchestrator && make generate\n\n# DB: Run migrations\nmake migrate\n\n# Run single test\ncd packages/ && go test -v -run TestName ./path/to/package\n```\n\n### Publishing Artifacts\n```bash\n# Build and upload all service images to your GCP project\nmake build-and-upload\n\n# Build specific service\nmake build-and-upload/api\nmake build-and-upload/orchestrator\n```\n\n## Architecture Overview\n\n### Service Communication Flow\n```\nClient → Client-Proxy → API (REST) ⟷ PostgreSQL\n ↓ ⟷ Redis\n Orchestrator ⟷ ClickHouse\n ↓ (gRPC)\n Firecracker VMs\n ↓\n Envd (in-VM daemon)\n```\n\n### Core Services\n\n**API (`packages/api/`)** - REST API using Gin framework\n- Entry point: `main.go`\n- Core logic: `internal/handlers/store.go` (APIStore)\n- Authentication: API keys and OIDC auth provider JWTs\n- OpenAPI code generation: `internal/api/*.gen.go`\n- Port: 80\n\n**Dashboard API (`packages/dashboard-api/`)** - REST backend for the web dashboard, not the SDK\n- Entry point: `main.go`\n- Spec: `spec/openapi-dashboard.yml`; template tags, `/v1/management`\n- Talks to Postgres and ClickHouse; never to orchestrators (see `docs/ARCHITECTURE.md`)\n- Port: 3010\n\n**Orchestrator (`packages/orchestrator/`)** - Firecracker microVM orchestration\n- Entry point: `main.go`\n- VM management: `pkg/sandbox/`\n- Firecracker integration: `pkg/sandbox/fc/`\n- Networking: `pkg/sandbox/network/`\n- Storage: `pkg/sandbox/nbd/` (Network Block Device)\n- Template caching: `pkg/sandbox/template/`\n- Template building: `pkg/template/`\n- gRPC server: `pkg/server/`\n- Utilities: `cmd/clean-nfs-cache/`, `cmd/inspect-build/`, `cmd/dummy-orchestrator/`\n\n**Envd (`packages/envd/`)** - In-VM daemon using Connect RPC\n- Runs inside each Firecracker VM\n- Process management API: `packages/envd/spec/process/process.proto`\n- Filesystem API: `packages/envd/spec/filesystem/filesystem.proto`\n- Port: 49983\n- **`pkg/version.go` is release-please's**: it is set from the Conventional Commits merged since the last envd release, so do not bump it by hand. A behavioral change ships under a `feat` or `fix` type so a version exists for the orchestrator to gate on; comment or docs-only changes use `docs`, which the changelog hides, and cut no release (`chore` and `refactor` are visible and cut one)\n\n**Client Proxy (`packages/client-proxy/`)** - Edge routing layer\n- Service discovery via `packages/shared/pkg/servicediscovery`\n- Request routing to orchestrators\n- Redis-backed state management\n\n**Shared (`packages/shared/`)** - Common utilities\n- Proto definitions: `pkg/grpc/orchestrator/`, `pkg/grpc/envd/`\n- Telemetry: `pkg/telemetry/` (OpenTelemetry)\n- Logging: `pkg/logger/` (Zap + OTEL)\n- Database: `pkg/db/` (ent ORM)\n- Models: `pkg/models/`\n- Storage: `pkg/storage/` (GCS/S3 clients)\n- Feature flags: `pkg/featureflags/` (LaunchDarkly)\n\n**Database (`packages/db/`)** - PostgreSQL layer\n- Migrations: `migrations/*.sql` (goose)\n- Queries: `queries/*.sql` (sqlc)\n- Generated code: `queries/` (plus `pkg/auth/queries/`, `pkg/dashboard/queries/`)\n\n### Key Technologies\n\n- **go 1.26.10** with workspaces (`go.work`)\n- **Firecracker** for microVM virtualization\n- **PostgreSQL** for primary data (sqlc for queries)\n- **ClickHouse** for analytics\n- **Redis** for caching and state\n- **OpenTelemetry** for observability (Grafana stack: Loki, Tempo, Mimir)\n- **gRPC/Connect RPC** for service communication\n- **Gin** (API), **chi** (Envd) for HTTP\n\n### Code Generation\n\nThe codebase uses several code generators:\n\n1. **Protocol Buffers** (`packages/orchestrator/generate.Dockerfile`)\n - Generates: `packages/shared/pkg/grpc/*/`\n - Run: `make generate/orchestrator`\n\n2. **OpenAPI** (`oapi-codegen`)\n - Spec: `spec/openapi.yml`\n - Generates: API handlers, types, specs\n - Run: `make generate/api`\n\n3. **SQL** (`sqlc`)\n - Queries: `packages/db/queries/*.sql`\n - Generates: Type-safe DB code\n - Run: `make generate/db`\n\n4. **Mocks** (`mockery`)\n - Config: `.mockery.yaml`\n - Run: `make generate-mocks`\n\n### Testing Patterns\n\n- **Unit tests**: Use `testify/assert` and `testify/require`\n- **Database tests**: `packages/db/pkg/testutils.SetupDatabase` gives each test its own database on the PostgreSQL server `PG_TEST_DATABASE_URL` names (see [DEV-LOCAL.md](DEV-LOCAL.md#unit-tests-that-need-postgresql))\n- **Integration tests**: `tests/integration/` with shared test utilities\n- **Mocking**: Generated mocks in `mocks/` directories\n- **Race detection**: Tests run with `-race` flag\n\nExample test invocation:\n```bash\n# Single package\ngo test -race -v ./internal/handlers\n\n# Specific test\ngo test -race -v -run TestCreateSandbox ./internal/handlers\n```\n\n## Important Development Notes\n\n### Working with Proto/gRPC\n- Proto files: `packages/envd/spec/process/`, `packages/envd/spec/filesystem/`, orchestrator protos at `packages/orchestrator/*.proto`\n- Shared protos: `packages/shared/pkg/grpc/`\n- After editing proto files, run `make generate/orchestrator` and `make generate/shared`\n\n### Database Migrations\n- Migrations: `packages/db/migrations/`\n- Create: `cd packages/db && make create-migration NAME=your-migration-name` — this generates the file with a correct `YYYYMMDDHHMMSS` timestamp. Do NOT hand-create migration files; placeholder timestamps like `120000`/`000000` cause same-day version collisions and are rejected by the out-of-order-migrations CI check.\n- Apply: `make migrate` (requires POSTGRES_CONNECTION_STRING)\n- Code generation: `make generate/db` (regenerates sqlc code)\n\n### Environment Variables\n- Local development defaults: `packages//.env.local`; see [DEV-LOCAL.md](DEV-LOCAL.md).\n- Self-hosting configuration: [embed/compose/.env](embed/compose/.env); see [embed/README.md](embed/README.md).\n- Custom environment configs: `.env.`, selected with `make switch-env ENV=`.\n\n### Firecracker & VM Management\n- Orchestrator requires **sudo** to run (Firecracker needs root)\n- VM networking uses `iptables` and Linux `netlink`\n- Storage uses NBD (Network Block Device)\n- Templates cached in GCS bucket (configurable via TEMPLATE_BUCKET_NAME)\n- Kernel/Firecracker versions: `packages/fc-versions/`\n\n### Observability\n- All services export OpenTelemetry traces/metrics/logs\n- Local stack includes Grafana + Loki + Tempo + Mimir\n- Telemetry setup: `packages/shared/pkg/telemetry/`\n- Logger: `packages/shared/pkg/logger/` (Zap with OTEL)\n- Profiling: API exposes pprof on `/debug/pprof/` (see `packages/api/Makefile` profiler target)\n\n### CI/CD Workflows\n- `.github/workflows/pull-request.yml` - PR orchestrator (lint, OpenAPI, unit, arm64, integration)\n- `.github/workflows/pr-tests.yml` - Unit test shards\n- `.github/workflows/integration_tests.yml` - Integration test suite\n\n## Architecture Patterns\n\n1. **Service Isolation**: Each service runs in containers with defined gRPC/HTTP interfaces\n2. **Shared Libraries**: Cross-cutting concerns (logging, telemetry, DB) in `packages/shared`\n3. **Event-Driven**: ClickHouse + Redis pub/sub for async operations\n4. **Caching Strategy**: Redis for templates, auth tokens, performance optimization\n5. **Feature Flags**: LaunchDarkly for gradual rollouts\n6. **Graceful Shutdown**: Services handle SIGTERM with context cancellation\n7. **Health Checks**: gRPC health protocol + HTTP health endpoints\n\n## Debugging\n\n### Logs\n- Local: Docker logs in `make local-infra`\n- Production: Grafana Loki\n\n\n# E2B Infrastructure Architecture\n\nThis document explains what this repository implements, what each service does, and how the\nservices interact. It is the fastest way to build a mental model of the codebase before diving\ninto the code.\n\n> **Keep this document updated.** If a code change alters anything described here (service\n> responsibilities, ports, protocols, data stores, flows, deployment topology), update this file\n> in the same PR.\n\n## What this repository implements\n\nE2B provides **sandboxes**: isolated Linux VMs that start almost instantly (they resume pre-booted\nsnapshots instead of cold-booting), run arbitrary code (typically\ngenerated by AI agents), and can be paused, snapshotted, and resumed. This repo contains the whole\nbackend: the control-plane REST API, the data-plane VM orchestration built on\n**Firecracker microVMs**, the in-VM agent, the edge routing layer, and template building.\n\nTwo ideas drive the design:\n\n1. **A sandbox is a resumed snapshot.** Templates are pre-booted VM snapshots (memory + disk +\n VM state) stored in object storage. \"Creating\" a sandbox means restoring a snapshot, which is\n why startup is fast. Memory pages are loaded lazily on page-fault (userfaultfd) and the root\n filesystem is a copy-on-write overlay, so only touched data is ever fetched. A template can\n also be filesystem-only (rootfs, no memfile), taken by a filesystem-only pause or checkpoint;\n sandboxes created from it cold-boot from disk instead of resuming.\n2. **Control plane and data plane are separate.** The API decides *where* a sandbox runs and\n tracks *that* it runs (Postgres/Redis); the orchestrator on each node owns *how* it runs\n (Firecracker, networking, storage). Sandbox traffic never passes through the API.\n\n## System overview\n\n```mermaid\nflowchart TB\n subgraph clients[\"Clients\"]\n SDK[\"SDK / CLI\"]\n Browser[\"Browser / HTTP clients\"]\n end\n\n LB[\"Load balancer
api.* | *.domain wildcard\"]\n\n VC[\"volume-content API (belt)
api.<domain>\"]\n\n subgraph controlplane[\"Control plane (API node pool)\"]\n API[\"API
REST :80, gRPC :5009/:5109\"]\n DashAPI[\"dashboard-api :3010\"]\n CP[\"client-proxy
:3002\"]\n end\n\n subgraph datastores[\"State\"]\n PG[(\"PostgreSQL
teams, templates, builds, snapshots\")]\n RD[(\"Redis
running sandboxes, routing catalog, caches\")]\n CH[(\"ClickHouse
metrics, events, optional logs\")]\n OS[(\"Object storage GCS/S3
template + snapshot artifacts\")]\n end\n\n subgraph clientnode[\"Sandbox nodes (one orchestrator per node)\"]\n ORCH[\"orchestrator
gRPC :5008, proxy :5007\"]\n subgraph vm[\"Firecracker microVM (per sandbox)\"]\n ENVD[\"envd :49983\"]\n USERPROC[\"user processes\"]\n end\n end\n\n subgraph buildnode[\"Build nodes\"]\n TM[\"template-manager
(orchestrator binary, gRPC :5008)\"]\n end\n\n SDK -->|REST| LB --> API\n Browser -->|\"port-sandboxid.domain\"| LB --> CP\n API -.->|\"mint content token + domain\"| SDK\n SDK -->|\"volume content (token-authed)
api.<BYOC or default domain>\"| VC\n API -->|\"gRPC Create/Delete/Pause\"| ORCH\n API -->|\"gRPC TemplateCreate\"| TM\n CP -->|\"lookup sandbox → node\"| RD\n CP -->|\"forward :5007\"| ORCH\n CP -.->|\"gRPC auto-resume\"| API\n ORCH --> ENVD\n ENVD --> USERPROC\n API --> PG & RD & CH\n DashAPI --> PG & CH\n ORCH --> OS & CH\n TM --> OS\n```\n\n## Services\n\n| Service | Package | Runs on | Purpose |\n|---|---|---|---|\n| API | `packages/api` | API nodes | Public REST API; sandbox lifecycle, placement, auth, quotas |\n| Orchestrator | `packages/orchestrator` | every sandbox node | Runs Firecracker VMs; sandbox create/pause/resume/kill |\n| Template manager | `packages/orchestrator` (role) | build nodes | Builds templates from Docker images |\n| Client proxy | `packages/client-proxy` | API nodes | Edge router: sandbox URL → correct node |\n| Envd | `packages/envd` | inside every VM | In-VM agent: process/filesystem API for SDKs |\n| Dashboard API | `packages/dashboard-api` | API nodes | Backend for the web dashboard and workspace control plane (sandbox records, admin, `/v1/management`) |\n\nSupporting packages: `packages/shared` (protos, telemetry, storage clients, feature flags),\n`packages/auth` (authentication library), `packages/db` (Postgres migrations + sqlc queries),\n`packages/clickhouse` (ClickHouse schema + clients), `packages/otel-collector` (collector config),\n`packages/nomad-nodepool-apm` (autoscaler plugin), `packages/local-dev` (local stack).\n\n### API (`packages/api`)\n\nThe control-plane entry point (Gin, OpenAPI-generated from `spec/openapi.yml`, port 80).\n\n- **Resources**: sandboxes (create/list/kill/pause/resume/connect/timeout/metrics/logs),\n templates and builds (including the web dashboard's build listings under `/unstable/builds`),\n teams, volumes, API keys, secrets, sandbox events and webhooks, admin operations.\n- **Default templates**: `env_defaults` names the platform's default templates. Admin routes\n under `/unstable/admin/templates` add, describe and remove default templates and aliases\n without a namespace, and clear the alias cache after each alias change so name resolution\n sees it at once.\n- **Spec document**: `GET /openapi.json` serves, without authentication, the spec the request\n validator runs. An `ETag` over the document answers a matching `If-None-Match` with 304. It is\n registered before the validator, which rejects undeclared paths.\n- **Auth** (via `packages/auth`): team API keys (`X-API-Key`, `e2b_` prefix), auth-provider JWTs\n (OIDC), and either an admin token or a service JWT verified from the configured admin JWKS.\n Backed by an auth DB (Postgres) with a Redis team cache.\n- **API-group rate limits**: operations declare a group through the `x-api-group` OpenAPI\n extension. After authentication succeeds, its callback wrapper reads the validator's\n already-matched operation and stores the API group directly in Gin's context. The rate limiter\n reads that group and maps it to a field in the authenticated team's limits. Grouped operations\n require team-identifying authentication; public operations do not populate an API group.\n The `list` group uses `api_team_rps_list`; the `delete` group uses `api_team_rps_delete` only\n for sandbox and template deletion. Other DELETE operations are ungrouped.\n Delete rates default to zero until configured.\n A group's rate is one of the team's limits and resolves with the rest of them in the statement\n that loads the team, which authentication caches; it has no source of its own.\n Propagated project limits carry both rates, and each accepted revision replaces\n them. A team with no `project_limits` row uses the deployment's `default_limits`; zero\n disables group limiting. The migration that added `default_limits` gave every existing team a\n row holding its limits at the time, so a change to the defaults reaches only teams created since.\n A positive rate sets both requests per second and burst capacity. Redis enforces a separate\n budget for each team, HTTP method, and route template across API replicas: groups select\n rates, but routes do not share a budget. Exhausted budgets return HTTP 429 with rate-limit\n and `Retry-After` headers. Ungrouped operations and unconfigured rates bypass group limiting;\n Redis failures are logged and counted while requests proceed.\n V1 route limits from `rate-limit-config` are always enforced first. For requests that pass V1,\n LaunchDarkly's `rate-limit-v2-mode` is the master switch for all V2 checks, and\n `rate-limit-delete-mode` can further restrict delete checks. Both default to `disabled`;\n missing, invalid, or unknown modes are treated as disabled. Deletes skip V2 if either flag is\n disabled and enforce only when both are enabled. If both are active and either is `shadow`,\n deletes run in shadow mode. Lists follow the global mode.\n The groups use the same limiter logic: `shadow` observes V2 decisions while\n preserving V1 headers, and `enabled` also enforces V2. Ungrouped operations, unconfigured V2\n rates, and V2 Redis errors retain V1 enforcement. A rejection reports the rejecting limiter's\n headers; when both checks allow a request in enabled mode, V2 supplies the rate-limit headers.\n- **Sandbox list snapshots**: a list request reads the team's whole sandbox index from Redis\n (its running and pausing records) and pages in memory, so with the LaunchDarkly\n `sandbox-list-max-lag-ms` flag set for a team, each API replica shares one read between\n that team's concurrent list requests and keeps it for the flag's value, counted from the\n start of the read; a request that joins a read already in flight gets that read's result\n however long it takes. A lowered flag applies to a live snapshot; a raised one from the\n next read. Within that lag a sandbox created, killed, paused or resumed can be missing from\n the list or shown in its previous state on the replica holding the snapshot, and one\n resumed inside it can appear on two pages of a walk served by different replicas. The\n default, zero, gives every request its own read with the same store commands as before.\n Only the two list endpoints use the snapshot; the evictor, reconcile, template delete and\n team teardown keep reading the store directly.\n- **Workload identity**: sandbox create accepts an optional `iam.tokens` map of caller-named\n workload-token definitions (each an exact `audience` and `tokenType`). A non-empty, validated\n map enables workload identity, whose identity the orchestrator derives from the sandbox's\n already-authoritative team/sandbox/execution/template IDs; the definitions are passed to the\n orchestrator in `SandboxConfig.iam`. The API mints no credential and delivers nothing into the\n sandbox; file-based delivery is rejected at admission. Definitions are persisted in the\n running-sandbox (Redis) and paused-snapshot (Postgres) state so they survive pause/resume and\n orchestrator re-sync; a fork starts a new workload and does not inherit them.\n- **Placement**: keeps a live map of orchestrator nodes (discovered via Nomad, Kubernetes, or a\n static list). Chooses a node per sandbox with a **best-of-K** algorithm\n (`internal/orchestrator/placement/`): sample K ready nodes, score by the\n higher of CPU load and hugepage-pool load, pick the lowest; retry on\n exhausted nodes. A node that refuses a create (`ResourceExhausted`) stays\n eligible, and each refusal waits a full-jitter backoff that doubles up to a\n cap, so a placement that finds every node busy does not re-send creates as\n fast as they are refused. The hugepage-pool load is measured against the node's\n base pool (`nr_hugepages`, allocated at boot), so a node that has borrowed surplus\n pages scores above 1 instead of reading as just full; a pool that is all overcommit\n is measured against its ceiling, and an orchestrator that does not report the base\n pool against `HugePages_Total` as before. A node that reports no pool scores 0.5. Hugepage scoring\n is on by default (`BEST_OF_K_HUGEPAGE_MEMORY`); set it false to rank on\n CPU alone. K, overcommit ratio, and alpha are tunable live via feature\n flags. With `placement-hugepage-limit-percent` above 0, a hugepage sandbox\n is also refused by any node whose committed hugepages would pass that share\n of the node's pool ceiling (base pool plus overcommit allowance, reported by\n the orchestrator); within an API replica the check and the in-flight record\n are one step per node, and when the compatible nodes are all full placement\n answers with a capacity error, not \"no compatible node\". With the\n orchestrator's `create-memory-refusal` flag on (`CREATE_MEMORY_REFUSAL` for a\n deployment without LaunchDarkly), a create whose guest memory the pool\n cannot back while a sandbox on that node is still tearing down is refused\n with `ResourceExhausted`, so placement retries the node once the pages are\n back; with nothing tearing down it fails at once and says why. Off, it fails\n as an internal error and the node is excluded.\n- **State**: writes sandbox records to Redis (source of truth for *running* sandboxes). It does\n not write routing: client-proxy reads the orchestrator-written `sandbox:routing:{id}` (see\n \"Sandbox routing records\"). Persistent entities (templates, builds, snapshots, teams) live in\n Postgres.\n- **Secrets**: `/secrets` is the only public surface for secret management (create, list, get,\n update, delete). The API authenticates the caller with the customer alternatives above, converts\n the authenticated team UUID to the project UUID the backend knows, checks the `customer-secrets`\n feature flag, and forwards a metadata-only request over a unary gRPC contract\n (`e2b.secretsstore.management.v1`) to the secrets store backend named by\n `SECRETS_STORE_BACKEND_GRPC_ADDRESS`. No caller credential, header or client-supplied tenant\n crosses that hop, and no response - here or in a log, a span or an error - carries a secret\n value. What a caller stores is a runtime marker, not a resolved secret; orchestrator-ee resolves\n a marker to a value at sandbox egress, never here. Without a configured address, or with the flag\n off, the routes stay registered and answer 403. Responses are `Cache-Control: no-store`, request bodies are capped at 512 KiB, and values\n at 64 KiB. A create carries the cap on live secrets the backend enforces: `max_secrets` is one\n of the team's limits and resolves with the rest of them in the statement that loads the team (its\n `project_limits` value, else the deployment's `default_limits`).\n- **Sandbox events and webhooks**: `/events/sandboxes` reads a team's sandbox lifecycle events,\n and `/events/webhooks` manages the webhook subscriptions those events are delivered to, along\n with each subscription's delivery attempts and delivery statistics. The API stores none of\n this: it authenticates the caller, resolves the team's cluster, and forwards the request over\n two unary gRPC contracts (`e2b.webhooks.events.v1` and `e2b.webhooks.management.v1`) to the\n backend named by `WEBHOOKS_BACKEND_GRPC_ADDRESS`. The team and cluster come from the\n authenticated context, never from the request, and a team with no cluster selects the local\n one. Without a configured address the routes stay registered and answer 503, so the surface\n does not depend on how a deployment is wired. A backend failure the caller cannot act on is\n answered with the endpoint's own message; the outcomes that keep their identity are\n not-found, a rejected request, the webhook-count limit, and a timeout. A create or update\n body carries the webhook's signing secret, so a request rejected before the handler has its\n body kept out of the response, the access log and the span. Reads are capped at 100\n deliveries per page.\n- **Rig passthrough**: admin endpoints under `/clusters/{clusterID}/rigs` manage a cluster's\n orchestrator node pools (\"rigs\"). They list rigs, change rig capacity, list and terminate\n instances, and read recent scaling errors. Every handler delegates to the cluster's edge service\n (`/v1/rigs/...`) through the per-cluster HTTP client that the cluster sync keeps fresh. The API\n holds no cloud credentials and makes no scaling decisions. The caller authenticates with an\n admin credential and never holds the cluster's edge secret. Edge status codes pass through unchanged\n (400, 404, 409, 501). An edge 401 becomes a 500: it means the API is misconfigured against the\n cluster, not that the caller's token is invalid. A cluster with no rig management configured\n returns an empty list. The local cluster has no edge deployment and answers 501 for every rig\n operation.\n- **Extra listeners**: internal gRPC :5009 and edge gRPC :5109 expose `ResumeSandbox` so\n client-proxy can wake paused sandboxes on incoming traffic.\n- **Outbox** (`internal/outbox`): a River client on the shared database's `river` schema\n works jobs that need the orchestrator connections. Its job arguments live in\n `packages/db/pkg/outbox`. The one kind today, `teardown_team_resources`, stops a deleted\n team's workloads; the API's own project deletion enqueues it. A\n guard runs first on every attempt: a team that exists and is not blocked fails the attempt,\n and the job retries rather than kill a live team's sandboxes; a missing team row does not\n stop it. Then\n `kill_sandboxes` kills the team's running sandboxes and succeeds only once the sandbox store\n holds none of them in any state, so a sandbox still pausing or being killed makes the job\n retry. A sandbox that is not running cannot be paused, so no new snapshot of the team\n appears afterwards. Killing through the orchestrator is what removes the sandboxes' store\n records, so the job deletes no Redis state itself, and every attempt is safe to run again.\n Builds are not cancelled: they end on their own build timeout. An attempt is bounded at\n 30 minutes; retries back off from 30 seconds to a 15-minute cap, so the 100 attempts last\n about a day before River discards the job and keeps it. Volumes, templates, snapshots and\n the team row are left alone. Steps report\n `api.outbox.steps.finished` and `api.outbox.step.duration`; finished jobs report\n `outbox.jobs.finished`, and the backlog reports `outbox.jobs` and `outbox.oldest_*_age`.\n `OUTBOX_MAX_WORKERS` (default 10) and `OUTBOX_BACKLOG_INTERVAL` (default 30s) configure it;\n shutdown stops it after the sandbox-work drain, before the database and Redis clients close.\n- Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to\n Loki, with a LaunchDarkly-gated ClickHouse read path (`logs-read-config`) for local-cluster logs\n during the log storage migration. `LOKI_URL` is optional: without it the api has no Loki client\n and those reads fail until the flag routes them to ClickHouse. `LOGS_READ_CONFIG=true|false` sets\n the value the flag falls back to when LaunchDarkly has none (no LaunchDarkly, or the flag not\n defined there); a LaunchDarkly value wins. LaunchDarkly feature flags also gate placement\n parameters, rate limits, and rollouts.\n\n#### Project management writes (`/unstable/management`)\n\n`PUT /unstable/management/projects/{projectID}/settings` applies a revisioned settings\nprojection from the owning management service. This endpoint belongs to the API\nand accepts only service JWTs through `AdminJWTAuth`. It locks the team and checks\nthat it has not been deleted before applying the projection. The shared `ProjectSettings` type\ncontains `SandboxSettings`, with the optional ingress default under\n`sandboxSettings.defaultAllowPublicTraffic`. Storage uses `projection.project_settings`,\nwith `revision` and nullable `sandbox_default_allow_public_traffic` in the same row.\nThe receiver applies both in one guarded upsert. Equal or older revisions cannot\noverwrite it. Clearing stores NULL at a newer revision, retaining the ordering fence.\nCache invalidation follows commit and is retried on duplicate\ndelivery after a cache failure. There is no rollout flag check on accepted delivery.\nSandbox creation resolves an absent default to true and honors a saved default\nindependently of rollout eligibility. New sandboxes can use an older value while\ndelivery or cache refresh is pending.\n\nThe owning management service also drives the project writes under\n`/unstable/management/projects/{projectID}`. Like the settings route, each accepts only service\nJWTs through `AdminJWTAuth`, and each is idempotent, because the caller is level-triggered and\nretries.\n\n- `PUT /unstable/management/projects/{projectID}` creates or reconciles a project and answers 201\n or 200 to say which. A project is a `public.teams` row created from a caller-supplied UUID, with\n no tier. An optional cluster ID is stored on the project; omitting it preserves the current\n assignment. A changed slug renames the project and, in the same transaction, gives each of its\n namespaced template aliases under the new slug as well, keeping the old ones and leaving an\n alias the new slug already holds as it is. It answers 409 when another project already holds the slug, when the\n request assigns a cluster whose deletion is recorded, or when a concurrent request created the\n same project first.\n- `PUT .../members/{userID}` applies the desired presence for one user, gated by a monotonic\n per-project/user revision in `projection.project_members`; a duplicate or older revision\n succeeds without changing target state. A present projection carries the user's OIDC\n issuer/subject identities. Every projected user has at least one, and an identity already owned\n by a different user returns 409. A revocation removes only that user's `users_teams` row;\n projected users and identities are retained. Auth caches member authorization, so each accepted\n command evicts that user's authorization for the project after commit.\n- `PUT .../limits` syncs the project's effective limits into `project_limits`, which the team\n lookups read in preference to `default_limits`. It is gated by a monotonic per-project revision\n in `projection.project_limits`: the caller raises it whenever the limits it resolved change, and\n a delivery at or below the recorded revision is dropped and still answers 204. The ledger and the\n values advance in one transaction, so a revision is never recorded without the values it\n admitted.\n- `PUT .../block` applies a revision-fenced block state: the `projection.project_blocks` revision\n and `teams.is_blocked` / `blocked_reason` commit together, and an older or duplicate revision\n leaves stored state unchanged. Every accepted delivery, a dropped one included, evicts the team's\n auth cache; a failed eviction answers an error so the retry finishes it after commit. The API\n reads this state when admitting billable work. Apart from project deletion, which blocks the\n team for good, this is the only route that writes it.\n- `DELETE /unstable/management/projects/{projectID}` soft-deletes the project, described below.\n\nThe revision fences are the target's for the same reason: the caller can only fence what it\nsends, so two deliveries in flight arrive in whichever order the network gives them, and the older\none has to be refused where it lands. The project, membership, limits and block writes live in\n`internal/management` with their post-commit cache evictions rather than in the handlers.\n\nA limits or block delivery may carry `decided_at`, when the caller decided that revision; the\nledger row stores it beside the revision. Each delivery the ledger accepts records the seconds\nsince then in `dashboard-api.management.apply_lag` with `projection` `project_limits` or\n`project_blocks`; one without `decided_at`\ncounts in `dashboard-api.management.origin_unknown` instead, never as zero lag. Dropped, refused\nand failed deliveries record nothing. The names keep the dashboard-api prefix from when\ndashboard-api served these writes, so the alert and dashboard reading them carry on unchanged.\n\nProject deletion soft-deletes any project, including a member's default project; whether a project\nmay be deleted is the caller's decision. `envs`, `snapshots` and `volumes` reference `teams` with\n`ON DELETE NO ACTION`, so the team row stays. One transaction:\n\n- locks the row (404 if it is missing) and sets `teams.deleted_at`;\n- blocks the team (a CHECK keeps every deleted team blocked) and clears its cluster assignment;\n- scrubs its name and email to `DELETED` and its slug to `deleted-`, releasing the\n old slug for reuse;\n- deletes its memberships and API keys;\n- soft-deletes its templates, snapshot templates and paused-sandbox snapshots the way a template\n deletion does, releasing their aliases and build tracking;\n- enqueues `teardown_team_resources` with `InsertTx` on the outbox's own River client, so the\n teardown runs exactly when the deletion commits.\n\nAfter commit it evicts the deleted keys' auth cache entries best-effort; the team's other cached\nentries refresh or expire on their own. Repeating the call answers 204 and runs every step again,\nkeeping only the first deletion's time. That undoes a block reason, rename or cluster assignment\nwritten by a delivery that passed its deleted-project check just before the first call committed;\ndeletes memberships, keys and envs created since (a sandbox that paused before the teardown killed\nit leaves a snapshot); and enqueues the teardown again, which is a no-op while one is still open.\n\nAfterwards the API's deleted-project middleware, running after authentication, answers 404 to every\nnon-GET route under `/unstable/management/` that names `{projectID}`: the upsert, member, limits,\nblock and settings writes. The DELETE is exempt so a retry still reaches it. dashboard-api guards its own\nmanagement and admin writes the same way. Shared auth refuses a membership or API key still left\nfor the project with 403, and the API leaves the project out of a user's team list. Volumes, builds\nand stored artifacts are not reclaimed.\n\n#### Team bans (`/unstable/admin/teams`)\n\n`PUT /unstable/admin/teams/{teamID}/ban` sets `teams.is_banned`, evicts the team's auth cache so\nits API keys stop authenticating, then kills its running sandboxes and cancels its builds through\nthe same orchestrator and template-manager calls as `POST /admin/teams/{teamID}/sandboxes/kill`\nand `.../builds/cancel`. The ban commits first: a kill or cancel step that fails is reported in\nthe 200 response and does not undo it. The handler detaches the request from the caller and gives\nit one 25-second budget at the same moment, so a caller that disconnects does not stop it and\nnothing it runs (deletion checks, ban writes, cache evictions, kills, cancellations) blocks past\nthe budget. A ban write or eviction that runs out of it fails like any other (500, or `failed` in\nthe bulk route). Each team's kill and cancel run side by side; a step the budget cuts short is\nreported as failed with what it counted so far. A build delete already under way keeps its own 30\nseconds, so a request ends within about 55 seconds. A failed cache eviction answers 500, so the\ncaller's retry repeats the ban and its containment; banning a banned team contains it again. `DELETE .../ban` clears the flag and evicts the cache.\n`POST /unstable/admin/teams/bans` takes up to 100 teams. It stores every ban first, five teams at\na time, then contains the banned teams, five at a time, all under the one budget, and sorts each team\ninto banned, not found or failed; one team's failure does not stop the others. A deleted project\nanswers 404 to the single-team routes and counts as not found in the bulk route. The logic lives\nin `internal/team`.\n\n### Orchestrator (`packages/orchestrator`)\n\nA single Go binary running on every sandbox node (as root). `ORCHESTRATOR_SERVICES` selects its\nroles: `orchestrator` (run sandboxes) and/or `template-manager` (build templates). Code lives\nunder `pkg/`, almost all Linux-only.\n\ngRPC services on :5008 (`pkg/server/`, `pkg/service/`, `pkg/template/server/`, `pkg/volumes/`):\n\n- **SandboxService** — `Create`, `Update`, `List`, `Delete`, `Pause`, `Checkpoint`.\n `Update` callbacks are serialized per sandbox. A successful transition out of the live\n map closes update admission and joins any admitted callback before stop/checkpoint\n work or stopping notifications begin; `Close` repeats that check for crash cleanup.\n An update using an inactive sandbox returns gRPC `NotFound`, which the API maps to\n HTTP 404 for timeout and network updates. Teardown waits for admitted callbacks;\n it cannot preempt one that does not return.\n- **TemplateService** — `TemplateCreate`, `TemplateBuildStatus`, `TemplateBuildDelete` (template-manager role only).\n- **InfoService** — node identity, roles, capacity, health status (used by API node discovery).\n- **ChunkService / VolumeService** — peer-to-peer template chunk serving; persistent volumes.\n\nProcess shutdown, triggered by `SIGINT`, `SIGTERM`, `SIGUSR1`, or a service failure, moves\nthe node to `ShuttingDown` before draining builds and sandboxes. Like `Draining`, this excludes new\nplacement while existing work stays reachable, and `/health` returns HTTP 200 with status\n`draining`. Unlike reversible `Draining`, `ShuttingDown` is terminal for the process;\nstatus overrides cannot enter or leave it. `FORCE_STOP` skips waiting for sandboxes to exit.\nThe sandbox drain ends only when no sandbox is live and the node's outstanding work (below) is\nzero, so a create or a resume-fresh checkpoint in flight, whose sandbox is briefly not live, holds\nit until its sandbox goes live and then ends.\n\nAfter sandbox drain and snapshot uploads complete, sandbox proxy cleanup attempts graceful HTTP\nserver shutdown with a separate deadline, then forces server closure if needed. Successful forced\nclosure recovers graceful deadline expiry; other server errors still propagate. Pprof stays\navailable through service teardown; its final shutdown also has a deadline and forced-close fallback.\n\n`InfoService.ServiceInfo` reports `outstanding_work` for sandbox, template-builder, and\nmixed-role nodes. It counts overlapping work holds, not distinct sandboxes or builds, and includes\ntracked background persistence and cleanup. The producer reports zero when idle. The API caches\nthis report and exposes it as required top-level `outstandingWork` in admin node list and detail\nresponses, where zero also covers a node that has not reported. This observational count does not\nby itself authorize node deletion.\n\n`InfoService.ServiceInfo.max_sandboxes` reports the cached node limit for every service role.\nAdmin `/nodes` and `/nodes/{nodeID}` return required `maxSandboxes`, including zero. Admission,\nmetrics, and node info share the value refreshed from node-scoped flags every 30 seconds.\n\nKey mechanisms (all under `pkg/sandbox/`):\n\n- **Firecracker** (`fc/`): each sandbox is one Firecracker process in its own cgroup and network\n namespace. The FC HTTP API (unix socket) configures machine, drives, network, and snapshots.\n Guest metadata (sandbox ID, envd access token hash) is passed via MMDS.\n A sandbox may use fewer CPUs than its VM has (`SandboxConfig.max_vcpus` above `vcpu`): the VM\n is created with `max_vcpus` vCPUs, a boot puts `maxcpus=` on the kernel command line, and\n the `fc_vcpu` threads are held to `vcpu` CPUs by a `cpu.max` quota in a threaded `vcpu` child of\n the sandbox cgroup, so the VMM's own threads stay outside it. The quota is in place before the\n guest runs an instruction: a boot limits the whole cgroup until the vCPU threads exist and then\n moves the limit onto them; a resume confines the paused threads before the VM resumes, falls\n back to the whole cgroup, and fails if neither holds. The snapshot's metadata records the VM's\n vCPU count (`vcpu_count`, stamped at build and on every pause), so a resume limits below it\n without trusting the request or discovering capacity from cgroup threads. A legacy snapshot\n carries its stored original vCPU count as `max_vcpus`; if unavailable, it may only resume at its\n unchanged `vcpu`.\n `/init` carries the count as `cpuCount`\n so envd onlines or offlines guest CPUs to match and reports the outcome on `X-Envd-Cpus` and\n `/metrics`.\n- **Lazy memory / UFFD** (`uffd/`): on resume, Firecracker restores the VM without loading\n memory; a userfaultfd handler serves page faults directly from the template's memfile, so only\n touched pages are read. An optional prefetcher warms known-hot pages.\n- **Copy-on-write rootfs** (`rootfs/`, `nbd/`, `block/`): the template rootfs stays read-only;\n writes go to a per-sandbox COW cache exposed to Firecracker as an NBD block device served by\n an in-process userspace NBD server. On pause, the dirty blocks are exported as a diff.\n- **Template cache** (`template/`): templates are fetched lazily from object storage and cached\n on local disk (and optionally on a shared NFS chunk cache, or fetched peer-to-peer from other\n nodes before upload completes). Every reader of a cached template holds a pin on it — a\n running sandbox, a build, a peer stream, an ancestor lookup — and eviction never closes a\n pinned template. A snapshot's own upload holds a pin while it runs and records on the entry\n whether the snapshot landed in storage; a layer that did not land is never released, so a\n failed upload's local copy stays until its TTL expires. When a pause, or a\n checkpoint that resumes a fresh sandbox, publishes a new layer, the layer its sandbox ran\n from is marked superseded if this node inserted it for an earlier pause, because no resume\n can ask for it again. With `snapshot-cache-release-superseded` and\n `snapshot-cache-ancestor-storage-fallback` both on, a superseded layer is released once no\n pin holds it, its devices have resolved and its upload has landed, instead of staying until\n its TTL expires. A descendant's upload then heals that ancestor's build data from storage,\n and writes the same header bytes it would have written with the ancestor still cached.\n The sandbox and template caches start empty after a restart: the template cache empties the\n build-data directory, and startup reclaim empties the sandbox, template and template-build\n scratch directories.\n- **Networking** (`network/`): each sandbox gets a slot — a network namespace with a veth pair\n and a tap device, unique host-side IP (from a /16), NAT, and per-slot nftables egress firewall\n (with SNI/Host-inspecting TCP firewall for domain allow/deny lists). Slots are pooled and\n reused; slot indexes are allocated locally against the node's netns state (leftover\n namespaces from a previous run are torn down by startup reclaim).\n- **Sandbox proxy** (:5007, `pkg/proxy/`): reverse-proxies incoming traffic from client-proxy to\n the sandbox's slot IP and requested port over HTTP or configured HTTPS, enforcing per-sandbox\n traffic access tokens through the shared ingress policy. HTTPS backends may use self-signed certificates.\n It refuses a full-router sandbox with 503 `sandbox_feature_unsupported` before dialing the\n slot.\n Each admitted lifecycle owns its proxy handlers and backend sockets, including pending dials\n and upgrades. Stopping seals admission and interrupts connections; network release waits for\n that work to finish. Restarting envd resets the owner while allowing the live lifecycle to\n admit a replacement. Retired streams have one second to write their terminal response before\n a stalled frontend is interrupted.\n Shared proxy shutdown waits for H2C and upgraded connections to close and includes them in\n forced connection cleanup. Failed H2C handshakes release their hijacked connections when the\n handshake handler returns.\n- Writes sandbox lifecycle **events** and cgroup **host stats** to ClickHouse; exports metrics via\n OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy\n collector remains the fallback primary destination, and configured shadow destinations can mirror\n writes during collector/storage migrations without changing sandbox behavior.\n\nDetached sandbox-event publication holds node work until the publisher returns. This covers\ndelivery attempts, not end-to-end delivery: the ClickHouse target enqueues into an in-memory\nbatcher, whose flushing remains part of shutdown.\n\n### Envd (`packages/envd`)\n\nThe agent inside every VM (started by systemd very early in boot), port 49983, chi + Connect RPC.\n\n- **Process service** (`spec/process/process.proto`): start/list/connect to processes, stream\n stdout/stderr, stdin, signals, PTYs — this is what SDKs use to \"run code\". A start is refused\n with `unavailable` while the workload cgroups are frozen for a pause, and the freeze waits for\n in-flight starts, so a pause does not capture a child stopped before exec in a frozen cgroup. A\n start into a cgroup the guest froze itself is refused the same way. Starts carrying the system\n tag are unaffected: they stay in envd's own cgroup, which no freeze covers.\n- **Filesystem service** (`spec/filesystem/filesystem.proto`): stat/list/make/move/remove/watch.\n- **REST**: `/health`, `/metrics`, `/envs`, `/files` upload/download, `/init` (orchestrator pushes\n env vars, access token, metadata and the online CPU count after boot/resume), `/upgrade` (live self-upgrade, below),\n freeze/thaw hooks used during pause.\n- **Public vs. control-plane routes**: the control routes (`/init`, `/collapse`, `/upgrade`, and the\n freeze/thaw hooks) are marked `x-internal: true` in `spec/envd.yaml`, and `/upgrade` — which the spec does not\n describe — is listed alongside them in the orchestrator's `pkg/sandbox/envd`. The orchestrator\n reaches them over the host network at the sandbox slot IP; the sandbox proxy refuses them with a\n 404, for every method, so they are not reachable through a sandbox URL. Adding a control route\n therefore means marking it in the spec (`go generate` carries the marker into the proxy's\n rejection list) — otherwise it ships reachable from the internet.\n- **Guest trust** (`internal/host/cacerts.go`): `/init` supplies the desired CA bundle.\n envd atomically records its PEM and owned DER identities in `/run/e2b/ca/state.json`,\n then publishes a complete replacement of the active bundle with the requested roots\n before returning. Ordinary installs copy the existing bundle without decoding it;\n recovery first repairs interrupted writes. All supplied identities are owned,\n including identical preexisting copies. A background loop resumes saved intent at startup\n and retries cleanup every second. It atomically removes retired roots from the active\n bundle before rootfs work, so source-file failures cannot delay that cleanup.\n It persists each requested certificate as\n `e2b-.crt` in `/usr/local/share/ca-certificates`, removes retired managed files,\n and retains retirement identities until both bundle and source cleanup succeed.\n Superseded cleanup cannot publish an older bundle. Legacy-file removal checks the latest\n desired roots under the init lock, preserving hash links to a still-requested certificate.\n State survives envd process restart,\n not guest reboot. Retired roots may be trusted until cleanup succeeds, and an independent\n trust-store rebuild can restore a copy from a customer source file. Applications that\n cached their trust store must reload it.\n- **Auth**: `X-Access-Token` header checked against a token delivered via Firecracker MMDS;\n signed URLs for file endpoints. `/init` is exempt (it is what *delivers* the token), which is the\n main reason the proxy refuses it outright.\n- **Live upgrade** (`internal/services/process/upgrade.go`): an authenticated `POST /upgrade` lets\n the orchestrator swap envd inside a *running* sandbox at resume. It streams the new binary in the\n request body and envd `syscall.Exec`s into it **with the same PID**, carrying the workload's\n stdio/PTY fds, process table, recently-retained exit codes and filesystem watchers forward via a\n tmpfs handover blob. The workload cgroups stay frozen until the post-upgrade `/init` restores the\n access token (so no re-adopted process runs unauthenticated), and the handover outcome\n (procs/watchers re-adopted, plus any failures) rides back on that `/init`'s `X-Envd-Handover`\n header for fleet visibility.\n- Scans guest ports and forwards them so any port a user process opens becomes reachable through\n sandbox URLs. **`pkg/version.go` carries the released version**, maintained by release-please from\n the Conventional Commits merged since the last envd release; do not edit it by hand. The API and\n the orchestrator gate features on the envd version recorded in each template build — the\n orchestrator's held `/init` is the one gate that reads the version of the envd expected to\n answer instead (see sandbox creation) — so a behavioral change ships under a `feat`/`fix`\n commit type that cuts a version it can be gated on.\n\n### Client proxy (`packages/client-proxy`)\n\nThe stateless edge for all sandbox traffic (port 3002; health on 3003). Terminates\n`https://-.` requests (host parsing in `packages/shared/pkg/proxy/host.go`),\nlooks the sandbox up in the Redis routing record to find the owning node, and reverse-proxies to\nthat node's orchestrator proxy on :5007 by default. `ORCHESTRATOR_PROXY_PORT` selects a\ndifferent downstream port when the node proxy listens elsewhere. If the sandbox is not in the record (paused), it calls\nthe API's `ResumeSandbox` gRPC — paused sandboxes wake transparently on traffic.\nThe record is the orchestrator-owned `sandbox:routing:{id}` (see \"Sandbox routing records\"\nbelow).\n\nThe record on a hit, or the `ResumeSandbox` answer on a miss, is the sandbox's route, placement\nincluded. A miss uses the resume answer directly and does not read the record again. A legacy\n(no placement) or egress-router route goes to `orchestrator_ip` on the orchestrator proxy port,\nnever to the placement's ingress endpoint. A full-router route goes to the Router's ingress\nendpoint saved in its placement, never to an orchestrator, over the same pooled, kept-alive\nconnections as the orchestrator hop: the proxies never resend a request themselves (only its\ndial is retried), and net/http's resend of a bodyless idempotent request whose reused connection\nclosed before any response is accepted on both hops. A full-router route without that endpoint,\nan unknown mode or a missing `orchestrator_ip` answers the not-found page before anything is\ndialed.\n\nConcurrent requests that miss the record for the same sandbox, port and credentials share one\n**resume flight** per client-proxy process: the first request's call runs detached from its\nconnection under a 90 s ceiling and later requests join it, so one resume RPC serves them all\nand a client that disconnects mid-resume does not fail the others. A call whose requests have\nall left runs to its end, so a client that comes back finds the sandbox awake, but no further\nattempt starts for nobody, and a process keeps at most 64 such callerless calls; beyond that a\ndeparture cancels the proxy's call. That bound and that cancellation apply to the proxy's flights\nonly: the API's resume entry point lets a create already under way run on past the proxy's\ncancellation, so the node finishes the create and the API registers the sandbox inside the\nreservation's 90 s window (the create may run until ten seconds before the pending reservation can\nbe reaped and the registration until five seconds before, so the run-on never lets a fresh\nreservation start a second resume of the same snapshot), and the API's shutdown drain waits for\nthat work, bounded at 95 s. The run-on is granted when the resume is admitted, and an API already\ndraining admits none: a resume that arrives then keeps its caller's cut, so a create the proxy\ncancels mid-drain is cut with it and the reservation lapses. The sandbox otherwise lives its\nordinary timeout or auto-pause, billed, with nobody waiting for it. With\n`client-proxy-resume-parking` on (default off), a request whose flight got a still-transitioning\nor `Unavailable` answer parks instead of failing: the flight retries\nwhile a parked request waits, under a budget (`client-proxy-resume-parking-budget-ms`, 15 s from\nthe first retryable failure) that gates only whether another retry starts: every attempt, a late\nretry included, runs under its full 90 s ceiling while a caller waits on it, so a parked request\ncan wait up to the budget plus one attempt. The proxy cuts a retry at the budget's end only once\nevery caller has left, and that cut ends the proxy's wait, not the create on the node.\nA process keeps at most\n`client-proxy-resume-parking-lot-size` (512) parked requests, shedding the rest. A resume the\ncontrol plane cannot carry out right now (no capacity, no node, the API unreachable, the budget\nspent) answers **503** with `Retry-After: 2` and no error code; a request shed because the lot was\nfull answers the same **503** without the header and with `proxy_parking_lot_full` as its\n`error_code`, so a client, or one behind an ingress that strips the header, still tells the two\napart. The SDKs do not act on the header either way: their status-based retry\nwraps only the control-plane API client, and sandbox traffic through the proxy is retried only on\na connection failure, never on a 503. The omission is for hand-rolled retry loops and browsers\nthat honour the header, which would otherwise come back in one wave while the lot is still full,\nand for a client that does the SDKs' jittered backoff on its own; a placement no retry helps, which\nthe API names in an `ErrorInfo` detail on the RPC status, is not retried and answers without the\nheader, with the status the API carried: **503** for a feature no orchestrator in the cluster\nsupports, **500** (`sandbox_create_refused`) for a sandbox the node refuses to create, such as a\nstored build whose vCPU count is outside the node's bounds; a sandbox still pausing\nanswers **409** with the header on a user port and the same page as **503** on the envd port (the\nSDK reads an envd 409 as \"already exists\"); a request whose client left before it was routed\nanswers 408.\n\n### Dashboard API (`packages/dashboard-api`)\n\nA separate REST service (port 3010, spec `spec/openapi-dashboard.yml`) consumed by the web\ndashboard and the workspace control plane, not the SDK: the killed-sandbox record, admin routes\n(clusters, team cluster assignment, user-profile resolution) and the `/v1/management`\noperations. Template reads are served by the API, not here. Its admin routes accept either a\nshared admin token or a short-lived service JWT verified against the configured admin JWKS. The\nsandbox record route accepts dashboard user auth only. Its workspace-agnostic `/v1/management` operations are defined in the\nsame dashboard OpenAPI contract and registered on the existing router. Their `AdminJWTAuth`\nOpenAPI security scheme accepts only short-lived service JWTs verified against the workspace-api\n`/.well-known/jwks.json` endpoint, with accepted signing methods derived from each JWK's required\n`alg` metadata. Issuers and audiences are configured through the JSON `ADMIN_AUTH_PROVIDER_CONFIG` value —\nthe same config shape as `AUTH_PROVIDER_CONFIG`. Talks to Postgres and ClickHouse; never talks to\norchestrators. The web dashboard itself is a separate repository (github.com/e2b-dev/dashboard)\npublished as an image; E2B Embed runs it on port 3001 beside dashboard-api.\n\nAn issuer normally configures at least one accepted audience, binding a JWT to its intended target.\nAn issuer may omit `audiences` only with no `audienceMatchPolicy`; this deliberately disables\naudience matching while issuer, signature, and temporal-claim verification remain required.\n\nThe `/v1/management` operations are the cluster's half of a contract the workspace residency owns:\nthe cluster lifecycle below. Project upsert, membership, limits, block state and deletion are\nserved by the API under `/unstable/management/projects/{projectID}`, and team bans under\n`/unstable/admin/teams` (see API above).\n\nThe management surface owns a replay-safe cluster lifecycle. A caller registers a stable\ncluster UUID, assigns it only to the named project, detaches that exact assignment before provider\ncleanup, and deletes the cluster only after no project references it. Replaying the same\nregistration or assignment succeeds. Registering the cluster again with the same name and auth\norganization may change its endpoint, TLS setting, sandbox proxy domain and token, for example when\nthe cluster moves to a new domain; the API rebuilds that cluster's clients on its next sync and gives\nsandboxes the new domain. A changed name or auth organization, a different assignment, an inexact\ndetach, or deletion of a referenced cluster returns a conflict. The\nassignment does not check the project's tier; the caller decides which projects may use a cluster.\nA cluster first registered through this surface starts without deletion protection; one created any\nother way starts protected, a later registration never changes it, and deleting a protected cluster\nor checking its destroy readiness returns a conflict.\nA caller that deletes a cluster's projects before the cluster first lists them with\n`GET /v1/management/clusters/{clusterID}/projects`, which returns every project whose team is\nassigned to the cluster however it was assigned, leaving out deleted teams, and records the deletion with\n`PUT /v1/management/clusters/{clusterID}/deletion` before deleting any of them. That request\nlocks the cluster row, returns a conflict while deletion protection is on, and sets\n`clusters.deletion_requested_at`; it succeeds for an absent cluster and on repeat. From then on a\ncheck constraint keeps deletion protection off, and a trigger on `teams` refuses to newly assign a\nteam to the cluster (`cluster_assignment_cluster_deleting` from the assignment route, a conflict\nfrom the API's project upsert carrying the cluster ID). The trigger takes a key-share lock on the cluster row\nbefore it checks, so an assignment either commits before the deletion is recorded, where the\ncaller's next listing finds it, or is refused. Deleting each project clears its team's cluster,\nthe cluster delete clears it from any team deleted earlier, and then succeeds.\nThe shared error response supports an optional `error_code`, defined by the dashboard OpenAPI\n`ErrorCode` schema. Registration and assignment rejections currently populate it; other errors\nmay omit it. Callers use that stable code rather than the human-readable message, and treat\nuncoded or unknown reasons as generic errors.\n\nA project deletion keeps the team row, so dashboard-api middleware, running after\nauthentication, answers 404 to every write under `/v1/management/` or `/admin/` that names a\ndeleted project by `projectID` or `teamID` (\n\n\n\n\n# Redis Reservation Storage\n\nThis package coordinates sandbox creation reservations across API instances.\n\n## Keys\n\n- Storage index: `sandbox:storage:{teamID}:index`\n- Pending zset: `sandbox:storage:{teamID}:reservations:pending`\n- Result key: `sandbox:storage:{teamID}:reservations:{sandboxID}:result`\n- PubSub routing key: `sandbox:storage:{teamID}:reservations:{sandboxID}:notify`\n\n## Flow\n\n`Reserve` runs a Lua script that atomically removes stale pending entries, checks whether the sandbox already exists or has already pending start, enforces the team limit using `SCARD(storage index) + ZCARD(pending zset)`, deletes any stale result key, and adds the sandbox ID to the pending zset.\n\nWhen creation completes, it removes the sandbox from the pending zset, writes a TTL result key containing either the sandbox or the creation error, and publishes the routing key.\n\nA waiter subscribes to the routing key, probes the result key immediately, then waits for PubSub notifications or the 1 second fallback ticker. PubSub is best-effort; the fallback ticker is required for correctness.\n\n`Release` is called when the sandbox is removed from storage (`Store.Remove`). It removes the sandbox from the pending zset, deletes the result key, and publishes the routing key.\n\n\nThe python SDK had a bug: new fields introduced to the API would throw exceptions when processed. In order to work around this, we convert all responses to a list of structs that were valid when the python SDK still had the bug.\n\nWe copy the proto files to this folder in order to keep track of what was available when this SDK was supported. These should never be modified.\n\nThis entire package can be removed once we no longer have clients that send a user agent of `connect-python`.\n\n\n# envd\n\nDaemon that runs inside a sandbox that allows interacting with the sandbox via calls from the SDK.\n\n## Development\n\n### Versioning\n\nThe envd version in `pkg/version.go` is maintained by release-please from the Conventional Commits merged since the last envd release; do not edit it by hand. A change that affects behavior (code, dependencies) ships under a `feat` or `fix` commit type, so that a version the orchestrator can gate on is cut. Pure comment or documentation changes use `docs`, a type the changelog hides, and cut no release; `chore` and `refactor` are visible in it and do cut one.\n\n### Running locally\n\nRun the following command to (re)build the envd daemon and start a Docker container with envd running inside:\n\n```bash\nmake start-docker\n```\n\nYou can use E2B SDKs with env var `E2B_DEBUG=true` or with a debug parameter set to `true` when creating or connecting to a sandbox, to connect to the envd started with this command.\n\n### Generating API server stubs\n\nRun the following command to install the necessary dependencies for generating server stubs:\n\n```bash\nmake init-generate\n```\n\n#### Generating\n\nAfter changing the API specs in `./spec/` run the following command to generate the server stubs:\n\n```bash\nmake generate\n```\n\n### Debugging\n\n- \n- \n\nRun `make run-debug` and then connect to the port `2345` with a debugger or use the VSCode run/debug and run the \"Debug envd\" to build the envd, Docker, and start the debugging.\n\n\n# Nomad Node Pool Autoscaler Plugins\n\nCustom plugins for the [Nomad Autoscaler](https://github.com/hashicorp/nomad-autoscaler) that keep a service-job allocation count aligned with the number of nodes in a Nomad node pool.\n\n## Purpose\n\nThe package builds two plugins because Nomad Autoscaler supports only one plugin type per external binary:\n\n- `nomad-nodepool-apm` reports the number of ready nodes in a pool.\n- `nomad-deployment-aware-target` fails an active deployment before applying the requested task-group count.\n\nTogether they let service jobs replicate system-job placement while retaining rolling updates. The deployment-aware target is used by the `template-manager` and `orchestrator-ee` service jobs. When the task-group count must change, scaling intentionally abandons a conflicting in-progress rollout (Nomad rejects scaling while a deployment is active); Nomad records that deployment as `failed`, rather than `cancelled`. The rollout spawned by the scale itself is left to run, and no deployment is touched when the count already matches. The target requires `auto_revert = false` on the scaled group (otherwise failing a deployment would restore an older job version and ping-pong); it refuses to scale groups with `auto_revert = true` and returns an error on every attempt.\n\n## Usage\n\n### Building\n\n```bash\nmake build\n```\n\n### Uploading to GCS\n\n```bash\nGCP_PROJECT_ID=your-project-id make build-and-upload\n```\n\n### Configuration\n\nIn your Nomad Autoscaler configuration:\n\n```hcl\napm \"nomad-nodepool-apm\" {\n driver = \"nomad-nodepool-apm\"\n config = {\n nomad_address = \"http://localhost:4646\" # Optional, uses NOMAD_ADDR env var\n nomad_token = \"your-token\" # Optional, uses NOMAD_TOKEN env var\n nomad_region = \"global\" # Optional\n }\n}\n\ntarget \"nomad-deployment-aware-target\" {\n driver = \"nomad-deployment-aware-target\"\n config = {\n nomad_address = \"http://localhost:4646\"\n nomad_token = \"your-token\"\n nomad_region = \"global\"\n }\n}\n```\n\n### In Job Scaling Policy\n\n```hcl\nscaling {\n enabled = true\n min = 1\n max = 50\n\n policy {\n evaluation_interval = \"30s\"\n cooldown = \"2m\"\n\n target \"nomad-deployment-aware-target\" {}\n\n check \"match_node_count\" {\n source = \"nomad-nodepool-apm\"\n query = \"orchestrator\" # Node pool name\n\n strategy \"pass-through\" {} # Use node count directly as desired count\n }\n }\n}\n```\n\n## How It Works\n\n1. The plugin queries the Nomad API to list nodes filtered by the specified node pool\n2. It counts only nodes that are:\n - Status: `ready`\n3. Returns the count as a metric for the autoscaler to use\n4. With the `pass-through` strategy, this count becomes the desired number of allocations\n5. The target serializes scaling per namespaced job and is a no-op when the durable count already matches\n6. When the count must change it fails a conflicting active deployment, rereads the job, and scales with the current job modify index\n7. It verifies the final durable count; the rollout spawned by the scale proceeds normally\n8. Concurrent job or deployment changes are retried from fresh state with a bounded attempt count\n\nDry-run actions do not fail deployments or write task-group counts.\n\n## Query Format\n\nThe `query` parameter should be the name of the node pool you want to count nodes from.\n\nExample: `query = \"orchestrator\"` counts all ready nodes in the `orchestrator` node pool.\n\n## Configuration Options\n\n| Option | Description | Default |\n|--------|-------------|---------|\n| `nomad_address` | Nomad server address | `NOMAD_ADDR` env var or `http://127.0.0.1:4646` |\n| `nomad_token` | Nomad ACL token | `NOMAD_TOKEN` env var |\n| `nomad_region` | Nomad region | Default region |\n| `nomad_namespace` | Nomad namespace | Default namespace |\n\n\nTo run inside of a host in GCS, trust this fancy one-liner:\n\n host=@ && bucket= && go build -o simulator . && ssh $host rm ./simulator && scp ./simulator $host:~/simulator && ssh $host -- ./simulator -csv-path gcs.csv -test-duration=5s $bucket && ssh $host cat gcs.csv\n\n\n# Orchestrator\n\n## Commands\n\n**Prerequisite:** Enable NBD module first:\n\n```bash\nmodprobe nbd nbds_max=4096\n\ncat </etc/udev/rules.d/97-nbd-device.rules\n# Disable inotify watching of change events for NBD devices\nACTION==\"add|change\", KERNEL==\"nbd*\", OPTIONS:=\"nowatch\"\nEOH\nudevadm control --reload-rules\nudevadm trigger\n```\n\n**Prerequisite:** Allocate enough 2MB HugeTLB pages (hugepages) for testing:\n\n```bash\necho 1024 | sudo tee /proc/sys/vm/nr_hugepages # for 2GB hugepages, adjust as needed\n```\n\n### Create Build\n\nCreate a new build.\n\n```bash\nsudo go run ./cmd/create-build -to-build -storage .local-build\n```\n\nFlags:\n\n- `-to-build ` - Output build ID (UUID, required)\n- `-from-build ` - Base build ID for incremental builds\n- `-template ` - Template ID (default: `local-template`)\n- `-storage ` - Local path or `gs://bucket` (enables local mode with auto-download of kernel/FC)\n- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)\n- `-kernel ` - Kernel version (default: `vmlinux-6.1.158`)\n- `-firecracker ` - Firecracker version (default: `v1.14.1_458ca91`)\n- `-vcpu ` - vCPUs (default: `1`)\n- `-memory ` - Memory in MB (default: `512`)\n- `-disk ` - Disk in MB (default: `1000`)\n- `-hugepages` - Use 2MB huge pages (default: `true`, set `false` for 4KB pages)\n- `-start-cmd ` - Start command\n- `-ready-cmd ` - Ready check command\n\n> You can use the `$(uuidgen)` command to generate a random UUID.\n>\n> If you are using Mise, you can run the command as `sudo $(which go) run ./cmd/create-build -to-build -storage .local-build`.\n\n### Resume Build\n\nResume a sandbox from a build.\n\n```bash\n# Local storage\nsudo go run ./cmd/resume-build -from-build -storage .local-build -iterations 10\n\n# Remote GCS storage (pass credentials to sudo)\nsudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \\\n go run ./cmd/resume-build -from-build -storage gs://bucket -iterations 10\n\n# Pause mode: resume, run command, then snapshot\nsudo go run ./cmd/resume-build -from-build -to-build \\\n -storage .local-build -cmd-pause \"pip install numpy\"\n\n# Pause mode: start command through envd, then wait for SIGUSR1 before snapshot\nsudo go run ./cmd/resume-build -from-build -to-build \\\n -storage .local-build -cmd-signal-pause \"python3 /home/user/workspace/job.py\"\n```\n\nFlags:\n\n- `-from-build ` - Build ID (UUID) to resume from (required)\n- `-to-build ` - Output build ID (UUID) for pause snapshot (auto-generated if not specified)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)\n- `-iterations ` - Number of iterations, 0 = interactive (default: `0`)\n- `-cold` - Clear cache between iterations (cold start each time)\n- `-no-prefetch` - Disable memory prefetching\n- `-v` - Verbose logging\n- `-pause` - Start and immediately pause (create snapshot)\n- `-signal-pause ` - Wait for signal before pause (e.g., `SIGTERM`, `SIGUSR1`)\n- `-cmd-pause ` - Execute command in sandbox, then pause on success\n- `-cmd-signal-pause ` - Execute command in sandbox, then wait for `SIGUSR1` before pause\n- `-envd-version ` - The guest's envd version (default: a placeholder — see the caveat below)\n\n**envd version caveat:**\n\n`resume-build` has no database and the snapshot metadata does not record the build's envd\nversion (only the kernel and Firecracker ones), so unless `-envd-version` is given the tool\nreports a fixed placeholder (`1.0.0`). Every `envd.version` field in its output and logs is\nthen that placeholder, **not** the guest's version — don't reason from it. To read the real\none, mount the build's rootfs (`mount-build-rootfs`) and run its `/usr/bin/envd -version`,\nor read `X-Envd-Version`, which a running envd reports on `/init`.\n\nThe placeholder sits above every envd version gate, so the gated capabilities (`/freeze`,\n`/fsfreeze`, `/collapse`, the KVM clock, the upgrade `from_version`) stay on. Against an\nolder guest that is a path production would not take, so pass `-envd-version` with the\nguest's real version when the behavior under test depends on one of those gates. The tool\nprints which of the two it is using when it loads the build.\n\n**Pause mode example:**\n\n```bash\n# Chain builds with explicit build IDs\nsudo go run ./cmd/resume-build -from-build $BUILD1 -to-build $BUILD2 \\\n -storage .local-build -cmd-pause \"apt install curl\"\n\nsudo go run ./cmd/resume-build -from-build $BUILD2 -to-build $BUILD3 \\\n -storage .local-build -cmd-pause \"pip install requests\"\n\n# Start a long-running command through envd, then trigger pause from the host\nsudo go run ./cmd/resume-build -from-build $BUILD3 -to-build $BUILD4 \\\n -storage .local-build -cmd-signal-pause \"python3 /home/user/workspace/job.py\"\n# In another shell:\nps -ef | grep 'cmd/resume-build' | grep -v grep\nsudo kill -SIGUSR1 \n```\n\n### Copy Build\n\nCopy a build between storage locations (local or GCS).\n\n```bash\n# Local to local\ngo run ./cmd/copy-build -build -from .local-build -to /other/path\n\n# Local to GCS\ngo run ./cmd/copy-build -build -from .local-build -to gs://bucket\n\n# GCS to GCS\ngo run ./cmd/copy-build -build -from gs://bucket1 -to gs://bucket2\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required)\n- `-from ` - Source: local path or `gs://bucket`\n- `-to ` - Destination: local path or `gs://bucket`\n- `-team ` - Team UUID (if set, prints SQL to populate DB on stdout)\n- `-envd-version ` - Envd version (required if team provided) — must match the version present in the template\n- `-vcpu ` - vCPUs (default: `2`)\n- `-memory ` - Memory in MB (default: `1024`)\n- `-disk ` - Disk in MB (default: `1024`)\n- `-tag ` - Build assignment tag (default: `default`)\n\n**Print DB seed SQL:**\n\nWhen `-team` is set, the command reads `metadata.json` from the destination to get `kernel_version` and `firecracker_version`, generates a new env ID, and prints SQL statements (`BEGIN`/`COMMIT` wrapped) to stdout. Pipe directly to `psql` to populate the database:\n\n```bash\ngo run ./cmd/copy-build -build -from .local-build -to gs://bucket \\\n -team -envd-version 0.2.11 -vcpu 2 -memory 1024 -disk 1024 \\\n | psql $POSTGRES_CONNECTION_STRING\n```\n\n### Mount Build Rootfs\n\n```bash\n# Local storage\nsudo go run ./cmd/mount-build-rootfs -build -storage .local-build -mount /mnt/rootfs\n\n# Remote GCS storage (pass credentials to sudo)\nsudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \\\n go run ./cmd/mount-build-rootfs -build -storage gs://bucket -mount /mnt/rootfs\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required unless `-empty`)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-mount ` - Mount path\n- `-verify` - Verify rootfs integrity (requires `-mount`)\n- `-log` - Enable logging\n- `-empty` - Create an empty rootfs instead of loading a build\n- `-size ` - Size of empty rootfs (default: 1GB)\n- `-block-size ` - Block size (default: 4096)\n\n### Inspect Build\n\nInspect build artifacts (headers and data blocks).\n\n```bash\n# Inspect memfile header (default)\ngo run ./cmd/inspect-build -build -storage .local-build\n\n# Inspect rootfs header\ngo run ./cmd/inspect-build -build -storage .local-build -rootfs\n\n# Inspect header + data blocks\ngo run ./cmd/inspect-build -build -storage .local-build -data\n\n# Inspect first 100 data blocks\ngo run ./cmd/inspect-build -build -storage .local-build -data -end 100\n\n# Remote GCS storage\ngo run ./cmd/inspect-build -build -storage gs://bucket -rootfs -data\n```\n\nFlags:\n\n- `-build ` - Build ID (UUID, required)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-memfile` - Inspect memfile artifact (default)\n- `-rootfs` - Inspect rootfs artifact\n- `-data` - Also inspect data blocks (not just header)\n- `-start ` - Start block index (default: `0`, only with `-data`)\n- `-end ` - End block index (default: all, only with `-data`)\n\n### Show Build Diff\n\nShow the diff between two builds and preview merge result.\n\n```bash\n# Memfile (default)\ngo run ./cmd/show-build-diff -from-build -to-build -storage .local-build\n\n# Rootfs\ngo run ./cmd/show-build-diff -from-build -to-build -storage .local-build -rootfs\n\n# With visualization\ngo run ./cmd/show-build-diff -from-build -to-build -storage gs://bucket -visualize\n```\n\nFlags:\n\n- `-from-build ` - Base build ID (required)\n- `-to-build ` - Diff build ID (required)\n- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)\n- `-memfile` - Inspect memfile artifact (default)\n- `-rootfs` - Inspect rootfs artifact\n- `-visualize` - Visualize the headers\n\n---\n\n## Architecture (ARM64) Support\n\nThe orchestrator supports both `amd64` (x86_64) and `arm64` (aarch64) architectures. Architecture is detected automatically via `runtime.GOARCH` at compile time.\n\n### Architecture naming convention\n\nThis project uses **Go/Docker/Debian naming** (`amd64`/`arm64`) for architecture directories in binary paths and GCS buckets:\n\n| Convention | x86_64 name | ARM64 name | Used by |\n|------------|-------------|------------|---------|\n| **Go/Docker/Debian** | `amd64` | `arm64` | This repo, Docker, `dpkg --print-architecture` |\n| Linux/GNU | `x86_64` | `aarch64` | `uname -m`, kernel Makefiles |\n\nBinary paths follow the `{version}/{arch}/` layout:\n\n```\n# Firecracker (GCS bucket or FIRECRACKER_VERSIONS_DIR)\nfc-versions/v1.12.1_210cbac/amd64/firecracker\nfc-versions/v1.12.1_210cbac/arm64/firecracker\n\n# Kernels (GCS bucket or HOST_KERNELS_DIR)\nkernels/vmlinux-6.1.158/amd64/vmlinux.bin\nkernels/vmlinux-6.1.158/arm64/vmlinux.bin\n```\n\n> **Note:** The [fc-kernels](https://github.com/e2b-dev/fc-kernels) repo currently uses `x86_64` instead of `amd64` for its directory names. This will be aligned in a follow-up change.\n\n### ARM64-specific behavior\n\n- **SMT** is disabled (ARM processors don't support simultaneous multi-threading)\n- **CPU detection** uses fallback values since `gopsutil` doesn't populate Family/Model on ARM64\n- **OCI platform** is set to the target architecture instead of hardcoded `amd64`\n- **Busybox binaries** are fetched per architecture by `scripts/fetch-busybox.sh` and selected at runtime from `///busybox`\n- **Guest kernel command line** drops the x86-only `i8042.*` and `clocksource=kvm-clock` parameters on arm64; the console stays `ttyS0` (Firecracker exposes a 16550-compatible UART on both architectures)\n\n### Cross-architecture deployment\n\n`TARGET_ARCH` is a **runtime** environment variable that overrides the architecture used for path resolution and OCI image pulls. When unset, defaults to the host architecture (`runtime.GOARCH`).\n\n```bash\n# Run orchestrator targeting amd64 paths from an arm64 host\nTARGET_ARCH=amd64 ./bin/orchestrator\n\n# Or in .env file (read at runtime)\necho \"TARGET_ARCH=amd64\" >> .env.local\n```\n\n`TARGET_ARCH` affects:\n- Firecracker and kernel binary path resolution (`{version}/{arch}/...`)\n- OCI image platform for container pulls\n\nIt does **not** affect:\n- Makefile compilation (see `BUILD_ARCH` below)\n- Hardware-dependent runtime behavior (SMT detection, CPU info) which always uses the actual host architecture\n\n### Building for ARM64\n\n`BUILD_ARCH` defaults to `amd64`. To build ARM64 binaries:\n\n```bash\n# Single command\nBUILD_ARCH=arm64 make build-local\n\n# Or set in .env.local for persistent override\necho \"BUILD_ARCH=arm64\" >> .env.local\n```\n\nThis applies to all services: orchestrator, envd, client-proxy.\n\n---\n\n## Environment Variables\n\nAutomatically set in local mode. Set before running to override:\n\n- `TARGET_ARCH` - Target architecture override (`amd64` or `arm64`; default: host architecture)\n- `HOST_ENVD_PATH` - Envd binary path (default: `../envd/bin/envd`)\n- `HOST_KERNELS_DIR` - Kernel versions dir (local: `{storage}/kernels`, prod: `/fc-kernels`)\n- `FIRECRACKER_VERSIONS_DIR` - Firecracker versions dir (local: `{storage}/fc-versions`, prod: `/fc-versions`)\n- `ORCHESTRATOR_BASE_PATH` - Base orchestrator data (local: `{storage}/orchestrator`, prod: `/orchestrator`)\n- `SANDBOX_DIR` - Sandbox working dir (default: `/fc-vm`)\n\n## Limitations\n\n- Custom template builds support base images from the declared distro families: **Debian/Ubuntu** (apt), the **RHEL family** — Fedora, CentOS Stream, Rocky, Alma — (dnf/microdnf/yum), **Arch** (pacman), and **Alpine** (apk, OpenRC). The distro is resolved from the image's `/etc/os-release` `ID` — never by probing for package managers. Images without an os-release identity (distroless, scratch) or from an undeclared family are rejected fast, with a build-log error naming the reason and the supported families (see `pkg/template/build/phases/base/distro`). Minimal/restricted-repo images may fail provisioning if their repos don't carry the required packages; the failure is surfaced in the build log.\n- Sandboxes always boot the kernel E2B supplies, never one from the base image: `/lib/modules` is empty, no kernel modules can be loaded, and SELinux is disabled on the kernel command line (`selinux=0`), so a base image that ships an SELinux policy boots without it. `kernel-devel` from a distro's repos resolves against a kernel that isn't running. **RHEL** (`rhel`, incl. UBI), **Oracle Linux** (`ol`) and **Amazon Linux** (`amzn`) are therefore not accepted even though they are RPM/dnf images — they are chosen for kABI, signed kernel modules and UEK, which this cannot honour. The community rebuilds above are supported because they are chosen for the userland.\n\n\n# Resource IDs at HTTP boundaries\n\n`apiid.ProjectID` and `apiid.WorkspaceID` accept a legacy UUID or the matching\npublic ID. Both inputs produce the same struct:\n\n```go\nprojectID.UUID // uuid.UUID\nprojectID.PublicID // canonical prj_ string\n```\n\nThe existing `pkg/id` codecs remain unchanged. These structs are HTTP input\nvalues; database and domain functions can continue taking `uuid.UUID`.\n\n## Adding or changing an API ID\n\nUpdate [`id`](../id/README.md#adding-or-changing-a-resource-id) and this package\nin the same PR whenever adding a resource kind to either package. Follow the\nlinked checklist: `id` owns the codec, and `apiid` provides the matching input\nstruct and binding support. Do not add an API-only encoding or a new codec\nwithout its API type. Keep existing kinds' encoding and validation changes,\ntests, and documentation synchronized across both packages.\n\n## OpenAPI and generated Gin handlers\n\nUse the standard oapi-codegen type override on a string schema:\n\n```yaml\nprojectID:\n name: projectID\n in: path\n required: true\n schema:\n type: string\n description: Project UUID or prj_ public ID.\n x-go-type: apiid.ProjectID\n x-go-type-import:\n name: apiid\n path: github.com/e2b-dev/infra/packages/shared/pkg/apiid\n```\n\nRegenerate the server and dependent clients using the API's generation command.\nNo middleware, registry, or generator fork is needed:\n\n```go\nfunc (s *Server) GetProject(c *gin.Context, projectID api.ProjectID) {\n project, err := s.projects.Get(c.Request.Context(), projectID.UUID)\n // Handle err and build the response as usual.\n}\n```\n\nUse `apiid.WorkspaceID` for workspace parameters. Apply the same schema override\nto query or header parameters and to JSON body properties or array items:\n\n```yaml\nworkspace_id:\n type: string\n description: Workspace UUID or wrk_ public ID.\n x-go-type: apiid.WorkspaceID\n x-go-type-import:\n name: apiid\n path: github.com/e2b-dev/infra/packages/shared/pkg/apiid\n```\n\nAfter `c.ShouldBindJSON(&body)`, use `body.WorkspaceId.UUID` and\n`body.WorkspaceId.PublicID`. Keep response UUID fields unchanged unless a\nresponse-contract change is intended. Do not keep `format: uuid` on inputs\nthat also accept public IDs.\n\n## Handwritten Gin binding\n\n```go\nvar params struct {\n ProjectID apiid.ProjectID `uri:\"projectID\" binding:\"required\"`\n}\nif err := c.ShouldBindUri(¶ms); err != nil {\n c.AbortWithStatus(http.StatusBadRequest)\n return\n}\nprojectUUID := params.ProjectID.UUID\n```\n\n`ShouldBindQuery` and `ShouldBindHeader` work with `form` and `header` tags.\n`UnmarshalText` supports generated path binding; `Bind` supports oapi-codegen's\nexploded query binding; `UnmarshalParam` supports Gin's own binders. Tests cover\neach path because these binders do not all use the same interface.\n\n## Construction and encoding\n\nUse `apiid.NewProjectID(u)` or `apiid.NewWorkspaceID(u)` when constructing an ID\nin Go. Decoders and constructors populate both fields. Malformed input and\nwrong-kind public IDs fail without modifying the receiver. JSON accepts only\nstrings; encoding emits a public ID string, not a JSON object.\n\nThe fields are exported for direct reads. Treat decoded values as immutable;\nGo cannot stop a caller from manually making them inconsistent. Encoding derives\nthe public string from `UUID`, rather than trusting a manually edited `PublicID`.\nA zero-valued struct denotes an unset input; requiredness belongs in the request\nschema or Gin validation. Use a pointer for optional inputs when absence matters.\n\nOther APIs adopt the types by updating their schemas and regeneration, not by\ninstalling middleware. This package does not change their contracts automatically.\n\n\n# LaunchDarkly client authentication\n\nThe client selects authentication in this order:\n\n1. A non-empty `LAUNCH_DARKLY_API_KEY` selects SDK-key authentication and ignores\n the OAuth variables.\n2. Without an SDK key, OAuth variables select OAuth client-credentials authentication.\n Incomplete OAuth configuration fails startup.\n3. Without either, the client uses the offline flag store.\n\nFor direct LaunchDarkly access, set `LAUNCH_DARKLY_API_KEY`. An optional\n`LAUNCH_DARKLY_BASE_URL` redirects streaming, polling, events, and diagnostics\nto a Relay proxy.\n\nFor an OAuth gateway, leave `LAUNCH_DARKLY_API_KEY` unset and configure:\n\n| Variable | Value |\n| --- | --- |\n| `LAUNCH_DARKLY_BASE_URL` | HTTPS gateway URL |\n| `LAUNCH_DARKLY_OAUTH_TOKEN_URL` | HTTPS OAuth token endpoint |\n| `LAUNCH_DARKLY_OAUTH_CLIENT_ID` | Deployment client ID |\n| `LAUNCH_DARKLY_OAUTH_CLIENT_SECRET` | Deployment client secret |\n| `LAUNCH_DARKLY_OAUTH_AUDIENCE` | Optional audience parameter for the token request |\n\nUse the existing deployment OAuth credentials and audience when the gateway\naccepts that identity. The token endpoint must support the `client_credentials`\ngrant with credentials in the request body.\n\nBoth client constructors enable live flag updates without an SDK key in OAuth\nmode.\nOAuth URLs must not contain embedded credentials, query parameters, or fragments.\n\nThe client caches tokens and refreshes them before expiry. Every new HTTP\nrequest uses the current bearer token, including stream reconnections, events,\nand diagnostics. The gateway must bound stream duration because an open stream\ndoes not reauthenticate when its token expires.\n\nToken requests have a five-second timeout. OAuth requests do not follow\nredirects. Authentication failures do not switch between SDK-key and OAuth modes.\n\nThe client mints its first token at startup. If the token endpoint answers with\na non-5xx response, construction fails, including with `WithStartOnInitTimeout`.\nIf the token endpoint is unreachable or returns 5xx, the SDK retries and the\nexisting initialization timeout policy applies.\n\nA gateway `401` or `403` before the gateway has accepted any request stops the\nSDK, as a rejected SDK key does. After the first accepted request, the client\ndrops its cached token on `401` or `403` and the SDK retries with a fresh one.\nThe SDK's own backoff bounds the retry and mint rate.\n\n\n# Typed public IDs\n\nThis package converts resource UUIDs to and from human-readable public IDs:\n\n```text\nprj_7mfb0gpp6c8dsaasre0asc7n3s\nwrk_2n1t201rmv87aae5j4csam8000\nsec_01kyjhkfvse96rg4d2qzd9enft\n```\n\nEvery public ID has a stable resource prefix, an underscore, and a\n26-character UUID suffix. E2B's public representation is always lowercase.\n\n## UUID ownership\n\nResource owners remain responsible for generating and storing UUIDs. UUIDs\nmay, for example, be generated by application code or a database. This package\ndoes not use `typeid.New` or `typeid.MustNew`.\n\n[`github.com/sumup/typeid`](https://github.com/sumup/typeid) is used only to\nconvert an existing UUID to its prefixed public representation and parse that\nrepresentation back into the UUID-backed resource type. PostgreSQL and other\npersistence layers continue to store the underlying UUID, not the prefixed\nstring.\n\n```go\nprojectUUID := uuid.New()\nprojectID := id.ProjectID(projectUUID)\n\npublicID := projectID.String()\nparsedID, err := id.ParseProjectID(publicID)\nif err != nil {\n\treturn err\n}\n\nprojectUUID = uuid.UUID(parsedID)\n```\n\nProject and workspace UUIDs are normally UUIDv4 and use `typeid.Random`.\nSecret UUIDs are UUIDv7 and use `typeid.Sortable`. SumUp renders Random IDs in\nuppercase, so the E2B wrapper normalizes them to the lowercase public contract.\n\n## Team IDs\n\nTeams are projects: a team UUID is also its project UUID.\n`ConvertTeamIDToProjectID` renders a team UUID as its public `prj_` ID, and\n`ParseTeamID` accepts either spelling wherever an API takes a team ID.\n\n## Adding or changing a resource ID\n\nKeep this package and [`apiid`](../apiid/README.md) in sync in the same PR.\nAdding a resource kind here requires its matching API input type in `apiid`;\nadding a type in `apiid` requires a matching codec here. Changes to an existing\nkind's prefix, encoding, or validation must also update the affected API type\nand tests. Encoding rules belong here; `apiid` must reuse them, not duplicate them.\n\nTo add another public resource ID:\n\n1. Choose a short, stable, lowercase prefix that keeps the full public ID at\n the expected fixed width.\n2. Add a private prefix type implementing `Prefix()`.\n3. Select `typeid.Random` for a UUIDv4 resource or `typeid.Sortable` for a\n UUIDv7 resource.\n4. Add a distinct UUID-backed public Go type with `String` and parse methods.\n5. Keep UUID generation in the resource-owning application or database.\n6. Test the exact prefix, lowercase output, fixed length, invalid-prefix\n rejection, and UUID round trip.\n7. Add the matching `apiid` struct with `UUID` and `PublicID`, its constructor,\n and its text, JSON, generated-query, and Gin binding methods.\n8. Test that UUID and public inputs produce identical API structs, wrong kinds\n are rejected, and encoding agrees with this package's codec. Update both\n READMEs and regenerate affected API contracts and clients.\n\nDo not add JSON, text, SQL, or UUID-generation behavior implicitly. Those\ninterfaces should be introduced only when a concrete caller needs them.\n\n\n# Shared sandbox types\n\nThis package owns `RuntimeMetadata`, `SandboxType`, and `EgressClass`, including\nidentity logging and the empty-type fallback to a regular sandbox. It contains\nno worker resources, network initialization, or VM lifecycle management.\n\nCallers import these types directly. `SandboxType.EgressClass()` selects the\ntraffic class; DSCP configuration and socket handling belong to worker networking.\nThe resource-owning `Sandbox` stays in the orchestrator package. Runtime team\nmetadata is attribution, not trusted authorization data.\n"}