# E2B Runtime
**The open-source runtime behind [E2B](https://e2b.dev/?utm_source=github&utm_medium=referral&utm_campaign=readme&utm_content=runtime), the AI agent cloud. Firecracker microVMs that resume from a snapshot, run untrusted agent code, and pause when the agent stops.**
[](./LICENSE)
[](./go.work)
[](https://github.com/e2b-dev/runtime/stargazers)
[](https://discord.gg/e2b)
[](https://x.com/e2b)
**[Docs](https://docs.e2b.dev/?utm_source=github&utm_medium=referral&utm_campaign=readme&utm_content=runtime)**
| [Architecture](./docs/ARCHITECTURE.md)
| [Run it yourself](./embed/README.md)
| [SDKs & CLI](https://github.com/e2b-dev/E2B)
| [Cookbook](https://github.com/e2b-dev/e2b-cookbook)
| [Contributing](./CONTRIBUTING.md)
---
## What is E2B Runtime?
E2B Runtime is the complete backend that powers [E2B Cloud](https://e2b.dev/?utm_source=github&utm_medium=referral&utm_campaign=readme&utm_content=runtime): the control-plane API, the per-node orchestrator that drives Firecracker, the agent that runs inside every VM, the edge router for sandbox traffic, and the template builder. It gives every agent session its own isolated Linux machine that boots from a snapshot, runs whatever the agent asks it to, and can be paused and resumed as if nothing happened.
It is written in Go, licensed under Apache-2.0, and built by [E2B](https://e2b.dev/?utm_source=github&utm_medium=referral&utm_campaign=readme&utm_content=runtime). The same code serves the public cloud, dedicated enterprise deployments, and the single-machine [Embed](./embed/README.md) package you can run on your own hardware.
## Why it's fast
Two ideas drive the design.
**A sandbox is a resumed snapshot.** Templates are pre-booted VMs (memory, disk, and machine state) stored in object storage. "Creating" a sandbox means restoring one, not booting a kernel. Memory pages are served lazily on page fault through `userfaultfd`, and the root filesystem is a copy-on-write overlay over a read-only image, so only the data a sandbox actually touches is ever fetched. Fresh creates, resumes after a pause, and forks all take the same path.
**Control plane and data plane never mix.** The API decides *where* a sandbox runs and records *that* it runs. The orchestrator on each node owns *how* it runs: the Firecracker process, the network namespace, the block device, the cgroup. Sandbox traffic goes straight from the edge to the node and never passes through the API.
## What you get
- **Hardware-isolated sandboxes.** One Firecracker microVM per sandbox, in its own cgroup and network namespace, with a per-sandbox nftables egress firewall and SNI/Host-inspecting domain allow and deny lists.
- **Pause and resume.** Pause diffs memory and disk against the template and ships the diff to object storage. Resume prefers the node that still has it cached. Idle sandboxes auto-pause, and incoming traffic wakes them transparently.
- **Fork a running sandbox.** Fork checkpoints a live sandbox in place and starts new sandboxes from that snapshot, up to a hundred per request, while the original keeps running untouched. A paused sandbox has the same artifact shape as a template, so resuming one takes the same fast path as creating one.
- **Templates built from your recipe.** Layered builds from Docker images and build steps, each layer hashed and cached, with a final optimize pass that records which pages a boot actually touches.
- **A real API inside every VM.** `envd` exposes processes, PTYs, filesystem operations, file watchers, and port forwarding over Connect RPC and REST. It is what the SDKs talk to when they "run code", and it can be live-upgraded inside a running sandbox without dropping the workload.
- **Sandbox URLs for anything that listens.** `https://-.` reaches any port a process opens, routed at the edge with per-sandbox access tokens.
- **Persistent volumes, secrets, workload identity.** Volumes outlive sandboxes. Secrets are metadata-only on the control plane: no value ever crosses the API, a log, or a span. Workload identity gives a sandbox short-lived tokens without the API ever minting a credential.
- **Observability built in.** Everything exports OpenTelemetry. Sandbox lifecycle events, host stats, and metrics land in ClickHouse.
## Run it
**On E2B Cloud.** The fastest way to use the runtime is to not run it. Grab an API key at [e2b.dev](https://e2b.dev/?utm_source=github&utm_medium=referral&utm_campaign=readme&utm_content=runtime) and start a sandbox from the [JavaScript or Python SDK](https://github.com/e2b-dev/E2B):
```python
from e2b import Sandbox
with Sandbox.create() as sandbox:
result = sandbox.commands.run('echo "Hello from E2B!"')
print(result.stdout)
```
**On one machine you own.** [E2B Embed](./embed/README.md) is the whole stack, real Firecracker sandboxes included, on a single Linux host with KVM. Two files and one command:
```bash
mkdir e2b && cd e2b
curl -fsSL --remote-name-all "https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/compose/{compose.yaml,.env}"
docker compose up -d --wait
```
The same package ships as Terraform for GCP, AWS, and Azure and as a Kubernetes manifest. Embed is one machine by design; when one node is not enough, see below.
**In your own cloud, for production.** E2B runs the runtime as a dedicated deployment inside your account, with your data staying there. See [e2b.dev/enterprise](https://e2b.dev/enterprise?utm_source=github&utm_medium=referral&utm_campaign=readme&utm_content=runtime).
## How it fits together
```
SDK / CLI ──REST──▶ API ──gRPC──▶ orchestrator ──▶ Firecracker microVM ──▶ envd ──▶ your processes
│ │
├── PostgreSQL ├── object storage (templates, snapshots)
├── Redis └── ClickHouse (events, metrics)
└── ClickHouse
browser ──https://-.──▶ client-proxy ──▶ orchestrator proxy ──▶ envd
```
| Service | Package | Runs on | Purpose |
|---|---|---|---|
| API | `packages/api` | control plane | Public REST API: sandbox lifecycle, placement, auth, quotas |
| Orchestrator | `packages/orchestrator` | every sandbox node | Runs Firecracker VMs: create, pause, resume, kill, checkpoint |
| Template manager | `packages/orchestrator` (role) | build nodes | Builds templates from Docker images and build steps |
| Client proxy | `packages/client-proxy` | control plane | Edge router: sandbox URL to the right node, auto-resume on traffic |
| Envd | `packages/envd` | inside every VM | Process, filesystem, and port API the SDKs use |
| Dashboard API | `packages/dashboard-api` | control plane | Backend for the web dashboard |
[`docs/ARCHITECTURE.md`](./docs/ARCHITECTURE.md) has the full picture: sequence diagrams for creation, traffic, pause and resume, and template builds, plus the data stores and the deployment topology. Read it first.
## Who is this for?
- **Teams building agents** who want to know exactly what their sandbox is, down to the kernel, and to run the same runtime locally that they run in production.
- **Platform teams** who need agent execution inside their own cloud account, on infrastructure they control, without a black box.
- **Infrastructure engineers** interested in Firecracker, lazy memory restore, copy-on-write block devices, and snapshot-based scheduling at scale.
## Developing
The runtime needs Linux with KVM. [`DEV-LOCAL.md`](./DEV-LOCAL.md) walks through the host prep, the local stack, and running services from source.
```bash
make local-infra # PostgreSQL, Redis, ClickHouse, monitoring
make test # unit tests across packages
make test-integration # against a live deployment
make generate # OpenAPI, proto, sqlc
make fmt lint tidy
```
Releases are per package and follow conventional commits; see [`docs/RELEASING.md`](./docs/RELEASING.md).
## Contributing
Bug fixes and docs fixes are welcome as pull requests. For anything larger, open an issue first so we can agree on direction before you write code. [`CONTRIBUTING.md`](./CONTRIBUTING.md) has the details, including what we are unlikely to merge and why.
## Community
- [Discord](https://discord.gg/e2b) for questions and help
- [GitHub Issues](https://github.com/e2b-dev/runtime/issues) for bugs
- [X](https://x.com/e2b) for release notes and what we are building
## License
Apache-2.0. See [`LICENSE`](./LICENSE).
---
### Overview (`embed/README.md`)
# E2B Embed
**E2B Embed runs the whole E2B stack, every feature included, on a single
machine in the environment of your choice.** Ship it inside your product, or
run it inside a customer's own environment, and bring E2B sandboxes to
customers whose data has to stay put, in government and in regulated
industries such as finance and healthcare. Same SDK, same API as E2B Cloud.
Open source, Apache-2.0.
[Docker Compose](#docker-compose)
| [Terraform on GCP](#terraform-on-gcp)
| [Terraform on AWS](#terraform-on-aws)
| [Terraform on Azure](#terraform-on-azure)
| [Kubernetes](#kubernetes)
| [Reference](docs/REFERENCE.md)
## How to run
Choose the setup that fits what you already have.
| Run it with | You need | The command | Install |
|-------------|----------|-------------|---------|
| Docker Compose | a Linux host you own | `docker compose up -d --wait` | [Docker Compose](#docker-compose) |
| Terraform on GCP | a Google Cloud project | `terraform apply` | [Terraform on GCP](#terraform-on-gcp) |
| Terraform on AWS | an AWS account | `terraform apply` | [Terraform on AWS](#terraform-on-aws) |
| Terraform on Azure | an Azure subscription | `terraform apply` | [Terraform on Azure](#terraform-on-azure) |
| Kubernetes | one node in your cluster | `kubectl apply -k` | [Kubernetes](#kubernetes) |
Every guide ends with your first sandbox, a few minutes after you start. If
you'd like to scale in production, please talk with us at
[e2b.dev/enterprise](https://e2b.dev/enterprise).
## What runs on the node
One node holds the complete E2B runtime. Your application talks to it
through the SDK, your team through the dashboard. The control plane is the
same API as E2B Cloud, the data plane runs the same E2B sandboxes, storage
keeps your templates and snapshots, and telemetry keeps your logs and
metrics. Nothing leaves the node. It is the same node wherever you run it,
on a Linux host you own with Docker Compose, on one VM on Google Cloud,
AWS or Azure with Terraform, or on one node in your Kubernetes cluster.
## What you get
- **The same sandboxes as E2B Cloud.** Isolated, paused and resumed on
demand.
- **The same SDK and API.** Moving your application over from E2B Cloud is a
configuration change.
- **Everything stays on one single machine.** Templates, sandbox logs and data live
on its disk and go nowhere else.
- **Private by design.** Only your application and your dashboard users need
to reach the node, and the Terraform installs let in only the addresses you
name. The [reference](docs/REFERENCE.md#ports-on-the-host-network) lists
what to open.
- **Your own templates.** Build custom sandbox templates through the same API
and start sandboxes from them right away.
- **The dashboard.** Your sandboxes, templates and logs in the browser, with a
terminal and a file browser for every sandbox.
- **Open source.** Apache-2.0, in the public
[E2B runtime repository](https://github.com/e2b-dev/runtime). No E2B
account and no license key needed.
### Docker Compose
Run it on a Linux machine you already have. [Full guide](compose/README.md#install).
```bash
mkdir e2b && cd e2b
curl -fsSL --remote-name-all "https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/compose/{compose.yaml,.env}"
docker compose up -d --wait
```
### Terraform on GCP
Create the machine in your Google Cloud project. [Full guide](terraform/gcp/README.md#install).
```hcl
module "e2b" {
source = "github.com/e2b-dev/runtime//embed/terraform/gcp?ref=main"
project_id = "my-project"
client_cidrs = ["203.0.113.0/24"] # where your SDK clients connect from
}
```
```bash
terraform init && terraform apply
```
### Terraform on AWS
Create the machine in your AWS account. [Full guide](terraform/aws/README.md#install).
```hcl
module "e2b" {
source = "github.com/e2b-dev/runtime//embed/terraform/aws?ref=main"
client_cidrs = ["203.0.113.0/24"] # where your SDK clients connect from
}
```
```bash
terraform init && terraform apply
```
### Terraform on Azure
Create the machine in your Azure subscription. [Full guide](terraform/azure/README.md#install).
```hcl
provider "azurerm" {
features {}
storage_use_azuread = true
}
module "e2b" {
source = "github.com/e2b-dev/runtime//embed/terraform/azure?ref=main"
client_cidrs = ["203.0.113.0/24"] # where your SDK clients connect from
}
```
```bash
terraform init && terraform apply
```
### Kubernetes
Run it on a node in your existing cluster. [Full guide](kubernetes/README.md#install).
```bash
kubectl create namespace e2b
kubectl -n e2b create secret generic e2b-api --from-literal=ADMIN_TOKEN="$(openssl rand -hex 32)" --from-literal=SANDBOX_ACCESS_TOKEN_HASH_SEED="$(openssl rand -hex 32)"
kubectl apply -k "https://github.com/e2b-dev/runtime//embed/kubernetes?ref=main"
```
## When one node is not enough
Embed is one machine, by design. When you need more, E2B runs three other
ways, with the same SDK and API, so moving is a configuration change.
- **Private cloud**, in development. The whole platform inside your boundary,
built for networks nothing may leave. Design partners welcome.
- **[Bring Your Own Cloud](https://e2b.dev/enterprise).** Your cloud account,
operated by E2B.
- **[E2B Cloud](https://e2b.dev).** Managed by E2B, nothing to host.
[See E2B for enterprise](https://e2b.dev/enterprise) for private cloud and
Bring Your Own Cloud.
## Limitations
- **Runs on Linux.** Embed needs a Linux machine with hardware
virtualization, and the Terraform installs create one for you. On a Mac, it
runs inside a Linux virtual machine (Apple silicon M3 or newer, macOS 15 or
newer).
- **Plain HTTP.** Embed answers over HTTP at the machine's own address, with
no TLS of its own, which is why it belongs inside your network. A port
inside a sandbox is reached at that same address. The
[reference](docs/REFERENCE.md#reaching-a-port-inside-a-sandbox) shows how.
## Reference
For whoever operates the machine, [`docs/REFERENCE.md`](docs/REFERENCE.md)
has the detail on networking, logs and metrics, secrets, releases, building
templates and the developer tooling.
---
### Overview (`embed/compose/README.md`)
# E2B Embed with Docker Compose
Runs the whole of Embed, the control plane and a real Firecracker
orchestrator alike, directly on one Linux host you own: no Terraform, no cloud
account, no wrapper scripts, and `docker compose up` as the whole interface.
It is a single-machine package rather than a deployment pattern;
the hub is [`../README.md`](../README.md).
## Requirements
- Linux x86-64 or arm64 with KVM and a 4 KiB-page kernel (Ubuntu's default
on both), on a host you own and may mutate: bare metal, or a VM created with
nested virtualization enabled (on GCE, `--enable-nested-virtualization`; on
EC2, `--cpu-options NestedVirtualization=enabled` on an Intel type that
offers it, such as `m8i`; on Apple silicon, a Lima or other
Virtualization.framework VM with nested virtualization on, which needs an
M3 or newer and macOS 15 or newer). x86-64
is what the guides were written and tested on. arm64 is verified end to
end on bare metal: on 2026-09-14 a bare-metal arm64 host (a GCE `c4a` metal
instance, Ubuntu 26.04, kernel 7.0) reached `ready` in under two minutes
and passed the smoke test, sandbox create, pause and resume, template
builds from arm64 images and 64 concurrent sandboxes. arm64 needs kernel
6.10 or newer: restoring a sandbox needs userfaultfd write-protect, which
arm64 gained in Linux 6.10, so on Ubuntu 24.04's stock 6.8 kernel the stack
is healthy but every sandbox start fails with `Failed to UFFD object` until
`linux-generic-hwe-24.04` is installed. The nine pinned images are
published for both architectures, and `fetch-artifacts` verifies the arm64
orchestrator and envd against the `.sha256` their release writes beside
the object. A pin with no such object still stops with a `FIX:` line
naming it.
- Ubuntu 24.04 is the recommended host: kernel 6.8 or newer (6.10 or newer on
arm64, above), glibc 2.34 or newer (the released orchestrator's floor),
cgroup v2 (systemd's default),
and `iptables`, `rsync`, `e2fsprogs` and `iproute2` installed. Ubuntu
server has all four but not `python3-venv`, which [Try it](#try-it)
needs. Refresh the index first, or a package that is genuinely missing
fails to fetch:
`sudo apt-get update && sudo apt-get install -y iptables rsync e2fsprogs iproute2 python3-venv`.
- `/dev/kvm` and `/dev/net/tun` present. On a VM, `/dev/kvm` means nested
virtualization is enabled for it; on GCE, stop the VM, run
`gcloud compute instances update --enable-nested-virtualization`, and
start it again; on EC2, stop the instance, run
`aws ec2 modify-instance-cpu-options --instance-id --nested-virtualization enabled`,
and start it again. Without either device, `up` stops at `preflight` with a
`FIX:` line and nothing that needs KVM is started.
- Docker Engine 27 or newer with Compose 2.24 or newer, and your user in the
`docker` group. Docker's own apt repository is the simplest way to get
both: `curl -fsSL https://get.docker.com | sh && sudo usermod -aG docker
$USER`, then log in again. Ubuntu 24.04's `docker-compose-v2` package also
clears the Compose floor. `docker version` and `docker compose version`
show what you have; an
older Compose rejects the compose file with an error on `required:` before
anything starts.
- 12 GiB RAM recommended and 20 GiB free disk. A pause the disk cannot hold is
refused before it starts, keeping 1 GiB free
(`PAUSE_ADMISSION_DISK_HEADROOM_MIB`), instead of failing midway; the
estimate counts the sandbox's whole memory, so a large sandbox needs that
much free disk to pause. `HUGEPAGES=2048` reserves
4 GiB of that RAM for sandboxes, and `preflight` does not check RAM, so on
a smaller host the first signal is `host-setup` failing with `FIX: give the
host more memory (12 GiB recommended) or lower HUGEPAGES`. Lower
`HUGEPAGES` rather than assuming 8 GiB is always enough. Size it by the
sandboxes you will run at once: every sandbox takes its guest memory from
this reservation as it touches pages, up to its full size (256 of the 2 MiB
pages for a 512 MiB `base` sandbox, about 90 while idle), so the default
carries eight sandboxes at full memory or about twenty idle ones. Past
that the stack stays healthy and each create fails with `500: Failed to
place sandbox: this node cannot map 512 MiB of guest memory: its memory
pool is too small or in use by running sandboxes`, with the sandbox's own
size in the message, while `docker compose logs orchestrator` shows
`create refused: guest memory not mappable`. Set a larger `HUGEPAGES` in
`.env` and run `up` again: `host-setup` re-applies the sysctl on every
start and the orchestrator needs no restart. A killed sandbox gives its
pages back a few hundred milliseconds after the kill returns; a create
sent in that window is retried on this host until they are back
(`CREATE_MEMORY_REFUSAL`) instead of failing.
- Outbound HTTPS to Docker Hub, `us-docker.pkg.dev`, `storage.googleapis.com`,
`github.com` and `raw.githubusercontent.com`. No account and no token:
every image and binary the stack pulls is public.
## Install
Two files, nothing else:
```bash
mkdir e2b && cd e2b
curl -fsSL --remote-name-all "https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/compose/{compose.yaml,.env}"
docker compose up -d --wait
```
The first run checks the host, prepares it, downloads the Firecracker
artifacts (about 60 MB), pulls the pinned images, migrates and seeds the
databases and builds the `base` template. It exits 0 only when the base
template is ready, about 2 minutes on an 8-vCPU / 32 GiB VM; a second `up`
takes about a minute, the healthcheck start periods. Smaller hosts take
longer. Nothing is rolled back if it fails, and nothing needs to be: see
[If `up` fails](#if-up-fails).
## Try it
Put this install's three SDK variables and the dashboard URL, its own team
API key included, in your shell:
```bash
eval "$(docker compose exec ready cat /run/e2b/sdk.env)"
```
Open the dashboard at `$E2B_DASHBOARD_URL` (http://localhost:3001) and paste
`$E2B_API_KEY` into its key form. On the machine itself, or through the
tunnel in [Ports](#ports), that is all. From another machine without a
tunnel, the browser also has to reach sandbox traffic on port 3002 at an
address it can open: set `E2B_DASHBOARD_HOST=` in
`.env` and run `docker compose up -d --wait` again, which recreates only the
`dashboard` service.
Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's `pip` refuses to write to. On Ubuntu `venv`
comes from `python3-venv`: install it first, which the apt line in
[Requirements](#requirements) does, or this fails with `ensurepip is not
available`.
```bash
python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0
```
Then create a sandbox. Save this and run it with `.venv/bin/python`:
```python
from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()
```
Or run the packaged smoke test, which does the same with the JavaScript SDK
in a container and then reaches a port inside the sandbox through
client-proxy:
```bash
docker compose --profile test run --rm smoke
```
## Remove
```bash
docker compose down # stop; running sandboxes end, the data and host state stay, so the next up is fast
docker compose down -v # also the databases and the team key; the next up rebuilds base
docker compose --profile purge run --rm host-teardown # undo the host setup
```
The purge is valid only after `down -v`. If you purged after a plain `down`,
run `down -v` before the next `up`, or the database still lists templates
whose files the purge removed.
## Reference
### Everyday commands
| Command | Effect |
|---------|--------|
| `docker compose up -d --wait` | start (or reconcile) everything and wait until the `base` template exists |
| `docker compose ps` | service status; `ready` running means the stack is usable |
| `docker compose logs -f api orchestrator` | follow logs |
| `docker compose logs ready` | the four `export` lines, this install's team API key and dashboard URL included |
| `docker compose --profile test run --rm smoke` | run the SDK smoke test in a container |
One stack per host; there are no VM-name or port knobs. `E2B_DASHBOARD_HOST`
(default `localhost`) is the address a browser uses for sandbox traffic on
port 3002 from the dashboard; set it in `.env` to this host's address when
you open the dashboard from another machine without a tunnel. `HUGEPAGES`
(default 2048) and `PF_MIN_FREE_GIB` (default 20) can be lowered in `.env` or
the environment for small hosts and CI. `NBDS_MAX` (default 64) is the number
of NBD devices `host-setup` asks the kernel for. Every running sandbox holds
one and a template build in flight holds one or two more, so the loaded
module's `nbds_max` is the ceiling on concurrent sandboxes: with every device
taken, a create waits for one until the api gives up (`500: Failed to place
sandbox: sandbox creation failed` or `504: placement timed out`), and the
creates queued behind it in a burst get `503: not enough capacity`, the node's
refusal to have more than three sandboxes starting at once. A module already
loaded with more devices than `NBDS_MAX` is accepted and raises the ceiling.
The module takes the number only when it loads, so set `NBDS_MAX` in `.env`
before the first `up`, or with the stack down: once `ls /sys/block/nbd*/pid`
prints nothing, `modprobe -r nbd` on the host and `up` again; `host-setup`
stops with a `FIX:` line when the loaded module has fewer devices than asked.
The orchestrator also refuses the 201st running sandbox (`max number of
running sandboxes on node reached (200)`), a limit this stack cannot raise.
`NBD_POOL_SIZE` (32) is the orchestrator's warm buffer of claimed devices,
not a ceiling, and stays below `NBDS_MAX` on purpose: at parity the pool's
refill loop spins and logs `no free slots` every few seconds on an idle host.
`FORCE_REBUILD=1`, settable in the
same two places, forces `base-template` to rebuild even when a `base` row
already shows `ready`, for a stale or broken template. `TEAM_API_KEY`,
`ADMIN_TOKEN` and `SANDBOX_ACCESS_TOKEN_HASH_SEED` are under Secrets below,
and `E2B_OTEL_COLLECTOR_GRPC_ENDPOINT` under Telemetry.
An optional, git-ignored `env/api.local.env` can add api variables. Create
the `env/` directory next to `compose.yaml` yourself, since the two-file
install does not ship one. It cannot override a key already set in the api
`environment:` block, because Compose gives `environment:` precedence over
`env_file`, so change those in `compose.yaml`.
### What runs where
The reference's [What runs where](../docs/REFERENCE.md#what-runs-where) has the services.
This is what they do to the host, which is why it should be a dedicated host
or VM: on every `up`, `host-setup`
- loads the `nbd` (with `nbds_max=NBDS_MAX`), `tun` and `kvm` kernel modules;
- writes `/etc/modules-load.d/e2b.conf`, `/etc/modprobe.d/e2b-nbd.conf`,
`/etc/udev/rules.d/97-nbd-device.rules` and `/etc/sysctl.d/90-e2b.conf`,
which also raises `net.ipv4.tcp_max_syn_backlog` and `vm.max_map_count`;
- adds one iptables mangle rule, the TCP MSS clamp;
- reserves the hugepages;
- creates `/var/lib/e2b`, `/fc-*`, `/orchestrator` and `/var/run/netns`.
`host-teardown` removes the files, the directories, the hugepage reservation
and the iptables rule, and unloads `nbd` when no device is still connected
(it says `nbd left loaded` when one is). It removes the iptables rule only if
`host-setup` was the one that added it: a host that already had an identical
TCP MSS clamp keeps its own firewall configuration, and the purge says
`MSS clamp: not added by host-setup, kept`. `kvm` and `tun` are left loaded
on purpose, because other software on the host may need them, and the two
extra sysctls stay at their raised values; all of that reverts on the next
reboot.
After a host reboot, run `docker compose up -d --wait` once. The long-running
services return on their own (`restart: unless-stopped`), but the TCP MSS
clamp and `/var/run/netns` are the two host mutations that do not persist
across a reboot (the clamp is an iptables rule, `/var/run/netns` lives on a
tmpfs), and `host-setup` recreates both idempotently on that run.
### If `up` fails
Nothing is rolled back and nothing needs to be: every step is idempotent. Read
the `FIX:` line in the failing service's log (`docker compose logs preflight`,
`host-setup`, `fetch-artifacts` or `base-template`), fix what it names, and
run `docker compose up -d --wait` again; the steps that already succeeded are
skipped.
A failed `preflight` has changed nothing on the host. A failed `host-setup`
leaves behind whatever it had already done, all of it idempotent: the config
files, the loaded modules and, depending on where it stopped, the reserved
hugepages (`HUGEPAGES` times 2 MiB, 4 GiB by default) and the TCP MSS rule,
which `host-teardown` or a reboot releases. A failed download leaves only fully verified files behind.
A failed template build leaves a template row marked `error` that the next
build replaces; the `base-template` step already runs one Firecracker build VM
through the orchestrator, which the same SIGTERM and `host-teardown` sweep
cleans up like any other sandbox. `ready` is the marker that the stack is
usable for the sandboxes you create.
To undo everything instead:
```bash
docker compose down -v && docker compose --profile purge run --rm host-teardown
```
### Ports
The thirteen ports and what each is for are in the hub. Compose binds them on
every host interface: let trusted clients reach 3000, 3001 and 3002, and make
sure the other ten are reachable on no address the host holds, its own public
one included, not only from the network edge.
From another machine, tunnel rather than open, and use the same four
variables with `127.0.0.1` in place of `localhost`. Template builds with
`copy()` steps upload through the orchestrator's port 5008, which stays closed
to the network, so tunnel that too; the upload URL the stack hands out
(`http://127.0.0.1:5008/...`) is then valid unchanged.
```bash
ssh -N -L 3000:127.0.0.1:3000 -L 3001:127.0.0.1:3001 -L 3002:127.0.0.1:3002 -L 5008:127.0.0.1:5008 @
```
### Secrets
The team API key is this install's own. The seed generates it on the first
`up` and keeps it in the `seed-state` volume (`/run/e2b/team-api-key`), where
a later seed run finds it again and `base-template`, `smoke` and `ready` read
it; `docker compose logs ready` prints it and
`docker compose exec ready cat /run/e2b/sdk.env` returns it as shell exports.
`down -v` drops that volume together with the databases, so the next `up`
generates a new key and there is never a key without its database or a
database without its key.
To choose or rotate the key, set `TEAM_API_KEY` (`e2b_` plus at least 32 hex
characters) in `.env` and run `up`. To rotate to a fresh generated key, remove
the file and run `up`:
```bash
docker compose run --rm --no-deps --entrypoint rm seed /run/e2b/team-api-key
docker compose up -d --wait
```
Either way the seed inserts the new key and revokes its earlier one, which
stops working within five minutes (the api caches team lookups in Redis for
that long). Within a few seconds `ready` re-renders `/run/e2b/sdk.env` and
`docker compose logs ready` prints the new key: `up` re-runs the seed but does
not recreate `ready`, so `ready` watches the key file instead. Keys created
any other way are never touched, and nothing in the two shipped files carries
a key ([`../tests/team-api-key.bats`](../tests/team-api-key.bats)).
The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the dashboard is served over plain http;
sign-out clears it, and rotating the key signs every browser out.
The api's own two secrets, `ADMIN_TOKEN` and
`SANDBOX_ACCESS_TOKEN_HASH_SEED`, are this install's own in the same way. The
`api-secrets` one-shot generates both on the first `up` and keeps them in the
`seed-state` volume as `/run/e2b/api.env`, a file only root can read, as
`GEN_ADMIN_TOKEN` and `GEN_SANDBOX_ACCESS_TOKEN_HASH_SEED`; a later run finds
them again and the entrypoints of the api and dashboard-api read them;
nothing prints them, and `down -v` drops them with the databases. The
reference's [Secrets](../docs/REFERENCE.md#secrets) says what the two are
for.
To choose your own instead, put them in `.env` before the first `up`. A value
there wins over the generated one, one variable at a time, and the install is
still two files:
```bash
printf 'ADMIN_TOKEN=%s\nSANDBOX_ACCESS_TOKEN_HASH_SEED=%s\n' "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" >> .env
```
To rotate the generated pair, remove the file, write a new one and recreate
the api on it:
```bash
docker compose run --rm --no-deps --entrypoint rm api-secrets /run/e2b/api.env
docker compose run --rm --no-deps api-secrets
docker compose up -d --no-deps --force-recreate api dashboard-api
```
`--no-deps` throughout: the two one-shot runs need no other service, and
recreating the api's dependencies would restart the orchestrator and end every
running sandbox. The new hash seed invalidates the traffic and envd tokens of
the `secure` sandboxes that survive the api's restart, and the new admin token
invalidates every admin client. A value pinned in `.env` is not rotated by
this: change the line and recreate the api and dashboard-api.
`docker compose exec api env` shows both empty: the entrypoint fills them in
for the api process only. Read them from the volume instead:
`docker compose run --rm --no-deps --entrypoint cat
api-secrets /run/e2b/api.env`.
### Telemetry
Nothing is exported by default, so the dashboard's monitoring charts and the
SDK's `getMetrics` (`get_metrics` in Python) stay empty. To run the built-in
OpenTelemetry collector, which keeps those metrics in this stack's
ClickHouse, uncomment the two lines that ship commented at the end of the
optional block in `.env`:
```bash
E2B_OTEL_COLLECTOR_GRPC_ENDPOINT=127.0.0.1:4317
COMPOSE_PROFILES=otel
```
Then run `docker compose up -d --wait`, which starts `otel-collector` and
recreates the four services that export to it; recreating the orchestrator
ends the sandboxes running on it. While the collector runs,
`curl -s 127.0.0.1:13133` answers `Server available`, and once a sandbox has
been running for a few seconds its rows show up:
`docker compose exec clickhouse clickhouse-client --user clickhouse --password clickhouse --query "SELECT count() FROM sandbox_metrics_gauge"`.
For a collector of your own, uncomment only the first line and put that
collector's address in it. To turn the built-in one off again, comment both
lines out, run `docker compose rm -sf otel-collector` and then
`docker compose up -d --wait`. The reference's
[Observability](../docs/REFERENCE.md#observability) says what each choice
gives you and how to forward traces and logs as well.
### Upgrading
Download `compose.yaml` and `.env` again, with the same command as
[Install](#install), and run `docker compose up -d --wait --pull always`.
Before that `up`, carry the optional lines you had set, such as
`E2B_DASHBOARD_HOST` or the two under [Telemetry](#telemetry), over into the
new `.env`: it ships them commented again, and without the Telemetry pair the
services stop exporting. The `.env` you get pins the stack images that go with
those files, so the two always move together. To take a particular commit
rather than the newest, put the commit in place of `main` in the two URLs, as
in `raw.githubusercontent.com/e2b-dev/runtime//embed/compose/…`; the raw
host ignores `?ref=`. Binary checksums live inside the tools
image, so bumping a Firecracker artifact means taking newer files rather than
editing anything on the host. Running sandboxes end when the orchestrator
restarts; a single machine has nowhere to drain them to.
### Known limitations
- Nothing on this stack authenticates a network peer except api's own API key
and admin token check. Thirteen ports bind every host interface, and the
orchestrator's 5008 in particular is an unauthenticated control API that
creates and kills sandboxes. Expose 3000, 3001 and 3002 to trusted clients if
you must, and keep the other ten unreachable on every address the host holds,
its own public one included.
- The team API key rotates only through the seed (set `TEAM_API_KEY`, or
remove the key file, then `up`), not through an API call, and the old key
keeps working for up to five minutes afterwards.
- A pause the node refuses — it is still persisting the sandbox's parent, or
the disk cannot hold the capture — leaves the sandbox running and answers
503 (`PAUSE_REFUSAL_RESTORE`); retry it.
- Container logs are capped: every service keeps at most five 50 MiB log
files (`x-logging` in `compose.yaml`), so `docker compose logs` shows about
the last 250 MiB per service and older lines are gone. Without the cap a
service that fails on a full disk writes error output faster than anything
frees space and holds the disk at 100%. Raise the cap in your copy for a
longer history.
- Expected log noise, all harmless: an OIDC warning from api at startup,
because no identity provider is configured, and, every ten seconds from both
api and the orchestrator, `failed to upload metrics: exporter export
timeout`, because no OpenTelemetry collector is configured and the endpoint
is empty. That line stops once `E2B_OTEL_COLLECTOR_GRPC_ENDPOINT` names a
collector that answers ([Telemetry](#telemetry)).
dashboard-api also logs `ADMIN_AUTH_PROVIDER_CONFIG is not configured` once
at startup; the management endpoints it guards are not used here.
- To run without the dashboard, remove the `dashboard` and `dashboard-api`
services from your copy of `compose.yaml`, and `dashboard` from `ready`'s
`depends_on`; the rest of the stack runs.
### Troubleshooting
- **Logs missing for a sandbox or a build.** `docker compose logs vector`
shows insert errors (ClickHouse down, or a row it refused), and
`docker compose exec clickhouse clickhouse-client --user clickhouse --password clickhouse --query "SELECT count() FROM sandbox_logs"`
shows whether rows arrive at all. Vector buffers and retries while
ClickHouse is down; the api's log endpoints fail during that time.
- **`preflight` failed.** Read the `FIX:` line in
`docker compose logs preflight`. The most common cause is a missing
`/dev/kvm`: on a VM, nested virtualization was off when it was created.
- **`up --wait` failed at `base-template`.** Its log ends with `FIX: read the
api and orchestrator logs for the template-manager error, then start the
stack again`, and `docker compose logs orchestrator` has the
template-manager error itself. A line saying `orchestrator not
registered with the api yet, retrying` is normal for the first seconds after
a start.
- **The smoke test failed.** Its output ends with `FIX: read the api,
orchestrator and client-proxy logs`; read those three logs. If the last line
is instead `FIX: the exposed port could not be reached; read the
client-proxy and orchestrator logs`, the sandbox was created and ran the
command, but the listener the test started on port 8080 did not answer
through client-proxy within 15 seconds. If it is `FIX: the sandbox could not
be killed; read the api and orchestrator logs`, the sandbox was created and
ran the command but the stack could not end it. That still fails the smoke
test, and the sandbox may still be running.
- **`Stopping orchestrator, success: false`** and `orchestrator-launch:
orchestrator exited with status 1` at the end of the orchestrator's log
after `docker compose stop` are the binary's normal SIGTERM path, not a
crash; only a line starting `FIX:` means an actual failure. Once `down` has
removed the container there is no log left to read.
- **Template builds fail inside the sandbox with `Hash Sum mismatch`, `Remote
end closed connection` or `Unable to fetch some archives`** while small
downloads work: the MSS clamp is missing. This is expected on every GCE VM,
whose NIC MTU is below 1500, and can also happen after a reboot.
`sudo iptables -t mangle -S FORWARD | grep TCPMSS` must show one rule; rerun
`docker compose up -d --wait` to have `host-setup` add it.
- **Sandboxes fail to start although the orchestrator is healthy.** Confirm
that the `orchestrator` service has `cgroup: host` in `compose.yaml` and
that its launcher's `nsenter` call includes `-C`, in
[`scripts/orchestrator-launch.sh`](scripts/orchestrator-launch.sh) rather
than in `compose.yaml`; the symptom is `fork/exec /usr/bin/unshare: no such
file or directory` in `docker compose logs orchestrator`.
- **A missing or incomplete `.env` beside `compose.yaml`** is the likeliest
two-file mistake. Compose stops before starting anything and names every
variable it could not resolve:
```
error while interpolating services.api.image: required variable
E2B_API_IMAGE is missing a value: fetch the .env that ships beside
compose.yaml (README, Install)
```
Fetch `.env`, or the lines it is missing, with the Install command above and
retry. A partial `.env` fails the same way, naming only what is missing.
- **`/sys/module/nbd/parameters/max_part` shows 31** for the configured 16.
That is the kernel's rounding, and it is expected.
- **`host-teardown` refuses to run while the stack is up**, with `FIX: run
docker compose down -v first, then docker compose --profile purge run --rm
host-teardown`.
- **`host-teardown` said `nbd left loaded`.** A device was still connected. It
unloads on reboot, or run `sudo modprobe -r nbd` once `pgrep firecracker`
prints nothing.
- **Stuck stack.** `docker compose down -v && docker compose --profile purge
run --rm host-teardown && docker compose up -d --wait` recreates everything
in a few minutes once the images are cached.
### Developing
This directory holds the two files the Install downloads together with the
scripts, configs and Dockerfiles behind the images they pin. From a checkout,
with the package root as the working directory, the Python smoke test runs
directly:
```bash
python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0
.venv/bin/python compose/smoke/smoke_test.py
```
It needs the `python3-venv` package on Ubuntu, and
[`smoke/smoke_test.py`](smoke/smoke_test.py) is not shipped in either stack
image, so it is not reachable from a two-file install. The stack is tested
with the 2.46 SDK line (`e2b@2.46.1`, `e2b==2.46.0`), the versions the
container smoke test and the images pin.
Changing an E2B component pin means editing `.env` and, for a binary, the
checksums in [`scripts/fetch-artifacts.sh`](scripts/fetch-artifacts.sh); then
`make images` builds the three stack images locally under their pinned tags,
and `docker compose up -d --wait` **without** `--pull always` uses the images
just built rather than the published ones. The reference's
[Developing](../docs/REFERENCE.md#developing) covers `make lint`, `make test`,
`make sync-configs` and `make stores-check`, all of which run from the package
root.
---
### Overview (`embed/kubernetes/README.md`)
# E2B Embed on one Kubernetes node
One StatefulSet that translates [`../compose/compose.yaml`](../compose/compose.yaml):
the stores, Vector, the orchestrator, api and client-proxy run as sidecar init
containers in the compose start order, the one-shots as init containers
between them, and the pod is Ready once the `base` template exists. It is a
single-node package rather than a Kubernetes deployment pattern;
the hub is [`../README.md`](../README.md).
## Requirements
- Kubernetes 1.29 or newer, for sidecar init containers.
- One Linux x86-64 or arm64 node with KVM (`/dev/kvm` present, which on a VM
means nested virtualization is on) and a 4 KiB-page kernel: Ubuntu 24.04, or
another host with kernel 6.8 or newer, glibc 2.34 or newer, cgroup v2 and
the `iptables`, `rsync`, `e2fsprogs` and `iproute2` tools; 12 GiB RAM and
20 GiB free on `/`. No Container-Optimized OS. arm64 is verified end to
end on a bare-metal arm64 host with Docker Compose (2026-09-14, kernel
7.0); no arm64 node has run this StatefulSet yet. arm64 needs kernel 6.10
or newer (userfaultfd write-protect, see the Compose guide). The nine
pinned images are published for both architectures, and the `fetch-artifacts`
init container verifies the arm64 orchestrator and envd against the
`.sha256` their release writes beside the object. A pin with no such object
still stops with a `FIX:` line naming it.
- 4 GiB of 2 MiB hugepages, reserved before the kubelet starts, because it
advertises only what it saw then. Every sandbox needs them, and the
orchestrator's request is what stops the kubelet capping the pod at zero.
- The node labelled `e2b.dev/single-node=true`: the databases live on its
disk, so the pod is pinned to it.
- `git` on the machine you run `kubectl` from: the remote `apply -k` URL
below and the `kubectl kustomize` against the same URL under Try it are git
sources, and kustomize clones them with that binary.
## Install
On a fresh node, in this order (the reservation precedes the kubelet):
```bash
# the orchestrator execs the first four on the node, so they have to be there;
# python3-venv is for the SDK in Try it. Refresh the index first, or a package
# that is genuinely missing fails to fetch
sudo apt-get update && sudo apt-get install -y iptables rsync e2fsprogs iproute2 python3-venv
echo 'vm.nr_hugepages=2048' | sudo tee /etc/sysctl.d/90-e2b.conf && sudo sysctl -p /etc/sysctl.d/90-e2b.conf
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.4+k3s1 INSTALL_K3S_EXEC="--node-label e2b.dev/single-node=true --write-kubeconfig-mode 644" sh -
```
Then `export KUBECONFIG=/etc/rancher/k3s/k3s.yaml`, which is what makes the
`kubectl` commands below work as your own user; k3s writes that file mode 600
and owned by root unless it is told otherwise. Mode 644 hands that file, a
cluster-admin credential, to every local user of the node, which is fine on a
machine that is yours alone; on a shared one drop the flag and run every
command as `sudo k3s kubectl` instead.
On a node that already runs Kubernetes the k3s line does not apply, and its
kubeconfig is whatever that cluster already gave you: install the same
packages, set the same sysctl, restart the kubelet
(`sudo systemctl restart k3s` on k3s) and run
`kubectl label node e2b.dev/single-node=true`; the in-pod `host-setup`
writes that sysctl file and value too, so it changes nothing here. Either way
the node has to advertise `hugepages-2Mi: 4Gi` before you go on, or the pod
stays Pending.
Then install Embed:
```bash
kubectl create namespace e2b
kubectl -n e2b create secret generic e2b-api \
--from-literal=ADMIN_TOKEN="$(openssl rand -hex 32)" \
--from-literal=SANDBOX_ACCESS_TOKEN_HASH_SEED="$(openssl rand -hex 32)"
kubectl apply -k "https://github.com/e2b-dev/runtime//embed/kubernetes?ref=main"
kubectl -n e2b rollout status statefulset/e2b --timeout=15m
kubectl -n e2b logs e2b-0 -c ready
```
Ready takes about 3 minutes on `n4-standard-4`: 2 m 38 s from `apply -k` to
Ready in the reference run, 49 s of it the `base` template build. The
15-minute timeout above is a ceiling, not the expectation.
The namespace and the Secret come before `apply -k` so the api never starts
without them. Until the Secret exists the api container cannot be created:
the pod sits at `Init:CreateContainerConfigError` with the event
`secret "e2b-api" not found`, and the kubelet retries with a backoff of up to
five minutes. The kustomization carries the namespace too, so `apply -k`
reports `namespace/e2b configured` and warns that the one you created by hand
has no last-applied annotation; that is kubectl adding its annotation and
nothing else. It also brings a headless Service and the StatefulSet. From a
checkout: `kubectl apply -k embed/kubernetes`.
## Try it
Put this install's three SDK variables and the dashboard URL in your shell:
```bash
eval "$(kubectl -n e2b exec e2b-0 -c ready -- cat /run/e2b/sdk.env)"
```
Open the dashboard at `$E2B_DASHBOARD_URL`, which is the node's IP on port
3001, and paste `$E2B_API_KEY` into its key form. The browser also reaches
sandbox traffic on the node's IP, port 3002, so both have to be open to it.
Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's `pip` refuses to write to. On Ubuntu `venv`
comes from `python3-venv`: install it first, which the apt line in
[Install](#install) does, or this fails with `ensurepip is not available`.
```bash
python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0
```
Then create a sandbox. Save this and run it with `.venv/bin/python`:
```python
from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()
```
There is no `smoke` service here. To run the same check with the JavaScript
SDK, which also reaches a port inside the sandbox through client-proxy, run
the image that carries it on the node. The image name comes out of the same
kustomization the install applied, so this needs no checkout:
```bash
IMG="$(kubectl kustomize "https://github.com/e2b-dev/runtime//embed/kubernetes?ref=main" | awk '/image: .*node-e2b/{print $2; exit}')"
kubectl -n e2b run smoke --rm -i --restart=Never --image="$IMG" --overrides='{"spec":{"hostNetwork":true}}' \
--env E2B_API_URL="$E2B_API_URL" --env E2B_SANDBOX_URL="$E2B_SANDBOX_URL" --env E2B_API_KEY="$E2B_API_KEY" -- node /app/smoke.mjs
```
`--env E2B_API_KEY=...` puts the team key in the pod spec, so for as long as
the pod runs anyone who can read pods in the `e2b` namespace can read the key
out of `kubectl -n e2b get pod smoke -o yaml`, and, on a cluster whose audit
policy records request bodies, it lands in the API server's audit log. `--rm`
removes the pod when the test ends.
## Remove
Three tiers, matching compose's `down`, `down -v` and a purge:
```bash
# 1. the pod; databases, key and node state stay. Wait for it: the data
# directory below must not go while the pod is still running.
kubectl -n e2b delete statefulset e2b && kubectl -n e2b wait --for=delete pod/e2b-0 --timeout=5m
# 2. on the node: the databases and the team key
sudo rm -rf /var/lib/e2b/data
# 3. the Secret and the namespace. kubectl delete -k ""
# --ignore-not-found does the same; without that flag it exits 1 on the
# StatefulSet step 1 already removed, having deleted the namespace anyway
kubectl delete namespace e2b
```
The data directory goes as one piece: the team API key lives in `seed-state`
beside the databases and must not outlive them, or the reverse. Past that, the
node keeps what `host-setup` wrote until it is recycled: the files under
`/etc`, the rest of `/var/lib/e2b` (its `storage` holds the built templates and
their cache, about 2 GiB after one install), `/fc-*`, the hugepage reservation
and the iptables rule.
There is no Kubernetes teardown one-shot; Compose's `host-teardown`
has no counterpart here.
## Reference
### What runs where
One pod on the labelled node. The stores, Vector, the orchestrator, api,
client-proxy, dashboard-api and the dashboard are sidecar init containers
started in the compose order, host preparation first and the dashboard pair
last, so the `base` template never waits on the browser UI; the one-shots are
ordinary init containers between them; `ready` turns the pod Ready once the
`base` template exists and the dashboard answers.
The pod uses the node's network and pid namespaces, runs three privileged
containers (`preflight`, `host-setup` and the orchestrator launcher, which
`nsenter` into the node), mounts the node's `/` for the artifact download and
`/sys/fs/cgroup` for the sandbox sweep, and prepares the node the way the
compose `host-setup` does: kernel modules, sysctls, hugepages, a udev rule, an
iptables rule and directories under `/`. Use a dedicated node. Sandboxes end
when the pod is deleted or restarted.
The databases and the team API key are `hostPath` directories under
`/var/lib/e2b/data` on that node, which is why the manifest pins the pod there
with a `nodeSelector`; a pod started anywhere else would begin with an empty
stack.
### Ports
The node exposes the same thirteen ports as a compose host, and Postgres,
Redis, ClickHouse and Vector listen on loopback here too, as does the
collector under Telemetry once it is patched in. The SDK variables
`ready` prints use the node's IP: reach 3000, 3001 and 3002 on it and firewall
the other ten (3003, 3010, 5007, 5008, 5009, 5109 and the sandbox egress
proxies 5010, 5016, 5017 and 5018), since 5008 is an unauthenticated control
API and the egress proxies expect no external client.
k3s adds two ports of its own on top of those thirteen: `*:6443`, the
Kubernetes API server, and `*:10250`, the kubelet. They belong to the
cluster rather than to the stack, and they want the same firewall treatment
as the ten, more urgently if anything: the install above writes the
cluster-admin kubeconfig mode 644, so 6443 is the port that hands out the
node.
Vector's log listener is the one port that differs. It binds
`127.0.0.1:20006` rather than compose's 30006, because 30000 to 32767 is
Kubernetes' default NodePort range and a NodePort allocated 30006 would DNAT
the pod's own loopback log traffic away from Vector.
### Secrets
The api's two secrets, `ADMIN_TOKEN` and `SANDBOX_ACCESS_TOKEN_HASH_SEED`,
come from the `e2b-api` Secret created during the install; nothing in the
manifest carries them. Compose generates them per install in an
`api-secrets` one-shot, which has no counterpart here: rotate them by
replacing the Secret and restarting the pod. dashboard-api takes `ADMIN_TOKEN`
from the same Secret.
The team API key is different: the seed generates it on the first start and
keeps it in `/var/lib/e2b/data/seed-state/team-api-key` on the node, where
`ready` reads it. To rotate it, remove the file on the node and restart the
pod:
```bash
sudo rm /var/lib/e2b/data/seed-state/team-api-key # on the node
kubectl -n e2b delete pod e2b-0
```
The seed generates a new key, revokes the ones it issued earlier, and `ready`
prints the new exports. To pin a key instead of generating one, patch the seed
init container's `SEED_TEAM_API_KEY` with a kustomize patch; the value is
`e2b_` followed by at least 32 hex characters, as compose's `TEAM_API_KEY`
documents.
The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the node serves plain http; sign-out
clears it.
### Telemetry
api, the orchestrator, client-proxy and dashboard-api take their OpenTelemetry
collector from the `OTEL_COLLECTOR_GRPC_ENDPOINT` literal of the
`e2b-settings` ConfigMap in [`kustomization.yaml`](kustomization.yaml). It
ships empty, which exports nothing; `127.0.0.1:4317`, or any `host:port` the
node can reach, sends their metrics, traces and logs there over OTLP/gRPC.
The built-in collector, the compose `otel` profile's service, is the one
commented line at the end of that file:
```yaml
# patches: [{ path: patches/otel-collector.yaml, target: { kind: StatefulSet, name: e2b } }]
```
From a checkout, set the literal to `127.0.0.1:4317`, uncomment that line and
run `kubectl apply -k embed/kubernetes`.
[`patches/otel-collector.yaml`](patches/otel-collector.yaml) appends an
`otel-collector` sidecar after the dashboard pair, reading
[`config/otel-collector.yaml`](config/otel-collector.yaml) from the
`otel-config` ConfigMap, and `curl -s 127.0.0.1:13133` on the node answers
while it runs. Unlike compose, where nothing waits for it, its startup probe
comes before `ready`, so a collector that never answers leaves the pod
unready. With the dashboard patch under Limitations as well, both entries go
in the one `patches:` list.
The URL install has no file to edit. An overlay of your own does the same: a
directory holding this `kustomization.yaml` and a copy of the patch file
(`curl -fsSLO https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/kubernetes/patches/otel-collector.yaml`),
applied with `kubectl apply -k` on that directory.
```yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: e2b
resources:
- https://github.com/e2b-dev/runtime//embed/kubernetes?ref=main
configMapGenerator:
- name: e2b-settings
behavior: merge
literals:
- OTEL_COLLECTOR_GRPC_ENDPOINT=127.0.0.1:4317
patches:
- path: otel-collector.yaml
target: { kind: StatefulSet, name: e2b }
```
For a collector of your own, give its address instead and leave out
`patches:`. The reference's
[Observability](../docs/REFERENCE.md#observability) says what the built-in
collector keeps and what reads it.
The built-in collector discards traces and logs. To forward them from the
overlay, copy its config in as well
(`curl -fsSL --create-dirs -o config/otel-collector.yaml https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/kubernetes/config/otel-collector.yaml`),
make the edit its comments describe, and replace the `otel-config` ConfigMap
with that copy through one more entry under `configMapGenerator:`:
```yaml
- name: otel-config
behavior: replace
files:
- config.yaml=config/otel-collector.yaml
```
### Upgrading
The generated ConfigMaps carry a content hash in their names, so changing a
pin in [`kustomization.yaml`](kustomization.yaml) or a file under
[`config/`](config/) and re-applying rolls the pod onto the new map. The
previous maps are left behind, because kustomize does not prune; delete them
by name if they bother you. A rolling update waits for the current pod to be
Ready, so a pod that is not (a crash-looping container, a Pending pod) adopts
nothing from a re-apply until you delete it: `kubectl -n e2b delete pod
e2b-0`, and the StatefulSet recreates it from the new template.
### Hugepages
`HUGEPAGES`, given to `host-setup` in
[`statefulset.yaml`](statefulset.yaml), and the orchestrator's `hugepages-2Mi`
request and limit are one number and have to move together: 2048 pages of
2 MiB is `4Gi`. [`../tests/kubernetes.bats`](../tests/kubernetes.bats) fails
when they disagree.
### Limitations
- Template builds with `copy()` steps upload through the orchestrator's port
5008 (`LOCAL_UPLOAD_BASE_URL`), the same as compose from another machine.
Build from the node (a `hostNetwork` pod, or `kubectl exec`), or tunnel 5008
to your loopback (`ssh -N -L 5008:127.0.0.1:5008 @`) so the
handed-out `http://127.0.0.1:5008/...` URL is valid unchanged. Do not open
5008 to the network.
- `env/api.local.env` has no counterpart here. To add api environment, patch
the api container with a kustomize patch.
- There is no `smoke` service. Try it above has the recipe that replaces it.
- A node that carries a taint needs a toleration, added with a kustomize
patch; a commented example is in [`statefulset.yaml`](statefulset.yaml).
- To run without the dashboard, drop both of its containers with a
strategic-merge patch beside [`kustomization.yaml`](kustomization.yaml),
referenced from it as `patches: [{ path: no-dashboard.yaml }]`; the rest of
the stack runs.
```yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: e2b
spec:
template:
spec:
initContainers:
- name: dashboard-api
$patch: delete
- name: dashboard
$patch: delete
```
### Troubleshooting
- **The pod stays Pending with `Insufficient hugepages-2Mi`.** The node is not
advertising the reservation. `kubectl get node -o
jsonpath='{.items[0].status.allocatable.hugepages-2Mi}'` must print `4Gi`;
the kubelet advertises only the hugepages it saw when it started, so set the
sysctl and restart it.
- **The pod sits at `Init:CreateContainerConfigError`, with the event
`secret "e2b-api" not found`.** The `e2b-api` Secret does not exist, so the
kubelet cannot create the api container. Create it as the Install section
shows; the kubelet retries with a backoff of up to five minutes, so the pod
recovers on its own once the Secret is there.
- **The k3s install fails with `SSL certificate problem` or `Download failed`
before anything E2B runs.** The unpinned installer asks k3s's channel
server for the current `stable` release; when that service is down the
installer treats the word `stable` as a version and fails. The install line
above pins `INSTALL_K3S_VERSION`, which skips the channel lookup; keep the
pin, or set it to the release you want.
- **`kubectl` says permission denied on `/etc/rancher/k3s/k3s.yaml`.** k3s
wrote the kubeconfig mode 600 and owned by root, which is its default
without `--write-kubeconfig-mode 644`. Either
`sudo chmod 644 /etc/rancher/k3s/k3s.yaml` once and
`export KUBECONFIG=/etc/rancher/k3s/k3s.yaml`, or run every command as
`sudo k3s kubectl` instead.
---
### Overview (`firecracker/fc-busybox/README.md`)
# fc-busybox
BusyBox static builds for [E2B](https://e2b.dev) sandbox VMs.
## What this builds
A BusyBox binary with default config (all standard applets including ash shell), statically linked with musl libc for both **amd64** and **arm64**.
The sandbox init script uses `#!/usr/bin/busybox ash`, so ash and the full default applet set must be present.
## Build
```bash
sudo ./build.sh # host arch
sudo TARGET_ARCH=arm64 ./build.sh # arm64
sudo BUSYBOX_VERSION=1.37.0 ./build.sh # custom version
```
Output: `builds/{version}/{arch}/busybox`
---
### Overview (`firecracker/fc-kernels/README.md`)
# fc-kernels
## Overview
This project builds custom Linux kernels for Firecracker microVMs from the same kernel sources as the official Firecracker repo, using the configuration files (and optional patches) that live in this repo.
## Prerequisites
- Linux environment (for building kernels)
## Building locally
1. **Configure kernel versions:**
- Edit `kernel_versions.txt` to specify which kernel versions to build (one per line, e.g. `6.1.158`).
- Place the corresponding config(s) in `configs/x86_64/.config` and `configs/arm64/.config`.
- (Optional) Drop `*.patch` files into `patches//` to apply on top of the upstream tree before build.
2. **Build:**
```sh
make build # builds all versions in kernel_versions.txt for x86_64
make build-arm64 # same, for arm64
./build.sh 6.1.158 # build a single version (x86_64)
./build.sh 6.1.158 arm64
```
Output: `builds/vmlinux-//vmlinux.bin` where `` is `amd64` or `arm64` (Go/OCI convention). For x86_64 a legacy copy is also placed at `builds/vmlinux-/vmlinux.bin`.
## Adding a patch release
A version is built from the newest `microvm-kernel*-*.amzn*` tag of [amazonlinux/linux](https://github.com/amazonlinux/linux), so that is where the available patch releases come from. Match the prefix loosely: upstream puts the series there on some tags (`microvm-kernel-6.1.158-15.288.amzn2023`, but `microvm-kernel6.18-6.18.36-71.136.amzn2023`), and a pattern that assumes the hyphen quietly hides the newest releases of a series.
```sh
git ls-remote --tags https://github.com/amazonlinux/linux | grep -E 'microvm-kernel[^ ]*-6\.1\.[0-9]+-'
```
A new version needs its own config, seeded from the version it supersedes and then settled against the tree it will be built from:
```sh
for arch in x86_64 arm64; do cp configs/$arch/6.1.158.config configs/$arch/6.1.177.config; done
cp -r patches/6.1.158 patches/6.1.177 # only if that version carries patches
sudo env CONFIG_ONLY=1 ./build.sh 6.1.177 # root: build.sh installs its build dependencies
sudo env CONFIG_ONLY=1 ./build.sh 6.1.177 arm64
sudo chown -R "$(id -u):$(id -g)" configs # the resolved configs come back root-owned
```
`CONFIG_ONLY=1` stops after `olddefconfig` and writes the resolved config back to `configs//.config`, so the symbols the newer tree added are recorded and reviewable rather than silently taking upstream defaults at build time. It applies `patches//` on the way, which is where a carried patch that no longer applies shows up, and it sets `CONFIG_BUILD_SALT` from the tag being built, which `olddefconfig` leaves alone: a seeded config would otherwise ship the version it was seeded from. Run it per architecture on that architecture's machine: Kconfig probes the compiler, so a config resolved cross is not the one a native build resolves.
Then add the version to `kernel_versions.txt` and build.
`make build` builds every version in the file, so keep the list to the versions you still want built. Dropping a pin is safe: published artifacts are never overwritten or deleted, and a template records the kernel version it was built with.
## Flavours
A flavour is the same upstream tree and patches built under a different config, named `-`: `6.1.158-android` is `6.1.158` with binder, DMA-BUF heaps, Android's networking options, a small patch that answers Android's ashmem ioctls on memfd files instead of an ashmem driver, and PSI (pressure stall information) on by default, so a sandbox can run Android in a container. The other configs build PSI in but leave it off unless the guest cmdline says `psi=1`; Android's low-memory killer daemon needs `/proc/pressure`, and without it the UI never comes up. `build.sh` takes the upstream tag and the `patches/` directory from the stem (`6.1.158`) and everything else — the config, the output directory, and so the published name a template selects — from the full name. Seed a flavour's config from its stem's and settle it the same way as a new version (`sudo env CONFIG_ONLY=1 ./build.sh 6.1.158-android`, per architecture). A flavour is a sibling of its stem, not a patch release: the weekly bump neither seeds from it nor bumps it. A flavour may also carry patches of its own under `patches/-/`; they apply after the stem's, and a change there rebuilds only that flavour.
## New kernel in E2B's infra
_Note: these steps should give you a new kernel on your self-hosted E2B using https://github.com/e2b-dev/infra_
- Build the kernel from the new config or patch.
- Update `DefaultKernelVersion` in [packages/shared/pkg/featureflags/flags.go](https://github.com/e2b-dev/infra/blob/main/packages/shared/pkg/featureflags/flags.go) if you changed the kernel version. It is the fallback: `DEFAULT_KERNEL_VERSION` in the environment overrides it, and a `build-kernel-version` feature flag overrides both.
- Build and deploy `api`.
## Architecture naming
Output directories use Go's `runtime.GOARCH` convention (`amd64`, `arm64`) so they match the infra orchestrator's `TargetArch()` path resolution. The build-time variable `TARGET_ARCH` (`x86_64`, `arm64`) is only used internally for config paths and cross-compilation flags.
## License
This project is licensed under the Apache License 2.0. See [LICENSE](../../LICENSE) for details.
---
### Overview (`firecracker/fc-versions/README.md`)
# fc-versions
## Overview
This project builds custom Firecracker versions for E2B sandbox VMs.
## Prerequisites
- Linux environment (for building firecracker)
- `FIRECRACKER_REPO_TOKEN` set, to clone the Firecracker source
## Build
```sh
./build.sh [arch]
```
`arch` is `amd64` (the default) or `arm64`. Run the script from this directory. It resolves its output through a relative path.
Output: `builds///firecracker`
## Releases
A release is named by the tag it is built from, `vX.Y-`: the
upstream minor line we track, then our own version of the patches carried on
top of it (for example `v1.14-0.1.0`). The tag is cut by hand on the fork
before the release is built — the pipeline resolves it and refuses anything
else, so it never composes a release name of its own.
Releases named `_` predate this scheme and are never rebuilt.
## License
This project is licensed under the Apache License 2.0. See [LICENSE](../../LICENSE) for details.
---
### Overview (`packages/envd/README.md`)
# envd
Daemon that runs inside a sandbox that allows interacting with the sandbox via calls from the SDK.
## Development
### Versioning
The envd version in `pkg/version.go` is maintained by release-please from the Conventional Commits merged since the last envd release; do not edit it by hand. A change that affects behavior (code, dependencies) ships under a `feat` or `fix` commit type, so that a version the orchestrator can gate on is cut. Pure comment or documentation changes use `docs`, a type the changelog hides, and cut no release; `chore` and `refactor` are visible in it and do cut one.
### Running locally
Run the following command to (re)build the envd daemon and start a Docker container with envd running inside:
```bash
make start-docker
```
You can use E2B SDKs with env var `E2B_DEBUG=true` or with a debug parameter set to `true` when creating or connecting to a sandbox, to connect to the envd started with this command.
### Generating API server stubs
Run the following command to install the necessary dependencies for generating server stubs:
```bash
make init-generate
```
#### Generating
After changing the API specs in `./spec/` run the following command to generate the server stubs:
```bash
make generate
```
### Debugging
-
-
Run `make run-debug` and then connect to the port `2345` with a debugger or use the VSCode run/debug and run the "Debug envd" to build the envd, Docker, and start the debugging.
---
### Overview (`packages/nomad-nodepool-apm/README.md`)
# Nomad Node Pool Autoscaler Plugins
Custom plugins for the [Nomad Autoscaler](https://github.com/hashicorp/nomad-autoscaler) that keep a service-job allocation count aligned with the number of nodes in a Nomad node pool.
## Purpose
The package builds two plugins because Nomad Autoscaler supports only one plugin type per external binary:
- `nomad-nodepool-apm` reports the number of ready nodes in a pool.
- `nomad-deployment-aware-target` fails an active deployment before applying the requested task-group count.
Together they let service jobs replicate system-job placement while retaining rolling updates. The deployment-aware target is used by the `template-manager` and `orchestrator-ee` service jobs. When the task-group count must change, scaling intentionally abandons a conflicting in-progress rollout (Nomad rejects scaling while a deployment is active); Nomad records that deployment as `failed`, rather than `cancelled`. The rollout spawned by the scale itself is left to run, and no deployment is touched when the count already matches. The target requires `auto_revert = false` on the scaled group (otherwise failing a deployment would restore an older job version and ping-pong); it refuses to scale groups with `auto_revert = true` and returns an error on every attempt.
## Usage
### Building
```bash
make build
```
### Uploading to GCS
```bash
GCP_PROJECT_ID=your-project-id make build-and-upload
```
### Configuration
In your Nomad Autoscaler configuration:
```hcl
apm "nomad-nodepool-apm" {
driver = "nomad-nodepool-apm"
config = {
nomad_address = "http://localhost:4646" # Optional, uses NOMAD_ADDR env var
nomad_token = "your-token" # Optional, uses NOMAD_TOKEN env var
nomad_region = "global" # Optional
}
}
target "nomad-deployment-aware-target" {
driver = "nomad-deployment-aware-target"
config = {
nomad_address = "http://localhost:4646"
nomad_token = "your-token"
nomad_region = "global"
}
}
```
### In Job Scaling Policy
```hcl
scaling {
enabled = true
min = 1
max = 50
policy {
evaluation_interval = "30s"
cooldown = "2m"
target "nomad-deployment-aware-target" {}
check "match_node_count" {
source = "nomad-nodepool-apm"
query = "orchestrator" # Node pool name
strategy "pass-through" {} # Use node count directly as desired count
}
}
}
```
## How It Works
1. The plugin queries the Nomad API to list nodes filtered by the specified node pool
2. It counts only nodes that are:
- Status: `ready`
3. Returns the count as a metric for the autoscaler to use
4. With the `pass-through` strategy, this count becomes the desired number of allocations
5. The target serializes scaling per namespaced job and is a no-op when the durable count already matches
6. When the count must change it fails a conflicting active deployment, rereads the job, and scales with the current job modify index
7. It verifies the final durable count; the rollout spawned by the scale proceeds normally
8. Concurrent job or deployment changes are retried from fresh state with a bounded attempt count
Dry-run actions do not fail deployments or write task-group counts.
## Query Format
The `query` parameter should be the name of the node pool you want to count nodes from.
Example: `query = "orchestrator"` counts all ready nodes in the `orchestrator` node pool.
## Configuration Options
| Option | Description | Default |
|--------|-------------|---------|
| `nomad_address` | Nomad server address | `NOMAD_ADDR` env var or `http://127.0.0.1:4646` |
| `nomad_token` | Nomad ACL token | `NOMAD_TOKEN` env var |
| `nomad_region` | Nomad region | Default region |
| `nomad_namespace` | Nomad namespace | Default namespace |
---
### Overview (`packages/orchestrator/README.md`)
# Orchestrator
## Commands
**Prerequisite:** Enable NBD module first:
```bash
modprobe nbd nbds_max=4096
cat </etc/udev/rules.d/97-nbd-device.rules
# Disable inotify watching of change events for NBD devices
ACTION=="add|change", KERNEL=="nbd*", OPTIONS:="nowatch"
EOH
udevadm control --reload-rules
udevadm trigger
```
**Prerequisite:** Allocate enough 2MB HugeTLB pages (hugepages) for testing:
```bash
echo 1024 | sudo tee /proc/sys/vm/nr_hugepages # for 2GB hugepages, adjust as needed
```
### Create Build
Create a new build.
```bash
sudo go run ./cmd/create-build -to-build -storage .local-build
```
Flags:
- `-to-build ` - Output build ID (UUID, required)
- `-from-build ` - Base build ID for incremental builds
- `-template ` - Template ID (default: `local-template`)
- `-storage ` - Local path or `gs://bucket` (enables local mode with auto-download of kernel/FC)
- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)
- `-kernel ` - Kernel version (default: `vmlinux-6.1.158`)
- `-firecracker ` - Firecracker version (default: `v1.14.1_458ca91`)
- `-vcpu ` - vCPUs (default: `1`)
- `-memory ` - Memory in MB (default: `512`)
- `-disk ` - Disk in MB (default: `1000`)
- `-hugepages` - Use 2MB huge pages (default: `true`, set `false` for 4KB pages)
- `-start-cmd ` - Start command
- `-ready-cmd ` - Ready check command
> You can use the `$(uuidgen)` command to generate a random UUID.
>
> If you are using Mise, you can run the command as `sudo $(which go) run ./cmd/create-build -to-build -storage .local-build`.
### Resume Build
Resume a sandbox from a build.
```bash
# Local storage
sudo go run ./cmd/resume-build -from-build -storage .local-build -iterations 10
# Remote GCS storage (pass credentials to sudo)
sudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \
go run ./cmd/resume-build -from-build -storage gs://bucket -iterations 10
# Pause mode: resume, run command, then snapshot
sudo go run ./cmd/resume-build -from-build -to-build \
-storage .local-build -cmd-pause "pip install numpy"
# Pause mode: start command through envd, then wait for SIGUSR1 before snapshot
sudo go run ./cmd/resume-build -from-build -to-build \
-storage .local-build -cmd-signal-pause "python3 /home/user/workspace/job.py"
```
Flags:
- `-from-build ` - Build ID (UUID) to resume from (required)
- `-to-build ` - Output build ID (UUID) for pause snapshot (auto-generated if not specified)
- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)
- `-sandbox-dir ` - Override `SANDBOX_DIR` (the rootfs path baked into the snapshot)
- `-iterations ` - Number of iterations, 0 = interactive (default: `0`)
- `-cold` - Clear cache between iterations (cold start each time)
- `-no-prefetch` - Disable memory prefetching
- `-v` - Verbose logging
- `-pause` - Start and immediately pause (create snapshot)
- `-signal-pause ` - Wait for signal before pause (e.g., `SIGTERM`, `SIGUSR1`)
- `-cmd-pause ` - Execute command in sandbox, then pause on success
- `-cmd-signal-pause ` - Execute command in sandbox, then wait for `SIGUSR1` before pause
- `-envd-version ` - The guest's envd version (default: a placeholder — see the caveat below)
**envd version caveat:**
`resume-build` has no database and the snapshot metadata does not record the build's envd
version (only the kernel and Firecracker ones), so unless `-envd-version` is given the tool
reports a fixed placeholder (`1.0.0`). Every `envd.version` field in its output and logs is
then that placeholder, **not** the guest's version — don't reason from it. To read the real
one, mount the build's rootfs (`mount-build-rootfs`) and run its `/usr/bin/envd -version`,
or read `X-Envd-Version`, which a running envd reports on `/init`.
The placeholder sits above every envd version gate, so the gated capabilities (`/freeze`,
`/fsfreeze`, `/collapse`, the KVM clock, the upgrade `from_version`) stay on. Against an
older guest that is a path production would not take, so pass `-envd-version` with the
guest's real version when the behavior under test depends on one of those gates. The tool
prints which of the two it is using when it loads the build.
**Pause mode example:**
```bash
# Chain builds with explicit build IDs
sudo go run ./cmd/resume-build -from-build $BUILD1 -to-build $BUILD2 \
-storage .local-build -cmd-pause "apt install curl"
sudo go run ./cmd/resume-build -from-build $BUILD2 -to-build $BUILD3 \
-storage .local-build -cmd-pause "pip install requests"
# Start a long-running command through envd, then trigger pause from the host
sudo go run ./cmd/resume-build -from-build $BUILD3 -to-build $BUILD4 \
-storage .local-build -cmd-signal-pause "python3 /home/user/workspace/job.py"
# In another shell:
ps -ef | grep 'cmd/resume-build' | grep -v grep
sudo kill -SIGUSR1
```
### Copy Build
Copy a build between storage locations (local or GCS).
```bash
# Local to local
go run ./cmd/copy-build -build -from .local-build -to /other/path
# Local to GCS
go run ./cmd/copy-build -build -from .local-build -to gs://bucket
# GCS to GCS
go run ./cmd/copy-build -build -from gs://bucket1 -to gs://bucket2
```
Flags:
- `-build ` - Build ID (UUID, required)
- `-from ` - Source: local path or `gs://bucket`
- `-to ` - Destination: local path or `gs://bucket`
- `-team ` - Team UUID (if set, prints SQL to populate DB on stdout)
- `-envd-version ` - Envd version (required if team provided) — must match the version present in the template
- `-vcpu ` - vCPUs (default: `2`)
- `-memory ` - Memory in MB (default: `1024`)
- `-disk ` - Disk in MB (default: `1024`)
- `-tag ` - Build assignment tag (default: `default`)
**Print DB seed SQL:**
When `-team` is set, the command reads `metadata.json` from the destination to get `kernel_version` and `firecracker_version`, generates a new env ID, and prints SQL statements (`BEGIN`/`COMMIT` wrapped) to stdout. Pipe directly to `psql` to populate the database:
```bash
go run ./cmd/copy-build -build -from .local-build -to gs://bucket \
-team -envd-version 0.2.11 -vcpu 2 -memory 1024 -disk 1024 \
| psql $POSTGRES_CONNECTION_STRING
```
### Mount Build Rootfs
```bash
# Local storage
sudo go run ./cmd/mount-build-rootfs -build -storage .local-build -mount /mnt/rootfs
# Remote GCS storage (pass credentials to sudo)
sudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \
go run ./cmd/mount-build-rootfs -build -storage gs://bucket -mount /mnt/rootfs
```
Flags:
- `-build ` - Build ID (UUID, required unless `-empty`)
- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)
- `-mount ` - Mount path
- `-verify` - Verify rootfs integrity (requires `-mount`)
- `-log` - Enable logging
- `-empty` - Create an empty rootfs instead of loading a build
- `-size ` - Size of empty rootfs (default: 1GB)
- `-block-size ` - Block size (default: 4096)
### Inspect Build
Inspect build artifacts (headers and data blocks).
```bash
# Inspect memfile header (default)
go run ./cmd/inspect-build -build -storage .local-build
# Inspect rootfs header
go run ./cmd/inspect-build -build -storage .local-build -rootfs
# Inspect header + data blocks
go run ./cmd/inspect-build -build -storage .local-build -data
# Inspect first 100 data blocks
go run ./cmd/inspect-build -build -storage .local-build -data -end 100
# Remote GCS storage
go run ./cmd/inspect-build -build -storage gs://bucket -rootfs -data
```
Flags:
- `-build ` - Build ID (UUID, required)
- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)
- `-memfile` - Inspect memfile artifact (default)
- `-rootfs` - Inspect rootfs artifact
- `-data` - Also inspect data blocks (not just header)
- `-start ` - Start block index (default: `0`, only with `-data`)
- `-end ` - End block index (default: all, only with `-data`)
### Show Build Diff
Show the diff between two builds and preview merge result.
```bash
# Memfile (default)
go run ./cmd/show-build-diff -from-build -to-build -storage .local-build
# Rootfs
go run ./cmd/show-build-diff -from-build -to-build -storage .local-build -rootfs
# With visualization
go run ./cmd/show-build-diff -from-build -to-build -storage gs://bucket -visualize
```
Flags:
- `-from-build ` - Base build ID (required)
- `-to-build ` - Diff build ID (required)
- `-storage ` - Local path or `gs://bucket` (default: `.local-build`)
- `-memfile` - Inspect memfile artifact (default)
- `-rootfs` - Inspect rootfs artifact
- `-visualize` - Visualize the headers
---
## Architecture (ARM64) Support
The orchestrator supports both `amd64` (x86_64) and `arm64` (aarch64) architectures. Architecture is detected automatically via `runtime.GOARCH` at compile time.
### Architecture naming convention
This project uses **Go/Docker/Debian naming** (`amd64`/`arm64`) for architecture directories in binary paths and GCS buckets:
| Convention | x86_64 name | ARM64 name | Used by |
|------------|-------------|------------|---------|
| **Go/Docker/Debian** | `amd64` | `arm64` | This repo, Docker, `dpkg --print-architecture` |
| Linux/GNU | `x86_64` | `aarch64` | `uname -m`, kernel Makefiles |
Binary paths follow the `{version}/{arch}/` layout:
```
# Firecracker (GCS bucket or FIRECRACKER_VERSIONS_DIR)
fc-versions/v1.12.1_210cbac/amd64/firecracker
fc-versions/v1.12.1_210cbac/arm64/firecracker
# Kernels (GCS bucket or HOST_KERNELS_DIR)
kernels/vmlinux-6.1.158/amd64/vmlinux.bin
kernels/vmlinux-6.1.158/arm64/vmlinux.bin
```
> **Note:** The [fc-kernels](https://github.com/e2b-dev/fc-kernels) repo currently uses `x86_64` instead of `amd64` for its directory names. This will be aligned in a follow-up change.
### ARM64-specific behavior
- **SMT** is disabled (ARM processors don't support simultaneous multi-threading)
- **CPU detection** uses fallback values since `gopsutil` doesn't populate Family/Model on ARM64
- **OCI platform** is set to the target architecture instead of hardcoded `amd64`
- **Busybox binaries** are fetched per architecture by `scripts/fetch-busybox.sh` and selected at runtime from `///busybox`
- **Guest kernel command line** drops the x86-only `i8042.*` and `clocksource=kvm-clock` parameters on arm64; the console stays `ttyS0` (Firecracker exposes a 16550-compatible UART on both architectures)
### Cross-architecture deployment
`TARGET_ARCH` is a **runtime** environment variable that overrides the architecture used for path resolution and OCI image pulls. When unset, defaults to the host architecture (`runtime.GOARCH`).
```bash
# Run orchestrator targeting amd64 paths from an arm64 host
TARGET_ARCH=amd64 ./bin/orchestrator
# Or in .env file (read at runtime)
echo "TARGET_ARCH=amd64" >> .env.local
```
`TARGET_ARCH` affects:
- Firecracker and kernel binary path resolution (`{version}/{arch}/...`)
- OCI image platform for container pulls
It does **not** affect:
- Makefile compilation (see `BUILD_ARCH` below)
- Hardware-dependent runtime behavior (SMT detection, CPU info) which always uses the actual host architecture
### Building for ARM64
`BUILD_ARCH` defaults to `amd64`. To build ARM64 binaries:
```bash
# Single command
BUILD_ARCH=arm64 make build-local
# Or set in .env.local for persistent override
echo "BUILD_ARCH=arm64" >> .env.local
```
This applies to all services: orchestrator, envd, client-proxy.
---
## Environment Variables
Automatically set in local mode. Set before running to override:
- `TARGET_ARCH` - Target architecture override (`amd64` or `arm64`; default: host architecture)
- `HOST_ENVD_PATH` - Envd binary path (default: `../envd/bin/envd`)
- `HOST_KERNELS_DIR` - Kernel versions dir (local: `{storage}/kernels`, prod: `/fc-kernels`)
- `FIRECRACKER_VERSIONS_DIR` - Firecracker versions dir (local: `{storage}/fc-versions`, prod: `/fc-versions`)
- `ORCHESTRATOR_BASE_PATH` - Base orchestrator data (local: `{storage}/orchestrator`, prod: `/orchestrator`)
- `SANDBOX_DIR` - Sandbox working dir (default: `/fc-vm`)
## Limitations
- Custom template builds support base images from the declared distro families: **Debian/Ubuntu** (apt), the **RHEL family** — Fedora, CentOS Stream, Rocky, Alma — (dnf/microdnf/yum), **Arch** (pacman), and **Alpine** (apk, OpenRC). The distro is resolved from the image's `/etc/os-release` `ID` — never by probing for package managers. Images without an os-release identity (distroless, scratch) or from an undeclared family are rejected fast, with a build-log error naming the reason and the supported families (see `pkg/template/build/phases/base/distro`). Minimal/restricted-repo images may fail provisioning if their repos don't carry the required packages; the failure is surfaced in the build log.
- Sandboxes always boot the kernel E2B supplies, never one from the base image: `/lib/modules` is empty, no kernel modules can be loaded, and SELinux is disabled on the kernel command line (`selinux=0`), so a base image that ships an SELinux policy boots without it. `kernel-devel` from a distro's repos resolves against a kernel that isn't running. **RHEL** (`rhel`, incl. UBI), **Oracle Linux** (`ol`) and **Amazon Linux** (`amzn`) are therefore not accepted even though they are RPM/dnf images — they are chosen for kABI, signed kernel modules and UEK, which this cannot honour. The community rebuilds above are supported because they are chosen for the userland.
---
### Overview (`tests/integration/README.md`)
# Integration Tests
Package for defining integration tests. Currently, there is a setup for API and Orchestrator testing.
## Run locally
1. Configure test variables in a root `.env.` file and select it with
`make switch-env ENV=` from the repository root. Use
[`.env.local`](.env.local) as a reference for local API values;
`TESTS_ORCHESTRATOR_HOST` must point to the orchestrator for tests that call it directly.
2. If you made changes to the `api` or `envd` protobuf spec, run `make generate` from this folder (and don't forget to generate it in `envd` if changes apply there too).
3. If the orchestrator is not reachable directly, tunnel its gRPC port (`TESTS_ORCHESTRATOR_HOST`, e.g. `localhost:5008`) to a sandbox node before running the tests
4. Run `make test` in this folder or `make test-integration` from the repository root.
Narrow the run with `make test/`, e.g. `make test/api/templates`
or `make test/api/templates/build_template_test.go:TestTemplateBuildCOPY`.
## What CI runs
The `uncompressed` config runs the whole suite. The compressed configs (`zstd1`,
`lz4`) re-run only `scripts/compression-tests.tsv`, the explicit allow-list of
tests whose subject is writing or reading back a snapshot; that file documents
the inclusion criteria and the code paths the compression knobs reach. Select it
locally with `TESTS_ONLY=scripts/compression-tests.tsv make test`. An entry that
no longer matches a real test is an error, so the list cannot rot into silently
reduced coverage.
## Usage of clients (api, orchestrator, envd)
All tests are in the folder internal/tests. You can see the usage of different clients in the tests. Here are just basics.
### API
HTTP client. In order to pass the API key, use the `setup.WithAPIKey()` option.
```go
client := setup.GetAPIClient()
sbxTimeout := int32(60)
resp, err := client.PostSandboxesWithResponse(ctx, api.NewSandbox{
Timeout: &sbxTimeout,
}, setup.WithAPIKey())
```
### Orchestrator
GRPC client. There is no authentication needed as it runs behind API in production.
```go
client := setup.GetOrchestratorClient(t, ctx)
resp, err := client.List(ctx, &emptypb.Empty{})
```
### Envd
Envd client is used for interacting with the sandbox.
There are two API types—HTTP and GRPC.
Each of them provides different methods for interacting with the sandbox; you need to check which ones you need.
#### HTTP
`setup.WithSandbox(...)` sets the sandbox URL and sends the envd access token when the sandbox has one. Pass the sandbox response to authenticate envd requests.
```go
client := setup.GetEnvdClient(t, ctx)
resp, err := client.HTTPClient.PostFilesWithBodyWithResponse(
ctx,
...,
setup.WithSandbox(t, sbx.JSON201),
)
```
To intentionally omit the envd access token, use the tokenless helper:
```go
setup.WithInsecureSandbox(t, sbx.JSON201.SandboxID)
```
#### GRPC
`setup.SetSandboxHeader(...)` sets the sandbox URL and sends the envd access token when the sandbox has one.
For gRPC requests that intentionally omit the token, use `setup.SetInsecureSandboxHeader(...)`.
All methods also expect a user (`user`/`root`) to be set in the header.
You can achieve it using `setup.SetUserHeader(...)`.
```go
client := setup.GetEnvdClient(t, ctx)
req := connect.NewRequest(&filesystem.ListDirRequest{
Path: "/",
})
setup.SetSandboxHeader(t, req.Header(), sbx.JSON201)
setup.SetUserHeader(req.Header(), "user")
resp, err := client.FilesystemClient.ListDir(ctx, req)
```
---
### Overview (`embed/compose/env/README.md`)
This directory exists for the optional, git-ignored `api.local.env` override; see `compose.yaml`.
---
### Overview (`embed/terraform/aws/README.md`)
# E2B Embed on one EC2 instance with Terraform
Creates one Ubuntu 24.04 instance with nested virtualization, in an Auto
Scaling group of one, which installs Docker, writes the two files Embed ships
and runs `docker compose up -d --wait`. It is a single-machine package
rather than a deployment pattern; the hub is
[`../../README.md`](../../README.md).
## Requirements
- Terraform 1.7.5 or newer.
- An AWS account and credentials the AWS provider can read (an SSO profile
in `AWS_PROFILE`, or any other source it supports), allowed to create a
VPC and its network, an Elastic IP, a security group, an IAM role and
instance profile, an S3 bucket, a launch template and an Auto Scaling
group; to pass the instance role to EC2 (`iam:PassRole`); to launch from
the template (`ec2:RunInstances` and `ec2:CreateTags`, which Auto Scaling
checks against the caller when it creates the group); to read Canonical's
public Ubuntu SSM parameter (`ssm:GetParameter`) when `ami` is empty; and,
in an account that has never had an Auto Scaling group, to create Auto
Scaling's service-linked role (`iam:CreateServiceLinkedRole`).
- The root volume is encrypted with the account's default EBS key. If that
default is a customer-managed KMS key, its key policy has to let Auto
Scaling's service-linked role use it, or every launch fails with
`Client.InternalError`.
- A region that offers a nested-virtualization instance type, `m8i.xlarge`
by default. The module puts the instance in the first of the region's own
zones (not a Local or Wavelength Zone) that offers the type.
- The AWS CLI v2 and its Session Manager plugin on your own machine, for the
`session_command` output, the port forwarding under Limitations and the
instance refresh under Upgrading. `terraform apply` needs neither.
- No machine of your own. The module creates it.
- The module source has to be the repository, not a copy of this directory:
it reads `../../compose/compose.yaml` and `../../compose/.env` relative to
itself, which the `github.com/...//` source form below provides by cloning.
A copy of this directory alone fails at plan time, and so does an absolute
local path, which Terraform copies the same way. A checkout of your own
works through a relative `source`, such as `../runtime/embed/terraform/aws`.
## Install
```hcl
provider "aws" {
region = "us-east-1"
}
module "e2b" {
source = "github.com/e2b-dev/runtime//embed/terraform/aws?ref="
client_cidrs = ["203.0.113.0/24"] # where your SDK clients connect from
}
output "api_url" { value = module.e2b.api_url }
output "sandbox_url" { value = module.e2b.sandbox_url }
output "dashboard_url" { value = module.e2b.dashboard_url }
output "session_command" { value = module.e2b.session_command }
output "e2b_api_key" {
value = module.e2b.e2b_api_key
sensitive = true
}
```
```bash
terraform init && terraform apply
```
A module's outputs are not the root module's, so the five `output` blocks are
what makes `terraform output` see them; without them the Try it commands below
print nothing.
`?ref=` pins the source to a commit of the public repository's `main`
branch, so an `init` on another day fetches the same `compose.yaml` and `.env`;
upgrading is a deliberate change of that ref. Use `main` itself for the newest
code. The instance answers `GET /health` about three minutes after `apply`
returns, the Docker install and the image pulls behind it. The `base`
template build runs on past that, so the first sandbox can be created about
four minutes after `apply` returns.
## Try it
Put this install's three SDK variables in your shell:
```bash
export E2B_API_URL="$(terraform output -raw api_url)"
export E2B_SANDBOX_URL="$(terraform output -raw sandbox_url)"
export E2B_API_KEY="$(terraform output -raw e2b_api_key)"
```
Open `terraform output -raw dashboard_url` in a browser and paste the key
(`terraform output -raw e2b_api_key`) into its key form. The instance's `.env`
carries `E2B_DASHBOARD_HOST=`, so the browser reaches sandbox
traffic at the same address, no tunnel needed.
Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's `pip` refuses to write to. This runs on your own
machine, not on the instance; on Ubuntu install `python3-venv` first with
`sudo apt-get update && sudo apt-get install -y python3-venv`, or it fails
with `ensurepip is not available`.
```bash
python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0
```
Then create a sandbox. Save this and run it with `.venv/bin/python`:
```python
from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()
```
The first attempt can fail with `404: tag 'default' does not exist for
template 'local-dev-team/base'`. That is the `base` build still finishing,
not a broken install: wait a minute and run the snippet again.
`curl -s -H "X-API-Key: $E2B_API_KEY" $E2B_API_URL/v2/templates` lists
`base` from the moment its build starts rather than when it finishes, so a
listing is not the go-ahead for the snippet above. Or run the packaged smoke
test on the instance itself: it does the same with the JavaScript SDK in a
container, then reaches a port inside the sandbox through client-proxy:
```bash
eval "$(terraform output -raw session_command)"
cd /opt/e2b && sudo docker compose --profile test run --rm smoke
```
## Remove
```bash
terraform destroy # the instance, the bucket with its objects, and every other resource the module made
```
## Reference
### What it creates
Twenty-seven resources with one client CIDR, three more for each further
one: a VPC with one public subnet, an internet gateway, a route table and
its default route; an Elastic IP; a security group, a rule per client port
and CIDR, and one egress rule; an IAM role with its instance profile and
its one inline policy, and a 30-second `time_sleep` that gives the new
profile time to reach EC2 (Timings); a private, versioned, encrypted S3
bucket holding the two files; a launch template (`m8i.xlarge`, a 50 GB
encrypted gp3 root volume, nested virtualization on, IMDSv2 only, the
startup script as user data); an Auto Scaling group of one; and the three
secrets under Secrets below. The providers are aws, random and time.
The group replaces the instance when the instance's own watchdog reports it
unhealthy: `GET :3000/health` has failed for ten minutes. A replacement
starts from a fresh volume, so it rebuilds the `base` template and the
templates you built are gone.
### Security group
Three ports, and nothing else reaches the instance from outside: 3000, 3001
and 3002 from `client_cidrs`. There is no port 22 and no health-checker
range: Session Manager reaches the instance through its agent's outbound
connection, and the group hears about the instance's health from the
instance itself. The orchestrator's unauthenticated control port 5008 in
particular stays inside the instance.
### Timings
| Event | Time |
|-------|------|
| a new install's `terraform apply`, between the instance profile and the Auto Scaling group | 30 seconds |
| first boot, until `GET /health` answers | about 3 minutes |
| first boot, until the first `Sandbox.create` succeeds | about 4 minutes |
| a replacement after `/health` starts failing | about 14 minutes: 11 of failing, then a first boot |
| an instance refresh (see Upgrading), until `GET /health` answers on the new instance | about 3 minutes |
| a reboot of the instance | about 1 minute |
Expected on the default `m8i.xlarge` in `us-east-1`, but for the first row,
which is a fixed wait: IAM takes some seconds to make a new instance profile
usable by EC2, and a launch that names it sooner fails with
`Authentication Failure. Launching EC2 instance failed.` If the first launch
still fails that way, run `terraform apply` again.
To follow a boot:
```bash
eval "$(terraform output -raw session_command)"
sudo tail -f /var/log/cloud-init-output.log # the first boot
sudo journalctl -u e2b-embed -f # the stack's up, on every boot
```
The `session_command` output is a shell snippet that looks the current
instance id up, which is why it is used through `eval`. From outside,
`aws ec2 get-console-output --region --instance-id --latest --query Output --output text`
prints the same first-boot log (`` as in the `id=` line under
Limitations).
### Secrets
`ADMIN_TOKEN`, `SANDBOX_ACCESS_TOKEN_HASH_SEED` and the team API key are all
generated per install, here by Terraform itself: the startup script writes
all three into the instance's `.env` before the first `up`, so the pair the
`api-secrets` one-shot generates on the instance is never read. To rotate
either api secret, replace its resource and then replace the instance as
Upgrading describes: `terraform apply -replace=module.e2b.random_bytes.admin_token`
or `-replace=module.e2b.random_bytes.sandbox_access_token_hash_seed` makes a
new launch-template version; a new hash seed ends the tokens of running
`secure` sandboxes. They live in the Terraform state, so treat the state file
and `terraform show` as secret, and they reach the instance through its user
data, which every version of the launch template keeps. Anyone in the
account allowed `ec2:DescribeLaunchTemplateVersions` can read it: that
action cannot be limited to one launch template, and read-only access such
as AWS's `ReadOnlyAccess` managed policy (`ec2:Describe*`) includes it. So
can anyone allowed `ec2:DescribeInstanceAttribute` or
`ec2:GetLaunchTemplateData`, which read it from the instance, and any
process on the host, root or not, through the metadata service. The
metadata service takes IMDSv2 tokens only, one network hop away, so
containers on Docker's bridge network cannot reach it; the host, its
host-network services and a Session Manager shell can. The startup script
does not print them; read the key with `terraform output -raw e2b_api_key`.
Set `team_api_key` to choose the key instead; change it and replace the
instance to rotate it. Keeping the three out of user data, in SSM Parameter
Store parameters only the instance's role can read, would narrow who can see
them; this module does not do that.
The instance's role may read the two files in the bucket, move this
install's Elastic IP to itself, mark its own instance unhealthy, and make
the calls the SSM agent needs for Session Manager and Run Command:
`ssm:UpdateInstanceInformation`, its heartbeat, and the `ssmmessages` and
`ec2messages` actions that carry sessions and commands. Those are on every
resource: the two message services have no resource to name, and the
heartbeat's instance does not exist when the policy is written. The role
reads no Parameter Store parameter and no other S3 object, and it does not
carry AWS's managed `AmazonSSMManagedInstanceCore` policy, which would let
it read any parameter in the account and region by name. Systems Manager
features beyond sessions and Run Command, such as State Manager
associations, Inventory and Patch Manager, are not granted. Anyone with a
shell on the instance, and any process on the host, holds that role.
The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the instance serves plain http; sign-out
clears it.
### Upgrading
A newer ref points at newer files. `terraform apply` uploads them to the
bucket and records a new launch-template version, which the group uses for
the next instance it launches, but it does not replace the running one, and
that instance wrote its `compose.yaml` and `.env` once, on first boot.
Replace it deliberately:
```bash
aws autoscaling start-instance-refresh --region --auto-scaling-group-name --preferences '{"MinHealthyPercentage":0,"MaxHealthyPercentage":100,"InstanceWarmup":300}'
```
`` is the module's `name`, `e2b-embed` unless you set it. The two
percentages matter: the instance holds the Elastic IP, so the refresh must
terminate it before it launches the new one (the group's own maintenance
policy says the same for every replacement). That is a fresh instance, so
the `base` template is rebuilt and the templates you built are gone. The
warm-up only shortens the wait for the refresh's own status, which
otherwise sits out the group's 900-second grace period. Expected:
`aws autoscaling describe-instance-refreshes --region --auto-scaling-group-name --query 'InstanceRefreshes[0].Status'`
reads `Successful` about 5 minutes after the refresh starts, a couple of
minutes after the stack answers on the new instance.
Unlike the GCP module's group, a replacement this group makes on its own,
after the watchdog reports the instance unhealthy, also boots the newest
launch-template version, so it comes up on the newest files. The AMI
follows Canonical's parameter the same way: an apply after Canonical
publishes a new Ubuntu 24.04 image records it in a new launch-template
version, and the next replacement boots it. Set `ami` to pin one.
`compose_base_url` points the first boot at the Compose files of a specific
commit instead of the files the module ships, for example
`https://raw.githubusercontent.com/e2b-dev/runtime//embed/compose`.
### Telemetry
`otel_collector = true` runs the built-in OpenTelemetry collector on the
instance, keeping the sandbox and team metrics behind the dashboard's
monitoring charts in its own ClickHouse, and `otel_collector_grpc_endpoint`
sends the services' metrics, traces and logs to a collector of your own. Set
one or the other: with both, the services export to your endpoint and the
built-in collector receives nothing, so the dashboard's charts stay empty.
Both reach the instance's `.env` at first boot only, so changing either later
takes the replacement under Upgrading; the reference's
[Observability](../../docs/REFERENCE.md#observability) has the rest.
### Limitations
- No persistent data volume. Every replacement starts from a fresh volume,
rebuilds the `base` template and loses the templates you built.
- No load-balancer health check. The instance's own watchdog is what reports
it unhealthy, and the group's EC2 status checks cover an instance that
stops responding altogether. Stopping the stack for more than ten minutes,
with `docker compose down` for example, therefore gets the instance
replaced: stop the watchdog first with
`sudo systemctl stop e2b-embed-watchdog.timer`, and start it again only
once `GET /health` answers, since it counts from the last answer it saw.
- Template builds with `copy()` steps upload through the orchestrator's port
5008, which the security group keeps closed. Run them on the instance, or
forward the four ports through Session Manager and point the SDK at
`http://127.0.0.1:3000` and `http://127.0.0.1:3002`. One session per port:
```bash
for port in 3000 3001 3002 5008; do
eval "$(terraform output -raw session_command) --document-name AWS-StartPortForwardingSession --parameters portNumber=$port,localPortNumber=$port" &
done
```
The four sessions are jobs of your shell and run until you stop them:
`kill %1 %2 %3 %4`, or the numbers `jobs` shows. Or one `ssh` through
Session Manager forwards all four, in the foreground until Ctrl-C. Ubuntu's
image ships EC2 Instance Connect, which accepts a key of your own for 60
seconds, so nothing is installed on the instance and no port 22 is opened;
`ssh` has to connect within that minute, and your credentials need
`ec2-instance-connect:SendSSHPublicKey`:
```bash
id="$(aws autoscaling describe-auto-scaling-groups --region --auto-scaling-group-names --query 'AutoScalingGroups[0].Instances[?LifecycleState==`InService`] | [0].InstanceId' --output text)"
aws ec2-instance-connect send-ssh-public-key --region --instance-id "$id" --instance-os-user ubuntu --ssh-public-key file://~/.ssh/id_ed25519.pub
ssh -i ~/.ssh/id_ed25519 -N -L 3000:127.0.0.1:3000 -L 3001:127.0.0.1:3001 -L 3002:127.0.0.1:3002 -L 5008:127.0.0.1:5008 \
-o IdentitiesOnly=yes -o ProxyCommand="aws ssm start-session --region --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p" ubuntu@"$id"
```
- x86-64 only. Graviton instances offer no nested virtualization. A Graviton
`.metal` instance has a bare `/dev/kvm` and would run the stack on kernel
6.10 or newer (the Compose guide's Requirements say why), but this module
does not build an arm64 instance.
- A public subnet only. The instance needs a public address for its
downloads and for its Elastic IP. In a private subnet it would need a NAT
gateway, and a restricted egress would have to allow the hosts the Compose
guide's Requirements list, `download.docker.com` and the Ubuntu apt mirrors
for the first boot, the EC2, Auto Scaling and S3 APIs the instance calls,
Session Manager's `ssm`, `ssmmessages` and `ec2messages` endpoints, and
DNS to 8.8.8.8 for the sandboxes.
- AWS GovCloud (US) is untested. The module takes the partition of every ARN
from the provider and Canonical publishes the same AMI parameter there, so
it should work with the provider pointed at a GovCloud region.
### Variables
| Input | Default | What it is |
|-------|---------|------------|
| `client_cidrs` | required | the IPv4 CIDRs allowed to reach 3000, 3001 and 3002, as network addresses (`203.0.113.0/24`, `198.51.100.7/32`); at least one, and a browser's has to be its public IPv4 address; each CIDR takes three of the security group's inbound rules, 60 by default, so up to 20 CIDRs unless the account's quota is raised |
| `name` | `e2b-embed` | name prefix for every resource; the IAM role and instance profile, the security group and the Auto Scaling group take it as is, and IAM names are account-wide, so a second install in the same account needs its own `name`, in any region |
| `instance_type` | `m8i.xlarge` | a nested-virtualization type or an x86-64 `.metal` type: AWS supports nested virtualization on the 7th- and 8th-generation Intel families (`m7i`, `c7i` and `r7i`, `m8i`, `c8i` and `r8i`, and the `-flex` variants AWS offers of them), and this module was validated on `m8i.xlarge`; no arm64 or Mac type; 12 GiB RAM recommended, the default has 16 |
| `availability_zone` | the first that offers `instance_type` | the zone for the subnet and the instance, one of the region's own (not a Local or Wavelength Zone), read when the subnet is created; it has to offer `instance_type` |
| `vpc_cidr` | `10.10.0.0/16` | the dedicated VPC, as a network address from /16 to /20; it must not overlap `10.11.0.0/16` or `10.12.0.0/16`, the orchestrator's sandbox networks; read when the VPC is created: changing it on an existing install fails at apply, so destroy the install first |
| `ami` | Ubuntu 24.04 LTS | an AMI id; empty reads Canonical's current Ubuntu 24.04 amd64 image at each plan, and a set id skips that read; it must be known at plan time (a literal, a variable or a data source read at plan), because it decides whether the parameter is read, so an id from a resource created in the same apply fails the plan with `Invalid count argument`; the module wants Ubuntu 24.04 on x86-64 (the startup script installs Docker from Docker's `noble` repository) with its root device at `/dev/sda1`, where the launch template's root volume settings apply; the stack itself needs apt, a writable `/etc` and glibc 2.34 or newer |
| `root_volume_size_gb` | `50` | 20 GiB has to stay free after the OS and Docker |
| `root_volume_type` | `gp3` | the root volume's EBS type |
| `hugepages` | `2048` | 2 MiB hugepages reserved for sandboxes; 2048 is 4 GiB |
| `team_api_key` | generated | `e2b_` plus an even number of lowercase hex characters, at least 32 |
| `compose_base_url` | the shipped files | a directory URL to fetch the two files from at first boot |
| `otel_collector_grpc_endpoint` | empty | host:port of an OTLP/gRPC collector the services export metrics, traces and logs to, with no scheme; empty disables export unless `otel_collector` is on |
| `otel_collector` | `false` | run the built-in collector on the instance, writing sandbox and team metrics into its own ClickHouse; implies `127.0.0.1:4317` when the endpoint is empty |
| `tags` | `{}` | tags for every resource that takes them except the two bucket objects (S3 allows an object only 10), the instance, its volume and its network interface included |
| Output | What it is |
|--------|------------|
| `api_url` | `E2B_API_URL` for the SDK |
| `sandbox_url` | `E2B_SANDBOX_URL` for the SDK |
| `dashboard_url` | the dashboard, for a browser; sign in with `e2b_api_key` |
| `e2b_api_key` | `E2B_API_KEY`, the team API key the seed inserted (sensitive) |
| `autoscaling_group` | the name of the Auto Scaling group |
| `session_command` | a shell snippet for `eval`: a Session Manager shell on the group's in-service instance |
A worked call is in [`examples/basic/main.tf`](examples/basic/main.tf), which
`make lint` validates along with the module.
---
### Overview (`embed/terraform/azure/README.md`)
# E2B Embed on one Azure instance with Terraform
Creates one Ubuntu 24.04 instance with nested virtualization, in a scale set
of one, which installs Docker, writes the two files Embed ships and runs
`docker compose up -d --wait`. It is a single-machine package rather than a
deployment pattern; the hub is [`../../README.md`](../../README.md).
## Requirements
- Terraform 1.7.5 or newer.
- An Azure subscription and credentials the azurerm provider can read
(`az login`, or any other source it supports), with `ARM_SUBSCRIPTION_ID`
set or `subscription_id` in the provider block.
- Permission to create a resource group and, in it, a virtual network, a
security group, two public IP addresses, a NAT gateway, a user-assigned
managed identity, a storage account and a scale set; **and** to write role
assignments and one custom role definition, which Owner or the pair
Contributor plus User Access Administrator grants. The module makes three
role assignments and one role definition (Identity below).
- `storage_use_azuread = true` in the provider block. The storage account the
module creates refuses shared keys, so Terraform writes the two files with
the identity you ran it as rather than with an account key, which is what
keeps a key and a SAS out of the state file.
- A region that offers the instance size, `Standard_D4ds_v7` by default, in
the zone you ask for.
- The Azure CLI on your own machine, for the `run_command` output and the
serial console under Reaching the instance. `terraform apply` needs neither.
- No machine of your own. The module creates it.
- The module source has to be the repository, not a copy of this directory:
it reads `../../compose/compose.yaml` and `../../compose/.env` relative to
itself, which the `github.com/...//` source form below provides by cloning.
A copy of this directory alone fails at plan time, and so does an absolute
local path, which Terraform copies the same way. A checkout of your own
works through a relative `source`, such as `../runtime/embed/terraform/azure`.
## Install
```hcl
provider "azurerm" {
features {}
storage_use_azuread = true
}
module "e2b" {
source = "github.com/e2b-dev/runtime//embed/terraform/azure?ref="
client_cidrs = ["203.0.113.0/24"] # where your SDK clients connect from
}
output "api_url" { value = module.e2b.api_url }
output "sandbox_url" { value = module.e2b.sandbox_url }
output "dashboard_url" { value = module.e2b.dashboard_url }
output "run_command" { value = module.e2b.run_command }
output "e2b_api_key" {
value = module.e2b.e2b_api_key
sensitive = true
}
output "admin_password" {
value = module.e2b.admin_password
sensitive = true
}
```
```bash
terraform init && terraform apply
```
A module's outputs are not the root module's, so the six `output` blocks are
what makes `terraform output` see them; without them the Try it commands below
print nothing.
`?ref=` pins the source to a commit of the public repository's `main`
branch, so an `init` on another day fetches the same `compose.yaml` and `.env`;
upgrading is a deliberate change of that ref. Use `main` itself for the newest
code. The instance answers `GET /health` about two minutes after `apply`
returns -- most of the Docker install and the image pulls happen while `apply`
is still waiting on the scale set. The `base` template build runs on past
that, so the first sandbox can be created about seven minutes after `apply`
returns; the Timings table below carries the measured numbers.
## Try it
Put this install's three SDK variables in your shell:
```bash
export E2B_API_URL="$(terraform output -raw api_url)"
export E2B_SANDBOX_URL="$(terraform output -raw sandbox_url)"
export E2B_API_KEY="$(terraform output -raw e2b_api_key)"
```
Open `terraform output -raw dashboard_url` in a browser and paste the key
(`terraform output -raw e2b_api_key`) into its key form. The instance's `.env`
carries `E2B_DASHBOARD_HOST=`, so the browser reaches sandbox
traffic at the same address, no tunnel needed.
Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's `pip` refuses to write to. This runs on your own
machine, not on the instance; on Ubuntu install `python3-venv` first with
`sudo apt-get update && sudo apt-get install -y python3-venv`, or it fails
with `ensurepip is not available`.
```bash
python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0
```
Then create a sandbox. Save this and run it with `.venv/bin/python`:
```python
from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()
```
The first attempt can fail with `404: tag 'default' does not exist for
template 'local-dev-team/base'`. That is the `base` build still finishing,
not a broken install: wait a minute and run the snippet again.
`curl -s -H "X-API-Key: $E2B_API_KEY" $E2B_API_URL/v2/templates` lists
`base` from the moment its build starts rather than when it finishes, so a
listing is not the go-ahead for the snippet above. Or run the packaged smoke
test on the instance itself: it does the same with the JavaScript SDK in a
container, then reaches a port inside the sandbox through client-proxy:
```bash
eval "$(terraform output -raw run_command) 'cd /opt/e2b && docker compose --profile test run --rm smoke'"
```
## Remove
```bash
terraform destroy # the resource group and everything the module made in it
```
## Reference
### What it creates
One resource group and, in it: a virtual network with one subnet; a security
group with one inbound rule and its subnet association; a static Standard
public IP for clients, and a second one behind a NAT gateway for egress; a
user-assigned managed identity, a custom role definition and three role
assignments, and a 60-second `time_sleep` that gives them time to propagate
(Timings); a storage account that refuses shared keys, with one private
container holding the two files; a scale set of one
(`Standard_D4ds_v7`, a 50 GB Premium SSD OS disk, the Application Health
extension, the startup script as custom data); and the four secrets under
Secrets below. The providers are azurerm, random and time.
The scale set replaces the instance when the health extension reports
`GET :3000/health` dead past the grace period. The extension runs in Binary
Health States: a 200 is healthy, anything else -- another status, a timeout,
a refused connection -- is not. A replacement starts from a fresh disk, so it
rebuilds the `base` template and the templates you built are gone.
### Why there is a NAT gateway
Instances of a Flexible scale set get no outbound address of their own, and
the client address is not on the instance's network interface until the first
boot has already installed Docker and asked Resource Manager to attach it. The
NAT gateway is what carries that boot, and every later egress; it also gives
the install one stable outbound address, which an upstream allowlist can name.
It is the one resource the AWS and GCP modules have no counterpart for, and
the one line item on this shape's bill that those two do not have.
### Security group
Three ports, and nothing else reaches the instance from outside: 3000, 3001
and 3002 from `client_cidrs`. There is no port 22 and no probe range: the
serial console and Run Command reach the instance through the platform rather
than through the network, and the health extension asks `/health` from inside
the instance. The orchestrator's unauthenticated control port 5008 in
particular stays inside the instance.
### Identity
The instance's identity holds two grants, and anyone with a shell on the
instance holds them with it:
- **Storage Blob Data Reader on the container**, not on the account: it reads
the two files the module uploaded and nothing else.
- **A custom role scoped to the resource group** with five actions:
`Microsoft.Network/networkInterfaces/read` and `/write`,
`Microsoft.Network/publicIPAddresses/read` and `/join/action`, and
`Microsoft.Network/virtualNetworks/subnets/join/action`. That is the
narrowest grant that attaches the address. The built-in role for the job is
Network Contributor, which carries every other write in `Microsoft.Network`
with it. The `join` on the subnet is there because a network interface is
written whole: adding the address also re-states the subnet the interface is
already in, and Azure authorizes that with a join. The resource group is the
scope because the interface does not exist until the scale set launches an
instance, so there is no narrower scope to name it at; the group holds this
install and nothing else.
The third role assignment is not the instance's: it gives whoever runs
`terraform apply` Storage Blob Data Contributor on the same container, because
with shared keys refused, writing a blob is authorized the same way reading
one is.
Custom role definition names are unique per tenant, so a second install in the
same tenant needs its own `name`.
### Timings
| Event | Time |
|-------|------|
| `terraform apply` on a clean subscription, start to finish | about 7 minutes |
| of which, the wait for the role assignments to propagate | 60 seconds, fixed |
| after `apply` returns, until `GET /health` answers | about 2 minutes |
| after `apply` returns, until the first `Sandbox.create` succeeds | about 7 minutes |
| a replacement (Upgrading), until `GET /health` answers again | about 2 minutes |
| a replacement, until the rebuilt `base` takes a sandbox | about 4 minutes |
| `/health` dying, until the extension reports the instance unhealthy | about 2 minutes |
| a reboot of the instance | about 1 minute |
Measured on `Standard_D4ds_v7` in `eastus`. Most of the first boot happens
while `apply` is still waiting for the scale set, which is why `/health`
answers so soon after it returns. The 60-second row is a fixed wait: a role
assignment takes some time to reach the service that enforces it, and a call
inside that window is refused as if the assignment were never made. If an
apply still fails on a blob with `AuthorizationPermissionMismatch`, run
`terraform apply` again.
To follow a boot:
```bash
eval "$(terraform output -raw run_command) 'tail -n 100 /var/log/cloud-init-output.log'"
eval "$(terraform output -raw run_command) 'journalctl -u e2b-embed -n 100'"
```
From outside, `az vm boot-diagnostics get-boot-log --ids ` prints the
console log of an instance that never got far enough to run a command.
### Reaching the instance
`run_command` is a shell snippet that looks the current instance up and runs a
script on it through the guest agent, which is why it is used through `eval`;
put the script in quotes inside the `eval`, as the examples above do. It needs
no open port.
When the stack is too broken for that, the serial console is the way in:
`az serial-console connect --resource-group --name `, user
`e2b`, password `terraform output -raw admin_password`. That is the only thing
the password is for; no port 22 is opened, on purpose.
### Upgrading
A newer ref points at newer files. `terraform apply` uploads them to the
container and records a new scale-set model, which the next instance uses, but
it does not replace the running one, and that instance wrote its
`compose.yaml` and `.env` once, on first boot. Replace it deliberately:
```bash
az vmss scale --resource-group --name --new-capacity 0
az vmss scale --resource-group --name --new-capacity 1
```
`` is the module's `name`, `e2b-embed` unless you set it, for both the
resource group and the scale set. The two steps are the point: the instance
holds the public IP, so it has to go before the next one comes up, and
scaling to zero and back is what makes that explicit. The new instance boots
the newest model, attaches the same address and rebuilds base from a fresh
disk, so the templates you built are gone. A replacement the scale set makes
on its own, after the health extension reports the instance dead, uses the
newest model too.
The image follows the same rule: `version = "latest"` records a new model when
Canonical publishes a new Ubuntu 24.04 image, and the next replacement boots
it. Set `image.version` to pin one.
`compose_base_url` points the first boot at the Compose files of a specific
commit instead of the files the module ships, for example
`https://raw.githubusercontent.com/e2b-dev/runtime//embed/compose`.
### Secrets
`ADMIN_TOKEN`, `SANDBOX_ACCESS_TOKEN_HASH_SEED` and the team API key are all
generated per install, here by Terraform itself: the startup script writes all
three into the instance's `.env` before the first `up`, so the pair the
`api-secrets` one-shot generates on the instance is never read. To rotate
either api secret, replace its resource and then replace the instance as
Upgrading describes: `terraform apply -replace=module.e2b.random_bytes.admin_token`
or `-replace=module.e2b.random_bytes.sandbox_access_token_hash_seed` records a
new model; a new hash seed ends the tokens of running `secure` sandboxes.
They live in the Terraform state, so treat the state file and `terraform show`
as secret, and they reach the instance in its custom data, which the scale-set
model keeps. Anyone allowed `Microsoft.Compute/virtualMachineScaleSets/read`
on the resource group can read it, and read-only access such as the built-in
Reader role includes that action. So can any process on the host, root or not,
through the instance metadata service. The startup script does not print them;
read the key with `terraform output -raw e2b_api_key`. Set `team_api_key` to
choose the key instead; change it and replace the instance to rotate it.
Keeping the three out of custom data, in a key vault only the instance's
identity can read, would narrow who can see them; this module does not do
that.
The fourth secret is the `e2b` user's password, for the serial console only
(Reaching the instance). It exists because Azure refuses a Linux instance with
neither a password nor an SSH key.
Azure encrypts the OS disk at rest with a platform key, so the `.env` and the
three secrets in it are encrypted without a setting here. The storage account
refuses shared keys, so no account key and no SAS exists for the container:
the instance reads the two files with its identity's token, and `terraform
apply` writes them with yours.
The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the instance serves plain http; sign-out
clears it.
### Telemetry
`otel_collector = true` runs the built-in OpenTelemetry collector on the
instance, keeping the sandbox and team metrics behind the dashboard's
monitoring charts in its own ClickHouse, and `otel_collector_grpc_endpoint`
sends the services' metrics, traces and logs to a collector of your own. Set
one or the other: with both, the services export to your endpoint and the
built-in collector receives nothing, so the dashboard's charts stay empty.
Both reach the instance's `.env` at first boot only, so changing either later
takes the replacement under Upgrading; the reference's
[Observability](../../docs/REFERENCE.md#observability) has the rest.
### Limitations
- No persistent data volume. Every replacement starts from a fresh disk,
rebuilds the `base` template and loses the templates you built.
- Stopping the stack gets the instance replaced. The health extension reports
it unhealthy about two minutes after `/health` stops answering, and the
scale set then replaces it from a fresh disk. So `docker compose down` on
the instance is not a pause: suspend repairs first with
`az vmss update --resource-group --name --enable-automatic-repairs false`,
and turn them back on once `/health` answers again with
`az vmss update --resource-group --name --enable-automatic-repairs true --automatic-repairs-grace-period 15`.
Re-enabling demands the grace period restated, and the flag takes bare
minutes, not an ISO 8601 span -- the CLI wraps the value itself, so `PT15M`
is refused as `PTPT15MM`. A later `terraform apply` also restores the
policy, since the suspension lives outside the Terraform state.
- Nested virtualization cannot be checked before the apply. Azure decides it
by the size alone, with no per-instance flag and no API to ask at plan time,
so `instance_size` is held to the generations that have it rather than to
what your region and subscription actually offer. A size either does not
offer fails at apply with Azure's own message; a size without `/dev/kvm`
would boot and then refuse every sandbox. Check with
`az vm list-skus -l --resource-type virtualMachines` before
changing it: SKU availability varies by subscription offer as well as by
region, and the default is not available in every one.
- Template builds with `copy()` steps upload through the orchestrator's port
5008, which the security group keeps closed -- `client_cidrs` opens only
3000, 3001 and 3002, and the upload URL the API hands back points at the
instance's own loopback besides. Run such builds on the instance itself
(the `run_command` output); building them remotely is not supported.
- x86-64 only. The Cobalt sizes are arm64 and the module's default image is
x86-64 Ubuntu.
- One region, one zone. The instance is a scale set of one; `zone` moves it,
and emptying `zone` places it regionally for a region that has no zones.
- Azure Government and the sovereign clouds are untested. Canonical publishes
the same image there and the module names no endpoint of its own, so it
should work with the provider pointed at one of them.
### Variables
| Input | Default | What it is |
|-------|---------|------------|
| `client_cidrs` | required | the IPv4 CIDRs allowed to reach 3000, 3001 and 3002, as network addresses (`203.0.113.0/24`, `198.51.100.7/32`); at least one, and a browser's has to be its public IPv4 address |
| `name` | `e2b-embed` | name prefix for the resource group and everything in it; the custom role definition takes it too, and role definition names are unique per tenant, so a second install in the same tenant needs its own `name` |
| `location` | `eastus` | the region for the resource group and everything in it |
| `zone` | `1` | the availability zone for the instance, as a bare number; empty places it regionally, for a region that has no zones |
| `instance_size` | `Standard_D4ds_v7` | a general-purpose D size with premium storage, generation 3 or newer (`Dsv3` and up, with or without the `a` and `d` letters), or an `Lsv3` and up; no burstable and no arm64 Cobalt size; 12 GiB RAM recommended, the default has 16. Which generations you can actually use depends on the region **and on your subscription's offer** — `az vm list-skus -l --resource-type virtualMachines` is what answers it, and a size it marks `NotAvailableForSubscription` fails at apply |
| `image` | Ubuntu 24.04 LTS | the marketplace image, as publisher, offer, sku and version; the module wants Ubuntu 24.04 on x86-64 and a generation 2 image, and `latest` records a new model when Canonical publishes one |
| `os_disk_size_gb` | `50` | 20 GiB has to stay free after the OS and Docker |
| `os_disk_type` | `Premium_LRS` | the OS disk's storage type |
| `hugepages` | `2048` | 2 MiB hugepages reserved for sandboxes; 2048 is 4 GiB |
| `team_api_key` | generated | `e2b_` plus an even number of lowercase hex characters, at least 32 |
| `compose_base_url` | the shipped files | a directory URL to fetch the two files from at first boot |
| `otel_collector_grpc_endpoint` | empty | host:port of an OTLP/gRPC collector the services export metrics, traces and logs to, with no scheme; empty disables export unless `otel_collector` is on |
| `otel_collector` | `false` | run the built-in collector on the instance, writing sandbox and team metrics into its own ClickHouse; implies `127.0.0.1:4317` when the endpoint is empty |
| `tags` | `{}` | tags for every resource in the module's resource group |
| Output | What it is |
|--------|------------|
| `api_url` | `E2B_API_URL` for the SDK |
| `sandbox_url` | `E2B_SANDBOX_URL` for the SDK |
| `dashboard_url` | the dashboard, for a browser; sign in with `e2b_api_key` |
| `e2b_api_key` | `E2B_API_KEY`, the team API key the seed inserted (sensitive) |
| `scale_set` | the name of the scale set of one |
| `run_command` | a shell snippet for `eval`: run a script on the instance through the guest agent |
| `admin_password` | the `e2b` user's password, for the serial console (sensitive) |
A worked call is in [`examples/basic/main.tf`](examples/basic/main.tf), which
`make lint` validates along with the module.
---
### Overview (`embed/terraform/gcp/README.md`)
# E2B Embed on one GCE instance with Terraform
Creates one Ubuntu 24.04 VM with nested virtualization, in a managed instance
group of one, which installs Docker, writes the two files Embed ships and runs
`docker compose up -d --wait`. It is a single-machine package rather
than a deployment pattern; the hub is
[`../../README.md`](../../README.md).
## Requirements
- Terraform 1.7.5 or newer.
- A GCP project with the Compute Engine and Cloud Resource Manager APIs
enabled (the second one grants the instance's service account its logging
role), and credentials for it (`gcloud auth application-default login`, or
a service account).
- No machine of your own. The module creates it.
- The module source has to be the repository, not a copy of this directory:
it reads `../../compose/compose.yaml` and `../../compose/.env` relative to
itself, which the `github.com/...//` source form below provides by cloning.
A copy of this directory alone fails at plan time.
## Install
```hcl
module "e2b" {
source = "github.com/e2b-dev/runtime//embed/terraform/gcp?ref="
project_id = "my-project"
client_cidrs = ["203.0.113.0/24"] # where your SDK clients connect from
}
output "api_url" { value = module.e2b.api_url }
output "sandbox_url" { value = module.e2b.sandbox_url }
output "dashboard_url" { value = module.e2b.dashboard_url }
output "ssh_command" { value = module.e2b.ssh_command }
output "e2b_api_key" {
value = module.e2b.e2b_api_key
sensitive = true
}
```
```bash
terraform init && terraform apply
```
A module's outputs are not the root module's, so the five `output` blocks are
what makes `terraform output` see them; without them the Try it commands below
print nothing.
`?ref=` pins the source to a commit of the public repository's `main`
branch, so an `init` on another day fetches the same `compose.yaml` and `.env`;
upgrading is a deliberate change of that ref. Use `main` itself for the newest
code. The first instance answers `GET /health` about two
minutes after `apply` returns, the Docker install and the image pulls behind
it. The `base` template build runs on past that, so the first sandbox can be
created about four minutes after `apply` returns.
## Try it
Put this install's three SDK variables in your shell:
```bash
export E2B_API_URL="$(terraform output -raw api_url)"
export E2B_SANDBOX_URL="$(terraform output -raw sandbox_url)"
export E2B_API_KEY="$(terraform output -raw e2b_api_key)"
```
Open `terraform output -raw dashboard_url` in a browser and paste the key
(`terraform output -raw e2b_api_key`) into its key form. The instance's `.env`
carries `E2B_DASHBOARD_HOST=`, so the browser reaches sandbox
traffic at the same address, no tunnel needed.
Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's `pip` refuses to write to. This runs on your own
machine, not on the instance; on Ubuntu install `python3-venv` first with
`sudo apt-get update && sudo apt-get install -y python3-venv`, or it fails
with `ensurepip is not available`.
```bash
python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0
```
Then create a sandbox. Save this and run it with `.venv/bin/python`:
```python
from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()
```
The first attempt can fail with `404: tag 'default' does not exist for
template 'local-dev-team/base'`. That is the `base` build still finishing,
not a broken install: wait a minute and run the snippet again.
`curl -s -H "X-API-Key: $E2B_API_KEY" $E2B_API_URL/v2/templates` lists
`base` from the moment its build starts rather than when it finishes, so a
listing is not the go-ahead for the snippet above. Or run the packaged smoke
test on the instance itself: it does the same with the JavaScript SDK in a
container, then reaches a port inside the sandbox through client-proxy:
```bash
eval "$(terraform output -raw ssh_command)"
cd /opt/e2b && sudo docker compose --profile test run --rm smoke
```
## Remove
```bash
terraform destroy # the instance and every other resource the module made
```
## Reference
### What it creates
Fourteen resources: a VPC and a subnet; a reserved external address; a
service account and its `roles/logging.logWriter` binding; an instance
template (`n4-standard-4`, a 50 GB Hyperdisk, nested virtualization on, the
two files in instance metadata); a health check and a zonal managed instance
group of one that auto-heals on `GET :3000/health`; three firewall rules; and
the three secrets under Secrets below.
The group recreates the instance when it fails its health check. A recreated
instance starts from a fresh disk, so it rebuilds the `base` template and the
templates you built are gone.
### Firewall
Three rules, and nothing else reaches the instance from outside: 3000, 3001
and 3002 from `client_cidrs`, 3000 from Google's health checkers, and 22 from
IAP. The orchestrator's unauthenticated control port 5008 in particular stays
inside the VM.
### Timings
| Event | Time |
|-------|------|
| first boot, until `GET /health` answers | about 2 minutes |
| first boot, until the first `Sandbox.create` succeeds | about 4 minutes |
| an auto-healing repair, or a replacement after an out-of-band delete | about 3 minutes |
| `rolling-action replace` (see Upgrading) | 5 to 6 minutes |
| a plain reset of the VM | about 1 minute |
Measured on the default `n4-standard-4`. A replacement takes longer than it
looks: the old instance keeps answering `/health` for the first 40 seconds or
so, so wait for the group's `currentAction` to read `NONE` before timing it.
To follow a boot:
```bash
eval "$(terraform output -raw ssh_command)"
sudo journalctl -u google-startup-scripts -f
```
The `ssh_command` output is a shell snippet that looks the current instance
name up, which is why it is used through `eval`.
### Secrets
`ADMIN_TOKEN`, `SANDBOX_ACCESS_TOKEN_HASH_SEED` and the team API key are all
generated per install, here by Terraform itself: the startup script writes
all three into the instance's `.env` before the first `up`, so the pair the
`api-secrets` one-shot generates on the instance is never read. To rotate
either api secret, replace its resource and then replace the instance as
Upgrading describes: `terraform apply -replace=module.e2b.random_bytes.admin_token`
or `-replace=module.e2b.random_bytes.sandbox_access_token_hash_seed` changes
the instance template; a new hash seed ends the tokens of running `secure`
sandboxes. They live in the Terraform state, so treat the state file and
`terraform show` as secret, and they reach the instance through its
startup-script metadata, which anyone with `compute.instances.get` on the
project can read and which any process on the instance can read from the
metadata server without authentication. The startup script does not print
them; read the key with `terraform output -raw e2b_api_key`. Set
`team_api_key` to choose the key instead; change it and recreate the instance
to rotate it.
The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the instance serves plain http; sign-out
clears it.
### Upgrading
A newer ref points at newer files, which changes the instance template.
`terraform apply` records the new template but does not replace the running
instance, because the group's update policy is opportunistic, and an
auto-healing recreate in the meantime uses the template the instance was
created from rather than the new one. Replace it deliberately:
```bash
gcloud compute instance-groups managed rolling-action replace --project --zone --replacement-method=recreate --max-surge=0 --max-unavailable=1
```
The three flags matter: the instance holds a reserved address, so the
replacement must be delete-before-create. That is a fresh instance, so the
`base` template is rebuilt and the templates you built are gone. The command
also sets the group's update policy type to proactive; the next
`terraform apply` sets it back, which is the only change it will show.
The instance's `.env` and `compose.yaml` are written once, on first boot, so
an install created before the dashboard existed keeps running the Compose file
it was created with, which has no dashboard, until that replace.
`compose_base_url` points the first boot at the Compose files of a specific
commit instead of the files the module ships, for example
`https://raw.githubusercontent.com/e2b-dev/runtime//embed/compose`.
### Telemetry
`otel_collector = true` runs the built-in OpenTelemetry collector on the
instance, keeping the sandbox and team metrics behind the dashboard's
monitoring charts in its own ClickHouse, and `otel_collector_grpc_endpoint`
sends the services' metrics, traces and logs to a collector of your own. Set
one or the other: with both, the services export to your endpoint and the
built-in collector receives nothing, so the dashboard's charts stay empty.
Both reach the instance's `.env` at first boot only, so changing either later
takes the replace under Upgrading; the reference's
[Observability](../../docs/REFERENCE.md#observability) has the rest.
### Limitations
- No persistent data disk. Every recreate or replace starts from a fresh
disk, rebuilds the `base` template and loses the templates you built.
- Template builds with `copy()` steps upload through the orchestrator's port
5008, which the firewall keeps closed. Run them on the instance, or tunnel
the four ports and point the SDK at `http://127.0.0.1:3000` and
`http://127.0.0.1:3002`:
```bash
eval "$(terraform output -raw ssh_command) -- -N -L 3000:127.0.0.1:3000 -L 3001:127.0.0.1:3001 -L 3002:127.0.0.1:3002 -L 5008:127.0.0.1:5008"
```
### Variables
| Input | Default | What it is |
|-------|---------|------------|
| `project_id` | required | the GCP project; the Compute Engine and Cloud Resource Manager APIs must be enabled |
| `client_cidrs` | required | the CIDRs allowed to reach 3000, 3001 and 3002; at least one, and a browser's has to be its public IPv4 address |
| `zone` | `us-west1-b` | the zone; the subnet's and the address's region follows from it |
| `name` | `e2b-embed` | name prefix for every resource |
| `machine_type` | `n4-standard-4` | 12 GiB RAM recommended; the default has 16 |
| `boot_disk_size_gb` | `50` | 20 GiB has to stay free after the OS and Docker |
| `boot_disk_type` | `hyperdisk-balanced` | n4 machine types support only Hyperdisk |
| `image` | Ubuntu 24.04 LTS | the stack needs apt, a writable `/etc` and glibc 2.34 or newer; x86-64 only (this template is a nested-virtualization VM with Docker's amd64 repository in its startup script; GCE's arm64 machine types have no nested virtualization, metal included; the metal types run the stack on their bare `/dev/kvm`, so an arm64 install would be a metal instance, not this nested-virt VM template) |
| `hugepages` | `2048` | 2 MiB hugepages reserved for sandboxes; 2048 is 4 GiB |
| `team_api_key` | generated | `e2b_` plus an even number of lowercase hex characters, at least 32 |
| `compose_base_url` | the shipped files | a directory URL to fetch the two files from at first boot |
| `otel_collector_grpc_endpoint` | empty | host:port of an OTLP/gRPC collector the services export metrics, traces and logs to, with no scheme; empty disables export unless `otel_collector` is on |
| `otel_collector` | `false` | run the built-in collector on the instance, writing sandbox and team metrics into its own ClickHouse; implies `127.0.0.1:4317` when the endpoint is empty |
| `labels` | `{}` | labels for the instance template and the address |
| Output | What it is |
|--------|------------|
| `api_url` | `E2B_API_URL` for the SDK |
| `sandbox_url` | `E2B_SANDBOX_URL` for the SDK |
| `dashboard_url` | the dashboard, for a browser; sign in with `e2b_api_key` |
| `e2b_api_key` | `E2B_API_KEY`, the team API key the seed inserted (sensitive) |
| `instance_group` | the self link of the managed instance group |
| `ssh_command` | a shell snippet for `eval`: SSH through IAP to the current instance |
A worked call is in [`examples/basic/main.tf`](examples/basic/main.tf), which
`make lint` validates along with the module.
---
### Overview (`packages/orchestrator/cmd/simulate-gcs-traffic/readme.md`)
To run inside of a host in GCS, trust this fancy one-liner:
host=@ && bucket= && go build -o simulator . && ssh $host rm ./simulator && scp ./simulator $host:~/simulator && ssh $host -- ./simulator -csv-path gcs.csv -test-duration=5s $bucket && ssh $host cat gcs.csv
---
### Overview (`packages/shared/pkg/apiid/README.md`)
# Resource IDs at HTTP boundaries
`apiid.ProjectID` and `apiid.WorkspaceID` accept a legacy UUID or the matching
public ID. Both inputs produce the same struct:
```go
projectID.UUID // uuid.UUID
projectID.PublicID // canonical prj_ string
```
The existing `pkg/id` codecs remain unchanged. These structs are HTTP input
values; database and domain functions can continue taking `uuid.UUID`.
## Adding or changing an API ID
Update [`id`](../id/README.md#adding-or-changing-a-resource-id) and this package
in the same PR whenever adding a resource kind to either package. Follow the
linked checklist: `id` owns the codec, and `apiid` provides the matching input
struct and binding support. Do not add an API-only encoding or a new codec
without its API type. Keep existing kinds' encoding and validation changes,
tests, and documentation synchronized across both packages.
## OpenAPI and generated Gin handlers
Use the standard oapi-codegen type override on a string schema:
```yaml
projectID:
name: projectID
in: path
required: true
schema:
type: string
description: Project UUID or prj_ public ID.
x-go-type: apiid.ProjectID
x-go-type-import:
name: apiid
path: github.com/e2b-dev/infra/packages/shared/pkg/apiid
```
Regenerate the server and dependent clients using the API's generation command.
No middleware, registry, or generator fork is needed:
```go
func (s *Server) GetProject(c *gin.Context, projectID api.ProjectID) {
project, err := s.projects.Get(c.Request.Context(), projectID.UUID)
// Handle err and build the response as usual.
}
```
Use `apiid.WorkspaceID` for workspace parameters. Apply the same schema override
to query or header parameters and to JSON body properties or array items:
```yaml
workspace_id:
type: string
description: Workspace UUID or wrk_ public ID.
x-go-type: apiid.WorkspaceID
x-go-type-import:
name: apiid
path: github.com/e2b-dev/infra/packages/shared/pkg/apiid
```
After `c.ShouldBindJSON(&body)`, use `body.WorkspaceId.UUID` and
`body.WorkspaceId.PublicID`. Keep response UUID fields unchanged unless a
response-contract change is intended. Do not keep `format: uuid` on inputs
that also accept public IDs.
## Handwritten Gin binding
```go
var params struct {
ProjectID apiid.ProjectID `uri:"projectID" binding:"required"`
}
if err := c.ShouldBindUri(¶ms); err != nil {
c.AbortWithStatus(http.StatusBadRequest)
return
}
projectUUID := params.ProjectID.UUID
```
`ShouldBindQuery` and `ShouldBindHeader` work with `form` and `header` tags.
`UnmarshalText` supports generated path binding; `Bind` supports oapi-codegen's
exploded query binding; `UnmarshalParam` supports Gin's own binders. Tests cover
each path because these binders do not all use the same interface.
## Construction and encoding
Use `apiid.NewProjectID(u)` or `apiid.NewWorkspaceID(u)` when constructing an ID
in Go. Decoders and constructors populate both fields. Malformed input and
wrong-kind public IDs fail without modifying the receiver. JSON accepts only
strings; encoding emits a public ID string, not a JSON object.
The fields are exported for direct reads. Treat decoded values as immutable;
Go cannot stop a caller from manually making them inconsistent. Encoding derives
the public string from `UUID`, rather than trusting a manually edited `PublicID`.
A zero-valued struct denotes an unset input; requiredness belongs in the request
schema or Gin validation. Use a pointer for optional inputs when absence matters.
Other APIs adopt the types by updating their schemas and regeneration, not by
installing middleware. This package does not change their contracts automatically.
---
### Overview (`packages/shared/pkg/id/README.md`)
# Typed public IDs
This package converts resource UUIDs to and from human-readable public IDs:
```text
prj_7mfb0gpp6c8dsaasre0asc7n3s
wrk_2n1t201rmv87aae5j4csam8000
sec_01kyjhkfvse96rg4d2qzd9enft
```
Every public ID has a stable resource prefix, an underscore, and a
26-character UUID suffix. E2B's public representation is always lowercase.
## UUID ownership
Resource owners remain responsible for generating and storing UUIDs. UUIDs
may, for example, be generated by application code or a database. This package
does not use `typeid.New` or `typeid.MustNew`.
[`github.com/sumup/typeid`](https://github.com/sumup/typeid) is used only to
convert an existing UUID to its prefixed public representation and parse that
representation back into the UUID-backed resource type. PostgreSQL and other
persistence layers continue to store the underlying UUID, not the prefixed
string.
```go
projectUUID := uuid.New()
projectID := id.ProjectID(projectUUID)
publicID := projectID.String()
parsedID, err := id.ParseProjectID(publicID)
if err != nil {
return err
}
projectUUID = uuid.UUID(parsedID)
```
Project and workspace UUIDs are normally UUIDv4 and use `typeid.Random`.
Secret UUIDs are UUIDv7 and use `typeid.Sortable`. SumUp renders Random IDs in
uppercase, so the E2B wrapper normalizes them to the lowercase public contract.
## Team IDs
Teams are projects: a team UUID is also its project UUID.
`ConvertTeamIDToProjectID` renders a team UUID as its public `prj_` ID, and
`ParseTeamID` accepts either spelling wherever an API takes a team ID.
## Adding or changing a resource ID
Keep this package and [`apiid`](../apiid/README.md) in sync in the same PR.
Adding a resource kind here requires its matching API input type in `apiid`;
adding a type in `apiid` requires a matching codec here. Changes to an existing
kind's prefix, encoding, or validation must also update the affected API type
and tests. Encoding rules belong here; `apiid` must reuse them, not duplicate them.
To add another public resource ID:
1. Choose a short, stable, lowercase prefix that keeps the full public ID at
the expected fixed width.
2. Add a private prefix type implementing `Prefix()`.
3. Select `typeid.Random` for a UUIDv4 resource or `typeid.Sortable` for a
UUIDv7 resource.
4. Add a distinct UUID-backed public Go type with `String` and parse methods.
5. Keep UUID generation in the resource-owning application or database.
6. Test the exact prefix, lowercase output, fixed length, invalid-prefix
rejection, and UUID round trip.
7. Add the matching `apiid` struct with `UUID` and `PublicID`, its constructor,
and its text, JSON, generated-query, and Gin binding methods.
8. Test that UUID and public inputs produce identical API structs, wrong kinds
are rejected, and encoding agrees with this package's codec. Update both
READMEs and regenerate affected API contracts and clients.
Do not add JSON, text, SQL, or UUID-generation behavior implicitly. Those
interfaces should be introduced only when a concrete caller needs them.
---
### Overview (`packages/shared/pkg/sandboxtypes/README.md`)
# Shared sandbox types
This package owns `RuntimeMetadata`, `SandboxType`, and `EgressClass`, including
identity logging and the empty-type fallback to a regular sandbox. It contains
no worker resources, network initialization, or VM lifecycle management.
Callers import these types directly. `SandboxType.EgressClass()` selects the
traffic class; DSCP configuration and socket handling belong to worker networking.
The resource-owning `Sandbox` stays in the orchestrator package. Runtime team
metadata is attribution, not trusted authorization data.
---
### Overview (`packages/envd/internal/services/legacy/readme.md`)
The python SDK had a bug: new fields introduced to the API would throw exceptions when processed. In order to work around this, we convert all responses to a list of structs that were valid when the python SDK still had the bug.
We copy the proto files to this folder in order to keep track of what was available when this SDK was supported. These should never be modified.
This entire package can be removed once we no longer have clients that send a user agent of `connect-python`.
---
### Overview (`packages/api/internal/sandbox/reservations/redis/README.md`)
# Redis Reservation Storage
This package coordinates sandbox creation reservations across API instances.
## Keys
- Storage index: `sandbox:storage:{teamID}:index`
- Pending zset: `sandbox:storage:{teamID}:reservations:pending`
- Result key: `sandbox:storage:{teamID}:reservations:{sandboxID}:result`
- PubSub routing key: `sandbox:storage:{teamID}:reservations:{sandboxID}:notify`
## Flow
`Reserve` runs a Lua script that atomically removes stale pending entries, checks whether the sandbox already exists or has already pending start, enforces the team limit using `SCARD(storage index) + ZCARD(pending zset)`, deletes any stale result key, and adds the sandbox ID to the pending zset.
When creation completes, it removes the sandbox from the pending zset, writes a TTL result key containing either the sandbox or the creation error, and publishes the routing key.
A waiter subscribes to the routing key, probes the result key immediately, then waits for PubSub notifications or the 1 second fallback ticker. PubSub is best-effort; the fallback ticker is required for correctness.
`Release` is called when the sandbox is removed from storage (`Store.Remove`). It removes the sandbox from the pending zset, deletes the result key, and publishes the routing key.
---
### ARCHITECTURE (`docs/ARCHITECTURE.md`)
# E2B Infrastructure Architecture
This document explains what this repository implements, what each service does, and how the
services interact. It is the fastest way to build a mental model of the codebase before diving
into the code.
> **Keep this document updated.** If a code change alters anything described here (service
> responsibilities, ports, protocols, data stores, flows, deployment topology), update this file
> in the same PR.
## What this repository implements
E2B provides **sandboxes**: isolated Linux VMs that start almost instantly (they resume pre-booted
snapshots instead of cold-booting), run arbitrary code (typically
generated by AI agents), and can be paused, snapshotted, and resumed. This repo contains the whole
backend: the control-plane REST API, the data-plane VM orchestration built on
**Firecracker microVMs**, the in-VM agent, the edge routing layer, and template building.
Two ideas drive the design:
1. **A sandbox is a resumed snapshot.** Templates are pre-booted VM snapshots (memory + disk +
VM state) stored in object storage. "Creating" a sandbox means restoring a snapshot, which is
why startup is fast. Memory pages are loaded lazily on page-fault (userfaultfd) and the root
filesystem is a copy-on-write overlay, so only touched data is ever fetched. A template can
also be filesystem-only (rootfs, no memfile), taken by a filesystem-only pause or checkpoint;
sandboxes created from it cold-boot from disk instead of resuming.
2. **Control plane and data plane are separate.** The API decides *where* a sandbox runs and
tracks *that* it runs (Postgres/Redis); the orchestrator on each node owns *how* it runs
(Firecracker, networking, storage). Sandbox traffic never passes through the API.
## System overview
```
/* Detailed source-code truncated for AI context efficiency. */
```
## Services
| Service | Package | Runs on | Purpose |
|---|---|---|---|
| API | `packages/api` | API nodes | Public REST API; sandbox lifecycle, placement, auth, quotas |
| Orchestrator | `packages/orchestrator` | every sandbox node | Runs Firecracker VMs; sandbox create/pause/resume/kill |
| Template manager | `packages/orchestrator` (role) | build nodes | Builds templates from Docker images |
| Client proxy | `packages/client-proxy` | API nodes | Edge router: sandbox URL → correct node |
| Envd | `packages/envd` | inside every VM | In-VM agent: process/filesystem API for SDKs |
| Dashboard API | `packages/dashboard-api` | API nodes | Backend for the web dashboard (templates, builds, admin) |
Supporting packages: `packages/shared` (protos, telemetry, storage clients, feature flags),
`packages/auth` (authentication library), `packages/db` (Postgres migrations + sqlc queries),
`packages/clickhouse` (ClickHouse schema + clients), `packages/otel-collector` (collector config),
`packages/nomad-nodepool-apm` (autoscaler plugin), `packages/local-dev` (local stack).
### API (`packages/api`)
The control-plane entry point (Gin, OpenAPI-generated from `spec/openapi.yml`, port 80).
`PUT /unstable/management/projects/{projectID}/settings` applies a revisioned settings
projection from the owning management service. This endpoint belongs to the API
and accepts only service JWTs through `AdminJWTAuth`. It locks the team and checks
that it has not been deleted before applying the projection. The shared `ProjectSettings` type
contains `SandboxSettings`, with the optional ingress default under
`sandboxSettings.defaultAllowPublicTraffic`. Storage uses `projection.project_settings`,
with `revision` and nullable `sandbox_default_allow_public_traffic` in the same row.
The receiver applies both in one guarded upsert. Equal or older revisions cannot
overwrite it. Clearing stores NULL at a newer revision, retaining the ordering fence.
Cache invalidation follows commit and is retried on duplicate
delivery after a cache failure. There is no rollout flag check on accepted delivery.
Sandbox creation resolves an absent default to true and honors a saved default
independently of rollout eligibility. New sandboxes can use an older value while
delivery or cache refresh is pending.
- **Resources**: sandboxes (create/list/kill/pause/resume/connect/timeout/metrics/logs),
templates and builds, teams, volumes, API keys, secrets, sandbox events and webhooks,
admin operations.
- **Spec document**: `GET /openapi.json` serves, without authentication, the spec the request
validator runs. An `ETag` over the document answers a matching `If-None-Match` with 304. It is
registered before the validator, which rejects undeclared paths.
- **Auth** (via `packages/auth`): team API keys (`X-API-Key`, `e2b_` prefix), auth-provider JWTs
(OIDC), and either an admin token or a service JWT verified from the configured admin JWKS.
Backed by an auth DB (Postgres) with a Redis team cache.
- **API-group rate limits**: operations declare a group through the `x-api-group` OpenAPI
extension. After authentication succeeds, its callback wrapper reads the validator's
already-matched operation and stores the API group directly in Gin's context. The rate limiter
reads that group and maps it to a field in the authenticated team's limits. Grouped operations
require team-identifying authentication; public operations do not populate an API group.
The `list` group uses `api_team_rps_list`; the `delete` group uses `api_team_rps_delete` only
for sandbox and template deletion. Other DELETE operations are ungrouped.
Delete rates default to zero until configured.
A group's rate is one of the team's limits and resolves with the rest of them in the statement
that loads the team, which authentication caches; it has no source of its own.
Propagated project limits carry both rates, and each accepted revision replaces
them. A team with no `project_limits` row uses the deployment's `default_limits`; zero
disables group limiting. The migration that added `default_limits` gave every existing team a
row holding its limits at the time, so a change to the defaults reaches only teams created since.
A positive rate sets both requests per second and burst capacity. Redis enforces a separate
budget for each team, HTTP method, and route template across API replicas: groups select
rates, but routes do not share a budget. Exhausted budgets return HTTP 429 with rate-limit
and `Retry-After` headers. Ungrouped operations and unconfigured rates bypass group limiting;
Redis failures are logged and counted while requests proceed.
V1 route limits from `rate-limit-config` are always enforced first. For requests that pass V1,
LaunchDarkly's `rate-limit-v2-mode` is the master switch for all V2 checks, and
`rate-limit-delete-mode` can further restrict delete checks. Both default to `disabled`;
missing, invalid, or unknown modes are treated as disabled. Deletes skip V2 if either flag is
disabled and enforce only when both are enabled. If both are active and either is `shadow`,
deletes run in shadow mode. Lists follow the global mode.
The groups use the same limiter logic: `shadow` observes V2 decisions while
preserving V1 headers, and `enabled` also enforces V2. Ungrouped operations, unconfigured V2
rates, and V2 Redis errors retain V1 enforcement. A rejection reports the rejecting limiter's
headers; when both checks allow a request in enabled mode, V2 supplies the rate-limit headers.
- **Sandbox list snapshots**: a list request reads the team's whole sandbox index from Redis
(its running and pausing records) and pages in memory, so with the LaunchDarkly
`sandbox-list-max-lag-ms` flag set for a team, each API replica shares one read between
that team's concurrent list requests and keeps it for the flag's value, counted from the
start of the read; a request that joins a read already in flight gets that read's result
however long it takes. A lowered flag applies to a live snapshot; a raised one from the
next read. Within that lag a sandbox created, killed, paused or resumed can be missing from
the list or shown in its previous state on the replica holding the snapshot, and one
resumed inside it can appear on two pages of a walk served by different replicas. The
default, zero, gives every request its own read with the same store commands as before.
Only the two list endpoints use the snapshot; the evictor, reconcile, template delete and
team teardown keep reading the store directly.
- **Workload identity**: sandbox create accepts an optional `iam.tokens` map of caller-named
workload-token definitions (each an exact `audience` and `tokenType`). A non-empty, validated
map enables workload identity, whose identity the orchestrator derives from the sandbox's
already-authoritative team/sandbox/execution/template IDs; the definitions are passed to the
orchestrator in `SandboxConfig.iam`. The API mints no credential and delivers nothing into the
sandbox; file-based delivery is rejected at admission. Definitions are persisted in the
running-sandbox (Redis) and paused-snapshot (Postgres) state so they survive pause/resume and
orchestrator re-sync; a fork starts a new workload and does not inherit them.
- **Placement**: keeps a live map of orchestrator nodes (discovered via Nomad, Kubernetes, or a
static list). Chooses a node per sandbox with a **best-of-K** algorithm
(`internal/orchestrator/placement/`): sample K ready nodes, score by the
higher of CPU load and hugepage-pool load, pick the lowest; retry on
exhausted nodes. A node that refuses a create (`ResourceExhausted`) stays
eligible, and each refusal waits a full-jitter backoff that doubles up to a
cap, so a placement that finds every node busy does not re-send creates as
fast as they are refused. The hugepage-pool load is measured against the node's
base pool (`nr_hugepages`, allocated at boot), so a node that has borrowed surplus
pages scores above 1 instead of reading as just full; a pool that is all overcommit
is measured against its ceiling, and an orchestrator that does not report the base
pool against `HugePages_Total` as before. A node that reports no pool scores 0.5. Hugepage scoring
is on by default (`BEST_OF_K_HUGEPAGE_MEMORY`); set it false to rank on
CPU alone. K, overcommit ratio, and alpha are tunable live via feature
flags. With `placement-hugepage-limit-percent` above 0, a hugepage sandbox
is also refused by any node whose committed hugepages would pass that share
of the node's pool ceiling (base pool plus overcommit allowance, reported by
the orchestrator); within an API replica the check and the in-flight record
are one step per node, and when the compatible nodes are all full placement
answers with a capacity error, not "no compatible node". With the
orchestrator's `create-memory-refusal` flag on (`CREATE_MEMORY_REFUSAL` for a
deployment without LaunchDarkly), a create whose guest memory the pool
cannot back while a sandbox on that node is still tearing down is refused
with `ResourceExhausted`, so placement retries the node once the pages are
back; with nothing tearing down it fails at once and says why. Off, it fails
as an internal error and the node is excluded.
- **State**: writes sandbox records to Redis (source of truth for *running* sandboxes). It does
not write routing: client-proxy reads the orchestrator-written `sandbox:routing:{id}` (see
"Sandbox routing records"). Persistent entities (templates, builds, snapshots, teams) live in
Postgres.
- **Secrets**: `/secrets` is the only public surface for secret management (create, list, get,
update, delete). The API authenticates the caller with the customer alternatives above, converts
the authenticated team UUID to the project UUID the backend knows, checks the `customer-secrets`
feature flag, and forwards a metadata-only request over a unary gRPC contract
(`e2b.secretsstore.management.v1`) to the secrets store backend named by
`SECRETS_STORE_BACKEND_GRPC_ADDRESS`. No caller credential, header or client-supplied tenant
crosses that hop, and no response - here or in a log, a span or an error - carries a secret
value. What a caller stores is a runtime marker, not a resolved secret; orchestrator-ee resolves
a marker to a value at sandbox egress, never here. Without a configured address, or with the flag
off, the routes stay registered and answer 403. Responses are `Cache-Control: no-store`, request bodies are capped at 512 KiB, and values
at 64 KiB. A create carries the cap on live secrets the backend enforces: `max_secrets` is one
of the team's limits and resolves with the rest of them in the statement that loads the team (its
`project_limits` value, else the deployment's `default_limits`).
- **Sandbox events and webhooks**: `/events/sandboxes` reads a team's sandbox lifecycle events,
and `/events/webhooks` manages the webhook subscriptions those events are delivered to, along
with each subscription's delivery attempts and delivery statistics. The API stores none of
this: it authenticates the caller, resolves the team's cluster, and forwards the request over
two unary gRPC contracts (`e2b.webhooks.events.v1` and `e2b.webhooks.management.v1`) to the
backend named by `WEBHOOKS_BACKEND_GRPC_ADDRESS`. The team and cluster come from the
authenticated context, never from the request, and a team with no cluster selects the local
one. Without a configured address the routes stay registered and answer 503, so the surface
does not depend on how a deployment is wired. A backend failure the caller cannot act on is
answered with the endpoint's own message; the outcomes that keep their identity are
not-found, a rejected request, the webhook-count limit, and a timeout. A create or update
body carries the webhook's signing secret, so a request rejected before the handler has its
body kept out of the response, the access log and the span. Reads are capped at 100
deliveries per page.
- **Rig passthrough**: admin endpoints under `/clusters/{clusterID}/rigs` manage a cluster's
orchestrator node pools ("rigs"). They list rigs, change rig capacity, list and terminate
instances, and read recent scaling errors. Every handler delegates to the cluster's edge service
(`/v1/rigs/...`) through the per-cluster HTTP client that the cluster sync keeps fresh. The API
holds no cloud credentials and makes no scaling decisions. The caller authenticates with an
admin credential and never holds the cluster's edge secret. Edge status codes pass through unchanged
(400, 404, 409, 501). An edge 401 becomes a 500: it means the API is misconfigured against the
cluster, not that the caller's token is invalid. A cluster with no rig management configured
returns an empty list. The local cluster has no edge deployment and answers 501 for every rig
operation.
- **Extra listeners**: internal gRPC :5009 and edge gRPC :5109 expose `ResumeSandbox` so
client-proxy can wake paused sandboxes on incoming traffic.
- **Outbox** (`internal/outbox`): a River client on the shared database's `river` schema
works jobs that other services enqueue there for work needing the orchestrator
connections. Its job arguments live in `packages/db/pkg/outbox`. The one kind today,
`teardown_team_resources`, stops a deleted team's workloads; dashboard-api enqueues it when
it deletes a project. A guard runs first on every
attempt: a team that exists and is not blocked fails the attempt, and the job retries rather
than kill a live team's sandboxes; a missing team row does not stop it. Then
`kill_sandboxes` kills the team's running sandboxes and succeeds only once the sandbox store
holds none of them in any state, so a sandbox still pausing or being killed makes the job
retry. A sandbox that is not running cannot be paused, so no new snapshot of the team
appears afterwards. Killing through the orchestrator is what removes the sandboxes' store
records, so the job deletes no Redis state itself, and every attempt is safe to run again.
Builds are not cancelled: they end on their own build timeout. An attempt is bounded at
30 minutes; retries back off from 30 seconds to a 15-minute cap, so the 100 attempts last
about a day before River discards the job and keeps it. Volumes, templates, snapshots and
the team row are left alone. Steps report
`api.outbox.steps.finished` and `api.outbox.step.duration`; finished jobs report
`outbox.jobs.finished`, and the backlog reports `outbox.jobs` and `outbox.oldest_*_age`.
`OUTBOX_MAX_WORKERS` (default 10) and `OUTBOX_BACKLOG_INTERVAL` (default 30s) configure it;
shutdown stops it after the sandbox-work drain, before the database and Redis clients close.
- Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to
Loki, with a LaunchDarkly-gated ClickHouse read path (`logs-read-config`) for local-cluster logs
during the log storage migration. `LOKI_URL` is optional: without it the api has no Loki client
and those reads fail until the flag routes them to ClickHouse. `LOGS_READ_CONFIG=true|false` sets
the value the flag falls back to when LaunchDarkly has none (no LaunchDarkly, or the flag not
defined there); a LaunchDarkly value wins. LaunchDarkly feature flags also gate placement
parameters, rate limits, and rollouts.
### Orchestrator (`packages/orchestrator`)
A single Go binary running on every sandbox node (as root). `ORCHESTRATOR_SERVICES` selects its
roles: `orchestrator` (run sandboxes) and/or `template-manager` (build templates). Code lives
under `pkg/`, almost all Linux-only.
gRPC services on :5008 (`pkg/server/`, `pkg/service/`, `pkg/template/server/`, `pkg/volumes/`):
- **SandboxService** — `Create`, `Update`, `List`, `Delete`, `Pause`, `Checkpoint`.
`Update` callbacks are serialized per sandbox. A successful transition out of the live
map closes update admission and joins any admitted callback before stop/checkpoint
work or stopping notifications begin; `Close` repeats that check for crash cleanup.
An update using an inactive sandbox returns gRPC `NotFound`, which the API maps to
HTTP 404 for timeout and network updates. Teardown waits for admitted callbacks;
it cannot preempt one that does not return.
- **TemplateService** — `TemplateCreate`, `TemplateBuildStatus`, `TemplateBuildDelete` (template-manager role only).
- **InfoService** — node identity, roles, capacity, health status (used by API node discovery).
- **ChunkService / VolumeService** — peer-to-peer template chunk serving; persistent volumes.
Process shutdown, triggered by `SIGINT`, `SIGTERM`, `SIGUSR1`, or a service failure, moves
the node to `ShuttingDown` before draining builds and sandboxes. Like `Draining`, this excludes new
placement while existing work stays reachable, and `/health` returns HTTP 200 with status
`draining`. Unlike reversible `Draining`, `ShuttingDown` is terminal for the process;
status overrides cannot enter or leave it. `FORCE_STOP` skips waiting for sandboxes to exit.
The sandbox drain ends only when no sandbox is live and the node's outstanding work (below) is
zero, so a create or a resume-fresh checkpoint in flight, whose sandbox is briefly not live, holds
it until its sandbox goes live and then ends.
After sandbox drain and snapshot uploads complete, sandbox proxy cleanup attempts graceful HTTP
server shutdown with a separate deadline, then forces server closure if needed. Successful forced
closure recovers graceful deadline expiry; other server errors still propagate. Pprof stays
available through service teardown; its final shutdown also has a deadline and forced-close fallback.
`InfoService.ServiceInfo` reports `outstanding_work` for sandbox, template-builder, and
mixed-role nodes. It counts overlapping work holds, not distinct sandboxes or builds, and includes
tracked background persistence and cleanup. The producer reports zero when idle. The API caches
this report and exposes it as required top-level `outstandingWork` in admin node list and detail
responses, where zero also covers a node that has not reported. This observational count does not
by itself authorize node deletion.
`InfoService.ServiceInfo.max_sandboxes` reports the cached node limit for every service role.
Admin `/nodes` and `/nodes/{nodeID}` return required `maxSandboxes`, including zero. Admission,
metrics, and node info share the value refreshed from node-scoped flags every 30 seconds.
Key mechanisms (all under `pkg/sandbox/`):
- **Firecracker** (`fc/`): each sandbox is one Firecracker process in its own cgroup and network
namespace. The FC HTTP API (unix socket) configures machine, drives, network, and snapshots.
Guest metadata (sandbox ID, envd access token hash) is passed via MMDS.
A sandbox may use fewer CPUs than its VM has (`SandboxConfig.max_vcpus` above `vcpu`): the VM
is created with `max_vcpus` vCPUs, a boot puts `maxcpus=` on the kernel command line, and
the `fc_vcpu` threads are held to `vcpu` CPUs by a `cpu.max` quota in a threaded `vcpu` child of
the sandbox cgroup, so the VMM's own threads stay outside it. The quota is in place before the
guest runs an instruction: a boot limits the whole cgroup until the vCPU threads exist and then
moves the limit onto them; a resume confines the paused threads before the VM resumes, falls
back to the whole cgroup, and fails if neither holds. The snapshot's metadata records the VM's
vCPU count (`vcpu_count`, stamped at build and on every pause), so a resume limits below it
without trusting the request or discovering capacity from cgroup threads. A legacy snapshot
carries its stored original vCPU count as `max_vcpus`; if unavailable, it may only resume at its
unchanged `vcpu`.
`/init` carries the count as `cpuCount`
so envd onlines or offlines guest CPUs to match and reports the outcome on `X-Envd-Cpus` and
`/metrics`.
- **Lazy memory / UFFD** (`uffd/`): on resume, Firecracker restores the VM without loading
memory; a userfaultfd handler serves page faults directly from the template's memfile, so only
touched pages are read. An optional prefetcher warms known-hot pages.
- **Copy-on-write rootfs** (`rootfs/`, `nbd/`, `block/`): the template rootfs stays read-only;
writes go to a per-sandbox COW cache exposed to Firecracker as an NBD block device served by
an in-process userspace NBD server. On pause, the dirty blocks are exported as a diff.
- **Template cache** (`template/`): templates are fetched lazily from object storage and cached
on local disk (and optionally on a shared NFS chunk cache, or fetched peer-to-peer from other
nodes before upload completes). Every reader of a cached template holds a pin on it — a
running sandbox, a build, a peer stream, an ancestor lookup — and eviction never closes a
pinned template. A snapshot's own upload holds a pin while it runs and records on the entry
whether the snapshot landed in storage; a layer that did not land is never released, so a
failed upload's local copy stays until its TTL expires. When a pause, or a
checkpoint that resumes a fresh sandbox, publishes a new layer, the layer its sandbox ran
from is marked superseded if this node inserted it for an earlier pause, because no resume
can ask for it again. With `snapshot-cache-release-superseded` and
`snapshot-cache-ancestor-storage-fallback` both on, a superseded layer is released once no
pin holds it, its devices have resolved and its upload has landed, instead of staying until
its TTL expires. A descendant's upload then heals that ancestor's build data from storage,
and writes the same header bytes it would have written with the ancestor still cached.
The sandbox and template caches start empty after a restart: the template cache empties the
build-data directory, and startup reclaim empties the sandbox, template and template-build
scratch directories.
- **Networking** (`network/`): each sandbox gets a slot — a network namespace with a veth pair
and a tap device, unique host-side IP (from a /16), NAT, and per-slot nftables egress firewall
(with SNI/Host-inspecting TCP firewall for domain allow/deny lists). Slots are pooled and
reused; slot indexes are allocated locally against the node's netns state (leftover
namespaces from a previous run are torn down by startup reclaim).
- **Sandbox proxy** (:5007, `pkg/proxy/`): reverse-proxies incoming traffic from client-proxy to
the sandbox's slot IP and requested port over HTTP or configured HTTPS, enforcing per-sandbox
traffic access tokens through the shared ingress policy. HTTPS backends may use self-signed certificates.
It refuses a full-router sandbox with 503 `sandbox_feature_unsupported` before dialing the
slot.
Each admitted lifecycle owns its proxy handlers and backend sockets, including pending dials
and upgrades. Stopping seals admission and interrupts connections; network release waits for
that work to finish. Restarting envd resets the owner while allowing the live lifecycle to
admit a replacement. Retired streams have one second to write their terminal response before
a stalled frontend is interrupted.
Shared proxy shutdown waits for H2C and upgraded connections to close and includes them in
forced connection cleanup. Failed H2C handshakes release their hijacked connections when the
handshake handler returns.
- Writes sandbox lifecycle **events** and cgroup **host stats** to ClickHouse; exports metrics via
OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy
collector remains the fallback primary destination, and configured shadow destinations can mirror
writes during collector/storage migrations without changing sandbox behavior.
Detached sandbox-event publication holds node work until the publisher returns. This covers
delivery attempts, not end-to-end delivery: the ClickHouse target enqueues into an in-memory
batcher, whose flushing remains part of shutdown.
### Envd (`packages/envd`)
The agent inside every VM (started by systemd very early in boot), port 49983, chi + Connect RPC.
- **Process service** (`spec/process/process.proto`): start/list/connect to processes, stream
stdout/stderr, stdin, signals, PTYs — this is what SDKs use to "run code". A start is refused
with `unavailable` while the workload cgroups are frozen for a pause, and the freeze waits for
in-flight starts, so a pause does not capture a child stopped before exec in a frozen cgroup. A
start into a cgroup the guest froze itself is refused the same way. Starts carrying the system
tag are unaffected: they stay in envd's own cgroup, which no freeze covers.
- **Filesystem service** (`spec/filesystem/filesystem.proto`): stat/list/make/move/remove/watch.
- **REST**: `/health`, `/metrics`, `/envs`, `/files` upload/download, `/init` (orchestrator pushes
env vars, access token, metadata and the online CPU count after boot/resume), `/upgrade` (live self-upgrade, below),
freeze/thaw hooks used during pause.
- **Public vs. control-plane routes**: the control routes (`/init`, `/collapse`, `/upgrade`, and the
freeze/thaw hooks) are marked `x-internal: true` in `spec/envd.yaml`, and `/upgrade` — which the spec does not
describe — is listed alongside them in the orchestrator's `pkg/sandbox/envd`. The orchestrator
reaches them over the host network at the sandbox slot IP; the sandbox proxy refuses them with a
404, for every method, so they are not reachable through a sandbox URL. Adding a control route
therefore means marking it in the spec (`go generate` carries the marker into the proxy's
rejection list) — otherwise it ships reachable from the internet.
- **Guest trust** (`internal/host/cacerts.go`): `/init` supplies the desired CA bundle.
envd atomically records its PEM and owned DER identities in `/run/e2b/ca/state.json`,
then publishes a complete replacement of the active bundle with the requested roots
before returning. Ordinary installs copy the existing bundle without decoding it;
recovery first repairs interrupted writes. All supplied identities are owned,
including identical preexisting copies. A background loop resumes saved intent at startup
and retries cleanup every second. It atomically removes retired roots from the active
bundle before rootfs work, so source-file failures cannot delay that cleanup.
It persists each requested certificate as
`e2b-.crt` in `/usr/local/share/ca-certificates`, removes retired managed files,
and retains retirement identities until both bundle and source cleanup succeed.
Superseded cleanup cannot publish an older bundle. Legacy-file removal checks the latest
desired roots under the init lock, preserving hash links to a still-requested certificate.
State survives envd process restart,
not guest reboot. Retired roots may be trusted until cleanup succeeds, and an independent
trust-store rebuild can restore a copy from a customer source file. Applications that
cached their trust store must reload it.
- **Auth**: `X-Access-Token` header checked against a token delivered via Firecracker MMDS;
signed URLs for file endpoints. `/init` is exempt (it is what *delivers* the token), which is the
main reason the proxy refuses it outright.
- **Live upgrade** (`internal/services/process/upgrade.go`): an authenticated `POST /upgrade` lets
the orchestrator swap envd inside a *running* sandbox at resume. It streams the new binary in the
request body and envd `syscall.Exec`s into it **with the same PID**, carrying the workload's
stdio/PTY fds, process table, recently-retained exit codes and filesystem watchers forward via a
tmpfs handover blob. The workload cgroups stay frozen until the post-upgrade `/init` restores the
access token (so no re-adopted process runs unauthenticated), and the handover outcome
(procs/watchers re-adopted, plus any failures) rides back on that `/init`'s `X-Envd-Handover`
header for fleet visibility.
- Scans guest ports and forwards them so any port a user process opens becomes reachable through
sandbox URLs. **`pkg/version.go` carries the released version**, maintained by release-please from
the Conventional Commits merged since the last envd release; do not edit it by hand. The API and
the orchestrator gate features on the envd version recorded in each template build — the
orchestrator's held `/init` is the one gate that reads the version of the envd expected to
answer instead (see sandbox creation) — so a behavioral change ships under a `feat`/`fix`
commit type that cuts a version it can be gated on.
### Client proxy (`packages/client-proxy`)
The stateless edge for all sandbox traffic (port 3002; health on 3003). Terminates
`https://-.` requests (host parsing in `packages/shared/pkg/proxy/host.go`),
looks the sandbox up in the Redis routing record to find the owning node, and reverse-proxies to
that node's orchestrator proxy on :5007 by default. `ORCHESTRATOR_PROXY_PORT` selects a
different downstream port when the node proxy listens elsewhere. If the sandbox is not in the record (paused), it calls
the API's `ResumeSandbox` gRPC — paused sandboxes wake transparently on traffic.
The record is the orchestrator-owned `sandbox:routing:{id}` (see "Sandbox routing records"
below).
The record on a hit, or the `ResumeSandbox` answer on a miss, is the sandbox's route, placement
included. A miss uses the resume answer directly and does not read the record again. A legacy
(no placement) or egress-router route goes to `orchestrator_ip` on the orchestrator proxy port,
never to the placement's ingress endpoint. A full-router route answers 503
`sandbox_feature_unsupported` without a Retry-After header and before anything is dialed. An
unknown mode or a missing `orchestrator_ip` answers the not-found page.
Concurrent requests that miss the record for the same sandbox, port and credentials share one
**resume flight** per client-proxy process: the first request's call runs detached from its
connection under a 90 s ceiling and later requests join it, so one resume RPC serves them all
and a client that disconnects mid-resume does not fail the others. A call whose requests have
all left runs to its end, so a client that comes back finds the sandbox awake, but no further
attempt starts for nobody, and a process keeps at most 64 such callerless calls; beyond that a
departure cancels the proxy's call. That bound and that cancellation apply to the proxy's flights
only: the API's resume entry point lets a create already under way run on past the proxy's
cancellation, so the node finishes the create and the API registers the sandbox inside the
reservation's 90 s window (the create may run until ten seconds before the pending reservation can
be reaped and the registration until five seconds before, so the run-on never lets a fresh
reservation start a second resume of the same snapshot), and the API's shutdown drain waits for
that work, bounded at 95 s. The run-on is granted when the resume is admitted, and an API already
draining admits none: a resume that arrives then keeps its caller's cut, so a create the proxy
cancels mid-drain is cut with it and the reservation lapses. The sandbox otherwise lives its
ordinary timeout or auto-pause, billed, with nobody waiting for it. With
`client-proxy-resume-parking` on (default off), a request whose flight got a still-transitioning
or `Unavailable` answer parks instead of failing: the flight retries
while a parked request waits, under a budget (`client-proxy-resume-parking-budget-ms`, 15 s from
the first retryable failure) that gates only whether another retry starts: every attempt, a late
retry included, runs under its full 90 s ceiling while a caller waits on it, so a parked request
can wait up to the budget plus one attempt. The proxy cuts a retry at the budget's end only once
every caller has left, and that cut ends the proxy's wait, not the create on the node.
A process keeps at most
`client-proxy-resume-parking-lot-size` (512) parked requests, shedding the rest. A resume the
control plane cannot carry out right now (no capacity, no node, the API unreachable, the budget
spent) answers **503** with `Retry-After: 2` and no error code; a request shed because the lot was
full answers the same **503** without the header and with `proxy_parking_lot_full` as its
`error_code`, so a client, or one behind an ingress that strips the header, still tells the two
apart. The SDKs do not act on the header either way: their status-based retry
wraps only the control-plane API client, and sandbox traffic through the proxy is retried only on
a connection failure, never on a 503. The omission is for hand-rolled retry loops and browsers
that honour the header, which would otherwise come back in one wave while the lot is still full,
and for a client that does the SDKs' jittered backoff on its own; a placement no retry helps, which
the API names in an `ErrorInfo` detail on the RPC status, is not retried and answers without the
header, with the status the API carried: **503** for a feature no orchestrator in the cluster
supports, **500** (`sandbox_create_refused`) for a sandbox the node refuses to create, such as a
stored build whose vCPU count is outside the node's bounds; a sandbox still pausing
answers **409** with the header on a user port and the same page as **503** on the envd port (the
SDK reads an envd 409 as "already exists"); a request whose client left before it was routed
answers 408.
### Dashboard API (`packages/dashboard-api`)
A separate REST service (port 3010, spec `spec/openapi-dashboard.yml`) consumed by the web
dashboard, not the SDK: template tags, build listings, and admin profiles. Its admin
routes accept either a shared admin token or
a short-lived service JWT verified against the configured admin JWKS. Team-scoped template and build read routes accept either dashboard user auth or
team API key auth (`X-API-Key`). Its workspace-agnostic `/v1/management` operations are defined in the
same dashboard OpenAPI contract and registered on the existing router. Their `AdminJWTAuth`
OpenAPI security scheme accepts only short-lived service JWTs verified against the workspace-api
`/.well-known/jwks.json` endpoint, with accepted signing methods derived from each JWK's required
`alg` metadata. Issuers and audiences are configured through the JSON `ADMIN_AUTH_PROVIDER_CONFIG` value —
the same config shape as `AUTH_PROVIDER_CONFIG`. Talks to Postgres and ClickHouse; never talks to
orchestrators. The web dashboard itself is a separate repository (github.com/e2b-dev/dashboard)
published as an image; E2B Embed runs it on port 3001 beside dashboard-api.
An issuer normally configures at least one accepted audience, binding a JWT to its intended target.
An issuer may omit `audiences` only with no `audienceMatchPolicy`; this deliberately disables
audience matching while issuer, signature, and temporal-claim verification remain required.
The `/v1/management` operations are the cluster's half of a contract the workspace residency owns:
project upsert (a project is a `public.teams` row created from a caller-supplied UUID; the tier is
assigned once at creation from a local default and no push moves it; an optional cluster ID is
stored on the project, with omission preserving its current assignment; a changed slug renames the project and, in the same transaction, gives each of its namespaced
template aliases under the new slug as well, keeping the old ones and leaving an alias the new
slug already holds as it is), per-member projection,
and limit sync (into `project_limits`, which the team lookups read in preference to `default_limits`). All are
idempotent, because the caller is level-triggered and retries. `PUT
/v1/management/projects/{projectID}/members/{userID}` applies the desired presence for one user,
gated by a monotonic per-project/user revision stored in `projection.project_members`; duplicate or
older revisions succeed without changing target state. `PUT
/v1/management/projects/{projectID}/limits` is gated the same way, by a monotonic per-project
revision in `projection.project_limits`: the caller raises it whenever the limits it resolved for a
project change, and a delivery at or below the recorded revision is dropped and still answers 204.
The ledger and the values in `public.project_limits` advance in one transaction, so a revision is
never recorded without the values it admitted. Both fences are the target's, and both exist for the
same reason — the caller can only fence what it sends, so two deliveries in flight arrive in
whichever order the network gives them and the older one has to be refused where it lands. A present projection includes that User's
OIDC issuer/subject identities. Every projected user has at least one identity, and an identity
already owned by a different user returns 409. A revocation removes only that User's
`users_teams` row; projected Users and identities are retained. Membership writes live in
`internal/management` with their post-commit cache eviction rather than in the handlers: auth
caches member authorization, so each accepted command invalidates that User's authorization for
the Project after commit.
`PUT /v1/management/projects/{projectID}/block` applies a revision-fenced block
state. The `projection.project_blocks` revision and `public.teams.is_blocked` /
`blocked_reason` commit together. Older or duplicate revisions leave stored state
unchanged. Every accepted delivery invalidates the team auth cache; eviction
failures return an error so retries can finish invalidation after commit. The API
uses this state when admitting billable work. It is the only route that writes the
team's block state.
Limit and block deliveries may carry `decided_at`, when the caller decided that
revision. The ledger row stores it beside the revision. Each delivery the ledger
accepts records the seconds since then in `dashboard-api.management.apply_lag`, by
`projection` (`project_limits`, `project_blocks`); one without `decided_at` counts
in `dashboard-api.management.origin_unknown` instead, never as zero lag. Dropped,
refused and failed deliveries record nothing.
The management surface also owns a replay-safe cluster lifecycle. A caller registers a stable
cluster UUID, assigns it only to the named project, detaches that exact assignment before provider
cleanup, and deletes the cluster only after no project references it. Replaying the same
registration or assignment succeeds. Registering the cluster again with the same name and auth
organization may change its endpoint, TLS setting, sandbox proxy domain and token, for example when
the cluster moves to a new domain; the API rebuilds that cluster's clients on its next sync and gives
sandboxes the new domain. A changed name or auth organization, a different assignment, an inexact
detach, or deletion of a referenced cluster returns a conflict. The
assignment does not check the project's tier; the caller decides which projects may use a cluster.
A cluster first registered through this surface starts without deletion protection; one created any
other way starts protected, a later registration never changes it, and deleting a protected cluster
or checking its destroy readiness returns a conflict.
A caller that deletes a cluster's projects before the cluster first lists them with
`GET /v1/management/clusters/{clusterID}/projects`, which returns every project whose team is
assigned to the cluster however it was assigned, leaving out deleted teams, and records the deletion with
`PUT /v1/management/clusters/{clusterID}/deletion` before deleting any of them. That request
locks the cluster row, returns a conflict while deletion protection is on, and sets
`clusters.deletion_requested_at`; it succeeds for an absent cluster and on repeat. From then on a
check constraint keeps deletion protection off, and a trigger on `teams` refuses to newly assign a
team to the cluster (`cluster_assignment_cluster_deleting` from the assignment route, a conflict
from an upsert carrying the cluster ID). The trigger takes a key-share lock on the cluster row
before it checks, so an assignment either commits before the deletion is recorded, where the
caller's next listing finds it, or is refused. Deleting each project clears its team's cluster,
the cluster delete clears it from any team deleted earlier, and then succeeds.
The shared error response supports an optional `error_code`, defined by the dashboard OpenAPI
`ErrorCode` schema. Registration and assignment rejections currently populate it; other errors
may omit it. Callers use that stable code rather than the human-readable message, and treat
uncoded or unknown reasons as generic errors.
`DELETE /v1/management/projects/{projectID}` soft-deletes any project, including a member's
default project; whether a project may be deleted is the caller's decision. `envs`,
`snapshots` and `volumes` reference `teams` with `ON DELETE NO ACTION`, so the team row
stays. One transaction locks the row (404 if it is missing), sets `teams.deleted_at`,
blocks the team (a CHECK keeps every deleted team blocked), clears its
cluster assignment, scrubs its name and email to `DELETED` and its slug to
`deleted-` (releasing the old slug for reuse), deletes its memberships and API
keys, soft-deletes its templates, snapshot templates and paused-sandbox snapshots the way a
template deletion does (releasing their aliases and build tracking), and enqueues the API's
`teardown_team_resources` outbox job with River's `InsertTx`, so the teardown runs exactly
when the deletion commits. dashboard-api holds an insert-only River client for this; it works
no jobs. After commit it evicts the deleted keys' auth cache entries best-effort; the team's
other cached entries refresh or expire on their own. Repeating the call answers 204 and runs
every step again, keeping only the first deletion's time: it re-blocks and re-scrubs the team
and clears its cluster, undoing a block reason, rename or cluster assignment (from the cluster
route or an upsert carrying a cluster ID) written by a delivery that passed its
deleted-project check just before the first call committed; it deletes memberships, keys and
envs created since (a sandbox that paused before the teardown killed it leaves a snapshot);
and it enqueues the teardown again, which is a no-op while one is still open. Afterwards a
dashboard-api middleware, running after authentication, answers 404 to every write under
`/v1/management/` or `/admin/` that names the project by `projectID` or `teamID` (project
upsert, member, limits and block deliveries, cluster assignment and detachment, admin ban);
the project DELETE is exempt so a retry still reaches it. Shared auth refuses a
membership or API key still left for the project with 403, and the API leaves it out of a
user's team list. Volumes, builds and stored artifacts are not reclaimed.
## Data stores
| Store | Owner packages | What lives there |
|---|---|---|
| **PostgreSQL** | `packages/db` (goose migrations, sqlc) | Durable control-plane state: `teams`, `users`, `default_limits` (the one row of quota defaults), `project_limits` (per-team quota overrides pushed in by the owning service; the team lookups read it in preference to `default_limits`), `tiers` and `addons` (read for limits only by the legacy `team_limits` view), `envs` (templates), `env_builds` (build rows: vcpu, ram_mb, status, versions), `env_aliases`, `snapshots` (paused sandboxes), `team_api_keys`, `volumes`, `clusters` |
| **Redis** | API, client-proxy, orchestrator | Ephemeral runtime state: running-sandbox store (source of truth), sandbox→node routing catalog, team/template/snapshot caches, rate limiting, P2P chunk peer registry |
| **ClickHouse** | `packages/clickhouse` | Time-series/analytics: `metrics_gauge`/`metrics_sum` (written by the OTel collector), `sandbox_events`, `sandbox_host_stats` (written by orchestrator), team metrics, and optionally `sandbox_logs` during the log migration. Read by API and dashboard-api |
| **Object storage** (GCS/S3/local, `packages/shared/pkg/storage`) | orchestrator, template-manager | Template & snapshot artifacts, keyed by build ID: `{buildID}/memfile`, `{buildID}/rootfs.ext4`, `{buildID}/snapfile`, `{buildID}/metadata.json` + `.header` index files |
The API and dashboard-api share `packages/db/migrations`, tracked in PostgreSQL's
`_migrations` table. This includes `env_defaults`, which migration `20260915204740`
creates on fresh databases and adopts where it already exists, preserving its data.
Any existing `_dashboard_migrations` table is retained as historical bookkeeping and
requires no updates. Run the db-migrator before deploying updated services to a
database that lacks the table; no separate dashboard migration runner is needed.
Rolling back this migration retains the table and its data.
The same database holds River's job tables in the `river` schema, which a goose migration
creates. The db-migrator applies the goose migrations and then River's own, under one advisory
lock; `make migrate` runs it, because the goose CLI cannot apply River's. At startup the API and
dashboard-api refuse a database whose goose version is older than the one they were built against,
or whose River migrations are not current. The API works jobs from that schema (see its Outbox).
A template and a paused-sandbox snapshot have the **same artifact shape** — a snapshot is just a
new build whose memfile/rootfs are stored as diffs against the template it came from (diff chains
are resolved through the `.header` files).
## Core flows
### Sandbox creation
```mermaid
sequenceDiagram
autonumber
participant C as SDK
participant API as API
participant R as Redis
participant O as Orchestrator (chosen node)
participant FC as Firecracker
participant E as envd (in VM)
C->>API: POST /sandboxes {templateID}
API->>API: auth team, resolve template alias → ready build (Postgres/cache)
API->>API: mint envd access token (build's envd must support secured access)
API->>API: best-of-K placement → pick node
API->>O: gRPC SandboxService.Create(SandboxConfig)
O->>O: fetch template (local cache / NFS / object storage)
O->>O: acquire network slot + NBD rootfs overlay + uffd memory
O->>FC: load snapshot, resume VM
O->>E: POST /init (env vars, access token) — held until envd reads it (older envds: retried until ready)
E-->>O: 204
O-->>API: Create OK
API->>R: store running sandbox
API-->>C: 201 sandbox {sandboxID, domain, envdAccessToken}
```
The API blocks on the gRPC `Create`, which itself blocks on envd's `/init` — when the client
gets a response, the sandbox is fully usable. Fresh creates are internally a *resume* of the
template's base snapshot (cold boots happen for filesystem-only templates and builds, or when
an explicit resume requests one — see pause and resume below; template creates never do).
A create that leaves `network.allowPublicTraffic` unset or null uses the project's
`ProjectSettings.SandboxSettings.DefaultAllowPublicTraffic`, read from the cached team.
A missing key, including a team cached before settings existed, keeps public traffic. An explicit value wins.
With no `network` object, a public sandbox stores no network config and a private one stores only
the ingress setting. Either way a private sandbox gets a traffic access token at create, and
resume, auto-resume and fork keep the setting saved with the sandbox rather than reading the
project default again.
V2 create idempotency keeps the former `allowPublicTraffic: true` fingerprint for an omitted
flag on unconfigured projects, so retries across API versions still replay. Configured projects
distinguish omission from an explicit boolean. The first configuration change can make an older
key return 409 until its idempotency record expires; it does not alter the original sandbox.
How that `/init` reaches envd depends on the envd expected to answer it. Behind the
`envd-init-held-request` flag, and for an envd at or above 0.10.0, the orchestrator sends one
**held request**: it dials the guest first, writes a single `/init` with `Expect: 100-continue`
over that connection, and withholds the body — timestamp included — until envd's handler asks
for it, so the body is minted after the request has cleared the guest's accept queue and
dispatch rather than before. The request waits for envd up to the caller's envd-wait budget and
is re-sent only after a transport failure (a refused connect, or a connection that dropped under
it), never on a timer. Older envds, and every sandbox with the flag off, keep the **legacy loop**:
a complete `/init` posted and retried until envd answers. The gate reads the envd version it
expects to answer, in this order: the version a confirmed live upgrade delivered in this life,
else the version an offline upgrade swapped onto the rootfs before this boot, else the version
the running envd last reported on `/init` when it parses as a version, else the running version
the snapshot recorded at pause, else the version baked into the template build. The first two
are binaries the orchestrator staged itself; the rest are the guest's word, which only ever
chooses between the two loops for that guest.
#### Opt-in request idempotency
`POST /v2/sandboxes` supports an optional, team-scoped `Idempotency-Key` to prevent retries from
repeating sandbox creation while a Redis reservation exists. A generic executor owns reservation
and response replay across API replicas. Participating handlers parse and authorize the request,
then call `idempotency.Execute(c, requestContent, operation)` with all mutations inside the callback.
The executor buffers finite HTTP responses, so streaming and connection upgrades cannot use it.
##### Admission and request identity
The `api-idempotency-routes` JSON flag enables participating handlers by HTTP method and Gin route
pattern. For sandbox creation:
```json
{"POST /v2/sandboxes": true}
```
The flag defaults to null, which disables idempotency. The idempotency middleware bypasses requests
without a key, an authenticated team, or an enabled route. Authentication, schema validation,
rate limits, and blocked-team checks apply before the executor, including on retries. A key is
1 to 255 lowercase ASCII letters, digits, or hyphens (`a-z`, `0-9`, `-`), so a lowercase UUID is
valid as is. Only the route's OpenAPI schema defines that format, and it applies regardless of the
flag, so a participating route must declare the header with the same schema. The validator checks
only the first header value; the generated parameter binding rejects a repeated header before the
handler runs.
The request fingerprint hashes the HTTP method, concrete URI including query string, and
handler-supplied JSON-serializable content with SHA-256. Handlers include behavior-affecting
headers or caller information in that content. Sandbox creation fingerprints its parsed request
after schema defaults, so whitespace and map ordering do not change identity; array order and
parsed field values do. Template aliases and business configuration resolve during execution.
##### Reservation and replay
The executor reserves a Redis hash keyed by team ID and the SHA-256 digest of the idempotency key.
The namespace spans participating routes within a team. An atomic reservation script selects one
owner across API replicas and stores the request fingerprint, a random owner token, and the
response deadline in `expected_response_at`. The owner token also guards completion and cleanup,
so an expired request cannot modify a replacement reservation.
Reservation and execution share the request context's deadline. The response deadline adds a
five-second persistence budget and uses Redis's clock for comparison. The executor reads the
reservation and server time together, then selects the response:
| Reservation state | Result |
|---|---|
| New reservation | Run the operation callback. |
| Different request fingerprint | Return 409 with `error_code: idempotency_request_mismatch`. |
| Matching fingerprint with a cached response | Replay its status, body, and allowlisted headers. |
| Matching fingerprint without a response, before its deadline | Return 409 with `error_code: idempotency_in_progress`. |
| Matching fingerprint without a response, with an elapsed or unreadable deadline | Return 422 with `error_code: idempotency_outcome_unknown`. |
| Redis failure, invalid retention configuration, or missing request deadline | Return 503. |
Retries against a reservation do not wait for or repeat the operation. The `error_code` field
distinguishes the two conflict cases without requiring clients to parse messages. An unknown
outcome means the API cannot establish the result, so retrying with a new key can duplicate work.
A late response from the owner remains replayable while the reservation exists. Admission and
retry errors do not replace the owner's cached result.
##### Response storage and delivery
The executor captures the handler's status and body, then attempts to persist them before
delivery. Redis stores one JSON value containing the status, base64-encoded body, allowlisted
headers, and optional retryability value. Base64 preserves arbitrary response bytes; the separate serialized
buffer lets Redis finish a write after the HTTP capture buffer has been released. The cache includes
handler errors as well as successful responses, keeping retries attached to one attempt without
inferring which errors permit safe re-execution.
The handler edits response headers in place while the executor buffers status and body writes.
`Content-Type`, when present, enters the cache. Replay applies that header over the current
request's headers, preserving fresh request IDs, rate limits, and cross-origin resource sharing
(CORS) values. The capture pool reuses allocations up to 64 KiB and erases body contents before
reuse because responses can contain credentials.
Persistence has a detached five-second wait budget and retains request context values for
tracing. If it fails, the API logs the failure and delivers the handler's response: a caller should
receive a known creation result even when Redis cannot store it. Subsequent retries can replay
only a stored response; otherwise they receive the pending or unknown-outcome error.
##### Cancellation and retention
Before execution starts, cancellation or a reservation error triggers best-effort cleanup. The
reservation worker waits for its Redis call to return before attempting deletion because socket
I/O can outlive request cancellation. Cleanup gets its own five-second timeout and deletes only
an unfinished record with the same owner token. Failed cleanup or an ambiguous Redis write can
leave the reservation until expiry.
Once execution starts, cancellation, a panic, or failed persistence leaves the reservation in
place. An operation can have side effects before writing any response, so releasing its key
would allow duplicate work. The executor does not reconcile an unknown operation outcome.
The `api-idempotency-ttl-seconds` flag sets retention, defaulting to 86,400 seconds when
unavailable. Configured values must be positive integers. The executor sets the time to live
(TTL) at reservation; completion, retries, flag changes, and deleting the sandbox do not extend
or remove it. Operators must choose retention longer than the operation window: expiry permits
a new owner even if the first operation is still running. Redis eviction or data loss, a different
key, and feature bypass also permit another execution. Redis capacity and access controls must
account for cached response volume and the sandbox credentials those responses can contain.
#### Sandbox creation retry hints
`POST /v2/sandboxes` answers errors with `Should-Retry`, exposed through CORS. The header states
whether the condition behind the error may clear, so that retrying the request could succeed:
`true` for a transient failure such as missing capacity, a rate limit, or a creation still in
progress (retry after `Retry-After` when present, otherwise with exponential backoff); `false`
for a failure a retry cannot fix, such as a malformed request or a configuration fault, and for
an outcome that is unknown, where a retry could create a second sandbox. The header says nothing
about idempotency keys. The executor records every handler result under the key, success or
error, and replays it for as long as the record lives; the caller decides whether a retry keeps
the key or takes a new one, and the error code names the situation that decision depends on.
Each error carries its classification in `APIError.Retryable`, a `Retryability` enum, and the
error writer (`httperror.Write`, behind `SendAPIError`) sets the header from it: `CanRetry` and
`NoRetry` map to `true` and `false`. `UndefinedRetry` (the zero value) leaves the header unchanged:
an absent header stays absent, and a value set earlier is preserved. Callers that discard a
response must explicitly remove its retry hint.
The v2 handler, the rate limiter on that route, and the idempotency executor classify their
errors. Other creation routes share the creation path, so they reset `Retryable` to
`UndefinedRetry` before sending. Successful responses omit the header, and so do admission
errors that carry no classification. A recovered panic removes any hint the handler wrote,
because the hint described a response the panic replaced. Idempotency stores the header with
the cached response and replays it, including its absence.
| Response | Should-Retry |
|---|---|
| Successful creation or replay | absent |
| Authentication/schema/parameter validation errors (including 400, 408, and 5xx), blocked-team rejection, oversized body, or recovered panic | absent |
| Handler input validation, permissions, or missing template/tag | `false` |
| Idempotency key rejected by the middleware's own check, or idempotency configuration fault (retention flag unreadable or not positive, no request deadline) | `false` |
| Request rate limit, with or without a key | `true` |
| Idempotency reservation/lookup failure, or original request still in progress (`idempotency_in_progress`) | `true` |
| Idempotency request mismatch, or unknown outcome of the original request (`idempotency_outcome_unknown`) | `false` |
| Cancellation after acquiring a new idempotency reservation but before execution | `true` |
| Template/volume lookup failure, token generation failure, concurrency reservation error/limit, or unavailable cluster information | `true` |
| Invalid stored build versions or unsupported orchestrator capability | `false` |
| No available/compatible node, only capacity refusals, or placement timeout before any non-refused create attempt | `true` |
| Failed/uncertain create RPC, timeout after a create attempt, or sandbox-store write failure | `false` |
Placement retains uncertainty from earlier attempts even when the last node reports a permanent
error. A sandbox-store write error follows a successful create RPC and only starts asynchronous
cleanup, so a retry could run alongside the sandbox being removed. Classification is not based
on HTTP status alone.
### Sandbox traffic
```mermaid
sequenceDiagram
autonumber
participant U as Client
participant CP as client-proxy :3002
participant R as Redis catalog
participant API as API
participant OP as orchestrator proxy :5007
participant E as envd / user process
U->>CP: https://3000-i7fa3.domain
CP->>CP: parse host → port 3000, sandbox i7fa3
CP->>R: GetSandbox(i7fa3)
alt running
R-->>CP: node IP
else paused / unknown
CP->>API: gRPC ResumeSandbox(i7fa3) — one flight shared by concurrent requests
API-->>CP: orchestrator IP and network placement (after resume), or 409 / 503 + Retry-After to the client
end
CP->>OP: forward to http://nodeIP:5007
OP->>OP: lookup sandbox, check traffic access token
OP->>E: http://slotIP:3000 (via veth/tap into VM)
E-->>U: response
```
### Sandbox routing records
client-proxy resolves the node IP of a sandbox from one routing record in Redis, written by the
orchestrator. Its JSON shape (`sandbox_catalog.SandboxInfo` in `packages/shared/pkg/sandbox-catalog`):
`orchestrator_id`, `orchestrator_ip`, `execution_id`, `sandbox_started_at`,
`sandbox_max_length_in_hours`.
| Record | Key | Writer | Written | Deleted |
|---|---|---|---|---|
| Orchestrator-owned | `sandbox:routing:{sandboxID}` | orchestrator, `packages/orchestrator/pkg/routing` | on `MarkRunning` (sandbox enters the live map, envd is ready) | on `MarkStopping` (kill, pause, checkpoint, crash) |
**The orchestrator-owned record is the routing source.** The orchestrator writes
`sandbox:routing:{id}` on `MarkRunning` and deletes it on `MarkStopping`. A failed write is logged
and counted (`orchestrator.routing.publish.total{result=error}`); the sandbox keeps running. Build
sandboxes are skipped. The delete is guarded by `execution_id` in a Lua script, so a stale
lifecycle never removes the record of a newer execution. On a miss client-proxy goes to the
auto-resume path (`ResumeSandbox` gRPC to the API).
The TTL of the record is `sandbox_max_length_in_hours` from the write time. The record is
deleted earlier in every normal stop path.
A routed sandbox adds `lifecycle_id` (its Firecracker process) and `network_placement` with the
mode, Router, Portable IP and ingress endpoint (`ingress_address`, `ingress_port`). The endpoint is
the Router for full-router and the worker for egress-router. The record carries location only. The
orchestrator owns the Router configuration, which never travels through this record. An absent
placement means the legacy per-worker network. The orchestrator publisher does not write a
placement yet.
The API keeps the placement returned by the winning orchestrator create on its own sandbox record
in Redis (`networkPlacement`), and callers that join the same start get the same placement. A new
execution carries only its own create's placement. On a completed checkpoint response, the API
records the resulting VM's placement together with the return to Running. If the caller abandons
the checkpoint, the node can finish without returning its placement to the API, which can retain
the old placement. Recovery of that outcome is not implemented.
A `ResumeSandbox` answer for a routed sandbox carries `network_placement` for the execution it
reports: the winning start's placement for a fresh or joined resume, and the record's placement
for a sandbox already running or back to Running after a pause or checkpoint wait. A record
without a placement is a legacy sandbox and is answered with `orchestrator_ip` alone, as before.
A placement from another execution, an unknown mode, or a full-router placement without Router,
address and port fails the RPC. A full-router answer has an empty `orchestrator_ip` and never
looks up an orchestrator. Egress-router and legacy answers still carry `orchestrator_ip`.
client-proxy reads this placement and refuses full-router routes, as does the orchestrator proxy.
### Volume content
Persistent volumes (`packages/orchestrator/pkg/volumes/`) are managed through the control-plane
API (`POST/GET /volumes`), but their **content** — reading and writing files — is served by a
separate volume-content API (belt, `e2b-dev/belt`) that the SDK talks to directly, not through the
control-plane API. The API's role is to mint the credential and tell the SDK where to send content
traffic.
```mermaid
sequenceDiagram
autonumber
participant U as SDK
participant API as API
participant PG as PostgreSQL
participant VC as volume-content API (belt)
U->>API: POST /volumes (create) or GET /volumes/{id}
API->>PG: persist / load volume row
API->>API: mint JWT (aud = https://api.#lt;domain#gt;) resolve domain
API-->>U: { volumeID, name, token, domain? }
Note over U: domain is returned only for BYOC teams. SDK stores it and falls back to api.#lt;E2B_DOMAIN#gt; otherwise
U->>VC: /volumecontent/{id}/... at api.#lt;domain#gt; Authorization: Bearer token
VC->>VC: verify token (audience must match its own origin)
VC-->>U: file content
```
- **Domain selection.** The token's audience and the content host are the same origin,
`https://api.`. For teams on a **custom (BYOC) cluster** (`team.ClusterID` set), the API
returns that cluster's domain (`cluster.SandboxDomain`, resolved in
`handlers.volumeContentDomain`) so content traffic goes to the BYOC cluster's edge instead of the
control-plane host. For teams on the default cluster the response omits `domain` and the SDK uses
its configured default (`api.`); the audience then uses the deployment's `DOMAIN_NAME`.
- **Token.** A short-lived JWT (`handlers.generateVolumeContentToken`, config in
`cfg.VolumesTokenConfig`) signed by the API, scoped to the team and volume, presented as a bearer
token on every content request. Its `aud` claim is `https://api.`, so a token minted for
one cluster's origin is not accepted by another.
### Pause and resume
- **Pause**: API records a snapshot row in Postgres, then gRPC `Pause` to the node. The
orchestrator pauses the VM, snapshots it, diffs memory (dirty-page tracking) and rootfs (COW
cache) against the template, caches the snapshot locally, and uploads asynchronously to object
storage (with a retry budget). The orchestrator deletes the sandbox's routing record on
`MarkStopping`. Once the API acquires the pause transition, the snapshot DB upsert and the node
RPC share a detached 80-second budget; the Redis transition key has a 95-second TTL.
The node inherits that deadline for admission and snapshotting. Before anything destructive
the node also refuses, retryably, a pause whose capture the filesystem holding its build
directory cannot take: guest memory plus the rootfs cache's size plus a snapfile allowance,
doubled when template storage is a directory on that filesystem, counting captures already
admitted and not yet on disk, with the headroom the `pause-admission-disk-headroom-mib`
flag names left free. The check is off by default; the flag's fallback reads the
`PAUSE_ADMISSION_DISK_HEADROOM_MIB` environment variable, which is how E2B Embed turns it
on. Caller cancellation cannot
abandon the snapshot or its RPC result; terminal build-status writes have a separate detached
ten-second budget. Snapshot uploads and sandbox teardown retain their separate background
lifetimes. Shutdown drains pauses still in flight — after the HTTP and gRPC drains, before the
database and Redis clients close — so a deploy cannot cut a snapshot short. The drain stops
admitting pauses first, so the eviction sweep cannot start one behind it; a refused pause
leaves the sandbox running for another replica. The wait is bounded by the pause budget plus
the terminal build-status write it falls through to, plus a short grace.
- **Deferred rootfs export** (gated by the `deferred-rootfs-export` flag in
`packages/shared/pkg/featureflags`): instead of diffing the rootfs on the pause critical
path, the orchestrator ejects the writable COW cache during pause and returns, then seals it
into the rootfs diff (reflink) in the background. This moves the rootfs-diff latency off the
pause, but the local snapshot's rootfs body isn't materialized until the seal finishes, so the
async upload — and any origin-node resume/prefetch that reads the rootfs diff — waits on the
seal. A seal failure is permanent (it never re-runs), so the upload fails fast rather than
retrying.
- **Per-sandbox pause rate limit** (gated by the `sandbox-pause-rate-limit` flag,
`{"limit": N, "period_s": S}` with S at most 600): before the pause transition, the pause
endpoint reads the sandbox record and counts the pause against a budget of N pauses per period
(GCRA, burst N) in Redis under `ratelimit:sandbox-pause:{sandboxID}`. An exhausted budget
answers HTTP 429 with `Retry-After` and leaves the sandbox running. The count is per
execution ID, so concurrent requests for one execution use one slot; a record that is
missing or already pausing is not counted, and Redis errors let the pause through. The
pause is pinned (`ExpectExecutionID`) to the record the limiter read, so a resume landing
between the check and the pause sends the new execution back through the limiter instead
of pausing it uncounted. The key expires once the budget is full again, at most one period
after the last counted pause. Timeout auto-pauses are not limited.
- **Resume**: same path as creation, but placement prefers the **origin node** — if the snapshot
is still in its local cache, resume avoids any object-storage reads. The origin node gets
the same hugepage-limit check as any other node, so a full origin is skipped. `Checkpoint` is a
pause+resume in place used to persist state while keeping the sandbox running.
- **Post-checkpoint re-init and the pre-pause cut**: after an in-place checkpoint the sandbox
keeps running, so the orchestrator re-sends `/init` in the background to re-sync the guest
clock. The re-init is best-effort, bounded by the envd-wait budget and by the VM's liveness,
and published on the sandbox before it starts, because a held re-init whose `/init` landed
after the pre-pause freeze would have envd serve it behind that freeze and thaw the workload
the snapshot is about to capture. A Pause therefore cuts the published re-init before it
freezes — a held one is cancelled wherever it is, one that has not yet chosen its loop is
refused when it does, one already on the legacy loop is left to run as before — and holds
publication, so no fresh re-init starts between the cut and the freeze. The hold is released
once the freeze lands, or once the Pause gives up before it; in the latter case the Pause's
cleanup restarts the re-init it cut, since the VM never paused and the clock re-sync is still
owed. The **prefetch harvest** that follows a pause or an in-place checkpoint resumes a
network-isolated throwaway of the new snapshot to record which memory pages its resume
touches; the throwaway's held `/init` asks envd to skip the thaw, so the workload never runs
and the trace holds only the resume's own faults. A throwaway the gate routes to the legacy
loop sends no such request, and envd thaws inline as before.
- **Resume-fresh checkpoint ownership**: when the in-place route is not available, a checkpoint
snapshots the sandbox and resumes a fresh VM from the new build under the same execution. From
the reservation on, the old VM is lost on any failure and the resume is the only way the sandbox
survives, so that work belongs to the node, not to the caller. The node claims the sandbox for
the checkpoint at the reservation, while its live entry still exists, and holds the claim until
the resumed sandbox is live or, on failure, until the rollback has run and the reservation is
released, so no interleaving finds the sandbox neither live nor in flight. Everything past the
reservation runs detached from the caller's cancellation. The snapshot phase and the resume
phase (template, resume, envd upgrade, registration) each have their own ten-minute budget,
observed by the steps that take the context (the Firecracker pause on this route does) and
between steps: a step that does not take it outlives the budget. Only a Delete for the sandbox
or the orchestrator's shutdown ends the checkpoint early. A Delete that arrives during the window
cancels the checkpoint and waits for
it, so the rollback is that delete and no VM the API stopped tracking goes live later. It
answers success when its cancellation is what ends the sandbox's lifecycle, and not found when
the checkpoint ended for its own cause or never held the sandbox, even when the Delete's own
deadline expires first; a reservation still being taken at that deadline counts as held, and the
acknowledgement below covers it. A Delete answering for a checkpoint that ended for its own
cause or never held the sandbox is first handed any sandbox of the checkpoint's that is live
again, the original the registry put back after a failure in the reservation, and answers
success for it. A publication the cancellation did not reach can still go live, and the
checkpoint then kills it with the Delete's reason: the Delete records its acknowledgement with
its final read of the live map under the operation's mutex, and the publication reads that
acknowledgement after its insert under the same mutex, so one of the two always sees the other.
While a checkpoint is in flight, Pause answers the same retryable refusal a busy node gives,
and a second Checkpoint answers operation-in-progress. The checkpoint marker is the sandbox's
own and is shared by both routes, so an in-place checkpoint and a resume-fresh one on the same
sandbox refuse each other. Shutdown cancels a detached checkpoint after the sandbox drain, as
the gRPC server closes, so it cannot hold a handler open for minutes. The kill accounting
follows the cause: a checkpoint a Delete ended records the kill and publishes the killed event
and counter with the Delete's reason, as the in-place path does for its own teardown, whereas
a resumed VM torn down by a budget or by shutdown is not labelled killed.
Two limits remain. The in-place checkpoint path is still bound to the caller's context, so
this ownership covers only the resume-fresh route. And the API still marks the build failed
when its own request ends before the node answers, even when the node's work later succeeds,
because the outcome is unknown to it; it restores the sandbox to Running rather than kill it.
A caller timeout therefore no longer means the snapshot or the VM creation stopped, only that
the build row records a failure the node may have gone on to complete.
- **Fork**: checkpoints the original in place, then starts each fork from that snapshot as a
new sandbox rather than a resume — like a sandbox created from a snapshot template — so
placement spreads the forks across nodes instead of preferring the original's node.
- **Filesystem-only checkpoint**: `memory: false` on snapshot-template creation exports the
rootfs and no memfile, resumes the same Firecracker process, and produces a template whose
sandboxes cold-boot; the source keeps running. Routed by its own flag,
`filesystem-only-checkpoint`, independent of `in-place-checkpoint` (no memory is exported, so
the sync-WP tracking, the Firecracker release gate and the balloon gate play no part). The
API refuses it before the sandbox leaves Running when the flag is off for the team (400,
`snapshot_filesystem_only_disabled`) or when the sandbox's node runs an orchestrator
release below the one that reads the request field (409,
`snapshot_filesystem_only_unsupported_node`, the placement feature floor
`filesystem-only-checkpoint`; a checkpoint cannot move to another node, so the options are a
full snapshot or a pause and resume). The request is never downgraded to a memory snapshot.
The snapshot row records the kind only once the build is ready, and the pause and fork paths
keep the same rule, so a refused or failed request leaves the previous build resumed as what
it is. Each snapshot build also records its own kind (`env_builds.filesystem_only`) when it is
created. A sandbox created from a snapshot template reads that, because the source's row
follows its later snapshots, and placement holds such a cold boot to the
`fs-only-resume-cpu-model` CPU model as it does a filesystem-only resume.
- **Explicit filesystem-only resume**: `memory: false` on resume/connect demands a cold boot
(`RebootSandbox`) even when the snapshot includes memory, as a self-serve rescue when the
restored memory state is unusable. Gated per team by the `fs-only-resume-api` flag; when off
the request is rejected with an explicit error, never silently downgraded to a memory restore.
The disk has crash-recovery semantics (unflushed pre-pause writes are lost), nothing durable
is mutated (the memory snapshot survives untouched), and auto-resume never takes this path —
traffic always memory-resumes. The one way a memory-preferring sandbox ends up with a
filesystem-only snapshot is at pause time, not resume time: when the node keeps refusing a
timeout auto-pause (its parent memfile header still deduplicating after the admission grace)
for longer than the retry budget counted from the first refusal
(`auto-pause-overstay-budget-milliseconds`, 120 s by default; 0 degrades at the first
refusal, negative never degrades), the evictor requests a filesystem-only snapshot instead, so
the sandbox stops overstaying its expiry; the next resume of that snapshot is a cold boot. The
degrade is only ever decided on a refusal in the same sweep, so eviction lag alone never
degrades anything, and refusals only survive with `pause-refusal-restore` on (its fallback
reads the `PAUSE_REFUSAL_RESTORE` environment variable, which is how E2B Embed, with no flag
service, turns it on). The node
refuses before `MarkStopping`, so its routing record stays live and nothing needs to restore
it; with the flag off the refused sandbox is killed on the node together with its record.
- **Pre-boot filesystem recovery**: every cold boot of a rootfs that was not frozen at pause
(`fs_quiesced` false/absent) runs a jailed `e2fsck -p -E journal_only` before the VM starts —
journal replay only, the same recovery the guest kernel would do at mount — so a `memory: false`
rescue and a legacy sync-fallback filesystem-only snapshot both mount a consistent disk. It
replays and exits without a full consistency scan, so the cost is bounded by journal content,
not filesystem size. Runs under the same confinement as the offline envd swap (unprivileged
transient unit, device access pinned to the sandbox's own NBD node). A clean replay boots;
anything else fails the start with the snapshot untouched but stays retryable. Journal replay
never condemns a snapshot: its exit codes cannot tell an unmountable filesystem apart from a
transient device fault, so every non-replayed outcome — an operational failure (timeout, I/O,
device error) or an e2fsck exit that is not a clean replay — is retryable, never a permanent
customer-facing verdict. In-file corruption that still mounts is likewise not condemned — replay
does not scan, so it boots. Whole-filesystem repair and condemning a rootfs are left to a
separate opt-in full-filesystem repair path. Gated by the `preboot-fs-recovery` flag, separate
from `fs-only-resume-api` because it also changes the behavior of existing filesystem-only cold
boots.
- **Envd live-upgrade on resume**: the orchestrator can upgrade the sandbox's envd to a newer
node-local build during resume (gated by the `envd-upgrade-target` flag in
`packages/shared/pkg/featureflags`), via envd's `POST /upgrade` (see the envd section). It is
best-effort — a delivery failure before the `exec` leaves the old envd serving — except an
unrecoverable post-`exec` failure (the new envd never re-initializes), which fails the resume
rather than return a permanently unusable sandbox.
- **Envd offline-upgrade on cold-boot resume**: reaches envd too old for the live `/upgrade`
handover (below `MinEnvdVersionForUpgrade`). When a *filesystem-only* snapshot cold-boots
(`RebootSandbox`), the orchestrator rewrites `/usr/bin/envd` in the rootfs **before** the VM
boots (`PreBootFn` → `pkg/sandbox/rootfs.SwapEnvdBinary`), entirely in userspace via a jailed
`debugfs` — never a host-kernel mount of the tenant image. The old envd never participates, so
the method is version-agnostic. Gated by the `envd-offline-upgrade-target` flag (a sibling of
`envd-upgrade-target` sharing the same version-remap resolver), and applied only when the
snapshot's rootfs was captured frozen (`fs_quiesced`, so it is crash-consistent); best-effort
(a swap failure boots the original envd). Because the swap keys on the snapshot's *built-with*
version, which it does not advance, it re-fires idempotently on each cold-boot resume until a
re-pause re-bakes the running version.
- **Both upgrade paths read the host envd binary through a node-local cache** (gated by the
`envd-binary-cache` flag): the version probe, the live delivery and the offline swap's staging
copy all read a local copy rather than the read-only artifact mount, which an `exec` would
otherwise demand-page in small random reads. The cache is why neither path performs bulk I/O
against that mount on the resume path.
It adds a third condition to both paths, so an eligible snapshot is not unconditionally
upgraded: an upgrade whose binary is not cached on this node is **deferred to a later resume**
rather than served from the mount, since reading it there would put tens of seconds on a path a
customer is waiting on. A node whose cache is cold — any miss, and in particular a ramp keyed on
sandbox or team, where the boot-time warm does not evaluate — defers that upgrade while a
background warm populates the cache; on a node-scoped rule the boot warm normally lands before
the first resume. Both upgrades are idempotent and re-fire per resume, so a deferral costs one
cycle. Ramp the flag on a node-scoped context kind — the cache is
shared by every sandbox on the node.
- Auto-pause/auto-resume make sandboxes effectively serverless: idle sandboxes pause, traffic
resumes them (see traffic flow above).
### Template build
```mermaid
sequenceDiagram
autonumber
participant C as SDK
participant API as API
participant TM as template-manager (build node)
participant FC as Firecracker build VMs
participant OS as Object storage
C->>API: POST /v3/templates (register build: cpu, ram, free-disk target) → Postgres env_builds
C->>API: POST /v2/templates/{id}/builds/{buildID} (recipe: steps, start/ready cmd)
API->>TM: gRPC TemplateCreate(TemplateConfig)
TM->>TM: pull image → inject envd/provisioning → extract ext4 rootfs
TM->>FC: boot VM per phase: provision → user steps
TM->>TM: resize disk on host
TM->>FC: boot VM per phase: finalize → optimize
TM->>OS: upload layers + final {buildID}/memfile, rootfs.ext4, snapfile, metadata
API->>TM: poll TemplateBuildStatus
API->>API: mark build ready in Postgres
```
The project provisioning system enqueues a base build when a cluster-backed project becomes active.
The job registers and starts an ordinary base build, then completes when the API accepts it.
A failed delivery may rebuild on retry; regional build tracking owns completion and timeouts.
Ordinary projects do not enqueue this job.
Builds are **layered** (`pkg/template/build/phases/`): base → user → one layer per recipe step →
resize disk → finalize → optimize. Each layer is hashed and cached, so rebuilds only re-run changed
steps. Resize disk grows the quiescent rootfs on the host; the other non-cached phases run in a real
Firecracker VM and their pause-diffs become layers. The optimize phase records which memory pages a
fresh resume touches, producing prefetch hints that speed up future sandbox starts.
Template creation holds node work through foreground setup and asynchronous build completion,
including layer uploads, synchronous cleanup, and final status publication. Cancellation can mark
a build failed before execution finishes; the work hold remains until execution unwinds. Template
build deletion owns a separate hold until object-storage cleanup returns.
`TemplateBuildDelete` cancels a running build and removes its build-ID prefix from template
storage. It does not delete container registry images; build outputs are VM snapshots in
object storage.
## Deployment topology
The services are scheduler-agnostic binaries and containers; the supported way to run them is the
Kubernetes-based distribution. The roles below hold regardless of how the nodes are provisioned.
E2B Embed (`embed/`) runs the same containers on one machine, the web dashboard included — with
Compose, with Terraform on GCP, AWS or Azure, or on Kubernetes; its ports, start order and images
are in `embed/docs/REFERENCE.md`.
```mermaid
flowchart TB
LB["Load balancer + TLS api.* → API | *.domain → client-proxy"]
subgraph control["control plane"]
AJ["api, dashboard-api, client-proxy, redis, otel-collector"]
end
subgraph sandboxnodes["sandbox nodes (autoscaled)"]
OJ["orchestrator + Firecracker sandboxes"]
end
subgraph buildnodes["build nodes (autoscaled)"]
TJ["template-manager"]
end
subgraph ch["analytics"]
CJ["clickhouse"]
end
LB --> control
AJ -->|gRPC| OJ & TJ
```
- **Control-plane services** (api, dashboard-api, client-proxy) are stateless containers and the
only LB backends. The api discovers orchestrator nodes through
`packages/shared/pkg/servicediscovery` (Kubernetes, DNS, a static list, or the legacy Nomad
backend the code still carries).
- **Sandbox nodes** run the orchestrator directly on the host (it needs root for Firecracker,
namespaces, NBD, cgroups), configured with hugepages and local template caches. Autoscaled.
- **Build nodes** run the same binary in template-manager mode.
- PostgreSQL is external (connection string via secrets); Redis is a managed service or a
single in-cluster instance; ClickHouse runs on its own nodes.
- Observability: everything exports OTel; the collector fans out to ClickHouse (product metrics)
and Grafana Cloud/stack. The API and orchestrator export synchronous counters and histograms
as deltas, so per-team series do not accumulate in process memory; the collector converts
them back to cumulative (`deltatocumulative`) before any Prometheus backend, which only works
when each of those processes sends to a collector on its own node. Logs default to the legacy Vector → Loki path; dynamic log routing can
select a primary collector and shadow collectors, and local-cluster log reads can be switched to
ClickHouse with `logs-read-config` after `sandbox_logs` is populated (`LOGS_READ_CONFIG` is the
flag's fallback where LaunchDarkly has no value). Once reads are on ClickHouse, Loki can be left
out of a deployment and the api started without `LOKI_URL`.
## Repository layout
```
packages/
api/ Control-plane REST API
orchestrator/ Sandbox runtime + template builder (one binary, per-node)
client-proxy/ Edge router for sandbox traffic
envd/ In-VM agent (pkg/version.go is release-please's; see Envd)
dashboard-api/ Web-dashboard backend
shared/ Protos, telemetry, storage clients, proxy engine, feature flags
auth/ AuthN library (API keys, JWT/OIDC) used by api + dashboard-api
db/ Postgres migrations (goose) + queries (sqlc)
clickhouse/ ClickHouse schema, batching writers, query clients
otel-collector/ Collector config
nomad-nodepool-apm/ Nomad autoscaler metric and deployment-aware target plugins
local-dev/ docker-compose local stack + DB seeding
spec/ OpenAPI specs (public, edge, dashboard) — codegen sources
tests/integration/ Integration tests against a live deployment
embed/ E2B Embed: the whole stack on one machine (Compose, Terraform, Kubernetes)
```
Cross-service contracts are all generated: OpenAPI specs in `spec/`, gRPC protos in
`packages/orchestrator/*.proto` and `packages/envd/spec/`, SQL in `packages/db/queries/`.
Run `make generate` after changing any of them.
---
### RELEASING (`docs/RELEASING.md`)
# Releasing packages to e2b-artifacts
This repository is a read-only mirror: its source of truth is E2B's internal
monorepo, which exports the code here via copybara. Releases are cut there
too — the package list, SemVer release PRs, auto-deploy tags, and the
publish workflow all live in the monorepo. Each directory in the table
below is what gets tagged and published. No release tags or GitHub Releases
appear in this repository.
| Component | Package directory | Published artifact |
|---|---|---|
| api | `packages/api` | images `us-docker.pkg.dev/e2b-artifacts/api/api` and `…/api/db-migrator` (released as one unit) |
| client-proxy | `packages/client-proxy` | image `us-docker.pkg.dev/e2b-artifacts/client-proxy/client-proxy` |
| clickhouse-migrator | `packages/clickhouse` | image `us-docker.pkg.dev/e2b-artifacts/clickhouse-migrator/clickhouse-migrator` |
| dashboard-api | `packages/dashboard-api` | image `us-docker.pkg.dev/e2b-artifacts/dashboard-api/dashboard-api` |
| embed | `embed` | images `us-docker.pkg.dev/e2b-artifacts/embed/tools`, `…/embed/node-e2b` and `…/embed/seed` (released as one unit; the release moves every platform pin in `embed/compose/.env` and `embed/kubernetes/kustomization.yaml` — api, db-migrator, dashboard-api, client-proxy, clickhouse-migrator, orchestrator and these three — to its own version) |
| envd | `packages/envd` | binary `https://storage.googleapis.com/e2b-artifact-binaries/envd/v/envd` |
| nomad-nodepool-apm | `packages/nomad-nodepool-apm` | binaries `nomad-nodepool-apm`, `nomad-deployment-aware-target` under `…/nomad-nodepool-apm/v/` |
| orchestrator | `packages/orchestrator` | binaries `orchestrator`, `clean-nfs-cache` under `…/orchestrator/v/` |
## How a release happens
Two identities:
1. Every commit on the monorepo's default branch that touches a package
directory is tagged
`-vMAJOR.MINOR.YYYYMMDDHHMM-` for that package only, on
that package's current version line. That publish does not increment
the patch. The timestamp sits in the patch slot, so these tags sort
above every `MAJOR.MINOR.x` release — pin exact tags, not "highest".
2. Conventional commits (`feat:`, `fix:`) accumulate into a release PR.
api, client-proxy, clickhouse-migrator, dashboard-api,
nomad-nodepool-apm and orchestrator share one coordinated SemVer with
the rest of the platform: merging that PR tags each of them
`-vX.Y.Z` at the same number, with changelog. embed has
its own release PR, opened once the platform release is tagged, at
that same number. envd is versioned on its own and has its own
release PR. (The same commits arrive here through the copybara
export.)
3. Either tag is a git tag only — no GitHub Release. The tag push
publishes to `e2b-artifacts` — images as `:v`, binaries as
versioned objects in the public `e2b-artifact-binaries` bucket at
`/v/`, each with a `.sha256` beside it
(sha256sum format). Client-bucket copies use the same string:
`.v`, never a bare commit SHA. The `v` is on every tag,
image tag, bucket path and object suffix; pin with it.
A manual tag (below) still publishes that commit.
## Release candidates / manual publishes
Push a `-v` tag by hand in the monorepo (e.g.
`client-proxy-v2.0.0-rc1`) and the publish workflow publishes that commit as
`:v2.0.0-rc1`. The registry has immutable tags and the binaries bucket is
create-only, so a manual tag can never overwrite an existing version — a bad
published version is fixed by cutting the next version, never by
re-publishing.
## Recovery
All recovery happens in the monorepo:
- **Publish failed**: re-run that tag's publish run (any time), or dispatch
the publish workflow manually with the tag name.
- **Release merged but never tagged** (e.g. the merge wasn't the head commit
of its push): push the `-v` tag on the merge commit by
hand, and swap the release PR's label from `autorelease: pending` to
`autorelease: tagged` — otherwise release-please refuses to open new
release PRs for any package.
---
### Embed Docs REFERENCE (`embed/docs/REFERENCE.md`)
# E2B Embed reference
The [hub README](../README.md) says what E2B Embed is and how to run it.
This page holds the rest: what runs where and on which ports, logs and
telemetry, the secrets, how images and pins are released, building templates,
and the developer tooling. The 5 guides cover what differs on Compose,
Terraform and Kubernetes.
## What runs where
The 4 stores run in containers on a bridge network with their ports on
`127.0.0.1`. api, client-proxy, dashboard-api, the dashboard and the released
orchestrator run on the machine's own network, and so does the opt-in
collector ([Observability](#observability)). The orchestrator is a host
process, launched through `nsenter` by a privileged container, the same
pattern E2B's own Kubernetes deployment uses. Its launcher ends every sandbox
when it stops.
8 one-shots run before the stack is usable, in this order:
- `preflight` fails fast with a `FIX:` line when the machine is unsuitable.
- `api-secrets` generates the api's admin token and sandbox-token hash seed.
It needs only the volume, so it finishes first.
- `host-setup` prepares the machine on every start.
- `fetch-artifacts` downloads and verifies the 5 Firecracker binaries.
- `db-migrator` and `clickhouse-migrator` bring the 2 databases to the schema
their pinned images expect.
- `seed` writes the default team and generates this install's API key.
- `base-template` builds the default template through the API.
Kubernetes runs 7 of them: there the api's 2 secrets come from a Secret
instead. `ready` is the marker that everything above worked. It is a
long-running container, not a one-shot, and its readiness is the whole
stack's.
dashboard-api starts once `db-migrator` and `api-secrets` are done and reaches
the same stores as the api; the dashboard starts once api and dashboard-api
are healthy.
### Ports on the host network
13 ports listen on every interface of the machine. The SDK needs 3000 and
3002; a browser needs 3001 for the dashboard. The other 10 must not be
reachable on any address the machine holds: give it no public address of its
own, or firewall those 10 ports for that address as well, not only at the
network edge.
| Port | Service | Reachable from | Purpose |
|------|---------|----------------|---------|
| 3000 | api | trusted clients | the REST API the SDK calls |
| 3001 | dashboard | trusted clients | the web dashboard, for a browser |
| 3002 | client-proxy | trusted clients | sandbox traffic (header routing) |
| 3003 | client-proxy | the machine only | health |
| 3010 | dashboard-api | the machine only | the dashboard's backend |
| 5007 | orchestrator | the machine only | the orchestrator's proxy into sandboxes, behind 3002 |
| 5008 | orchestrator | the machine only | **unauthenticated** gRPC control API |
| 5009 | api | the machine only | internal gRPC |
| 5010 | orchestrator | the machine only | sandbox egress: hyperloop proxy |
| 5016 | orchestrator | the machine only | sandbox egress: TCP firewall proxy (HTTP) |
| 5017 | orchestrator | the machine only | sandbox egress: TCP firewall proxy (TLS) |
| 5018 | orchestrator | the machine only | sandbox egress: TCP firewall proxy (other) |
| 5109 | api | the machine only | edge gRPC |
Port 5008 is the one to guard most. It creates and kills sandboxes and starts
template builds; nothing authenticates it, the api dials it directly as
`LOCAL_ORCHESTRATOR_ADDRESS`, and anyone who reaches it has the whole
orchestrator. The 4 egress-proxy ports (5010 and 5016 to 5018) take the
sandbox traffic the orchestrator redirects inside each sandbox's network
namespace. They expect no client from outside the machine and have no
authentication of their own.
### Ports on loopback
Everything outside the table above stays on loopback. Postgres is on
5432, Redis on 6379, ClickHouse on 8123 and 9000. On Kubernetes ClickHouse
also listens on 9004, 9005 and 9009, its MySQL and PostgreSQL wire protocols
and its interserver port, since the pod shares the node's network. Vector's
log listener is on 30006 (20006 on Kubernetes, for the reason that guide
gives) and its API on 44313, on Kubernetes only. The 2 pprof endpoints are
6060 for api and 6061 for the orchestrator. The built-in collector, when it
runs, listens on 4317 and 13133 ([Observability](#observability)).
`sudo ss -ltnp` on the machine confirms the split: the 13 ports in the table
above show a `*:` address, everything here shows `127.0.0.1:`.
## Observability
What the stack keeps about itself, out of the box and with 1 setting.
### Logs
Sandbox and template-build logs live in ClickHouse. The orchestrator and the
api ship their lines to Vector: envd's output from inside the VM, the
orchestrator's per-sandbox events, the template-manager's build output and
the api's own. Vector turns each into a row of the `sandbox_logs` table. The
api reads that table for the SDK's `getLogs` and `e2b template build`, and
the dashboard shows the same logs through it. `LOGS_READ_CONFIG=true` sets
the api's `logs-read-config` flag, since there is no LaunchDarkly here to set
it. Retention is the table's 7 days. There is no Loki in this stack, and the
api needs no `LOKI_URL`.
`PAUSE_REFUSAL_RESTORE=true` sets the api's `pause-refusal-restore` flag the
same way, in an api release that reads it: a pause the node refuses — it is
still persisting the sandbox's parent — keeps the sandbox running and
answers 503 to retry, instead of killing it.
Every service also writes its own log to stdout, which is what
`docker compose logs ` and `kubectl -n e2b logs e2b-0 -c `
print. None of this needs a collector.
### The collector setting
api, the orchestrator, client-proxy and dashboard-api carry the platform's
OpenTelemetry instrumentation, and 1 setting points it at a collector:
`E2B_OTEL_COLLECTOR_GRPC_ENDPOINT` in [`compose/.env`](../compose/.env), a
`host:port` the 4 services reach over OTLP/gRPC without TLS. It ships
commented, and empty or absent it exports nothing, which is how the stack runs
by default. Set, each of the 4 sends its metrics every 15 seconds, its traces
and its logs there. api and the orchestrator also export the `e2b.*` product
metrics on their own: a team's running and created sandboxes, and each
sandbox's CPU, memory and disk. With no endpoint those exporters time out and
log `failed to upload metrics: exporter export timeout`; that line stops once
the endpoint names a collector that answers.
### The built-in collector
`COMPOSE_PROFILES=otel`, the line under the endpoint in `.env` and also
shipped commented, starts `otel-collector`: the OpenTelemetry Collector contrib
image, on the machine's network, pinned inline in
[`compose/compose.yaml`](../compose/compose.yaml) like the stores. It receives
on `127.0.0.1:4317`, so the endpoint that goes with it is that address, and
uncommenting both lines is the whole switch. Its config is
[`compose/config/otel/otel-collector.yaml`](../compose/config/otel/otel-collector.yaml),
inlined into the compose file as `otel-collector-config`.
It keeps the `e2b.*` metrics and writes them into the stack's own ClickHouse,
into `metrics_gauge` and `metrics_sum`. The `clickhouse-migrator` creates
both, as routing tables that store nothing themselves: their materialized
views fill `sandbox_metrics_gauge` (kept 7 days) and `team_metrics_gauge` and
`team_metrics_sum` (kept 90), so those 3 are the ones to query. It drops
every other metric, accepts traces and logs and discards them, and its health
check answers `curl -s 127.0.0.1:13133` on the machine. The image has no
shell, so Compose runs it without a healthcheck, and no service depends on
it, `ready` included, so the rest of the stack starts whether it runs or not.
It is not an observability stack. Nothing here stores or shows traces, logs
or the services' own metrics. To keep traces and logs, forward them: the
config sends them to a `nop` exporter and carries a commented `otlp/upstream`
one. In your copy of `compose.yaml`, uncomment `otlp/upstream`, set its
endpoint, name it in place of `nop` in the traces and logs pipelines, and run
`docker compose up -d --force-recreate otel-collector`, since Compose does not
recreate a container for a changed inline config on its own. On Kubernetes,
make the same edit in `kubernetes/config/otel-collector.yaml`, or in an
overlay's copy for the URL install (the
[Kubernetes guide](../kubernetes/README.md#telemetry) has the overlay), and
run `kubectl apply -k` again. The services' own metrics stop at the filter.
To keep those too, send everything to a collector of your own instead: leave
the profile off and put that collector's address in the endpoint. It then
gets the `e2b.*` metrics as well, and this ClickHouse none.
### What reads the metrics
The api serves the 3 tables above: `GET /teams/{teamID}/metrics` and
`/teams/{teamID}/metrics/max` for a team's concurrent sandboxes and start
rate, and `GET /sandboxes/metrics` and `/sandboxes/{sandboxID}/metrics` for a
sandbox's CPU, memory and disk, which the SDK's `getMetrics` (`get_metrics` in
Python) reads. The dashboard's monitoring charts, a team's concurrent
sandboxes and start rate and a sandbox's resource usage, are drawn from the
same endpoints, so they stay empty until the built-in collector runs, or a
collector of your own writes those metrics into this ClickHouse. Sandbox and
build logs do not depend on it: they reach ClickHouse through Vector.
### Ports, Kubernetes and Terraform
4317 and 13133 bind `127.0.0.1` only, so they are not among the 13 ports in
[the host-network table](#ports-on-the-host-network), and no firewall needs
to change for them.
Kubernetes carries the same setting as the `OTEL_COLLECTOR_GRPC_ENDPOINT`
literal of the `e2b-settings` ConfigMap, and the built-in collector as one
commented patch line in the kustomization; the
[Kubernetes guide](../kubernetes/README.md#telemetry) has both, and where the
collector sits in the pod. Terraform has the `otel_collector_grpc_endpoint` and
`otel_collector` variables, which the startup script writes into the
instance's `.env`; the [GCP](../terraform/gcp/README.md#variables),
[AWS](../terraform/aws/README.md#variables) and
[Azure](../terraform/azure/README.md#variables) guides list them.
## Secrets
The team API key is per install. The seed generates it on the first start and
keeps it beside the databases, so an install never has a key without its
database or a database without its key. Every install prints it with the SDK
URLs and the dashboard URL. Each guide's Secrets section says where its copy
lives and how to pin or rotate it. A rotation revokes the old key, which keeps
working for up to 5 minutes: the api caches team lookups in Redis for that
long.
The dashboard takes that same key in its key form and keeps it in an httpOnly
browser cookie for a year; sign-out clears it, and rotating the key signs
every browser out. Embed serves the dashboard over plain http, so the cookie
is not marked Secure (`DASHBOARD_COOKIE_SECURE=false`); a browser on a
plain-http address would otherwise drop it and the key form would loop.
`ADMIN_TOKEN` and `SANDBOX_ACCESS_TOKEN_HASH_SEED` are the api's own 2, and
every install generates them too. Compose writes them on the first
start into the volume that holds the team key (`/run/e2b/api.env` in
`seed-state`). Terraform writes generated ones into the instance's `.env` at
first boot. Kubernetes reads the Secret the install creates. On Compose,
setting either one in `.env` pins it and leaves the other generated; each
guide's Secrets section says how to rotate what it holds. dashboard-api
reads the admin token the same way everywhere, so rotating it means
recreating dashboard-api as well as the api.
Both are worth guarding. The admin token is admin over the seeded team on
port 3000 without the team API key, including minting and revoking API keys;
the team's id is a public constant. The hash seed makes the traffic and envd
tokens of `secure` sandboxes computable from a sandbox ID by anyone who
reaches 3002.
## Images and pins
[`compose/.env`](../compose/.env) is the source of truth for every E2B
version the stack uses: the 5 released E2B service images (api, db-migrator,
dashboard-api, client-proxy, clickhouse-migrator), the dashboard image, the 3
stack images Embed builds itself, and the 5 Firecracker binaries. The 4 store
images and the built-in collector's are pinned inline in
[`compose/compose.yaml`](../compose/compose.yaml) and repeated in the
StatefulSet and its collector patch. The stack images carry everything that
is not a released E2B service: the host scripts, the SDK scripts and the
database seeder. Terraform ships the `.env` to the instance, and Kubernetes
repeats its pins in
[`kubernetes/kustomization.yaml`](../kubernetes/kustomization.yaml).
The 3 stack images live in the `embed` repository of the `e2b-artifacts`
registry.
Embed is released at the platform version once that release is tagged.
That release moves every platform pin in both files to it: api, db-migrator,
dashboard-api, client-proxy, clickhouse-migrator, the orchestrator and the 3
stack images, the lines carrying a release marker. A checkout at a release
therefore names one version everywhere and pulls exactly what that release
published; see [RELEASING.md](../../docs/RELEASING.md). envd has its own
release line, and the kernel, Firecracker and BusyBox are not released here,
so those 4 are pinned by hand. A bump adds the binary's checksum to
[`compose/scripts/fetch-artifacts.sh`](../compose/scripts/fetch-artifacts.sh)
first. The orchestrator needs no row: each release writes a `.sha256` beside
the binary it publishes, and `fetch-artifacts` verifies against it. The
dashboard image is released from its own repository
(github.com/e2b-dev/dashboard, tags `vX.Y.Z`) and is pinned by hand too; its
line carries no release marker. All of it is public and pulled anonymously;
the stores and the collector come from Docker Hub.
The 9 pinned images are published for both architectures; `fetch-artifacts`
verifies the arm64 orchestrator and envd against the `.sha256` their release
writes beside the object; a pin with no such
object stops with a `FIX:` line naming it.
To pin an install, pin the commit. The Compose files come from raw URLs, so
put the commit in place of `main` in their path. The Terraform `source` and
the `kubectl apply -k` URL are git URLs and take `?ref=`. The `main`
URLs the guides use give the newest.
Top to bottom: the source files, the Dockerfiles that copy them, the images
(3 built here, 10 pulled ready-made) and the compose services. Arrows in the
last band are `depends_on` gates, in start order. The stripe on each service
says which image it runs. 3 things the picture leaves out: the ClickHouse,
Vector and collector configs are inlined into
[`compose/compose.yaml`](../compose/compose.yaml) rather than shipped in an
image; the 5 Firecracker artifacts never enter an image at all, since
`fetch-artifacts` writes them onto the machine and the orchestrator reads them
there; and the opt-in `otel-collector` service, an 11th image pulled
ready-made, is not drawn.
## Beyond the first sandbox
`Template.build` builds your own template through the same API, whichever way
you run the stack:
```python
from e2b import Template, Sandbox
tpl = Template().from_python_image("3.12").run_cmd("pip install requests")
Template.build(tpl, alias="py-requests", cpu_count=2, memory_mb=1024, on_build_logs=lambda e: print(str(e)))
sbx = Sandbox.create("py-requests")
print(sbx.commands.run("python3 -c 'import requests; print(requests.__version__)'").stdout)
sbx.kill()
```
The build runs in a sandbox of its own on the machine and pulls the image
from Docker Hub; no Docker daemon is involved. How long it takes depends on
how much of the base image is already cached.
### Reaching a port inside a sandbox
`sandbox.get_host(port)` returns `{port}-{id}.e2b.app`, which does not resolve
here. Reach a port inside a sandbox through client-proxy's header routing
instead:
```bash
curl -H "E2b-Sandbox-Id: $SANDBOX_ID" -H "E2b-Sandbox-Port: 8080" http://localhost:3002/
```
The dashboard at `http://localhost:3001`, the `E2B_DASHBOARD_URL` that
`ready` prints, shows the same sandboxes and templates: paste `$E2B_API_KEY`
into its key form. Its terminal and filesystem inspector reach a sandbox
through the same header routing, at the address in `E2B_DASHBOARD_HOST`
(default `localhost`). Terraform writes it into the instance's `.env`, and
the Kubernetes manifest sets the same address from the node's IP; on Compose
set it in `.env` when a browser on another machine opens the dashboard
without a tunnel.
## Developing
`make` is a developer convenience; the operator path is only `docker compose`,
`terraform` or `kubectl`.
| Target | What it does |
|--------|--------------|
| `make images` | build the 3 stack images locally under their pinned tags |
| `make lint` | render the compose file and the kustomization, validate the Vector config and the 3 Terraform modules, shellcheck the scripts and the tests |
| `make test` | run the bats suite in `tests/` |
| `make stores-check` | the store-level integration check |
| `make sync-configs` | re-inline the 3 configs into the compose file |
### What the targets need
`make lint` wants `shellcheck`, `terraform` (1.7.5 or newer) and `kubectl`,
for `kubectl kustomize`. `terraform init` downloads the google, aws, azurerm,
random and time providers, once each into a shared cache, so it needs network;
the aws provider alone is about 180 MB to download and about 800 MB unpacked
in the cache. `make test` wants `bats` plus `kubectl`, which
`tests/kubernetes.bats` renders the manifest with.
`tests/terraform.bats` reads every module's files as text and needs no
terraform.
Both also need a working Docker daemon with the compose plugin, and `jq` on
`PATH`. `make lint` renders `compose/compose.yaml`, validates the Vector
config with the Vector image (`make vector-validate`) and the collector's
with the collector image `compose.yaml` pins (`make otel-validate`).
`tests/inline-configs.bats` diffs the rendered inline configs against the
copies under `compose/config/`. `tests/vector-rows.bats` replays the fixture
log lines in `tests/fixtures/vector/` through the shipped Vector config with
that image, with `compose/scripts/dev/vector-testconfig.py` swapping only the
source for stdin and the sink for a JSON console, and asserts the
`sandbox_logs` row each becomes or that it is dropped. It needs Docker,
python3 and `jq` as well. `tests/pins.bats` keeps the 3 pins, the 2 dashboard
pins, their Kubernetes counterparts and the registry the images are built
into in step with each other.
### Inline configs
After editing `compose/config/vector/vector.toml`,
`compose/config/clickhouse/config.xml` or
`compose/config/otel/otel-collector.yaml`, run `make sync-configs` (python3).
It rewrites the inline copies in `compose/compose.yaml`, the
`VECTOR_CONFIG_SHA256`, `CLICKHOUSE_CONFIG_SHA256` and `OTEL_CONFIG_SHA256`
stamps, and the Kubernetes copies under `kubernetes/config/`. The stamps make
Compose recreate the container on a config change, which it does not do for
inline `configs:` content on its own. `tests/config-hashes.bats` fails until
the stamps match, and `tests/kubernetes.bats` until the copies do.
### The stores check
`make stores-check` runs what CI's stores job runs: both stores, their
migrators, the seed, and the assertions that the seed wrote the team API key
file, that the seeded row hashes that key, and that both migration ledgers
landed. It wants Docker, network access and the images from `make images`,
and needs no KVM. It deliberately leaves the containers up so a failure can
be inspected, so finish with
`docker compose --project-directory compose down -v`.
### Rebuilding the stack images
The scripts under `compose/scripts/` and `compose/scripts/node/` travel inside
the 2 stack images that carry them, and the compose file mounts nothing from
the repository. After editing either directory, run `make images` before the
next start, or the stack silently keeps running the old code. Without `make`,
this is the `images` target (its `$$` becomes a single `$` outside of make):
```bash
set -a; . ./compose/.env; set +a
docker buildx bake --load \
--set "*.platform=linux/$(docker version -f '{{.Server.Arch}}')" \
--set "seed.context=https://github.com/e2b-dev/runtime.git#${RUNTIME_COMMIT}" \
--set seed.args.SRC=packages \
--set "tools.tags=${E2B_TOOLS_IMAGE}" \
--set "node-e2b.tags=${E2B_NODE_E2B_IMAGE}" \
--set "seed.tags=${E2B_SEED_IMAGE}"
```
Binary checksums live in the tools image's `compose/scripts/fetch-artifacts.sh`,
so bumping a Firecracker artifact means new stack images, not an edit on a
machine.
---
--- METRICS ---
- Files Extracted: 24
- Estimated Token Budget: ~67505 tokens
- Canonical Reference: https://codewiki.google/github.com/e2b-dev/infra
