infra Documentation

Technical Documentation: e2b-dev/infra

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

What is E2B Runtime? Copy Chapter

E2B Runtime is the complete backend that powers E2B Cloud: the control-plane API, the per-node orchestrator that drives Firecracker, the agent that runs inside every VM, the edge router for sandbox traffic, and the template builder. It gives every agent session its own isolated Linux machine that boots from a snapshot, runs whatever the agent asks it to, and can be paused and resumed as if nothing happened.

It is written in Go, licensed under Apache-2.0, and built by E2B. The same code serves the public cloud, dedicated enterprise deployments, and the single-machine Embed package you can run on your own hardware.

Why it's fast Copy Chapter

Two ideas drive the design.

A sandbox is a resumed snapshot. Templates are pre-booted VMs (memory, disk, and machine state) stored in object storage. "Creating" a sandbox means restoring one, not booting a kernel. Memory pages are served lazily on page fault through userfaultfd, and the root filesystem is a copy-on-write overlay over a read-only image, so only the data a sandbox actually touches is ever fetched. Fresh creates, resumes after a pause, and forks all take the same path.

Control plane and data plane never mix. The API decides where a sandbox runs and records that it runs. The orchestrator on each node owns how it runs: the Firecracker process, the network namespace, the block device, the cgroup. Sandbox traffic goes straight from the edge to the node and never passes through the API.

What you get Copy Chapter

• Hardware-isolated sandboxes. One Firecracker microVM per sandbox, in its own cgroup and network namespace, with a per-sandbox nftables egress firewall and SNI/Host-inspecting domain allow and deny lists.
• Pause and resume. Pause diffs memory and disk against the template and ships the diff to object storage. Resume prefers the node that still has it cached. Idle sandboxes auto-pause, and incoming traffic wakes them transparently.
• Fork a running sandbox. Fork checkpoints a live sandbox in place and starts new sandboxes from that snapshot, up to a hundred per request, while the original keeps running untouched. A paused sandbox has the same artifact shape as a template, so resuming one takes the same fast path as creating one.
• Templates built from your recipe. Layered builds from Docker images and build steps, each layer hashed and cached, with a final optimize pass that records which pages a boot actually touches.
• A real API inside every VM. envd exposes processes, PTYs, filesystem operations, file watchers, and port forwarding over Connect RPC and REST. It is what the SDKs talk to when they "run code", and it can be live-upgraded inside a running sandbox without dropping the workload.
• Sandbox URLs for anything that listens. https://-<sandbox>.<domain> reaches any port a process opens, routed at the edge with per-sandbox access tokens.
• Persistent volumes, secrets, workload identity. Volumes outlive sandboxes. Secrets are metadata-only on the control plane: no value ever crosses the API, a log, or a span. Workload identity gives a sandbox short-lived tokens without the API ever minting a credential.
• Observability built in. Everything exports OpenTelemetry. Sandbox lifecycle events, host stats, and metrics land in ClickHouse.

Run it Copy Chapter

On E2B Cloud. The fastest way to use the runtime is to not run it. Grab an API key at e2b.dev and start a sandbox from the JavaScript or Python SDK:

from e2b import Sandbox

with Sandbox.create() as sandbox:
    result = sandbox.commands.run('echo "Hello from E2B!"')
    print(result.stdout)

On one machine you own. E2B Embed is the whole stack, real Firecracker sandboxes included, on a single Linux host with KVM. Two files and one command:

mkdir e2b && cd e2b
curl -fsSL --remote-name-all "https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/compose/{compose.yaml,.env}"
docker compose up -d --wait

The same package ships as Terraform for GCP, AWS, and Azure and as a Kubernetes manifest. Embed is one machine by design; when one node is not enough, see below.

In your own cloud, for production. E2B runs the runtime as a dedicated deployment inside your account, with your data staying there. See e2b.dev/enterprise.

How it fits together Copy Chapter

SDK / CLI ──REST──▶ API ──gRPC──▶ orchestrator ──▶ Firecracker microVM ──▶ envd ──▶ your processes
                     │                 │
                     ├── PostgreSQL    ├── object storage (templates, snapshots)
                     ├── Redis         └── ClickHouse (events, metrics)
                     └── ClickHouse

browser ──https://<port>-<sandbox>.<domain>──▶ client-proxy ──▶ orchestrator proxy ──▶ envd

docs/ARCHITECTURE.md has the full picture: sequence diagrams for creation, traffic, pause and resume, and template builds, plus the data stores and the deployment topology. Read it first.

Who is this for? Copy Chapter

• Teams building agents who want to know exactly what their sandbox is, down to the kernel, and to run the same runtime locally that they run in production.
• Platform teams who need agent execution inside their own cloud account, on infrastructure they control, without a black box.
• Infrastructure engineers interested in Firecracker, lazy memory restore, copy-on-write block devices, and snapshot-based scheduling at scale.

Developing Copy Chapter

The runtime needs Linux with KVM. DEV-LOCAL.md walks through the host prep, the local stack, and running services from source.

make local-infra        # PostgreSQL, Redis, ClickHouse, monitoring
make test               # unit tests across packages
make test-integration   # against a live deployment
make generate           # OpenAPI, proto, sqlc
make fmt lint tidy

Releases are per package and follow conventional commits; see docs/RELEASING.md.

Contributing Copy Chapter

Bug fixes and docs fixes are welcome as pull requests. For anything larger, open an issue first so we can agree on direction before you write code. CONTRIBUTING.md has the details, including what we are unlikely to merge and why.

Community Copy Chapter

• Discord for questions and help
• GitHub Issues for bugs
• X for release notes and what we are building

License Copy Chapter

Apache-2.0. See LICENSE.

Overview (`embed/README.md`) Copy Section

E2B Embed

E2B Embed runs the whole E2B stack, every feature included, on a single
machine in the environment of your choice. Ship it inside your product, or
run it inside a customer's own environment, and bring E2B sandboxes to
customers whose data has to stay put, in government and in regulated
industries such as finance and healthcare. Same SDK, same API as E2B Cloud.
Open source, Apache-2.0.

Docker Compose
| Terraform on GCP
| Terraform on AWS
| Terraform on Azure
| Kubernetes
| Reference

How to run Copy Chapter

Choose the setup that fits what you already have.

Every guide ends with your first sandbox, a few minutes after you start. If
you'd like to scale in production, please talk with us at
e2b.dev/enterprise.

What runs on the node Copy Chapter

One node holds the complete E2B runtime. Your application talks to it
through the SDK, your team through the dashboard. The control plane is the
same API as E2B Cloud, the data plane runs the same E2B sandboxes, storage
keeps your templates and snapshots, and telemetry keeps your logs and
metrics. Nothing leaves the node. It is the same node wherever you run it,
on a Linux host you own with Docker Compose, on one VM on Google Cloud,
AWS or Azure with Terraform, or on one node in your Kubernetes cluster.

What you get Copy Chapter

• The same sandboxes as E2B Cloud. Isolated, paused and resumed on
  demand.
• The same SDK and API. Moving your application over from E2B Cloud is a
  configuration change.
• Everything stays on one single machine. Templates, sandbox logs and data live
  on its disk and go nowhere else.
• Private by design. Only your application and your dashboard users need
  to reach the node, and the Terraform installs let in only the addresses you
  name. The reference lists
  what to open.
• Your own templates. Build custom sandbox templates through the same API
  and start sandboxes from them right away.
• The dashboard. Your sandboxes, templates and logs in the browser, with a
  terminal and a file browser for every sandbox.
• Open source. Apache-2.0, in the public
  E2B runtime repository. No E2B
  account and no license key needed.

Docker Compose Copy Section

Run it on a Linux machine you already have. Full guide.

mkdir e2b && cd e2b
curl -fsSL --remote-name-all "https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/compose/{compose.yaml,.env}"
docker compose up -d --wait

Terraform on GCP Copy Section

Create the machine in your Google Cloud project. Full guide.

module "e2b" {
  source       = "github.com/e2b-dev/runtime//embed/terraform/gcp?ref=main"
  project_id   = "my-project"
  client_cidrs = ["203.0.113.0/24"]   # where your SDK clients connect from
}

terraform init && terraform apply

Terraform on AWS Copy Section

Create the machine in your AWS account. Full guide.

module "e2b" {
  source       = "github.com/e2b-dev/runtime//embed/terraform/aws?ref=main"
  client_cidrs = ["203.0.113.0/24"]   # where your SDK clients connect from
}

terraform init && terraform apply

Terraform on Azure Copy Section

Create the machine in your Azure subscription. Full guide.

provider "azurerm" {
  features {}
  storage_use_azuread = true
}

module "e2b" {
  source       = "github.com/e2b-dev/runtime//embed/terraform/azure?ref=main"
  client_cidrs = ["203.0.113.0/24"]   # where your SDK clients connect from
}

terraform init && terraform apply

Kubernetes Copy Section

Run it on a node in your existing cluster. Full guide.

kubectl create namespace e2b
kubectl -n e2b create secret generic e2b-api --from-literal=ADMIN_TOKEN="$(openssl rand -hex 32)" --from-literal=SANDBOX_ACCESS_TOKEN_HASH_SEED="$(openssl rand -hex 32)"
kubectl apply -k "https://github.com/e2b-dev/runtime//embed/kubernetes?ref=main"

When one node is not enough Copy Chapter

Embed is one machine, by design. When you need more, E2B runs three other
ways, with the same SDK and API, so moving is a configuration change.

• Private cloud, in development. The whole platform inside your boundary,
  built for networks nothing may leave. Design partners welcome.
• Bring Your Own Cloud. Your cloud account,
  operated by E2B.
• E2B Cloud. Managed by E2B, nothing to host.

See E2B for enterprise for private cloud and
Bring Your Own Cloud.

Limitations Copy Chapter

• Runs on Linux. Embed needs a Linux machine with hardware
  virtualization, and the Terraform installs create one for you. On a Mac, it
  runs inside a Linux virtual machine (Apple silicon M3 or newer, macOS 15 or
  newer).
• Plain HTTP. Embed answers over HTTP at the machine's own address, with
  no TLS of its own, which is why it belongs inside your network. A port
  inside a sandbox is reached at that same address. The
  reference shows how.

Reference Copy Chapter

For whoever operates the machine, docs/REFERENCE.md
has the detail on networking, logs and metrics, secrets, releases, building
templates and the developer tooling.

Overview (`embed/compose/README.md`) Copy Section

E2B Embed with Docker Compose

Runs the whole of Embed, the control plane and a real Firecracker
orchestrator alike, directly on one Linux host you own: no Terraform, no cloud
account, no wrapper scripts, and docker compose up as the whole interface.
It is a single-machine package rather than a deployment pattern;
the hub is ../README.md.

Requirements Copy Chapter

• Linux x86-64 or arm64 with KVM and a 4 KiB-page kernel (Ubuntu's default
  on both), on a host you own and may mutate: bare metal, or a VM created with
  nested virtualization enabled (on GCE, --enable-nested-virtualization; on
  EC2, --cpu-options NestedVirtualization=enabled on an Intel type that
  offers it, such as m8i; on Apple silicon, a Lima or other
  Virtualization.framework VM with nested virtualization on, which needs an
  M3 or newer and macOS 15 or newer). x86-64
  is what the guides were written and tested on. arm64 is verified end to
  end on bare metal: on 2026-09-14 a bare-metal arm64 host (a GCE c4a metal
  instance, Ubuntu 26.04, kernel 7.0) reached ready in under two minutes
  and passed the smoke test, sandbox create, pause and resume, template
  builds from arm64 images and 64 concurrent sandboxes. arm64 needs kernel
  6.10 or newer: restoring a sandbox needs userfaultfd write-protect, which
  arm64 gained in Linux 6.10, so on Ubuntu 24.04's stock 6.8 kernel the stack
  is healthy but every sandbox start fails with Failed to UFFD object until
  linux-generic-hwe-24.04 is installed. The nine pinned images are
  published for both architectures, and fetch-artifacts verifies the arm64
  orchestrator and envd against the .sha256 their release writes beside
  the object. A pin with no such object still stops with a FIX: line
  naming it.
• Ubuntu 24.04 is the recommended host: kernel 6.8 or newer (6.10 or newer on
  arm64, above), glibc 2.34 or newer (the released orchestrator's floor),
  cgroup v2 (systemd's default),
  and iptables, rsync, e2fsprogs and iproute2 installed. Ubuntu
  server has all four but not python3-venv, which Try it
  needs. Refresh the index first, or a package that is genuinely missing
  fails to fetch:
  sudo apt-get update && sudo apt-get install -y iptables rsync e2fsprogs iproute2 python3-venv.
• /dev/kvm and /dev/net/tun present. On a VM, /dev/kvm means nested
  virtualization is enabled for it; on GCE, stop the VM, run
  gcloud compute instances update <vm> --enable-nested-virtualization, and
  start it again; on EC2, stop the instance, run
  aws ec2 modify-instance-cpu-options --instance-id --nested-virtualization enabled,
  and start it again. Without either device, up stops at preflight with a
  FIX: line and nothing that needs KVM is started.
• Docker Engine 27 or newer with Compose 2.24 or newer, and your user in the
  docker group. Docker's own apt repository is the simplest way to get
  both: curl -fsSL https://get.docker.com | sh && sudo usermod -aG docker $USER, then log in again. Ubuntu 24.04's docker-compose-v2 package also
  clears the Compose floor. docker version and docker compose version
  show what you have; an
  older Compose rejects the compose file with an error on required: before
  anything starts.
• 12 GiB RAM recommended and 20 GiB free disk. A pause the disk cannot hold is
  refused before it starts, keeping 1 GiB free
  (PAUSE_ADMISSION_DISK_HEADROOM_MIB), instead of failing midway; the
  estimate counts the sandbox's whole memory, so a large sandbox needs that
  much free disk to pause. HUGEPAGES=2048 reserves
  4 GiB of that RAM for sandboxes, and preflight does not check RAM, so on
  a smaller host the first signal is host-setup failing with FIX: give the host more memory (12 GiB recommended) or lower HUGEPAGES. Lower
  HUGEPAGES rather than assuming 8 GiB is always enough. Size it by the
  sandboxes you will run at once: every sandbox takes its guest memory from
  this reservation as it touches pages, up to its full size (256 of the 2 MiB
  pages for a 512 MiB base sandbox, about 90 while idle), so the default
  carries eight sandboxes at full memory or about twenty idle ones. Past
  that the stack stays healthy and each create fails with 500: Failed to place sandbox: this node cannot map 512 MiB of guest memory: its memory pool is too small or in use by running sandboxes, with the sandbox's own
  size in the message, while docker compose logs orchestrator shows
  create refused: guest memory not mappable. Set a larger HUGEPAGES in
  .env and run up again: host-setup re-applies the sysctl on every
  start and the orchestrator needs no restart. A killed sandbox gives its
  pages back a few hundred milliseconds after the kill returns; a create
  sent in that window is retried on this host until they are back
  (CREATE_MEMORY_REFUSAL) instead of failing.
• Outbound HTTPS to Docker Hub, us-docker.pkg.dev, storage.googleapis.com,
  github.com and raw.githubusercontent.com. No account and no token:
  every image and binary the stack pulls is public.

Install Copy Chapter

Two files, nothing else:

mkdir e2b && cd e2b
curl -fsSL --remote-name-all "https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/compose/{compose.yaml,.env}"
docker compose up -d --wait

The first run checks the host, prepares it, downloads the Firecracker
artifacts (about 60 MB), pulls the pinned images, migrates and seeds the
databases and builds the base template. It exits 0 only when the base
template is ready, about 2 minutes on an 8-vCPU / 32 GiB VM; a second up
takes about a minute, the healthcheck start periods. Smaller hosts take
longer. Nothing is rolled back if it fails, and nothing needs to be: see
If up fails.

Try it Copy Chapter

Put this install's three SDK variables and the dashboard URL, its own team
API key included, in your shell:

eval "$(docker compose exec ready cat /run/e2b/sdk.env)"

Open the dashboard at $E2B_DASHBOARD_URL (http://localhost:3001) and paste
$E2B_API_KEY into its key form. On the machine itself, or through the
tunnel in Ports, that is all. From another machine without a
tunnel, the browser also has to reach sandbox traffic on port 3002 at an
address it can open: set E2B_DASHBOARD_HOST= in
.env and run docker compose up -d --wait again, which recreates only the
dashboard service.

Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's pip refuses to write to. On Ubuntu venv
comes from python3-venv: install it first, which the apt line in
Requirements does, or this fails with ensurepip is not available.

python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0

Then create a sandbox. Save this and run it with .venv/bin/python:

from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()

Or run the packaged smoke test, which does the same with the JavaScript SDK
in a container and then reaches a port inside the sandbox through
client-proxy:

docker compose --profile test run --rm smoke

Remove Copy Chapter

docker compose down    # stop; running sandboxes end, the data and host state stay, so the next up is fast
docker compose down -v # also the databases and the team key; the next up rebuilds base
docker compose --profile purge run --rm host-teardown  # undo the host setup

The purge is valid only after down -v. If you purged after a plain down,
run down -v before the next up, or the database still lists templates
whose files the purge removed.

Reference Copy Chapter

Everyday commands Copy Section

One stack per host; there are no VM-name or port knobs. E2B_DASHBOARD_HOST
(default localhost) is the address a browser uses for sandbox traffic on
port 3002 from the dashboard; set it in .env to this host's address when
you open the dashboard from another machine without a tunnel. HUGEPAGES
(default 2048) and PF_MIN_FREE_GIB (default 20) can be lowered in .env or
the environment for small hosts and CI. NBDS_MAX (default 64) is the number
of NBD devices host-setup asks the kernel for. Every running sandbox holds
one and a template build in flight holds one or two more, so the loaded
module's nbds_max is the ceiling on concurrent sandboxes: with every device
taken, a create waits for one until the api gives up (500: Failed to place sandbox: sandbox creation failed or 504: placement timed out), and the
creates queued behind it in a burst get 503: not enough capacity, the node's
refusal to have more than three sandboxes starting at once. A module already
loaded with more devices than NBDS_MAX is accepted and raises the ceiling.
The module takes the number only when it loads, so set NBDS_MAX in .env
before the first up, or with the stack down: once ls /sys/block/nbd*/pid
prints nothing, modprobe -r nbd on the host and up again; host-setup
stops with a FIX: line when the loaded module has fewer devices than asked.
The orchestrator also refuses the 201st running sandbox (max number of running sandboxes on node reached (200)), a limit this stack cannot raise.
NBD_POOL_SIZE (32) is the orchestrator's warm buffer of claimed devices,
not a ceiling, and stays below NBDS_MAX on purpose: at parity the pool's
refill loop spins and logs no free slots every few seconds on an idle host.
FORCE_REBUILD=1, settable in the
same two places, forces base-template to rebuild even when a base row
already shows ready, for a stale or broken template. TEAM_API_KEY,
ADMIN_TOKEN and SANDBOX_ACCESS_TOKEN_HASH_SEED are under Secrets below,
and E2B_OTEL_COLLECTOR_GRPC_ENDPOINT under Telemetry.

An optional, git-ignored env/api.local.env can add api variables. Create
the env/ directory next to compose.yaml yourself, since the two-file
install does not ship one. It cannot override a key already set in the api
environment: block, because Compose gives environment: precedence over
env_file, so change those in compose.yaml.

What runs where Copy Section

The reference's What runs where has the services.
This is what they do to the host, which is why it should be a dedicated host
or VM: on every up, host-setup

• loads the nbd (with nbds_max=NBDS_MAX), tun and kvm kernel modules;
• writes /etc/modules-load.d/e2b.conf, /etc/modprobe.d/e2b-nbd.conf,
  /etc/udev/rules.d/97-nbd-device.rules and /etc/sysctl.d/90-e2b.conf,
  which also raises net.ipv4.tcp_max_syn_backlog and vm.max_map_count;
• adds one iptables mangle rule, the TCP MSS clamp;
• reserves the hugepages;
• creates /var/lib/e2b, /fc-*, /orchestrator and /var/run/netns.

host-teardown removes the files, the directories, the hugepage reservation
and the iptables rule, and unloads nbd when no device is still connected
(it says nbd left loaded when one is). It removes the iptables rule only if
host-setup was the one that added it: a host that already had an identical
TCP MSS clamp keeps its own firewall configuration, and the purge says
MSS clamp: not added by host-setup, kept. kvm and tun are left loaded
on purpose, because other software on the host may need them, and the two
extra sysctls stay at their raised values; all of that reverts on the next
reboot.

After a host reboot, run docker compose up -d --wait once. The long-running
services return on their own (restart: unless-stopped), but the TCP MSS
clamp and /var/run/netns are the two host mutations that do not persist
across a reboot (the clamp is an iptables rule, /var/run/netns lives on a
tmpfs), and host-setup recreates both idempotently on that run.

If `up` fails Copy Section

Nothing is rolled back and nothing needs to be: every step is idempotent. Read
the FIX: line in the failing service's log (docker compose logs preflight,
host-setup, fetch-artifacts or base-template), fix what it names, and
run docker compose up -d --wait again; the steps that already succeeded are
skipped.

A failed preflight has changed nothing on the host. A failed host-setup
leaves behind whatever it had already done, all of it idempotent: the config
files, the loaded modules and, depending on where it stopped, the reserved
hugepages (HUGEPAGES times 2 MiB, 4 GiB by default) and the TCP MSS rule,
which host-teardown or a reboot releases. A failed download leaves only fully verified files behind.
A failed template build leaves a template row marked error that the next
build replaces; the base-template step already runs one Firecracker build VM
through the orchestrator, which the same SIGTERM and host-teardown sweep
cleans up like any other sandbox. ready is the marker that the stack is
usable for the sandboxes you create.

To undo everything instead:

docker compose down -v && docker compose --profile purge run --rm host-teardown

Ports Copy Section

The thirteen ports and what each is for are in the hub. Compose binds them on
every host interface: let trusted clients reach 3000, 3001 and 3002, and make
sure the other ten are reachable on no address the host holds, its own public
one included, not only from the network edge.

From another machine, tunnel rather than open, and use the same four
variables with 127.0.0.1 in place of localhost. Template builds with
copy() steps upload through the orchestrator's port 5008, which stays closed
to the network, so tunnel that too; the upload URL the stack hands out
(http://127.0.0.1:5008/...) is then valid unchanged.

ssh -N -L 3000:127.0.0.1:3000 -L 3001:127.0.0.1:3001 -L 3002:127.0.0.1:3002 -L 5008:127.0.0.1:5008 <user>@<host>

Secrets Copy Section

The team API key is this install's own. The seed generates it on the first
up and keeps it in the seed-state volume (/run/e2b/team-api-key), where
a later seed run finds it again and base-template, smoke and ready read
it; docker compose logs ready prints it and
docker compose exec ready cat /run/e2b/sdk.env returns it as shell exports.
down -v drops that volume together with the databases, so the next up
generates a new key and there is never a key without its database or a
database without its key.

To choose or rotate the key, set TEAM_API_KEY (e2b_ plus at least 32 hex
characters) in .env and run up. To rotate to a fresh generated key, remove
the file and run up:

docker compose run --rm --no-deps --entrypoint rm seed /run/e2b/team-api-key
docker compose up -d --wait

Either way the seed inserts the new key and revokes its earlier one, which
stops working within five minutes (the api caches team lookups in Redis for
that long). Within a few seconds ready re-renders /run/e2b/sdk.env and
docker compose logs ready prints the new key: up re-runs the seed but does
not recreate ready, so ready watches the key file instead. Keys created
any other way are never touched, and nothing in the two shipped files carries
a key (../tests/team-api-key.bats).

The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the dashboard is served over plain http;
sign-out clears it, and rotating the key signs every browser out.

The api's own two secrets, ADMIN_TOKEN and
SANDBOX_ACCESS_TOKEN_HASH_SEED, are this install's own in the same way. The
api-secrets one-shot generates both on the first up and keeps them in the
seed-state volume as /run/e2b/api.env, a file only root can read, as
GEN_ADMIN_TOKEN and GEN_SANDBOX_ACCESS_TOKEN_HASH_SEED; a later run finds
them again and the entrypoints of the api and dashboard-api read them;
nothing prints them, and down -v drops them with the databases. The
reference's Secrets says what the two are
for.

To choose your own instead, put them in .env before the first up. A value
there wins over the generated one, one variable at a time, and the install is
still two files:

printf 'ADMIN_TOKEN=%s\nSANDBOX_ACCESS_TOKEN_HASH_SEED=%s\n' "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" >> .env

To rotate the generated pair, remove the file, write a new one and recreate
the api on it:

docker compose run --rm --no-deps --entrypoint rm api-secrets /run/e2b/api.env
docker compose run --rm --no-deps api-secrets
docker compose up -d --no-deps --force-recreate api dashboard-api

--no-deps throughout: the two one-shot runs need no other service, and
recreating the api's dependencies would restart the orchestrator and end every
running sandbox. The new hash seed invalidates the traffic and envd tokens of
the secure sandboxes that survive the api's restart, and the new admin token
invalidates every admin client. A value pinned in .env is not rotated by
this: change the line and recreate the api and dashboard-api.
docker compose exec api env shows both empty: the entrypoint fills them in
for the api process only. Read them from the volume instead:
docker compose run --rm --no-deps --entrypoint cat api-secrets /run/e2b/api.env.

Telemetry Copy Section

Nothing is exported by default, so the dashboard's monitoring charts and the
SDK's getMetrics (get_metrics in Python) stay empty. To run the built-in
OpenTelemetry collector, which keeps those metrics in this stack's
ClickHouse, uncomment the two lines that ship commented at the end of the
optional block in .env:

E2B_OTEL_COLLECTOR_GRPC_ENDPOINT=127.0.0.1:4317
COMPOSE_PROFILES=otel

Then run docker compose up -d --wait, which starts otel-collector and
recreates the four services that export to it; recreating the orchestrator
ends the sandboxes running on it. While the collector runs,
curl -s 127.0.0.1:13133 answers Server available, and once a sandbox has
been running for a few seconds its rows show up:
docker compose exec clickhouse clickhouse-client --user clickhouse --password clickhouse --query "SELECT count() FROM sandbox_metrics_gauge".

For a collector of your own, uncomment only the first line and put that
collector's address in it. To turn the built-in one off again, comment both
lines out, run docker compose rm -sf otel-collector and then
docker compose up -d --wait. The reference's
Observability says what each choice
gives you and how to forward traces and logs as well.

Upgrading Copy Section

Download compose.yaml and .env again, with the same command as
Install, and run docker compose up -d --wait --pull always.
Before that up, carry the optional lines you had set, such as
E2B_DASHBOARD_HOST or the two under Telemetry, over into the
new .env: it ships them commented again, and without the Telemetry pair the
services stop exporting. The .env you get pins the stack images that go with
those files, so the two always move together. To take a particular commit
rather than the newest, put the commit in place of main in the two URLs, as
in raw.githubusercontent.com/e2b-dev/runtime/<commit>/embed/compose/…; the raw
host ignores ?ref=. Binary checksums live inside the tools
image, so bumping a Firecracker artifact means taking newer files rather than
editing anything on the host. Running sandboxes end when the orchestrator
restarts; a single machine has nowhere to drain them to.

Known limitations Copy Section

• Nothing on this stack authenticates a network peer except api's own API key
  and admin token check. Thirteen ports bind every host interface, and the
  orchestrator's 5008 in particular is an unauthenticated control API that
  creates and kills sandboxes. Expose 3000, 3001 and 3002 to trusted clients if
  you must, and keep the other ten unreachable on every address the host holds,
  its own public one included.
• The team API key rotates only through the seed (set TEAM_API_KEY, or
  remove the key file, then up), not through an API call, and the old key
  keeps working for up to five minutes afterwards.
• A pause the node refuses — it is still persisting the sandbox's parent, or
  the disk cannot hold the capture — leaves the sandbox running and answers
  503 (PAUSE_REFUSAL_RESTORE); retry it.
• Container logs are capped: every service keeps at most five 50 MiB log
  files (x-logging in compose.yaml), so docker compose logs shows about
  the last 250 MiB per service and older lines are gone. Without the cap a
  service that fails on a full disk writes error output faster than anything
  frees space and holds the disk at 100%. Raise the cap in your copy for a
  longer history.
• Expected log noise, all harmless: an OIDC warning from api at startup,
  because no identity provider is configured, and, every ten seconds from both
  api and the orchestrator, failed to upload metrics: exporter export timeout, because no OpenTelemetry collector is configured and the endpoint
  is empty. That line stops once E2B_OTEL_COLLECTOR_GRPC_ENDPOINT names a
  collector that answers (Telemetry).
  dashboard-api also logs ADMIN_AUTH_PROVIDER_CONFIG is not configured once
  at startup; the management endpoints it guards are not used here.
• To run without the dashboard, remove the dashboard and dashboard-api
  services from your copy of compose.yaml, and dashboard from ready's
  depends_on; the rest of the stack runs.

Troubleshooting Copy Section

• Logs missing for a sandbox or a build. docker compose logs vector
  shows insert errors (ClickHouse down, or a row it refused), and
  docker compose exec clickhouse clickhouse-client --user clickhouse --password clickhouse --query "SELECT count() FROM sandbox_logs"
  shows whether rows arrive at all. Vector buffers and retries while
  ClickHouse is down; the api's log endpoints fail during that time.

• preflight failed. Read the FIX: line in
  docker compose logs preflight. The most common cause is a missing
  /dev/kvm: on a VM, nested virtualization was off when it was created.

• up --wait failed at base-template. Its log ends with FIX: read the api and orchestrator logs for the template-manager error, then start the stack again, and docker compose logs orchestrator has the
  template-manager error itself. A line saying orchestrator not registered with the api yet, retrying is normal for the first seconds after
  a start.

• The smoke test failed. Its output ends with FIX: read the api, orchestrator and client-proxy logs; read those three logs. If the last line
  is instead FIX: the exposed port could not be reached; read the client-proxy and orchestrator logs, the sandbox was created and ran the
  command, but the listener the test started on port 8080 did not answer
  through client-proxy within 15 seconds. If it is FIX: the sandbox could not be killed; read the api and orchestrator logs, the sandbox was created and
  ran the command but the stack could not end it. That still fails the smoke
  test, and the sandbox may still be running.

• Stopping orchestrator, success: false and orchestrator-launch: orchestrator exited with status 1 at the end of the orchestrator's log
  after docker compose stop are the binary's normal SIGTERM path, not a
  crash; only a line starting FIX: means an actual failure. Once down has
  removed the container there is no log left to read.

• Template builds fail inside the sandbox with Hash Sum mismatch, Remote end closed connection or Unable to fetch some archives while small
  downloads work: the MSS clamp is missing. This is expected on every GCE VM,
  whose NIC MTU is below 1500, and can also happen after a reboot.
  sudo iptables -t mangle -S FORWARD | grep TCPMSS must show one rule; rerun
  docker compose up -d --wait to have host-setup add it.

• Sandboxes fail to start although the orchestrator is healthy. Confirm
  that the orchestrator service has cgroup: host in compose.yaml and
  that its launcher's nsenter call includes -C, in
  scripts/orchestrator-launch.sh rather
  than in compose.yaml; the symptom is fork/exec /usr/bin/unshare: no such file or directory in docker compose logs orchestrator.

• A missing or incomplete .env beside compose.yaml is the likeliest
  two-file mistake. Compose stops before starting anything and names every
  variable it could not resolve:

  text Copy

  error while interpolating services.api.image: required variable
  E2B_API_IMAGE is missing a value: fetch the .env that ships beside
  compose.yaml (README, Install)

  Fetch .env, or the lines it is missing, with the Install command above and
  retry. A partial .env fails the same way, naming only what is missing.

• /sys/module/nbd/parameters/max_part shows 31 for the configured 16.
  That is the kernel's rounding, and it is expected.

• host-teardown refuses to run while the stack is up, with FIX: run docker compose down -v first, then docker compose --profile purge run --rm host-teardown.

• host-teardown said nbd left loaded. A device was still connected. It
  unloads on reboot, or run sudo modprobe -r nbd once pgrep firecracker
  prints nothing.

• Stuck stack. docker compose down -v && docker compose --profile purge run --rm host-teardown && docker compose up -d --wait recreates everything
  in a few minutes once the images are cached.

Developing Copy Section

This directory holds the two files the Install downloads together with the
scripts, configs and Dockerfiles behind the images they pin. From a checkout,
with the package root as the working directory, the Python smoke test runs
directly:

python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0
.venv/bin/python compose/smoke/smoke_test.py

It needs the python3-venv package on Ubuntu, and
smoke/smoke_test.py is not shipped in either stack
image, so it is not reachable from a two-file install. The stack is tested
with the 2.46 SDK line ([email protected], e2b==2.46.0), the versions the
container smoke test and the images pin.

Changing an E2B component pin means editing .env and, for a binary, the
checksums in scripts/fetch-artifacts.sh; then
make images builds the three stack images locally under their pinned tags,
and docker compose up -d --wait without --pull always uses the images
just built rather than the published ones. The reference's
Developing covers make lint, make test,
make sync-configs and make stores-check, all of which run from the package
root.

Overview (`embed/kubernetes/README.md`) Copy Section

E2B Embed on one Kubernetes node

One StatefulSet that translates ../compose/compose.yaml:
the stores, Vector, the orchestrator, api and client-proxy run as sidecar init
containers in the compose start order, the one-shots as init containers
between them, and the pod is Ready once the base template exists. It is a
single-node package rather than a Kubernetes deployment pattern;
the hub is ../README.md.

Requirements Copy Chapter

• Kubernetes 1.29 or newer, for sidecar init containers.
• One Linux x86-64 or arm64 node with KVM (/dev/kvm present, which on a VM
  means nested virtualization is on) and a 4 KiB-page kernel: Ubuntu 24.04, or
  another host with kernel 6.8 or newer, glibc 2.34 or newer, cgroup v2 and
  the iptables, rsync, e2fsprogs and iproute2 tools; 12 GiB RAM and
  20 GiB free on /. No Container-Optimized OS. arm64 is verified end to
  end on a bare-metal arm64 host with Docker Compose (2026-09-14, kernel
  7.0); no arm64 node has run this StatefulSet yet. arm64 needs kernel 6.10
  or newer (userfaultfd write-protect, see the Compose guide). The nine
  pinned images are published for both architectures, and the fetch-artifacts
  init container verifies the arm64 orchestrator and envd against the
  .sha256 their release writes beside the object. A pin with no such object
  still stops with a FIX: line naming it.
• 4 GiB of 2 MiB hugepages, reserved before the kubelet starts, because it
  advertises only what it saw then. Every sandbox needs them, and the
  orchestrator's request is what stops the kubelet capping the pod at zero.
• The node labelled e2b.dev/single-node=true: the databases live on its
  disk, so the pod is pinned to it.
• git on the machine you run kubectl from: the remote apply -k URL
  below and the kubectl kustomize against the same URL under Try it are git
  sources, and kustomize clones them with that binary.

Install Copy Chapter

On a fresh node, in this order (the reservation precedes the kubelet):

# the orchestrator execs the first four on the node, so they have to be there;
# python3-venv is for the SDK in Try it. Refresh the index first, or a package
# that is genuinely missing fails to fetch
sudo apt-get update && sudo apt-get install -y iptables rsync e2fsprogs iproute2 python3-venv
echo 'vm.nr_hugepages=2048' | sudo tee /etc/sysctl.d/90-e2b.conf && sudo sysctl -p /etc/sysctl.d/90-e2b.conf
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.36.4+k3s1 INSTALL_K3S_EXEC="--node-label e2b.dev/single-node=true --write-kubeconfig-mode 644" sh -

Then export KUBECONFIG=/etc/rancher/k3s/k3s.yaml, which is what makes the
kubectl commands below work as your own user; k3s writes that file mode 600
and owned by root unless it is told otherwise. Mode 644 hands that file, a
cluster-admin credential, to every local user of the node, which is fine on a
machine that is yours alone; on a shared one drop the flag and run every
command as sudo k3s kubectl instead.

On a node that already runs Kubernetes the k3s line does not apply, and its
kubeconfig is whatever that cluster already gave you: install the same
packages, set the same sysctl, restart the kubelet
(sudo systemctl restart k3s on k3s) and run
kubectl label node <node> e2b.dev/single-node=true; the in-pod host-setup
writes that sysctl file and value too, so it changes nothing here. Either way
the node has to advertise hugepages-2Mi: 4Gi before you go on, or the pod
stays Pending.

Then install Embed:

kubectl create namespace e2b
kubectl -n e2b create secret generic e2b-api \
  --from-literal=ADMIN_TOKEN="$(openssl rand -hex 32)" \
  --from-literal=SANDBOX_ACCESS_TOKEN_HASH_SEED="$(openssl rand -hex 32)"
kubectl apply -k "https://github.com/e2b-dev/runtime//embed/kubernetes?ref=main"
kubectl -n e2b rollout status statefulset/e2b --timeout=15m
kubectl -n e2b logs e2b-0 -c ready

Ready takes about 3 minutes on n4-standard-4: 2 m 38 s from apply -k to
Ready in the reference run, 49 s of it the base template build. The
15-minute timeout above is a ceiling, not the expectation.

The namespace and the Secret come before apply -k so the api never starts
without them. Until the Secret exists the api container cannot be created:
the pod sits at Init:CreateContainerConfigError with the event
secret "e2b-api" not found, and the kubelet retries with a backoff of up to
five minutes. The kustomization carries the namespace too, so apply -k
reports namespace/e2b configured and warns that the one you created by hand
has no last-applied annotation; that is kubectl adding its annotation and
nothing else. It also brings a headless Service and the StatefulSet. From a
checkout: kubectl apply -k embed/kubernetes.

Try it Copy Chapter

Put this install's three SDK variables and the dashboard URL in your shell:

eval "$(kubectl -n e2b exec e2b-0 -c ready -- cat /run/e2b/sdk.env)"

Open the dashboard at $E2B_DASHBOARD_URL, which is the node's IP on port
3001, and paste $E2B_API_KEY into its key form. The browser also reaches
sandbox traffic on the node's IP, port 3002, so both have to be open to it.

Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's pip refuses to write to. On Ubuntu venv
comes from python3-venv: install it first, which the apt line in
Install does, or this fails with ensurepip is not available.

python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0

Then create a sandbox. Save this and run it with .venv/bin/python:

from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()

There is no smoke service here. To run the same check with the JavaScript
SDK, which also reaches a port inside the sandbox through client-proxy, run
the image that carries it on the node. The image name comes out of the same
kustomization the install applied, so this needs no checkout:

IMG="$(kubectl kustomize "https://github.com/e2b-dev/runtime//embed/kubernetes?ref=main" | awk '/image: .*node-e2b/{print $2; exit}')"
kubectl -n e2b run smoke --rm -i --restart=Never --image="$IMG" --overrides='{"spec":{"hostNetwork":true}}' \
  --env E2B_API_URL="$E2B_API_URL" --env E2B_SANDBOX_URL="$E2B_SANDBOX_URL" --env E2B_API_KEY="$E2B_API_KEY" -- node /app/smoke.mjs

--env E2B_API_KEY=... puts the team key in the pod spec, so for as long as
the pod runs anyone who can read pods in the e2b namespace can read the key
out of kubectl -n e2b get pod smoke -o yaml, and, on a cluster whose audit
policy records request bodies, it lands in the API server's audit log. --rm
removes the pod when the test ends.

Remove Copy Chapter

Three tiers, matching compose's down, down -v and a purge:

# 1. the pod; databases, key and node state stay. Wait for it: the data
#    directory below must not go while the pod is still running.
kubectl -n e2b delete statefulset e2b && kubectl -n e2b wait --for=delete pod/e2b-0 --timeout=5m
# 2. on the node: the databases and the team key
sudo rm -rf /var/lib/e2b/data
# 3. the Secret and the namespace. kubectl delete -k "<the same URL>"
#    --ignore-not-found does the same; without that flag it exits 1 on the
#    StatefulSet step 1 already removed, having deleted the namespace anyway
kubectl delete namespace e2b

The data directory goes as one piece: the team API key lives in seed-state
beside the databases and must not outlive them, or the reverse. Past that, the
node keeps what host-setup wrote until it is recycled: the files under
/etc, the rest of /var/lib/e2b (its storage holds the built templates and
their cache, about 2 GiB after one install), /fc-*, the hugepage reservation
and the iptables rule.
There is no Kubernetes teardown one-shot; Compose's host-teardown
has no counterpart here.

Reference Copy Chapter

What runs where Copy Section

One pod on the labelled node. The stores, Vector, the orchestrator, api,
client-proxy, dashboard-api and the dashboard are sidecar init containers
started in the compose order, host preparation first and the dashboard pair
last, so the base template never waits on the browser UI; the one-shots are
ordinary init containers between them; ready turns the pod Ready once the
base template exists and the dashboard answers.

The pod uses the node's network and pid namespaces, runs three privileged
containers (preflight, host-setup and the orchestrator launcher, which
nsenter into the node), mounts the node's / for the artifact download and
/sys/fs/cgroup for the sandbox sweep, and prepares the node the way the
compose host-setup does: kernel modules, sysctls, hugepages, a udev rule, an
iptables rule and directories under /. Use a dedicated node. Sandboxes end
when the pod is deleted or restarted.

The databases and the team API key are hostPath directories under
/var/lib/e2b/data on that node, which is why the manifest pins the pod there
with a nodeSelector; a pod started anywhere else would begin with an empty
stack.

Ports Copy Section

The node exposes the same thirteen ports as a compose host, and Postgres,
Redis, ClickHouse and Vector listen on loopback here too, as does the
collector under Telemetry once it is patched in. The SDK variables
ready prints use the node's IP: reach 3000, 3001 and 3002 on it and firewall
the other ten (3003, 3010, 5007, 5008, 5009, 5109 and the sandbox egress
proxies 5010, 5016, 5017 and 5018), since 5008 is an unauthenticated control
API and the egress proxies expect no external client.

k3s adds two ports of its own on top of those thirteen: *:6443, the
Kubernetes API server, and *:10250, the kubelet. They belong to the
cluster rather than to the stack, and they want the same firewall treatment
as the ten, more urgently if anything: the install above writes the
cluster-admin kubeconfig mode 644, so 6443 is the port that hands out the
node.

Vector's log listener is the one port that differs. It binds
127.0.0.1:20006 rather than compose's 30006, because 30000 to 32767 is
Kubernetes' default NodePort range and a NodePort allocated 30006 would DNAT
the pod's own loopback log traffic away from Vector.

Secrets Copy Section

The api's two secrets, ADMIN_TOKEN and SANDBOX_ACCESS_TOKEN_HASH_SEED,
come from the e2b-api Secret created during the install; nothing in the
manifest carries them. Compose generates them per install in an
api-secrets one-shot, which has no counterpart here: rotate them by
replacing the Secret and restarting the pod. dashboard-api takes ADMIN_TOKEN
from the same Secret.

The team API key is different: the seed generates it on the first start and
keeps it in /var/lib/e2b/data/seed-state/team-api-key on the node, where
ready reads it. To rotate it, remove the file on the node and restart the
pod:

sudo rm /var/lib/e2b/data/seed-state/team-api-key   # on the node
kubectl -n e2b delete pod e2b-0

The seed generates a new key, revokes the ones it issued earlier, and ready
prints the new exports. To pin a key instead of generating one, patch the seed
init container's SEED_TEAM_API_KEY with a kustomize patch; the value is
e2b_ followed by at least 32 hex characters, as compose's TEAM_API_KEY
documents.

The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the node serves plain http; sign-out
clears it.

Telemetry Copy Section

api, the orchestrator, client-proxy and dashboard-api take their OpenTelemetry
collector from the OTEL_COLLECTOR_GRPC_ENDPOINT literal of the
e2b-settings ConfigMap in kustomization.yaml. It
ships empty, which exports nothing; 127.0.0.1:4317, or any host:port the
node can reach, sends their metrics, traces and logs there over OTLP/gRPC.
The built-in collector, the compose otel profile's service, is the one
commented line at the end of that file:

# patches: [{ path: patches/otel-collector.yaml, target: { kind: StatefulSet, name: e2b } }]

From a checkout, set the literal to 127.0.0.1:4317, uncomment that line and
run kubectl apply -k embed/kubernetes.
patches/otel-collector.yaml appends an
otel-collector sidecar after the dashboard pair, reading
config/otel-collector.yaml from the
otel-config ConfigMap, and curl -s 127.0.0.1:13133 on the node answers
while it runs. Unlike compose, where nothing waits for it, its startup probe
comes before ready, so a collector that never answers leaves the pod
unready. With the dashboard patch under Limitations as well, both entries go
in the one patches: list.

The URL install has no file to edit. An overlay of your own does the same: a
directory holding this kustomization.yaml and a copy of the patch file
(curl -fsSLO https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/kubernetes/patches/otel-collector.yaml),
applied with kubectl apply -k on that directory.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: e2b
resources:
  - https://github.com/e2b-dev/runtime//embed/kubernetes?ref=main
configMapGenerator:
  - name: e2b-settings
    behavior: merge
    literals:
      - OTEL_COLLECTOR_GRPC_ENDPOINT=127.0.0.1:4317
patches:
  - path: otel-collector.yaml
    target: { kind: StatefulSet, name: e2b }

For a collector of your own, give its address instead and leave out
patches:. The reference's
Observability says what the built-in
collector keeps and what reads it.

The built-in collector discards traces and logs. To forward them from the
overlay, copy its config in as well
(curl -fsSL --create-dirs -o config/otel-collector.yaml https://raw.githubusercontent.com/e2b-dev/runtime/main/embed/kubernetes/config/otel-collector.yaml),
make the edit its comments describe, and replace the otel-config ConfigMap
with that copy through one more entry under configMapGenerator::

- name: otel-config
    behavior: replace
    files:
      - config.yaml=config/otel-collector.yaml

Upgrading Copy Section

The generated ConfigMaps carry a content hash in their names, so changing a
pin in kustomization.yaml or a file under
config/ and re-applying rolls the pod onto the new map. The
previous maps are left behind, because kustomize does not prune; delete them
by name if they bother you. A rolling update waits for the current pod to be
Ready, so a pod that is not (a crash-looping container, a Pending pod) adopts
nothing from a re-apply until you delete it: kubectl -n e2b delete pod e2b-0, and the StatefulSet recreates it from the new template.

Hugepages Copy Section

HUGEPAGES, given to host-setup in
statefulset.yaml, and the orchestrator's hugepages-2Mi
request and limit are one number and have to move together: 2048 pages of
2 MiB is 4Gi. ../tests/kubernetes.bats fails
when they disagree.

Limitations Copy Section

• Template builds with copy() steps upload through the orchestrator's port
  5008 (LOCAL_UPLOAD_BASE_URL), the same as compose from another machine.
  Build from the node (a hostNetwork pod, or kubectl exec), or tunnel 5008
  to your loopback (ssh -N -L 5008:127.0.0.1:5008 @<node>) so the
  handed-out http://127.0.0.1:5008/... URL is valid unchanged. Do not open
  5008 to the network.

• env/api.local.env has no counterpart here. To add api environment, patch
  the api container with a kustomize patch.

• There is no smoke service. Try it above has the recipe that replaces it.

• A node that carries a taint needs a toleration, added with a kustomize
  patch; a commented example is in statefulset.yaml.

• To run without the dashboard, drop both of its containers with a
  strategic-merge patch beside kustomization.yaml,
  referenced from it as patches: [{ path: no-dashboard.yaml }]; the rest of
  the stack runs.

  yaml Copy

  apiVersion: apps/v1
  kind: StatefulSet
  metadata:
    name: e2b
  spec:
    template:
      spec:
        initContainers:
          - name: dashboard-api
            $patch: delete
          - name: dashboard
            $patch: delete

Troubleshooting Copy Section

• The pod stays Pending with Insufficient hugepages-2Mi. The node is not
  advertising the reservation. kubectl get node -o jsonpath='{.items[0].status.allocatable.hugepages-2Mi}' must print 4Gi;
  the kubelet advertises only the hugepages it saw when it started, so set the
  sysctl and restart it.
• The pod sits at Init:CreateContainerConfigError, with the event
  secret "e2b-api" not found. The e2b-api Secret does not exist, so the
  kubelet cannot create the api container. Create it as the Install section
  shows; the kubelet retries with a backoff of up to five minutes, so the pod
  recovers on its own once the Secret is there.
• The k3s install fails with SSL certificate problem or Download failed
  before anything E2B runs. The unpinned installer asks k3s's channel
  server for the current stable release; when that service is down the
  installer treats the word stable as a version and fails. The install line
  above pins INSTALL_K3S_VERSION, which skips the channel lookup; keep the
  pin, or set it to the release you want.
• kubectl says permission denied on /etc/rancher/k3s/k3s.yaml. k3s
  wrote the kubeconfig mode 600 and owned by root, which is its default
  without --write-kubeconfig-mode 644. Either
  sudo chmod 644 /etc/rancher/k3s/k3s.yaml once and
  export KUBECONFIG=/etc/rancher/k3s/k3s.yaml, or run every command as
  sudo k3s kubectl instead.

Overview (`firecracker/fc-busybox/README.md`) Copy Section

fc-busybox

BusyBox static builds for E2B sandbox VMs.

What this builds Copy Chapter

A BusyBox binary with default config (all standard applets including ash shell), statically linked with musl libc for both amd64 and arm64.

The sandbox init script uses #!/usr/bin/busybox ash, so ash and the full default applet set must be present.

Build Copy Chapter

sudo ./build.sh                          # host arch
sudo TARGET_ARCH=arm64 ./build.sh        # arm64
sudo BUSYBOX_VERSION=1.37.0 ./build.sh   # custom version

Output: builds/{version}/{arch}/busybox

Overview (`firecracker/fc-kernels/README.md`) Copy Section

fc-kernels

Overview Copy Chapter

This project builds custom Linux kernels for Firecracker microVMs from the same kernel sources as the official Firecracker repo, using the configuration files (and optional patches) that live in this repo.

Prerequisites Copy Chapter

• Linux environment (for building kernels)

Building locally Copy Chapter

1. Configure kernel versions:

   • Edit kernel_versions.txt to specify which kernel versions to build (one per line, e.g. 6.1.158).
   • Place the corresponding config(s) in configs/x86_64/<version>.config and configs/arm64/<version>.config.
   • (Optional) Drop *.patch files into patches/<version>/ to apply on top of the upstream tree before build.

2. Build:

   sh Copy

   make build              # builds all versions in kernel_versions.txt for x86_64
   make build-arm64        # same, for arm64
   ./build.sh 6.1.158      # build a single version (x86_64)
   ./build.sh 6.1.158 arm64

   Output: builds/vmlinux-<version>/<arch>/vmlinux.bin where <arch> is amd64 or arm64 (Go/OCI convention). For x86_64 a legacy copy is also placed at builds/vmlinux-<version>/vmlinux.bin.

Adding a patch release Copy Chapter

A version is built from the newest microvm-kernel*<version>-*.amzn* tag of amazonlinux/linux, so that is where the available patch releases come from. Match the prefix loosely: upstream puts the series there on some tags (microvm-kernel-6.1.158-15.288.amzn2023, but microvm-kernel6.18-6.18.36-71.136.amzn2023), and a pattern that assumes the hyphen quietly hides the newest releases of a series.

git ls-remote --tags https://github.com/amazonlinux/linux | grep -E 'microvm-kernel[^ ]*-6\.1\.[0-9]+-'

A new version needs its own config, seeded from the version it supersedes and then settled against the tree it will be built from:

for arch in x86_64 arm64; do cp configs/$arch/6.1.158.config configs/$arch/6.1.177.config; done
cp -r patches/6.1.158 patches/6.1.177          # only if that version carries patches
sudo env CONFIG_ONLY=1 ./build.sh 6.1.177      # root: build.sh installs its build dependencies
sudo env CONFIG_ONLY=1 ./build.sh 6.1.177 arm64
sudo chown -R "$(id -u):$(id -g)" configs      # the resolved configs come back root-owned

CONFIG_ONLY=1 stops after olddefconfig and writes the resolved config back to configs/<arch>/<version>.config, so the symbols the newer tree added are recorded and reviewable rather than silently taking upstream defaults at build time. It applies patches/<version>/ on the way, which is where a carried patch that no longer applies shows up, and it sets CONFIG_BUILD_SALT from the tag being built, which olddefconfig leaves alone: a seeded config would otherwise ship the version it was seeded from. Run it per architecture on that architecture's machine: Kconfig probes the compiler, so a config resolved cross is not the one a native build resolves.

Then add the version to kernel_versions.txt and build.

make build builds every version in the file, so keep the list to the versions you still want built. Dropping a pin is safe: published artifacts are never overwritten or deleted, and a template records the kernel version it was built with.

Flavours Copy Chapter

A flavour is the same upstream tree and patches built under a different config, named <version>-<flavour>: 6.1.158-android is 6.1.158 with binder, DMA-BUF heaps, Android's networking options, a small patch that answers Android's ashmem ioctls on memfd files instead of an ashmem driver, and PSI (pressure stall information) on by default, so a sandbox can run Android in a container. The other configs build PSI in but leave it off unless the guest cmdline says psi=1; Android's low-memory killer daemon needs /proc/pressure, and without it the UI never comes up. build.sh takes the upstream tag and the patches/ directory from the stem (6.1.158) and everything else — the config, the output directory, and so the published name a template selects — from the full name. Seed a flavour's config from its stem's and settle it the same way as a new version (sudo env CONFIG_ONLY=1 ./build.sh 6.1.158-android, per architecture). A flavour is a sibling of its stem, not a patch release: the weekly bump neither seeds from it nor bumps it. A flavour may also carry patches of its own under patches/<version>-<flavour>/; they apply after the stem's, and a change there rebuilds only that flavour.

New kernel in E2B's infra Copy Chapter

Note: these steps should give you a new kernel on your self-hosted E2B using https://github.com/e2b-dev/infra

• Build the kernel from the new config or patch.
• Update DefaultKernelVersion in packages/shared/pkg/featureflags/flags.go if you changed the kernel version. It is the fallback: DEFAULT_KERNEL_VERSION in the environment overrides it, and a build-kernel-version feature flag overrides both.
• Build and deploy api.

Architecture naming Copy Chapter

Output directories use Go's runtime.GOARCH convention (amd64, arm64) so they match the infra orchestrator's TargetArch() path resolution. The build-time variable TARGET_ARCH (x86_64, arm64) is only used internally for config paths and cross-compilation flags.

License Copy Chapter

This project is licensed under the Apache License 2.0. See LICENSE for details.

Overview (`firecracker/fc-versions/README.md`) Copy Section

fc-versions

Overview Copy Chapter

This project builds custom Firecracker versions for E2B sandbox VMs.

Prerequisites Copy Chapter

• Linux environment (for building firecracker)
• FIRECRACKER_REPO_TOKEN set, to clone the Firecracker source

Build Copy Chapter

./build.sh <commit_hash> <version_name> [arch]

arch is amd64 (the default) or arm64. Run the script from this directory. It resolves its output through a relative path.

Output: builds/<version_name>/<arch>/firecracker

Releases Copy Chapter

A release is named by the tag it is built from, vX.Y-<e2b-semver>: the
upstream minor line we track, then our own version of the patches carried on
top of it (for example v1.14-0.1.0). The tag is cut by hand on the fork
before the release is built — the pipeline resolves it and refuses anything
else, so it never composes a release name of its own.

Releases named <tag>_<sha> predate this scheme and are never rebuilt.

License Copy Chapter

This project is licensed under the Apache License 2.0. See LICENSE for details.

Overview (`packages/envd/README.md`) Copy Section

envd

Daemon that runs inside a sandbox that allows interacting with the sandbox via calls from the SDK.

Development Copy Chapter

Versioning Copy Section

The envd version in pkg/version.go is maintained by release-please from the Conventional Commits merged since the last envd release; do not edit it by hand. A change that affects behavior (code, dependencies) ships under a feat or fix commit type, so that a version the orchestrator can gate on is cut. Pure comment or documentation changes use docs, a type the changelog hides, and cut no release; chore and refactor are visible in it and do cut one.

Running locally Copy Section

Run the following command to (re)build the envd daemon and start a Docker container with envd running inside:

make start-docker

You can use E2B SDKs with env var E2B_DEBUG=true or with a debug parameter set to true when creating or connecting to a sandbox, to connect to the envd started with this command.

Generating API server stubs Copy Section

Run the following command to install the necessary dependencies for generating server stubs:

make init-generate

Generating

After changing the API specs in ./spec/ run the following command to generate the server stubs:

make generate

Debugging Copy Section

• https://golangforall.com/en/post/go-docker-delve-remote-debug.html
• https://github.com/golang/vscode-go/blob/master/docs/debugging.md

Run make run-debug and then connect to the port 2345 with a debugger or use the VSCode run/debug and run the "Debug envd" to build the envd, Docker, and start the debugging.

Overview (`packages/nomad-nodepool-apm/README.md`) Copy Section

Nomad Node Pool Autoscaler Plugins

Custom plugins for the Nomad Autoscaler that keep a service-job allocation count aligned with the number of nodes in a Nomad node pool.

Purpose Copy Chapter

The package builds two plugins because Nomad Autoscaler supports only one plugin type per external binary:

• nomad-nodepool-apm reports the number of ready nodes in a pool.
• nomad-deployment-aware-target fails an active deployment before applying the requested task-group count.

Together they let service jobs replicate system-job placement while retaining rolling updates. The deployment-aware target is used by the template-manager and orchestrator-ee service jobs. When the task-group count must change, scaling intentionally abandons a conflicting in-progress rollout (Nomad rejects scaling while a deployment is active); Nomad records that deployment as failed, rather than cancelled. The rollout spawned by the scale itself is left to run, and no deployment is touched when the count already matches. The target requires auto_revert = false on the scaled group (otherwise failing a deployment would restore an older job version and ping-pong); it refuses to scale groups with auto_revert = true and returns an error on every attempt.

Usage Copy Chapter

Building Copy Section

make build

Uploading to GCS Copy Section

GCP_PROJECT_ID=your-project-id make build-and-upload

Configuration Copy Section

In your Nomad Autoscaler configuration:

apm "nomad-nodepool-apm" {
  driver = "nomad-nodepool-apm"
  config = {
    nomad_address = "http://localhost:4646"  # Optional, uses NOMAD_ADDR env var
    nomad_token   = "your-token"             # Optional, uses NOMAD_TOKEN env var
    nomad_region  = "global"                 # Optional
  }
}

target "nomad-deployment-aware-target" {
  driver = "nomad-deployment-aware-target"
  config = {
    nomad_address = "http://localhost:4646"
    nomad_token   = "your-token"
    nomad_region  = "global"
  }
}

In Job Scaling Policy Copy Section

scaling {
  enabled = true
  min     = 1
  max     = 50

  policy {
    evaluation_interval = "30s"
    cooldown            = "2m"

    target "nomad-deployment-aware-target" {}

    check "match_node_count" {
      source = "nomad-nodepool-apm"
      query  = "orchestrator"  # Node pool name

      strategy "pass-through" {}  # Use node count directly as desired count
    }
  }
}

How It Works Copy Chapter

1. The plugin queries the Nomad API to list nodes filtered by the specified node pool
2. It counts only nodes that are:
   • Status: ready
3. Returns the count as a metric for the autoscaler to use
4. With the pass-through strategy, this count becomes the desired number of allocations
5. The target serializes scaling per namespaced job and is a no-op when the durable count already matches
6. When the count must change it fails a conflicting active deployment, rereads the job, and scales with the current job modify index
7. It verifies the final durable count; the rollout spawned by the scale proceeds normally
8. Concurrent job or deployment changes are retried from fresh state with a bounded attempt count

Dry-run actions do not fail deployments or write task-group counts.

Query Format Copy Chapter

The query parameter should be the name of the node pool you want to count nodes from.

Example: query = "orchestrator" counts all ready nodes in the orchestrator node pool.

Configuration Options Copy Chapter

Overview (`packages/orchestrator/README.md`) Copy Section

Orchestrator

Commands Copy Chapter

Prerequisite: Enable NBD module first:

modprobe nbd nbds_max=4096

cat <<EOH >/etc/udev/rules.d/97-nbd-device.rules
# Disable inotify watching of change events for NBD devices
ACTION=="add|change", KERNEL=="nbd*", OPTIONS:="nowatch"
EOH
udevadm control --reload-rules
udevadm trigger

Prerequisite: Allocate enough 2MB HugeTLB pages (hugepages) for testing:

echo 1024 | sudo tee /proc/sys/vm/nr_hugepages   # for 2GB hugepages, adjust as needed

Create Build Copy Section

Create a new build.

sudo go run ./cmd/create-build -to-build <uuid> -storage .local-build

Flags:

• -to-build - Output build ID (UUID, required)
• -from-build - Base build ID for incremental builds
• -template - Template ID (default: local-template)
• -storage - Local path or gs://bucket (enables local mode with auto-download of kernel/FC)
• -sandbox-dir - Override SANDBOX_DIR (the rootfs path baked into the snapshot)
• -kernel <version> - Kernel version (default: vmlinux-6.1.158)
• -firecracker <version> - Firecracker version (default: v1.14.1_458ca91)
• -vcpu <n> - vCPUs (default: 1)
• -memory <mb> - Memory in MB (default: 512)
• -disk <mb> - Disk in MB (default: 1000)
• -hugepages - Use 2MB huge pages (default: true, set false for 4KB pages)
• -start-cmd <cmd> - Start command
• -ready-cmd <cmd> - Ready check command

You can use the $(uuidgen) command to generate a random UUID.

If you are using Mise, you can run the command as sudo $(which go) run ./cmd/create-build -to-build -storage .local-build.

Resume Build Copy Section

Resume a sandbox from a build.

# Local storage
sudo go run ./cmd/resume-build -from-build <uuid> -storage .local-build -iterations 10

# Remote GCS storage (pass credentials to sudo)
sudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \
  go run ./cmd/resume-build -from-build <uuid> -storage gs://bucket -iterations 10

# Pause mode: resume, run command, then snapshot
sudo go run ./cmd/resume-build -from-build <uuid> -to-build <new-uuid> \
  -storage .local-build -cmd-pause "pip install numpy"

# Pause mode: start command through envd, then wait for SIGUSR1 before snapshot
sudo go run ./cmd/resume-build -from-build <uuid> -to-build <new-uuid> \
  -storage .local-build -cmd-signal-pause "python3 /home/user/workspace/job.py"

Flags:

• -from-build - Build ID (UUID) to resume from (required)
• -to-build - Output build ID (UUID) for pause snapshot (auto-generated if not specified)
• -storage - Local path or gs://bucket (default: .local-build)
• -sandbox-dir - Override SANDBOX_DIR (the rootfs path baked into the snapshot)
• -iterations <n> - Number of iterations, 0 = interactive (default: 0)
• -cold - Clear cache between iterations (cold start each time)
• -no-prefetch - Disable memory prefetching
• -v - Verbose logging
• -pause - Start and immediately pause (create snapshot)
• -signal-pause <signal> - Wait for signal before pause (e.g., SIGTERM, SIGUSR1)
• -cmd-pause <cmd> - Execute command in sandbox, then pause on success
• -cmd-signal-pause <cmd> - Execute command in sandbox, then wait for SIGUSR1 before pause
• -envd-version <version> - The guest's envd version (default: a placeholder — see the caveat below)

envd version caveat:

resume-build has no database and the snapshot metadata does not record the build's envd
version (only the kernel and Firecracker ones), so unless -envd-version is given the tool
reports a fixed placeholder (1.0.0). Every envd.version field in its output and logs is
then that placeholder, not the guest's version — don't reason from it. To read the real
one, mount the build's rootfs (mount-build-rootfs) and run its /usr/bin/envd -version,
or read X-Envd-Version, which a running envd reports on /init.

The placeholder sits above every envd version gate, so the gated capabilities (/freeze,
/fsfreeze, /collapse, the KVM clock, the upgrade from_version) stay on. Against an
older guest that is a path production would not take, so pass -envd-version with the
guest's real version when the behavior under test depends on one of those gates. The tool
prints which of the two it is using when it loads the build.

Pause mode example:

# Chain builds with explicit build IDs
sudo go run ./cmd/resume-build -from-build $BUILD1 -to-build $BUILD2 \
  -storage .local-build -cmd-pause "apt install curl"

sudo go run ./cmd/resume-build -from-build $BUILD2 -to-build $BUILD3 \
  -storage .local-build -cmd-pause "pip install requests"

# Start a long-running command through envd, then trigger pause from the host
sudo go run ./cmd/resume-build -from-build $BUILD3 -to-build $BUILD4 \
  -storage .local-build -cmd-signal-pause "python3 /home/user/workspace/job.py"
# In another shell:
ps -ef | grep 'cmd/resume-build' | grep -v grep
sudo kill -SIGUSR1 <resume-build-pid>

Copy Build Copy Section

Copy a build between storage locations (local or GCS).

# Local to local
go run ./cmd/copy-build -build <uuid> -from .local-build -to /other/path

# Local to GCS
go run ./cmd/copy-build -build <uuid> -from .local-build -to gs://bucket

# GCS to GCS
go run ./cmd/copy-build -build <uuid> -from gs://bucket1 -to gs://bucket2

Flags:

• -build - Build ID (UUID, required)
• -from - Source: local path or gs://bucket
• -to - Destination: local path or gs://bucket
• -team - Team UUID (if set, prints SQL to populate DB on stdout)
• -envd-version <version> - Envd version (required if team provided) — must match the version present in the template
• -vcpu <n> - vCPUs (default: 2)
• -memory <mb> - Memory in MB (default: 1024)
• -disk <mb> - Disk in MB (default: 1024)
• -tag <name> - Build assignment tag (default: default)

Print DB seed SQL:

When -team is set, the command reads metadata.json from the destination to get kernel_version and firecracker_version, generates a new env ID, and prints SQL statements (BEGIN/COMMIT wrapped) to stdout. Pipe directly to psql to populate the database:

go run ./cmd/copy-build -build <uuid> -from .local-build -to gs://bucket \
  -team <team-uuid> -envd-version 0.2.11 -vcpu 2 -memory 1024 -disk 1024 \
  | psql $POSTGRES_CONNECTION_STRING

Mount Build Rootfs Copy Section

# Local storage
sudo go run ./cmd/mount-build-rootfs -build <uuid> -storage .local-build -mount /mnt/rootfs

# Remote GCS storage (pass credentials to sudo)
sudo GOOGLE_APPLICATION_CREDENTIALS=$HOME/.config/gcloud/application_default_credentials.json \
  go run ./cmd/mount-build-rootfs -build <uuid> -storage gs://bucket -mount /mnt/rootfs

Flags:

• -build - Build ID (UUID, required unless -empty)
• -storage - Local path or gs://bucket (default: .local-build)
• -mount - Mount path
• -verify - Verify rootfs integrity (requires -mount)
• -log - Enable logging
• -empty - Create an empty rootfs instead of loading a build
• -size - Size of empty rootfs (default: 1GB)
• -block-size - Block size (default: 4096)

Inspect Build Copy Section

Inspect build artifacts (headers and data blocks).

# Inspect memfile header (default)
go run ./cmd/inspect-build -build <uuid> -storage .local-build

# Inspect rootfs header
go run ./cmd/inspect-build -build <uuid> -storage .local-build -rootfs

# Inspect header + data blocks
go run ./cmd/inspect-build -build <uuid> -storage .local-build -data

# Inspect first 100 data blocks
go run ./cmd/inspect-build -build <uuid> -storage .local-build -data -end 100

# Remote GCS storage
go run ./cmd/inspect-build -build <uuid> -storage gs://bucket -rootfs -data

Flags:

• -build - Build ID (UUID, required)
• -storage - Local path or gs://bucket (default: .local-build)
• -memfile - Inspect memfile artifact (default)
• -rootfs - Inspect rootfs artifact
• -data - Also inspect data blocks (not just header)
• -start <n> - Start block index (default: 0, only with -data)
• -end <n> - End block index (default: all, only with -data)

Show Build Diff Copy Section

Show the diff between two builds and preview merge result.

# Memfile (default)
go run ./cmd/show-build-diff -from-build <uuid> -to-build <uuid> -storage .local-build

# Rootfs
go run ./cmd/show-build-diff -from-build <uuid> -to-build <uuid> -storage .local-build -rootfs

# With visualization
go run ./cmd/show-build-diff -from-build <uuid> -to-build <uuid> -storage gs://bucket -visualize

Flags:

• -from-build - Base build ID (required)
• -to-build - Diff build ID (required)
• -storage - Local path or gs://bucket (default: .local-build)
• -memfile - Inspect memfile artifact (default)
• -rootfs - Inspect rootfs artifact
• -visualize - Visualize the headers

Architecture (ARM64) Support Copy Chapter

The orchestrator supports both amd64 (x86_64) and arm64 (aarch64) architectures. Architecture is detected automatically via runtime.GOARCH at compile time.

Architecture naming convention Copy Section

This project uses Go/Docker/Debian naming (amd64/arm64) for architecture directories in binary paths and GCS buckets:

Binary paths follow the {version}/{arch}/ layout:

# Firecracker (GCS bucket or FIRECRACKER_VERSIONS_DIR)
fc-versions/v1.12.1_210cbac/amd64/firecracker
fc-versions/v1.12.1_210cbac/arm64/firecracker

# Kernels (GCS bucket or HOST_KERNELS_DIR)
kernels/vmlinux-6.1.158/amd64/vmlinux.bin
kernels/vmlinux-6.1.158/arm64/vmlinux.bin

Note: The fc-kernels repo currently uses x86_64 instead of amd64 for its directory names. This will be aligned in a follow-up change.

ARM64-specific behavior Copy Section

• SMT is disabled (ARM processors don't support simultaneous multi-threading)
• CPU detection uses fallback values since gopsutil doesn't populate Family/Model on ARM64
• OCI platform is set to the target architecture instead of hardcoded amd64
• Busybox binaries are fetched per architecture by scripts/fetch-busybox.sh and selected at runtime from /<version>/<GOARCH>/busybox
• Guest kernel command line drops the x86-only i8042.* and clocksource=kvm-clock parameters on arm64; the console stays ttyS0 (Firecracker exposes a 16550-compatible UART on both architectures)

Cross-architecture deployment Copy Section

TARGET_ARCH is a runtime environment variable that overrides the architecture used for path resolution and OCI image pulls. When unset, defaults to the host architecture (runtime.GOARCH).

# Run orchestrator targeting amd64 paths from an arm64 host
TARGET_ARCH=amd64 ./bin/orchestrator

# Or in .env file (read at runtime)
echo "TARGET_ARCH=amd64" >> .env.local

TARGET_ARCH affects:

• Firecracker and kernel binary path resolution ({version}/{arch}/...)
• OCI image platform for container pulls

It does not affect:

• Makefile compilation (see BUILD_ARCH below)
• Hardware-dependent runtime behavior (SMT detection, CPU info) which always uses the actual host architecture

Building for ARM64 Copy Section

BUILD_ARCH defaults to amd64. To build ARM64 binaries:

# Single command
BUILD_ARCH=arm64 make build-local

# Or set in .env.local for persistent override
echo "BUILD_ARCH=arm64" >> .env.local

This applies to all services: orchestrator, envd, client-proxy.

Environment Variables Copy Chapter

Automatically set in local mode. Set before running to override:

• TARGET_ARCH - Target architecture override (amd64 or arm64; default: host architecture)
• HOST_ENVD_PATH - Envd binary path (default: ../envd/bin/envd)
• HOST_KERNELS_DIR - Kernel versions dir (local: {storage}/kernels, prod: /fc-kernels)
• FIRECRACKER_VERSIONS_DIR - Firecracker versions dir (local: {storage}/fc-versions, prod: /fc-versions)
• ORCHESTRATOR_BASE_PATH - Base orchestrator data (local: {storage}/orchestrator, prod: /orchestrator)
• SANDBOX_DIR - Sandbox working dir (default: /fc-vm)

Limitations Copy Chapter

• Custom template builds support base images from the declared distro families: Debian/Ubuntu (apt), the RHEL family — Fedora, CentOS Stream, Rocky, Alma — (dnf/microdnf/yum), Arch (pacman), and Alpine (apk, OpenRC). The distro is resolved from the image's /etc/os-release ID — never by probing for package managers. Images without an os-release identity (distroless, scratch) or from an undeclared family are rejected fast, with a build-log error naming the reason and the supported families (see pkg/template/build/phases/base/distro). Minimal/restricted-repo images may fail provisioning if their repos don't carry the required packages; the failure is surfaced in the build log.
• Sandboxes always boot the kernel E2B supplies, never one from the base image: /lib/modules is empty, no kernel modules can be loaded, and SELinux is disabled on the kernel command line (selinux=0), so a base image that ships an SELinux policy boots without it. kernel-devel from a distro's repos resolves against a kernel that isn't running. RHEL (rhel, incl. UBI), Oracle Linux (ol) and Amazon Linux (amzn) are therefore not accepted even though they are RPM/dnf images — they are chosen for kABI, signed kernel modules and UEK, which this cannot honour. The community rebuilds above are supported because they are chosen for the userland.

Overview (`tests/integration/README.md`) Copy Section

Integration Tests

Package for defining integration tests. Currently, there is a setup for API and Orchestrator testing.

Run locally Copy Chapter

1. Configure test variables in a root .env.<environment> file and select it with
   make switch-env ENV=<environment> from the repository root. Use
   .env.local as a reference for local API values;
   TESTS_ORCHESTRATOR_HOST must point to the orchestrator for tests that call it directly.
2. If you made changes to the api or envd protobuf spec, run make generate from this folder (and don't forget to generate it in envd if changes apply there too).
3. If the orchestrator is not reachable directly, tunnel its gRPC port (TESTS_ORCHESTRATOR_HOST, e.g. localhost:5008) to a sandbox node before running the tests
4. Run make test in this folder or make test-integration from the repository root.

Narrow the run with make test/, e.g. make test/api/templates
or make test/api/templates/build_template_test.go:TestTemplateBuildCOPY.

What CI runs Copy Chapter

The uncompressed config runs the whole suite. The compressed configs (zstd1,
lz4) re-run only scripts/compression-tests.tsv, the explicit allow-list of
tests whose subject is writing or reading back a snapshot; that file documents
the inclusion criteria and the code paths the compression knobs reach. Select it
locally with TESTS_ONLY=scripts/compression-tests.tsv make test. An entry that
no longer matches a real test is an error, so the list cannot rot into silently
reduced coverage.

Usage of clients (api, orchestrator, envd) Copy Chapter

All tests are in the folder internal/tests. You can see the usage of different clients in the tests. Here are just basics.

API Copy Section

HTTP client. In order to pass the API key, use the setup.WithAPIKey() option.

client := setup.GetAPIClient()

sbxTimeout := int32(60)
resp, err := client.PostSandboxesWithResponse(ctx, api.NewSandbox{
    Timeout:    &sbxTimeout,
}, setup.WithAPIKey())

Orchestrator Copy Section

GRPC client. There is no authentication needed as it runs behind API in production.

client := setup.GetOrchestratorClient(t, ctx)
resp, err := client.List(ctx, &emptypb.Empty{})

Envd Copy Section

Envd client is used for interacting with the sandbox.
There are two API types—HTTP and GRPC.
Each of them provides different methods for interacting with the sandbox; you need to check which ones you need.

HTTP

setup.WithSandbox(...) sets the sandbox URL and sends the envd access token when the sandbox has one. Pass the sandbox response to authenticate envd requests.

client := setup.GetEnvdClient(t, ctx)
resp, err := client.HTTPClient.PostFilesWithBodyWithResponse(
    ctx,
    ...,
    setup.WithSandbox(t, sbx.JSON201),
)

To intentionally omit the envd access token, use the tokenless helper:

setup.WithInsecureSandbox(t, sbx.JSON201.SandboxID)

GRPC

setup.SetSandboxHeader(...) sets the sandbox URL and sends the envd access token when the sandbox has one.
For gRPC requests that intentionally omit the token, use setup.SetInsecureSandboxHeader(...).

All methods also expect a user (user/root) to be set in the header.
You can achieve it using setup.SetUserHeader(...).

client := setup.GetEnvdClient(t, ctx)
req := connect.NewRequest(&filesystem.ListDirRequest{
    Path: "/",
})
setup.SetSandboxHeader(t, req.Header(), sbx.JSON201)
setup.SetUserHeader(req.Header(), "user")
resp, err := client.FilesystemClient.ListDir(ctx, req)

Overview (`embed/compose/env/README.md`) Copy Section

This directory exists for the optional, git-ignored api.local.env override; see compose.yaml.

Overview (`embed/terraform/aws/README.md`) Copy Section

E2B Embed on one EC2 instance with Terraform

Creates one Ubuntu 24.04 instance with nested virtualization, in an Auto
Scaling group of one, which installs Docker, writes the two files Embed ships
and runs docker compose up -d --wait. It is a single-machine package
rather than a deployment pattern; the hub is
../../README.md.

Requirements Copy Chapter

• Terraform 1.7.5 or newer.
• An AWS account and credentials the AWS provider can read (an SSO profile
  in AWS_PROFILE, or any other source it supports), allowed to create a
  VPC and its network, an Elastic IP, a security group, an IAM role and
  instance profile, an S3 bucket, a launch template and an Auto Scaling
  group; to pass the instance role to EC2 (iam:PassRole); to launch from
  the template (ec2:RunInstances and ec2:CreateTags, which Auto Scaling
  checks against the caller when it creates the group); to read Canonical's
  public Ubuntu SSM parameter (ssm:GetParameter) when ami is empty; and,
  in an account that has never had an Auto Scaling group, to create Auto
  Scaling's service-linked role (iam:CreateServiceLinkedRole).
• The root volume is encrypted with the account's default EBS key. If that
  default is a customer-managed KMS key, its key policy has to let Auto
  Scaling's service-linked role use it, or every launch fails with
  Client.InternalError.
• A region that offers a nested-virtualization instance type, m8i.xlarge
  by default. The module puts the instance in the first of the region's own
  zones (not a Local or Wavelength Zone) that offers the type.
• The AWS CLI v2 and its Session Manager plugin on your own machine, for the
  session_command output, the port forwarding under Limitations and the
  instance refresh under Upgrading. terraform apply needs neither.
• No machine of your own. The module creates it.
• The module source has to be the repository, not a copy of this directory:
  it reads ../../compose/compose.yaml and ../../compose/.env relative to
  itself, which the github.com/...// source form below provides by cloning.
  A copy of this directory alone fails at plan time, and so does an absolute
  local path, which Terraform copies the same way. A checkout of your own
  works through a relative source, such as ../runtime/embed/terraform/aws.

Install Copy Chapter

provider "aws" {
  region = "us-east-1"
}

module "e2b" {
  source       = "github.com/e2b-dev/runtime//embed/terraform/aws?ref=<commit>"
  client_cidrs = ["203.0.113.0/24"]   # where your SDK clients connect from
}

output "api_url" { value = module.e2b.api_url }
output "sandbox_url" { value = module.e2b.sandbox_url }
output "dashboard_url" { value = module.e2b.dashboard_url }
output "session_command" { value = module.e2b.session_command }
output "e2b_api_key" {
  value     = module.e2b.e2b_api_key
  sensitive = true
}

terraform init && terraform apply

A module's outputs are not the root module's, so the five output blocks are
what makes terraform output see them; without them the Try it commands below
print nothing.

?ref=<commit> pins the source to a commit of the public repository's main
branch, so an init on another day fetches the same compose.yaml and .env;
upgrading is a deliberate change of that ref. Use main itself for the newest
code. The instance answers GET /health about three minutes after apply
returns, the Docker install and the image pulls behind it. The base
template build runs on past that, so the first sandbox can be created about
four minutes after apply returns.

Try it Copy Chapter

Put this install's three SDK variables in your shell:

export E2B_API_URL="$(terraform output -raw api_url)"
export E2B_SANDBOX_URL="$(terraform output -raw sandbox_url)"
export E2B_API_KEY="$(terraform output -raw e2b_api_key)"

Open terraform output -raw dashboard_url in a browser and paste the key
(terraform output -raw e2b_api_key) into its key form. The instance's .env
carries E2B_DASHBOARD_HOST=, so the browser reaches sandbox
traffic at the same address, no tunnel needed.

Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's pip refuses to write to. This runs on your own
machine, not on the instance; on Ubuntu install python3-venv first with
sudo apt-get update && sudo apt-get install -y python3-venv, or it fails
with ensurepip is not available.

python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0

Then create a sandbox. Save this and run it with .venv/bin/python:

from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()

The first attempt can fail with 404: tag 'default' does not exist for template 'local-dev-team/base'. That is the base build still finishing,
not a broken install: wait a minute and run the snippet again.

curl -s -H "X-API-Key: $E2B_API_KEY" $E2B_API_URL/v2/templates lists
base from the moment its build starts rather than when it finishes, so a
listing is not the go-ahead for the snippet above. Or run the packaged smoke
test on the instance itself: it does the same with the JavaScript SDK in a
container, then reaches a port inside the sandbox through client-proxy:

eval "$(terraform output -raw session_command)"
cd /opt/e2b && sudo docker compose --profile test run --rm smoke

Remove Copy Chapter

terraform destroy   # the instance, the bucket with its objects, and every other resource the module made

Reference Copy Chapter

What it creates Copy Section

Twenty-seven resources with one client CIDR, three more for each further
one: a VPC with one public subnet, an internet gateway, a route table and
its default route; an Elastic IP; a security group, a rule per client port
and CIDR, and one egress rule; an IAM role with its instance profile and
its one inline policy, and a 30-second time_sleep that gives the new
profile time to reach EC2 (Timings); a private, versioned, encrypted S3
bucket holding the two files; a launch template (m8i.xlarge, a 50 GB
encrypted gp3 root volume, nested virtualization on, IMDSv2 only, the
startup script as user data); an Auto Scaling group of one; and the three
secrets under Secrets below. The providers are aws, random and time.

The group replaces the instance when the instance's own watchdog reports it
unhealthy: GET :3000/health has failed for ten minutes. A replacement
starts from a fresh volume, so it rebuilds the base template and the
templates you built are gone.

Security group Copy Section

Three ports, and nothing else reaches the instance from outside: 3000, 3001
and 3002 from client_cidrs. There is no port 22 and no health-checker
range: Session Manager reaches the instance through its agent's outbound
connection, and the group hears about the instance's health from the
instance itself. The orchestrator's unauthenticated control port 5008 in
particular stays inside the instance.

Timings Copy Section

Expected on the default m8i.xlarge in us-east-1, but for the first row,
which is a fixed wait: IAM takes some seconds to make a new instance profile
usable by EC2, and a launch that names it sooner fails with
Authentication Failure. Launching EC2 instance failed. If the first launch
still fails that way, run terraform apply again.

To follow a boot:

eval "$(terraform output -raw session_command)"
sudo tail -f /var/log/cloud-init-output.log   # the first boot
sudo journalctl -u e2b-embed -f               # the stack's up, on every boot

The session_command output is a shell snippet that looks the current
instance id up, which is why it is used through eval. From outside,
aws ec2 get-console-output --region <region> --instance-id --latest --query Output --output text
prints the same first-boot log (`` as in the id= line under
Limitations).

Secrets Copy Section

ADMIN_TOKEN, SANDBOX_ACCESS_TOKEN_HASH_SEED and the team API key are all
generated per install, here by Terraform itself: the startup script writes
all three into the instance's .env before the first up, so the pair the
api-secrets one-shot generates on the instance is never read. To rotate
either api secret, replace its resource and then replace the instance as
Upgrading describes: terraform apply -replace=module.e2b.random_bytes.admin_token
or -replace=module.e2b.random_bytes.sandbox_access_token_hash_seed makes a
new launch-template version; a new hash seed ends the tokens of running
secure sandboxes. They live in the Terraform state, so treat the state file
and terraform show as secret, and they reach the instance through its user
data, which every version of the launch template keeps. Anyone in the
account allowed ec2:DescribeLaunchTemplateVersions can read it: that
action cannot be limited to one launch template, and read-only access such
as AWS's ReadOnlyAccess managed policy (ec2:Describe*) includes it. So
can anyone allowed ec2:DescribeInstanceAttribute or
ec2:GetLaunchTemplateData, which read it from the instance, and any
process on the host, root or not, through the metadata service. The
metadata service takes IMDSv2 tokens only, one network hop away, so
containers on Docker's bridge network cannot reach it; the host, its
host-network services and a Session Manager shell can. The startup script
does not print them; read the key with terraform output -raw e2b_api_key.
Set team_api_key to choose the key instead; change it and replace the
instance to rotate it. Keeping the three out of user data, in SSM Parameter
Store parameters only the instance's role can read, would narrow who can see
them; this module does not do that.

The instance's role may read the two files in the bucket, move this
install's Elastic IP to itself, mark its own instance unhealthy, and make
the calls the SSM agent needs for Session Manager and Run Command:
ssm:UpdateInstanceInformation, its heartbeat, and the ssmmessages and
ec2messages actions that carry sessions and commands. Those are on every
resource: the two message services have no resource to name, and the
heartbeat's instance does not exist when the policy is written. The role
reads no Parameter Store parameter and no other S3 object, and it does not
carry AWS's managed AmazonSSMManagedInstanceCore policy, which would let
it read any parameter in the account and region by name. Systems Manager
features beyond sessions and Run Command, such as State Manager
associations, Inventory and Patch Manager, are not granted. Anyone with a
shell on the instance, and any process on the host, holds that role.

The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the instance serves plain http; sign-out
clears it.

Upgrading Copy Section

A newer ref points at newer files. terraform apply uploads them to the
bucket and records a new launch-template version, which the group uses for
the next instance it launches, but it does not replace the running one, and
that instance wrote its compose.yaml and .env once, on first boot.
Replace it deliberately:

aws autoscaling start-instance-refresh --region <region> --auto-scaling-group-name <name> --preferences '{"MinHealthyPercentage":0,"MaxHealthyPercentage":100,"InstanceWarmup":300}'

<name> is the module's name, e2b-embed unless you set it. The two
percentages matter: the instance holds the Elastic IP, so the refresh must
terminate it before it launches the new one (the group's own maintenance
policy says the same for every replacement). That is a fresh instance, so
the base template is rebuilt and the templates you built are gone. The
warm-up only shortens the wait for the refresh's own status, which
otherwise sits out the group's 900-second grace period. Expected:
aws autoscaling describe-instance-refreshes --region <region> --auto-scaling-group-name <name> --query 'InstanceRefreshes[0].Status'
reads Successful about 5 minutes after the refresh starts, a couple of
minutes after the stack answers on the new instance.

Unlike the GCP module's group, a replacement this group makes on its own,
after the watchdog reports the instance unhealthy, also boots the newest
launch-template version, so it comes up on the newest files. The AMI
follows Canonical's parameter the same way: an apply after Canonical
publishes a new Ubuntu 24.04 image records it in a new launch-template
version, and the next replacement boots it. Set ami to pin one.

compose_base_url points the first boot at the Compose files of a specific
commit instead of the files the module ships, for example
https://raw.githubusercontent.com/e2b-dev/runtime/<commit>/embed/compose.

Telemetry Copy Section

otel_collector = true runs the built-in OpenTelemetry collector on the
instance, keeping the sandbox and team metrics behind the dashboard's
monitoring charts in its own ClickHouse, and otel_collector_grpc_endpoint
sends the services' metrics, traces and logs to a collector of your own. Set
one or the other: with both, the services export to your endpoint and the
built-in collector receives nothing, so the dashboard's charts stay empty.
Both reach the instance's .env at first boot only, so changing either later
takes the replacement under Upgrading; the reference's
Observability has the rest.

Limitations Copy Section

• No persistent data volume. Every replacement starts from a fresh volume,
  rebuilds the base template and loses the templates you built.

• No load-balancer health check. The instance's own watchdog is what reports
  it unhealthy, and the group's EC2 status checks cover an instance that
  stops responding altogether. Stopping the stack for more than ten minutes,
  with docker compose down for example, therefore gets the instance
  replaced: stop the watchdog first with
  sudo systemctl stop e2b-embed-watchdog.timer, and start it again only
  once GET /health answers, since it counts from the last answer it saw.

• Template builds with copy() steps upload through the orchestrator's port
  5008, which the security group keeps closed. Run them on the instance, or
  forward the four ports through Session Manager and point the SDK at
  http://127.0.0.1:3000 and http://127.0.0.1:3002. One session per port:

  bash Copy

  for port in 3000 3001 3002 5008; do
    eval "$(terraform output -raw session_command) --document-name AWS-StartPortForwardingSession --parameters portNumber=$port,localPortNumber=$port" &
  done

  The four sessions are jobs of your shell and run until you stop them:
  kill %1 %2 %3 %4, or the numbers jobs shows. Or one ssh through
  Session Manager forwards all four, in the foreground until Ctrl-C. Ubuntu's
  image ships EC2 Instance Connect, which accepts a key of your own for 60
  seconds, so nothing is installed on the instance and no port 22 is opened;
  ssh has to connect within that minute, and your credentials need
  ec2-instance-connect:SendSSHPublicKey:

  bash Copy

  id="$(aws autoscaling describe-auto-scaling-groups --region <region> --auto-scaling-group-names <name> --query 'AutoScalingGroups[0].Instances[?LifecycleState==`InService`] | [0].InstanceId' --output text)"
  aws ec2-instance-connect send-ssh-public-key --region <region> --instance-id "$id" --instance-os-user ubuntu --ssh-public-key file://~/.ssh/id_ed25519.pub
  ssh -i ~/.ssh/id_ed25519 -N -L 3000:127.0.0.1:3000 -L 3001:127.0.0.1:3001 -L 3002:127.0.0.1:3002 -L 5008:127.0.0.1:5008 \
    -o IdentitiesOnly=yes -o ProxyCommand="aws ssm start-session --region <region> --target %h --document-name AWS-StartSSHSession --parameters portNumber=%p" ubuntu@"$id"

• x86-64 only. Graviton instances offer no nested virtualization. A Graviton
  .metal instance has a bare /dev/kvm and would run the stack on kernel
  6.10 or newer (the Compose guide's Requirements say why), but this module
  does not build an arm64 instance.

• A public subnet only. The instance needs a public address for its
  downloads and for its Elastic IP. In a private subnet it would need a NAT
  gateway, and a restricted egress would have to allow the hosts the Compose
  guide's Requirements list, download.docker.com and the Ubuntu apt mirrors
  for the first boot, the EC2, Auto Scaling and S3 APIs the instance calls,
  Session Manager's ssm, ssmmessages and ec2messages endpoints, and
  DNS to 8.8.8.8 for the sandboxes.

• AWS GovCloud (US) is untested. The module takes the partition of every ARN
  from the provider and Canonical publishes the same AMI parameter there, so
  it should work with the provider pointed at a GovCloud region.

Variables Copy Section

A worked call is in examples/basic/main.tf, which
make lint validates along with the module.

Overview (`embed/terraform/azure/README.md`) Copy Section

E2B Embed on one Azure instance with Terraform

Creates one Ubuntu 24.04 instance with nested virtualization, in a scale set
of one, which installs Docker, writes the two files Embed ships and runs
docker compose up -d --wait. It is a single-machine package rather than a
deployment pattern; the hub is ../../README.md.

Requirements Copy Chapter

• Terraform 1.7.5 or newer.
• An Azure subscription and credentials the azurerm provider can read
  (az login, or any other source it supports), with ARM_SUBSCRIPTION_ID
  set or subscription_id in the provider block.
• Permission to create a resource group and, in it, a virtual network, a
  security group, two public IP addresses, a NAT gateway, a user-assigned
  managed identity, a storage account and a scale set; and to write role
  assignments and one custom role definition, which Owner or the pair
  Contributor plus User Access Administrator grants. The module makes three
  role assignments and one role definition (Identity below).
• storage_use_azuread = true in the provider block. The storage account the
  module creates refuses shared keys, so Terraform writes the two files with
  the identity you ran it as rather than with an account key, which is what
  keeps a key and a SAS out of the state file.
• A region that offers the instance size, Standard_D4ds_v7 by default, in
  the zone you ask for.
• The Azure CLI on your own machine, for the run_command output and the
  serial console under Reaching the instance. terraform apply needs neither.
• No machine of your own. The module creates it.
• The module source has to be the repository, not a copy of this directory:
  it reads ../../compose/compose.yaml and ../../compose/.env relative to
  itself, which the github.com/...// source form below provides by cloning.
  A copy of this directory alone fails at plan time, and so does an absolute
  local path, which Terraform copies the same way. A checkout of your own
  works through a relative source, such as ../runtime/embed/terraform/azure.

Install Copy Chapter

provider "azurerm" {
  features {}
  storage_use_azuread = true
}

module "e2b" {
  source       = "github.com/e2b-dev/runtime//embed/terraform/azure?ref=<commit>"
  client_cidrs = ["203.0.113.0/24"]   # where your SDK clients connect from
}

output "api_url" { value = module.e2b.api_url }
output "sandbox_url" { value = module.e2b.sandbox_url }
output "dashboard_url" { value = module.e2b.dashboard_url }
output "run_command" { value = module.e2b.run_command }
output "e2b_api_key" {
  value     = module.e2b.e2b_api_key
  sensitive = true
}
output "admin_password" {
  value     = module.e2b.admin_password
  sensitive = true
}

terraform init && terraform apply

A module's outputs are not the root module's, so the six output blocks are
what makes terraform output see them; without them the Try it commands below
print nothing.

?ref=<commit> pins the source to a commit of the public repository's main
branch, so an init on another day fetches the same compose.yaml and .env;
upgrading is a deliberate change of that ref. Use main itself for the newest
code. The instance answers GET /health about two minutes after apply
returns -- most of the Docker install and the image pulls happen while apply
is still waiting on the scale set. The base template build runs on past
that, so the first sandbox can be created about seven minutes after apply
returns; the Timings table below carries the measured numbers.

Try it Copy Chapter

Put this install's three SDK variables in your shell:

export E2B_API_URL="$(terraform output -raw api_url)"
export E2B_SANDBOX_URL="$(terraform output -raw sandbox_url)"
export E2B_API_KEY="$(terraform output -raw e2b_api_key)"

Open terraform output -raw dashboard_url in a browser and paste the key
(terraform output -raw e2b_api_key) into its key form. The instance's .env
carries E2B_DASHBOARD_HOST=, so the browser reaches sandbox
traffic at the same address, no tunnel needed.

Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's pip refuses to write to. This runs on your own
machine, not on the instance; on Ubuntu install python3-venv first with
sudo apt-get update && sudo apt-get install -y python3-venv, or it fails
with ensurepip is not available.

python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0

Then create a sandbox. Save this and run it with .venv/bin/python:

from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()

The first attempt can fail with 404: tag 'default' does not exist for template 'local-dev-team/base'. That is the base build still finishing,
not a broken install: wait a minute and run the snippet again.

curl -s -H "X-API-Key: $E2B_API_KEY" $E2B_API_URL/v2/templates lists
base from the moment its build starts rather than when it finishes, so a
listing is not the go-ahead for the snippet above. Or run the packaged smoke
test on the instance itself: it does the same with the JavaScript SDK in a
container, then reaches a port inside the sandbox through client-proxy:

eval "$(terraform output -raw run_command) 'cd /opt/e2b && docker compose --profile test run --rm smoke'"

Remove Copy Chapter

terraform destroy   # the resource group and everything the module made in it

Reference Copy Chapter

What it creates Copy Section

One resource group and, in it: a virtual network with one subnet; a security
group with one inbound rule and its subnet association; a static Standard
public IP for clients, and a second one behind a NAT gateway for egress; a
user-assigned managed identity, a custom role definition and three role
assignments, and a 60-second time_sleep that gives them time to propagate
(Timings); a storage account that refuses shared keys, with one private
container holding the two files; a scale set of one
(Standard_D4ds_v7, a 50 GB Premium SSD OS disk, the Application Health
extension, the startup script as custom data); and the four secrets under
Secrets below. The providers are azurerm, random and time.

The scale set replaces the instance when the health extension reports
GET :3000/health dead past the grace period. The extension runs in Binary
Health States: a 200 is healthy, anything else -- another status, a timeout,
a refused connection -- is not. A replacement starts from a fresh disk, so it
rebuilds the base template and the templates you built are gone.

Why there is a NAT gateway Copy Section

Instances of a Flexible scale set get no outbound address of their own, and
the client address is not on the instance's network interface until the first
boot has already installed Docker and asked Resource Manager to attach it. The
NAT gateway is what carries that boot, and every later egress; it also gives
the install one stable outbound address, which an upstream allowlist can name.
It is the one resource the AWS and GCP modules have no counterpart for, and
the one line item on this shape's bill that those two do not have.

Security group Copy Section

Three ports, and nothing else reaches the instance from outside: 3000, 3001
and 3002 from client_cidrs. There is no port 22 and no probe range: the
serial console and Run Command reach the instance through the platform rather
than through the network, and the health extension asks /health from inside
the instance. The orchestrator's unauthenticated control port 5008 in
particular stays inside the instance.

Identity Copy Section

The instance's identity holds two grants, and anyone with a shell on the
instance holds them with it:

• Storage Blob Data Reader on the container, not on the account: it reads
  the two files the module uploaded and nothing else.
• A custom role scoped to the resource group with five actions:
  Microsoft.Network/networkInterfaces/read and /write,
  Microsoft.Network/publicIPAddresses/read and /join/action, and
  Microsoft.Network/virtualNetworks/subnets/join/action. That is the
  narrowest grant that attaches the address. The built-in role for the job is
  Network Contributor, which carries every other write in Microsoft.Network
  with it. The join on the subnet is there because a network interface is
  written whole: adding the address also re-states the subnet the interface is
  already in, and Azure authorizes that with a join. The resource group is the
  scope because the interface does not exist until the scale set launches an
  instance, so there is no narrower scope to name it at; the group holds this
  install and nothing else.

The third role assignment is not the instance's: it gives whoever runs
terraform apply Storage Blob Data Contributor on the same container, because
with shared keys refused, writing a blob is authorized the same way reading
one is.

Custom role definition names are unique per tenant, so a second install in the
same tenant needs its own name.

Timings Copy Section

Measured on Standard_D4ds_v7 in eastus. Most of the first boot happens
while apply is still waiting for the scale set, which is why /health
answers so soon after it returns. The 60-second row is a fixed wait: a role
assignment takes some time to reach the service that enforces it, and a call
inside that window is refused as if the assignment were never made. If an
apply still fails on a blob with AuthorizationPermissionMismatch, run
terraform apply again.

To follow a boot:

eval "$(terraform output -raw run_command) 'tail -n 100 /var/log/cloud-init-output.log'"
eval "$(terraform output -raw run_command) 'journalctl -u e2b-embed -n 100'"

From outside, az vm boot-diagnostics get-boot-log --ids prints the
console log of an instance that never got far enough to run a command.

Reaching the instance Copy Section

run_command is a shell snippet that looks the current instance up and runs a
script on it through the guest agent, which is why it is used through eval;
put the script in quotes inside the eval, as the examples above do. It needs
no open port.

When the stack is too broken for that, the serial console is the way in:
az serial-console connect --resource-group <name> --name , user
e2b, password terraform output -raw admin_password. That is the only thing
the password is for; no port 22 is opened, on purpose.

Upgrading Copy Section

A newer ref points at newer files. terraform apply uploads them to the
container and records a new scale-set model, which the next instance uses, but
it does not replace the running one, and that instance wrote its
compose.yaml and .env once, on first boot. Replace it deliberately:

az vmss scale --resource-group <name> --name <name> --new-capacity 0
az vmss scale --resource-group <name> --name <name> --new-capacity 1

<name> is the module's name, e2b-embed unless you set it, for both the
resource group and the scale set. The two steps are the point: the instance
holds the public IP, so it has to go before the next one comes up, and
scaling to zero and back is what makes that explicit. The new instance boots
the newest model, attaches the same address and rebuilds base from a fresh
disk, so the templates you built are gone. A replacement the scale set makes
on its own, after the health extension reports the instance dead, uses the
newest model too.

The image follows the same rule: version = "latest" records a new model when
Canonical publishes a new Ubuntu 24.04 image, and the next replacement boots
it. Set image.version to pin one.

compose_base_url points the first boot at the Compose files of a specific
commit instead of the files the module ships, for example
https://raw.githubusercontent.com/e2b-dev/runtime/<commit>/embed/compose.

Secrets Copy Section

ADMIN_TOKEN, SANDBOX_ACCESS_TOKEN_HASH_SEED and the team API key are all
generated per install, here by Terraform itself: the startup script writes all
three into the instance's .env before the first up, so the pair the
api-secrets one-shot generates on the instance is never read. To rotate
either api secret, replace its resource and then replace the instance as
Upgrading describes: terraform apply -replace=module.e2b.random_bytes.admin_token
or -replace=module.e2b.random_bytes.sandbox_access_token_hash_seed records a
new model; a new hash seed ends the tokens of running secure sandboxes.

They live in the Terraform state, so treat the state file and terraform show
as secret, and they reach the instance in its custom data, which the scale-set
model keeps. Anyone allowed Microsoft.Compute/virtualMachineScaleSets/read
on the resource group can read it, and read-only access such as the built-in
Reader role includes that action. So can any process on the host, root or not,
through the instance metadata service. The startup script does not print them;
read the key with terraform output -raw e2b_api_key. Set team_api_key to
choose the key instead; change it and replace the instance to rotate it.
Keeping the three out of custom data, in a key vault only the instance's
identity can read, would narrow who can see them; this module does not do
that.

The fourth secret is the e2b user's password, for the serial console only
(Reaching the instance). It exists because Azure refuses a Linux instance with
neither a password nor an SSH key.

Azure encrypts the OS disk at rest with a platform key, so the .env and the
three secrets in it are encrypted without a setting here. The storage account
refuses shared keys, so no account key and no SAS exists for the container:
the instance reads the two files with its identity's token, and terraform apply writes them with yours.

The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the instance serves plain http; sign-out
clears it.

Telemetry Copy Section

otel_collector = true runs the built-in OpenTelemetry collector on the
instance, keeping the sandbox and team metrics behind the dashboard's
monitoring charts in its own ClickHouse, and otel_collector_grpc_endpoint
sends the services' metrics, traces and logs to a collector of your own. Set
one or the other: with both, the services export to your endpoint and the
built-in collector receives nothing, so the dashboard's charts stay empty.
Both reach the instance's .env at first boot only, so changing either later
takes the replacement under Upgrading; the reference's
Observability has the rest.

Limitations Copy Section

• No persistent data volume. Every replacement starts from a fresh disk,
  rebuilds the base template and loses the templates you built.
• Stopping the stack gets the instance replaced. The health extension reports
  it unhealthy about two minutes after /health stops answering, and the
  scale set then replaces it from a fresh disk. So docker compose down on
  the instance is not a pause: suspend repairs first with
  az vmss update --resource-group <name> --name <name> --enable-automatic-repairs false,
  and turn them back on once /health answers again with
  az vmss update --resource-group <name> --name <name> --enable-automatic-repairs true --automatic-repairs-grace-period 15.
  Re-enabling demands the grace period restated, and the flag takes bare
  minutes, not an ISO 8601 span -- the CLI wraps the value itself, so PT15M
  is refused as PTPT15MM. A later terraform apply also restores the
  policy, since the suspension lives outside the Terraform state.
• Nested virtualization cannot be checked before the apply. Azure decides it
  by the size alone, with no per-instance flag and no API to ask at plan time,
  so instance_size is held to the generations that have it rather than to
  what your region and subscription actually offer. A size either does not
  offer fails at apply with Azure's own message; a size without /dev/kvm
  would boot and then refuse every sandbox. Check with
  az vm list-skus -l <region> --resource-type virtualMachines before
  changing it: SKU availability varies by subscription offer as well as by
  region, and the default is not available in every one.
• Template builds with copy() steps upload through the orchestrator's port
  5008, which the security group keeps closed -- client_cidrs opens only
  3000, 3001 and 3002, and the upload URL the API hands back points at the
  instance's own loopback besides. Run such builds on the instance itself
  (the run_command output); building them remotely is not supported.
• x86-64 only. The Cobalt sizes are arm64 and the module's default image is
  x86-64 Ubuntu.
• One region, one zone. The instance is a scale set of one; zone moves it,
  and emptying zone places it regionally for a region that has no zones.
• Azure Government and the sovereign clouds are untested. Canonical publishes
  the same image there and the module names no endpoint of its own, so it
  should work with the provider pointed at one of them.

Variables Copy Section

A worked call is in examples/basic/main.tf, which
make lint validates along with the module.

Overview (`embed/terraform/gcp/README.md`) Copy Section

E2B Embed on one GCE instance with Terraform

Creates one Ubuntu 24.04 VM with nested virtualization, in a managed instance
group of one, which installs Docker, writes the two files Embed ships and runs
docker compose up -d --wait. It is a single-machine package rather
than a deployment pattern; the hub is
../../README.md.

Requirements Copy Chapter

• Terraform 1.7.5 or newer.
• A GCP project with the Compute Engine and Cloud Resource Manager APIs
  enabled (the second one grants the instance's service account its logging
  role), and credentials for it (gcloud auth application-default login, or
  a service account).
• No machine of your own. The module creates it.
• The module source has to be the repository, not a copy of this directory:
  it reads ../../compose/compose.yaml and ../../compose/.env relative to
  itself, which the github.com/...// source form below provides by cloning.
  A copy of this directory alone fails at plan time.

Install Copy Chapter

module "e2b" {
  source       = "github.com/e2b-dev/runtime//embed/terraform/gcp?ref=<commit>"
  project_id   = "my-project"
  client_cidrs = ["203.0.113.0/24"]   # where your SDK clients connect from
}

output "api_url" { value = module.e2b.api_url }
output "sandbox_url" { value = module.e2b.sandbox_url }
output "dashboard_url" { value = module.e2b.dashboard_url }
output "ssh_command" { value = module.e2b.ssh_command }
output "e2b_api_key" {
  value     = module.e2b.e2b_api_key
  sensitive = true
}

terraform init && terraform apply

A module's outputs are not the root module's, so the five output blocks are
what makes terraform output see them; without them the Try it commands below
print nothing.

?ref=<commit> pins the source to a commit of the public repository's main
branch, so an init on another day fetches the same compose.yaml and .env;
upgrading is a deliberate change of that ref. Use main itself for the newest
code. The first instance answers GET /health about two
minutes after apply returns, the Docker install and the image pulls behind
it. The base template build runs on past that, so the first sandbox can be
created about four minutes after apply returns.

Try it Copy Chapter

Put this install's three SDK variables in your shell:

export E2B_API_URL="$(terraform output -raw api_url)"
export E2B_SANDBOX_URL="$(terraform output -raw sandbox_url)"
export E2B_API_KEY="$(terraform output -raw e2b_api_key)"

Open terraform output -raw dashboard_url in a browser and paste the key
(terraform output -raw e2b_api_key) into its key form. The instance's .env
carries E2B_DASHBOARD_HOST=, so the browser reaches sandbox
traffic at the same address, no tunnel needed.

Install the SDK in a virtual environment, which is what keeps it off the
system Python that Ubuntu's pip refuses to write to. This runs on your own
machine, not on the instance; on Ubuntu install python3-venv first with
sudo apt-get update && sudo apt-get install -y python3-venv, or it fails
with ensurepip is not available.

python3 -m venv .venv && .venv/bin/pip install e2b==2.46.0

Then create a sandbox. Save this and run it with .venv/bin/python:

from e2b import Sandbox
sbx = Sandbox.create("base")
print(sbx.commands.run("echo hello from the sandbox").stdout)
sbx.kill()

The first attempt can fail with 404: tag 'default' does not exist for template 'local-dev-team/base'. That is the base build still finishing,
not a broken install: wait a minute and run the snippet again.

curl -s -H "X-API-Key: $E2B_API_KEY" $E2B_API_URL/v2/templates lists
base from the moment its build starts rather than when it finishes, so a
listing is not the go-ahead for the snippet above. Or run the packaged smoke
test on the instance itself: it does the same with the JavaScript SDK in a
container, then reaches a port inside the sandbox through client-proxy:

eval "$(terraform output -raw ssh_command)"
cd /opt/e2b && sudo docker compose --profile test run --rm smoke

Remove Copy Chapter

terraform destroy   # the instance and every other resource the module made

Reference Copy Chapter

What it creates Copy Section

Fourteen resources: a VPC and a subnet; a reserved external address; a
service account and its roles/logging.logWriter binding; an instance
template (n4-standard-4, a 50 GB Hyperdisk, nested virtualization on, the
two files in instance metadata); a health check and a zonal managed instance
group of one that auto-heals on GET :3000/health; three firewall rules; and
the three secrets under Secrets below.

The group recreates the instance when it fails its health check. A recreated
instance starts from a fresh disk, so it rebuilds the base template and the
templates you built are gone.

Firewall Copy Section

Three rules, and nothing else reaches the instance from outside: 3000, 3001
and 3002 from client_cidrs, 3000 from Google's health checkers, and 22 from
IAP. The orchestrator's unauthenticated control port 5008 in particular stays
inside the VM.

Timings Copy Section

Measured on the default n4-standard-4. A replacement takes longer than it
looks: the old instance keeps answering /health for the first 40 seconds or
so, so wait for the group's currentAction to read NONE before timing it.
To follow a boot:

eval "$(terraform output -raw ssh_command)"
sudo journalctl -u google-startup-scripts -f

The ssh_command output is a shell snippet that looks the current instance
name up, which is why it is used through eval.

Secrets Copy Section

ADMIN_TOKEN, SANDBOX_ACCESS_TOKEN_HASH_SEED and the team API key are all
generated per install, here by Terraform itself: the startup script writes
all three into the instance's .env before the first up, so the pair the
api-secrets one-shot generates on the instance is never read. To rotate
either api secret, replace its resource and then replace the instance as
Upgrading describes: terraform apply -replace=module.e2b.random_bytes.admin_token
or -replace=module.e2b.random_bytes.sandbox_access_token_hash_seed changes
the instance template; a new hash seed ends the tokens of running secure
sandboxes. They live in the Terraform state, so treat the state file and
terraform show as secret, and they reach the instance through its
startup-script metadata, which anyone with compute.instances.get on the
project can read and which any process on the instance can read from the
metadata server without authentication. The startup script does not print
them; read the key with terraform output -raw e2b_api_key. Set
team_api_key to choose the key instead; change it and recreate the instance
to rotate it.

The dashboard keeps the key you paste in an httpOnly browser cookie for a
year, not marked Secure because the instance serves plain http; sign-out
clears it.

Upgrading Copy Section

A newer ref points at newer files, which changes the instance template.
terraform apply records the new template but does not replace the running
instance, because the group's update policy is opportunistic, and an
auto-healing recreate in the meantime uses the template the instance was
created from rather than the new one. Replace it deliberately:

gcloud compute instance-groups managed rolling-action replace <name> --project <project> --zone <zone> --replacement-method=recreate --max-surge=0 --max-unavailable=1

The three flags matter: the instance holds a reserved address, so the
replacement must be delete-before-create. That is a fresh instance, so the
base template is rebuilt and the templates you built are gone. The command
also sets the group's update policy type to proactive; the next
terraform apply sets it back, which is the only change it will show.

The instance's .env and compose.yaml are written once, on first boot, so
an install created before the dashboard existed keeps running the Compose file
it was created with, which has no dashboard, until that replace.

compose_base_url points the first boot at the Compose files of a specific
commit instead of the files the module ships, for example
https://raw.githubusercontent.com/e2b-dev/runtime/<commit>/embed/compose.

Telemetry Copy Section

otel_collector = true runs the built-in OpenTelemetry collector on the
instance, keeping the sandbox and team metrics behind the dashboard's
monitoring charts in its own ClickHouse, and otel_collector_grpc_endpoint
sends the services' metrics, traces and logs to a collector of your own. Set
one or the other: with both, the services export to your endpoint and the
built-in collector receives nothing, so the dashboard's charts stay empty.
Both reach the instance's .env at first boot only, so changing either later
takes the replace under Upgrading; the reference's
Observability has the rest.

Limitations Copy Section

• No persistent data disk. Every recreate or replace starts from a fresh
  disk, rebuilds the base template and loses the templates you built.

• Template builds with copy() steps upload through the orchestrator's port
  5008, which the firewall keeps closed. Run them on the instance, or tunnel
  the four ports and point the SDK at http://127.0.0.1:3000 and
  http://127.0.0.1:3002:

  bash Copy

  eval "$(terraform output -raw ssh_command) -- -N -L 3000:127.0.0.1:3000 -L 3001:127.0.0.1:3001 -L 3002:127.0.0.1:3002 -L 5008:127.0.0.1:5008"

Variables Copy Section

A worked call is in examples/basic/main.tf, which
make lint validates along with the module.

Overview (`packages/orchestrator/cmd/simulate-gcs-traffic/readme.md`) Copy Section

To run inside of a host in GCS, trust this fancy one-liner:

host=@ && bucket= && go build -o simulator . && ssh $host rm ./simulator && scp ./simulator $host:~/simulator && ssh $host -- ./simulator -csv-path gcs.csv -test-duration=5s $bucket && ssh $host cat gcs.csv

Overview (`packages/shared/pkg/apiid/README.md`) Copy Section

Resource IDs at HTTP boundaries

apiid.ProjectID and apiid.WorkspaceID accept a legacy UUID or the matching
public ID. Both inputs produce the same struct:

projectID.UUID     // uuid.UUID
projectID.PublicID // canonical prj_ string

The existing pkg/id codecs remain unchanged. These structs are HTTP input
values; database and domain functions can continue taking uuid.UUID.

Adding or changing an API ID Copy Chapter

Update id and this package
in the same PR whenever adding a resource kind to either package. Follow the
linked checklist: id owns the codec, and apiid provides the matching input
struct and binding support. Do not add an API-only encoding or a new codec
without its API type. Keep existing kinds' encoding and validation changes,
tests, and documentation synchronized across both packages.

OpenAPI and generated Gin handlers Copy Chapter

Use the standard oapi-codegen type override on a string schema:

projectID:
  name: projectID
  in: path
  required: true
  schema:
    type: string
    description: Project UUID or prj_ public ID.
    x-go-type: apiid.ProjectID
    x-go-type-import:
      name: apiid
      path: github.com/e2b-dev/infra/packages/shared/pkg/apiid

Regenerate the server and dependent clients using the API's generation command.
No middleware, registry, or generator fork is needed:

func (s *Server) GetProject(c *gin.Context, projectID api.ProjectID) {
    project, err := s.projects.Get(c.Request.Context(), projectID.UUID)
    // Handle err and build the response as usual.
}

Use apiid.WorkspaceID for workspace parameters. Apply the same schema override
to query or header parameters and to JSON body properties or array items:

workspace_id:
  type: string
  description: Workspace UUID or wrk_ public ID.
  x-go-type: apiid.WorkspaceID
  x-go-type-import:
    name: apiid
    path: github.com/e2b-dev/infra/packages/shared/pkg/apiid

After c.ShouldBindJSON(&body), use body.WorkspaceId.UUID and
body.WorkspaceId.PublicID. Keep response UUID fields unchanged unless a
response-contract change is intended. Do not keep format: uuid on inputs
that also accept public IDs.

Handwritten Gin binding Copy Chapter

var params struct {
    ProjectID apiid.ProjectID `uri:"projectID" binding:"required"`
}
if err := c.ShouldBindUri(&params); err != nil {
    c.AbortWithStatus(http.StatusBadRequest)
    return
}
projectUUID := params.ProjectID.UUID

ShouldBindQuery and ShouldBindHeader work with form and header tags.
UnmarshalText supports generated path binding; Bind supports oapi-codegen's
exploded query binding; UnmarshalParam supports Gin's own binders. Tests cover
each path because these binders do not all use the same interface.

Construction and encoding Copy Chapter

Use apiid.NewProjectID(u) or apiid.NewWorkspaceID(u) when constructing an ID
in Go. Decoders and constructors populate both fields. Malformed input and
wrong-kind public IDs fail without modifying the receiver. JSON accepts only
strings; encoding emits a public ID string, not a JSON object.

The fields are exported for direct reads. Treat decoded values as immutable;
Go cannot stop a caller from manually making them inconsistent. Encoding derives
the public string from UUID, rather than trusting a manually edited PublicID.
A zero-valued struct denotes an unset input; requiredness belongs in the request
schema or Gin validation. Use a pointer for optional inputs when absence matters.

Other APIs adopt the types by updating their schemas and regeneration, not by
installing middleware. This package does not change their contracts automatically.

Overview (`packages/shared/pkg/id/README.md`) Copy Section

Typed public IDs

This package converts resource UUIDs to and from human-readable public IDs:

prj_7mfb0gpp6c8dsaasre0asc7n3s
wrk_2n1t201rmv87aae5j4csam8000
sec_01kyjhkfvse96rg4d2qzd9enft

Every public ID has a stable resource prefix, an underscore, and a
26-character UUID suffix. E2B's public representation is always lowercase.

UUID ownership Copy Chapter

Resource owners remain responsible for generating and storing UUIDs. UUIDs
may, for example, be generated by application code or a database. This package
does not use typeid.New or typeid.MustNew.

github.com/sumup/typeid is used only to
convert an existing UUID to its prefixed public representation and parse that
representation back into the UUID-backed resource type. PostgreSQL and other
persistence layers continue to store the underlying UUID, not the prefixed
string.

projectUUID := uuid.New()
projectID := id.ProjectID(projectUUID)

publicID := projectID.String()
parsedID, err := id.ParseProjectID(publicID)
if err != nil {
	return err
}

projectUUID = uuid.UUID(parsedID)

Project and workspace UUIDs are normally UUIDv4 and use typeid.Random.
Secret UUIDs are UUIDv7 and use typeid.Sortable. SumUp renders Random IDs in
uppercase, so the E2B wrapper normalizes them to the lowercase public contract.

Team IDs Copy Chapter

Teams are projects: a team UUID is also its project UUID.
ConvertTeamIDToProjectID renders a team UUID as its public prj_ ID, and
ParseTeamID accepts either spelling wherever an API takes a team ID.

Adding or changing a resource ID Copy Chapter

Keep this package and apiid in sync in the same PR.
Adding a resource kind here requires its matching API input type in apiid;
adding a type in apiid requires a matching codec here. Changes to an existing
kind's prefix, encoding, or validation must also update the affected API type
and tests. Encoding rules belong here; apiid must reuse them, not duplicate them.

To add another public resource ID:

1. Choose a short, stable, lowercase prefix that keeps the full public ID at
   the expected fixed width.
2. Add a private prefix type implementing Prefix().
3. Select typeid.Random for a UUIDv4 resource or typeid.Sortable for a
   UUIDv7 resource.
4. Add a distinct UUID-backed public Go type with String and parse methods.
5. Keep UUID generation in the resource-owning application or database.
6. Test the exact prefix, lowercase output, fixed length, invalid-prefix
   rejection, and UUID round trip.
7. Add the matching apiid struct with UUID and PublicID, its constructor,
   and its text, JSON, generated-query, and Gin binding methods.
8. Test that UUID and public inputs produce identical API structs, wrong kinds
   are rejected, and encoding agrees with this package's codec. Update both
   READMEs and regenerate affected API contracts and clients.

Do not add JSON, text, SQL, or UUID-generation behavior implicitly. Those
interfaces should be introduced only when a concrete caller needs them.

Overview (`packages/shared/pkg/sandboxtypes/README.md`) Copy Section

Shared sandbox types

This package owns RuntimeMetadata, SandboxType, and EgressClass, including
identity logging and the empty-type fallback to a regular sandbox. It contains
no worker resources, network initialization, or VM lifecycle management.

Callers import these types directly. SandboxType.EgressClass() selects the
traffic class; DSCP configuration and socket handling belong to worker networking.
The resource-owning Sandbox stays in the orchestrator package. Runtime team
metadata is attribution, not trusted authorization data.

Overview (`packages/envd/internal/services/legacy/readme.md`) Copy Section

The python SDK had a bug: new fields introduced to the API would throw exceptions when processed. In order to work around this, we convert all responses to a list of structs that were valid when the python SDK still had the bug.

We copy the proto files to this folder in order to keep track of what was available when this SDK was supported. These should never be modified.

This entire package can be removed once we no longer have clients that send a user agent of connect-python.

Overview (`packages/api/internal/sandbox/reservations/redis/README.md`) Copy Section

Redis Reservation Storage

This package coordinates sandbox creation reservations across API instances.

Keys Copy Chapter

• Storage index: sandbox:storage:{teamID}:index
• Pending zset: sandbox:storage:{teamID}:reservations:pending
• Result key: sandbox:storage:{teamID}:reservations:{sandboxID}:result
• PubSub routing key: sandbox:storage:{teamID}:reservations:{sandboxID}:notify

Flow Copy Chapter

Reserve runs a Lua script that atomically removes stale pending entries, checks whether the sandbox already exists or has already pending start, enforces the team limit using SCARD(storage index) + ZCARD(pending zset), deletes any stale result key, and adds the sandbox ID to the pending zset.

When creation completes, it removes the sandbox from the pending zset, writes a TTL result key containing either the sandbox or the creation error, and publishes the routing key.

A waiter subscribes to the routing key, probes the result key immediately, then waits for PubSub notifications or the 1 second fallback ticker. PubSub is best-effort; the fallback ticker is required for correctness.

Release is called when the sandbox is removed from storage (Store.Remove). It removes the sandbox from the pending zset, deletes the result key, and publishes the routing key.

ARCHITECTURE (`docs/ARCHITECTURE.md`) Copy Section

E2B Infrastructure Architecture

This document explains what this repository implements, what each service does, and how the
services interact. It is the fastest way to build a mental model of the codebase before diving
into the code.

Keep this document updated. If a code change alters anything described here (service
responsibilities, ports, protocols, data stores, flows, deployment topology), update this file
in the same PR.

What this repository implements Copy Chapter

E2B provides sandboxes: isolated Linux VMs that start almost instantly (they resume pre-booted
snapshots instead of cold-booting), run arbitrary code (typically
generated by AI agents), and can be paused, snapshotted, and resumed. This repo contains the whole
backend: the control-plane REST API, the data-plane VM orchestration built on
Firecracker microVMs, the in-VM agent, the edge routing layer, and template building.

Two ideas drive the design:

1. A sandbox is a resumed snapshot. Templates are pre-booted VM snapshots (memory + disk +
   VM state) stored in object storage. "Creating" a sandbox means restoring a snapshot, which is
   why startup is fast. Memory pages are loaded lazily on page-fault (userfaultfd) and the root
   filesystem is a copy-on-write overlay, so only touched data is ever fetched. A template can
   also be filesystem-only (rootfs, no memfile), taken by a filesystem-only pause or checkpoint;
   sandboxes created from it cold-boot from disk instead of resuming.
2. Control plane and data plane are separate. The API decides where a sandbox runs and
   tracks that it runs (Postgres/Redis); the orchestrator on each node owns how it runs
   (Firecracker, networking, storage). Sandbox traffic never passes through the API.

System overview Copy Chapter

/* Detailed source-code truncated for AI context efficiency. */

Services Copy Chapter

Supporting packages: packages/shared (protos, telemetry, storage clients, feature flags),
packages/auth (authentication library), packages/db (Postgres migrations + sqlc queries),
packages/clickhouse (ClickHouse schema + clients), packages/otel-collector (collector config),
packages/nomad-nodepool-apm (autoscaler plugin), packages/local-dev (local stack).

API (`packages/api`) Copy Section

The control-plane entry point (Gin, OpenAPI-generated from spec/openapi.yml, port 80).

PUT /unstable/management/projects/{projectID}/settings applies a revisioned settings
projection from the owning management service. This endpoint belongs to the API
and accepts only service JWTs through AdminJWTAuth. It locks the team and checks
that it has not been deleted before applying the projection. The shared ProjectSettings type
contains SandboxSettings, with the optional ingress default under
sandboxSettings.defaultAllowPublicTraffic. Storage uses projection.project_settings,
with revision and nullable sandbox_default_allow_public_traffic in the same row.
The receiver applies both in one guarded upsert. Equal or older revisions cannot
overwrite it. Clearing stores NULL at a newer revision, retaining the ordering fence.
Cache invalidation follows commit and is retried on duplicate
delivery after a cache failure. There is no rollout flag check on accepted delivery.
Sandbox creation resolves an absent default to true and honors a saved default
independently of rollout eligibility. New sandboxes can use an older value while
delivery or cache refresh is pending.

• Resources: sandboxes (create/list/kill/pause/resume/connect/timeout/metrics/logs),
  templates and builds, teams, volumes, API keys, secrets, sandbox events and webhooks,
  admin operations.
• Spec document: GET /openapi.json serves, without authentication, the spec the request
  validator runs. An ETag over the document answers a matching If-None-Match with 304. It is
  registered before the validator, which rejects undeclared paths.
• Auth (via packages/auth): team API keys (X-API-Key, e2b_ prefix), auth-provider JWTs
  (OIDC), and either an admin token or a service JWT verified from the configured admin JWKS.
  Backed by an auth DB (Postgres) with a Redis team cache.
• API-group rate limits: operations declare a group through the x-api-group OpenAPI
  extension. After authentication succeeds, its callback wrapper reads the validator's
  already-matched operation and stores the API group directly in Gin's context. The rate limiter
  reads that group and maps it to a field in the authenticated team's limits. Grouped operations
  require team-identifying authentication; public operations do not populate an API group.
  The list group uses api_team_rps_list; the delete group uses api_team_rps_delete only
  for sandbox and template deletion. Other DELETE operations are ungrouped.
  Delete rates default to zero until configured.
  A group's rate is one of the team's limits and resolves with the rest of them in the statement
  that loads the team, which authentication caches; it has no source of its own.
  Propagated project limits carry both rates, and each accepted revision replaces
  them. A team with no project_limits row uses the deployment's default_limits; zero
  disables group limiting. The migration that added default_limits gave every existing team a
  row holding its limits at the time, so a change to the defaults reaches only teams created since.
  A positive rate sets both requests per second and burst capacity. Redis enforces a separate
  budget for each team, HTTP method, and route template across API replicas: groups select
  rates, but routes do not share a budget. Exhausted budgets return HTTP 429 with rate-limit
  and Retry-After headers. Ungrouped operations and unconfigured rates bypass group limiting;
  Redis failures are logged and counted while requests proceed.
  V1 route limits from rate-limit-config are always enforced first. For requests that pass V1,
  LaunchDarkly's rate-limit-v2-mode is the master switch for all V2 checks, and
  rate-limit-delete-mode can further restrict delete checks. Both default to disabled;
  missing, invalid, or unknown modes are treated as disabled. Deletes skip V2 if either flag is
  disabled and enforce only when both are enabled. If both are active and either is shadow,
  deletes run in shadow mode. Lists follow the global mode.
  The groups use the same limiter logic: shadow observes V2 decisions while
  preserving V1 headers, and enabled also enforces V2. Ungrouped operations, unconfigured V2
  rates, and V2 Redis errors retain V1 enforcement. A rejection reports the rejecting limiter's
  headers; when both checks allow a request in enabled mode, V2 supplies the rate-limit headers.
• Sandbox list snapshots: a list request reads the team's whole sandbox index from Redis
  (its running and pausing records) and pages in memory, so with the LaunchDarkly
  sandbox-list-max-lag-ms flag set for a team, each API replica shares one read between
  that team's concurrent list requests and keeps it for the flag's value, counted from the
  start of the read; a request that joins a read already in flight gets that read's result
  however long it takes. A lowered flag applies to a live snapshot; a raised one from the
  next read. Within that lag a sandbox created, killed, paused or resumed can be missing from
  the list or shown in its previous state on the replica holding the snapshot, and one
  resumed inside it can appear on two pages of a walk served by different replicas. The
  default, zero, gives every request its own read with the same store commands as before.
  Only the two list endpoints use the snapshot; the evictor, reconcile, template delete and
  team teardown keep reading the store directly.
• Workload identity: sandbox create accepts an optional iam.tokens map of caller-named
  workload-token definitions (each an exact audience and tokenType). A non-empty, validated
  map enables workload identity, whose identity the orchestrator derives from the sandbox's
  already-authoritative team/sandbox/execution/template IDs; the definitions are passed to the
  orchestrator in SandboxConfig.iam. The API mints no credential and delivers nothing into the
  sandbox; file-based delivery is rejected at admission. Definitions are persisted in the
  running-sandbox (Redis) and paused-snapshot (Postgres) state so they survive pause/resume and
  orchestrator re-sync; a fork starts a new workload and does not inherit them.
• Placement: keeps a live map of orchestrator nodes (discovered via Nomad, Kubernetes, or a
  static list). Chooses a node per sandbox with a best-of-K algorithm
  (internal/orchestrator/placement/): sample K ready nodes, score by the
  higher of CPU load and hugepage-pool load, pick the lowest; retry on
  exhausted nodes. A node that refuses a create (ResourceExhausted) stays
  eligible, and each refusal waits a full-jitter backoff that doubles up to a
  cap, so a placement that finds every node busy does not re-send creates as
  fast as they are refused. The hugepage-pool load is measured against the node's
  base pool (nr_hugepages, allocated at boot), so a node that has borrowed surplus
  pages scores above 1 instead of reading as just full; a pool that is all overcommit
  is measured against its ceiling, and an orchestrator that does not report the base
  pool against HugePages_Total as before. A node that reports no pool scores 0.5. Hugepage scoring
  is on by default (BEST_OF_K_HUGEPAGE_MEMORY); set it false to rank on
  CPU alone. K, overcommit ratio, and alpha are tunable live via feature
  flags. With placement-hugepage-limit-percent above 0, a hugepage sandbox
  is also refused by any node whose committed hugepages would pass that share
  of the node's pool ceiling (base pool plus overcommit allowance, reported by
  the orchestrator); within an API replica the check and the in-flight record
  are one step per node, and when the compatible nodes are all full placement
  answers with a capacity error, not "no compatible node". With the
  orchestrator's create-memory-refusal flag on (CREATE_MEMORY_REFUSAL for a
  deployment without LaunchDarkly), a create whose guest memory the pool
  cannot back while a sandbox on that node is still tearing down is refused
  with ResourceExhausted, so placement retries the node once the pages are
  back; with nothing tearing down it fails at once and says why. Off, it fails
  as an internal error and the node is excluded.
• State: writes sandbox records to Redis (source of truth for running sandboxes). It does
  not write routing: client-proxy reads the orchestrator-written sandbox:routing:{id} (see
  "Sandbox routing records"). Persistent entities (templates, builds, snapshots, teams) live in
  Postgres.
• Secrets: /secrets is the only public surface for secret management (create, list, get,
  update, delete). The API authenticates the caller with the customer alternatives above, converts
  the authenticated team UUID to the project UUID the backend knows, checks the customer-secrets
  feature flag, and forwards a metadata-only request over a unary gRPC contract
  (e2b.secretsstore.management.v1) to the secrets store backend named by
  SECRETS_STORE_BACKEND_GRPC_ADDRESS. No caller credential, header or client-supplied tenant
  crosses that hop, and no response - here or in a log, a span or an error - carries a secret
  value. What a caller stores is a runtime marker, not a resolved secret; orchestrator-ee resolves
  a marker to a value at sandbox egress, never here. Without a configured address, or with the flag
  off, the routes stay registered and answer 403. Responses are Cache-Control: no-store, request bodies are capped at 512 KiB, and values
  at 64 KiB. A create carries the cap on live secrets the backend enforces: max_secrets is one
  of the team's limits and resolves with the rest of them in the statement that loads the team (its
  project_limits value, else the deployment's default_limits).
• Sandbox events and webhooks: /events/sandboxes reads a team's sandbox lifecycle events,
  and /events/webhooks manages the webhook subscriptions those events are delivered to, along
  with each subscription's delivery attempts and delivery statistics. The API stores none of
  this: it authenticates the caller, resolves the team's cluster, and forwards the request over
  two unary gRPC contracts (e2b.webhooks.events.v1 and e2b.webhooks.management.v1) to the
  backend named by WEBHOOKS_BACKEND_GRPC_ADDRESS. The team and cluster come from the
  authenticated context, never from the request, and a team with no cluster selects the local
  one. Without a configured address the routes stay registered and answer 503, so the surface
  does not depend on how a deployment is wired. A backend failure the caller cannot act on is
  answered with the endpoint's own message; the outcomes that keep their identity are
  not-found, a rejected request, the webhook-count limit, and a timeout. A create or update
  body carries the webhook's signing secret, so a request rejected before the handler has its
  body kept out of the response, the access log and the span. Reads are capped at 100
  deliveries per page.
• Rig passthrough: admin endpoints under /clusters/{clusterID}/rigs manage a cluster's
  orchestrator node pools ("rigs"). They list rigs, change rig capacity, list and terminate
  instances, and read recent scaling errors. Every handler delegates to the cluster's edge service
  (/v1/rigs/...) through the per-cluster HTTP client that the cluster sync keeps fresh. The API
  holds no cloud credentials and makes no scaling decisions. The caller authenticates with an
  admin credential and never holds the cluster's edge secret. Edge status codes pass through unchanged
  (400, 404, 409, 501). An edge 401 becomes a 500: it means the API is misconfigured against the
  cluster, not that the caller's token is invalid. A cluster with no rig management configured
  returns an empty list. The local cluster has no edge deployment and answers 501 for every rig
  operation.
• Extra listeners: internal gRPC :5009 and edge gRPC :5109 expose ResumeSandbox so
  client-proxy can wake paused sandboxes on incoming traffic.
• Outbox (internal/outbox): a River client on the shared database's river schema
  works jobs that other services enqueue there for work needing the orchestrator
  connections. Its job arguments live in packages/db/pkg/outbox. The one kind today,
  teardown_team_resources, stops a deleted team's workloads; dashboard-api enqueues it when
  it deletes a project. A guard runs first on every
  attempt: a team that exists and is not blocked fails the attempt, and the job retries rather
  than kill a live team's sandboxes; a missing team row does not stop it. Then
  kill_sandboxes kills the team's running sandboxes and succeeds only once the sandbox store
  holds none of them in any state, so a sandbox still pausing or being killed makes the job
  retry. A sandbox that is not running cannot be paused, so no new snapshot of the team
  appears afterwards. Killing through the orchestrator is what removes the sandboxes' store
  records, so the job deletes no Redis state itself, and every attempt is safe to run again.
  Builds are not cancelled: they end on their own build timeout. An attempt is bounded at
  30 minutes; retries back off from 30 seconds to a 15-minute cap, so the 100 attempts last
  about a day before River discards the job and keeps it. Volumes, templates, snapshots and
  the team row are left alone. Steps report
  api.outbox.steps.finished and api.outbox.step.duration; finished jobs report
  outbox.jobs.finished, and the backlog reports outbox.jobs and outbox.oldest_*_age.
  OUTBOX_MAX_WORKERS (default 10) and OUTBOX_BACKLOG_INTERVAL (default 30s) configure it;
  shutdown stops it after the sandbox-work drain, before the database and Redis clients close.
• Reads ClickHouse for sandbox/team metrics endpoints. Sandbox and template-build logs default to
  Loki, with a LaunchDarkly-gated ClickHouse read path (logs-read-config) for local-cluster logs
  during the log storage migration. LOKI_URL is optional: without it the api has no Loki client
  and those reads fail until the flag routes them to ClickHouse. LOGS_READ_CONFIG=true|false sets
  the value the flag falls back to when LaunchDarkly has none (no LaunchDarkly, or the flag not
  defined there); a LaunchDarkly value wins. LaunchDarkly feature flags also gate placement
  parameters, rate limits, and rollouts.

Orchestrator (`packages/orchestrator`) Copy Section

A single Go binary running on every sandbox node (as root). ORCHESTRATOR_SERVICES selects its
roles: orchestrator (run sandboxes) and/or template-manager (build templates). Code lives
under pkg/, almost all Linux-only.

gRPC services on :5008 (pkg/server/, pkg/service/, pkg/template/server/, pkg/volumes/):

• SandboxService — Create, Update, List, Delete, Pause, Checkpoint.
  Update callbacks are serialized per sandbox. A successful transition out of the live
  map closes update admission and joins any admitted callback before stop/checkpoint
  work or stopping notifications begin; Close repeats that check for crash cleanup.
  An update using an inactive sandbox returns gRPC NotFound, which the API maps to
  HTTP 404 for timeout and network updates. Teardown waits for admitted callbacks;
  it cannot preempt one that does not return.
• TemplateService — TemplateCreate, TemplateBuildStatus, TemplateBuildDelete (template-manager role only).
• InfoService — node identity, roles, capacity, health status (used by API node discovery).
• ChunkService / VolumeService — peer-to-peer template chunk serving; persistent volumes.

Process shutdown, triggered by SIGINT, SIGTERM, SIGUSR1, or a service failure, moves
the node to ShuttingDown before draining builds and sandboxes. Like Draining, this excludes new
placement while existing work stays reachable, and /health returns HTTP 200 with status
draining. Unlike reversible Draining, ShuttingDown is terminal for the process;
status overrides cannot enter or leave it. FORCE_STOP skips waiting for sandboxes to exit.
The sandbox drain ends only when no sandbox is live and the node's outstanding work (below) is
zero, so a create or a resume-fresh checkpoint in flight, whose sandbox is briefly not live, holds
it until its sandbox goes live and then ends.

After sandbox drain and snapshot uploads complete, sandbox proxy cleanup attempts graceful HTTP
server shutdown with a separate deadline, then forces server closure if needed. Successful forced
closure recovers graceful deadline expiry; other server errors still propagate. Pprof stays
available through service teardown; its final shutdown also has a deadline and forced-close fallback.

InfoService.ServiceInfo reports outstanding_work for sandbox, template-builder, and
mixed-role nodes. It counts overlapping work holds, not distinct sandboxes or builds, and includes
tracked background persistence and cleanup. The producer reports zero when idle. The API caches
this report and exposes it as required top-level outstandingWork in admin node list and detail
responses, where zero also covers a node that has not reported. This observational count does not
by itself authorize node deletion.

InfoService.ServiceInfo.max_sandboxes reports the cached node limit for every service role.
Admin /nodes and /nodes/{nodeID} return required maxSandboxes, including zero. Admission,
metrics, and node info share the value refreshed from node-scoped flags every 30 seconds.

Key mechanisms (all under pkg/sandbox/):

• Firecracker (fc/): each sandbox is one Firecracker process in its own cgroup and network
  namespace. The FC HTTP API (unix socket) configures machine, drives, network, and snapshots.
  Guest metadata (sandbox ID, envd access token hash) is passed via MMDS.
  A sandbox may use fewer CPUs than its VM has (SandboxConfig.max_vcpus above vcpu): the VM
  is created with max_vcpus vCPUs, a boot puts maxcpus=<vcpu> on the kernel command line, and
  the fc_vcpu threads are held to vcpu CPUs by a cpu.max quota in a threaded vcpu child of
  the sandbox cgroup, so the VMM's own threads stay outside it. The quota is in place before the
  guest runs an instruction: a boot limits the whole cgroup until the vCPU threads exist and then
  moves the limit onto them; a resume confines the paused threads before the VM resumes, falls
  back to the whole cgroup, and fails if neither holds. The snapshot's metadata records the VM's
  vCPU count (vcpu_count, stamped at build and on every pause), so a resume limits below it
  without trusting the request or discovering capacity from cgroup threads. A legacy snapshot
  carries its stored original vCPU count as max_vcpus; if unavailable, it may only resume at its
  unchanged vcpu.
  /init carries the count as cpuCount
  so envd onlines or offlines guest CPUs to match and reports the outcome on X-Envd-Cpus and
  /metrics.
• Lazy memory / UFFD (uffd/): on resume, Firecracker restores the VM without loading
  memory; a userfaultfd handler serves page faults directly from the template's memfile, so only
  touched pages are read. An optional prefetcher warms known-hot pages.
• Copy-on-write rootfs (rootfs/, nbd/, block/): the template rootfs stays read-only;
  writes go to a per-sandbox COW cache exposed to Firecracker as an NBD block device served by
  an in-process userspace NBD server. On pause, the dirty blocks are exported as a diff.
• Template cache (template/): templates are fetched lazily from object storage and cached
  on local disk (and optionally on a shared NFS chunk cache, or fetched peer-to-peer from other
  nodes before upload completes). Every reader of a cached template holds a pin on it — a
  running sandbox, a build, a peer stream, an ancestor lookup — and eviction never closes a
  pinned template. A snapshot's own upload holds a pin while it runs and records on the entry
  whether the snapshot landed in storage; a layer that did not land is never released, so a
  failed upload's local copy stays until its TTL expires. When a pause, or a
  checkpoint that resumes a fresh sandbox, publishes a new layer, the layer its sandbox ran
  from is marked superseded if this node inserted it for an earlier pause, because no resume
  can ask for it again. With snapshot-cache-release-superseded and
  snapshot-cache-ancestor-storage-fallback both on, a superseded layer is released once no
  pin holds it, its devices have resolved and its upload has landed, instead of staying until
  its TTL expires. A descendant's upload then heals that ancestor's build data from storage,
  and writes the same header bytes it would have written with the ancestor still cached.
  The sandbox and template caches start empty after a restart: the template cache empties the
  build-data directory, and startup reclaim empties the sandbox, template and template-build
  scratch directories.
• Networking (network/): each sandbox gets a slot — a network namespace with a veth pair
  and a tap device, unique host-side IP (from a /16), NAT, and per-slot nftables egress firewall
  (with SNI/Host-inspecting TCP firewall for domain allow/deny lists). Slots are pooled and
  reused; slot indexes are allocated locally against the node's netns state (leftover
  namespaces from a previous run are torn down by startup reclaim).
• Sandbox proxy (:5007, pkg/proxy/): reverse-proxies incoming traffic from client-proxy to
  the sandbox's slot IP and requested port over HTTP or configured HTTPS, enforcing per-sandbox
  traffic access tokens through the shared ingress policy. HTTPS backends may use self-signed certificates.
  It refuses a full-router sandbox with 503 sandbox_feature_unsupported before dialing the
  slot.
  Each admitted lifecycle owns its proxy handlers and backend sockets, including pending dials
  and upgrades. Stopping seals admission and interrupts connections; network release waits for
  that work to finish. Restarting envd resets the owner while allowing the live lifecycle to
  admit a replacement. Retired streams have one second to write their terminal response before
  a stalled frontend is interrupted.
  Shared proxy shutdown waits for H2C and upgraded connections to close and includes them in
  forced connection cleanup. Failed H2C handshakes release their hijacked connections when the
  handshake handler returns.
• Writes sandbox lifecycle events and cgroup host stats to ClickHouse; exports metrics via
  OTel. Sandbox and template-build log writes go through a flag-resolved HTTP route: the legacy
  collector remains the fallback primary destination, and configured shadow destinations can mirror
  writes during collector/storage migrations without changing sandbox behavior.

Detached sandbox-event publication holds node work until the publisher returns. This covers
delivery attempts, not end-to-end delivery: the ClickHouse target enqueues into an in-memory
batcher, whose flushing remains part of shutdown.

Envd (`packages/envd`) Copy Section

The agent inside every VM (started by systemd very early in boot), port 49983, chi + Connect RPC.

• Process service (spec/process/process.proto): start/list/connect to processes, stream
  stdout/stderr, stdin, signals, PTYs — this is what SDKs use to "run code". A start is refused
  with unavailable while the workload cgroups are frozen for a pause, and the freeze waits for
  in-flight starts, so a pause does not capture a child stopped before exec in a frozen cgroup. A
  start into a cgroup the guest froze itself is refused the same way. Starts carrying the system
  tag are unaffected: they stay in envd's own cgroup, which no freeze covers.
• Filesystem service (spec/filesystem/filesystem.proto): stat/list/make/move/remove/watch.
• REST: /health, /metrics, /envs, /files upload/download, /init (orchestrator pushes
  env vars, access token, metadata and the online CPU count after boot/resume), /upgrade (live self-upgrade, below),
  freeze/thaw hooks used during pause.
• Public vs. control-plane routes: the control routes (/init, /collapse, /upgrade, and the
  freeze/thaw hooks) are marked x-internal: true in spec/envd.yaml, and /upgrade — which the spec does not
  describe — is listed alongside them in the orchestrator's pkg/sandbox/envd. The orchestrator
  reaches them over the host network at the sandbox slot IP; the sandbox proxy refuses them with a
  404, for every method, so they are not reachable through a sandbox URL. Adding a control route
  therefore means marking it in the spec (go generate carries the marker into the proxy's
  rejection list) — otherwise it ships reachable from the internet.
• Guest trust (internal/host/cacerts.go): /init supplies the desired CA bundle.
  envd atomically records its PEM and owned DER identities in /run/e2b/ca/state.json,
  then publishes a complete replacement of the active bundle with the requested roots
  before returning. Ordinary installs copy the existing bundle without decoding it;
  recovery first repairs interrupted writes. All supplied identities are owned,
  including identical preexisting copies. A background loop resumes saved intent at startup
  and retries cleanup every second. It atomically removes retired roots from the active
  bundle before rootfs work, so source-file failures cannot delay that cleanup.
  It persists each requested certificate as
  e2b-<SHA-256>.crt in /usr/local/share/ca-certificates, removes retired managed files,
  and retains retirement identities until both bundle and source cleanup succeed.
  Superseded cleanup cannot publish an older bundle. Legacy-file removal checks the latest
  desired roots under the init lock, preserving hash links to a still-requested certificate.
  State survives envd process restart,
  not guest reboot. Retired roots may be trusted until cleanup succeeds, and an independent
  trust-store rebuild can restore a copy from a customer source file. Applications that
  cached their trust store must reload it.
• Auth: X-Access-Token header checked against a token delivered via Firecracker MMDS;
  signed URLs for file endpoints. /init is exempt (it is what delivers the token), which is the
  main reason the proxy refuses it outright.
• Live upgrade (internal/services/process/upgrade.go): an authenticated POST /upgrade lets
  the orchestrator swap envd inside a running sandbox at resume. It streams the new binary in the
  request body and envd syscall.Execs into it with the same PID, carrying the workload's
  stdio/PTY fds, process table, recently-retained exit codes and filesystem watchers forward via a
  tmpfs handover blob. The workload cgroups stay frozen until the post-upgrade /init restores the
  access token (so no re-adopted process runs unauthenticated), and the handover outcome
  (procs/watchers re-adopted, plus any failures) rides back on that /init's X-Envd-Handover
  header for fleet visibility.
• Scans guest ports and forwards them so any port a user process opens becomes reachable through
  sandbox URLs. pkg/version.go carries the released version, maintained by release-please from
  the Conventional Commits merged since the last envd release; do not edit it by hand. The API and
  the orchestrator gate features on the envd version recorded in each template build — the
  orchestrator's held /init is the one gate that reads the version of the envd expected to
  answer instead (see sandbox creation) — so a behavioral change ships under a feat/fix
  commit type that cuts a version it can be gated on.

Client proxy (`packages/client-proxy`) Copy Section

The stateless edge for all sandbox traffic (port 3002; health on 3003). Terminates
https://-<sandboxID>.<domain> requests (host parsing in packages/shared/pkg/proxy/host.go),
looks the sandbox up in the Redis routing record to find the owning node, and reverse-proxies to
that node's orchestrator proxy on :5007 by default. ORCHESTRATOR_PROXY_PORT selects a
different downstream port when the node proxy listens elsewhere. If the sandbox is not in the record (paused), it calls
the API's ResumeSandbox gRPC — paused sandboxes wake transparently on traffic.
The record is the orchestrator-owned sandbox:routing:{id} (see "Sandbox routing records"
below).

The record on a hit, or the ResumeSandbox answer on a miss, is the sandbox's route, placement
included. A miss uses the resume answer directly and does not read the record again. A legacy
(no placement) or egress-router route goes to orchestrator_ip on the orchestrator proxy port,
never to the placement's ingress endpoint. A full-router route answers 503
sandbox_feature_unsupported without a Retry-After header and before anything is dialed. An
unknown mode or a missing orchestrator_ip answers the not-found page.

Concurrent requests that miss the record for the same sandbox, port and credentials share one
resume flight per client-proxy process: the first request's call runs detached from its
connection under a 90 s ceiling and later requests join it, so one resume RPC serves them all
and a client that disconnects mid-resume does not fail the others. A call whose requests have
all left runs to its end, so a client that comes back finds the sandbox awake, but no further
attempt starts for nobody, and a process keeps at most 64 such callerless calls; beyond that a
departure cancels the proxy's call. That bound and that cancellation apply to the proxy's flights
only: the API's resume entry point lets a create already under way run on past the proxy's
cancellation, so the node finishes the create and the API registers the sandbox inside the
reservation's 90 s window (the create may run until ten seconds before the pending reservation can
be reaped and the registration until five seconds before, so the run-on never lets a fresh
reservation start a second resume of the same snapshot), and the API's shutdown drain waits for
that work, bounded at 95 s. The run-on is granted when the resume is admitted, and an API already
draining admits none: a resume that arrives then keeps its caller's cut, so a create the proxy
cancels mid-drain is cut with it and the reservation lapses. The sandbox otherwise lives its
ordinary timeout or auto-pause, billed, with nobody waiting for it. With
client-proxy-resume-parking on (default off), a request whose flight got a still-transitioning
or Unavailable answer parks instead of failing: the flight retries
while a parked request waits, under a budget (client-proxy-resume-parking-budget-ms, 15 s from
the first retryable failure) that gates only whether another retry starts: every attempt, a late
retry included, runs under its full 90 s ceiling while a caller waits on it, so a parked request
can wait up to the budget plus one attempt. The proxy cuts a retry at the budget's end only once
every caller has left, and that cut ends the proxy's wait, not the create on the node.
A process keeps at most
client-proxy-resume-parking-lot-size (512) parked requests, shedding the rest. A resume the
control plane cannot carry out right now (no capacity, no node, the API unreachable, the budget
spent) answers 503 with Retry-After: 2 and no error code; a request shed because the lot was
full answers the same 503 without the header and with proxy_parking_lot_full as its
error_code, so a client, or one behind an ingress that strips the header, still tells the two
apart. The SDKs do not act on the header either way: their status-based retry
wraps only the control-plane API client, and sandbox traffic through the proxy is retried only on
a connection failure, never on a 503. The omission is for hand-rolled retry loops and browsers
that honour the header, which would otherwise come back in one wave while the lot is still full,
and for a client that does the SDKs' jittered backoff on its own; a placement no retry helps, which
the API names in an ErrorInfo detail on the RPC status, is not retried and answers without the
header, with the status the API carried: 503 for a feature no orchestrator in the cluster
supports, 500 (sandbox_create_refused) for a sandbox the node refuses to create, such as a
stored build whose vCPU count is outside the node's bounds; a sandbox still pausing
answers 409 with the header on a user port and the same page as 503 on the envd port (the
SDK reads an envd 409 as "already exists"); a request whose client left before it was routed
answers 408.

Dashboard API (`packages/dashboard-api`) Copy Section

A separate REST service (port 3010, spec spec/openapi-dashboard.yml) consumed by the web
dashboard, not the SDK: template tags, build listings, and admin profiles. Its admin
routes accept either a shared admin token or
a short-lived service JWT verified against the configured admin JWKS. Team-scoped template and build read routes accept either dashboard user auth or
team API key auth (X-API-Key). Its workspace-agnostic /v1/management operations are defined in the
same dashboard OpenAPI contract and registered on the existing router. Their AdminJWTAuth
OpenAPI security scheme accepts only short-lived service JWTs verified against the workspace-api
/.well-known/jwks.json endpoint, with accepted signing methods derived from each JWK's required
alg metadata. Issuers and audiences are configured through the JSON ADMIN_AUTH_PROVIDER_CONFIG value —
the same config shape as AUTH_PROVIDER_CONFIG. Talks to Postgres and ClickHouse; never talks to
orchestrators. The web dashboard itself is a separate repository (github.com/e2b-dev/dashboard)
published as an image; E2B Embed runs it on port 3001 beside dashboard-api.

An issuer normally configures at least one accepted audience, binding a JWT to its intended target.
An issuer may omit audiences only with no audienceMatchPolicy; this deliberately disables
audience matching while issuer, signature, and temporal-claim verification remain required.

The /v1/management operations are the cluster's half of a contract the workspace residency owns:
project upsert (a project is a public.teams row created from a caller-supplied UUID; the tier is
assigned once at creation from a local default and no push moves it; an optional cluster ID is
stored on the project, with omission preserving its current assignment; a changed slug renames the project and, in the same transaction, gives each of its namespaced
template aliases under the new slug as well, keeping the old ones and leaving an alias the new
slug already holds as it is), per-member projection,
and limit sync (into project_limits, which the team lookups read in preference to default_limits). All are
idempotent, because the caller is level-triggered and retries. PUT /v1/management/projects/{projectID}/members/{userID} applies the desired presence for one user,
gated by a monotonic per-project/user revision stored in projection.project_members; duplicate or
older revisions succeed without changing target state. PUT /v1/management/projects/{projectID}/limits is gated the same way, by a monotonic per-project
revision in projection.project_limits: the caller raises it whenever the limits it resolved for a
project change, and a delivery at or below the recorded revision is dropped and still answers 204.
The ledger and the values in public.project_limits advance in one transaction, so a revision is
never recorded without the values it admitted. Both fences are the target's, and both exist for the
same reason — the caller can only fence what it sends, so two deliveries in flight arrive in
whichever order the network gives them and the older one has to be refused where it lands. A present projection includes that User's
OIDC issuer/subject identities. Every projected user has at least one identity, and an identity
already owned by a different user returns 409. A revocation removes only that User's
users_teams row; projected Users and identities are retained. Membership writes live in
internal/management with their post-commit cache eviction rather than in the handlers: auth
caches member authorization, so each accepted command invalidates that User's authorization for
the Project after commit.

PUT /v1/management/projects/{projectID}/block applies a revision-fenced block
state. The projection.project_blocks revision and public.teams.is_blocked /
blocked_reason commit together. Older or duplicate revisions leave stored state
unchanged. Every accepted delivery invalidates the team auth cache; eviction
failures return an error so retries can finish invalidation after commit. The API
uses this state when admitting billable work. It is the only route that writes the
team's block state.

Limit and block deliveries may carry decided_at, when the caller decided that
revision. The ledger row stores it beside the revision. Each delivery the ledger
accepts records the seconds since then in dashboard-api.management.apply_lag, by
projection (project_limits, project_blocks); one without decided_at counts
in dashboard-api.management.origin_unknown instead, never as zero lag. Dropped,
refused and failed deliveries record nothing.

The management surface also owns a replay-safe cluster lifecycle. A caller registers a stable
cluster UUID, assigns it only to the named project, detaches that exact assignment before provider
cleanup, and deletes the cluster only after no project references it. Replaying the same
registration or assignment succeeds. Registering the cluster again with the same name and auth
organization may change its endpoint, TLS setting, sandbox proxy domain and token, for example when
the cluster moves to a new domain; the API rebuilds that cluster's clients on its next sync and gives
sandboxes the new domain. A changed name or auth organization, a different assignment, an inexact
detach, or deletion of a referenced cluster returns a conflict. The
assignment does not check the project's tier; the caller decides which projects may use a cluster.
A cluster first registered through this surface starts without deletion protection; one created any
other way starts protected, a later registration never changes it, and deleting a protected cluster
or checking its destroy readiness returns a conflict.
A caller that deletes a cluster's projects before the cluster first lists them with
GET /v1/management/clusters/{clusterID}/projects, which returns every project whose team is
assigned to the cluster however it was assigned, leaving out deleted teams, and records the deletion with
PUT /v1/management/clusters/{clusterID}/deletion before deleting any of them. That request
locks the cluster row, returns a conflict while deletion protection is on, and sets
clusters.deletion_requested_at; it succeeds for an absent cluster and on repeat. From then on a
check constraint keeps deletion protection off, and a trigger on teams refuses to newly assign a
team to the cluster (cluster_assignment_cluster_deleting from the assignment route, a conflict
from an upsert carrying the cluster ID). The trigger takes a key-share lock on the cluster row
before it checks, so an assignment either commits before the deletion is recorded, where the
caller's next listing finds it, or is refused. Deleting each project clears its team's cluster,
the cluster delete clears it from any team deleted earlier, and then succeeds.
The shared error response supports an optional error_code, defined by the dashboard OpenAPI
ErrorCode schema. Registration and assignment rejections currently populate it; other errors
may omit it. Callers use that stable code rather than the human-readable message, and treat
uncoded or unknown reasons as generic errors.

DELETE /v1/management/projects/{projectID} soft-deletes any project, including a member's
default project; whether a project may be deleted is the caller's decision. envs,
snapshots and volumes reference teams with ON DELETE NO ACTION, so the team row
stays. One transaction locks the row (404 if it is missing), sets teams.deleted_at,
blocks the team (a CHECK keeps every deleted team blocked), clears its
cluster assignment, scrubs its name and email to DELETED and its slug to
deleted- (releasing the old slug for reuse), deletes its memberships and API
keys, soft-deletes its templates, snapshot templates and paused-sandbox snapshots the way a
template deletion does (releasing their aliases and build tracking), and enqueues the API's
teardown_team_resources outbox job with River's InsertTx, so the teardown runs exactly
when the deletion commits. dashboard-api holds an insert-only River client for this; it works
no jobs. After commit it evicts the deleted keys' auth cache entries best-effort; the team's
other cached entries refresh or expire on their own. Repeating the call answers 204 and runs
every step again, keeping only the first deletion's time: it re-blocks and re-scrubs the team
and clears its cluster, undoing a block reason, rename or cluster assignment (from the cluster
route or an upsert carrying a cluster ID) written by a delivery that passed its
deleted-project check just before the first call committed; it deletes memberships, keys and
envs created since (a sandbox that paused before the teardown killed it leaves a snapshot);
and it enqueues the teardown again, which is a no-op while one is still open. Afterwards a
dashboard-api middleware, running after authentication, answers 404 to every write under
/v1/management/ or /admin/ that names the project by projectID or teamID (project
upsert, member, limits and block deliveries, cluster assignment and detachment, admin ban);
the project DELETE is exempt so a retry still reaches it. Shared auth refuses a
membership or API key still left for the project with 403, and the API leaves it out of a
user's team list. Volumes, builds and stored artifacts are not reclaimed.

Data stores Copy Chapter

The API and dashboard-api share packages/db/migrations, tracked in PostgreSQL's
_migrations table. This includes env_defaults, which migration 20260915204740
creates on fresh databases and adopts where it already exists, preserving its data.
Any existing _dashboard_migrations table is retained as historical bookkeeping and
requires no updates. Run the db-migrator before deploying updated services to a
database that lacks the table; no separate dashboard migration runner is needed.
Rolling back this migration retains the table and its data.

The same database holds River's job tables in the river schema, which a goose migration
creates. The db-migrator applies the goose migrations and then River's own, under one advisory
lock; make migrate runs it, because the goose CLI cannot apply River's. At startup the API and
dashboard-api refuse a database whose goose version is older than the one they were built against,
or whose River migrations are not current. The API works jobs from that schema (see its Outbox).

A template and a paused-sandbox snapshot have the same artifact shape — a snapshot is just a
new build whose memfile/rootfs are stored as diffs against the template it came from (diff chains
are resolved through the .header files).

Core flows Copy Chapter

Sandbox creation Copy Section

sequenceDiagram
    autonumber
    participant C as SDK
    participant API as API
    participant R as Redis
    participant O as Orchestrator (chosen node)
    participant FC as Firecracker
    participant E as envd (in VM)

    C->>API: POST /sandboxes {templateID}
    API->>API: auth team, resolve template alias → ready build (Postgres/cache)
    API->>API: mint envd access token (build's envd must support secured access)
    API->>API: best-of-K placement → pick node
    API->>O: gRPC SandboxService.Create(SandboxConfig)
    O->>O: fetch template (local cache / NFS / object storage)
    O->>O: acquire network slot + NBD rootfs overlay + uffd memory
    O->>FC: load snapshot, resume VM
    O->>E: POST /init (env vars, access token) — held until envd reads it (older envds: retried until ready)
    E-->>O: 204
    O-->>API: Create OK
    API->>R: store running sandbox
    API-->>C: 201 sandbox {sandboxID, domain, envdAccessToken}

The API blocks on the gRPC Create, which itself blocks on envd's /init — when the client
gets a response, the sandbox is fully usable. Fresh creates are internally a resume of the
template's base snapshot (cold boots happen for filesystem-only templates and builds, or when
an explicit resume requests one — see pause and resume below; template creates never do).

A create that leaves network.allowPublicTraffic unset or null uses the project's
ProjectSettings.SandboxSettings.DefaultAllowPublicTraffic, read from the cached team.
A missing key, including a team cached before settings existed, keeps public traffic. An explicit value wins.
With no network object, a public sandbox stores no network config and a private one stores only
the ingress setting. Either way a private sandbox gets a traffic access token at create, and
resume, auto-resume and fork keep the setting saved with the sandbox rather than reading the
project default again.

V2 create idempotency keeps the former allowPublicTraffic: true fingerprint for an omitted
flag on unconfigured projects, so retries across API versions still replay. Configured projects
distinguish omission from an explicit boolean. The first configuration change can make an older
key return 409 until its idempotency record expires; it does not alter the original sandbox.

How that /init reaches envd depends on the envd expected to answer it. Behind the
envd-init-held-request flag, and for an envd at or above 0.10.0, the orchestrator sends one
held request: it dials the guest first, writes a single /init with Expect: 100-continue
over that connection, and withholds the body — timestamp included — until envd's handler asks
for it, so the body is minted after the request has cleared the guest's accept queue and
dispatch rather than before. The request waits for envd up to the caller's envd-wait budget and
is re-sent only after a transport failure (a refused connect, or a connection that dropped under
it), never on a timer. Older envds, and every sandbox with the flag off, keep the legacy loop:
a complete /init posted and retried until envd answers. The gate reads the envd version it
expects to answer, in this order: the version a confirmed live upgrade delivered in this life,
else the version an offline upgrade swapped onto the rootfs before this boot, else the version
the running envd last reported on /init when it parses as a version, else the running version
the snapshot recorded at pause, else the version baked into the template build. The first two
are binaries the orchestrator staged itself; the rest are the guest's word, which only ever
chooses between the two loops for that guest.

Opt-in request idempotency

POST /v2/sandboxes supports an optional, team-scoped Idempotency-Key to prevent retries from
repeating sandbox creation while a Redis reservation exists. A generic executor owns reservation
and response replay across API replicas. Participating handlers parse and authorize the request,
then call idempotency.Execute(c, requestContent, operation) with all mutations inside the callback.
The executor buffers finite HTTP responses, so streaming and connection upgrades cannot use it.

Admission and request identity

The api-idempotency-routes JSON flag enables participating handlers by HTTP method and Gin route
pattern. For sandbox creation:

{"POST /v2/sandboxes": true}

The flag defaults to null, which disables idempotency. The idempotency middleware bypasses requests
without a key, an authenticated team, or an enabled route. Authentication, schema validation,
rate limits, and blocked-team checks apply before the executor, including on retries. A key is
1 to 255 lowercase ASCII letters, digits, or hyphens (a-z, 0-9, -), so a lowercase UUID is
valid as is. Only the route's OpenAPI schema defines that format, and it applies regardless of the
flag, so a participating route must declare the header with the same schema. The validator checks
only the first header value; the generated parameter binding rejects a repeated header before the
handler runs.

The request fingerprint hashes the HTTP method, concrete URI including query string, and
handler-supplied JSON-serializable content with SHA-256. Handlers include behavior-affecting
headers or caller information in that content. Sandbox creation fingerprints its parsed request
after schema defaults, so whitespace and map ordering do not change identity; array order and
parsed field values do. Template aliases and business configuration resolve during execution.

Reservation and replay

The executor reserves a Redis hash keyed by team ID and the SHA-256 digest of the idempotency key.
The namespace spans participating routes within a team. An atomic reservation script selects one
owner across API replicas and stores the request fingerprint, a random owner token, and the
response deadline in expected_response_at. The owner token also guards completion and cleanup,
so an expired request cannot modify a replacement reservation.

Reservation and execution share the request context's deadline. The response deadline adds a
five-second persistence budget and uses Redis's clock for comparison. The executor reads the
reservation and server time together, then selects the response:

Retries against a reservation do not wait for or repeat the operation. The error_code field
distinguishes the two conflict cases without requiring clients to parse messages. An unknown
outcome means the API cannot establish the result, so retrying with a new key can duplicate work.
A late response from the owner remains replayable while the reservation exists. Admission and
retry errors do not replace the owner's cached result.

Response storage and delivery

The executor captures the handler's status and body, then attempts to persist them before
delivery. Redis stores one JSON value containing the status, base64-encoded body, allowlisted
headers, and optional retryability value. Base64 preserves arbitrary response bytes; the separate serialized
buffer lets Redis finish a write after the HTTP capture buffer has been released. The cache includes
handler errors as well as successful responses, keeping retries attached to one attempt without
inferring which errors permit safe re-execution.

The handler edits response headers in place while the executor buffers status and body writes.
Content-Type, when present, enters the cache. Replay applies that header over the current
request's headers, preserving fresh request IDs, rate limits, and cross-origin resource sharing
(CORS) values. The capture pool reuses allocations up to 64 KiB and erases body contents before
reuse because responses can contain credentials.

Persistence has a detached five-second wait budget and retains request context values for
tracing. If it fails, the API logs the failure and delivers the handler's response: a caller should
receive a known creation result even when Redis cannot store it. Subsequent retries can replay
only a stored response; otherwise they receive the pending or unknown-outcome error.

Cancellation and retention

Before execution starts, cancellation or a reservation error triggers best-effort cleanup. The
reservation worker waits for its Redis call to return before attempting deletion because socket
I/O can outlive request cancellation. Cleanup gets its own five-second timeout and deletes only
an unfinished record with the same owner token. Failed cleanup or an ambiguous Redis write can
leave the reservation until expiry.

Once execution starts, cancellation, a panic, or failed persistence leaves the reservation in
place. An operation can have side effects before writing any response, so releasing its key
would allow duplicate work. The executor does not reconcile an unknown operation outcome.

The api-idempotency-ttl-seconds flag sets retention, defaulting to 86,400 seconds when
unavailable. Configured values must be positive integers. The executor sets the time to live
(TTL) at reservation; completion, retries, flag changes, and deleting the sandbox do not extend
or remove it. Operators must choose retention longer than the operation window: expiry permits
a new owner even if the first operation is still running. Redis eviction or data loss, a different
key, and feature bypass also permit another execution. Redis capacity and access controls must
account for cached response volume and the sandbox credentials those responses can contain.

Sandbox creation retry hints

POST /v2/sandboxes answers errors with Should-Retry, exposed through CORS. The header states
whether the condition behind the error may clear, so that retrying the request could succeed:
true for a transient failure such as missing capacity, a rate limit, or a creation still in
progress (retry after Retry-After when present, otherwise with exponential backoff); false
for a failure a retry cannot fix, such as a malformed request or a configuration fault, and for
an outcome that is unknown, where a retry could create a second sandbox. The header says nothing
about idempotency keys. The executor records every handler result under the key, success or
error, and replays it for as long as the record lives; the caller decides whether a retry keeps
the key or takes a new one, and the error code names the situation that decision depends on.

Each error carries its classification in APIError.Retryable, a Retryability enum, and the
error writer (httperror.Write, behind SendAPIError) sets the header from it: CanRetry and
NoRetry map to true and false. UndefinedRetry (the zero value) leaves the header unchanged:
an absent header stays absent, and a value set earlier is preserved. Callers that discard a
response must explicitly remove its retry hint.
The v2 handler, the rate limiter on that route, and the idempotency executor classify their
errors. Other creation routes share the creation path, so they reset Retryable to
UndefinedRetry before sending. Successful responses omit the header, and so do admission
errors that carry no classification. A recovered panic removes any hint the handler wrote,
because the hint described a response the panic replaced. Idempotency stores the header with
the cached response and replays it, including its absence.

Placement retains uncertainty from earlier attempts even when the last node reports a permanent
error. A sandbox-store write error follows a successful create RPC and only starts asynchronous
cleanup, so a retry could run alongside the sandbox being removed. Classification is not based
on HTTP status alone.

Sandbox traffic Copy Section

sequenceDiagram
    autonumber
    participant U as Client
    participant CP as client-proxy :3002
    participant R as Redis catalog
    participant API as API
    participant OP as orchestrator proxy :5007
    participant E as envd / user process

    U->>CP: https://3000-i7fa3.domain
    CP->>CP: parse host → port 3000, sandbox i7fa3
    CP->>R: GetSandbox(i7fa3)
    alt running
        R-->>CP: node IP
    else paused / unknown
        CP->>API: gRPC ResumeSandbox(i7fa3) — one flight shared by concurrent requests
        API-->>CP: orchestrator IP and network placement (after resume), or 409 / 503 + Retry-After to the client
    end
    CP->>OP: forward to http://nodeIP:5007
    OP->>OP: lookup sandbox, check traffic access token
    OP->>E: http://slotIP:3000 (via veth/tap into VM)
    E-->>U: response

Sandbox routing records Copy Section

client-proxy resolves the node IP of a sandbox from one routing record in Redis, written by the
orchestrator. Its JSON shape (sandbox_catalog.SandboxInfo in packages/shared/pkg/sandbox-catalog):
orchestrator_id, orchestrator_ip, execution_id, sandbox_started_at,
sandbox_max_length_in_hours.

The orchestrator-owned record is the routing source. The orchestrator writes
sandbox:routing:{id} on MarkRunning and deletes it on MarkStopping. A failed write is logged
and counted (orchestrator.routing.publish.total{result=error}); the sandbox keeps running. Build
sandboxes are skipped. The delete is guarded by execution_id in a Lua script, so a stale
lifecycle never removes the record of a newer execution. On a miss client-proxy goes to the
auto-resume path (ResumeSandbox gRPC to the API).

The TTL of the record is sandbox_max_length_in_hours from the write time. The record is
deleted earlier in every normal stop path.

A routed sandbox adds lifecycle_id (its Firecracker process) and network_placement with the
mode, Router, Portable IP and ingress endpoint (ingress_address, ingress_port). The endpoint is
the Router for full-router and the worker for egress-router. The record carries location only. The
orchestrator owns the Router configuration, which never travels through this record. An absent
placement means the legacy per-worker network. The orchestrator publisher does not write a
placement yet.

The API keeps the placement returned by the winning orchestrator create on its own sandbox record
in Redis (networkPlacement), and callers that join the same start get the same placement. A new
execution carries only its own create's placement. On a completed checkpoint response, the API
records the resulting VM's placement together with the return to Running. If the caller abandons
the checkpoint, the node can finish without returning its placement to the API, which can retain
the old placement. Recovery of that outcome is not implemented.

A ResumeSandbox answer for a routed sandbox carries network_placement for the execution it
reports: the winning start's placement for a fresh or joined resume, and the record's placement
for a sandbox already running or back to Running after a pause or checkpoint wait. A record
without a placement is a legacy sandbox and is answered with orchestrator_ip alone, as before.
A placement from another execution, an unknown mode, or a full-router placement without Router,
address and port fails the RPC. A full-router answer has an empty orchestrator_ip and never
looks up an orchestrator. Egress-router and legacy answers still carry orchestrator_ip.
client-proxy reads this placement and refuses full-router routes, as does the orchestrator proxy.

Volume content Copy Section

Persistent volumes (packages/orchestrator/pkg/volumes/) are managed through the control-plane
API (POST/GET /volumes), but their content — reading and writing files — is served by a
separate volume-content API (belt, e2b-dev/belt) that the SDK talks to directly, not through the
control-plane API. The API's role is to mint the credential and tell the SDK where to send content
traffic.

sequenceDiagram
    autonumber
    participant U as SDK
    participant API as API
    participant PG as PostgreSQL
    participant VC as volume-content API (belt)

    U->>API: POST /volumes (create) or GET /volumes/{id}
    API->>PG: persist / load volume row
    API->>API: mint JWT (aud = https://api.#lt;domain#gt;)<br/>resolve domain
    API-->>U: { volumeID, name, token, domain? }
    Note over U: domain is returned only for BYOC teams.<br/>SDK stores it and falls back to api.#lt;E2B_DOMAIN#gt; otherwise
    U->>VC: /volumecontent/{id}/... at api.#lt;domain#gt;<br/>Authorization: Bearer token
    VC->>VC: verify token (audience must match its own origin)
    VC-->>U: file content

• Domain selection. The token's audience and the content host are the same origin,
  https://api.<domain>. For teams on a custom (BYOC) cluster (team.ClusterID set), the API
  returns that cluster's domain (cluster.SandboxDomain, resolved in
  handlers.volumeContentDomain) so content traffic goes to the BYOC cluster's edge instead of the
  control-plane host. For teams on the default cluster the response omits domain and the SDK uses
  its configured default (api.<E2B_DOMAIN>); the audience then uses the deployment's DOMAIN_NAME.
• Token. A short-lived JWT (handlers.generateVolumeContentToken, config in
  cfg.VolumesTokenConfig) signed by the API, scoped to the team and volume, presented as a bearer
  token on every content request. Its aud claim is https://api.<domain>, so a token minted for
  one cluster's origin is not accepted by another.

Pause and resume Copy Section

• Pause: API records a snapshot row in Postgres, then gRPC Pause to the node. The
  orchestrator pauses the VM, snapshots it, diffs memory (dirty-page tracking) and rootfs (COW
  cache) against the template, caches the snapshot locally, and uploads asynchronously to object
  storage (with a retry budget). The orchestrator deletes the sandbox's routing record on
  MarkStopping. Once the API acquires the pause transition, the snapshot DB upsert and the node
  RPC share a detached 80-second budget; the Redis transition key has a 95-second TTL.
  The node inherits that deadline for admission and snapshotting. Before anything destructive
  the node also refuses, retryably, a pause whose capture the filesystem holding its build
  directory cannot take: guest memory plus the rootfs cache's size plus a snapfile allowance,
  doubled when template storage is a directory on that filesystem, counting captures already
  admitted and not yet on disk, with the headroom the pause-admission-disk-headroom-mib
  flag names left free. The check is off by default; the flag's fallback reads the
  PAUSE_ADMISSION_DISK_HEADROOM_MIB environment variable, which is how E2B Embed turns it
  on. Caller cancellation cannot
  abandon the snapshot or its RPC result; terminal build-status writes have a separate detached
  ten-second budget. Snapshot uploads and sandbox teardown retain their separate background
  lifetimes. Shutdown drains pauses still in flight — after the HTTP and gRPC drains, before the
  database and Redis clients close — so a deploy cannot cut a snapshot short. The drain stops
  admitting pauses first, so the eviction sweep cannot start one behind it; a refused pause
  leaves the sandbox running for another replica. The wait is bounded by the pause budget plus
  the terminal build-status write it falls through to, plus a short grace.

  • Deferred rootfs export (gated by the deferred-rootfs-export flag in
    packages/shared/pkg/featureflags): instead of diffing the rootfs on the pause critical
    path, the orchestrator ejects the writable COW cache during pause and returns, then seals it
    into the rootfs diff (reflink) in the background. This moves the rootfs-diff latency off the
    pause, but the local snapshot's rootfs body isn't materialized until the seal finishes, so the
    async upload — and any origin-node resume/prefetch that reads the rootfs diff — waits on the
    seal. A seal failure is permanent (it never re-runs), so the upload fails fast rather than
    retrying.
  • Per-sandbox pause rate limit (gated by the sandbox-pause-rate-limit flag,
    {"limit": N, "period_s": S} with S at most 600): before the pause transition, the pause
    endpoint reads the sandbox record and counts the pause against a budget of N pauses per period
    (GCRA, burst N) in Redis under ratelimit:sandbox-pause:{sandboxID}. An exhausted budget
    answers HTTP 429 with Retry-After and leaves the sandbox running. The count is per
    execution ID, so concurrent requests for one execution use one slot; a record that is
    missing or already pausing is not counted, and Redis errors let the pause through. The
    pause is pinned (ExpectExecutionID) to the record the limiter read, so a resume landing
    between the check and the pause sends the new execution back through the limiter instead
    of pausing it uncounted. The key expires once the budget is full again, at most one period
    after the last counted pause. Timeout auto-pauses are not limited.

• Resume: same path as creation, but placement prefers the origin node — if the snapshot
  is still in its local cache, resume avoids any object-storage reads. The origin node gets
  the same hugepage-limit check as any other node, so a full origin is skipped. Checkpoint is a
  pause+resume in place used to persist state while keeping the sandbox running.

• Post-checkpoint re-init and the pre-pause cut: after an in-place checkpoint the sandbox
  keeps running, so the orchestrator re-sends /init in the background to re-sync the guest
  clock. The re-init is best-effort, bounded by the envd-wait budget and by the VM's liveness,
  and published on the sandbox before it starts, because a held re-init whose /init landed
  after the pre-pause freeze would have envd serve it behind that freeze and thaw the workload
  the snapshot is about to capture. A Pause therefore cuts the published re-init before it
  freezes — a held one is cancelled wherever it is, one that has not yet chosen its loop is
  refused when it does, one already on the legacy loop is left to run as before — and holds
  publication, so no fresh re-init starts between the cut and the freeze. The hold is released
  once the freeze lands, or once the Pause gives up before it; in the latter case the Pause's
  cleanup restarts the re-init it cut, since the VM never paused and the clock re-sync is still
  owed. The prefetch harvest that follows a pause or an in-place checkpoint resumes a
  network-isolated throwaway of the new snapshot to record which memory pages its resume
  touches; the throwaway's held /init asks envd to skip the thaw, so the workload never runs
  and the trace holds only the resume's own faults. A throwaway the gate routes to the legacy
  loop sends no such request, and envd thaws inline as before.

• Resume-fresh checkpoint ownership: when the in-place route is not available, a checkpoint
  snapshots the sandbox and resumes a fresh VM from the new build under the same execution. From
  the reservation on, the old VM is lost on any failure and the resume is the only way the sandbox
  survives, so that work belongs to the node, not to the caller. The node claims the sandbox for
  the checkpoint at the reservation, while its live entry still exists, and holds the claim until
  the resumed sandbox is live or, on failure, until the rollback has run and the reservation is
  released, so no interleaving finds the sandbox neither live nor in flight. Everything past the
  reservation runs detached from the caller's cancellation. The snapshot phase and the resume
  phase (template, resume, envd upgrade, registration) each have their own ten-minute budget,
  observed by the steps that take the context (the Firecracker pause on this route does) and
  between steps: a step that does not take it outlives the budget. Only a Delete for the sandbox
  or the orchestrator's shutdown ends the checkpoint early. A Delete that arrives during the window
  cancels the checkpoint and waits for
  it, so the rollback is that delete and no VM the API stopped tracking goes live later. It
  answers success when its cancellation is what ends the sandbox's lifecycle, and not found when
  the checkpoint ended for its own cause or never held the sandbox, even when the Delete's own
  deadline expires first; a reservation still being taken at that deadline counts as held, and the
  acknowledgement below covers it. A Delete answering for a checkpoint that ended for its own
  cause or never held the sandbox is first handed any sandbox of the checkpoint's that is live
  again, the original the registry put back after a failure in the reservation, and answers
  success for it. A publication the cancellation did not reach can still go live, and the
  checkpoint then kills it with the Delete's reason: the Delete records its acknowledgement with
  its final read of the live map under the operation's mutex, and the publication reads that
  acknowledgement after its insert under the same mutex, so one of the two always sees the other.
  While a checkpoint is in flight, Pause answers the same retryable refusal a busy node gives,
  and a second Checkpoint answers operation-in-progress. The checkpoint marker is the sandbox's
  own and is shared by both routes, so an in-place checkpoint and a resume-fresh one on the same
  sandbox refuse each other. Shutdown cancels a detached checkpoint after the sandbox drain, as
  the gRPC server closes, so it cannot hold a handler open for minutes. The kill accounting
  follows the cause: a checkpoint a Delete ended records the kill and publishes the killed event
  and counter with the Delete's reason, as the in-place path does for its own teardown, whereas
  a resumed VM torn down by a budget or by shutdown is not labelled killed.

  Two limits remain. The in-place checkpoint path is still bound to the caller's context, so
  this ownership covers only the resume-fresh route. And the API still marks the build failed
  when its own request ends before the node answers, even when the node's work later succeeds,
  because the outcome is unknown to it; it restores the sandbox to Running rather than kill it.
  A caller timeout therefore no longer means the snapshot or the VM creation stopped, only that
  the build row records a failure the node may have gone on to complete.

• Fork: checkpoints the original in place, then starts each fork from that snapshot as a
  new sandbox rather than a resume — like a sandbox created from a snapshot template — so
  placement spreads the forks across nodes instead of preferring the original's node.

• Filesystem-only checkpoint: memory: false on snapshot-template creation exports the
  rootfs and no memfile, resumes the same Firecracker process, and produces a template whose
  sandboxes cold-boot; the source keeps running. Routed by its own flag,
  filesystem-only-checkpoint, independent of in-place-checkpoint (no memory is exported, so
  the sync-WP tracking, the Firecracker release gate and the balloon gate play no part). The
  API refuses it before the sandbox leaves Running when the flag is off for the team (400,
  snapshot_filesystem_only_disabled) or when the sandbox's node runs an orchestrator
  release below the one that reads the request field (409,
  snapshot_filesystem_only_unsupported_node, the placement feature floor
  filesystem-only-checkpoint; a checkpoint cannot move to another node, so the options are a
  full snapshot or a pause and resume). The request is never downgraded to a memory snapshot.
  The snapshot row records the kind only once the build is ready, and the pause and fork paths
  keep the same rule, so a refused or failed request leaves the previous build resumed as what
  it is. Each snapshot build also records its own kind (env_builds.filesystem_only) when it is
  created. A sandbox created from a snapshot template reads that, because the source's row
  follows its later snapshots, and placement holds such a cold boot to the
  fs-only-resume-cpu-model CPU model as it does a filesystem-only resume.

• Explicit filesystem-only resume: memory: false on resume/connect demands a cold boot
  (RebootSandbox) even when the snapshot includes memory, as a self-serve rescue when the
  restored memory state is unusable. Gated per team by the fs-only-resume-api flag; when off
  the request is rejected with an explicit error, never silently downgraded to a memory restore.
  The disk has crash-recovery semantics (unflushed pre-pause writes are lost), nothing durable
  is mutated (the memory snapshot survives untouched), and auto-resume never takes this path —
  traffic always memory-resumes. The one way a memory-preferring sandbox ends up with a
  filesystem-only snapshot is at pause time, not resume time: when the node keeps refusing a
  timeout auto-pause (its parent memfile header still deduplicating after the admission grace)
  for longer than the retry budget counted from the first refusal
  (auto-pause-overstay-budget-milliseconds, 120 s by default; 0 degrades at the first
  refusal, negative never degrades), the evictor requests a filesystem-only snapshot instead, so
  the sandbox stops overstaying its expiry; the next resume of that snapshot is a cold boot. The
  degrade is only ever decided on a refusal in the same sweep, so eviction lag alone never
  degrades anything, and refusals only survive with pause-refusal-restore on (its fallback
  reads the PAUSE_REFUSAL_RESTORE environment variable, which is how E2B Embed, with no flag
  service, turns it on). The node
  refuses before MarkStopping, so its routing record stays live and nothing needs to restore
  it; with the flag off the refused sandbox is killed on the node together with its record.

• Pre-boot filesystem recovery: every cold boot of a rootfs that was not frozen at pause
  (fs_quiesced false/absent) runs a jailed e2fsck -p -E journal_only before the VM starts —
  journal replay only, the same recovery the guest kernel would do at mount — so a memory: false
  rescue and a legacy sync-fallback filesystem-only snapshot both mount a consistent disk. It
  replays and exits without a full consistency scan, so the cost is bounded by journal content,
  not filesystem size. Runs under the same confinement as the offline envd swap (unprivileged
  transient unit, device access pinned to the sandbox's own NBD node). A clean replay boots;
  anything else fails the start with the snapshot untouched but stays retryable. Journal replay
  never condemns a snapshot: its exit codes cannot tell an unmountable filesystem apart from a
  transient device fault, so every non-replayed outcome — an operational failure (timeout, I/O,
  device error) or an e2fsck exit that is not a clean replay — is retryable, never a permanent
  customer-facing verdict. In-file corruption that still mounts is likewise not condemned — replay
  does not scan, so it boots. Whole-filesystem repair and condemning a rootfs are left to a
  separate opt-in full-filesystem repair path. Gated by the preboot-fs-recovery flag, separate
  from fs-only-resume-api because it also changes the behavior of existing filesystem-only cold
  boots.

• Envd live-upgrade on resume: the orchestrator can upgrade the sandbox's envd to a newer
  node-local build during resume (gated by the envd-upgrade-target flag in
  packages/shared/pkg/featureflags), via envd's POST /upgrade (see the envd section). It is
  best-effort — a delivery failure before the exec leaves the old envd serving — except an
  unrecoverable post-exec failure (the new envd never re-initializes), which fails the resume
  rather than return a permanently unusable sandbox.

• Envd offline-upgrade on cold-boot resume: reaches envd too old for the live /upgrade
  handover (below MinEnvdVersionForUpgrade). When a filesystem-only snapshot cold-boots
  (RebootSandbox), the orchestrator rewrites /usr/bin/envd in the rootfs before the VM
  boots (PreBootFn → pkg/sandbox/rootfs.SwapEnvdBinary), entirely in userspace via a jailed
  debugfs — never a host-kernel mount of the tenant image. The old envd never participates, so
  the method is version-agnostic. Gated by the envd-offline-upgrade-target flag (a sibling of
  envd-upgrade-target sharing the same version-remap resolver), and applied only when the
  snapshot's rootfs was captured frozen (fs_quiesced, so it is crash-consistent); best-effort
  (a swap failure boots the original envd). Because the swap keys on the snapshot's built-with
  version, which it does not advance, it re-fires idempotently on each cold-boot resume until a
  re-pause re-bakes the running version.

• Both upgrade paths read the host envd binary through a node-local cache (gated by the
  envd-binary-cache flag): the version probe, the live delivery and the offline swap's staging
  copy all read a local copy rather than the read-only artifact mount, which an exec would
  otherwise demand-page in small random reads. The cache is why neither path performs bulk I/O
  against that mount on the resume path.

  It adds a third condition to both paths, so an eligible snapshot is not unconditionally
  upgraded: an upgrade whose binary is not cached on this node is deferred to a later resume
  rather than served from the mount, since reading it there would put tens of seconds on a path a
  customer is waiting on. A node whose cache is cold — any miss, and in particular a ramp keyed on
  sandbox or team, where the boot-time warm does not evaluate — defers that upgrade while a
  background warm populates the cache; on a node-scoped rule the boot warm normally lands before
  the first resume. Both upgrades are idempotent and re-fire per resume, so a deferral costs one
  cycle. Ramp the flag on a node-scoped context kind — the cache is
  shared by every sandbox on the node.

• Auto-pause/auto-resume make sandboxes effectively serverless: idle sandboxes pause, traffic
  resumes them (see traffic flow above).

Template build Copy Section

sequenceDiagram
    autonumber
    participant C as SDK
    participant API as API
    participant TM as template-manager (build node)
    participant FC as Firecracker build VMs
    participant OS as Object storage

    C->>API: POST /v3/templates (register build: cpu, ram, free-disk target) → Postgres env_builds
    C->>API: POST /v2/templates/{id}/builds/{buildID} (recipe: steps, start/ready cmd)
    API->>TM: gRPC TemplateCreate(TemplateConfig)
    TM->>TM: pull image → inject envd/provisioning → extract ext4 rootfs
    TM->>FC: boot VM per phase: provision → user steps
    TM->>TM: resize disk on host
    TM->>FC: boot VM per phase: finalize → optimize
    TM->>OS: upload layers + final {buildID}/memfile, rootfs.ext4, snapfile, metadata
    API->>TM: poll TemplateBuildStatus
    API->>API: mark build ready in Postgres

The project provisioning system enqueues a base build when a cluster-backed project becomes active.
The job registers and starts an ordinary base build, then completes when the API accepts it.
A failed delivery may rebuild on retry; regional build tracking owns completion and timeouts.
Ordinary projects do not enqueue this job.

Builds are layered (pkg/template/build/phases/): base → user → one layer per recipe step →
resize disk → finalize → optimize. Each layer is hashed and cached, so rebuilds only re-run changed
steps. Resize disk grows the quiescent rootfs on the host; the other non-cached phases run in a real
Firecracker VM and their pause-diffs become layers. The optimize phase records which memory pages a
fresh resume touches, producing prefetch hints that speed up future sandbox starts.

Template creation holds node work through foreground setup and asynchronous build completion,
including layer uploads, synchronous cleanup, and final status publication. Cancellation can mark
a build failed before execution finishes; the work hold remains until execution unwinds. Template
build deletion owns a separate hold until object-storage cleanup returns.
TemplateBuildDelete cancels a running build and removes its build-ID prefix from template
storage. It does not delete container registry images; build outputs are VM snapshots in
object storage.

Deployment topology Copy Chapter

The services are scheduler-agnostic binaries and containers; the supported way to run them is the
Kubernetes-based distribution. The roles below hold regardless of how the nodes are provisioned.
E2B Embed (embed/) runs the same containers on one machine, the web dashboard included — with
Compose, with Terraform on GCP, AWS or Azure, or on Kubernetes; its ports, start order and images
are in embed/docs/REFERENCE.md.

flowchart TB
    LB["Load balancer + TLS<br/>api.* → API | *.domain → client-proxy"]

    subgraph control["control plane"]
        AJ["api, dashboard-api, client-proxy,<br/>redis, otel-collector"]
    end
    subgraph sandboxnodes["sandbox nodes (autoscaled)"]
        OJ["orchestrator<br/>+ Firecracker sandboxes"]
    end
    subgraph buildnodes["build nodes (autoscaled)"]
        TJ["template-manager"]
    end
    subgraph ch["analytics"]
        CJ["clickhouse"]
    end

    LB --> control
    AJ -->|gRPC| OJ & TJ

• Control-plane services (api, dashboard-api, client-proxy) are stateless containers and the
  only LB backends. The api discovers orchestrator nodes through
  packages/shared/pkg/servicediscovery (Kubernetes, DNS, a static list, or the legacy Nomad
  backend the code still carries).
• Sandbox nodes run the orchestrator directly on the host (it needs root for Firecracker,
  namespaces, NBD, cgroups), configured with hugepages and local template caches. Autoscaled.
• Build nodes run the same binary in template-manager mode.
• PostgreSQL is external (connection string via secrets); Redis is a managed service or a
  single in-cluster instance; ClickHouse runs on its own nodes.
• Observability: everything exports OTel; the collector fans out to ClickHouse (product metrics)
  and Grafana Cloud/stack. The API and orchestrator export synchronous counters and histograms
  as deltas, so per-team series do not accumulate in process memory; the collector converts
  them back to cumulative (deltatocumulative) before any Prometheus backend, which only works
  when each of those processes sends to a collector on its own node. Logs default to the legacy Vector → Loki path; dynamic log routing can
  select a primary collector and shadow collectors, and local-cluster log reads can be switched to
  ClickHouse with logs-read-config after sandbox_logs is populated (LOGS_READ_CONFIG is the
  flag's fallback where LaunchDarkly has no value). Once reads are on ClickHouse, Loki can be left
  out of a deployment and the api started without LOKI_URL.

Repository layout Copy Chapter

packages/
  api/                  Control-plane REST API
  orchestrator/         Sandbox runtime + template builder (one binary, per-node)
  client-proxy/         Edge router for sandbox traffic
  envd/                 In-VM agent (pkg/version.go is release-please's; see Envd)
  dashboard-api/        Web-dashboard backend
  shared/               Protos, telemetry, storage clients, proxy engine, feature flags
  auth/                 AuthN library (API keys, JWT/OIDC) used by api + dashboard-api
  db/                   Postgres migrations (goose) + queries (sqlc)
  clickhouse/           ClickHouse schema, batching writers, query clients
  otel-collector/       Collector config
  nomad-nodepool-apm/   Nomad autoscaler metric and deployment-aware target plugins
  local-dev/            docker-compose local stack + DB seeding
spec/                   OpenAPI specs (public, edge, dashboard) — codegen sources
tests/integration/      Integration tests against a live deployment
embed/                  E2B Embed: the whole stack on one machine (Compose, Terraform, Kubernetes)

Cross-service contracts are all generated: OpenAPI specs in spec/, gRPC protos in
packages/orchestrator/*.proto and packages/envd/spec/, SQL in packages/db/queries/.
Run make generate after changing any of them.

RELEASING (`docs/RELEASING.md`) Copy Section

Releasing packages to e2b-artifacts

This repository is a read-only mirror: its source of truth is E2B's internal
monorepo, which exports the code here via copybara. Releases are cut there
too — the package list, SemVer release PRs, auto-deploy tags, and the
publish workflow all live in the monorepo. Each directory in the table
below is what gets tagged and published. No release tags or GitHub Releases
appear in this repository.

How a release happens Copy Chapter

Two identities:

1. Every commit on the monorepo's default branch that touches a package
   directory is tagged
   <component>-vMAJOR.MINOR.YYYYMMDDHHMM-<sha> for that package only, on
   that package's current version line. That publish does not increment
   the patch. The timestamp sits in the patch slot, so these tags sort
   above every MAJOR.MINOR.x release — pin exact tags, not "highest".
2. Conventional commits (feat:, fix:) accumulate into a release PR.
   api, client-proxy, clickhouse-migrator, dashboard-api,
   nomad-nodepool-apm and orchestrator share one coordinated SemVer with
   the rest of the platform: merging that PR tags each of them
   <component>-vX.Y.Z at the same number, with changelog. embed has
   its own release PR, opened once the platform release is tagged, at
   that same number. envd is versioned on its own and has its own
   release PR. (The same commits arrive here through the copybara
   export.)
3. Either tag is a git tag only — no GitHub Release. The tag push
   publishes to e2b-artifacts — images as :v<version>, binaries as
   versioned objects in the public e2b-artifact-binaries bucket at
   <component>/v<version>/, each with a <name>.sha256 beside it
   (sha256sum format). Client-bucket copies use the same string:
   <name>.v<version>, never a bare commit SHA. The v is on every tag,
   image tag, bucket path and object suffix; pin with it.

A manual tag (below) still publishes that commit.

Release candidates / manual publishes Copy Chapter

Push a <component>-v<version> tag by hand in the monorepo (e.g.
client-proxy-v2.0.0-rc1) and the publish workflow publishes that commit as
:v2.0.0-rc1. The registry has immutable tags and the binaries bucket is
create-only, so a manual tag can never overwrite an existing version — a bad
published version is fixed by cutting the next version, never by
re-publishing.

Recovery Copy Chapter

All recovery happens in the monorepo:

• Publish failed: re-run that tag's publish run (any time), or dispatch
  the publish workflow manually with the tag name.
• Release merged but never tagged (e.g. the merge wasn't the head commit
  of its push): push the <component>-v<version> tag on the merge commit by
  hand, and swap the release PR's label from autorelease: pending to
  autorelease: tagged — otherwise release-please refuses to open new
  release PRs for any package.

Embed Docs REFERENCE (`embed/docs/REFERENCE.md`) Copy Section

E2B Embed reference

The hub README says what E2B Embed is and how to run it.
This page holds the rest: what runs where and on which ports, logs and
telemetry, the secrets, how images and pins are released, building templates,
and the developer tooling. The 5 guides cover what differs on Compose,
Terraform and Kubernetes.

What runs where Copy Chapter

The 4 stores run in containers on a bridge network with their ports on
127.0.0.1. api, client-proxy, dashboard-api, the dashboard and the released
orchestrator run on the machine's own network, and so does the opt-in
collector (Observability). The orchestrator is a host
process, launched through nsenter by a privileged container, the same
pattern E2B's own Kubernetes deployment uses. Its launcher ends every sandbox
when it stops.

8 one-shots run before the stack is usable, in this order:

• preflight fails fast with a FIX: line when the machine is unsuitable.
• api-secrets generates the api's admin token and sandbox-token hash seed.
  It needs only the volume, so it finishes first.
• host-setup prepares the machine on every start.
• fetch-artifacts downloads and verifies the 5 Firecracker binaries.
• db-migrator and clickhouse-migrator bring the 2 databases to the schema
  their pinned images expect.
• seed writes the default team and generates this install's API key.
• base-template builds the default template through the API.

Kubernetes runs 7 of them: there the api's 2 secrets come from a Secret
instead. ready is the marker that everything above worked. It is a
long-running container, not a one-shot, and its readiness is the whole
stack's.

dashboard-api starts once db-migrator and api-secrets are done and reaches
the same stores as the api; the dashboard starts once api and dashboard-api
are healthy.

Ports on the host network Copy Section

13 ports listen on every interface of the machine. The SDK needs 3000 and
3002; a browser needs 3001 for the dashboard. The other 10 must not be
reachable on any address the machine holds: give it no public address of its
own, or firewall those 10 ports for that address as well, not only at the
network edge.

Port 5008 is the one to guard most. It creates and kills sandboxes and starts
template builds; nothing authenticates it, the api dials it directly as
LOCAL_ORCHESTRATOR_ADDRESS, and anyone who reaches it has the whole
orchestrator. The 4 egress-proxy ports (5010 and 5016 to 5018) take the
sandbox traffic the orchestrator redirects inside each sandbox's network
namespace. They expect no client from outside the machine and have no
authentication of their own.

Ports on loopback Copy Section

Everything outside the table above stays on loopback. Postgres is on
5432, Redis on 6379, ClickHouse on 8123 and 9000. On Kubernetes ClickHouse
also listens on 9004, 9005 and 9009, its MySQL and PostgreSQL wire protocols
and its interserver port, since the pod shares the node's network. Vector's
log listener is on 30006 (20006 on Kubernetes, for the reason that guide
gives) and its API on 44313, on Kubernetes only. The 2 pprof endpoints are
6060 for api and 6061 for the orchestrator. The built-in collector, when it
runs, listens on 4317 and 13133 (Observability).
sudo ss -ltnp on the machine confirms the split: the 13 ports in the table
above show a *: address, everything here shows 127.0.0.1:.

Observability Copy Chapter

What the stack keeps about itself, out of the box and with 1 setting.

Logs Copy Section

Sandbox and template-build logs live in ClickHouse. The orchestrator and the
api ship their lines to Vector: envd's output from inside the VM, the
orchestrator's per-sandbox events, the template-manager's build output and
the api's own. Vector turns each into a row of the sandbox_logs table. The
api reads that table for the SDK's getLogs and e2b template build, and
the dashboard shows the same logs through it. LOGS_READ_CONFIG=true sets
the api's logs-read-config flag, since there is no LaunchDarkly here to set
it. Retention is the table's 7 days. There is no Loki in this stack, and the
api needs no LOKI_URL.

PAUSE_REFUSAL_RESTORE=true sets the api's pause-refusal-restore flag the
same way, in an api release that reads it: a pause the node refuses — it is
still persisting the sandbox's parent — keeps the sandbox running and
answers 503 to retry, instead of killing it.

Every service also writes its own log to stdout, which is what
docker compose logs <service> and kubectl -n e2b logs e2b-0 -c <container>
print. None of this needs a collector.

The collector setting Copy Section

api, the orchestrator, client-proxy and dashboard-api carry the platform's
OpenTelemetry instrumentation, and 1 setting points it at a collector:
E2B_OTEL_COLLECTOR_GRPC_ENDPOINT in compose/.env, a
host:port the 4 services reach over OTLP/gRPC without TLS. It ships
commented, and empty or absent it exports nothing, which is how the stack runs
by default. Set, each of the 4 sends its metrics every 15 seconds, its traces
and its logs there. api and the orchestrator also export the e2b.* product
metrics on their own: a team's running and created sandboxes, and each
sandbox's CPU, memory and disk. With no endpoint those exporters time out and
log failed to upload metrics: exporter export timeout; that line stops once
the endpoint names a collector that answers.

The built-in collector Copy Section

COMPOSE_PROFILES=otel, the line under the endpoint in .env and also
shipped commented, starts otel-collector: the OpenTelemetry Collector contrib
image, on the machine's network, pinned inline in
compose/compose.yaml like the stores. It receives
on 127.0.0.1:4317, so the endpoint that goes with it is that address, and
uncommenting both lines is the whole switch. Its config is
compose/config/otel/otel-collector.yaml,
inlined into the compose file as otel-collector-config.

It keeps the e2b.* metrics and writes them into the stack's own ClickHouse,
into metrics_gauge and metrics_sum. The clickhouse-migrator creates
both, as routing tables that store nothing themselves: their materialized
views fill sandbox_metrics_gauge (kept 7 days) and team_metrics_gauge and
team_metrics_sum (kept 90), so those 3 are the ones to query. It drops
every other metric, accepts traces and logs and discards them, and its health
check answers curl -s 127.0.0.1:13133 on the machine. The image has no
shell, so Compose runs it without a healthcheck, and no service depends on
it, ready included, so the rest of the stack starts whether it runs or not.

It is not an observability stack. Nothing here stores or shows traces, logs
or the services' own metrics. To keep traces and logs, forward them: the
config sends them to a nop exporter and carries a commented otlp/upstream
one. In your copy of compose.yaml, uncomment otlp/upstream, set its
endpoint, name it in place of nop in the traces and logs pipelines, and run
docker compose up -d --force-recreate otel-collector, since Compose does not
recreate a container for a changed inline config on its own. On Kubernetes,
make the same edit in kubernetes/config/otel-collector.yaml, or in an
overlay's copy for the URL install (the
Kubernetes guide has the overlay), and
run kubectl apply -k again. The services' own metrics stop at the filter.
To keep those too, send everything to a collector of your own instead: leave
the profile off and put that collector's address in the endpoint. It then
gets the e2b.* metrics as well, and this ClickHouse none.

What reads the metrics Copy Section

The api serves the 3 tables above: GET /teams/{teamID}/metrics and
/teams/{teamID}/metrics/max for a team's concurrent sandboxes and start
rate, and GET /sandboxes/metrics and /sandboxes/{sandboxID}/metrics for a
sandbox's CPU, memory and disk, which the SDK's getMetrics (get_metrics in
Python) reads. The dashboard's monitoring charts, a team's concurrent
sandboxes and start rate and a sandbox's resource usage, are drawn from the
same endpoints, so they stay empty until the built-in collector runs, or a
collector of your own writes those metrics into this ClickHouse. Sandbox and
build logs do not depend on it: they reach ClickHouse through Vector.

Ports, Kubernetes and Terraform Copy Section

4317 and 13133 bind 127.0.0.1 only, so they are not among the 13 ports in
the host-network table, and no firewall needs
to change for them.

Kubernetes carries the same setting as the OTEL_COLLECTOR_GRPC_ENDPOINT
literal of the e2b-settings ConfigMap, and the built-in collector as one
commented patch line in the kustomization; the
Kubernetes guide has both, and where the
collector sits in the pod. Terraform has the otel_collector_grpc_endpoint and
otel_collector variables, which the startup script writes into the
instance's .env; the GCP,
AWS and
Azure guides list them.

Secrets Copy Chapter

The team API key is per install. The seed generates it on the first start and
keeps it beside the databases, so an install never has a key without its
database or a database without its key. Every install prints it with the SDK
URLs and the dashboard URL. Each guide's Secrets section says where its copy
lives and how to pin or rotate it. A rotation revokes the old key, which keeps
working for up to 5 minutes: the api caches team lookups in Redis for that
long.

The dashboard takes that same key in its key form and keeps it in an httpOnly
browser cookie for a year; sign-out clears it, and rotating the key signs
every browser out. Embed serves the dashboard over plain http, so the cookie
is not marked Secure (DASHBOARD_COOKIE_SECURE=false); a browser on a
plain-http address would otherwise drop it and the key form would loop.

ADMIN_TOKEN and SANDBOX_ACCESS_TOKEN_HASH_SEED are the api's own 2, and
every install generates them too. Compose writes them on the first
start into the volume that holds the team key (/run/e2b/api.env in
seed-state). Terraform writes generated ones into the instance's .env at
first boot. Kubernetes reads the Secret the install creates. On Compose,
setting either one in .env pins it and leaves the other generated; each
guide's Secrets section says how to rotate what it holds. dashboard-api
reads the admin token the same way everywhere, so rotating it means
recreating dashboard-api as well as the api.

Both are worth guarding. The admin token is admin over the seeded team on
port 3000 without the team API key, including minting and revoking API keys;
the team's id is a public constant. The hash seed makes the traffic and envd
tokens of secure sandboxes computable from a sandbox ID by anyone who
reaches 3002.

Images and pins Copy Chapter

compose/.env is the source of truth for every E2B
version the stack uses: the 5 released E2B service images (api, db-migrator,
dashboard-api, client-proxy, clickhouse-migrator), the dashboard image, the 3
stack images Embed builds itself, and the 5 Firecracker binaries. The 4 store
images and the built-in collector's are pinned inline in
compose/compose.yaml and repeated in the
StatefulSet and its collector patch. The stack images carry everything that
is not a released E2B service: the host scripts, the SDK scripts and the
database seeder. Terraform ships the .env to the instance, and Kubernetes
repeats its pins in
kubernetes/kustomization.yaml.
The 3 stack images live in the embed repository of the e2b-artifacts
registry.

Embed is released at the platform version once that release is tagged.
That release moves every platform pin in both files to it: api, db-migrator,
dashboard-api, client-proxy, clickhouse-migrator, the orchestrator and the 3
stack images, the lines carrying a release marker. A checkout at a release
therefore names one version everywhere and pulls exactly what that release
published; see RELEASING.md. envd has its own
release line, and the kernel, Firecracker and BusyBox are not released here,
so those 4 are pinned by hand. A bump adds the binary's checksum to
compose/scripts/fetch-artifacts.sh
first. The orchestrator needs no row: each release writes a .sha256 beside
the binary it publishes, and fetch-artifacts verifies against it. The
dashboard image is released from its own repository
(github.com/e2b-dev/dashboard, tags vX.Y.Z) and is pinned by hand too; its
line carries no release marker. All of it is public and pulled anonymously;
the stores and the collector come from Docker Hub.

The 9 pinned images are published for both architectures; fetch-artifacts
verifies the arm64 orchestrator and envd against the .sha256 their release
writes beside the object; a pin with no such
object stops with a FIX: line naming it.

To pin an install, pin the commit. The Compose files come from raw URLs, so
put the commit in place of main in their path. The Terraform source and
the kubectl apply -k URL are git URLs and take ?ref=<commit>. The main
URLs the guides use give the newest.

Top to bottom: the source files, the Dockerfiles that copy them, the images
(3 built here, 10 pulled ready-made) and the compose services. Arrows in the
last band are depends_on gates, in start order. The stripe on each service
says which image it runs. 3 things the picture leaves out: the ClickHouse,
Vector and collector configs are inlined into
compose/compose.yaml rather than shipped in an
image; the 5 Firecracker artifacts never enter an image at all, since
fetch-artifacts writes them onto the machine and the orchestrator reads them
there; and the opt-in otel-collector service, an 11th image pulled
ready-made, is not drawn.

Beyond the first sandbox Copy Chapter

Template.build builds your own template through the same API, whichever way
you run the stack:

from e2b import Template, Sandbox
tpl = Template().from_python_image("3.12").run_cmd("pip install requests")
Template.build(tpl, alias="py-requests", cpu_count=2, memory_mb=1024, on_build_logs=lambda e: print(str(e)))
sbx = Sandbox.create("py-requests")
print(sbx.commands.run("python3 -c 'import requests; print(requests.__version__)'").stdout)
sbx.kill()

The build runs in a sandbox of its own on the machine and pulls the image
from Docker Hub; no Docker daemon is involved. How long it takes depends on
how much of the base image is already cached.

Reaching a port inside a sandbox Copy Section

sandbox.get_host(port) returns {port}-{id}.e2b.app, which does not resolve
here. Reach a port inside a sandbox through client-proxy's header routing
instead:

curl -H "E2b-Sandbox-Id: $SANDBOX_ID" -H "E2b-Sandbox-Port: 8080" http://localhost:3002/

The dashboard at http://localhost:3001, the E2B_DASHBOARD_URL that
ready prints, shows the same sandboxes and templates: paste $E2B_API_KEY
into its key form. Its terminal and filesystem inspector reach a sandbox
through the same header routing, at the address in E2B_DASHBOARD_HOST
(default localhost). Terraform writes it into the instance's .env, and
the Kubernetes manifest sets the same address from the node's IP; on Compose
set it in .env when a browser on another machine opens the dashboard
without a tunnel.

Developing Copy Chapter

make is a developer convenience; the operator path is only docker compose,
terraform or kubectl.

What the targets need Copy Section

make lint wants shellcheck, terraform (1.7.5 or newer) and kubectl,
for kubectl kustomize. terraform init downloads the google, aws, azurerm,
random and time providers, once each into a shared cache, so it needs network;
the aws provider alone is about 180 MB to download and about 800 MB unpacked
in the cache. make test wants bats plus kubectl, which
tests/kubernetes.bats renders the manifest with.
tests/terraform.bats reads every module's files as text and needs no
terraform.

Both also need a working Docker daemon with the compose plugin, and jq on
PATH. make lint renders compose/compose.yaml, validates the Vector
config with the Vector image (make vector-validate) and the collector's
with the collector image compose.yaml pins (make otel-validate).
tests/inline-configs.bats diffs the rendered inline configs against the
copies under compose/config/. tests/vector-rows.bats replays the fixture
log lines in tests/fixtures/vector/ through the shipped Vector config with
that image, with compose/scripts/dev/vector-testconfig.py swapping only the
source for stdin and the sink for a JSON console, and asserts the
sandbox_logs row each becomes or that it is dropped. It needs Docker,
python3 and jq as well. tests/pins.bats keeps the 3 pins, the 2 dashboard
pins, their Kubernetes counterparts and the registry the images are built
into in step with each other.

Inline configs Copy Section

After editing compose/config/vector/vector.toml,
compose/config/clickhouse/config.xml or
compose/config/otel/otel-collector.yaml, run make sync-configs (python3).
It rewrites the inline copies in compose/compose.yaml, the
VECTOR_CONFIG_SHA256, CLICKHOUSE_CONFIG_SHA256 and OTEL_CONFIG_SHA256
stamps, and the Kubernetes copies under kubernetes/config/. The stamps make
Compose recreate the container on a config change, which it does not do for
inline configs: content on its own. tests/config-hashes.bats fails until
the stamps match, and tests/kubernetes.bats until the copies do.

The stores check Copy Section

make stores-check runs what CI's stores job runs: both stores, their
migrators, the seed, and the assertions that the seed wrote the team API key
file, that the seeded row hashes that key, and that both migration ledgers
landed. It wants Docker, network access and the images from make images,
and needs no KVM. It deliberately leaves the containers up so a failure can
be inspected, so finish with
docker compose --project-directory compose down -v.

Rebuilding the stack images Copy Section

The scripts under compose/scripts/ and compose/scripts/node/ travel inside
the 2 stack images that carry them, and the compose file mounts nothing from
the repository. After editing either directory, run make images before the
next start, or the stack silently keeps running the old code. Without make,
this is the images target (its $$ becomes a single $ outside of make):

set -a; . ./compose/.env; set +a
docker buildx bake --load \
  --set "*.platform=linux/$(docker version -f '{{.Server.Arch}}')" \
  --set "seed.context=https://github.com/e2b-dev/runtime.git#${RUNTIME_COMMIT}" \
  --set seed.args.SRC=packages \
  --set "tools.tags=${E2B_TOOLS_IMAGE}" \
  --set "node-e2b.tags=${E2B_NODE_E2B_IMAGE}" \
  --set "seed.tags=${E2B_SEED_IMAGE}"

Binary checksums live in the tools image's compose/scripts/fetch-artifacts.sh,
so bumping a Firecracker artifact means new stack images, not an edit on a
machine.

--- METRICS ---

• Files Extracted: 24
• Estimated Token Budget: ~67505 tokens
• Canonical Reference: https://codewiki.google/github.com/e2b-dev/infra