# Technical Documentation: hexclave/stack-auth

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/hexclave/stack-auth)

### Overview (`README.md`)

# Hexclave
<div align="center">

<img src=".github/assets/hexclave-header.svg" alt="Hexclave" width="320"/>

<br/>

**The user infrastructure platform.**

Hexclave handles everything around your users: authentication, teams,
payments, emails, analytics, and much more. Get started with our AI
prompt below.

[Website](https://hexclave.com) · [Docs](https://docs.hexclave.com) · [Dashboard](https://app.hexclave.com) · [Discord](https://discord.hexclave.com)





</div>

---

<div align="center">
  <img src=".github/assets/comparison.webp" alt="Where Hexclave fits in the infrastructure stack" width="900"/>
</div>

## Get started

Setting up Hexclave is one prompt. Paste this into your coding agent of choice:

```text
Read skill.hexclave.com and help me setup hexclave in this project
```

## For AI agents

If you're an AI coding agent, use your web fetch tool or curl the skill documentation for up-to-date integration instructions:

```
curl -sSL "https://skill.hexclave.com?question=<your-question>&context=<your-context>"
```

## What's included

Hexclave ships as a catalog of apps you switch on as your product needs them.
Each one is built on the same user model, and new apps land regularly.

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/authentication.webp" alt="" width="40" align="top"/> &nbsp; Authentication

Authentication that just works with passkeys, OAuth, and CLI auth. Drop in one component and ship the whole flow; auth methods toggle from the dashboard with no code changes needed.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/authentication.webp" width="520" alt="Authentication"/>
</td>
</tr></table>

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/teams.webp" alt="" width="40" align="top"/> &nbsp; Teams

Build for teams, not just users, with workspaces, email invites, and roles that actually gate the work. The workspace switcher remembers selection, invites auto sign up new users, and permissions hold up under audit.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/teams.webp" width="520" alt="Teams"/>
</td>
</tr></table>

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/rbac.webp" alt="" width="40" align="top"/> &nbsp; RBAC

Permissions, sorted: roles that nest and one permission check that works the same on server or client. Define them in the dashboard, check them anywhere in your code.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/rbac.webp" width="520" alt="RBAC"/>
</td>
</tr></table>

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/api-keys.webp" alt="" width="40" align="top"/> &nbsp; API Keys

API keys without the footguns: leaked keys get auto-revoked, work for users and teams, and show the full secret only once. We never keep the plaintext after creation.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/api-keys.webp" width="520" alt="API Keys"/>
</td>
</tr></table>

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/payments.webp" alt="" width="40" align="top"/> &nbsp; Payments

Payments without the plumbing for subscriptions, one-time charges, and usage metering with credits. Bill a person or a whole team with one model, no separate codepath.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/payments.webp" width="520" alt="Payments"/>
</td>
</tr></table>

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/emails.webp" alt="" width="40" align="top"/> &nbsp; Emails

Email that delivers and tells you so, handling transactional and marketing sends from one API. Edit templates with an AI editor, theme once, and track every open and click.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/emails.webp" width="520" alt="Emails"/>
</td>
</tr></table>

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/analytics.webp" alt="" width="40" align="top"/> &nbsp; Analytics

Know your users with no data stack required, with live active user counts and session replays out of the box. Ask in plain English to build dashboards or write SQL to save queries, all with one flag enabled.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/analytics.webp" width="520" alt="Analytics"/>
</td>
</tr></table>

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/webhooks.webp" alt="" width="40" align="top"/> &nbsp; Webhooks

React to every user event in real time with signed, tamper-proof webhooks. Retries and backoff are handled for you; verify in five lines and manage endpoints from the dashboard.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/webhooks.webp" width="520" alt="Webhooks"/>
</td>
</tr></table>

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/data-vault.webp" alt="" width="40" align="top"/> &nbsp; Data Vault

A safe for the secrets your users hand you, locked with your secret so we never see the plaintext. Store and retrieve tokens in two lines each, server-only by design.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/data-vault.webp" width="520" alt="Data Vault"/>
</td>
</tr></table>

<table><tr>
<td width="50%" valign="middle">

### <img src=".github/assets/logos/launch-checklist.webp" alt="" width="40" align="top"/> &nbsp; Launch Checklist

Run through the must-do checks before flipping to production: domain setup, callbacks locked, secrets rotated. The progress tracker keeps your team aligned so nothing critical slips through on launch day.

</td>
<td width="50%" valign="middle" align="center">
<img src=".github/assets/app-shots/launch-checklist.webp" width="520" alt="Launch Checklist"/>
</td>
</tr></table>

## Contributing

Hexclave is open source, and contributions are welcome. Read
[`CONTRIBUTING.md`](./CONTRIBUTING.md) to get started, and say hello in
[Discord](https://discord.hexclave.com) before picking up anything large.
Found a security issue? Email security@hexclave.com.

## ❤ Contributors

<a href="https://github.com/hexclave/hexclave/graphs/contributors">
  <img src="https://contrib.rocks/image?repo=hexclave/hexclave&columns=9" alt="Contributors" width="100%" />
</a>

---

### Overview (`.devcontainer/README.md`)

# Hexclave Development Container

This development container provides a standardized development environment for working on Hexclave.

## Getting Started

1. Open this folder in VS Code with the Dev Containers extension installed
2. VS Code will prompt you to "Reopen in Container"
3. Once the container is built and started, the following commands will be run automatically:
   - `pnpm install`
   - `pnpm build:packages`
   - `pnpm codegen`

4. Start the dependencies and development server with:
   ```
   pnpm restart-deps
   pnpm dev
   ```

5. You can now access the dev launchpad at http://localhost:8100

For more information, read the README.md in the root of the repository.

---

### Overview (`docs-mintlify/README.md`)

# docs-mintlify

How to run the Mintlify docs preview locally from this repository.

## Prerequisites

- Node.js `>=20.17.0`
- `pnpm`
- Repository dependencies installed (`pnpm install` from repo root)
- OpenAPI specs in `openapi/` are **committed to git**. Hosted Mintlify cannot run monorepo codegen on deploy, so these files must be present in the repo for production docs.

  When you change API route OpenAPI metadata, regenerate and **commit** the four specs from the repo root:

  ```bash
  pnpm run --filter @hexclave/backend codegen-docs
  git add docs-mintlify/openapi/
  ```

  That writes `client.json`, `server.json`, `admin.json`, and `webhooks.json` into `docs-mintlify/openapi/`. CI fails if `pnpm codegen` produces different output than what is committed (see root lint-and-build workflow).

## Run locally

From the repository root:

```bash
pnpm -C docs-mintlify run dev
```

This starts Mintlify in `docs-mintlify` on `http://localhost:${NEXT_PUBLIC_HEXCLAVE_PORT_PREFIX:-81}04` (for example, `http://localhost:8104` with the default prefix).

From inside `docs-mintlify`, you can also run:

```bash
pnpm dev
```

Useful variants:

```bash
# Override the default port
pnpm -C docs-mintlify run dev -- --port 3333

# Skip OpenAPI processing for faster iteration
pnpm -C docs-mintlify run dev -- --disable-openapi
```

## Search + assistant in local preview

If you want local search and the Mintlify assistant:

```bash
pnpm -C docs-mintlify run login
pnpm -C docs-mintlify run status
```

Then re-run `pnpm -C docs-mintlify run dev`.

## Package scripts

From repo root:

```bash
pnpm -C docs-mintlify run lint
pnpm -C docs-mintlify run typecheck
pnpm -C docs-mintlify run build
pnpm -C docs-mintlify run clean
```

`lint` runs both `mint validate` and `mint broken-links`.

---

### Overview (`apps/deployment-gateway/README.md`)

# Deployment gateway

One Fly app proxies every `<suffix>-<mac>.deploy.built-with-hexclave.com` request to
`https://hxc-<suffix>.fly.dev`. Marshal uses the existing Fly app identity for `<suffix>`
and signs it: `<mac>` is the first 16 hex characters of an HMAC-SHA256 over the domain and
suffix under `HEXCLAVE_DEPLOYMENT_HOSTNAME_KEY`, a key only Marshal and this gateway hold
(`gateway.js` here, `platformHostnameMac` in `apps/marshal/src/platform-domain-names.ts`).
Fly app names are global across every Fly organization, so anyone can register an
`hxc-*` app of the right shape; without the key they cannot produce a hostname the gateway
routes to it, which keeps our wildcard certificate and domain off content we did not
deploy. There is no per-deployment routing table. Every other hostname, including vanity
names like `login.<domain>`, returns 421. Nginx (with its njs module) handles HTTP,
streaming uploads/downloads, and WebSockets.

The signature proves Marshal minted a name, not that Marshal still owns the app behind it.
Signatures do not expire or revoke: once a service is deleted and Marshal deletes its Fly
app, the app name is free again, and if Fly lets a stranger re-register it the old URL
(and any bookmark or OAuth redirect pointing at it) serves their content. Accepted for now;
closing it means either never deleting apps or an ownership lookup on the request path.

Source and deployment configuration live here. Hosted components remain on Vercel
under `<project-id>.built-with-hexclave.com`; they are not in this traffic path.
DNS can stay on Vercel. The gateway has one wildcard certificate, rather than requesting
a certificate for each deployment. Fly app certificates for customer-supplied domains
continue to be managed separately by Marshal.

This directory also holds [parked-page](parked-page/README.md), the page a STOPPED
deployment serves. It is a separate image with its own lifecycle: the gateway is one Fly app
this repository deploys, while the parked page is published to Docker Hub and pulled by
tenant apps whose services Marshal has parked. They live together because both are platform
infrastructure sitting in front of customer deployments, and neither is a tenant's own code.

## Initial setup

Run once for the shared gateway, before deploying the Marshal hostname change.
Choose an available app name and your Fly organization. These commands provision real
infrastructure; they are not part of the local test.

```sh
cd apps/deployment-gateway
export HEXCLAVE_GATEWAY_APP=hexclave-deployment-gateway
fly apps create "$HEXCLAVE_GATEWAY_APP" --org YOUR_FLY_ORG
fly ips allocate-v6 --app "$HEXCLAVE_GATEWAY_APP"
fly ips allocate-v4 --shared --app "$HEXCLAVE_GATEWAY_APP"
# The hostname signing key. Generate once, set it here, and set the SAME value as
# HEXCLAVE_DEPLOYMENT_HOSTNAME_KEY on the Marshal that serves this gateway's domain.
fly secrets set --app "$HEXCLAVE_GATEWAY_APP" --stage "HEXCLAVE_DEPLOYMENT_HOSTNAME_KEY=$(openssl rand -hex 32)"
fly deploy --app "$HEXCLAVE_GATEWAY_APP" --ha=true
fly scale count 2 --app "$HEXCLAVE_GATEWAY_APP"
fly certs add '*.deploy.built-with-hexclave.com' --app "$HEXCLAVE_GATEWAY_APP"
fly certs setup '*.deploy.built-with-hexclave.com' --app "$HEXCLAVE_GATEWAY_APP"
```

In Vercel DNS for `built-with-hexclave.com`:

1. Add `*.deploy` A and AAAA records pointing to the gateway's allocated IPv4/IPv6.
   Use the addresses reported by `fly ips list --app "$HEXCLAVE_GATEWAY_APP"`.
2. Add the DNS validation records **exactly as reported by `fly certs setup`**.
   Wildcard issuance uses DNS-01. Keep the validation records for automated renewal.
3. Keep the existing `*` record for hosted components pointing at Vercel. Do not
   move the root domain, root wildcard, or hosted-components certificate validation.
4. Verify with `fly certs check '*.deploy.built-with-hexclave.com' --app "$HEXCLAVE_GATEWAY_APP"`.
   Wait for a valid certificate before releasing the new Marshal URLs.

The gateway's default `.fly.dev` hostname intentionally returns 421: only deployment
hostnames route traffic. Health checks use `Host: gateway-health.internal` and `/healthz`,
so no customer URL path is reserved by the gateway.

The gateway refuses to start without a 64-hex-character `HEXCLAVE_DEPLOYMENT_HOSTNAME_KEY`,
and refuses the public development key from `apps/marshal/.env.development` unless
`HEXCLAVE_GATEWAY_ALLOW_DEVELOPMENT_KEY=1` (the Docker test sets it; production never does).
A gateway and Marshal holding different keys route nothing (every hostname is 421), which
is the failure to look for first when a fresh deployment's platform URL does not answer.
Rotating the key is a hard cutover, in either order: the gateway holds one key, so every
platform URL minted under the old key stops answering as soon as the gateway restarts with
the new one, and stays dead until its service is redeployed and the backend learns the new
URL. The first rollout of signed hostnames is the same event for every public service
deployed under the unsigned shape. Rotate on compromise, not on a schedule; if zero-downtime
rotation is ever needed, the gateway will need to accept an old and a new key at once.

For a separate preproduction gateway, create another Fly app from the same source and
use a separate wildcard domain. Set `HEXCLAVE_DEPLOYMENT_PLATFORM_DOMAIN` to the bare
suffix (for example, `deploy.example.net`) on the gateway with `fly deploy --env
HEXCLAVE_DEPLOYMENT_PLATFORM_DOMAIN=deploy.example.net`. Configure wildcard DNS and its
certificate using that suffix. The value must be a lowercase DNS domain, without `*.`.

Pass the same environment variable to the local Marshal live-test command and optionally
the Docker test command. Both use the production domain when the variable is unset.
The override does not change Fly app names, but it does change every signature (the
domain is bound into the mac), so a preproduction gateway needs its own Marshal
configuration even if it shares a key. The backend's production namespace reservation
is unchanged; this override supports testing the gateway and Marshal directly.

## Updates and operations

Deploy from this directory with `fly deploy --app "$HEXCLAVE_GATEWAY_APP"`. Source changes
are independent of Marshal and hosted-components releases. CI can run this same command
using a Fly deploy token scoped to the gateway app; keep that token in CI secrets.

The checked-in configuration keeps two machines running in the primary region and disables
autostop. It balances by connections because WebSockets can remain active for a long time.
Adjust machine count/size and concurrency from measured traffic. Two machines provide
machine redundancy, not regional failover. Health checks verify the gateway process, not
all downstream applications.

Upstream connections use HTTPS, SNI, and certificate verification. The HTTP Host must be
the target `.fly.dev` name so Fly can route it; the public hostname is forwarded in
`X-Forwarded-Host`. Apps generating absolute URLs should use their configured public URL
or correctly trust forwarded headers. Cookie domains and Location headers are preserved;
the gateway does not rewrite application authentication policy.

Response buffering, request buffering, and caching are disabled. The upstream read/write
idle timeout is one hour; WebSocket clients should send heartbeats and reconnect after
network interruptions or gateway deployments. This is an idle timeout, not a guarantee
that a connection survives machine replacement. Access logs omit query strings and cookies.

`HEXCLAVE_GATEWAY_RESOLVER` defaults to Fly's internal DNS resolver. The local integration
test overrides it with Docker DNS; this does not change the host-to-app mapping.

Useful commands:

```sh
fly checks list --app "$HEXCLAVE_GATEWAY_APP"
fly logs --app "$HEXCLAVE_GATEWAY_APP"
fly status --app "$HEXCLAVE_GATEWAY_APP"
```

## Tests

With Docker, Bun, and OpenSSL already installed, from the repository root:

```sh
bun test apps/deployment-gateway/gateway.test.mjs
```

This builds the actual gateway image and runs a pinned Bun fixture on an isolated Docker
network. A generated test-only TLS certificate is trusted only inside that disposable
container. Tests verify host rejection, signature verification (unsigned, mis-signed, wrongly
keyed and wrongly shaped hostnames all return 421, with one signature pinned against
Marshal's unit test), health routing, path/method forwarding, cookies,
TLS hostname validation, incremental SSE/chunked responses, and authenticated WebSocket
text/binary echo with clean close. Additional checks cover concurrent applications,
redirects and upstream errors, streaming uploads, disconnected clients, unavailable
upstreams, gateway restart/reconnection, and invalid domain configuration. Containers, network, generated image tag and test keys
are removed afterward. Docker may retain downloaded base images and build cache.

The test port defaults to `10070 + 100 * NEXT_PUBLIC_HEXCLAVE_PORT_PREFIX` (18170 by default).
Override it with `HEXCLAVE_GATEWAY_TEST_PORT` if occupied. The normal Vitest suite skips
this opt-in container test, and runs `parked-page`'s ordinary unit tests instead — the
vitest config here includes that directory by name rather than opting the whole app out:

```sh
pnpm test run apps/deployment-gateway
```

Once the gateway and wildcard DNS/TLS are ready:

```sh
pnpm -C apps/marshal test:platform-domains:live
```

This uses real Fly/S3 credentials in `apps/marshal/.env.local` together with the
gateway's `HEXCLAVE_DEPLOYMENT_HOSTNAME_KEY`, creates a disposable
application, and checks both its direct Fly URL and gateway URL. Open the two printed
`/compatibility` pages and click **Run browser checks** on each. It waits for both reports
before asserting the combined result, then redeploys and checks its stable URL. Cleanup
removes only the disposable application/state, never the shared gateway, DNS or wildcard
certificate. No Vercel alias or bypass secret is needed.

For browser cookie isolation, run two live tests in separate terminals. Before running
both compatibility checks, open the first app's `/compatibility/cookie-isolation` page
and click **Set test session**. Click **Read session** to see that app's marker. On the second app's isolation page,
**Read session** should return only null values.
Set a session on the second app and verify each still returns its own marker. Then run
the normal compatibility checks on both direct and gateway URLs so both runners can
verify redeploy and clean up. This checks host-only/current-host cookie isolation;
applications must still avoid setting cookies on a shared parent domain.

Before rollout, run this live check against the gateway and confirm every category passes.

---

### Overview (`apps/marshal/README.md`)

# Marshal

The Google Cloud-backed container runtime behind the Deployments app. Stateless: no database, its
only state is the S3-compatible bucket. All configuration is environment variables — see
[`.env`](./.env), which documents every one of them.

## Entrypoints

| File | Used by |
| --- | --- |
| `src/server.ts` | `pnpm start` / `pnpm dev` / the e2e workflows — owns the listener |
| `src/vercel.ts` | any host that owns its own listener — exports the app, binds nothing |
| `src/index.ts` | Vercel's entrypoint detection — re-exports the built `dist/vercel.mjs` |

`pnpm build` (tsdown) produces both `dist/server.mjs` and `dist/vercel.mjs`. Local development
and CI run the TypeScript directly through tsx and need no build.

## Deploying on Vercel

Project settings:

- **Root directory** `apps/marshal`, with "include source files outside of the root directory"
  enabled (this is a pnpm workspace).
- **Build command** `pnpm run build` — `src/index.ts` imports the artifact it produces.
- **Framework** is declared in [`vercel.json`](./vercel.json) as `elysia`; the function's
  `maxDuration` is declared in `src/index.ts`, where Vercel's builder statically reads it.
- **Deployment Protection must be off** (or carry a bypass) for the production domain. Two
  callers that hold no Vercel credential must reach it: the Hexclave backend on `/v1/*`, and
  the GCP builder VM on `/internal/deployments/:id/complete`.
- Attach a **stable custom domain** and set `MARSHAL_PUBLIC_URL` to its HTTPS origin. Builder VMs call
  back on that URL minutes after the request that started them, so a per-deployment URL would
  break every build the moment a newer deployment replaced it. Plain HTTP is accepted only when
  `MARSHAL_ALLOW_MOCKS=1` for local simulation.

Environment: set everything in `.env` that is marked required, plus `MARSHAL_PUBLIC_URL`.
`MARSHAL_PORT` is ignored — the platform owns the listener. Never set `MARSHAL_ALLOW_MOCKS`
in production; without it the mock builder and existing-project test override both fail closed.

Source uploads never pass through the function: `POST /v1/namespaces/:ns/uploads` returns a
presigned bucket URL that the CLI uploads the tarball to directly.

Domain claims, project-pool entries and its creation ledger, and namespace records are authenticated with
`HEXCLAVE_MARSHAL_DATA_ENCRYPTION_KEY`, with their object key bound into the MAC. Every write is
signed and a signed record that fails verification is refused. An UNSIGNED record — the shape every
Marshal before signing wrote — is currently still read as-is (see `readAuthenticatedControlPlaneState`
in `src/store.ts`): production's bucket predates the signature, and signing it in place needs the
key in hand, so this version boots on an unsigned bucket rather than failing closed. That leaves
records nobody has rewritten without tamper-evidence until `scripts/sign-control-plane-state.ts`
has been run against the bucket (Fly-era `domains/*.json` are the ones that exist); once it reports
`unsigned: 0`, the read path should be made to fail closed again. Automatically trusting AND
rewriting an unsigned object at read time would authenticate exactly the forgery this boundary is
intended to detect, which is why the script is offline and deliberate.

## Runtimes

Marshal runs a namespace's services on one of two infrastructure providers, chosen per
namespace and pinned in `tenants/<ns>.json` on first use (see `src/runtime.ts`):

- **Fly.io** — the default. A namespace with no record runs here, which is every namespace
  that existed before the second runtime did. One Fly app per service, machines for
  instances, volumes for disks, Fly certificates for custom domains. Enabled by
  `MARSHAL_FLY_API_TOKEN`; see `src/fly/`.
- **Google Cloud** — opted into per project with the internal `version` export in the deploy
  file (`export const version = "gcp-beta-1"`), which the backend sends as `runtime` on the
  deploy. Enabled by `HEXCLAVE_MARSHAL_GCP_PLATFORM_PROJECT_ID`; see `src/gcp/` and the
  sections below.

The two share everything above the provider seam (`src/provider.ts`): specs, deployments,
the build harness, the bucket, leases, redaction, and the HTTP API. A deploy that names a
runtime other than the namespace's pin is refused with a 409 unless the namespace holds no
services, in which case the pin moves. Both mocks (`fly-mock` on port-prefix 48, `gcp-mock`
on 49) run side by side locally, and `MARSHAL_BUILDER=mock` covers either runtime.

The Hexclave backend imposes a stricter rule for the internal GCP beta: the first deployment
attempt fixes the project runtime, including failed attempts and deployments whose services
were later removed. Choose the version before deploying; use a new project to test another
runtime. Existing projects can keep deploying on their selected runtime.

The internal beta rejects references to private serverless services because tenant networks
have no private Cloud Run DNS routing. Use a public serverless target or a private server.
Services with attached custom domains cannot change between server and serverless; detach
the domains first and reattach them after the new type has deployed.

## GCP tenancy and lifecycle

Marshal follows Google's [Cloud Run multi-tenant guidance](https://docs.cloud.google.com/run/docs/securing/multi-tenant): every namespace receives a dedicated GCP project. The project is the tenant security, quota, billing-attribution, monitoring, and deletion boundary. Put these projects in a folder reserved for untrusted tenant workloads; do not place first-party control-plane services in that folder.

Per that guidance, Marshal keeps a pool of pre-provisioned projects (`HEXCLAVE_MARSHAL_GCP_PROJECT_POOL_SIZE`, default 0) so the first deploy into a namespace is a bucket claim instead of a fifteen-minute project provisioning. Pool state lives in the bucket: `tenants/<ns>.json` is the namespace → project assignment (the idempotency anchor), `gcp-project-pool/<projectId>.json` holds one pool project's provisioning state, and `gcp-project-pool-ledger.json` is the creation-rate ledger. When the pool is empty or disabled, provisioning falls back to a deterministic per-namespace project created synchronously on first use, so a stalled pool degrades to the pre-pool latency rather than to a failed deploy.

Provisioning is a **cron-driven resumable state machine**, not background work. `creating → billing_pending → apis_pending → iam_pending → ready → claimed` are resume points: the entry is written *before* the Resource Manager create call, every wait is a state in the bucket, and each step is idempotent so it can be re-entered from whatever the bucket last recorded. This is required rather than merely tidy — the hosted deployment is frozen the moment a response is written, so work started in the background of a request does not run, and an unrecorded create would leave a billed project nothing could ever find.

Two authenticated maintenance endpoints drive it, scheduled in `vercel.json` (and by `apps/backend/scripts/run-cron-jobs.ts` locally). They live under `/v1/` and use the ordinary bearer check, which on this prefix also accepts `CRON_SECRET`. Set `CRON_SECRET` on the Vercel project to any independent value; it need not equal `MARSHAL_API_KEY`, and it opens nothing else. It must be **set**, though: with it unset Vercel sends no `Authorization` header and every invocation is rejected, which shows up only as a pool that never fills.

- `GET /v1/maintenance/project-pool/step` (every two minutes) advances every in-flight project concurrently, running each one as far as it can go and yielding only on a genuine Google Cloud wait — billing propagation and API enablement — or on its own ~75-second deadline. It then creates as many projects as the deficit allows, bounded by at most three in flight and ten creations per hour: deleted GCP projects hold organization project quota for thirty days, so an unbounded advancer would exhaust the organization rather than merely spend money.
- `GET /v1/maintenance/project-pool/reap` (hourly) condemns anything stuck in flight for more than forty-five minutes and deletes its project; returns to `ready` a claimed entry whose `tenants/<ns>.json` does not point back at it, once past a fifteen-minute grace that avoids racing a live claim; and deletes a claimed entry whose namespace *does* point at it, since the assignment is then the authority. That last case is what bounds `listPoolProjects()` — a LIST plus a GET per entry, on the deploy path — by pool size instead of by lifetime tenant count.

Both ticks run under one `withReconciliationLease("__platform__", "project-pool")`, because crons are at-least-once and ticks can overlap; a tick that loses the lease reports itself skipped instead of failing.

Service mapping:

- `serverless` → Cloud Run with the requested minimum/maximum instances and Direct VPC egress. Cloud Run has one ingress port, so a serverless spec must declare exactly one HTTP port. Marshal uses Cloud Run's recommended disabled Invoker IAM check instead of an `allUsers` binding, so public and load-balanced services work under domain-restricted-sharing organization policies; private services remain protected by their ingress setting.
- `server` → one `e2-micro` Compute Engine VM. Its persistent disk is grow-only and survives service deletion for later adoption. A public server receives a scale-to-zero Cloud Run nginx gateway; direct Internet ingress to the VM is not permitted by its firewall.
- source build → a short-lived Container-Optimized OS VM running the existing BuildKit harness. It obtains a short-lived Artifact Registry token from the metadata server; Marshal deletes the VM after recording the completion webhook.
- custom domain → Marshal first returns a tenant-bound TXT record at `_hexclave-verification.<hostname>` plus the shared frontend's A record. The user creates both during domain setup; polling the domain endpoint verifies the TXT proof before Marshal claims or routes the hostname. A verified hostname receives a per-domain serverless NEG and `EXTERNAL_MANAGED` backend service in the tenant project, routed through one environment-scoped global external Application Load Balancer in `HEXCLAVE_MARSHAL_GCP_PLATFORM_PROJECT_ID`. Certificate Manager certificates/map entries, the global IP, URL map, target HTTPS proxy, forwarding rule, and empty fallback backend remain in that platform project. This keeps Cloud Run and its backend security boundary tenant-local while avoiding one fixed-cost frontend per tenant or domain. Marshal serializes URL-map reconciliation with a distributed lease so concurrent tenant updates cannot discard routes.

The platform project and tenant projects must belong to the same organization. Global cross-project backend references do not require Shared VPC, but they do require the controller to have `compute.backendServices.use` on tenant backends. At larger fleet sizes, monitor the URL map's 1 MiB configuration limit and Certificate Manager map-entry quotas and shard tenants across additional environment-scoped frontends before reaching either limit. The shared frontend reduces fixed cost but creates a control-plane blast radius: isolate its project, restrict its administrators, and set the `compute.restrictCrossProjectServices` organization policy on the platform project — it is evaluated against the project holding the URL map, so it bounds which backends that frontend may reference to the tenant folder alone.

Cloud Run has no equivalent of Fly's request-triggered VM suspend/resume for a persistent server. A `server` with `min_instances: 0` therefore remains eligible to run as its single GCE instance; it preserves availability and disk semantics but does not guarantee scale-to-zero billing.

## Fly deployment platform domains

Public HTTP services advertise `https://<suffix>-<mac>.deploy.built-with-hexclave.com`, where
`hxc-<suffix>` is the existing Fly app name and `<mac>` signs it under
`HEXCLAVE_DEPLOYMENT_HOSTNAME_KEY`, shared with the gateway, so the gateway routes only names
this Marshal minted. The name remains stable across redeploys.
The dedicated Fly gateway in [apps/deployment-gateway](../deployment-gateway/README.md)
proxies HTTP, streaming, and WebSockets to the existing `.fly.dev` origin. Provision its
wildcard DNS/TLS before deploying this Marshal version. Hosted components remain on Vercel.

Customer custom domains keep their existing provisioning and display priority; private
services receive no generated platform URL. GCP routing is unchanged. The whole
`deploy.built-with-hexclave.com` namespace is reserved from customer-domain attachment.
There is no per-service DNS record, certificate issuance, or routing database, and no
automatic fallback to `.fly.dev` during a gateway outage.

## Parking

`POST /v1/namespaces/:ns/services/:key/park` stops a service and runs the platform's
parked page in its place; `POST .../unpark` puts the service's own image back. The backend
calls both from its Free-plan sweeper (`apps/backend/src/lib/deployments/parking.tsx`); the
page itself lives in
[apps/deployment-gateway/parked-page](../deployment-gateway/parked-page/README.md).

Parking swaps the image and the environment the machines run with, and nothing else. The
Fly app, ports, public IPs, certificates, custom domains, volumes and the stored spec all
survive, so the explanation answers on the platform hostname, on the `.fly.dev` name and on
every custom domain, and unparking is a re-apply of the spec that was stored all along. The
parked spec sets `min_instances: 0` and, on Fly, presents the service as `serverless`, so
the machine sleeps and Fly Proxy wakes it on the next request: a parked service costs
nothing to keep parked.

Park is idempotent (a service already parked for the same reason with no failed apply
behind it is left alone), and any ordinary apply unparks — which is what makes a redeploy
the other way back. `GET /v1/namespaces/:ns/services/:key` reports `status: "parked"` and a
`parked` object; a non-null `parked` with some other status is a park whose apply failed,
meaning the tenant's image is still serving.

A GCP `server` is the one gap: there the service type selects the resource kind rather than
a scaling policy, so its VM keeps running the parked page rather than stopping. Nothing
reaches it today, because a `server` is refused outright on the Free plan on GCP.

## Local GCP simulator

Development and provider-dependent backend E2E tests use `docker/dependencies/gcp-mock`. It implements only the Google REST resources Marshal owns; tests that do not cross the provider boundary continue to use focused `GcpClient` fakes. Set `HEXCLAVE_MARSHAL_GCP_MOCK_URL=local` to derive the simulator address from `NEXT_PUBLIC_HEXCLAVE_PORT_PREFIX`, or provide an explicit URL. Both forms require `MARSHAL_ALLOW_MOCKS=1`, and the introspection API also requires `HEXCLAVE_MARSHAL_GCP_MOCK_TOKEN` because it exposes resolved container environment values.

The simulator deliberately reproduces provider details Marshal depends on: permission-hidden unknown projects, long-running operations, newly-enabled API propagation errors, Cloud Run create/update body differences and stale reads, revision image digests, Compute Engine operation/resource shapes, grow-only persistent disks, Certificate Manager resources/states, URL-map fingerprint updates, same-organization cross-project backend validation, and Cloud Logging resource labels. The `src/gcp/mock-contract.test.ts` lifecycle is kept parallel to `src/gcp/live.test.ts`; changes to a provider adapter should update both. Deterministic `.verified.test` certificate activation and synthetic container readiness/log output are test controls, not claims that the simulator executes containers or provisions a real network.

## Required IAM

Credentials resolve in three ways, in this order: workload identity federation, an explicit
`GOOGLE_APPLICATION_CREDENTIALS` file, then the GCE metadata server.

Prefer federation for any hosted deployment; it is required on a host with no metadata server
(Vercel). Set `HEXCLAVE_MARSHAL_GCP_WORKLOAD_IDENTITY_AUDIENCE` to the provider resource and
`HEXCLAVE_MARSHAL_GCP_WORKLOAD_IDENTITY_SERVICE_ACCOUNT` to the controller service account it
impersonates — setting only one of the two is a startup error rather than a fallback. Also set
`HEXCLAVE_MARSHAL_GCP_WORKLOAD_IDENTITY_ASSERTION_AUDIENCE` to the `aud` the host mints into
its assertion (`https://vercel.com/<team-slug>` on Vercel; the provider's `--allowed-audiences`
value, which `bootstrap-gcp.sh` prints). It is a different value from the STS audience: the
exchange is addressed to the provider resource, but the assertion is addressed to the team URL,
and Marshal filters incoming assertion headers by the latter. Left unset it defaults to the
provider resource, with or without the `https:` scheme, matching a provider created without
`--allowed-audiences`. Marshal
exchanges the host's OIDC assertion for a federated token and impersonates the service account
with it, so no long-lived key exists anywhere. This matters more here than it usually does: the
controller identity can create, bill, and delete every tenant project, so a static key for it
would be the most valuable secret the system holds.

The assertion is read from the incoming request's `x-vercel-oidc-token` header, falling back to
`VERCEL_OIDC_TOKEN` (or whatever `HEXCLAVE_MARSHAL_GCP_WORKLOAD_IDENTITY_TOKEN_ENV` names) for
builds and local development. Those are genuinely different places, not redundancy: Vercel
populates the environment variable during builds and in `vercel dev`, but a running Function
receives the assertion as a per-invocation header and has no such variable, so a deployment
that only read the environment would authenticate locally and hold no credential in production.
OIDC federation must also be enabled on the Vercel project itself, or no assertion is issued at
all.

Use `GOOGLE_APPLICATION_CREDENTIALS` only for local administration, and never copy a key into
this repository or a tenant project.

[`scripts/bootstrap-gcp.sh`](./scripts/bootstrap-gcp.sh) provisions the IAM and org-policy
prerequisites below — tenant folder, platform project, controller service account, every role
binding, the org policies Marshal depends on, and the federation pool — and is idempotent. It
does not raise the organization's project-creation quota or touch the Logging default sink;
it prints both as remaining manual steps.

The Marshal controller needs:

- `roles/resourcemanager.projectCreator` and `roles/resourcemanager.projectDeleter` on the configured tenant-project parent.
- `roles/billing.user` on `HEXCLAVE_MARSHAL_GCP_BILLING_ACCOUNT`, **and** `roles/billing.projectManager` on the tenant-project parent. Both are needed: attaching billing to a new tenant project requires a permission on the billing account and another on the project itself, so `billing.user` alone fails the first deploy with a 403.
- `roles/browser` (or another role granting `resourcemanager.projects.get`) on the tenant-project parent, for the project lookups reconciliation performs.
- inherited access on the tenant-project folder equivalent to `roles/serviceusage.serviceUsageAdmin`, `roles/resourcemanager.projectIamAdmin`, `roles/compute.admin`, `roles/run.admin`, `roles/artifactregistry.admin`, `roles/iam.serviceAccountUser`, and `roles/logging.viewer`.
- `roles/compute.loadBalancerServiceUser` (or another role containing `compute.backendServices.use`) on tenant projects whose backends the platform URL map references.
- `roles/compute.loadBalancerAdmin` and `roles/certificatemanager.owner` on the platform project. Certificate Manager's predefined editor role omits resource deletion, so it cannot clean up certificates, map entries, or maps. Enable `compute.googleapis.com` and `certificatemanager.googleapis.com` there before starting Marshal.

The organization or folder Logging defaults must retain Cloud Run and Compute Engine entries in the project's `_Default` log bucket. Disabling that sink makes Marshal's logs API return no entries even when the controller has `roles/logging.viewer`.

Marshal enables Compute Engine, Cloud Run, Artifact Registry, IAM, and Cloud Logging in each tenant project. It grants only these runtime bindings inside the project:

- the default Compute service account: `roles/artifactregistry.writer` and `roles/logging.logWriter`;
- the Cloud Run service agent: `roles/compute.networkUser` for Direct VPC egress.

For a disposable project created out-of-band, `HEXCLAVE_MARSHAL_GCP_EXISTING_PROJECT_ID_FOR_TESTS` bypasses project creation. It is guarded by `MARSHAL_ALLOW_MOCKS=1` and must never point at a production project.

## Disposable live verification

### Fly gateway platform domains

From the repository root, run:

```sh
pnpm -C apps/marshal test:platform-domains:live
```

The runner reads real Fly and S3 credentials from `apps/marshal/.env.local`, accepting either
the `MARSHAL_*` names or `FLY_API_TOKEN`, `FLY_ORG_SLUG`, `S3_ACCESS_KEY_ID`,
`S3_SECRET_ACCESS_KEY`, `S3_API_ENDPOINT`, and `S3_BUCKET_NAME`, plus the gateway's hostname
signing key as `HEXCLAVE_DEPLOYMENT_HOSTNAME_KEY` (or `GATEWAY_HOSTNAME_KEY`) — the gateway
routes only hostnames signed with its own key. No Vercel API token is required.
Configure the dedicated gateway and wildcard DNS/TLS first, following its README.
For an isolated test gateway on a separate domain, pass
`HEXCLAVE_DEPLOYMENT_PLATFORM_DOMAIN=deploy.example.net` to the command and configure the
gateway with the same value. The production domain is the default when unset.
`--help` prints usage without loading credentials or calling providers.

The test creates one disposable Fly app/machine (which can incur usage charges), checks its
default Fly HTTPS response and branded HTTPS proxy, redeploys while retaining the hostname,
and verifies cleanup. It uses a pinned prebuilt Bun image and production service code;
source builds and the backend/dashboard deployment flow are outside its scope. No custom
certificate is requested, so the test does not consume custom-domain issuance quota.

After the HTTP checks pass, the runner prints two `/compatibility` URLs. Open **both**
in a browser and click **Run browser checks** on each page. Keep the terminal running. It waits up to ten minutes for reports from both origins before redeploy
and cleanup. No browser packages need to be installed.

The browser checks cover:
- SSE and chunked responses: first and final markers must arrive at least 1.5 seconds apart
  from an origin that waits 3 seconds, detecting a proxy that buffers the whole response.
- WebSockets: upgrade, session cookie authentication, subprotocol negotiation, text and
  binary echo, and a clean close.
- Cookies: Secure host-only `__Host-`/HttpOnly sessions, explicit current-host Domain cookies,
  Path scope, rejection of cookies scoped to the other origin, rotation, and logout.
  Separate Set-Cookie headers and SameSite/Secure/HttpOnly/Path/Domain attributes are also
  checked by the terminal probes.

A failure on either origin fails the run and still triggers cleanup. These are compatibility
fixtures, not tests of a particular application's OAuth flow. Cross-site iframe/third-party
cookie policies, cross-site SameSite enforcement, long-lived connection limits, and large
uploads remain outside this test. The fixture creates no parent-domain cookies.

Test state is isolated under a unique `live-domain-tests/<run-id>/` S3 prefix. The runner
cleans up after success, failure, or Ctrl+C. A complete pass exits **0**; test or cleanup
failures exit **1**. A private recovery file containing the disposable identity and encryption
key (no provider tokens) is printed before provisioning. If cleanup fails or the process is
killed, retain that file and run:

```sh
pnpm -C apps/marshal test:platform-domains:live --cleanup /path/printed/by/the/test.untracked.json
```

Use the same Fly/S3 account for cleanup. The file is removed only after cleanup is verified.
The live runner is not included in the normal Vitest suite.

### GCP

`src/gcp/live.test.ts` is opt-in because it creates billable resources. Set Application Default Credentials plus `HEXCLAVE_MARSHAL_GCP_LIVE_TEST=1`, `HEXCLAVE_MARSHAL_GCP_LIVE_BILLING_ACCOUNT`, and `HEXCLAVE_MARSHAL_GCP_LIVE_PLATFORM_PROJECT_ID`; optionally set `HEXCLAVE_MARSHAL_GCP_LIVE_PROJECT_PARENT=folders/<id>`. The existing platform project must have Compute Engine and Certificate Manager enabled and the controller roles documented above. Then run `pnpm -C apps/marshal test -- src/gcp/live.test.ts`.

The test creates one `hxctest-` tenant project and uses the configured existing platform project for an environment-scoped frontend. Its `try/finally` boundary removes the domain, every shared frontend/certificate resource created for the unique live-test environment, and the tenant project. It covers project/API/IAM provisioning, Artifact Registry, VPC creation, Cloud Run deployment and update, HTTP status and Cloud Logging, a persistent Compute Engine server and disk adoption across update, cross-project custom-domain load-balancer creation/status/removal, individual runtime deletion, and final cleanup. The custom hostname intentionally remains unconfigured, so the test verifies the returned DNS record and pending certificate state without changing public DNS.

---

### Overview (`devices/tv-box/README.md`)

# Hexclave TV Box Pilot Appliance

This directory contains the device-only layer for the Raspberry Pi Zero 2 W pilot. The box remains a thin client: it opens `https://app.hexclave.com/tv-box`; Hexclave continues to own pairing, display authorization, snapshots, profiles, privacy, and presentation decisions.

Customer onboarding, Wi-Fi password recovery, operator acceptance, and troubleshooting procedures are in [PILOT_RUNBOOK.md](PILOT_RUNBOOK.md).

## Build inputs

- Raspberry Pi `rpi-image-gen` pinned to commit `3f2c916086ad70197945bfc50ef953c1f6035f10` (v2.6.0).
- `HEXCLAVE_TV_BOX_WIFI_COUNTRY`: region-specific two-letter regulatory country.
- `HEXCLAVE_TV_BOX_SUPPORT_CA_PUBLIC_KEY_FILE`: OpenSSH CA **public** key. Never place the private key in this repository or image.
- `HEXCLAVE_TV_BOX_TEST_IMAGE`: optional, defaults to `false`. Set it to exactly `true` only for a non-shippable hardware-test image.

Run `scripts/build-image.sh` on a supported Raspberry Pi image-build host. Generated images, checksums and per-device state are release/manufacturing artifacts and must not be committed.

The copied appliance payload directories must also be free of ignored local files. Build checks reject those files even when Git reports a clean working tree; only regular Python bytecode in the runtime package is allowed because the image layer removes it. Reference documents and build artifacts outside the copied payload directories are not image inputs.

Before building, run `scripts/validate-source.sh`. Supplying `RPI_IMAGE_GEN_DIR` additionally validates all custom metadata and dependency resolution against the exact pinned builder. From the repository root, run `pnpm test run apps/dashboard/tv-box-runtime.test.js apps/dashboard/tv-box-app.test.js apps/dashboard/tv-box-bootstrap.test.js apps/dashboard/src/app/tv-box/document.test.ts apps/dashboard/src/app/tv-box/qa/route.test.ts apps/dashboard/src/app/tv/page-client.test.ts` for the framework-free renderer, shared request deadlines, and route contracts.

Pilot media must be at least 16 GB; the qualified hardware uses 32 GB high-endurance microSD cards. The MBR image contains a fixed boot partition, a 6 GB writable pilot root, a bounded 1 GB persistent-state partition, and a dedicated 2 GB swap partition. Creating these filesystems in the image avoids unsafe first-boot repartitioning and remains compatible with the Pi Zero 2 W boot layout.

The build checks the OS, architecture, and critical package versions against [image/qualified-runtime.json](image/qualified-runtime.json). Version drift stops the build: review and requalify a changed runtime before updating this baseline. This is not a complete reproducible repository snapshot; that remains future hardening.

The verification and manufacturing scripts require the raw disk image. Verify the compressed download's checksum first, then decompress it while keeping the original artifact (for example, `xz --decompress --keep IMAGE.img.xz` or `zstd --decompress --keep IMAGE.img.zst`). Attach that exact image read-only and mount its boot partition read-only and root/state partitions with `ro,noload`, not an extracted build directory. Decompress the builder's existing `manifest.zst` separately for the inventory input, then run:

```sh
scripts/verify-image.sh DISK_IMAGE ROOTFS_MOUNT STATE_MOUNT BOOT_MOUNT BUILDER_MANIFEST OUTPUT_DIRECTORY
```

Verification checks mount backing/offsets, runtime versions, exact installed-package inventory, clean unpaired state, and targeted credential contamination across boot/root/state. It rejects private keys and pre-issued user certificates; the sole packaged public cryptography-test-vector exception requires its exact path and file hash, with its package and upstream source recorded in the reviewed policy. These checks cannot prove the absence of every arbitrarily encoded secret. The receipt binds the raw image, inventory, source revision, and qualification policy using SHA-256, and records the reviewed builder revision that the build wrapper checks against its checkout. This is a build-host check, not a new Pi monitoring service or a signed release pipeline.

`scripts/manufacture.sh RAW_IMAGE BLOCK_DEVICE VERIFICATION_DIRECTORY` requires a matching production-channel receipt, validates the four-partition layout and target capacity, and requires explicit whole-device confirmation before writing. It flushes host buffers and verifies a read-back of the full image extent before reporting success. Raspberry Pi Imager remains an alternative that supports compressed input itself; let its verification stage finish. Neither check replaces first-boot and hardware qualification.

## Quick Tunnel test images

A test image may open an ephemeral Cloudflare Quick Tunnel before `/tv-box` is deployed. Build it with `HEXCLAVE_TV_BOX_TEST_IMAGE=true`; this names the artifact `hexclave-tv-box-test` and writes a test-channel marker into the root filesystem. After flashing, place a file named `hexclave-tv-box-test-origin.txt` in the Mac-editable boot volume containing exactly one origin such as:

```text
https://example-random-name.trycloudflare.com
```

The appliance validates one lowercase, single-label, HTTPS `*.trycloudflare.com` origin and appends `/tv-box` itself. Ports, paths, queries, fragments, credentials, wildcards, nested subdomains, additional lines, and other domains are rejected. A missing or rejected override retains the production URL and records a bounded configuration error in the local journal. Add or replace the file while the card is powered off, then boot or reboot the box; the network agent resolves the URL once when it starts.

Production images do not contain the build-time test marker and therefore ignore this boot file completely, even if it is later added. Never ship an image whose manifest says `image-channel=test`; rebuild a production image instead of trying to convert a flashed test image.

Test images also stop after exhausting bounded service restart attempts instead of rebooting and retain bounded Cage/Cog failure diagnostics for support collection. Production/pilot images retain automatic reboot recovery. Both channels use an eight-character, randomly generated temporary setup password from the same easy-to-read alphabet; this onboarding behavior does not require test-image mode.

## Runtime ownership

- `hexclave-tv-box-firstboot.service` creates the per-box device ID, hostname, machine-ID record, and SSH host keys after the state partition is mounted.
- `hexclave-tv-box-relay-identity.service` independently creates a unique relay-only key after first boot. `hexclave-tv-box-relay.service` is activated only by explicit per-device enrollment; its short validator replaces itself with OpenSSH, and systemd owns automatic retries. No Python relay supervisor stays resident; neither service is a dependency of Wi-Fi, the renderer, or local SSH.
- `hexclave-tv-box-network.service` is the sole privileged Wi-Fi policy owner. It drives NetworkManager and exposes a narrow local Unix-socket protocol to the unprivileged setup portal.
- `hexclave-tv-box-setup.service` serves only the local captive portal. Wi-Fi secrets cross that local socket once, enter `nmcli` through a mode-0600 password file, and remain in NetworkManager's state-partition-backed profiles.
- `hexclave-tv-box-setup-display.service` owns tty1 only during local setup and prints the temporary network credentials directly to HDMI. It refreshes them when a failed station attempt starts a new setup session. It does not use WebKit, write those credentials to the journal, or persist them.
- `hexclave-tv-box-kiosk.service` owns and explicitly activates tty1 through a dedicated logind session, gives Cage and Cog one private Wayland runtime directory, then runs Cage in its supported no-input appliance mode with Cog pinned to the Wayland platform, a persistent cookie jar, and a volatile cache. A narrow supervisor checks the exact Cage/Cog/WPE process tree, rejects stopped processes, and tolerates a short web-process replacement window. It adopts renderer orphans as a Linux child subreaper because PAM can place children outside the service cgroup. Shutdown targets only those descendants using process identities and pidfds: ten seconds for graceful exit, followed by up to five seconds for forced termination and reaping. The pre-existing PAM helper is excluded. systemd limits the unit to five starts in fifteen minutes, covering slow as well as rapid crash loops; production images retain reboot recovery, while test images stop for diagnosis.
- The `/tv-box` browser runtime distinguishes authoritative credential rejection from temporary network/backend failure. Only rejection returns the appliance to pairing; transient failure keeps local identity and retries with bounded backoff.

With no saved network, setup mode starts immediately. The browser-independent HDMI display remains available even if Cog/WPE cannot start, and the captive portal must pass a bounded local readiness check. Cage/Cog starts only for offline or connected application content. With a saved network, the appliance first performs one synchronous NetworkManager activation and recheck so a healthy boot launches the connected renderer only once; if station activation still fails, it shows offline content while retrying for five minutes, offers setup for fifteen minutes, and then alternates two-minute station retries with setup windows. Backend availability does not participate in this Wi-Fi state machine.

The supervisor also observes Cog's native document-load result without enabling page-console logging. Failed navigation retries after four minutes; navigation that never completes gets a two-minute load deadline followed by that retry delay. These retries stay below the crash-loop budget. The network agent's once-per-minute, credential-free origin probe can accelerate recovery, but only when the supervisor reports a failed or timed-out document. Flapping probes do not restart a healthy slideshow or change Wi-Fi state. Intentional network/document restarts reset the start counter only for an active unit with a valid healthy-process record; process failures and periodic reconciliation retain their budget. An inline bootstrap deadline reloads a document whose external application module failed to initialize: initially after thirty seconds, then with backoff capped at five minutes, without a fixed retry-count cutoff. Once the application is running, its request deadlines and retry logic own API outages and stale-state presentation.

Process/native-load readiness is not proof that every later JavaScript frame rendered successfully. A post-load busy JavaScript loop remains a qualification limit. Likewise, forcibly killing or indefinitely stopping the supervisor itself can prevent its exact descendant cleanup; the service cgroup alone is not a substitute for that cleanup across PAM scopes. Do not describe these as covered failure modes without additional validation and recovery design.

## Pilot display and identity policy

The qualified Pi appliance uses fixed 1920×1080 at 60 Hz for `/tv-box`, not EDID-preferred resolution selection. This retains the tested rendering budget. It is a device HDMI setting, not a web-wide resolution cap: `/tv` and its higher-resolution presentation on other devices are unchanged.

A build-generated, static transparent cursor is selected only for Cage/Cog through their private `XCURSOR_PATH`. No animation, polling daemon, or hosted `/tv` cursor change is involved. The Wi-Fi portal's timezone must decode as an installed timezone; it is persisted for the renderer's `TZ` environment, without modifying `/etc/localtime` or other OS timezone files. It is not an SSH access restriction.

The clean image has an empty `/etc/machine-id`; systemd creates a unique OS machine ID on first boot. That file remains on the writable pilot root, with a copy recorded in persistent state. Ordinary restarts preserve the OS ID, device UUID, host keys, saved network, and browser identity. After an explicit factory reset, the next boot regenerates the device UUID and SSH host keys and starts unpaired, but retains the OS machine ID. The OS ID is not a display credential. Read-only-root qualification must account for this layout; do not rewrite the running OS identity as part of ordinary support.

## Pilot support

SSH accepts only short-lived certificates for the `hexclave-tv-support` principal. The forced support interface exposes a fixed command allowlist; it does not provide an arbitrary shell. Pairing or factory reset must follow dashboard admin unpair so an offline reset cannot leave an authorized remote display record behind.

Support mutations are serialized across SSH sessions using a root-owned lock outside the network agent's runtime directory. While that agent is running, kiosk/network/reset transitions pass through its policy lock; a reset cannot start a browser over Wi-Fi setup or delete browser state while another support session restarts it. Read-only diagnostics remain available during these operations.

Office support can use the optional [reverse-SSH transport](PILOT_RUNBOOK.md#optional-support-relay). A configurable, trusted OpenSSH endpoint, per-device enrollment and operator authentication are required; there is no hosting-provider assumption or need for another backend. An isolated relay service may run on an approved existing host. The base image is unenrolled and makes no support connection. One relay-loopback listener reaches the appliance's loopback SSH service, preserving its certificate/forced-command boundary. Local support remains independently available from private/link-local sources on Wi-Fi, not globally routed addresses. This address rule is not a claim of same-subnet enforcement; a Mac can continue using the Pi's private LAN IPv4 address with no relay or jump host.

Factory reset returns **scheduled** and runs as a fixed systemd-owned job so closing its own support connection cannot interrupt cleanup. It takes the mutation lock, stops the relay and removes only the exact device state directories, including the relay identity. Revoke the old relay account separately and enroll the reset box again; dashboard unpair does not manage support registration.

Diagnostics distinguish relay enrollment from systemd process state and expose its public enrollment key. The no-argument `relay-metrics` support command reads CPU/memory/task accounting on demand for baseline, idle and reconnect measurements without a monitoring daemon. A running process is not proof of a working remote login. Network logs record transition reasons, durations and retry counts, with repeated failures coalesced. Support audits contain only validated command labels, outcomes, durations and process/operation IDs; individual certificate identity remains in native SSH authentication logs. Journal limits remain 32 MiB persistent/16 MiB runtime. There is no automatic log upload or raw page-console/SSH-debug export, and no credential, Wi-Fi-name or snapshot logging.

Pilot software updates remain serviced image replacements/reflashes with recorded image versions. The optional support transport is not an unattended updater or fleet service. Signed release, filesystem and broader qualification gates remain in Phase 2; fleet/OTA rollout is future scope, not an added requirement for this pilot slice.

The pilot image masks automatic package-update timers so field units cannot drift away from their recorded image artifact. Bluetooth is disabled because the appliance has no Bluetooth product function; Wi-Fi setup and restricted certificate-based support remain the only intended wireless and administrative paths.

## Acceptance and Phase 2 gates

The source-level implementation is followed by the per-device and soak checks in [PILOT_RUNBOOK.md](PILOT_RUNBOOK.md). The pilot does not complete the read-only-root conversion, signed image/SBOM pipeline, physical customer reset mechanism, fleet management, OTA updates, enterprise Wi-Fi, or full GA fault-injection matrix. These remain future scope; the 72-hour soak and broader hardware/security qualification are not completed by the pilot's 10-hour soak.

---

### Overview (`examples/better-auth-integration-demo/README.md`)

# Better Auth → Hexclave example

This example signs in with a local Better Auth instance, exchanges its provider
JWT with the local Hexclave backend, and displays the resulting session
metadata.

## Local setup

1. Copy `.env.local.example` to `.env.local`.
2. Set a local `BETTER_AUTH_SECRET` and the local Hexclave client and server
   keys and API URL.
3. Run `pnpm dev` from this directory. The script follows the repository's
   `NEXT_PUBLIC_HEXCLAVE_PORT_PREFIX` convention (`81` defaults to port `8114`).
4. Create a Better Auth user through the form, then sign in and exchange its
   token with Hexclave.

Before exchanging a token, enable the `better-auth-integration` provider in
the Hexclave dashboard at
`http://localhost:8101/projects/<project-id>/better-auth-integration`.
Configure the matching Better Auth values:

- issuer: `http://localhost:8114`
- audience: `better-auth-integration-demo`
- JWKS URL: `http://localhost:8114/api/auth/jwks`

If you use a custom port prefix, export `NEXT_PUBLIC_HEXCLAVE_PORT_PREFIX` in
your terminal before running `pnpm dev` (the `dev` script reads it from the
shell, not from `.env.local`), and then update the dashboard URL,
`BETTER_AUTH_URL`, the issuer and JWKS URL, and `NEXT_PUBLIC_HEXCLAVE_API_URL`
together; `.env` files do not derive ports automatically.

The example intentionally does not include provider credentials, access tokens,
or user credentials. The local SQLite database is created at runtime.

---

### Overview (`examples/cjs-test/README.md`)

This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).

## Getting Started

First, run the development server:

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.

You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.

This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.

## Learn More

To learn more about Next.js, take a look at the following resources:

- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.

You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!

## Deploy on Vercel

The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.

Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

---

### Overview (`examples/clerk-integration-demo/README.md`)

# Clerk → Hexclave example

This standalone example loads Clerk's browser SDK from its CDN, signs in a real
Clerk development-instance user, and exchanges the resulting provider JWT with
the local Hexclave backend.

## Local setup

1. Copy `.env.local.example` to `.env.local`.
2. Set `NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY` to the publishable key for your Clerk
   development instance.
3. Run `pnpm dev` from this directory. It uses port `8115` by default.
4. Configure the `clerk-integration` provider in the local dashboard at
   `http://localhost:8101/projects/<project-id>/clerk-integration`:
   - Issuer: `https://<your-clerk-domain>`
   - Authorized parties: `http://localhost:8115`
5. Sign in with Clerk. The page displays the decoded provider claims and the
   mapped Hexclave user profile.

The same-origin exchange route accepts POST only. It validates the provider
token input and every backend response before returning safe display data.
The provider token is never written to disk or committed.

## Getting the user's email and name

Clerk's default session token has no `email` or `name` claim, so Hexclave
creates the user with an empty profile. To map them, add the claims in the Clerk
dashboard under **Sessions → Customize session token**:

```json
{
  "email": "{{user.primary_email_address}}",
  "name": "{{user.full_name}}"
}
```

Hexclave maps these claims only when it first creates the user, so delete the
existing Hexclave user on the project's **Users** page, then sign out of Clerk
and sign in again. Deleting a user also removes its authentication methods, team
memberships and other project data, so do this only for a disposable test
identity or project.

---

### Overview (`examples/convex/README.md`)

# Welcome to your Convex + Next.js app

This is a [Convex](https://convex.dev/) project created with [`npm create convex`](https://www.npmjs.com/package/create-convex).

After the initial setup (<2 minutes) you'll have a working full-stack app using:

- Convex as your backend (database, server logic)
- [React](https://react.dev/) as your frontend (web page interactivity)
- [Next.js](https://nextjs.org/) for optimized web hosting and page routing
- [Tailwind](https://tailwindcss.com/) for building great looking accessible UI

## Get started

If you just cloned this codebase and didn't use `npm create convex`, run:

```
npm install
npm run dev
```

If you're reading this README on GitHub and want to use this template, run:

```
npm create convex@latest -- -t nextjs
```

## Learn more

To learn more about developing your project with Convex, check out:

- The [Tour of Convex](https://docs.convex.dev/get-started) for a thorough introduction to Convex principles.
- The rest of [Convex docs](https://docs.convex.dev/) to learn about all Convex features.
- [Stack](https://stack.convex.dev/) for in-depth articles on advanced topics.

## Join the community

Join thousands of developers building full-stack apps with Convex:

- Join the [Convex Discord community](https://convex.dev/community) to get help in real-time.
- Follow [Convex on GitHub](https://github.com/get-convex/), star and contribute to the open-source implementation of Convex.

---

### Overview (`examples/e-commerce/README.md`)

This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).

## Getting Started

First, run the development server:

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.

You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.

This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.

## Learn More

To learn more about Next.js, take a look at the following resources:

- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.

You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!

## Deploy on Vercel

The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.

Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

---

### Overview (`examples/lovable-react-18-example/README.md`)

# Welcome to your Lovable project

## Project info

**URL**: https://lovable.dev/projects/216a06f3-619b-4a7a-848e-54b52396418a

## How can I edit this code?

There are several ways of editing your application.

**Use Lovable**

Simply visit the [Lovable Project](https://lovable.dev/projects/216a06f3-619b-4a7a-848e-54b52396418a) and start prompting.

Changes made via Lovable will be committed automatically to this repo.

**Use your preferred IDE**

If you want to work locally using your own IDE, you can clone this repo and push changes. Pushed changes will also be reflected in Lovable.

The only requirement is having Node.js & npm installed - [install with nvm](https://github.com/nvm-sh/nvm#installing-and-updating)

Follow these steps:

```sh
# Step 1: Clone the repository using the project's Git URL.
git clone <YOUR_GIT_URL>

# Step 2: Navigate to the project directory.
cd <YOUR_PROJECT_NAME>

# Step 3: Install the necessary dependencies.
npm i

# Step 4: Start the development server with auto-reloading and an instant preview.
npm run dev
```

**Edit a file directly in GitHub**

- Navigate to the desired file(s).
- Click the "Edit" button (pencil icon) at the top right of the file view.
- Make your changes and commit the changes.

**Use GitHub Codespaces**

- Navigate to the main page of your repository.
- Click on the "Code" button (green button) near the top right.
- Select the "Codespaces" tab.
- Click on "New codespace" to launch a new Codespace environment.
- Edit files directly within the Codespace and commit and push your changes once you're done.

## What technologies are used for this project?

This project is built with:

- Vite
- TypeScript
- React
- shadcn-ui
- Tailwind CSS

## How can I deploy this project?

Simply open [Lovable](https://lovable.dev/projects/216a06f3-619b-4a7a-848e-54b52396418a) and click on Share -> Publish.

## Can I connect a custom domain to my Lovable project?

Yes, you can!

To connect a domain, navigate to Project > Settings > Domains and click Connect Domain.

Read more here: [Setting up a custom domain](https://docs.lovable.dev/features/custom-domain#custom-domain)

---

### Overview (`examples/middleware/README.md`)

This is a [Next.js](https://nextjs.org/) project bootstrapped with [`create-next-app`](https://github.com/vercel/next.js/tree/canary/packages/create-next-app).

## Getting Started

First, run the development server:

```bash
npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev
```

Open [http://localhost:3000](http://localhost:3000) with your browser to see the result.

You can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.

This project uses [`next/font`](https://nextjs.org/docs/basic-features/font-optimization) to automatically optimize and load Inter, a custom Google Font.

## Learn More

To learn more about Next.js, take a look at the following resources:

- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.

You can check out [the Next.js GitHub repository](https://github.com/vercel/next.js/) - your feedback and contributions are welcome!

## Deploy on Vercel

The easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.

Check out our [Next.js deployment documentation](https://nextjs.org/docs/deployment) for more details.

---

### Overview (`examples/react-example/README.md`)

# React + TypeScript + Vite

This template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.

Currently, two official plugins are available:

- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react/README.md) uses [Babel](https://babeljs.io/) for Fast Refresh
- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react-swc) uses [SWC](https://swc.rs/) for Fast Refresh

## Expanding the ESLint configuration

If you are developing a production application, we recommend updating the configuration to enable type aware lint rules:

- Configure the top-level `parserOptions` property like this:

```js
export default tseslint.config({
  languageOptions: {
    // other options...
    parserOptions: {
      project: ['./tsconfig.node.json', './tsconfig.app.json'],
      tsconfigRootDir: import.meta.dirname,
    },
  },
})
```

- Replace `tseslint.configs.recommended` to `tseslint.configs.recommendedTypeChecked` or `tseslint.configs.strictTypeChecked`
- Optionally add `...tseslint.configs.stylisticTypeChecked`
- Install [eslint-plugin-react](https://github.com/jsx-eslint/eslint-plugin-react) and update the config:

```js
// eslint.config.js
import react from 'eslint-plugin-react'

export default tseslint.config({
  // Set the react version
  settings: { react: { version: '18.3' } },
  plugins: {
    // Add the react plugin
    react,
  },
  rules: {
    // other rules...
    // Enable its recommended rules
    ...react.configs.recommended.rules,
    ...react.configs['jsx-runtime'].rules,
  },
})
```

---

### Overview (`examples/workos-integration-demo/README.md`)

# WorkOS AuthKit → Hexclave example

This example signs in with WorkOS AuthKit, exchanges the provider access token
with the local Hexclave backend, and displays the resulting session metadata.

## Local setup

1. Copy `.env.local.example` to `.env.local`.
2. Fill in the WorkOS AuthKit values for your own development tenant:
   - `WORKOS_CLIENT_ID`
   - `WORKOS_API_KEY`
   - `WORKOS_AUTHKIT_DOMAIN`
   - `WORKOS_COOKIE_PASSWORD` (use a local value at least 32 characters long)
3. Set the local Hexclave client and server keys and API URL.
4. Configure `http://localhost:8113/auth/callback` as an allowed WorkOS redirect in WorkOS. If you use a custom `NEXT_PUBLIC_HEXCLAVE_PORT_PREFIX`, update this URI in WorkOS and `.env.local` together; `.env` files cannot compute these ports from the prefix automatically.
5. Run `pnpm dev` from this directory.

Enable the `workos-integration` provider in the Hexclave dashboard at
`http://localhost:8101/projects/<project-id>/workos-integration` before
exchanging a token. Configure the WorkOS client ID and API key there; the
dashboard derives the issuer and JWKS URL from the client ID unless an explicit
override is required. If you use a custom port prefix, update the dashboard
URL, API URL, and redirect URI together; the redirect URI must be registered in
WorkOS for the port you use.

The example intentionally does not include provider credentials, access tokens,
or user credentials. The WorkOS issuer is derived from the configured client ID
by the Hexclave dashboard unless an explicit override is required.

## Getting the user's email and name

A WorkOS AuthKit access token has no `email` or `name` claim by default, so
Hexclave creates the user with an empty profile. To map them:

1. In the WorkOS dashboard, open **Authentication → Features → JWT Template**
   and save:

   ```json
   {
     "email": {{ user.email }},
     "name": "{{ user.first_name || '' }} {{ user.last_name || '' }}"
   }
   ```

2. Delete the existing Hexclave user on the project's **Users** page. Hexclave
   maps these claims only when it first creates the user. Deleting a user also
   removes its authentication methods, team memberships and other project data,
   so do this only for a disposable test identity or project.
3. Sign out and sign in again so WorkOS issues a token from the new template.

The email is stored as unverified unless the token has `email_verified: true`
(a boolean).

---

### Overview (`sdks/spec/README.md`)

# Hexclave SDK Specification

This folder contains the specification for Hexclave's SDKs.

When writing this specification, try to write imperative pseudocode as much as possible (be explicit about what things are named, etc.).

## Notation

The spec files use the following notation:

| Notation | Meaning |
|----------|---------|
| `[authenticated]` | Include access token, handle 401 refresh |
| `[server-only]` | Requires secretServerKey |
| `[BROWSER-LIKE]` | Requires browser or browser-like environment (browser, WebView, in-app browser). On mobile, open an in-app browser (ASWebAuthenticationSession on iOS, Custom Tabs on Android). On desktop, open the system browser with a registered URL scheme. |
| `[BROWSER-ONLY]` | Strictly requires browser environment (DOM, window object) |
| `[CLI-ONLY]` | Only in languages/platforms with an interactive terminal |
| `[JS-ONLY]` | Only available in the JavaScript SDK |
| `{ field, field }` | Request body (JSON) |
| `"Does not error"` | Function handles errors internally |
| `"Errors: ..."` | Lists possible errors with code/message |

See _utilities.spec.md for more details.

## Language Adaptation

The languages should adapt:

- **Naming conventions**: camelCase (JS), snake_case (Python), PascalCase (Go), etc.
- **Async patterns**: Promises (JS), async/await (Python), goroutines (Go)
- **Error handling**: Exceptions vs Result types (language preference)
- **Parameter conventions**: Objects vs. kwargs, etc.
- **Framework hooks**: Eg. for React, add `use*` equivalents to `get*`/`list*` methods
- **Everything else, wherever it makes sense**: Every language is unique and the patterns will differ. If you have to decide between what's idiomatic in a language vs. what was done in the Hexclave SDK for other languages, use the idiomatic pattern.

## Implementation Notes

### Object Construction

When constructing SDK objects (User, Team, etc.) from API responses:
1. Map naming conventions to your language's naming convention
2. Objects should hold a reference to the SDK client for making API calls
3. Objects can be mutable or immutable based on language conventions
4. `update()` methods should update local properties after successful API call

### Caching

Normal functions should not cache. Some frameworks, like React, have hooks that require caching; for these, require explicit guidance.

### Pagination

Most `list*` methods support pagination:
- Request with `cursor` and `limit` query params
- Response includes `pagination: { next_cursor?: string }`
- `next_cursor` is null or absent when no more pages
- Default limit is typically 100
- Note that not all backend APIs support pagination, and some just return all items at once.

### Date/Time Formats

- API uses milliseconds since epoch for timestamps (e.g., `signed_up_at_millis`)
- Convert to your language's native Date/DateTime type

---

### Overview (`apps/deployment-gateway/parked-page/README.md`)

# Deployment parked page

The page a stopped deployment serves.

When a project on the Free plan reaches the 24-hour limit, the backend's sweeper
asks Marshal to *park* each of its services (see `parkService` in
`apps/marshal/src/services.ts`). Parking rolls the service's machines onto this
image and leaves everything else about the service alone: same Fly app, same
ports, same public IPs, same certificates, same disks, same stored spec. So this
page answers on the deployment's platform hostname and on every custom domain
attached to it, and unparking is a roll back onto the tenant's own image.

The machine keeps `min_instances: 0`, so it sleeps when idle and Fly Proxy starts
it on the next request. A parked service therefore costs nothing to keep parked.

## What it serves

Every path and every method answers **503** with `Cache-Control: no-store`,
`X-Robots-Tag: noindex` and `X-Hexclave-Deployment-Stopped: <reason>`. HTML goes
to clients that asked for HTML, JSON to clients that asked for JSON, and plain
text to everything else (a wildcard `Accept` is a person at a terminal, not a
browser).

503 rather than 402 or 404 because a custom domain pointed at a parked service
would otherwise be deindexed: 503 is the status search engines treat as
temporary. `Retry-After` is deliberately omitted — a parked deployment comes back
when someone acts, not on a timer.

## Configuration

`PORT` — the service's standard-ports holder, which is what Fly maps 80/443 onto.
Marshal sets it at park time; it is the only per-service value this image takes.

`HEXCLAVE_PARKED_REASON` — which copy to render. `free_plan_24h` today; an
unrecognised value renders a generic page rather than an empty one, so an image
older than a newly added reason still works.

There is deliberately nothing else. The dashboard link is a constant that goes
through the project selector rather than naming a project id, which keeps one
published image serving every parked service in the fleet and keeps the project
id off a page the public can see.

## Publishing

Published as `docker.io/bgodil/deployment-parked-page`. It must stay PUBLIC: tenant Fly
machines pull it with no registry credentials, and a private repository would fail every
park. (Fly's own registry is not an option — its repositories are app-scoped, so an image
pushed to one app's repository is not reliably pullable from another app's machines.)

After any push, confirm an unauthenticated puller can still reach it:

```sh
curl -s "https://hub.docker.com/v2/repositories/bgodil/deployment-parked-page/" | grep -o '"is_private":[a-z]*'
```

```sh
cd apps/deployment-gateway/parked-page
docker buildx build --platform linux/amd64 -t bgodil/deployment-parked-page:2 --push .
docker buildx imagetools inspect bgodil/deployment-parked-page:2
```

Take the digest from that last command and set it on Marshal, which deploys on Vercel
(see [apps/marshal/README.md](../../marshal/README.md)):

```sh
vercel env add HEXCLAVE_DEPLOYMENT_PARKED_IMAGE production
# paste: bgodil/deployment-parked-page@sha256:<digest>
```

The digest currently published as `:2` is
`sha256:0ce015355b8f411367c6db5741fca6efe1e9c7475480f41a73642aeeae06d702`.

`:1` is superseded and must not be used: it ran as a non-root user and could not bind a
privileged port, so any service declaring port 80 crash-looped instead of showing the page.

A Vercel environment variable only reaches the running function on the next deployment, so
redeploy Marshal after setting it. Until then Marshal falls back to the tag in
DEFAULT_PARKED_IMAGE.

Pin the digest rather than the tag. Marshal hands this reference to Fly as the
image a parked machine runs, and a tag that moved under a fleet of already-parked
services would roll every one of them the next time it reconciled.

For the same reason, treat a published tag as immutable once anything is parked against
it: publish changes under the next tag and move the pinned digest deliberately, rather than
pushing over one already in use.

## Tests

`server.test.mjs` runs in the normal workspace suite (the gateway's vitest config
includes this directory by name; its own Docker integration test stays opt-in):

```sh
pnpm test run apps/deployment-gateway
```

To look at the page:

```sh
PORT=8080 node apps/deployment-gateway/parked-page/server.mjs
```

---

### Overview (`examples/convex/convex/README.md`)

# Welcome to your Convex functions directory!

Write your Convex functions here.
See https://docs.convex.dev/functions for more.

A query function that takes two arguments looks like:

```ts
// functions.js
import { query } from "./_generated/server";
import { v } from "convex/values";

export const myQueryFunction = query({
  // Validators for arguments.
  args: {
    first: v.number(),
    second: v.string(),
  },

  // Function implementation.
  handler: async (ctx, args) => {
    // Read the database as many times as you need here.
    // See https://docs.convex.dev/database/reading-data.
    const documents = await ctx.db.query("tablename").collect();

    // Arguments passed from the client are properties of the args object.
    console.log(args.first, args.second);

    // Write arbitrary JavaScript here: filter, aggregate, build derived data,
    // remove non-public properties, or create new objects.
    return documents;
  },
});
```

Using this query function in a React component looks like:

```ts
const data = useQuery(api.functions.myQueryFunction, {
  first: 10,
  second: "hello",
});
```

A mutation function looks like:

```ts
// functions.js
import { mutation } from "./_generated/server";
import { v } from "convex/values";

export const myMutationFunction = mutation({
  // Validators for arguments.
  args: {
    first: v.string(),
    second: v.string(),
  },

  // Function implementation.
  handler: async (ctx, args) => {
    // Insert or modify documents in the database here.
    // Mutations can also read from the database like queries.
    // See https://docs.convex.dev/database/writing-data.
    const message = { body: args.first, author: args.second };
    const id = await ctx.db.insert("messages", message);

    // Optionally, return a value from your mutation.
    return await ctx.db.get(id);
  },
});
```

Using this mutation function in a React component looks like:

```ts
const mutation = useMutation(api.functions.myMutationFunction);
function handleButtonPress() {
  // fire and forget, the most common way to use mutations
  mutation({ first: "Hello!", second: "me" });
  // OR
  // use the result once the mutation has completed
  mutation({ first: "Hello!", second: "me" }).then((result) =>
    console.log(result),
  );
}
```

Use the Convex CLI to push your functions to a deployment. See everything
the Convex CLI can do by running `npx convex -h` in your project root
directory. To learn more, launch the docs with `npx convex docs`.

---

### Overview (`sdks/implementations/swift/README.md`)

# Hexclave Swift SDK

Swift SDK for Hexclave. Supports iOS, macOS, watchOS, tvOS, and visionOS.

## Requirements

- Swift 5.9+
- iOS 15+ / macOS 12+ / watchOS 8+ / tvOS 15+ / visionOS 1+

## Installation

Add to your `Package.swift`:

```swift
dependencies: [
    .package(url: "https://github.com/hexclave/swift-sdk-prerelease", from: <version>)
]
```

## Quick Start

```swift
import StackAuth

let stack = StackClientApp(
    projectId: "your-project-id",
    publishableClientKey: "your-key"
)

// Sign in with email/password
try await stack.signInWithCredential(email: "user@example.com", password: "password")

// Get current user
if let user = try await stack.getUser() {
    print("Signed in as \(user.displayName ?? "Unknown")")
}

// Sign out
try await stack.signOut()
```

## Design Decisions

### Error Handling

All functions that can fail use Swift's native `throws`. Errors conform to `StackAuthError`:

```swift
do {
    try await stack.signInWithCredential(email: email, password: password)
} catch let error as StackAuthError {
    switch error.code {
    case "email_password_mismatch":
        print("Wrong password")
    default:
        print(error.message)
    }
}
```

### Token Storage

- **Default**: Keychain (secure, persists across app launches)
- **Option**: Memory (for testing or ephemeral sessions)
- **Option**: Custom `TokenStoreProtocol` implementation

```swift
// Memory storage (for testing)
let stack = StackClientApp(
    projectId: "...",
    publishableClientKey: "...",
    tokenStore: .memory
)

// Custom storage
let stack = StackClientApp(
    projectId: "...",
    publishableClientKey: "...",
    tokenStore: .custom(MyTokenStore())
)
```

### OAuth Flows

Two approaches for OAuth authentication:

**1. Integrated (recommended)** - Uses `ASWebAuthenticationSession`:

```swift
// Opens auth session, handles callback automatically
// Uses fixed callback scheme: hexclave-mobile-oauth-url://
try await stack.signInWithOAuth(provider: "google")
```

**2. Manual URL handling** - For custom implementations:

> **Note:** The `hexclave-mobile-oauth-url://` scheme is automatically accepted (the legacy `stack-auth-mobile-oauth-url://` scheme also remains accepted for backwards compatibility).

```swift
// Get the OAuth URL (must provide absolute URLs)
let oauth = try await stack.getOAuthUrl(
    provider: "google",
    redirectUrl: "hexclave-mobile-oauth-url://success",
    errorRedirectUrl: "hexclave-mobile-oauth-url://error"
)

// Open oauth.url in your own browser/webview
// Store oauth.state, oauth.codeVerifier, and oauth.redirectUrl

// When callback received:
try await stack.callOAuthCallback(
    url: callbackUrl,
    codeVerifier: oauth.codeVerifier,
    redirectUrl: oauth.redirectUrl
)
```

### Async/Await

All async operations use Swift's native concurrency:

```swift
Task {
    let user = try await stack.getUser()
    let teams = try await user?.listTeams()
}
```

## Key Differences from JavaScript SDK

| Aspect | JavaScript | Swift |
|--------|-----------|-------|
| Token Storage | Cookies | Keychain |
| OAuth | Browser redirect | ASWebAuthenticationSession |
| Redirect methods | Available | Not available (browser-only) |
| React hooks | `useUser()` etc. | Not applicable |

### Not Available in Swift

The following are browser-only and not exposed:

- `redirectToSignIn()`, `redirectToSignUp()`, etc.
- Cookie-based token storage
- `redirectMethod` constructor option

## Examples

Interactive example apps are available for testing all SDK functions:

### macOS Example

```bash
cd Examples/StackAuthMacOS
swift run
```

Features a sidebar-based UI for testing authentication, user management, teams, OAuth, tokens, and server-side operations.

### iOS Example

```bash
cd Examples/StackAuthiOS
open Package.swift  # Opens in Xcode
```

Features a tab-based UI optimized for iOS with the same comprehensive SDK coverage.

Both examples include:
- Configurable API endpoints
- Real-time operation logs
- Error testing scenarios (wrong password, unauthorized access, etc.)
- Client and server app operations

## Testing

Tests use Swift Testing framework against a running backend.

### Running Tests

1. Start the development server:
   ```bash
   pnpm dev
   ```

2. Run tests:
   ```bash
   cd sdks/implementations/swift
   swift test
   ```

The tests connect to `http://localhost:8102` (or `${NEXT_PUBLIC_HEXCLAVE_PORT_PREFIX}02`).

## API Reference

See the [SDK Specification](../../spec/README.md) for complete API documentation.

---

### Overview (`packages/shared/src/config/README.md`)

# Some notes on configs

The language in this file is very technical, if you're struggling, put it into ChatGPT and see if it can help you (with the usual hallucination disclaimer).
 
## Generic format vs. Hexclave

The config format is generally usable and not specific to Hexclave.

All the logic required for generic usage of the config format are in `format/`. The other files in this folder are specific to Hexclave's usage of it.

## Terminology

**Generic config format**:
- Config: Any config, as described in stack-info
- Normalized config: A config without `null` fields and dot notation

**Hexclave**: There are four levels, project, branch, environment, organization.
- Base config: The defaults that come with Hexclave
- `$Level` config override: Overrides that are applied to the base config (in the following order: project -> branch -> environment -> organization)
- `$Level` incomplete config: The base config after some overrides have been applied
- `$Level` rendered config: An incomplete config with those fields removed that can be overridden by a future override, deeply merged into the defaults and sanitized (using `apply{$Level}DefaultsAndSanitize`), and then normalized
- Complete config: The organization rendered config.
- `$Level` config override override: An override that overrides the `$Level` config override. This is most often used eg. in the REST API to let users make changes to the branch-level config, without overwriting the entire branch-level config override. *<sup>Note that, since config overrides (unlike configs) distinguish between `null` and a property missing (`undefined`), it is currently not possible to say "this property in the config override should be unset" (setting a property to `null` in the override override will simply also set it to `null` in the override). In the future, we'll have to think about how we handle this, probably with a sentinel value.</sup>*
- `$Level` config: Could refer to any of the above, depending on the context; if it's not clear, specify it.

<details>
<summary>Examples</summary>

Base config:
```js
{
  organizations: {},
  createTeamOnSignUp: false,
  sourceOfTruthConnectionString: null
}
```

---

Project config override:
```js
{
  sourceOfTruthConnectionString: 'postgresql://...',
}
```

Project incomplete config:
```js
// note: `organizations` and `createTeamOnSignUp` may be overridden by branch, environment, or organization configs! They are not final
{
  organizations: {},
  createTeamOnSignUp: false,
  sourceOfTruthConnectionString: 'postgresql://...',
}
```

Project rendered config:
```js
// since `organizations` and `createTeamOnSignUp` may change later, they are not included in the rendered config
{
  sourceOfTruthConnectionString: 'postgresql://...',
}
```

---

Branch config override:
```js
{
  organizations: {
    'my-org': {
      name: 'My Org',
    }
  }
}
```


Branch incomplete config:
```js
{
  organizations: {
    'my-org': {
      name: 'My Org',
    }
  },
  createTeamOnSignUp: true,
  sourceOfTruthConnectionString: 'postgresql://...',
}
```

Branch rendered config:
```js
// as above, `organizations` and `createTeamOnSignUp` are not included in the rendered config, as they may change later
{
  sourceOfTruthConnectionString: 'postgresql://...',
}
```

---

Environment config override:
```js
// no change from branch config
{}
```

Environment incomplete config:
```js
// no change from branch config
{
  organizations: {
    'my-org': {
      name: 'My Org',
    }
  },
  createTeamOnSignUp: true,
  sourceOfTruthConnectionString: 'postgresql://...',
}
```

Environment rendered config:
```js
// organizations can no longer change after this point, so they are included in the rendered config
{
  organizations: {
    'my-org': {
      name: 'My Org',
    }
  },
  createTeamOnSignUp: true,
  sourceOfTruthConnectionString: 'postgresql://...',
}
```

---

Organization config override:
```js
{
  createTeamOnSignUp: true,
}
```

Organization incomplete config = organization rendered config = complete config:
```js
{
  createTeamOnSignUp: true,
  sourceOfTruthConnectionString: 'postgresql://...',
}
```


</details>

---

### Overview (`sdks/implementations/swift/Examples/StackAuthMacOS/README.md`)

# Hexclave macOS Example

A comprehensive macOS SwiftUI application for testing all Hexclave SDK functions interactively.

## Prerequisites

- macOS 14.0+
- Swift 5.9+
- A running Hexclave backend (default: `http://localhost:8102`)

## Running the Example

1. Start the Hexclave backend:
   ```bash
   cd /path/to/stack-2
   pnpm run dev
   ```

2. Open and run the example:
   ```bash
   cd Examples/StackAuthMacOS
   swift run
   ```

   Or open in Xcode:
   ```bash
   open Package.swift
   ```

## Features

The example app provides a sidebar navigation with the following sections:

### Configuration
- **Settings**: Configure API base URL, project ID, and API keys
- **Logs**: View real-time logs of all SDK operations

### Client App Testing
- **Authentication**
  - Sign up with email/password
  - Sign in with credentials
  - Sign in with wrong password (error testing)
  - Sign out
  - Get current user
  - Get user (or throw)

- **User Management**
  - Set display name
  - Update client metadata
  - Update password
  - Get access/refresh tokens
  - Get auth headers
  - Get partial user from token

- **Teams**
  - Create team
  - List user's teams
  - Get team by ID
  - List team members

- **Contact Channels**
  - List contact channels

- **OAuth**
  - Generate OAuth URLs for Google, GitHub, Microsoft
  - Test PKCE code generation

- **Tokens**
  - Get access token (JWT format)
  - Get refresh token
  - Get auth headers
  - Test different token stores

### Server App Testing
- **Server Users**
  - Create user (basic and with all options)
  - List users with pagination
  - Get user by ID
  - Delete user

- **Server Teams**
  - Create team
  - List all teams
  - Add/remove users from teams
  - List team users
  - Delete team

- **Sessions**
  - Create session (impersonation)
  - Use session tokens with client app

## Default Configuration

The example is pre-configured for local development:
- Base URL: `http://localhost:8102`
- Project ID: `internal`
- Publishable Key: `this-publishable-client-key-is-for-local-development-only`
- Secret Key: `this-secret-server-key-is-for-local-development-only`

## SDK Functions Covered

| Category | Functions |
|----------|-----------|
| Auth | signUpWithCredential, signInWithCredential, signOut, getUser, getOAuthUrl |
| User | setDisplayName, update (metadata), updatePassword, getAccessToken, getRefreshToken, getAuthHeaders, getPartialUser |
| Teams | createTeam, listTeams, getTeam, listUsers (team members) |
| Contact | listContactChannels |
| Server Users | createUser, listUsers, getUser, delete, update (metadata, password) |
| Server Teams | createTeam, listTeams, getTeam, addUser, removeUser, listUsers, delete |
| Sessions | createSession |
| Errors | EmailPasswordMismatchError, UserNotSignedInError, PasswordConfirmationMismatchError |

---

### Overview (`sdks/implementations/swift/Examples/StackAuthiOS/README.md`)

# Hexclave iOS Example

An interactive iOS application for testing all Hexclave Swift SDK functions.

## Prerequisites

- Xcode 15.0 or later
- iOS 17.0+ Simulator or device
- Running Hexclave backend (default: `http://localhost:8102`)

## Running the Example

### Option 1: Xcode

1. Open the project in Xcode:
   ```bash
   open StackAuthiOS.xcodeproj
   ```

2. Select an iOS Simulator (e.g., "iPhone 15 Pro" or any available device) as the destination

3. Press ⌘R to build and run

### Option 2: Command Line

```bash
# Build (replace device name with an available simulator on your system)
xcodebuild -scheme StackAuthiOS -destination 'platform=iOS Simulator,name=iPhone 15 Pro' build

# Build and run (opens simulator)
xcodebuild -scheme StackAuthiOS -destination 'platform=iOS Simulator,name=iPhone 15 Pro' run
```

## Features

The app uses a tab-based interface optimized for mobile:

- **Settings**: Configure API endpoint, project ID, and keys
- **Auth**: Sign up, sign in, sign out, get current user
- **User**: Update display name, metadata, view tokens
- **Teams**: Create, list, and manage teams
- **Logs**: View all SDK calls with full details (tap for more, long-press to copy)

Additional functions are accessible via navigation links in Settings:
- Contact Channels
- OAuth URL generation
- Token operations
- Server Users (admin)
- Server Teams (admin)
- Sessions (impersonation)

## SDK Functions Covered

### Client App
- `signUpWithCredential(email:password:)`
- `signInWithCredential(email:password:)`
- `signOut()`
- `getUser()` / `getUser(or:)`
- `getAccessToken()` / `getRefreshToken()`
- `getAuthHeaders()`
- `getOAuthUrl(provider:)`

### Current User
- `setDisplayName(_:)`
- `update(clientMetadata:)`
- `listTeams()` / `getTeam(id:)`
- `createTeam(displayName:)`
- `listContactChannels()`

### Server App
- `createUser(email:password:...)`
- `listUsers(limit:)`
- `getUser(id:)`
- `createTeam(displayName:)`
- `listTeams()`
- `createSession(userId:)`

## Logging

The Logs tab shows all SDK activity in real-time:
- **Green checkmark**: Successful calls with full response data
- **Red X**: Errors with details
- **Blue info**: In-progress calls

Tap any log entry to see full details. Long-press to copy to clipboard.

## Network Configuration

For iOS Simulator to connect to your local backend:

1. The default `localhost:8102` should work in the simulator
2. For a real device, use your computer's local IP address instead

## Troubleshooting

### "Could not connect to server"
- Ensure your Hexclave backend is running
- Check the Base URL in Settings tab
- For real devices, use your computer's IP instead of localhost

### Build errors
- Make sure you have Xcode 15+ installed
- Try cleaning: Product → Clean Build Folder (⇧⌘K)

---

### Overview (`apps/backend/src/app/api/migrations/README.md`)

# API migrations

First, make sure you have a good reason to do an API migration. While they're relatively low-effort in our codebase, most changes don't break backwards compatibility, so it might not be required.

Examples of changes that are breaking and hence require an API migration:

- You are adding a required field to a request, or narrowing the allowed values of an existing request field
- You are removing a field of a response, or widening the allowed values for a response field
- You are renaming a field or endpoint
- You are removing an endpoint entirely
- The behavior changes in other ways that might affect clients
- ...

Release versions (eg. v1, v2) are documented thoroughly, while beta versions (eg. v2beta1, v3beta2) are mostly used by our own packages/SDKs and some external beta testers. **We still need to maintain backwards compatibility for beta versions**, so the only purpose of differentiating is to prevent "migration fatigue" if we were to announce a new API version every week. Beta versions come before release versions: `v1 < v2beta1 < v2beta2 < v2`, etc.

Each folder in `src/app/api/migrations` is a migration. The name of the folder is the name of the version you're migrating **to** — so, if you're migrating from `v2beta3` to `v2beta4`, the folder is called `v2beta4`. In other words, the files in `v2beta4` will process all requests for versions LESS than `v2beta4` (but not `v2beta4` itself).

To create a new migration, simply add a new folder in `src/app/api/migrations`. This folder has the same structure as `src/app/api/latest`, although it will fall back to that folder for routes that are not found. Additionally, this new folder should contain extra files: `beta-changes.txt` (the list of changes since the last beta version), and `release-changes.txt` (the list of changes since the last release version — only required for release versions). For every endpoint you migrate, you will likely also have to modify the most recent migration of that endpoint in previous versions (if any) to call your newly created endpoint, instead of the one that can be found in `latest`.

To understand the flow of old migrations, imagine a request for a `v2` endpoint. Instead of looking for a Next.js file in the `src/app/api` folder directly, the middleware will instead rewrite the request to `src/app/api/migrations/v2beta1`. If not found, it will check `v2beta2`, and so on. If no migration strictly newer than the requested version is found, it will return the route from `src/app/api/latest`.

---

### Overview (`packages/template/src/integrations/convex/component/README.md`)

# Hexclave Convex Component

This component is the official way to integrate Hexclave with your Convex project.

## Installation

To get started, follow the Hexclave setup prompt and choose Convex for the integration steps:

https://docs.hexclave.com/guides/getting-started/setup

## Get Started

[Create a new Hexclave project](https://app.hexclave.com) and set the environment variables in Convex to the project ID & API key environment variables from the Hexclave dashboard. Also, add the same values to the `.env.local` file.

Next, update or create a file in `convex/auth.config.ts`:

```ts
import { getConvexProvidersConfig } from "@hexclave/js/convex-auth.config";  // Vanilla JS
// or: import { getConvexProvidersConfig } from "@hexclave/react/convex-auth.config";  // React
// or: import { getConvexProvidersConfig } from "@hexclave/next/convex-auth.config";  // Next.js

export default {
  providers: getConvexProvidersConfig({
    projectId: process.env.HEXCLAVE_PROJECT_ID,  // or: process.env.NEXT_PUBLIC_HEXCLAVE_PROJECT_ID
  }),
}
```

Next, update or create a file in `convex/convex.config.ts`:

```ts
import { defineApp } from "convex/server";
import hexclaveComponent from "@hexclave/js/convex.config";  // Vanilla JS
// or: import hexclaveComponent from "@hexclave/react/convex.config";  // React
// or: import hexclaveComponent from "@hexclave/next/convex.config";  // Next.js


const app = defineApp();
app.use(hexclaveComponent);

export default app;
```

Then, update your Convex client to use Hexclave:

```ts
convexClient.setAuth(hexclaveClientApp.getConvexClientAuth({}));  // browser JS
convexReactClient.setAuth(hexclaveClientApp.getConvexClientAuth({}));  // React
convexHttpClient.setAuth(hexclaveClientApp.getConvexHttpClientAuth({ tokenStore: requestObject }));  // HTTP, see Hexclave docs for more information on tokenStore
```

Now, you'll be able to access Hexclave's functionality from your frontend & backend:

```ts
// MyPage.tsx
export function MyPage() {
  // see https://docs.hexclave.com for more information on how to use Hexclave
  const user = useUser();
  return <div>Your email is {user.email}</div>;
}

// myFunctions.ts
export const myQuery = query({
  handler: async (ctx, args) => {
    // In queries & mutations, use the special `getPartialUser` function to get user info
    const obj = await hexclaveServerApp.getPartialUser({ from: "convex", ctx });
    return JSON.stringify(obj);
  },
});
```

For more information on how to use Hexclave, see the [Hexclave docs](https://docs.hexclave.com).

---

### Overview (`apps/backend/src/app/api/latest/contact-channels/README.md`)

# Primary Email and Contact Channel Logic

This document explains the relationship between `primary_email` on user objects and contact channels, including all edge cases and expected behaviors.

## Overview

In Hexclave, users can have multiple **contact channels** (currently only email type is supported). One of these can be designated as the **primary email**, which is the main email address associated with the user account.

### Key Concepts

- **Contact Channel**: A communication endpoint (email) associated with a user
- **Primary Email**: The main email address for a user (reflected in `user.primary_email`)
- **Verified**: Whether the email ownership has been confirmed
- **Used for Auth**: Whether the email can be used to sign in

### Data Model

```
User
├── primary_email (derived from primary contact channel)
├── primary_email_verified (derived from primary contact channel)
├── primary_email_auth_enabled (derived from primary contact channel)
└── Contact Channels[]
    ├── value (email address)
    ├── type ("email")
    ├── is_primary (boolean)
    ├── is_verified (boolean)
    └── used_for_auth (boolean)
```

The `primary_email`, `primary_email_verified`, and `primary_email_auth_enabled` fields on the user object are **derived** from the contact channel where `is_primary = true`.

## Updating Primary Email via User Endpoint

When updating `primary_email` via `PATCH /api/v1/users/{user_id}` or `PATCH /api/v1/users/me`:

### Scenario 1: Setting primary_email when user has no email

**Behavior**: Creates a new contact channel marked as primary and unverified.

**Test**: [`users-primary-email.test.ts` - "should be able to set primary_email when user has no email"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

### Scenario 2: Changing to a new email address

**Behavior**:
1. The old primary email becomes a non-primary contact channel (demoted)
2. A new contact channel is created marked as primary
3. The new email is **always unverified** (even if the old one was verified)

**Tests**:
- [`users-primary-email.test.ts` - "should be able to change primary_email to a new email"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)
- [`users-primary-email.test.ts` - "should set new email as unverified even if old email was verified"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)
- [`users-primary-email.test.ts` - "old primary email should become non-primary contact channel after change"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

### Scenario 3: Changing to an existing non-primary contact channel

**Behavior**:
1. The old primary email is demoted to non-primary
2. The existing contact channel is **upgraded** to primary
3. The verification status is **preserved** (if it was verified, it stays verified)

**Tests**:
- [`users-primary-email.test.ts` - "should be able to change primary_email to an existing non-primary contact channel"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)
- [`users-primary-email.test.ts` - "should preserve verification status when switching to existing verified contact channel"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)
- [`users-primary-email.test.ts` - "should preserve unverified status when switching to existing unverified contact channel"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

### Scenario 4: Setting primary_email to null

**Behavior**: The primary contact channel is **demoted** to non-primary (not deleted). The contact channel remains on the user but is no longer the primary email.

**Note**: This means setting `primary_email` to a value and then to `null` is NOT a no-op—it leaves a non-primary contact channel behind.

**Test**: [`users-primary-email.test.ts` - "should be able to set primary_email to null (demotes to non-primary, does not delete)"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

### Scenario 5: Setting primary_email to the same value (no-op)

**Behavior**: No changes are made; verification status is preserved.

**Test**: [`users-primary-email.test.ts` - "should handle setting primary_email to the same email (no-op)"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

## Server-Side Additional Fields

When using server access, you can also set:

- `primary_email_verified`: Set the verification status directly
- `primary_email_auth_enabled`: Enable/disable using this email for authentication

**Tests**:
- [`users-primary-email.test.ts` - "should be able to set primary_email and primary_email_verified together"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)
- [`users-primary-email.test.ts` - "should be able to set primary_email_auth_enabled when changing primary_email"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

## Email Uniqueness Rules

### Uniqueness for `used_for_auth`

Emails that are `used_for_auth = true` must be unique across all users in a project. Two users cannot have the same email as an auth-enabled email.

**Test**: [`users-primary-email.test.ts` - "should not be able to set primary_email to email already used by another user with used_for_auth"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

### Non-auth emails

Emails that are NOT used for auth (`used_for_auth = false`) can be shared across users.

### Duplicate Contact Channels

A user cannot have two contact channels with the same email address. Attempting to create a duplicate will fail.

**Test**: [`contact-channels.test.ts` - "cannot create duplicate contact channels"](../../../../../../e2e/tests/backend/endpoints/api/v1/contact-channels/contact-channels.test.ts)

## Validation Rules

### Invalid Email Format

Emails must be valid email format. Invalid formats return 400 error.

**Test**: [`users-primary-email.test.ts` - "should reject invalid email format"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

### Empty String

Empty string is not a valid email. Returns 400 error.

**Test**: [`users-primary-email.test.ts` - "should reject empty string as email"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

### Case Insensitivity

Email addresses are treated case-insensitively for comparison purposes.

**Test**: [`users-primary-email.test.ts` - "should handle case insensitivity in email addresses"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

## Integration with Pending User Status

When `onboarding.requireEmailVerification` is enabled in project config:

### User becomes pending when:

1. Setting an unverified email as primary
2. Changing from a verified email to a new (unverified) email

**Tests**:
- [`users-primary-email.test.ts` - "should make user pending when setting unverified email with requireEmailVerification enabled"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)
- [`users-primary-email.test.ts` - "changing from verified to unverified email should make user pending"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

### User does NOT become pending when:

1. Switching to an already-verified existing contact channel

**Test**: [`users-primary-email.test.ts` - "switching to verified existing contact channel should not make user pending"](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts)

### Client-side Guardrails

The account settings UI prevents users from removing their primary verified email when `requireEmailVerification` is enabled, to avoid accidentally putting themselves in pending status.

**Note**: No E2E test currently covers this client-side behavior specifically.

## Contact Channel Endpoint Operations

### Creating Contact Channels

`POST /api/v1/contact-channels`

- New contact channels are **not primary** by default
- Client-created channels are **unverified** by default
- Server-created channels can set `is_verified: true`

**Tests**:
- [`contact-channels.test.ts` - "create contact channel on the client"](../../../../../../e2e/tests/backend/endpoints/api/v1/contact-channels/contact-channels.test.ts)
- [`contact-channels.test.ts` - "create contact channel on the server"](../../../../../../e2e/tests/backend/endpoints/api/v1/contact-channels/contact-channels.test.ts)

### Updating Contact Channel to Primary

`PATCH /api/v1/contact-channels/{user_id}/{channel_id}` with `is_primary: true`

- The previous primary channel (if any) is automatically demoted
- This is an alternative to updating `primary_email` on the user object

**Test**: [`contact-channels.test.ts` - "updates contact channel primary status"](../../../../../../e2e/tests/backend/endpoints/api/v1/contact-channels/contact-channels.test.ts)

### Setting Primary to Non-Primary

`PATCH /api/v1/contact-channels/{user_id}/{channel_id}` with `is_primary: false`

- Sets the channel to non-primary
- The user's `primary_email` becomes `null`

**Test**: [`contact-channels.test.ts` - "sets a primary contact channel to non-primary"](../../../../../../e2e/tests/backend/endpoints/api/v1/contact-channels/contact-channels.test.ts)

### Deleting Contact Channels

`DELETE /api/v1/contact-channels/{user_id}/{channel_id}`

- If deleting the primary contact channel, `primary_email` becomes `null`
- Associated auth methods are also deleted

**Test**: [`contact-channels.test.ts` - "delete contact channel on the client"](../../../../../../e2e/tests/backend/endpoints/api/v1/contact-channels/contact-channels.test.ts)

## Related Files

### Backend Logic
- [`../users/crud.tsx`](../users/crud.tsx) - User CRUD operations including primary email update logic
- [`./crud.tsx`](./crud.tsx) - Contact channel CRUD operations

### Client SDK
- [`packages/template/src/lib/stack-app/users/index.ts`](../../../../../../../packages/template/src/lib/stack-app/users/index.ts) - User types and update options

### E2E Tests
- [`users-primary-email.test.ts`](../../../../../../e2e/tests/backend/endpoints/api/v1/users-primary-email.test.ts) - Primary email update tests
- [`contact-channels.test.ts`](../../../../../../e2e/tests/backend/endpoints/api/v1/contact-channels/contact-channels.test.ts) - Contact channel tests
- [`pending-users.test.ts`](../../../../../../e2e/tests/backend/endpoints/api/v1/pending-users.test.ts) - Pending user status tests

---

### Overview (`apps/backend/src/app/api/latest/emails/README.md`)

# Email Infrastructure Overview

This folder contains the HTTP endpoints that sit on top of the new email outbox
pipeline. The pipeline is intentionally asynchronous: instead of sending mail
inside request handlers we persist work items to the `EmailOutbox` table and let
background workers render, queue, and deliver them.

## Execution Flow

1. **Enqueue** – API endpoints (and server-side helpers) call
   `sendEmailToMany` to persist one row per recipient. Each entry
   captures the template source, render variables, target recipient, priority,
   and scheduling metadata.
2. **Render** – `runEmailQueueStep` atomically
   claims rows that have not been rendered. Emails are rendered via Freestyle,
   producing HTML/Text/Subject snapshots while capturing
   render errors in structured fields.
3. **Queue** – Rendered rows whose `scheduled_at` is in the past are marked as
   ready (`isQueued = true`). Capacity is calculated per tenancy based on recent
   delivery performance to decide how many emails can be handed off to the
   sender during this iteration.
4. **Send** – Claimed rows are processed in parallel. Before delivery we fetch
   the latest user data, honour notification preferences and skip users who have
   unsubscribed or deleted their account. Provider responses are captured in the
   `sendServerError*` fields so the dashboard can surface actionable feedback.
5. **Delivery Stats** – The worker updates `EmailOutboxProcessingMetadata` so we
   can derive execution deltas and expose aggregated metrics via the
   `/emails/delivery-info` endpoint.

## Key Tables

- `EmailOutbox` – Durable queue of emails with full status history and audit
  data. Constraints ensure mutually exclusive sets of render/send error fields
  and guard against race conditions.
- `EmailOutboxProcessingMetadata` – Stores the last worker execution timestamp
  so we can compute accurate capacity budgets each run.

## Mutable vs. Immutable States

Emails can only be edited, paused, retried, or deleted **before** `startedSendingAt` is set.
Once sending begins, the entry becomes read-only. Retrying an email effectively resets its
place in the pipeline & queue, see the Prisma schema for more details.

---

--- METRICS ---
- Files Extracted: 26
- Estimated Token Budget: ~29974 tokens
- Canonical Reference: https://codewiki.google/github.com/hexclave/stack-auth
