## File: crates/openshell-providers/README.md # openshell-providers Provider discovery and normalization for credentials that sandboxes need at runtime. The gateway persists provider records. The sandbox supervisor fetches resolved provider environment from the gateway and injects credentials into agent child processes. This crate keeps provider-specific discovery and normalization logic out of the CLI and gateway control flow. ## Responsibilities - Discover local credentials from environment variables and known config files. - Normalize discovered data into provider records. - Keep provider-specific parsing rules in provider modules. - Use the shared authored-YAML compatibility boundary for profile imports and exports: reject duplicate keys and null objects without coercion, retain optional nulls, and preserve string types for YAML 1.1 readers. - Avoid logging credential values. ## Non-Responsibilities - Persisting provider records. - Authorizing provider CRUD operations. - Injecting credentials into sandbox child processes. - Routing inference requests. Those are owned by the gateway, sandbox supervisor, and router. ## Security Notes Provider data often contains API keys, bearer tokens, or local account configuration. Discovery code should return structured values without printing or tracing secrets. Callers that display provider data must redact sensitive fields by default. --- ## File: crates/openshell-prover/README.md # openshell-prover Formal verifier for OpenShell sandbox policies. Encodes a policy + its attached credential set + a binary capability registry as a Z3 SMT model, then runs reachability queries to detect credentialed-reach and capability changes a reviewer should be aware of. The crate also exposes an independent containment API for checking whether a fully composed candidate policy stays within an operator-supplied boundary. The `openshell-prover-cli` package wraps that API for local files. Containment and the legacy proposal-risk queries answer different questions; gateway callers continue to use the proposal-risk API until the managed-policy migration. Both prover entrypoints use the bounded `openshell-policy-schema` parser. Containment rejects unknown fields and managed `metadata`/`review` annotations as invalid input, then checks schema-valid controls against its supported model. It uses shared filesystem defaults, port selection, and access-preset semantics. The containment model accepts ASCII literals in network binary selectors, endpoint host and path selectors, and REST allow and deny method and path selectors. It returns `unsupported_policy_shape` when either policy uses a non-ASCII literal in one of those fields. This boundary does not apply to filesystem paths or unrelated policy text. Embedded NUL bytes in network selector fields are also unsupported. ASCII wildcards are modeled over the runtime match language and can therefore match non-ASCII runtime values. A solver string that cannot be decoded and validated exactly produces `invalid_witness` rather than counterexample evidence. ## Containment API compatibility The containment module is a reusable Rust API, but this contract does not make the entire `openshell-prover` crate a stable published SDK. Construct options through `CheckOptions::new`, and accept a policy only when the result is `CheckResult::Within`: ```rust use std::time::Duration; use openshell_prover::containment::{ CheckOptions, CheckResult, check_within_boundary, parse_policy_str, }; let boundary = parse_policy_str("version: 1\n")?; let candidate = parse_policy_str("version: 1\n")?; let result = check_within_boundary( &boundary, &candidate, CheckOptions::new(Duration::from_secs(10)), ); let authorized = matches!(result, CheckResult::Within(_)); # Ok::<(), Box>(()) ``` `ReasonCode`, `CheckDomain`, `Protocol`, and `Counterexample` are open to new variants. `CheckOptions`, `CheckCoverage`, `WithinEvidence`, and existing counterexample variants are open to new fields. Match these types with `..` and wildcard arms, use their accessors and `as_str()` identifiers, and treat unknown values as a fail-closed result. Existing identifier strings are stable. `CheckResult` is intentionally closed to the four states `Within`, `Exceeds`, `Unsupported`, and `Inconclusive`. `FilesystemAccess` is likewise closed to `Read` and `Write`. Adding a result state or filesystem access mode is a breaking API change. Rust source compatibility is separate from the CLI JSON schema contract. The CLI's numeric `schema_version` versions that JSON contract, while `prover_version` identifies the implementation that produced a result. The result's `coverage.domains` list is the machine-readable declaration of modeled authority. Used by the gateway to gate auto-approval of agent-authored policy proposals: any finding blocks auto-approval, an empty delta lets the chunk pass through (when the reviewer opts in via the `proposal_approval_mode` setting at either gateway or sandbox scope). ## What it decides The prover answers four formal questions. Each "yes" answer is its own categorical finding — there is no severity grade. The categories live in [`finding::category`](src/finding.rs). | Category | Question the prover decides | |---|---| | `link_local_reach` | Does this policy grant reach to a host in `169.254.0.0/16` or `fe80::/10`? | | `l7_bypass_credentialed` | Does it let a binary using a non-HTTP wire protocol (per the binary registry's `bypasses_l7` flag) reach a host where a credential is in scope? | | `credential_reach_expansion` | Does it let a binary reach a (host, port) with a credential in scope, where the binary couldn't reach that endpoint before? | | `capability_expansion` | On a (binary, host, port) the binary already reaches with credentials, does it add a new HTTP method? | The first two are unconditional risks. The latter two are *delta* properties — the gateway runs the prover on both the baseline policy and the merged policy and surfaces only the new paths. ## Evidence shape Each finding carries one or more [`FindingPath::Exfil`](src/finding.rs) entries: ```rust pub struct ExfilPath { pub binary: String, pub endpoint_host: String, pub endpoint_port: u16, pub mechanism: String, // human-readable description pub policy_name: String, // rule the path traverses pub category: String, // one of the category constants pub method: String, // populated for capability_expansion; empty otherwise } ``` The gateway's `finding_delta` keys paths by `(finding query, binary, host:port, path category, method)` so that adding a new method on an already-reached host surfaces as exactly one new path (not the whole re-emission of the existing method set). ### Category suppression at the delta layer `capability_expansion` paths whose `(binary, host, port)` tuple is also in the `credential_reach_expansion` delta are suppressed by the gateway. A brand-new credentialed reach is described by the reach-expansion finding alone, not also by N per-method findings. ## Adding a new category 1. Add a constant to `src/finding.rs::category`. 2. In `src/queries.rs::check_credential_safety`, add the branch that detects the new category and emits one `ExfilPath` per evidence row. Set `path.category` to the new constant. 3. In `src/report.rs::format_path_line`, add a `match` arm rendering the per-path display string the reviewer sees. 4. (Gateway) If the new category should be suppressed by another, add the suppression rule to `crates/openshell-server/src/grpc/policy.rs::finding_delta`. 5. Add a unit test in `src/queries.rs` and an integration test in `crates/openshell-server/src/grpc/policy.rs::tests`. The four v1 categories cover the formal properties the OpenShell auto-approval gate cares about today. Additional categories (e.g., "destructive method introduced," "new outbound TLS without SNI") would be additive — they don't displace existing categories. ## What the prover does *not* decide - **Semantic risk of an action.** The prover models *can the binary do this?*, not *is this destructive?*. `PUT /repos/.../contents/file.md` and `GET /repos/.../contents/file.md` are both authenticated actions; the reviewer (or a downstream layer like an LLM contextual reviewer or an intent file) decides if the action is desired. - **Cross-sandbox or cross-binary intent.** The model is per-sandbox. If two sandboxes share a credential through external policy, the prover reasons about each independently. - **Runtime behavior.** The prover analyzes the policy as written; it doesn't observe the proxy's actual decisions. The proxy is the enforcement layer; the prover is the change-review layer. ## Inputs - **Policy** — authored YAML decoded by `openshell-policy-schema` with the shared fail-closed parser, then projected into the prover-only `PolicyModel`. Missing `version` and unknown authored fields are rejected consistently with runtime parsing. An absent `filesystem_policy` uses the runtime-effective `include_workdir: true`; an explicitly present empty object remains `false`. Explicit `protocol: tcp` is treated as L4, matching runtime classification. - **Credential set** — built from the sandbox's attached providers in `crates/openshell-server/src/grpc/policy.rs::build_credential_set_for_sandbox`. v1 captures presence only (host-coarse); no scope modeling. - **Binary registry** — YAML descriptors at `crates/openshell-prover/registry/binaries/*.yaml`. Each describes the binary's protocols, `bypasses_l7` flag, and `can_exfiltrate` capability. Accepted-risk files, credential descriptors, and binary registries are separate file formats. They continue to use their own serde definitions and do not form part of the authored policy schema. ## Outputs - A list of `Finding` values, one per fired category. Each finding's `query` field holds the category name. - The legacy report renderers (`report::render_compact` / `render_report`) format proposal-risk findings for terminal consumers. - The gateway calls `report::finding_shorthand` to build the `validation_result` string persisted on each draft chunk. - The containment API returns typed evidence to `openshell-prover-cli`, which owns the standalone command's text and JSON formats and exit codes. ## Z3 model layout See `src/model.rs`. Briefly: - Endpoint facts are indexed by rule, endpoint position, and effective port. Sibling endpoints retain their own grants even when their destinations overlap. - Query entrypoints return an error if the base constraints are inconsistent or the solver cannot reach a conclusion. Callers must propagate that error; it cannot be accepted as a finding or compared as an empty finding list. - Bool sorts per `(binary, endpoint)` pair encode policy reachability, filtered by binary capability flags (`can_exfiltrate`, `bypasses_l7`). - Bool sorts per `(binary, host)` encode credential-in-scope (one credential set per sandbox). - The reachability formula composes these into the SAT query the `queries::check_credential_safety` loop iterates over. ## Tests - Unit tests in each module (`src/queries.rs`, `src/report.rs`, `src/policy.rs`) cover individual primitives and category emission. - Integration tests in `src/lib.rs::tests` exercise the full parse → build_model → run_all_queries pipeline against testdata policies in `testdata/`. - Gateway-level acceptance tests in `crates/openshell-server/src/grpc/policy.rs::tests` lock in the end-to-end `validation_result` shape and the auto-approval gate. --- ## File: crates/openshell-ocsf/schemas/ocsf/README.md # Vendored OCSF Schemas These schemas are vendored from the [OCSF Schema Server](https://schema.ocsf.io/) for offline test validation. ## Version - **OCSF v1.8.0** — fetched from `https://schema.ocsf.io/api/1.8.0/` ## Contents ### Classes (10) - `network_activity` [4001] - `http_activity` [4002] - `ssh_activity` [4007] - `process_activity` [1007] - `detection_finding` [2004] - `application_lifecycle` [6002] - `device_config_state_change` [5019] - `api_activity` [6003] - `authentication` [3002] - `base_event` [0] ### Objects (22) - `metadata`, `network_endpoint`, `network_proxy`, `process`, `actor` - `device`, `container`, `product`, `firewall_rule`, `finding_info` - `evidences`, `http_request`, `http_response`, `url`, `attack` - `remediation`, `connection_info`, `ai_model`, `api`, `user`, `service`, `os` ### Profiles (1) - `ai_operation` ## Updating To update to a new OCSF version: ```bash VERSION=1.8.0 for class in network_activity http_activity ssh_activity process_activity \ detection_finding application_lifecycle device_config_state_change \ api_activity authentication base_event; do curl -s "https://schema.ocsf.io/api/${VERSION}/classes/${class}" \ | python3 -m json.tool > "classes/${class}.json" done for object in metadata network_endpoint network_proxy process actor device \ container product firewall_rule finding_info evidences \ http_request http_response url attack remediation connection_info \ ai_model api user service os; do curl -s "https://schema.ocsf.io/api/${VERSION}/objects/${object}" \ | python3 -m json.tool > "objects/${object}.json" done echo "${VERSION}" > VERSION ``` Then update `OCSF_VERSION` in `crates/openshell-ocsf/src/lib.rs` to match. --- ## File: crates/openshell-driver-vm/runtime/README.md # VM Driver Runtime > Status: Experimental. VM support is under active development and may change. This directory owns the pinned runtime inputs for `openshell-driver-vm`: ```text runtime/ pins.env kernel/ openshell.kconfig ``` `openshell-driver-vm` embeds libkrun, libkrunfw, umoci for guest-side OCI image unpacking, and the portable capability-free `openshell-sandbox` role. VMs do not attach a guest NIC. The boundary carries control, mediated network, and DNS streams over the authenticated vsock channel. ## Why The stock `libkrunfw` kernel does not include every cgroup, seccomp, and Landlock feature the sandbox needs inside each microVM. `kernel/openshell.kconfig` extends the libkrunfw kernel so VM sandboxes retain guest-local process, network-syscall, and filesystem enforcement while the supervisor runs on the host. ## Build Scripts | Script | Platform | Purpose | |---|---|---| | `tasks/scripts/vm/build-libkrun.sh` | Linux | Builds libkrunfw and libkrun from source with the custom kernel config | | `tasks/scripts/vm/build-libkrun-macos.sh` | macOS | Builds portable libkrunfw and libkrun from a prebuilt `kernel.c` | | `tasks/scripts/vm/package-vm-runtime.sh` | Any | Packages `vm-runtime-.tar.zst` with libraries, umoci, and provenance | | `tasks/scripts/vm/download-kernel-runtime.sh` | Any | Downloads runtime tarballs from the `vm-runtime` release and stages compressed files | ## Local Flow ```shell # Download the current pre-built runtime and stage compressed artifacts mise run vm:setup # Build the portable Linux guest sandbox and static guest-init helper mise run vm:supervisor # Build the gateway, native host supervisor, and VM driver OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \ cargo build -p openshell-gateway -p openshell-supervisor -p openshell-driver-vm ``` Use `FROM_SOURCE=1 mise run vm:setup` to build the runtime from source instead of downloading `vm-runtime-.tar.zst`. ## CI Ownership `release-vm-kernel.yml` is the on-demand producer for: - `vm-runtime-linux-aarch64.tar.zst` - `vm-runtime-linux-x86_64.tar.zst` - `vm-runtime-darwin-aarch64.tar.zst` Those artifacts stay on the rolling `vm-runtime` release. Normal `dev` and `v*` release workflows download them, embed them into `openshell-driver-vm`, and publish the driver binary next to `openshell-gateway`. ## Provenance `package-vm-runtime.sh` writes `provenance.json` into each runtime tarball with the platform, libkrunfw commit, kernel version, and umoci version, GitHub SHA, and build time. The driver logs this metadata when it extracts and loads a runtime bundle. The release workflow also publishes GitHub artifact attestations for each runtime tarball. Verify a downloaded runtime with: ```bash gh attestation verify vm-runtime-linux-x86_64.tar.zst -R NVIDIA/OpenShell ``` --- ## File: crates/openshell-driver-vm/README.md # openshell-driver-vm > Status: Experimental. The VM compute driver is under active development. Standalone libkrun-backed [`ComputeDriver`](../../proto/compute_driver.proto) for OpenShell. The gateway spawns this binary as a subprocess and talks to it over the `openshell.compute.v1.ComputeDriver` Unix-socket surface. `openshell-supervisor` runs as a native host process, while `openshell-sandbox` runs as capability-free PID 1 inside each microVM and applies guest-local isolation over virtio-vsock. The driver embeds libkrun, libkrunfw, the guest OCI unpacker, the portable guest sandbox, and the custom kernel runtime. Each sandbox boots from a cached immutable bootstrap ext4 root disk plus a per-sandbox writable overlay disk. When the requested sandbox image differs from the bootstrap image, the driver prepares a read-only image ext4 disk inside a bootstrap VM and mounts that unpacked rootfs as the sandbox lowerdir. ## How it fits together Caller driver config is disabled by default, including upload workflows encoded in driver JSON. GPU devices and their trusted VFIO plumbing are temporarily exempt from approval labels; existing GPU selection validation still applies. Private rootfs staging does not authorize arbitrary host paths. The gateway passes its common policy to its managed VM subprocess. See [resource admission configuration](../../docs/how-it-works/gateways/configuration.mdx#external-resource-admission). ```mermaid flowchart LR subgraph host["Host"] gateway["openshell-gateway
(vm::spawn)"] driver["openshell-driver-vm
libkrun"] supervisor["openshell-supervisor
host policy supervisor"] gateway <-->|"gRPC over UDS
compute-driver.sock"| driver supervisor <-->|"authenticated gRPC
policy + relay"| gateway end subgraph guest["Per-sandbox microVM"] init["/srv/openshell-vm-
sandbox-init.sh"] sandbox["openshell-sandbox
capability-free guest PID 1"] workload["sandbox workload"] init --> sandbox --> workload end driver -->|"CreateSandbox
boots via libkrun"| guest supervisor <-->|"mutual TLS RFC 0012
over virtio-vsock"| sandbox client["openshell-cli"] -->|"connect / exec / forward"| gateway client -->|"CreateSandbox / Watch"| gateway ``` The supervisor owns gateway credentials, admitted policy, provider resolution, middleware, the network proxy, and relay registration. The sandbox receives no gateway JWT. Each VM generation receives distinct sandbox and supervisor channel keys; the guest consumes and unlinks its private bootstrap files before launching the workload. VM-specific RFC 0012 code under `src/isolation/` only chooses the vsock transport and binds immutable VM generation and image claims into the protected guest config and host descriptor. Lifecycle, authentication, process control, binary identity, forwarding, and streaming come from `openshell-isolation-interface` and `openshell-sandbox`. ## Quick start (recommended) ```shell mise run gateway:vm ``` First run takes a few minutes while `mise run vm:setup` stages libkrun/libkrunfw/umoci and `mise run vm:supervisor` builds the portable Linux guest sandbox plus its small static guest-init helper. The development task also builds the native host supervisor. Subsequent runs are cached. By default `mise run gateway:vm`: - Listens on plaintext HTTP at `127.0.0.1:18081`. - Uses `nvcr.io/nvidia/base/ubuntu:24.04` as the sandbox and bootstrap image. - Configures the gateway installation name as `vm-dev` and registers the same name with the CLI by writing `~/.config/openshell/gateways/vm-dev/metadata.json`. It does not modify the workspace `.env`. - Persists the gateway SQLite DB under `.cache/gateway-vm/gateway.db`. - Places the VM driver state (per-sandbox `overlay.ext4`, image cache, and `run/compute-driver.sock`) under `/tmp/openshell-vm-driver-$USER-vm-dev/` so the AF_UNIX socket path stays under macOS `SUN_LEN`. - Writes `.cache/gateway-vm/gateway.toml` with `[openshell.drivers.vm].driver_dir = "$PWD/target/debug"` so the freshly built `openshell-driver-vm` is used instead of an older installed copy from `~/.local/libexec/openshell`, `/usr/libexec/openshell`, or `/usr/local/libexec`. - Enables OTLP trace export to `http://127.0.0.1:4317` only when a local collector is listening there. Otherwise, it omits the OTLP configuration to avoid repeated export failures. For GPU passthrough (VFIO), pass `-- --gpu` and run with root privileges: ```shell sudo -E env "PATH=$PATH" mise run gateway:vm -- --gpu ``` GPU passthrough uses VFIO and requires host support for IOMMU, root privileges for bind/unbind operations, and a compatible sandbox image. The public GPU overview lives in the repository `README.md`. Point the CLI at the gateway with one of: ```shell openshell --gateway vm-dev status openshell gateway select vm-dev # then plain `openshell ` ``` Override defaults via environment: ```shell # custom port (fails fast if in use) OPENSHELL_SERVER_PORT=18091 mise run gateway:vm # custom gateway installation/CLI name + namespace OPENSHELL_VM_GATEWAY_NAME=vm-feature-a \ OPENSHELL_SANDBOX_NAMESPACE=vm-feature-a \ mise run gateway:vm # custom sandbox image OPENSHELL_SANDBOX_IMAGE=ghcr.io/example/sandbox:latest mise run gateway:vm # custom bootstrap image for the VM runtime used to prepare/boot target images OPENSHELL_VM_BOOTSTRAP_IMAGE=ghcr.io/example/bootstrap:latest mise run gateway:vm ``` Teardown: ```shell rm -rf /tmp/openshell-vm-driver-$USER-vm-dev .cache/gateway-vm rm -rf "${XDG_CONFIG_HOME:-$HOME/.config}/openshell/gateways/vm-dev" ``` ## Manual equivalent If you want to drive the launch yourself instead of using `mise run gateway:vm` (i.e. `tasks/scripts/gateway-vm.sh`): ```shell # 1. Stage runtime artifacts + guest sandbox into target/vm-runtime-compressed/ mise run vm:setup mise run vm:supervisor # builds the Linux guest sandbox and static guest-init helper # 2. Build gateway, native host supervisor, and driver OPENSHELL_VM_RUNTIME_COMPRESSED_DIR=$PWD/target/vm-runtime-compressed \ cargo build -p openshell-gateway -p openshell-supervisor -p openshell-driver-vm # 3. macOS only: codesign the driver for Hypervisor.framework codesign \ --entitlements crates/openshell-driver-vm/entitlements.plist \ --force -s - target/debug/openshell-driver-vm # 4. Start the gateway with the VM driver mkdir -p /tmp/openshell-vm-driver-$USER-vm-dev .cache/gateway-vm cat > .cache/gateway-vm/gateway.toml < ./scripts/bin/openshell sandbox connect demo ``` First sandbox takes 10–30 seconds to boot (image fetch/prepare/cache + libkrun + guest init). If `--from` is omitted, the VM driver uses the gateway's configured default sandbox image. Without either `--from` or `--sandbox-image`, VM sandbox creation fails. Subsequent creates reuse the prepared image cache and create only a sparse per-sandbox `overlay.ext4` before boot. `CreateSandbox` accepts the sandbox quickly and continues VM provisioning in the background. The driver publishes platform events for image resolution, cache hits/misses, layer pulls, rootfs preparation, overlay creation, and VM launcher startup so the CLI can show progress through the existing sandbox watch stream. The VM driver keeps two image caches. The bootstrap cache is a controlled `rootfs.ext4` used to boot the guest init and OpenShell supervisor. The prepared image cache is used when the requested sandbox image differs from the bootstrap image: the host downloads registry layers into a valid OCI layout, attaches that payload to a temporary bootstrap VM, and guest init runs `umoci raw unpack` onto Linux-owned ext4 storage. The resulting disk is cached under `/images//rootfs.ext4` and attached read-only to later sandboxes. Local Docker images are still exported as rootfs tar archives and prepared inside the bootstrap VM. The driver checks that a prepared disk contains the unpacked rootfs before caching it; on failure it caches nothing and reports the image-prep console tail. Set `OPENSHELL_VM_IMAGE_PULL_CONCURRENCY` to tune registry layer download parallelism (default `4`, maximum `16`). Registry pulls in both caches use the resolved digest as an immutable reference. The driver verifies the downloaded manifest against that digest before downloading layers. For multi-platform images, it verifies the index and then the selected platform manifest. Digest-pinned requests retain the caller's digest. Cache keys hash source identities to keep cache and staging filenames bounded. Bootstrap cache layout v6 and prepared-image layout v4 rebuild older entries on first use; older cache files remain on disk but are not reused for registry images. Both caches are scoped by source image identity and OpenShell version, so an OpenShell upgrade builds a fresh guest rootfs instead of reusing one with an old embedded supervisor. Host-side bootstrap layer assembly preserves relative symlinks that remain inside the rootfs, but rejects absolute or escaping cross-layer symlink traversal before applying later files or whiteouts. The layer applier performs rootfs mutations relative to opened directory handles and creates destination files without following symlinks. The requested sandbox image is never selected as the bootstrap image. Operators must configure either `bootstrap_image` or `default_image`; when both are empty, the driver fails during startup. Image and writable-overlay preparation run in owned worker processes. Stopping or deleting a sandbox cancels its worker and formatter processes, waits for them to exit, then removes the attempt's temporary files under `/images/preparations/`. New overlays, including retries after an interrupted first start, are built there and renamed into sandbox state only after completion. A file lock inherited by the worker's children keeps cleanup from deleting files that a process still owns. If cleanup cannot establish that the processes have stopped, the operation returns an error and retains both temporary files and sandbox state. At startup, the driver reclaims inactive attempts in this directory. It preserves active attempts, committed image caches, and unmarked staging from older releases. Concurrent preparations serialize image-cache publication; an interrupted attempt cannot publish a partial disk over an existing cache entry. Gateway upload slots for `rootfs_tar_path` keep their existing, separate lifecycle. Each sandbox gets its own sparse writable `/sandboxes//overlay.ext4`. Guest init mounts overlayfs as `/` with the prepared image rootfs as lowerdir when present, otherwise the bootstrap rootfs is used directly. Writes to `/sandbox` and other mutable paths land in the overlay while cached image disks remain unchanged. The overlay disk must be large enough to hold the compressed payload, unpacked rootfs, and sandbox writes during the first prepare. The driver also writes the accepted `DriverSandbox` launch request to `/sandboxes//sandbox.pb`. If the gateway restarts, it starts a new VM driver process. During graceful shutdown, the gateway first sends the shared `StopSandbox` request for each persisted running-intent sandbox, which stops its launcher while retaining the launch request and `overlay.ext4`. After driver initialization, the gateway sends the idempotent `StartSandbox` request for that retained intent. Explicitly stopped sandboxes remain excluded. Stop writes a marker in the sandbox state directory before terminating the launcher and releasing host GPU and network allocations. It retains `sandbox.pb`, `overlay.ext4`, and lifecycle-extension state. Startup registers marked sandboxes without launching compute. Start removes the marker and uses the normal persisted restore path with the existing overlay. Delete removes the entire sandbox state directory, including a stop marker and overlay. The host control writes and syncs a terminal tombstone when the canonical main process exits, before it reports completion and while it retains the boundary for exec and forwarding. Driver startup reports that sandbox as terminal instead of relaunching the VM, even when the process exited successfully. The driver embeds a platform-native host supervisor and extracts it into `/host-runtime`. It accepts a cached binary only when its SHA-256 content matches the embedded supervisor and it remains an executable regular file. Replacement is written and synced under a temporary name, then atomically renamed into place. `OPENSHELL_VM_SUPERVISOR_BIN` remains an explicit development override. ## Logs and debugging Raise log verbosity for both processes: ```shell RUST_LOG=openshell_server=debug,openshell_driver_vm=debug \ mise run gateway:vm ``` The VM guest's serial console is appended to `//console.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions and removes same-owner stale sockets. On clean shutdown, the gateway sends the managed driver `SIGTERM`, waits up to five seconds for it to flush telemetry and exit, then force-kills it if necessary and removes the socket. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development. The VM serial console is appended to `/sandboxes//rootfs-console.log`. Host-supervisor stdout and stderr are written beside it as `supervisor.log` and `supervisor.err.log`. Sandbox IDs must match `[A-Za-z0-9._-]{1,128}` before the driver uses them in host paths. The gateway-owned compute-driver socket lives at `/run/compute-driver.sock`; OpenShell creates `run/` with owner-only permissions, removes same-owner stale sockets, and the gateway removes the socket on clean shutdown via `ManagedDriverProcess::drop`. UDS clients must match the driver UID and provide the expected gateway process PID by default. Standalone same-UID UDS mode requires the explicit `--allow-same-uid-peer` development flag. TCP mode is disabled by default because it is unauthenticated; use `--allow-unauthenticated-tcp --bind-address 127.0.0.1:50061` only for local development. ## Network isolation VM sandboxes boot without a virtual NIC. The guest exposes only the protected vsock channel used by `openshell-sandbox`; `openshell-supervisor` performs DNS, policy evaluation, and external networking on the host. The driver does not create TAP devices or install nftables/iptables rules. ## Prerequisites - macOS on Apple Silicon, or Linux on aarch64/x86_64 with KVM - Rust toolchain - e2fsprogs (`mke2fs` or `mkfs.ext4`, plus `debugfs`) for root and overlay disk image creation and QEMU environment injection - Guest-sandbox cross-compile toolchain (needed on macOS, and on Linux when host arch differs from the guest): - Matching static guest target: `rustup target add aarch64-unknown-linux-musl` (or `x86_64-unknown-linux-musl` for an amd64 guest) - `cargo install --locked cargo-zigbuild` and `brew install zig` (or distro equivalent). `vm:supervisor` cross-compiles the Linux guest `openshell-sandbox` and its matching `openshell-supervisor`. - [mise](https://mise.jdx.dev/) task runner - Docker or Podman socket on the local CLI/gateway host when building an image before `openshell sandbox create --from `; the VM driver exports the image via the local container engine. Docker is tried first; if unavailable, the driver falls back to the Podman socket. On Linux, enable the Podman API socket with `systemctl --user start podman.socket` - `gh` CLI (used by `mise run vm:setup` to download pre-built runtime artifacts) ## Releases `openshell-driver-vm` is published as a normal OpenShell release artifact: - development builds: the rolling `dev` release - tagged builds: the corresponding `v*` release - runtime tarballs: the rolling `vm-runtime` release, rebuilt on demand by `release-vm-kernel.yml` On Debian-family Linux amd64 and arm64 systems, `install.sh` installs the Debian package from the selected `OPENSHELL_VERSION` release tag. That package includes `openshell-gateway` and `openshell-driver-vm`, but leaves `OPENSHELL_COMPUTE_DRIVER` unset so the gateway uses its normal runtime auto-detection. Set `OPENSHELL_COMPUTE_DRIVER=vm` to force the VM driver. On RPM-family Linux x86_64 and aarch64 systems, `install.sh` installs the `openshell` and `openshell-gateway` RPM packages from the selected release tag. The RPM gateway package is configured for the Podman driver. On Apple Silicon macOS, `install.sh` stages the generated `openshell.rb` formula from the selected release in the `nvidia/openshell` Homebrew tap. Homebrew installs `openshell`, `openshell-gateway`, and the self-contained `openshell-driver-vm` with its embedded native supervisor. It ad-hoc signs the driver with the Hypervisor entitlement in `post_install` and owns the `brew services` gateway lifecycle. The service also leaves `OPENSHELL_DRIVERS` unset so driver choice remains automatic unless the user explicitly overrides it. ## TODOs - The gateway still configures the driver via CLI args; this will move to a gRPC bootstrap call so the driver interface is uniform across backends. See the `TODO(driver-abstraction)` note in `crates/openshell-gateway/src/vm.rs`. - macOS local builds are codesigned by `tasks/scripts/gateway-vm.sh`; the generated Homebrew formula signs the release tarball driver for local installs. --- ## File: crates/openshell-driver-podman/README.md # openshell-driver-podman The Podman compute driver runs inside the gateway and uses the native libpod REST API over a Unix socket. Each sandbox has two independent containers: - `openshell-sandbox` owns the agent process in the workload container. - `openshell-supervisor` evaluates policy, holds gateway credentials, and proxies approved egress in a separate companion container. The driver provisions placement, identity, credentials, transport, and lifecycle. The shared isolation interface supplies exec, attach, signal, terminate, binary identity, DNS, TCP, and loopback-forwarding semantics. ## Runtime posture Caller driver config is disabled by default. Existing volumes require administrator-controlled approval labels; bind and supplemental image mounts are denied under enforcement. Private-volume names alone do not prove ownership. GPU devices are temporarily exempt. Admission runs before launch, restart, and periodically for running workloads. See [resource admission configuration](../../docs/how-it-works/gateways/configuration.mdx#external-resource-admission). | Property | Workload | Supervisor | |---|---|---| | UID/GID | Pinned non-root workload identity | Same mapped identity | | Capabilities | Drop all; add none | Drop all; add none | | Seccomp | Runtime default plus sandbox-installed filters | Runtime default | | Network | `none`; loopback only | Podman host network | | Gateway JWT and upstream credentials | Never mounted | Podman secrets | | User volumes and CDI devices | Workload only | Never mounted | | Channel | Private named volume, writable | Same volume, read-only | Podman creates the namespaces and volume ownership before the workload runs. Rootless operation uses the operator's Podman service and subordinate-ID configuration; it does not require adding capabilities to either container. The supervisor joins the workload's **user namespace only** to preserve UID/GID mapping for shared-volume access. PID, mount, and network namespaces remain separate. The channel volume uses shared SELinux relabeling (`:z`). Before starting either container, the driver uploads volume-relative archives directly to the channel and workspace volume destinations. A rootfs upload on a stopped Podman container does not populate nested named volumes. Restart restores only the channel bootstrap into the existing channel volume, preserving the workspace. The workload starts before the supervisor so its user namespace exists when the supervisor joins it; a stopped supervisor resolves that namespace again on its next start. The driver creates the managed workspace volume owned by the workload's final UID and GID, so the workload never starts as root. Custom image workspaces have no workspace volume or upload. The runtime must pass the sandbox's unprivileged enforcement probe, including nested seccomp notification and Landlock. Unsupported runtime defaults fail closed; do not switch to an unconfined profile or add capabilities. ## Protected channel and network enforcement ```text agent -> openshell-sandbox === authenticated gRPC / private UDS === supervisor -> network network=none TCP, DNS, control streams policy + gateway JWT ``` The workload has no external interface or published port. Seccomp socket mediation carries TCP and DNS through one authenticated gRPC connection. DNS remains supervisor-mediated; general UDP is unsupported. The driver sets `net.ipv4.ip_unprivileged_port_start=0` in the isolated workload network namespace so the sandbox's loopback DNS relay can bind port 53 without a capability. No nftables or nested network namespace setup runs in the sandbox. The channel contains the sandbox bootstrap and sandbox-side TLS identity only. Supervisor private keys and the runtime descriptor stay in the supervisor's private filesystem. Landlock denies agent access to the top-level `/.openshell` control hierarchy. The driver verifies Podman's reported `network=none` fence before launch and restart. Host networking applies to the supervisor, not the agent. Gateway sessions use the sandbox JWT and optional server-authenticated TLS. Only the gateway CA is delivered to the supervisor; user client certificates and private keys are not mounted into either container. The sandbox/supervisor channel always uses its separate, per-sandbox mutual TLS material. These are distinct authentication relationships. ## Identity and trusted binaries Both workload and supervisor images are pinned by immutable image ID. The driver reads account files from a stopped workload-image container; it never executes the image to resolve an account. Policy identity fields override OCI `USER` independently. Named users/groups resolve against that image, including supplementary groups. Root and unresolved identities fail before provisioning. Images must not prepopulate the reserved `/.openshell` hierarchy; this prevents image-controlled symlinks from aliasing private control state into user mounts. `sandbox_runtime_image` supplies the statically linked musl `/openshell-sandbox` binary. Podman's read-only image volume delivers it to the workload; user-namespace modes that cannot use image volumes retain the trusted binary extraction path. `supervisor_image` supplies the dynamically linked glibc `/openshell-supervisor` binary outside the workload. Image and request environment belong to agent children, never the supervisor process. ## OCI working directory OpenShell reads `WORKDIR` from the workload image. If it is unset, `/`, or `/sandbox`, OpenShell uses its managed `/sandbox` workspace volume. A custom path must be absolute and normalized, and cannot overlap `/proc`, `/sys`, `/dev`, OpenShell-reserved paths, or the workload's private control and CA mounts. Image volumes and driver mounts cannot cover it; mounts nested below it remain valid. A custom path stays in the image's container filesystem with its ownership and permissions. The workload starts as the final non-root user, which must be able to reach and write the directory. Agent commands use the path as their working directory. Custom workspace contents survive stop/start of the same workload container, but are removed with that container. They do not live in a managed named volume and are not copied to a replacement container. ## Lifecycle and readiness Create builds both stopped containers and stages the private archives before starting either container. The sandbox does not execute the agent until the supervisor authenticates and confirms the common boundary contract. Failed creation removes only containers created by that attempt, then cleans up driver-owned volumes and secrets. Stop retains both containers, workspace, channel, and secrets. Start restores the consumed sandbox bootstrap from a copy in the supervisor's private filesystem, verifies the fence, and starts the same pair. A failed supervisor start stops the workload. Delete removes the companion first, then the workload, channel, workspace, and driver-owned secrets. User-owned volumes are retained. Only workload containers appear in sandbox list/watch results. Readiness uses the supervisor's private health socket; there is no shell, legacy marker, or TCP-listener shortcut. Watch reconciliation and supervisor exit/removal events stop a running workload whose companion is unavailable. The gateway also requires the authenticated supervisor session before publishing Ready. ## Mounts, GPUs, and configuration User `bind`, `volume`, `tmpfs`, and `image` mounts and CDI GPU selection remain native Podman features and apply only to the workload. Bind mounts require the operator's `enable_bind_mounts` opt-in and disabled label admission. Supplemental image mounts also require disabled admission. Driver JSON requires `allow_driver_config = true`. Reserved control paths and the workspace root cannot be replaced. User-owned volumes are never created or deleted. See [gateway configuration](../../docs/how-it-works/gateways/configuration.mdx) for operator settings and [NETWORKING.md](NETWORKING.md) for supervisor networking. The supervisor uses Podman's host network and owns the upstream proxy settings. Omit `health_check_interval_secs` to disable Podman's periodic health command. Explicit zero is invalid. OpenShell still gates readiness on the supervisor's authenticated health signal. Gateway OTLP configuration continues to export compute-driver spans under the `openshell-driver-podman` service, preserving gateway trace context. --- ## File: crates/openshell-driver-kubernetes/README.md # openshell-driver-kubernetes Kubernetes-backed compute driver for OpenShell cluster deployments. Caller driver config is disabled by default. External resource references need administrator-controlled approval labels in every workspace mode, including before restart and scheduling-gate release. GPU devices are temporarily exempt. Image-pull Secrets are operator-selected gateway configuration rather than caller attachments. Managed mode stages an immutable copy for each sandbox runtime generation. See [resource admission configuration](../../docs/how-it-works/gateways/configuration.mdx#external-resource-admission). Caller PVC mounts cannot overlap the driver-managed `/.openshell` control tree or `/run`, including nested `subPath` mounts. Resource approval labels do not permit mounting over these paths. Use data paths such as `/data` or a directory below `/sandbox`. The driver uses the Kubernetes API to create, delete, fetch, and watch sandbox custom resources. It runs in-process with the gateway server and supports three workspace namespace modes via `workspace_mode`: - **Shared** (default): All sandboxes render into a single static namespace. Resource names use `{workspace}--{name}` for collision avoidance. - **Managed**: The driver auto-creates/deletes a K8s namespace per workspace (`openshell-{gateway_id}-{workspace_name}`) and creates a ServiceAccount in each. On OpenShift, it leaves SCC annotations to the namespace allocator and waits for the namespace's own MCS, UID-range, and supplemental-group annotations before provisioning sandbox resources. An existing namespace with a UID range but no MCS must be recreated so OpenShift can allocate a complete set of SCC annotations. - **Operator**: Workspace names map 1:1 to pre-provisioned namespaces discovered through exactly one source: either a label selector (`operator_namespace_label`) or a drop-in allowlist file (`operator_namespace_file`). Sandbox creation fails closed if the workspace namespace is not in the current allowlist. With a label selector, workspace checks that run before the watcher's first namespace listing wait up to 10 seconds for it and then return `Unavailable`, so a gateway restart does not reject labeled workspaces as unknown. Workspace deletion only removes gateway state; it never deletes or otherwise accesses the operator-managed Kubernetes namespace. When the gateway configures `[openshell.gateway.otlp]`, Kubernetes compute-driver spans export to the same OTLP/gRPC collector with the service name `openshell-driver-kubernetes`. The driver preserves the gateway trace context and uses the same compute-driver RPC span names in its in-process and standalone forms. Standalone deployments set `--gateway-name` or `OPENSHELL_GATEWAY_NAME` so exported spans carry the same `openshell.gateway.name` resource attribute as gateway spans. When it creates an Agent Sandbox resource, the driver serializes the active W3C trace context into the controller-reserved `opentelemetry.io/trace-context` annotation. An OTLP-enabled Agent Sandbox controller can therefore attach its asynchronous reconciliation spans to the originating OpenShell create trace. Workspace namespace modes assume exclusive control of the sandbox identity resource chain. In shared and managed modes, only the driver and its trusted Agent Sandbox controller may administer the sandbox namespace, Sandbox CRs, sandbox pods, or configured sandbox ServiceAccount. In operator mode, the platform operator owns namespace lifecycle but must prevent other principals from creating or mutating Sandbox CRs, creating sandbox pods with fabricated owner references, or using the configured sandbox ServiceAccount. Treat adding a namespace to the operator allowlist as granting this trust; the allowlist is not a tenant isolation boundary. ## Runtime Model The gateway stores platform state and delegates sandbox workload creation to this driver. Kubernetes owns scheduling and pod lifecycle. The workload Pod stages the statically linked musl `openshell-sandbox` binary from `sandbox_runtime_image`, while a directly managed Pod runs the dynamically linked glibc `openshell-supervisor` from `supervisor_image`. The sandbox owns the agent process, applies Landlock and child seccomp filters, identifies the binary behind each network syscall, and relays mediated streams to the supervisor. The supervisor authenticates to the gateway with a JWT, loads policy and provider state, performs destination and L7 authorization, and opens upstream connections. The workload receives no gateway credential, provider identity socket, or corporate-proxy credential. Both Pods run as the namespace-resolved non-root UID/GID with `allowPrivilegeEscalation: false`, `capabilities.drop: [ALL]`, and the runtime default seccomp profile. The sandbox installs a nested seccomp user-notification filter without requesting a capability in the Pod spec. Startup fails closed when the runtime blocks the required seccomp or Landlock operations. The supervisor Pod has a direct, non-controller owner reference to the Sandbox resource. This links its garbage-collection lifecycle to the sandbox without competing with the Agent Sandbox controller for workload-Pod ownership. When the gateway exports OTLP traces, the driver sets `OPENSHELL_OTLP_ENDPOINT` on the supervisor Pod to the gateway's endpoint and `TRACEPARENT` to the trace context of the operation that created the Pod. The supervisor exports its spans there and parents its startup span on that context. The endpoint is not configurable in driver TOML. The driver creates one namespace-wide `NetworkPolicy` before it releases any workload Pod. It selects every OpenShell workload, denies all workload egress, and permits OpenShell supervisor Pods to reach the sandbox TLS port. The authenticated Sandbox Protocol binds each connection to the exact sandbox and supervisor Pod identities. Supervisors have normal egress for gateway, DNS, and policy-approved upstream connections unless an operator policy restricts them. The cluster CNI must enforce ingress and egress `NetworkPolicy` for every sandbox namespace. Kubernetes accepts policy objects without confirming enforcement, so operators must verify CNI support before running sandboxes. Each sandbox generation uses two immutable bootstrap Secrets. A trusted init container stages the sandbox bootstrap into memory, and the sandbox removes it before starting untrusted code. The other Secret is mounted only by the supervisor; when `proxy_ca_bundle` is configured it also carries the operator's corporate proxy CA bundle, which the gateway reads from its own filesystem so the anchor stays in the gateway's trust domain rather than the sandbox namespace. The TLS channel binds the namespace, Sandbox CR, workload Pod, supervisor Pod, and shared network-policy identities. Stop deletes the workload and supervisor Pods. Start rotates both Secrets and creates a new supervisor Pod before releasing a new workload Pod. It also renders the trusted init container again from gateway config, so a sandbox created by an older gateway starts with the current sandbox runtime image. The shared network fence remains for the lifetime of the namespace. Kubernetes policies are additive, and the API does not attest that the CNI enforces them. Keep sandbox namespaces administrative: untrusted principals must not create permissive policies, create Pods, read bootstrap Secrets, or spoof the OpenShell role labels. Exact supervisor-to-sandbox authorization is still enforced by TLS, JWT claims, session generation, and recorded Pod UIDs. ## Sandbox Resource The driver works with the `agents.x-k8s.io` `Sandbox` custom resource. It detects the served Sandbox API at runtime, caches the selected API version for the gateway process, and uses `v1beta1` when available before falling back to `v1alpha1`. Restart the gateway after an in-place Agent Sandbox upgrade so the driver can detect served API versions again. Driver events map Kubernetes object state and platform events into the shared compute-driver protobuf surface used by the gateway. Kubernetes API calls use explicit timeouts so gRPC handlers do not block indefinitely when the API server is slow or unavailable. The request client's timeout starts when each request is sent, so time a pooled connection spends idle does not shorten it. Resource and Event watches recover in place with API-friendly backoff after transient watcher errors, avoiding a gateway-side watch restart and its associated watch gap. ## Workspace Persistence Sandbox pods use a PVC-backed `/sandbox` workspace. An init container seeds the PVC from the image's original `/sandbox` contents on first start and writes a sentinel so subsequent starts skip the copy. This is a stopgap persistence model. It preserves user files across pod rescheduling but duplicates the base workspace and does not automatically apply image updates to existing PVCs. Future snapshotting should replace it. Stop preserves the Agent Sandbox resource and workspace PVC while stopping its pod. The driver sets `spec.operatingMode: Suspended` for `v1beta1` or `spec.replicas: 0` for `v1alpha1`. Start sets `Running` or one replica for the same resource, so the replacement pod mounts the existing claim. Delete is the only lifecycle operation that removes the Sandbox resource and its owned storage. The driver confirms the stop from both the published `Suspended` condition and deletion of the backing pod. Legacy `v1alpha1` controllers omit a usable stopped condition, so pod deletion alone confirms their stop. The workspace PVC size defaults to `workspace_default_storage_size`. Set `workspace_storage_class` to pin the PVC to a specific `StorageClass`; an empty value omits `storageClassName` so the cluster's default `StorageClass` applies. Clusters with no default `StorageClass` must set this, otherwise the PVC stays `Pending` and the sandbox never starts. Both fields can also be supplied at runtime via `OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE` and `OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS`. Both apply only to the workspace PVC that OpenShell provisions automatically; they have no effect when a `driver_config` mount attaches an existing PVC under `/sandbox`, which skips the default PVC. ## Credentials, TLS, and Relay Both Pods set `automountServiceAccountToken: false`. The supervisor receives an explicit audience-bound projected token for the one-shot `IssueSandboxToken` exchange. The driver verifies that token and returns an opaque runtime identity derived from the namespace, immutable Sandbox resource UID, and supervisor Pod UID. Restart requires exactly one matching Sandbox resource and preserves its namespace and UID while rotating the supervisor Pod UID. The gateway requires the authenticated identity to match the durable binding before returning the generation-bound session JWT used by the supervisor. The sandbox Pod receives neither token. For HTTPS gateway connections, the supervisor reads only the CA from the configured TLS Secret. Shared mode projects `ca.crt` directly; managed and operator modes stage only the CA into the supervisor bootstrap Secret. User client certificates and private keys are not mounted into either Pod. The gateway uses the supervisor relay for connect, exec, logs, and file sync. Sandbox Pods do not need direct external ingress for SSH. The driver sends the canonical main-process specification only to the supervisor. The supervisor passes admitted launch state over the protected channel. Provider environment updates apply to future exec sessions. ## Container Security Context The sandbox, trusted bootstrap init container, and supervisor request no added Linux capability. They run as the same numeric non-root identity, disable privilege escalation, drop all capabilities, and inherit `RuntimeDefault` seccomp. The sandbox and agent must use the same complete UID, GID, and supplementary-group identity because the capability-free sandbox cannot change credentials after launch and must inspect its same-identity descendants. The workload Pod does not share host network, PID, IPC, or process namespaces. The driver uses a scheduling gate to inspect the admitted Pod and bind its UID into the bootstrap claims before kubelet starts it. Lifecycle RPCs and runtime reconciliation share a per-sandbox mutation gate across clones of the driver. Reconciliation skips busy sandboxes and refreshes the Sandbox CR under that gate before cleanup, so a stopped or stopping LIST snapshot cannot delete a supervisor created by a concurrent restart in the same driver instance. The gate preserves concurrency across sandboxes; it does not provide distributed exclusion between separate gateway or driver processes. ## GPU Support When a sandbox requests GPU support, the driver checks node allocatable capacity for `nvidia.com/gpu` and requests the configured GPU count in the workload spec. When no count is set, the driver requests one GPU resource. The sandbox image must provide the user-space libraries needed by the agent workload. ## Driver Config Following RFC 0006, this driver accepts the selected `SandboxTemplate.driver_config.kubernetes` block as `DriverSandboxTemplate.driver_config`. The Kubernetes driver owns the nested schema and currently accepts: - `pod.node_selector` - `pod.tolerations` - `pod.runtime_class_name` - `pod.priority_class_name` - `containers.agent.resources.requests` - `containers.agent.resources.limits` - `containers.agent.volume_mounts[].name` - `containers.agent.volume_mounts[].mount_path` - `containers.agent.volume_mounts[].sub_path` - `containers.agent.volume_mounts[].read_only` - `volumes[].name` - `volumes[].persistent_volume_claim.claim_name` - `volumes[].persistent_volume_claim.read_only` Nested keys inside the `kubernetes` block use snake_case. The top-level `driver_config` envelope is keyed by driver names, so `kubernetes` is not part of the nested schema. Set this through the CLI with the public driver-keyed envelope. The gateway forwards only the `kubernetes` object to this driver: ```shell openshell sandbox create \ --driver-config-json '{"kubernetes":{"pod":{"runtime_class_name":"kata-containers","node_selector":{"pool":"gpu"}}}}' \ -- claude ``` Resource keys use native Kubernetes resource names and quantity strings. The parser renders the keys listed above and rejects unknown fields. `pod.runtime_class_name` maps to PodSpec `runtimeClassName` and overrides the driver's configured `default_runtime_class_name`; the typed public `SandboxTemplate.runtime_class_name` still takes precedence when set. Use the public `--gpu` flag for the default GPU request, pass a count to `--gpu` for counted GPU requests, and use `driver_config` only for additional driver-owned resource details. Use PVC volumes to mount existing Kubernetes PersistentVolumeClaims into the agent container. PVC volumes and mounts default to read-only unless `read_only: false` is set explicitly. Read-write access requires `read_only: false` on both the PVC volume and each writable mount. The driver rejects duplicate volume names, invalid DNS-1123 volume labels or PVC claim subdomain names, mounts that reference unknown volumes, non-normalized or protected mount paths, and absolute or parent-traversing `sub_path` values. Any explicit driver-config mount under `/sandbox` disables the driver's default `/sandbox` workspace PVC injection for that sandbox. Only the explicit mount paths persist through the external PVC; other `/sandbox` paths come from the current sandbox image. ```shell openshell sandbox create \ --driver-config-json '{ "kubernetes": { "volumes": [{ "name": "user-data", "persistent_volume_claim": { "claim_name": "pvc-user-data-123", "read_only": false } }], "containers": { "agent": { "volume_mounts": [ { "name": "user-data", "mount_path": "/sandbox/.openshell/workspace", "sub_path": "workspace", "read_only": false }, { "name": "user-data", "mount_path": "/sandbox/.openshell/memory", "sub_path": "memory", "read_only": false } ] } } } }' \ -- claude ``` --- ## File: crates/openshell-driver-docker/README.md # openshell-driver-docker Docker-backed compute driver for local and remote OpenShell gateways. The driver uses `bollard` to manage sandbox resources through the configured Docker API socket. When `socket_path` is unset, it selects the first standard local socket that responds to an API ping. An explicitly selected Docker driver falls back to `/var/run/docker.sock` when no candidate responds. When the gateway configures `[openshell.gateway.otlp]`, the in-process driver exports spans to the same OTLP/gRPC collector as `openshell-driver-docker`. The standalone driver accepts `OPENSHELL_OTLP_ENDPOINT`, continues W3C trace context from gateway RPC metadata, and flushes spans during graceful shutdown. ## Runtime Model Caller driver config is disabled by default. Existing volumes require administrator-controlled approval labels; raw bind mounts have no supported label resolver and are denied under enforcement. GPU devices are temporarily exempt. Inspect labels again before launch, restart, and during reconciliation. See [resource admission configuration](../../docs/how-it-works/gateways/configuration.mdx#external-resource-admission). The driver creates two containers for each sandbox: - `openshell-sandbox` is PID 1 in the workload container. It owns the workload process tree, seccomp notification broker, mandatory Landlock baseline, binary identity, exec/signal/wait/PTY operations, and loopback forwarding. - `openshell-supervisor` runs in a separate companion container. It owns the gateway session, policy engine, credentials, interception CA, SSH relay, L7 inspection, DNS policy, and external upstream connections. Both containers are non-root, request no capabilities, and set no-new-privileges. A shared named volume carries the authenticated Unix socket and sandbox bootstrap material. A second supervisor-only volume carries the supervisor JWT and gateway client credentials and is never mounted into the workload. The workload uses `network_mode=none`. Its seccomp user-notification broker mediates every supported TCP and DNS operation, attributes it to the calling binary, and sends the request across the private channel. The supervisor authorizes the request before it opens an upstream connection. Docker's absent workload network is the mandatory outer fence if mediation fails or is bypassed. The trusted supervisor companion originates approved egress. Its operator-owned `supervisor_network_mode` defaults to `host`. Set `bridge` to run the supervisor on Docker's bridge network; `auto` also selects host. The driver copies trusted runtime bytes from the configured supervisor image through the Docker archive API. No workload launch depends on a host bind mount or a tool supplied by the workload image, so the same path works with local, remote, and VM-backed Docker daemons. ## Corporate Proxy Egress `[openshell.drivers.docker]` accepts the operator-owned corporate proxy fields `https_proxy`, `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`, `proxy_connect_by_hostname`, and `proxy_ca_bundle`. Sandbox environment, image contents, and per-sandbox driver configuration cannot select or override them. The driver validates the proxy URL and cross-field relationships at gateway startup. It bounded-reads proxy credentials and the PEM CA bundle before use. Missing, unreadable, empty, oversized, malformed, or certificate-free CA files fail closed. A CA bundle requires `https_proxy`, although the proxy URL may be `http://` when the proxy intercepts destination TLS. For each supervisor launch, Docker copies the credential and CA contents into the existing supervisor-only named volume. It passes only the fixed container paths `/.openshell/supervisor/upstream-proxy-auth` and `/.openshell/supervisor/upstream-proxy-ca-bundle.pem` to the supervisor. It does not bind-mount the gateway-host files, which preserves remote-daemon support and keeps host paths out of workload container metadata. The supervisor trusts the configured CA for its TLS connection to an HTTPS proxy and for destination certificates re-signed by a TLS-intercepting proxy. It also includes the corporate root in the generated combined trust bundle used by workload processes. ## Identity and Workspace Before creating the workload, the driver pins the image ID and reads its passwd/group databases through a stopped metadata container. It resolves the admitted policy identity, or the image `Config.User` fallback, into one exact non-root UID, primary GID, and supplementary-group set. Docker launches `openshell-sandbox` with that identity, and the sandbox uses the same identity for every canonical and exec process. UID or GID zero and unresolved symbolic identities are rejected. An absolute OCI working directory becomes the workspace. An empty, root (`/`), or explicit `/sandbox` declaration uses `/sandbox`. Any other workdir must already exist without symlink components. The resolved identity must be able to traverse every parent and write and enter the workdir; OpenShell does not change its ownership or mode. Image `VOLUME` declarations and user mounts must not cover the workdir, one of its parents, or the reserved `/.openshell` runtime/channel tree. OpenShell asks the kernel to validate access under the final identity, so POSIX ACL and host LSM decisions remain authoritative. ## Container Contract | Setting | Purpose | |---|---| | Exact non-root `user` and `group_add` | Gives sandbox and workload the same immutable UID/GID/group identity required for capability-free observation. | | `cap_drop = ALL`, no `cap_add`, no-new-privileges | Prevents either container from acquiring Linux capabilities. | | Docker default seccomp and AppArmor profiles | Retains runtime hardening; startup confirmation fails closed if nested seccomp notification is unavailable. | | `network_mode = none` on the workload | Removes direct external routes. | | `network_mode = host` on the supervisor | Lets the trusted supervisor reach the gateway's primary loopback listener and originate approved upstream connections. | | `restart_policy = no` | Keeps canonical main-process exit terminal. | | `PidsLimit` | Applies the configured sandbox PID budget. Omit `sandbox_pids_limit` to use OpenShell's default. Explicit zero is invalid. | | Private named volumes | One carries the authenticated sandbox/supervisor channel. The other is mounted only into the supervisor and contains its JWT and private gateway credentials. | | In-memory `/run/openshell-supervisor-ca` tmpfs | Holds only the public supervisor CA certificate and trust bundle without making all of `/run` writable. | | CDI GPU request | Assigns the exact validated CDI devices requested by driver config or count-based selection. | ## Stop, Start, and Delete Stop terminates the supervisor companion and stops the workload container without removing it. Docker retains the workload writable layer and attached volumes. Start stages a fresh sandbox bootstrap bundle, restarts that workload, and creates a new supervisor companion. A durably stopped sandbox stays stopped across gateway restarts. Delete force-removes both containers, the driver-owned runtime volumes, and the host-private runtime descriptor. Missing or altered descriptor and channel resources fail closed; the driver does not run an older combined-supervisor layout. ## Driver Config Mounts The gateway forwards the `docker` block from `--driver-config-json`. Supported mount types are: - `bind`: an absolute daemon-host path, allowed only when `[openshell.drivers.docker].enable_bind_mounts = true` and label admission is explicitly disabled. - `volume`: an existing named volume. The driver never creates or removes a user-supplied volume. Bind-backed local volumes require `enable_bind_mounts = true`. - `tmpfs`: an in-memory filesystem with optional size and mode. Host bind mounts are disabled by default because they expose daemon-host paths to sandbox requests. User bind and volume mounts are read-only by default. Targets must be absolute, normalized paths and cannot overlap the workspace root or OpenShell control paths. Example: ```shell docker volume create openshell-work openshell sandbox create \ --driver-config-json '{"docker":{"mounts":[{"type":"volume","source":"openshell-work","target":"/sandbox/work"}]}}' \ -- claude ``` ## Runtime Image `sandbox_runtime_image` contains the statically linked musl `/openshell-sandbox` binary. The driver extracts that binary as bytes and stages it into the stopped workload. `supervisor_image` contains the dynamically linked glibc `/openshell-supervisor` binary that runs in the supervisor companion container. Release and gateway image builds bake matching image tags into the binary. ## Gateway session and TLS `OPENSHELL_ENDPOINT` and gateway authentication material are injected only into the supervisor companion. The workload never receives the sandbox JWT, gateway client TLS key, policy authority, or interception CA private key. When no endpoint is configured, the supervisor connects to `127.0.0.1:`. Set `grpc_endpoint` when the gateway is not on the Docker daemon host. A configured HTTPS server certificate must include the endpoint host in its subject alternative names. In host mode, the driver publishes host loopback as the backend address for `host.openshell.internal`. Policy DNS resolves that reserved name through the mediated path, so policies can reach host services without a Docker bridge, container DNS alias, or another gateway listener. For HTTPS endpoints, the supervisor receives only the gateway CA and uses its sandbox-scoped bearer token to authenticate RPCs. User client certificates and private keys are not delivered to either container. Host mode on Docker Desktop requires host networking enabled in Settings. For bridge mode or a gateway endpoint using `host.docker.internal`, the driver resolves Docker's `host-gateway` once during initialization and caches it. A temporary container runs `--help` from the immutable supervisor image with no network, no capabilities, no privilege escalation, and a read-only root filesystem. The driver downloads Docker's generated `/etc/hosts` through the archive API, preferring a unique IPv4 address and otherwise a unique IPv6 address. Resolution has a 30-second timeout; forced cleanup has a separate 10-second timeout and runs after success or failure. The driver uses the selected address for both host aliases and the runtime descriptor's trust pin; workload DNS and hosts files cannot select it. Host mode with other gateway endpoints retains its gateway-address pin. An explicit operator-owned `host_gateway_ip` overrides discovery and skips the temporary container. No supervisor changes or per-sandbox discovery are required. When `grpc_endpoint` is omitted, bridge mode uses `host.docker.internal` on Desktop or for a wildcard gateway listener, and the concrete bind address for other non-loopback Linux listeners. The port is the gateway bind port; remapped container publications require an explicit endpoint. Native Linux loopback-only listeners cannot serve a bridge supervisor and produce an actionable error. Explicit endpoints retain TLS verification and are never rewritten; bridge rejects loopback/unspecified endpoint addresses. Host services bound only to loopback are also unreachable from a native Linux bridge supervisor. The selected network mode does not change the sandbox's mandatory Landlock and seccomp qualification. The supervisor owns these security-critical variables: - `OPENSHELL_ENDPOINT` - `OPENSHELL_SANDBOX_ID` - `OPENSHELL_SANDBOX` - `OPENSHELL_SANDBOX_TOKEN_FILE` - `OPENSHELL_SSH_SOCKET_PATH` - `OPENSHELL_MAIN_PROCESS_SPEC` - `OPENSHELL_TLS_CA` when HTTPS is enabled Template and sandbox environment is encoded in the protected bootstrap and exposed only to workload children. Workload input cannot override security-critical supervisor variables. --- ## File: crates/openshell-core/README.md # openshell-core Shared types, constants, configuration, and helpers used across OpenShell crates. ## Isolation registrations `isolation_registration` owns the operator registry's backend names, byte-stream endpoints, and shared validation. Its fallible conversions carry registrations through the compute-driver protobuf and back into the supervisor's JSON launch file. These routes contain no credentials or driver-owned launch coordinates. ## Object Metadata Top-level user-facing objects use a Kubernetes-style metadata convention. The metadata shape provides: - Stable server-generated ID. - Human-readable name. - Creation timestamp. - Optional labels for filtering and automation. Code that works with object metadata should use the traits in `openshell_core::metadata` instead of reaching into protobuf fields directly: ```rust use openshell_core::{ObjectId, ObjectLabels, ObjectName}; let id = sandbox.object_id(); let name = sandbox.object_name(); let labels = sandbox.object_labels(); ``` Trait methods must tolerate missing metadata and return safe empty values rather than panicking. ## Label Rules Labels follow Kubernetes-style key and value conventions: - Keys may include an optional DNS-prefix followed by `/`. - Names are limited to alphanumeric characters plus `-`, `_`, and `.`. - Values use the same character set and may be empty. - Selectors use comma-separated `key=value` pairs with AND semantics. Validate labels at API ingress before persisting objects. ## Inference Profiles Provider inference profiles live in this crate so the gateway, sandbox, and router agree on provider defaults. Profiles define: - Auth header style. - Default upstream headers. - Client-supplied passthrough headers. - Supported inference protocol shapes. Do not duplicate provider-specific inference behavior in callers. Add shared behavior here, then consume it from the gateway, sandbox, and router. ## Middleware Contracts Built-in supervisor middleware identifiers, host-selector matching, and pure configuration validation live in `openshell_core::middleware`. Policy admission and the supervisor runtime consume the same contract without introducing a dependency from the policy crate to the supervisor implementation. ## Protobuf Struct Conversion Use `openshell_core::proto_struct` when crossing between `serde_json` values and `prost_types::{Struct, Value}`. Both conversion directions live in this module; JSON-to-protobuf conversion is fallible so callers cannot silently replace an unrepresentable number. --- ## File: deploy/helm/openshell/README.md # OpenShell Helm Chart > **Experimental** - the Kubernetes deployment path is under active development. Expect rough edges and breaking changes. This chart deploys the OpenShell gateway into a Kubernetes cluster. It is published as an OCI artifact to GHCR at `oci://ghcr.io/nvidia/openshell/helm-chart`. By default, this chart also creates the namespace-scoped resources needed by sandboxes. For a shared-gateway deployment, install it with `workspaceResources.enabled=false`, then install the `deploy/helm/openshell-workspace` chart in every pre-provisioned workspace namespace. The gateway and workspace releases can then be upgraded and removed independently. Use Kubernetes `operator` workspace mode when one gateway serves multiple pre-provisioned workspace namespaces. ## Cluster-scoped vs namespaced objects Most objects in this chart are namespaced and land in the release namespace. Only two are cluster-scoped: | Object | Default name | | --- | --- | | `ClusterRole` | `-node-reader-` | | `ClusterRoleBinding` | `-node-reader-` | By default the release creates both, so an install by a cluster-admin is unchanged. On clusters where cluster-scoped RBAC is owned by a different team, split the install in two. A cluster-admin applies the cluster-scoped objects once per gateway ServiceAccount, rendered from the same values the release uses: ```shell helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ --namespace openshell -f my-values.yaml \ --set rbac.create=true \ --set rbac.clusterScoped.create=true \ --set agentSandbox.preflight.enabled=false \ --show-only templates/clusterrole.yaml \ --show-only templates/clusterrolebinding.yaml | kubectl apply -f - ``` A namespace-admin then installs and upgrades the release with cluster-scoped objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml) or the equivalent `--set`: ```shell helm upgrade --install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ --namespace openshell -f my-values.yaml \ --set rbac.clusterScoped.create=false ``` The gateway ServiceAccount name and namespace do not change, so the pre-created `ClusterRoleBinding` keeps matching the release. This works with `serviceAccount.create=false` too: the `ClusterRoleBinding` subject follows `serviceAccount.name`, so render the admin step with the same values. ### Which flag the installer needs In the default `shared` workspace mode the release also creates a namespaced sandbox `Role` granting Agent Sandbox (`agents.x-k8s.io`) permissions. Kubernetes forbids granting permissions you do not hold, and the built-in `admin` ClusterRole does not cover that CRD, so an installer holding only `admin` cannot create it. `rbac.clusterScoped.create=false` alone is then not enough and the install fails with `attempting to grant RBAC permissions not currently held`. | Workspace mode | Installer holds | Use | | --- | --- | --- | | `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC | | `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` | | `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` | `managed` and `operator` render no namespaced sandbox `Role`, so no extra grant is needed there. With `rbac.create=false` the cluster-admin applies the namespaced RBAC too, adding it to the same render: ```shell helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ --namespace openshell -f my-values.yaml \ --set rbac.create=true \ --set rbac.clusterScoped.create=true \ --set agentSandbox.preflight.enabled=false \ --show-only templates/clusterrole.yaml \ --show-only templates/clusterrolebinding.yaml \ --show-only templates/role.yaml \ --show-only templates/rolebinding.yaml \ --show-only templates/peer-role.yaml | kubectl apply -f - ``` To grant the installer the sandbox permissions instead, bind it to a Role carrying the same rules as the chart's `openshell-sandbox` Role. The certgen hook and credential driver RBAC keep their own flags (`pkiInitJob.enabled` and `server.credentialDrivers.kubernetesSecrets.rbac.create`). ### Migrating an existing release Helm deletes objects that leave a release manifest, so setting `rbac.clusterScoped.create=false` on a release that already owns the `ClusterRole` and `ClusterRoleBinding` deletes them. The gateway then loses TokenReview until a cluster-admin re-applies them. Hand ownership over first, as cluster-admin, so nothing is deleted: ```shell kubectl annotate clusterrole "openshell-node-reader-" \ helm.sh/resource-policy=keep --overwrite kubectl annotate clusterrolebinding "openshell-node-reader-" \ helm.sh/resource-policy=keep --overwrite ``` The objects then survive the upgrade that sets the flag, and the cluster-admin owns them from that point on. Fresh installs need no such step. `rbac.clusterScoped.create` is independent of `server.drivers.kubernetes.workspaceMode`. Managed and operator modes change what the `ClusterRole` contains, but they never force the namespaced release to apply it. Re-run the cluster-admin step after changing values that affect the `ClusterRole` rules. ## Prerequisites > **Required:** Your cluster CNI MUST enforce Kubernetes `NetworkPolicy` for > ingress and egress in every sandbox namespace. OpenShell creates the policies, > but Kubernetes accepts them even if no CNI enforces them. Without enforcement, > sandbox workloads may connect directly and bypass supervisor network policy. > Verify CNI support before installing OpenShell. The Kubernetes Agent Sandbox CRDs and controller must be installed on the cluster before deploying OpenShell. Install them with: ```shell kubectl apply -f https://github.com/kubernetes-sigs/agent-sandbox/releases/latest/download/sandbox.yaml ``` The chart does not install this cluster-scoped dependency. By default, it fails before creating gateway resources when the cluster serves neither supported Sandbox API (`agents.x-k8s.io/v1beta1` or `agents.x-k8s.io/v1alpha1`). Disable the check with `agentSandbox.preflight.enabled=false` for offline `helm template` rendering, where Helm cannot discover cluster APIs. ## Install on Kubernetes ```shell helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version ``` ## Install on OpenShift See the full [OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift) for details. Quick start: ```shell # Precreate the openshell namespace oc create ns openshell # Deploy openshell with overrides to allow SCC assignment of fsGroup and runAsUser for the gateway helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version -n openshell \ --set server.disableTls=true \ --set podSecurityContext.fsGroup=null \ --set securityContext.runAsUser=null ``` On OpenShift 4.22+, end-to-end TLS is supported via `BackendTLSPolicy`. See the [OpenShift install guide](https://docs.nvidia.com/openshell/latest/kubernetes/openshift#end-to-end-tls-openshift-422) for details. ## Available versions | Tag | Source | Notes | | --- | --- | --- | | `` (e.g. `0.6.0`) | Tagged GitHub release | Tracks the matching gateway, sandbox, and supervisor image versions. Recommended for production. | | `-pre.N` (e.g. `0.1.0-pre.3`) | A specific prerelease candidate | Immutable candidate pin that tracks images with the same exact version. | | `0.0.0-dev` | Latest commit on `main` | Floating tag, overwritten on every push. `appVersion` is `dev`, so images resolve to the `:dev` tag. | | `0.0.0-dev.` | A specific commit on `main` | Per-commit pin. Chart version and `appVersion` both use the full 40-character commit SHA, which matches the image tag pushed by CI. | Prerelease and `dev` tags are intended for testing changes ahead of a release. Production deployments should pin to a stable tagged release. ## Configuration See [`values.yaml`](values.yaml) for source defaults. Selected overlays: - [`ci/values-gateway.yaml`](ci/values-gateway.yaml) - gateway-only configuration - [`ci/values-cert-manager.yaml`](ci/values-cert-manager.yaml) - cert-manager integration - [`ci/values-keycloak.yaml`](ci/values-keycloak.yaml) - Keycloak OIDC integration - [`ci/values-high-availability.yaml`](ci/values-high-availability.yaml) - CI overlay for multi-replica external PostgreSQL testing - [`ci/values-autoscaling.yaml`](ci/values-autoscaling.yaml) - CI overlay for rendering the optional gateway HorizontalPodAutoscaler - [`ci/values-spire.yaml`](ci/values-spire.yaml) - SPIFFE/SPIRE provider token grants - [`ci/values-spire-stack.yaml`](ci/values-spire-stack.yaml) - SPIRE hardened chart values for local development ### Database backend By default, OpenShell uses SQLite and runs the gateway as a StatefulSet so the database is backed by a per-pod PVC: ```yaml server: dbUrl: "sqlite:/var/openshell/openshell.db" ``` #### Data PVC storage Size the gateway data claim, `openshell-data`, and select its StorageClass with `server.dataStorageSize` and `server.dataStorageClass`. Omitting both requests `1Gi` from the cluster's default StorageClass. Set `server.dataStorageClass` on clusters with no default StorageClass, otherwise the gateway PVC stays Pending. Both values are ignored under `workload.kind=deployment`, which mounts no PVC. ```yaml server: dataStorageSize: 20Gi dataStorageClass: fast-ssd ``` `volumeClaimTemplates` is immutable, so set both at install time. To grow an existing claim, expand it in place when its StorageClass allows expansion, then recreate the StatefulSet so its template matches: ```shell kubectl -n openshell patch pvc openshell-data-openshell-0 \ -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}' kubectl -n openshell delete statefulset openshell --cascade=orphan helm upgrade openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ --reuse-values -n openshell --set server.dataStorageSize=20Gi ``` A recreated StatefulSet re-adopts existing claims unchanged, so changing StorageClass means copying the data to a new claim. #### External PostgreSQL Use external PostgreSQL when the gateway should connect to a database managed outside this chart. The OpenShell chart does not deploy a database; install PostgreSQL separately using the chart, operator, or managed service that fits your environment, then pass the connection URI through a Secret. Create a Secret containing the PostgreSQL connection URI if one does not already exist: ```bash kubectl create secret generic my-pg-credentials -n openshell \ --from-literal=uri="postgresql://user:pass@host:5432/dbname" ``` Then install the chart pointing at that Secret: ```bash helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart --version \ -n openshell \ --set workload.kind=deployment \ --set server.externalDbSecret=my-pg-credentials ``` Use `workload.kind=deployment` for external database-backed multi-replica gateways. `workload.kind=statefulset` is still available for single-replica SQLite installs and for operators who explicitly need StatefulSet identity or storage semantics. ### Credential storage By default, the chart uses the gateway's encrypted database credential storage. The gateway writes encrypted provider credential envelopes to the OpenShell database. The chart creates a retained Kubernetes Secret with the shared key-encryption key and injects that key into every gateway pod, so the same default works for single-replica and external database-backed HA deployments. Use `gatewayConfig` to select `kubernetes-secrets` or `vault` when credentials should live in a cluster or external secret backend. Selecting an external credential driver disables the default credential-storage key-encryption-key Secret and environment injection. The map is rendered directly as gateway TOML, so it uses the gateway's snake_case field names: ```yaml gatewayConfig: openshell.gateway: credential_drivers: - vault openshell.credential_drivers.vault: address: https://vault.vault.svc.cluster.local:8200 mount: secret kv_version: "2" auth_method: kubernetes role: openshell-gateway ``` > `gatewayConfig` must contain only non-secret values. Helm serializes unknown > fields generically and cannot determine whether an arbitrary string, such as > `api_token`, is confidential. Do not put passwords, tokens, private keys, > database URLs, or other secret material in this map. Use Secret-backed > environment variables, files, volumes, or gateway credential drivers instead. > The chart rejects known unsafe forms such as `database_url`, inline URL > credentials, and PEM private keys; it is not a general secret scanner. For the Kubernetes Secret driver, use `openshell.credential_drivers.kubernetes-secrets.namespace` in the same map. The chart derives any required RBAC from the selected driver; use a dedicated namespace to limit access to OpenShell-managed Secrets. #### OpenShift Append these flags to any of the PostgreSQL commands above for OpenShift: ``` --set server.disableTls=true \ --set podSecurityContext.fsGroup=null \ --set securityContext.runAsUser=null ``` ### High availability Set `replicaCount` above `1` only with `server.externalDbSecret`; the default SQLite database is per pod and cannot coordinate multiple gateway replicas. The chart creates a headless peer Service for gateway-to-gateway relay traffic. StatefulSet pods use stable pod DNS names through that headless Service. Deployment pods advertise their pod IP with `OPENSHELL_PEER_ENDPOINT`, because Kubernetes does not assign stable per-pod DNS names to Deployment replicas. Gateway peer traffic uses Kubernetes ServiceAccount identity. Each gateway pod mounts a projected, pod-bound ServiceAccount token with audience `openshell-gateway-peer`; receiving replicas validate that token with the Kubernetes TokenReview API, verify the live pod UID and Helm selector labels, and authorize only the internal `PeerRelay` RPC. The chart does not create or accept a shared gateway peer Secret. With gateway TLS enabled, peer calls use the chart CA and client TLS Secret for server verification and mTLS. The client verifies the stable gateway Service DNS name while connecting directly to the owning pod. Custom TLS Secrets must include that Service DNS name in the server certificate and provide the CA and client credentials configured by `server.tls`. Set `autoscaling.enabled=true` to render an `autoscaling/v2` HorizontalPodAutoscaler for the gateway workload. The Deployment or StatefulSet then omits `spec.replicas`, and the chart applies its multi-replica checks to `autoscaling.maxReplicas`. CPU and memory targets need a matching `resources.requests` entry, or a `resources.limits` entry, which Kubernetes copies into the request. Add custom metrics from a metrics adapter with `autoscaling.metrics`. See the [High Availability guide](https://docs.nvidia.com/openshell/latest/kubernetes/high-availability) for the metrics to scale and alert on. Enabling autoscaling on an existing release removes `spec.replicas` from the workload in that upgrade. Kubernetes can reset the workload to one replica until the HPA scales it back to at least `autoscaling.minReplicas`, which disconnects supervisor sessions from the other gateway pods; they reconnect to the remaining replica. Enable autoscaling when you install the chart, or during a maintenance window. On an existing release, upgrade with `--reset-then-reuse-values`, which needs Helm 3.14 or later. With an older Helm, save the values with `helm get values -o yaml > values.yaml` and upgrade with `--reset-values -f values.yaml`. `--reuse-values` keeps the previous chart version's defaults, which lack the autoscaling values, including the scale-down `behavior`, when that release predates them. For background, refer to [Migrating Deployments and StatefulSets to horizontal autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#migrating-deployments-and-statefulsets-to-horizontal-autoscaling). Disabling autoscaling renders `spec.replicas` from `replicaCount` again, which defaults to 1, so set `replicaCount` to the replica count you want in that upgrade. ## Secret bootstrap By default, a pre-install/pre-upgrade hook Job runs `openshell-gateway generate-certs` to create the gateway's server/client mTLS Secrets and sandbox JWT signing Secret. The Job uses the gateway image itself, so air-gapped environments only need to mirror that one image (no separate openssl/alpine sidecar). When `certManager.enabled=true`, cert-manager owns the TLS Secrets and the chart runs the same hook in JWT-only mode because cert-manager does not create the sandbox JWT signing Secret. This precedence applies even if `pkiInitJob.enabled` remains true. Set `pkiInitJob.enabled=false` only when an external non-cert-manager TLS source manages TLS and you pre-create the sandbox JWT signing Secret. ## SPIFFE/SPIRE provider token grants Set `gatewayConfig.openshell.drivers.kubernetes.provider_spiffe_workload_api_socket_path` to let sandbox supervisors use SPIFFE JWT-SVIDs for dynamic provider token grants. The chart keeps supervisor-to-gateway authentication on gateway-minted sandbox JWTs and passes the configured socket path to the Kubernetes driver. ```yaml gatewayConfig: openshell.drivers.kubernetes: provider_spiffe_workload_api_socket_path: /spiffe-workload-api/spire-agent.sock ``` For local development, uncomment the SPIRE Helm releases in `skaffold.yaml` and add `ci/values-spire.yaml` to the OpenShell release values files. The gateway verifies supervisor JWT-SVIDs with JWT bundles fetched from the SPIFFE Workload API, so this path does not require access to the SPIRE OIDC discovery endpoint or its TLS CA. ## Values | Key | Type | Default | Description | |-----|------|---------|-------------| | affinity | object | `{}` | Affinity rules for the gateway pod. | | agentSandbox.preflight.enabled | bool | `true` | Check the live cluster for a supported Agent Sandbox API before rendering gateway resources. Disable only for offline rendering and linting. | | autoscaling.behavior | object | `{"scaleDown":{"policies":[{"periodSeconds":120,"type":"Pods","value":1}],"stabilizationWindowSeconds":300}}` | HPA scaling behavior. Scale-down hands the removed pod's supervisor sessions to the remaining replicas, so the default removes at most one replica every two minutes after a five-minute stabilization window. Helm merges maps: set autoscaling.behavior.scaleDown to null to drop the default. | | autoscaling.enabled | bool | `false` | Render a HorizontalPodAutoscaler and stop rendering spec.replicas. | | autoscaling.maxReplicas | int | `4` | Maximum gateway replicas. Each replica opens up to database_max_connections + database_lock_max_connections PostgreSQL connections (14 with the defaults; set them under gatewayConfig.openshell.gateway); size the database for rollouts at this count, as the High Availability guide describes. With a Deployment and the default pool sizes, the default of 4 needs `max_connections` of at least 126, more than the PostgreSQL default of 100. | | autoscaling.metrics | list | `[]` | Additional autoscaling/v2 MetricSpec entries appended verbatim, such as Pods metrics served by prometheus-adapter. | | autoscaling.minReplicas | int | `2` | Minimum gateway replicas. Use 2 or more to survive a pod failure. | | autoscaling.targetCPUUtilizationPercentage | int | `80` | Target average CPU utilization, as a percentage of resources.requests.cpu. Set to null to disable. Requires resources.requests.cpu, or resources.limits.cpu, which Kubernetes copies into the request. | | autoscaling.targetMemoryUtilizationPercentage | int | `nil` | Target average memory utilization, as a percentage of resources.requests.memory. Null disables it. Requires resources.requests.memory, or resources.limits.memory, which Kubernetes copies into the request. | | certManager.caSecretName | string | `"openshell-ca-tls"` | Secret created for the intermediate CA (Certificate with isCA: true). | | certManager.certificateDuration | string | `"8760h"` | Duration for cert-manager-issued certificates. | | certManager.certificateRenewBefore | string | `"720h"` | Renewal window for cert-manager-issued certificates. | | certManager.clientCaFromServerTlsSecret | bool | `true` | Mount gateway client CA from the internal server TLS secret's ca.crt. The internal server certificate is always signed by the chart CA — the same CA that signs the client (mTLS) certificate — so the default (true) is correct for all configurations, including when serverIssuerRef is set. Only set to false if you mount the client CA from a separate secret via server.tls.clientCaSecretName. | | certManager.enabled | bool | `false` | Create cert-manager Issuer and Certificate resources. When enabled, cert-manager owns TLS and the chart runs a JWT-only certgen hook to create the sandbox JWT signing Secret that cert-manager does not manage. | | certManager.serverDnsNames | list | `["openshell","openshell.openshell.svc","openshell.openshell.svc.cluster.local","localhost","openshell.localhost","*.openshell.localhost","host.docker.internal"]` | DNS SANs on the cert-manager-issued server certificate. | | certManager.serverIpAddresses | list | `["127.0.0.1"]` | IP SANs on the cert-manager-issued server certificate. | | certManager.serverIssuerRef | object | `{"group":"","kind":"","name":""}` | Override the issuerRef for the external server Certificate (e.g. a real LetsEncrypt/ACME ClusterIssuer for a publicly-trusted cert on an external hostname). When set, the chart creates a second server certificate from this issuer with only the hostnames in serverDnsNames; the internal server certificate is always signed by the chart's own CA. Leave name empty to use the chart CA for all server certificates (default). Requires certManager.enabled=true. | | credentialDrivers.vault.caConfigMapName | string | `""` | ConfigMap containing the private Vault/OpenBao CA certificate under the ca.crt key. Helm mounts it only when the Vault driver is selected. | | fullnameOverride | string | `""` | Override the full generated resource name. | | gateway.image.digest | string | `""` | Gateway image digest. When set, this takes precedence over tag. | | gateway.image.pullPolicy | string | `nil` | Gateway image pull policy. Empty uses global.image.pullPolicy. | | gateway.image.registry | string | `""` | Gateway image registry. Empty uses global.image.registry. | | gateway.image.repository | string | `"openshell/gateway"` | Gateway image repository. | | gateway.image.tag | string | `""` | Gateway image tag. Defaults to the chart appVersion when empty. | | gatewayConfig | object | `{"openshell":{"version":2}}` | Non-secret gateway application configuration. Top-level keys name TOML tables and are rendered into the mounted gateway.toml file. Kubernetes resource inputs remain outside this map; template expressions derive the corresponding runtime values from their resource owner. | | global.image.pullPolicy | string | `"IfNotPresent"` | Shared OpenShell image pull policy. Individual image pull policies take precedence. | | global.image.registry | string | `"ghcr.io/nvidia"` | Shared OpenShell image registry. Individual image registries take precedence. | | global.image.tag | string | `""` | Shared OpenShell image tag. Defaults to the chart appVersion when empty. | | grpcRoute.backendTLSPolicy.caCertificateConfigMapName | string | `""` | Name of the ConfigMap containing the CA certificate (key: ca.crt) used to validate the gateway pod's TLS certificate. Defaults to `-backend-ca` when empty. The certgen hook auto-creates this: with pkiInitJob (default), immediately on install/upgrade; with cert-manager, the hook polls for pkiInitJob.timeoutSeconds seconds waiting for cert-manager to issue the server certificate, then creates the ConfigMap. A single install usually succeeds; if cert-manager takes longer, increase pkiInitJob.timeoutSeconds. By default (pkiInitJob.failOnTimeout=true), the install fails if the timeout is reached; set failOnTimeout=false to allow the install to succeed and run `helm upgrade` after the certificate is issued. | | grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. | | grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (`..svc.cluster.local`) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. | | grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. | | grpcRoute.gateway.className | string | `"eg"` | GatewayClass for a chart-created Gateway. Set this to the name exposed by your Gateway API implementation. Envoy Gateway uses "eg" in the OpenShell examples. Ignored when gateway.create is false. | | grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. Use sectionName to select a specific listener when needed. | | grpcRoute.gateway.listener.allowedRoutes | string | `"Same"` | "Same" restricts attached routes to the release namespace; "All" allows any namespace. | | grpcRoute.gateway.listener.port | int | `80` | Listener port for the generated Gateway resource. Use 443 with protocol HTTPS. | | grpcRoute.gateway.listener.protocol | string | `"HTTP"` | Listener protocol for the generated Gateway resource: HTTP or HTTPS. HTTPS terminates TLS at the Gateway listener; pair it with server.disableTls=true so the proxy forwards plaintext to the gateway pod, and use OIDC for client identity (the gateway never sees the client certificate). | | grpcRoute.gateway.listener.tls.certificateRefs | list | `[]` | certificateRefs for the HTTPS listener. Required when protocol is HTTPS. Each entry needs a `name` pointing at a kubernetes.io/tls Secret in the Gateway's namespace. May reference a cert-manager-issued Secret or the existing openshell-server-tls Secret (its SANs must include the external hostname). | | grpcRoute.gateway.name | string | `""` | Name of the Gateway resource. Defaults to ``. Set a distinct name when the selected controller creates a proxy Service with the Gateway name. When gateway.create is false, set this to the name of the existing Gateway. | | grpcRoute.gateway.namespace | string | `""` | Namespace of the Gateway referenced by the GRPCRoute parentRef. Defaults to the release namespace. A chart-created Gateway always resides in the release namespace. | | grpcRoute.gateway.sectionName | string | `""` | Listener name on an existing Gateway. Leave empty to let Gateway API listener matching select compatible listeners. | | grpcRoute.hostnames | list | `[]` | Hostnames the GRPCRoute matches on. Leave empty to match all hosts. | | grpcRoute.replicaRouting.enabled | bool | `false` | Route requests carrying an `x-openshell-replica` header straight to that gateway replica. The CLI sets the header on long-lived sandbox connections (SSH, port forwards, exec) using the owner the gateway reports, so they skip the relay through a peer replica. Renders one Service per replica. Requires workload.kind=statefulset, workload.allowMultiReplicaStatefulSet=true, and at most 15 replicas. | | imagePullSecrets | list | `[]` | Image pull secrets attached to gateway and helper pods. | | nameOverride | string | `"openshell"` | Override the chart name used in generated resource names. | | networkPolicy.enabled | bool | `true` | Restrict SSH ingress on sandbox pods to the gateway. In managed mode, the driver applies the equivalent policy to each workspace namespace. | | nodeSelector | object | `{}` | Node selector for the gateway pod. | | openshiftRoute.annotations | object | `{}` | Extra annotations on the Route (e.g. haproxy.router.openshift.io/*). | | openshiftRoute.enabled | bool | `false` | Create an OpenShift Route with TLS passthrough. | | openshiftRoute.host | string | `""` | Hostname for the Route. Must match a SAN on the gateway's server cert. | | pkiInitJob.enabled | bool | `true` | Run a pre-install/pre-upgrade Job that creates gateway and client mTLS Secrets. When certManager.enabled=true, cert-manager owns TLS and this same hook runs in JWT-only mode even if pkiInitJob.enabled remains true. | | pkiInitJob.failOnTimeout | bool | `true` | Fail the helm install/upgrade if cert-manager does not issue the certificate within the polling timeout. When true (default), the install fails immediately if the timeout is reached, providing clear feedback that BackendTLSPolicy is non-functional. When false, the hook succeeds with a warning and you can run `helm upgrade` after cert-manager issues the certificate to create the backend CA ConfigMap. If you set this to false and see "TLS error: Secret is not supplied by SDS" when connecting to the gateway, check if the TLS secret exists and run `helm upgrade` to create the ConfigMap. | | pkiInitJob.securityContext.allowPrivilegeEscalation | bool | `false` | Whether the certgen hook container can gain additional privileges. | | pkiInitJob.securityContext.capabilities.drop | list | `["ALL"]` | Linux capabilities dropped from the certgen hook container. | | pkiInitJob.securityContext.runAsNonRoot | bool | `true` | Require the certgen hook container to run as a non-root user. | | pkiInitJob.securityContext.seccompProfile.type | string | `"RuntimeDefault"` | Seccomp profile applied to the certgen hook container. | | pkiInitJob.serverDnsNames | list | `[]` | Extra DNS SANs to append to the server certificate. | | pkiInitJob.serverIpAddresses | list | `[]` | Extra IP SANs to append to the server certificate. | | pkiInitJob.timeoutSeconds | int | `120` | Maximum time in seconds for the certgen hook to poll for cert-manager certificates. When using cert-manager with BackendTLSPolicy, the hook polls for this many seconds waiting for the certificate to be issued, then creates the backend CA ConfigMap. The Job deadline is set to (timeoutSeconds + 30) to allow time for ConfigMap creation and cleanup. Increase this if cert-manager takes longer than 120 seconds to issue certificates. | | podAnnotations | object | `{}` | Extra annotations to add to the gateway pod. | | podLabels | object | `{}` | Extra labels to add to the gateway pod. | | podLifecycle.terminationGracePeriodSeconds | int | `30` | Maximum time, in seconds, Kubernetes waits for the gateway to exit before killing it. The gateway exits as soon as shutdown completes; the limit covers supervisor session cleanup and draining queued OCSF records. | | podSecurityContext.fsGroup | int | `1000` | fsGroup assigned to the gateway pod. | | probes.liveness.failureThreshold | int | `3` | Liveness probe failure threshold before the container is restarted. | | probes.liveness.initialDelaySeconds | int | `2` | Liveness probe initial delay, in seconds. | | probes.liveness.periodSeconds | int | `5` | Liveness probe period, in seconds. | | probes.liveness.timeoutSeconds | int | `1` | Liveness probe timeout, in seconds. | | probes.readiness.failureThreshold | int | `3` | Readiness probe failure threshold before the pod is marked not ready. | | probes.readiness.initialDelaySeconds | int | `1` | Readiness probe initial delay, in seconds. | | probes.readiness.periodSeconds | int | `2` | Readiness probe period, in seconds. | | probes.readiness.timeoutSeconds | int | `1` | Readiness probe timeout, in seconds. | | probes.startup.failureThreshold | int | `30` | Startup probe failure threshold before the container is killed. | | probes.startup.periodSeconds | int | `2` | Startup probe period, in seconds. | | probes.startup.timeoutSeconds | int | `1` | Startup probe timeout, in seconds. | | rbac.clusterScoped.clusterRoleBindingName | string | `""` | Name for the ClusterRoleBinding. Empty uses the `-node-reader-` default. | | rbac.clusterScoped.clusterRoleName | string | `""` | Name for the ClusterRole. Empty uses the `-node-reader-` default. | | rbac.clusterScoped.create | bool | `true` | Create the cluster-scoped ClusterRole and ClusterRoleBinding. Disable for a namespace-admin install where a cluster-admin applies them separately; the gateway ServiceAccount name and namespace are unchanged, so a pre-created ClusterRoleBinding still matches. | | rbac.create | bool | `true` | Create the RBAC objects that grant the gateway ServiceAccount access. Disable to supply the namespaced sandbox and peer Role/RoleBinding and the cluster-scoped ClusterRole/ClusterRoleBinding out of band. The certgen hook and credential driver RBAC keep their own flags. | | replicaCount | int | `1` | Number of OpenShell gateway replicas. Values greater than 1 require server.externalDbSecret because the default SQLite backend is per pod. | | resources | object | `{}` | Gateway pod resource requests and limits. | | sandbox.image.digest | string | `""` | Sandbox image digest. When set, this takes precedence over tag. | | sandbox.image.pullPolicy | string | `nil` | Sandbox image pull policy. Leave unset to use the Kubernetes image default. | | sandbox.image.repository | string | `"nvcr.io/nvidia/base/ubuntu"` | Default standalone sandbox image repository. | | sandbox.image.tag | string | `"24.04"` | Sandbox image tag. Defaults to latest when empty. | | sandboxRuntime.image.digest | string | `""` | Sandbox runtime image digest. When set, this takes precedence over tag. | | sandboxRuntime.image.pullPolicy | string | `nil` | Sandbox runtime image pull policy. Empty uses global.image.pullPolicy. | | sandboxRuntime.image.registry | string | `""` | Sandbox runtime image registry. Empty uses global.image.registry. | | sandboxRuntime.image.repository | string | `"openshell/sandbox"` | Sandbox runtime image repository. | | sandboxRuntime.image.tag | string | `""` | Sandbox runtime image tag. Defaults to the chart appVersion when empty. | | sandboxServiceAccount.annotations | object | `{}` | Annotations to add to the generated sandbox service account. | | sandboxServiceAccount.create | bool | `true` | Create a service account for sandbox pods. | | sandboxServiceAccount.name | string | `""` | Existing service account name for sandbox pods when sandboxServiceAccount.create is false. | | securityContext.allowPrivilegeEscalation | bool | `false` | Whether the gateway container can gain additional privileges. | | securityContext.capabilities.drop | list | `["ALL"]` | Linux capabilities dropped from the gateway container. | | securityContext.runAsNonRoot | bool | `true` | Require the gateway container to run as a non-root user. | | securityContext.runAsUser | int | `1000` | UID assigned to the gateway container. | | securityContext.seccompProfile.type | string | `"RuntimeDefault"` | Seccomp profile applied to the gateway container. | | server.auth.allowUnauthenticatedUsers | bool | `false` | UNSAFE: accept unauthenticated CLI/user requests as a local developer principal. Intended only for trusted local Skaffold/k3d development or a fully trusted fronting proxy. Leave false for shared or production clusters. | | server.credentialDrivers.kubernetesSecrets.createNamespace | bool | `false` | Create the credential namespace. Requires a namespace other than the release namespace. The Namespace is retained on uninstall so stored credentials survive; an existing Namespace not owned by this release is left untouched. | | server.credentialDrivers.kubernetesSecrets.enabled | bool | `false` | Enable the in-tree Kubernetes Secret credential driver. WARNING: The RBAC Role grants read/write access to ALL Secrets in the configured namespace. Use a dedicated namespace to limit blast radius. | | server.credentialDrivers.kubernetesSecrets.namespace | string | `""` | Namespace where OpenShell-managed provider Secret objects are stored. Empty = Helm release namespace. A dedicated namespace is RECOMMENDED to isolate OpenShell-managed Secrets from other workloads. | | server.credentialDrivers.kubernetesSecrets.rbac.create | bool | `true` | Create a Role/RoleBinding granting the gateway ServiceAccount read/write access to managed provider Secrets. | | server.credentialDrivers.vault.address | string | `""` | Vault service base URL. Non-loopback endpoints must use HTTPS, for example https://vault.vault.svc.cluster.local:8200. | | server.credentialDrivers.vault.authMethod | string | `"kubernetes"` | Authentication method. Use "kubernetes" in-cluster or "token_file" for local/dev validation. | | server.credentialDrivers.vault.caConfigMapName | string | `""` | ConfigMap containing the private Vault CA certificate bundle in the ca.crt key. Leave empty to use platform trust roots. | | server.credentialDrivers.vault.enabled | bool | `false` | Enable the in-tree Vault credential driver. | | server.credentialDrivers.vault.kubernetesAuthMount | string | `"kubernetes"` | Vault Kubernetes auth mount. | | server.credentialDrivers.vault.kvVersion | string | `"2"` | Default KV engine version. Use "1" or "2". | | server.credentialDrivers.vault.mount | string | `"secret"` | Default KV mount name. | | server.credentialDrivers.vault.role | string | `""` | Vault Kubernetes auth role when authMethod is kubernetes. | | server.credentialDrivers.vault.serviceAccountTokenPath | string | `"/var/run/secrets/kubernetes.io/serviceaccount/token"` | ServiceAccount token path used for Kubernetes auth. | | server.credentialDrivers.vault.timeoutSecs | string | `""` | HTTP request timeout in seconds. Empty = driver default. | | server.credentialDrivers.vault.tokenPath | string | `""` | Mounted token file path when authMethod is token_file. | | server.credentialStorage.existingSecret | string | `""` | Name of a pre-existing Secret containing the key-encryption key. When set, the chart does NOT generate a new Secret; it references this one instead. The Secret must contain a key named "key-encryption-key" with a base64-encoded 32-byte value. Required for GitOps workflows that render manifests with `helm template` (where `lookup` is unavailable). | | server.dataStorageClass | string | `""` | Kubernetes StorageClass for the gateway data PVC (StatefulSet workload only). Empty (default) = omit storageClassName, using the cluster's default StorageClass. Set this on clusters with no default StorageClass, otherwise the gateway PVC stays Pending. Use a block-backed class, not NFS or other network filesystems. Immutable after install. | | server.dataStorageSize | string | `""` | Storage request for the gateway data PVC, `openshell-data` (StatefulSet workload only). Uses Kubernetes quantity syntax and requires an explicit unit ("20Gi", not "20"). Empty = built-in default (1Gi). Growing an existing release's claim is PVC expansion, not a values change. | | server.dbUrl | string | `"sqlite:/var/openshell/openshell.db"` | Gateway database URL (used for the default SQLite backend). SQLite runs in WAL mode and needs a local block-backed volume, not NFS or other network filesystems. | | server.defaultRuntimeClassName | string | `""` | Default Kubernetes runtimeClassName for sandbox pods. Applied when a CreateSandbox request does not specify one. Empty (default) = omit the field, using the cluster's default RuntimeClass. Set to a RuntimeClass name (e.g. "kata-containers", "nvidia") to apply it to all sandboxes that don't explicitly override it. | | server.disableTls | bool | `false` | Disable TLS entirely - the server listens on plaintext HTTP. Set to true when a reverse proxy / tunnel terminates TLS at the edge. | | server.drivers.kubernetes.allowDriverConfig | bool | `false` | Allow caller driver JSON; external resources still require approval. | | server.drivers.kubernetes.operatorNamespaceFile | string | `""` | Path to a JSON file containing an array of namespace names allowed in operator mode. Hot-reloaded on change. | | server.drivers.kubernetes.operatorNamespaceLabel | string | `""` | K8s label selector for namespace discovery in operator mode. The driver watches namespaces matching this label. | | server.drivers.kubernetes.resourceAdmission.enabled | bool | `true` | Require operator approval labels on external sandbox attachments (GPU attachments exempt). | | server.drivers.kubernetes.resourceAdmission.requiredLabels | string | `nil` | Replacement label map; null uses the built-in admission labels. Empty map is invalid when enabled. | | server.drivers.kubernetes.workspaceMode | string | `"shared"` | How workspaces map to Kubernetes namespaces. "shared" (default): all sandboxes in a single namespace. "managed": auto-creates per-workspace namespaces. "operator": uses pre-provisioned namespaces. | | server.enableLoopbackServiceHttp | bool | `true` | Enable plaintext HTTP routing for loopback sandbox service URLs on TLS-enabled gateways. | | server.enableUserNamespaces | bool | `false` | Enable Kubernetes user namespace isolation (hostUsers: false) for sandbox pods. Requires Kubernetes 1.33+ with user namespace support available (beta through 1.35, GA in 1.36+), plus a supporting container runtime and Linux 5.12+. When enabled, container UID 0 maps to an unprivileged host UID and capabilities become namespaced. | | server.enableWebsocketTunnel | bool | `false` | Enable the WebSocket tunnel used by CLI/SDK clients behind an authenticated edge proxy. Leave disabled for direct gateway installs. | | server.externalDbSecret | string | `""` | Name of a pre-existing Opaque Secret containing a PostgreSQL connection URI (key: uri). When set, the gateway reads OPENSHELL_DB_URL from this Secret instead of using dbUrl. The Secret must contain a `uri` key, e.g. postgresql://user:pass@host:5432/dbname. | | server.extraVolumeMounts | list | `[]` | Additional volume mounts for the gateway container. | | server.extraVolumes | list | `[]` | Additional volumes for the gateway pod. | | server.grpcEndpoint | string | `""` | gRPC endpoint sandboxes call back into the gateway. Leave empty to derive it from the chart fullname, release namespace, service port, and disableTls flag, for example https://openshell.openshell.svc.cluster.local:8080. Override only when sandboxes must reach the gateway via a different hostname (e.g. an external ingress or a host alias). | | server.grpcRateLimit.requests | int | `0` | Maximum gRPC requests allowed per window. Must be positive (alongside windowSeconds) to enable rate limiting; 0 (default) disables it. | | server.grpcRateLimit.windowSeconds | int | `0` | gRPC rate-limit window length in seconds. Must be positive (alongside requests) to enable rate limiting; 0 (default) disables it. | | server.hostGatewayIP | string | `""` | Host gateway IP for sandbox pod hostAliases. When set, sandbox pods get hostAliases entries mapping host.docker.internal and host.openshell.internal to this IP, allowing them to reach services running on the Docker host. Auto-detected by the cluster entrypoint script. | | server.logLevel | string | `"info"` | Gateway log level. | | server.name | string | `""` | Operator-facing gateway name. Defaults to the chart fullname so all replicas in one installation share an identity. Set explicitly when one telemetry collector receives spans from multiple namespaces or clusters. | | server.ocsfLog.enabled | bool | `false` | Write gateway OCSF events as JSONL. | | server.ocsfLog.maxFiles | int | `7` | Rotated files retained when rotation is daily. | | server.ocsfLog.path | string | `"/tmp/gateway-ocsf.jsonl"` | OCSF JSONL path. The default is writable in the gateway container but does not persist across restarts. To keep records, mount a volume with server.extraVolumes and server.extraVolumeMounts and set a path on it. When replicas share the volume, set subPathExpr: $(OPENSHELL_POD_NAME) on the mount so each replica writes its own file. | | server.ocsfLog.queueCapacity | int | `10000` | Maximum records waiting for the file writer. | | server.ocsfLog.queueMaxBytes | int | `16777216` | Maximum encoded bytes waiting for the file writer. | | server.ocsfLog.rotation | string | `"daily"` | Rotate the active file daily in UTC, or never. | | server.ocsfLog.schemaVersion | string | `""` | Optional OCSF downgrade target. Empty emits native OCSF 1.8.0. Supported values: "1.1", "1.3". | | server.oidc.adminRole | string | `""` | Role name for admin access. Leave empty (with userRole also empty) for authentication-only mode. Both must be set or both empty. | | server.oidc.audience | string | `"openshell-cli"` | Expected audience claim for the API resource server. This should match the server's --oidc-audience, NOT the CLI client ID. | | server.oidc.caConfigMapName | string | `""` | Name of a ConfigMap containing a CA certificate bundle (key: ca.crt) for verifying the OIDC issuer's TLS certificate. Required when the issuer uses a non-public CA (e.g. OpenShift ingress, private PKI). | | server.oidc.dangerouslyAllowInsecureHttp | bool | `false` | Development only: permit cleartext OIDC requests to numeric loopback addresses. This never permits HTTP to hostnames or non-loopback addresses. | | server.oidc.issuer | string | `""` | OIDC issuer URL (e.g. https://keycloak.example.com/realms/openshell). | | server.oidc.jwksAllowedOrigins | list | `[]` | Additional trusted HTTPS origins allowed to serve JWKS. The issuer origin is always allowed. Entries must not include a path or query. | | server.oidc.jwksTtl | int | `3600` | JWKS key cache TTL in seconds. Must be greater than zero. | | server.oidc.rolesClaim | string | `""` | Dot-separated path to the roles array in the JWT claims. Keycloak: "realm_access.roles", Entra ID: "roles", Okta: "groups". | | server.oidc.scopesClaim | string | `""` | Dot-separated path to the scopes array in the JWT claims. | | server.oidc.userRole | string | `""` | Role name for standard user access. | | server.otlp.endpoint | string | `""` | OTLP/gRPC collector endpoint, conventionally using port 4317. | | server.otlp.serviceName | string | `""` | Gateway OpenTelemetry service name. Empty uses openshell-gateway. | | server.policyValidationFailureMode | string | `"fail_closed"` | Posture when a candidate sandbox policy fails validation. `fail_closed` deactivates the previous policy; `retain_last_valid` keeps it active. | | server.providerTokenGrants.spiffe.enabled | bool | `false` | Mount the SPIFFE Workload API socket into gateway and sandbox pods for dynamic provider token grants. | | server.providerTokenGrants.spiffe.workloadApiSocketPath | string | `"/spiffe-workload-api/spire-agent.sock"` | Path to the SPIFFE Workload API socket mounted into gateway and sandbox pods. | | server.sandboxGid | string | `""` | GID for sandbox pods (`sandbox_gid`). Empty (default) = same as sandboxUid. Must be an integer between 1 and 4294967294. | | server.sandboxImagePullSecrets | list | `[]` | Image pull secrets attached to sandbox pods. Referenced Secrets must exist in the sandbox namespace. | | server.sandboxJwt.gatewayId | string | `""` | Stable gateway identity embedded in iss/aud of every minted token. Defaults to the release name so HA replicas share identity. | | server.sandboxJwt.k8sSaTokenTtlSecs | int | `3600` | Lifetime (seconds) of the projected ServiceAccount token kubelet writes into each sandbox pod for the IssueSandboxToken bootstrap exchange. Kubelet enforces a minimum of 600s; the driver clamps values outside [600, 86400]. Default 3600 — generous, since the supervisor consumes the token within seconds of pod start. | | server.sandboxJwt.secretDefaultMode | string | `""` | File mode for the mounted JWT signing key Secret. Default 0400 (owner-read only). Override to 0440 or 0444 if the container UID does not match the volume file owner. | | server.sandboxJwt.signingSecretName | string | `""` | Name of the Opaque Secret holding the signing key material. Empty falls back to the chart fullname with "-jwt-keys" appended. | | server.sandboxJwt.ttlSecs | int | `3600` | Token TTL in seconds. Defaults to 3600 (1h). | | server.sandboxNamespace | string | `""` | Namespace where sandbox pods are created. Defaults to the Helm release namespace (.Release.Namespace) when left empty. | | server.sandboxUid | string | `""` | UID for sandbox pods (`sandbox_uid`). Empty (default) = use the OpenShift SCC namespace annotation if present, otherwise the driver default. Must be an integer between 1 and 4294967294. | | server.telemetryEnabled | bool | `true` | Enable anonymous OpenShell telemetry from the gateway and the sandbox supervisors it launches. | | server.tls.certSecretName | string | `"openshell-server-tls"` | K8s secret (type kubernetes.io/tls) with tls.crt and tls.key for the server. | | server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Only used when enableMtls is true. Set to "" to disable client certificate verification for HTTPS-only mode. | | server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. | | server.tls.enableMtls | bool | `true` | Enable mTLS client certificate authentication. When false, the gateway runs HTTPS-only without requiring client certificates (use OIDC for auth instead). Must be false when using BackendTLSPolicy because ingress proxies cannot present client certificates to the backend. | | server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). | | server.workspaceStorageClass | string | `""` | Kubernetes StorageClass for the workspace PVC in sandbox pods. Empty (default) = omit storageClassName, using the cluster's default StorageClass. Set this on clusters with no default StorageClass, otherwise the workspace PVC stays Pending and the sandbox never starts. | | service.healthPort | int | `8081` | Gateway health service port. | | service.metricsPort | int | `9090` | Gateway metrics service port. | | service.port | int | `8080` | Gateway gRPC/HTTP service port. | | service.type | string | `"ClusterIP"` | Kubernetes Service type for the gateway. | | serviceAccount.annotations | object | `{}` | Annotations to add to the generated service account. | | serviceAccount.create | bool | `true` | Create a service account for the gateway. | | serviceAccount.name | string | `""` | Existing service account name to use when serviceAccount.create is false. | | supervisor.image.digest | string | `""` | Supervisor image digest. When set, this takes precedence over tag. | | supervisor.image.pullPolicy | string | `nil` | Supervisor image pull policy. Empty uses global.image.pullPolicy. Prefer always, if_not_present, or never; the chart also accepts legacy Kubernetes spellings Always, IfNotPresent, and Never. | | supervisor.image.registry | string | `""` | Supervisor image registry. Empty uses global.image.registry. | | supervisor.image.repository | string | `"openshell/supervisor"` | Supervisor image repository. | | supervisor.image.tag | string | `""` | Supervisor image tag. Defaults to the chart appVersion when empty. | | supervisor.sandboxRuntime.boundaryPort | int | `5500` | Workload boundary TLS listener port. | | tolerations | list | `[]` | Tolerations for the gateway pod. | | upstreamProxy | object | `{"authAllowInsecure":false,"authSecret":{"key":"","name":""},"caBundle":{"configMapName":"","key":"ca.crt"},"connectByHostname":false,"noProxy":"","url":""}` | Operator-owned corporate forward proxy for policy-approved TLS egress from Kubernetes sandboxes. The workload cannot select or override it. | | upstreamProxy.authAllowInsecure | bool | `false` | Required when authSecret is configured because Basic auth to an HTTP proxy is cleartext. | | upstreamProxy.authSecret.key | string | `""` | Secret key containing the proxy credential. | | upstreamProxy.authSecret.name | string | `""` | Existing Secret in the sandbox namespace containing a user:pass value. | | upstreamProxy.caBundle.configMapName | string | `""` | ConfigMap in the release namespace holding the corporate proxy CA bundle. Required for an https:// proxy with a private CA, and for a TLS-intercepting proxy that re-signs upstream certificates. The gateway reads it and stages it into each sandbox's immutable supervisor bootstrap Secret, so the anchor stays in the gateway's trust domain rather than the workload namespace. Supply only the CA that signs your proxy's certificate, or that the proxy re-signs intercepted upstream certificates with. Public roots already come from the supervisor image and its TLS stack, so a full merged trust bundle (for example an OpenShift config.openshift.io/inject-trusted-cabundle ConfigMap) adds hundreds of kilobytes of duplicated roots and can exceed the sandbox boundary's control-frame budget. | | upstreamProxy.caBundle.key | string | `"ca.crt"` | Key inside that ConfigMap. Change this only to reuse an existing ConfigMap whose key is not ca.crt. | | upstreamProxy.connectByHostname | bool | `false` | Last-resort option for hostname-filtering proxy ACLs. It lets the proxy resolve CONNECT targets. | | upstreamProxy.noProxy | string | `""` | Comma-separated destinations that bypass only the corporate proxy. | | upstreamProxy.url | string | `""` | Proxy URL in http://host:port or https://host:port form. An https:// proxy whose certificate is not publicly trusted also needs caBundle below. | | workload.allowMultiReplicaStatefulSet | bool | `false` | Allow replicaCount > 1 while rendering a StatefulSet. Prefer workload.kind=deployment for external database-backed multi-replica gateways; this override exists for operators who explicitly require StatefulSet identity or storage semantics. | | workload.kind | string | `"statefulset"` | Gateway workload controller kind. Use `statefulset` for the default SQLite database, or `deployment` when server.externalDbSecret points at an external database. | | workspaceResources.enabled | bool | `true` | Create the sandbox ServiceAccount, Role, RoleBinding, and NetworkPolicy from this chart. Disable for a gateway-only release. | ---------------------------------------------- Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)