## File: README.md [[⬇️ **Download]**](https://github.com/containerd/nerdctl/releases) [[📖 **Command reference]**](./docs/command-reference.md) [[❓**FAQs & Troubleshooting]**](./docs/faq.md) [[📚 **Additional documents]**](#additional-documents) # nerdctl: Docker-compatible CLI for containerd `nerdctl` is a Docker-compatible CLI for [contai**nerd**](https://containerd.io). ✅ Same UI/UX as `docker` ✅ Supports Docker Compose (`nerdctl compose up`) ✅ [Optional] Supports [rootless mode, without slirp overhead (bypass4netns)](./docs/rootless.md) ✅ [Optional] Supports lazy-pulling ([Stargz](./docs/stargz.md), [Nydus](./docs/nydus.md), [OverlayBD](./docs/overlaybd.md)) ✅ [Optional] Supports [encrypted images (ocicrypt)](./docs/ocicrypt.md) ✅ [Optional] Supports [P2P image distribution (IPFS)](./docs/ipfs.md) (\*1) ✅ [Optional] Supports [container image signing and verifying (cosign)](./docs/cosign.md) nerdctl is a **non-core** sub-project of containerd. \*1: P2P image distribution (IPFS) is completely optional. Your host is NOT connected to any P2P network, unless you opt in to [install and run IPFS daemon](https://docs.ipfs.io/install/). ## Examples ### Basic usage To run a container with the default `bridge` CNI network (10.4.0.0/24): ```console # nerdctl run -it --rm alpine ``` To build an image using BuildKit: ```console # nerdctl build -t foo /some-dockerfile-directory # nerdctl run -it --rm foo ``` To build and send output to a local directory using BuildKit: ```console # nerdctl build -o type=local,dest=. /some-dockerfile-directory ``` To run containers from `docker-compose.yaml`: ```console # nerdctl compose -f ./examples/compose-wordpress/docker-compose.yaml up ``` See also [`./examples/compose-wordpress`](./examples/compose-wordpress). ### Debugging Kubernetes To list local Kubernetes containers: ```console # nerdctl --namespace k8s.io ps -a ``` To build an image for local Kubernetes without using registry: ```console # nerdctl --namespace k8s.io build -t foo /some-dockerfile-directory # kubectl apply -f - < In addition to containerd, the following components should be installed: - [CNI plugins](https://github.com/containernetworking/plugins): for using `nerdctl run`. - v1.1.0 or later is highly recommended. - [BuildKit](https://github.com/moby/buildkit) (OPTIONAL): for using `nerdctl build`. BuildKit daemon (`buildkitd`) needs to be running. See also [the document about setting up BuildKit](./docs/build.md). - v0.11.0 or later is highly recommended. Some features, such as pruning caches with `nerdctl system prune`, do not work with older versions. - [RootlessKit](https://github.com/rootless-containers/rootlesskit) (OPTIONAL): for [Rootless mode](./docs/rootless.md) - RootlessKit needs to be v0.10.0 or later. v3.0.0 or later is recommended. These dependencies are included in `nerdctl-full---.tar.gz`, but not included in `nerdctl---.tar.gz`. ### Brew On Linux systems you can install nerdctl via [brew](https://brew.sh): ```bash brew install nerdctl ``` This is currently not supported for macOS. The section below shows how to install on macOS using brew. ### macOS [Lima](https://github.com/lima-vm/lima) project provides Linux virtual machines for macOS, with built-in integration for nerdctl. ```console $ brew install lima $ limactl start $ lima nerdctl run -d --name nginx -p 127.0.0.1:8080:80 nginx:alpine ``` ### Windows Install with [Scoop](https://scoop.sh): ``` scoop install nerdctl ``` Regarding compatibility, note that: - Linux containers: Known to work on WSL2 - Windows containers: experimental support for Windows (see below for features that are currently known to work) ### FreeBSD See [`./docs/freebsd.md`](docs/freebsd.md). ### Docker To run containerd and nerdctl inside Docker: ```bash docker build -t nerdctl . docker run -it --rm --privileged nerdctl ``` ## Motivation The goal of `nerdctl` is to facilitate experimenting the cutting-edge features of containerd that are not present in Docker (see below). Note that competing with Docker is _not_ the goal of `nerdctl`. Those cutting-edge features are expected to be eventually available in Docker as well. Also, `nerdctl` might be potentially useful for debugging Kubernetes clusters, but it is not the primary goal. ## Features present in `nerdctl` but not present in Docker Major: - On-demand image pulling (lazy-pulling) using [Stargz](./docs/stargz.md)/[Nydus](./docs/nydus.md)/[OverlayBD](./docs/overlaybd.md)/[SOCI](./docs/soci.md) Snapshotter: `nerdctl --snapshotter=stargz|nydus|overlaybd|soci run IMAGE` . - [Image encryption and decryption using ocicrypt (imgcrypt)](./docs/ocicrypt.md): `nerdctl image (encrypt|decrypt) SRC DST` - [P2P image distribution using IPFS](./docs/ipfs.md): `nerdctl run ipfs://CID` . P2P image distribution (IPFS) is completely optional. Your host is NOT connected to any P2P network, unless you opt in to [install and run IPFS daemon](https://docs.ipfs.io/install/). - [Cosign integration](./docs/cosign.md): `nerdctl pull --verify=cosign` and `nerdctl push --sign=cosign`, and [in Compose](./docs/cosign.md#cosign-in-compose) - [Accelerated rootless containers using bypass4netns](./docs/rootless.md): `nerdctl run --annotation nerdctl/bypass4netns=true` Minor: - Namespacing: `nerdctl --namespace= ps` . (NOTE: All Kubernetes containers are in the `k8s.io` containerd namespace regardless to Kubernetes namespaces) - Exporting Docker/OCI dual-format archives: `nerdctl save` . - Importing OCI archives as well as Docker archives: `nerdctl load` . - Specifying a non-image rootfs: `nerdctl run -it --rootfs /bin/sh` . The CLI syntax conforms to Podman convention. - Connecting a container to multiple networks at once: `nerdctl run --net foo --net bar` - Running [FreeBSD jails](./docs/freebsd.md). - Better multi-platform support, e.g., `nerdctl pull --all-platforms IMAGE` - Applying an (existing) AppArmor profile to rootless containers: `nerdctl run --security-opt apparmor=`. Use `sudo nerdctl apparmor load` to load the `nerdctl-default` profile. - Systemd compatibility support: `nerdctl run --systemd=always` Trivial: - Inspecting raw OCI config: `nerdctl container inspect --mode=native` . ## Features implemented in `nerdctl` ahead of Docker - Recursive read-only (RRO) bind-mount: `nerdctl run -v /mnt:/mnt:rro` (make children such as `/mnt/usb` to be read-only, too). Requires kernel >= 5.12. The same feature was later introduced in Docker v25 with a different syntax: read-only mounts are now recursively read-only by default when supported, and the behavior is customizable with `--mount type=bind,...,readonly,bind-recursive=`. nerdctl now supports the Docker v25 syntax too, and the old `rro` syntax is deprecated. ## Similar tools - [`ctr`](https://github.com/containerd/containerd/tree/main/cmd/ctr): incompatible with Docker CLI, and not friendly to users. Notably, `ctr` lacks the equivalents of the following nerdctl commands: - `nerdctl run -p ` - `nerdctl run --restart=always --net=bridge` - `nerdctl pull` with `~/.docker/config.json` and credential helper binaries such as `docker-credential-ecr-login` - `nerdctl logs` - `nerdctl build` - `nerdctl compose up` - [`crictl`](https://github.com/kubernetes-sigs/cri-tools): incompatible with Docker CLI, not friendly to users, and does not support non-CRI features - [k3c v0.2 (abandoned)](https://github.com/rancher/k3c/tree/v0.2.1): needs an extra daemon, and does not support non-CRI features - [Rancher Kim (nee k3c v0.3)](https://github.com/rancher/kim): needs Kubernetes, and only focuses on image management commands such as `kim build` and `kim push` - [PouchContainer (abandoned?)](https://github.com/alibaba/pouch): needs an extra daemon ## Developer guide nerdctl is a containerd **non-core** sub-project, licensed under the [Apache 2.0 license](./LICENSE). As a containerd non-core sub-project, you will find the: - [Project governance](https://github.com/containerd/project/blob/main/GOVERNANCE.md), - [Maintainers](./MAINTAINERS), - and [Contributing guidelines](https://github.com/containerd/project/blob/main/CONTRIBUTING.md) information in our [`containerd/project`](https://github.com/containerd/project) repository. ### Compiling nerdctl from source Run `make && sudo make install`. See the header of [`go.mod`](./go.mod) for the minimum supported version of Go. Using `go install github.com/containerd/nerdctl/v2/cmd/nerdctl` is possible, but unrecommended because it does not fill version strings printed in `nerdctl version` ### Testing See [testing nerdctl](docs/testing/README.md). ### Contributing to nerdctl Lots of commands and flags are currently missing. Pull requests are highly welcome. Please certify your [Developer Certificate of Origin (DCO)](https://developercertificate.org/), by signing off your commit with `git commit -s` and with your real name. # Command reference Moved to [`./docs/command-reference.md`](./docs/command-reference.md) # Additional documents Configuration guide: - [`./docs/config.md`](./docs/config.md): Configuration (`/etc/nerdctl/nerdctl.toml`, `~/.config/nerdctl/nerdctl.toml`) - [`./docs/registry.md`](./docs/registry.md): Registry authentication (`~/.docker/config.json`) Basic features: - [`./docs/compose.md`](./docs/compose.md): Compose - [`./docs/rootless.md`](./docs/rootless.md): Rootless mode - [`./docs/cni.md`](./docs/cni.md): CNI for containers network - [`./docs/build.md`](./docs/build.md): `nerdctl build` with BuildKit Advanced features: - [`./docs/stargz.md`](./docs/stargz.md): Lazy-pulling using Stargz Snapshotter - [`./docs/nydus.md`](./docs/nydus.md): Lazy-pulling using Nydus Snapshotter - [`./docs/soci.md`](./docs/soci.md): Lazy-pulling using SOCI Snapshotter - [`./docs/overlaybd.md`](./docs/overlaybd.md): Lazy-pulling using OverlayBD Snapshotter - [`./docs/ocicrypt.md`](./docs/ocicrypt.md): Running encrypted images - [`./docs/gpu.md`](./docs/gpu.md): Using GPUs inside containers - [`./docs/multi-platform.md`](./docs/multi-platform.md): Multi-platform mode Experimental features: - [`./docs/experimental.md`](./docs/experimental.md): Experimental features - [`./docs/freebsd.md`](./docs/freebsd.md): Running FreeBSD jails - [`./docs/ipfs.md`](./docs/ipfs.md): Distributing images on IPFS - [`./docs/builder-debug.md`](./docs/builder-debug.md): Interactive debugging of Dockerfile Implementation details: - [`./docs/dir.md`](./docs/dir.md): Directory layout (`/var/lib/nerdctl`) Misc: - [`./docs/faq.md`](./docs/faq.md): FAQs and Troubleshooting --- ## File: docs/dev/auditing_dockerfile.md # Auditing dockerfile Because of the nature of GitHub cache, and the time it takes to build the dockerfile for testing, it is desirable to be able to audit what is going on there. This document provides a few pointers on how to do that, and some results as of 2025-02-26 (run inside lima, nerdctl main, on a macbook pro M1). ## Intercept network traffic ### On macOS Use Charles: - start SSL proxying - enable SOCKS proxy - export the root certificate ### On linux Left as an exercise to the reader. ### If using lima - restart your lima instance with `HTTP_PROXY=http://X.Y.Z.W:8888 HTTPS_PROXY=socks5://X.Y.Z.W:8888 limactl start instance` - where XYZW is the local ip of the Charles proxy (non-localhost) ### On the host where you are running containerd - copy the root certificate from above into `/usr/local/share/ca-certificates/charles-ssl-proxying-certificate.crt` - update your host: `sudo update-ca-certificates` - now copy the root certificate again to your current nerdctl clone ### Hack the dockerfile to insert our certificate Add the following stages in the dockerfile: ```dockerfile FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-trixie AS hack-build-base-debian RUN apt-get update -qq; apt-get -qq install ca-certificates COPY charles-ssl-proxying-certificate.crt /usr/local/share/ca-certificates/ RUN update-ca-certificates FROM --platform=$BUILDPLATFORM golang:${GO_VERSION}-alpine AS hack-build-base RUN apk add --no-cache ca-certificates COPY charles-ssl-proxying-certificate.crt /usr/local/share/ca-certificates/ RUN update-ca-certificates FROM ubuntu:${UBUNTU_VERSION} AS hack-base RUN apt-get update -qq; apt-get -qq install ca-certificates COPY charles-ssl-proxying-certificate.crt /usr/local/share/ca-certificates/ RUN update-ca-certificates ``` Then replace any later "FROM" with our modified bases: ``` golang:${GO_VERSION}-trixie => hack-build-base-debian golang:${GO_VERSION}-alpine => hack-build-base ubuntu:${UBUNTU_VERSION} => hack-base ``` ## Mimicking what the CI is doing A quick helper: ```bash run(){ local no_cache="${1:-}" local platform="${2:-arm64}" local dockerfile="${3:-Dockerfile}" local target="${4:-test-integration}" local cache_shard="$CONTAINERD_VERSION"-"$platform" local shard="$cache_shard"-"$target"-"$UBUNTU_VERSION"-"$no_cache"-"$dockerfile" local cache_location=$HOME/bk-cache-"$cache_shard" local destination=$HOME/bk-output-"$shard" local logs="$HOME"/bk-debug-"$shard" if [ "$no_cache" != "" ]; then nerdctl system prune -af nerdctl builder prune -af rm -Rf "$cache_location" fi nerdctl build \ --build-arg UBUNTU_VERSION="$UBUNTU_VERSION" \ --build-arg CONTAINERD_VERSION="$CONTAINERD_VERSION" \ --platform="$platform" \ --output type=tar,dest="$destination" \ --progress plain \ --build-arg HTTP_PROXY=$HTTP_PROXY \ --build-arg HTTPS_PROXY=$HTTPS_PROXY \ --cache-to type=local,dest="$cache_location",mode=max \ --cache-from type=local,src="$cache_location" \ --target "$target" \ -f "$dockerfile" . 2>&1 | tee "$logs" } ``` And here is what the CI is doing: ```bash ci_run(){ local no_cache="${1:-}" export UBUNTU_VERSION=24.04 # The actual version may differ CONTAINERD_VERSION=v1.7.25 run "$no_cache" arm64 Dockerfile.origin build-dependencies UBUNTU_VERSION=22.04 CONTAINERD_VERSION=v1.7.25 run "" arm64 Dockerfile.origin test-integration CONTAINERD_VERSION=v2.0.3 run "$no_cache" arm64 Dockerfile.origin build-dependencies UBUNTU_VERSION=24.04 CONTAINERD_VERSION=v2.0.3 run "" arm64 Dockerfile.origin test-integration CONTAINERD_VERSION=v2.0.3 run "$no_cache" amd64 Dockerfile.origin build-dependencies UBUNTU_VERSION=24.04 CONTAINERD_VERSION=v2.0.3 run "" amd64 Dockerfile.origin test-integration } # To simulate what happens when there is no cache, go with: ci_run no_cache # Once you have a cached run, you can simulate what happens with cache # First modify something in the nerdctl tree # Then run it touch mimick_nerdctl_change ci_run ``` ## Analyzing results ### Network #### Full CI run, cold cache (the first three pipelines, and part of the fourth) The following numbers are based on the above script, with cold cache. Unfortunately golang did segfault on me during the last (cross-run targetting amd), so, these numbers should be taken as (slightly) underestimated. Total number of requests: 7190 Total network duration: 13 minutes 11 seconds Outbound: 1.31MB Inbound: 5202MB Breakdown per domain | Destination | # requests | through | duration | |----------------------------------------------|-------------------|------------|-----------------| | https://registry-1.docker.io | 123 (2 failed) | 1.22MB | 26s | | https://production.cloudflare.docker.com | 60 | 1242.41MB | 2m6s | | http://deb.debian.org | 207 | 107.14MB | 13s | | https://github.com | 105 | 977.88MB | 1m25s | | https://proxy.golang.org | 5343 (57 failed) | 753.69MB | 4m8s | | https://objects.githubusercontent.com | 42 | 900.22MB | 50s | | https://raw.githubusercontent.com | 8 | 92KB | 2s | | https://storage.googleapis.com | 19 (3 failed) | 537.21MB | 35s | | https://ghcr.io | 65 | 588.68KB | 13s | | https://auth.docker.io | 10 | 259KB | 5s | | https://pkg-containers.githubusercontent.com | 48 | 183.63MB | 20s | | http://ports.ubuntu.com | 300 | 165.36MB | 1m55s | | https://golang.org | 4 | 228.93KB | <1s | | https://go.dev | 4 | 95.51KB | <1s | | https://dl.google.com | 4 | 271.42MB | 11s | | https://sum.golang.org | 746 | 3.89MB | 17s | | http://security.ubuntu.com | 7 | 2.70MB | 3s | | http://archive.ubuntu.com | 95 | 55.95MB | 19s | | | - | - | - | | Total | 7190 | 5203MB | 13 mins 11 secs | #### Full CI run, warm cache (only the first three pipelines) | Destination | # requests | through | duration | |------------------------------------------|------------------|---------|----------------| | https://registry-1.docker.io | 25 | 537KB | 14s | | https://production.cloudflare.docker.com | 2 | 25MB | 1s | | https://github.com | 7 (1 failed) | 105KB | 2s | | https://proxy.golang.org | 930 (11 failed) | 150MB | 37s | | https://objects.githubusercontent.com | 4 | 86MB | 4s | | https://storage.googleapis.com | 3 | 112MB | 6s | | https://auth.docker.io | 1 | 26KB | <1s | | http://ports.ubuntu.com | 133 | 67MB | 50s | | https://golang.org | 2 | 114KB | <1s | | https://go.dev | 2 | 45KB | <1s | | https://dl.google.com | 2 | 134MB | 5s | | https://sum.golang.org | 484 | 3MB | 11s | | | - | - | - | | Total | 1595 (12 failed) | 579MB | 2 mins 10 secs | #### Analysis ##### Docker Hub Images from Docker Hub are clearly a source of concern (made even worse by the fact they apply strict limitations on the number of requests permitted). When the cache is cold, this is about 1GB per run, for 200 requests and 3 minutes. Actions: - [ ] reduce the number of images - we currently use 2 golang images, which does not make sense - [ ] reduce the round trips - there is no reason why any of the images should be queried more than once per build - [ ] move away from Hub golang image, and instead use a raw distro + golang download - Hub golang is a source of pain and issues (diverging version scheme forces ugly shell contorsions, delay in availability creates broken situations) - we are already downloading the go release tarball anyhow, so, this is just wasted bandwidth with no added value Success criteria: - on a cold cache, reduce the total number of requests against Docker properties by 50% or more - on a cold cache, cut the data transfer and time in half ##### Distro packages On a WARM cache, close to 1 minute is spent fetching Ubuntu packages. This should not happen, and distro downloads should always be cached. On a cold cache, distro packages download near 3 minutes. Very likely there is stage duplication that could be reduced and some of that could be cut of. Actions: - [ ] ensure distro package downloading is staged in a way we can cache it - [ ] review stages to reduce package installation duplication Success criteria: - [ ] 0 package installation on a warm cache - [ ] cut cold cache package install time by 50% (XXX not realistic?) ##### GitHub repositories Clones from GitHub do clock in at 1GB on a cold cache. Containerd alone counts for more than half of it (at 160MB+ x4). Hopefully, on a warm cache it is practically non-existent. But then, this is ridiculous. Actions: - [ ] shallow clone Success criteria: - [ ] reduce network traffic from cloning by 80% ##### Go modules At 750+MB and over 4 minutes, this is the number one speed bottleneck on a cold cache. On a warm cache, it is still over 150MB and 30+ seconds. In and of itself, this is hard to reduce, as we need these... Actions: - [ ] we could cache the module download location to reduce round-trips on modules that are shared across different projects - [ ] we are likely installing nerdctl modules six times - (once per architecture during the build phase, then once per ubuntu version and architecture during the tests runs (this is not even accounted for in the audit above)) - it should only happen twice (once per architecture) Success criteria: - [ ] achieve 20% reduction of total time spent downloading go modules ##### Other downloads 1. At 500MB+ and 30 seconds, storage.googleapis.com is serving a SINGLE go module that gets special treatment: klauspost/compress. This module is very small, but does serve along with it a very large `testdata` folder. The fact that nerdctl downloads its module multiple times is further compounding the effect. 2. the golang archive is downloaded multiple times - it should be downloaded only once per run, and only on a cold cache 3. some of the binary releases we are retrieving are also being retrieved with a warm cache, and they are generally quite large. We could consider building certain things from source instead, and in all cases ensure that we are only downloading with a cold cache. Success criteria: - [ ] 0 static downloads on a warm cache - [ ] cut extra downloads by 20% #### Duration Unscientific numbers, per pipeline dependencies, no cache: - 224 seconds total - 53 seconds exporting cache dependencies, with cache: - 12 seconds test-integration, no cache: - 282 seconds #### Caching Number of layers in cache: ``` after dependencies stage: 78 intermediate size: 1.5G after test-integration stage: 118 total size: 2.8G ``` ## Generic considerations ### Caching compression This is obviously heavily dependent on the runner properties. With local cache, on high-performance IO (laptop SSD), zstd is definitely considerably better (about twice as fast). With GHA, the impact is minimal, since network IO is heavily dominant, but zstd still has the upper hand with regard to cache size. ### Output Loading image into the Docker store comes at a somewhat significant cost. It is quite possible that a significant performance boost could be achieved by using buildkit containerd worker and nerdctl instead. --- ## File: docs/dev/store.md # About `pkg/store` ## TL;DR You _may_ want to read this if you are developing something in nerdctl that would involve storing persistent information. If there is a "store" already in the codebase (eg: volumestore, namestore, etc) that does provide the methods that you need, you are fine and should just stick to that. On the other hand, if you are curious, or if what you want to write is "new", then _you should_ have a look at this document: it does provide extended information about how we manage persistent data storage, especially with regard to concurrency and atomicity. ## Motivation The core of nerdctl aims at keeping its dependencies as lightweight as possible. For that reason, nerdctl does not use a database to store persistent information, but instead uses the filesystem, under a variety of directories. That "information" is typically volumes metadata, containers lifecycle info, the "name store" (which does ensure no two containers can be named the same), etc. However, storing data on the filesystem in a reliable way comes with challenges: - incomplete writes may happen (because of a system restart, or an application crash), leaving important structured files in a broken state - concurrent writes, or reading while writing would obviously be a problem as well, be it across goroutines, or between concurrent executions of the nerdctl binary, or embedded in a third-party application that does concurrently access resources The `pkg/store` package does provide a "storage" abstraction that takes care of these issues, generally providing guarantees that concurrent operations can be performed safely, and that writes are "atomic", ensuring we never brick user installs. For details about how, and what is done, read-on. ## The problem with writing a file A write may very well be interrupted. While reading the resulting mangled file will typically break `json.Unmarshall` for example, and while we should still handle such cases gracefully and provide meaningful information to the user about which file is damaged (which could be due to the user manually modifying them), using "atomic" writes will (almost always (*)) prevent this from happening on our part. An "atomic" write is usually performed by first writing data to a temporary file, and then, only if the write operation succeeded, move that temporary file to its final destination. The `rename` syscall (see https://man7.org/linux/man-pages/man2/rename.2.html) is indeed "atomic" (eg: it fully succeeds, or fails), providing said guarantees that you end-up with a complete file that has the entirety of what was meant to be written. This is an "almost always", as an _operating system crash_ MAY break that promise (this is highly dependent on specifics that are out of scope here, and that nerdctl has no control over). Though, crashing operating systems is (hopefully) a sufficiently rare event that we can consider we "always" have atomic writes. There is one caveat with "rename-based atomic writes" though: if you mount the file itself inside a container, an atomic write will not work as you expect, as the inode will (obviously) change when you modify the file, and these changes will not be propagated inside the container. This caveat is the reason why `hostsstore` does NOT use an atomic write to update the `hosts` file, but a traditional write. ## Concurrency between go routines This is a (simple) well-known problem. Just use a mutex to prevent concurrent modifications of the same object. Note that this is not much of a problem right now in nerdctl itself - but it might be in third-party applications using our codebase. This is just generally good hygiene when building concurrency-safe packages. ## Concurrency between distinct binary invocations This is much more of a problem. There are many good reasons and real-life scenarios where concurrent binary execution may happen. A third-party deployment tool (similar to terraform for eg), that will batch a bunch of operations to be performed to achieve a desired infrastructure state, and call many `nerdctl` invocations in parallel to achieve that. This is also common-place in testing (subpackages). And of course, a third-party tool that would be long-running and allow parallel execution, leveraging nerdctl codebase as a library, may certainly produce these circumstances. The known answer to that problem is to use a filesystem lock (or flock). Concretely, the first process will "lock" a directory. All other processes trying to do the same will then be put in a queue and wait for the prior lock to be released before they can "lock" themselves, in turn. Filesystem locking comes with its own set of challenges: - implementation is somewhat low-level (the golang core library keeps their implementation internal, and you have to reimplement your own with platform-dependent APIs and syscalls) - it is tricky to get right - there are reasons why golang core does not make it public - locking "scope" should be done carefully: having ONE global lock for everything will definitely hurt performance badly, as you will basically make everything "sequential", effectively destroying some of the benefits of parallelizing code in the first place... ## Lock design... While it is tempting to just provide self-locking, individual methods as an API (`Get`, `Set`), this is not the right answer. Imagine a context where consuming code would first like to check if something exists, then later on create it if it does not: ```golang if !store.Exists("something") { // do things // ... // Now, create store.Set([]byte("data"), "something") } ``` You do have two methods (`Get` and `Set`) that _may individually_ guarantee they are the sole user of that resource, but a concurrent change _in between_ these two calls may very well (and _will_) happen and change the state of the world. Effectively, in that case, `Set` might overwrite changes made by another go routine or concurrent execution, possibly wrecking havoc in another process. _When_ to lock, and _for how long_, is a decision that only the embedding code can make. A good example is container creation. It may require the creation of several different volumes. In that case, you want to lock at the start of the container creation process, and only release the lock when you are fully done creating the container - not just when done creating a volume (nor even when done creating all volumes). ## ... while safeguarding the developer nerdctl still provides some safeguards for the developer. Any store method that DOES require locking will fail loudly if it does not detect a lock. This is obviously not bullet-proof. For example, the lock may belong to another goroutine instead of the one we are in (and we cannot detect that). But this is still better than nothing, and will help developers making sure they **do** lock. ## Using the `store` api to implement your own storage While - as mentioned above - the store does _not_ lock on its own, specific "stores implementations" may, and should, provide higher-level methods that best fit their data-model usage, and that **do** lock on their own. For example, the namestore (which is the simplest store), does provide three simple methods: - Acquire - Release - Rename Users of the `namestore` do not have to bother with locking. These methods are safe to use concurrently. This is a good example of how to leverage core store primitives to implement a developer friendly, safe storage for "something" (in that case "names"). Finally note an important point - mentioned above: locking should be done to the smallest possible "segment" of sub-directories. Specifically, any store should lock only - at most - resources under the _namespace_ being manipulated. For example, a container lifecycle storage should not lock out any other container, but only its own private directory. ## Scope, ambitions and future `pkg/store` has no ambition whatsoever to be a generic solution, usable outside nerdctl. It is solely designed to fit nerdctl needs, and if it was to be made usable standalone, would probably have to be modified extensively, which is clearly out of scope here. Furthermore, there are already much more advanced generic solutions out there that you should use instead for outside-of-nerdctl projects. As for future, one nice thing we should consider is to implement read-only locks in addition to the exclusive, write-locks we currently use. The net benefit would be a performance boost in certain contexts (massively parallel, mostly read environments). --- ## File: docs/testing/README.md # Testing nerdctl This document covers basic usage of nerdctl testing tasks, and generic recommendations and principles about writing tests. For more comprehensive information about nerdctl test tools, see [tools.md](tools.md). ## Code, fix, lint, rinse, repeat ``` # Do hack # When done: # This will ensure proper import formatting, modules, and basic formatting of your code make fix # And this will run all linters and report if any modification is required make lint ``` Note that both these tasks will work on any OS, including macOS. Also note that `make lint` does call on subtask `make lint-commits` which is going to lint commits for the range `main..HEAD` by default (this is fine for most development flows that expect to be merged in main). If you are targeting a specific branch that is not `main` (for example working against `release/1.7`), you likely want to explicitly set the commit range to that instead. eg: `LINT_COMMIT_RANGE=target_branch..HEAD make lint-commits` or `LINT_COMMIT_RANGE=target_branch..HEAD make lint` ## Unit testing ``` # This will run basic unit testing make test ``` This task must be run on a supported OS (linux, windows, or freebsd). ## Integration testing ### TL;DR Be sure to first `make && sudo make install` ```bash # Test all with nerdctl (rootless mode, if running go as a non-root user) go test -p 1 ./cmd/nerdctl/... # Test all with nerdctl rootful go test -p 1 -exec sudo ./cmd/nerdctl/... # Test all with docker go test -p 1 ./cmd/nerdctl/... -args -test.target=docker # Test just the tests(s) which names match TestVolume.* go test -p 1 ./cmd/nerdctl/... -run "TestVolume.*" # Or alternatively, just test the subpackage go test ./cmd/nerdctl/volume ``` ### About parallelization By default, when `go test ./foo/...` finds subpackages, it does create _a separate test binary per sub-package_, and execute them _in parallel_. This effectively will make distinct tests in different subpackages to be executed in parallel, regardless of whether they called `t.Parallel` or not. The `-p 1` flag does inhibit this behavior, and forces go to run each sub-package sequentially. Note that this is different from the `--parallel` flag, which controls the amount of parallelization that a single go test binary will use when faced with tests that do explicitly allow it (with a call to `t.Parallel()`). ### Or provision a test environment with Docker-built artifacts Docker can be used to build all the dependencies needed to run the integration tests (containerd, runc, CNI plugins, BuildKit, snapshotters, etc.), which can then be installed on the host (this is what the CI does). These scripts substantially and irreversibly modify the host, so they refuse to run unless `GITHUB_ACTIONS=true` is set - only do this on a disposable machine: ```bash docker buildx build --target out-test-integration-artifacts --output type=local,dest=/tmp/nerdctl-test-artifacts . sudo env GITHUB_ACTIONS=true ./hack/provisioning/linux/test-integration-env.sh install /tmp/nerdctl-test-artifacts rootful ./hack/test-integration.sh -test.target=nerdctl -test.only-flaky=false ``` For rootless (`rootless`, or `rootless-port-slirp4netns`): ```bash sudo env GITHUB_ACTIONS=true ./hack/provisioning/linux/test-integration-env.sh install /tmp/nerdctl-test-artifacts rootless GITHUB_ACTIONS=true ./hack/test-integration-rootless.sh ./hack/test-integration.sh -test.target=nerdctl -test.only-flaky=false ``` ### Principles #### Tests should be parallelized (with best effort) ##### General case It should be possible to parallelize all tests - as such, please make sure you: - name all resources your test is manipulating after the test identifier (`data.Identifier()`) to guarantee your test will not interact with other tests - do NOT use `os.Setenv` - instead, use `Setenv` on the command you are running - use `t.Parallel()` at the beginning of your test (and subtests as well of course) - in the very exceptional case where your test for some reason can NOT be parallelized, be sure to mark it explicitly as such with a comment explaining why ##### For "blanket" destructive operations If you are going to use blanket destructive operations (like `prune`), please: - use a dedicated namespace: add `nerdtest.Private` to the test `Require`ments - since docker does not support namespaces, be sure to: - only enable `Parallel` if the target is NOT docker: ` if !nerdtest.IsDocker() { t.Parallel() }` - double check that what you do in the default namespace is safe #### Clean-up after (and before) yourself - do NOT use `defer`, use `t.Cleanup` - do NOT test the result of commands doing the cleanup - it is fine if they fail, and they are not test failure per-se - they are here to garbage collect - you should call your cleanup routine BEFORE doing anything, in case there is any leftovers from previous runs, typically: ``` tearDown := func(){ // Do some cleanup } tearDown() t.Cleanup(tearDown) ``` #### Test what you are testing, and not something else You should only test atomically. If you are testing `nerdctl volume create`, make sure that your test will not fail because of changes in `nerdctl volume inspect`. That obviously means there are certain things you cannot test "yet". Just put the right test in the right place with this simple rule of thumb: if your test requires another nerdctl command to validate the result, then it does not belong here. Instead, it should be a test for that other command. Of course, this is not perfect, and changes in `create` may now fail in `inspect` tests while `create` could be faulty, but it does beat the alternative, because of this principle: it is easier to walk *backwards* from a failure. --- ## File: docs/testing/tools.md # Nerdctl testing tools ## Preamble The integration test suite in nerdctl is meant to apply to both nerdctl and docker, and further support additional test properties to target specific contexts (ipv6, kube). Basic _usage_ is covered in the [testing docs](README.md). This here covers how to write tests, leveraging nerdctl `pkg/testutil/test` which has been specifically developed to take care of repetitive tasks, protect the developer against unintended side effects across tests, and generally encourage clear testing structure with good debug-ability and a relatively simple API for most cases. ## Using `test.Case` Starting from scratch, the simplest, basic structure of a new test is: ```go package main import ( "testing" "github.com/containerd/nerdctl/v2/pkg/testutil/nerdtest" "github.com/containerd/nerdctl/mod/tigron/test" "github.com/containerd/nerdctl/mod/tigron/require" "github.com/containerd/nerdctl/mod/tigron/expect" ) func TestMyThing(t *testing.T) { // Declare your test myTest := nerdtest.Setup() // This is going to run `nerdctl info` (or `docker info`) myTest.Command = test.Command("info") // Verify the command exits with 0, and stdout contains the word `Kernel` myTest.Expected = test.Expects(0, nil, expect.Contains("Kernel")) // Run it myTest.Run(t) } ``` ## Expectations There are a handful of helpers for "expectations". You already saw two (`test.Expects` and `test.Contains`): First, `test.Expects(exitCode int, errors []error, outputCompare Comparator)`, which is convenient to quickly describe what you expect overall. `exitCode` is obvious (note that passing -1 as an exit code will just verify the commands does fail without comparing the code, and -2 will not verify the exit code at all). `errors` is a slice of go `error`, that allows you to compare what is seen on stderr with existing errors (for example: `errdefs.ErrNotFound`), or more generally any string you want to match. `outputCompare` can be either your own comparison function, or one of the comparison helper. Secondly, `test.Contains` - which is a `Comparator`. ### Comparators Besides `expect.Contains(string)`, there are a few more: - `expect.DoesNotContain(string)` - `expect.Equals(string)` - `expect.Match(*regexp.Regexp)` - `expect.All(comparators ...Comparator)`, which allows you to bundle together a bunch of other comparators The following example shows how to implement your own custom `Comparator` (this is actually the `Equals` comparator). ```go package whatever import ( "testing" "gotest.tools/v3/assert" "github.com/containerd/nerdctl/mod/tigron/expect" "github.com/containerd/nerdctl/mod/tigron/require" "github.com/containerd/nerdctl/mod/tigron/test" "github.com/containerd/nerdctl/mod/tigron/tig" ) func MyComparator(compare string) test.Comparator { return func(stdout string, t tig.T) { t.Helper() assert.Assert(t, stdout == compare) } } ``` ### Advanced expectations You may want to have expectations that contain a certain piece of data that is being used in the command or at other stages of your test (Setup). For example, creating a container with a certain name, you might want to verify that this name is then visible in the list of containers. To achieve that, you should write your own `Manager`, leveraging test `Data`. Here is an example, where we are using `data.Labels().Get("sometestdata")`. ```go package main import ( "errors" "testing" "gotest.tools/v3/assert" "github.com/containerd/errdefs" "github.com/containerd/nerdctl/mod/tigron/expect" "github.com/containerd/nerdctl/mod/tigron/require" "github.com/containerd/nerdctl/mod/tigron/test" "github.com/containerd/nerdctl/mod/tigron/tig" "github.com/containerd/nerdctl/v2/pkg/testutil/nerdtest" ) func TestMyThing(t *testing.T) { nerdtest.Setup() // Declare your test myTest := &test.Case{ Data: test.WithLabels(map[string]string{"sometestdata": "blah"}), Command: test.Command("info"), Expected: func(data test.Data, helpers test.Helpers) *test.Expected { return &test.Expected{ ExitCode: 1, Errors: []error{ errors.New("foobla"), errdefs.ErrNotFound, }, Output: func(stdout string, t tig.T) { assert.Assert(t, stdout == data.Labels().Get("sometestdata")) }, } }, } myTest.Run(t) } ``` ## On `Data` `Data` is provided to allow storing mutable key-value information that pertain to the test. While it can be provided through `test.WithLabels(map[string]string{key: value})`, inside the testcase definition, it can also be dynamically manipulated inside `Setup`, or `Command`. Note that `Data` additionally exposes the following functions: - `Identifier(words ...string)` which returns a unique identifier associated with the _current_ test (or subtest) - `TempDir()` which returns the private, temporary directory associated with the test ... along with the `Get(key)` and `Set(key, value)` methods. Note that Data is copied down to subtests, which is convenient to pass "down" information relevant to a bunch of subtests (eg: like a registry IP). ## On Config `Config` is similar to `Data`, although it is meant specifically for predefined keys that impact the base behavior of the binary you are testing. You can initiate your config using `test.WithConfig(key, value)`, and you can manipulate it further using `helpers.Read` and`helpers.Write`. Currently, the following keys are defined: - `DockerConfig` allowing to set custom content for the `$DOCKER_CONFIG/config.json` file - `Namespace` (default to `nerdctl-test` if unspecified, but see "mode private") - `NerdctlToml` to set custom content for the `$NERDCTL_TOML` file - `HostsDir` to specify the value of the arg `--hosts-dir` - `DataRoot` to specify the value of the arg `--data-root` - `Debug` to enable debug (works for both nerdctl and docker) Note that config defined on the test case is copied over for subtests. ## Commands For simple cases, `test.Command(args ...string)` is the way to go. It will execute the binary to test (nerdctl or docker), with the provided arguments, and will by default get cwd inside the temporary directory associated with the test. ### Environment You can attach custom environment variables for your test in the `Env` property of your test. These will be automatically added to the environment for your command, and also your setup and cleanup routines (see below). If you would like to override the environment specifically for a command, but not for others (eg: in `Setup` or `Cleanup`), you can do so with custom commands (see below). Note that environment as defined statically in the test will be copied over for subtests. ### Working directory By default, the working directory of the command will be set to the temporary directory of the test. This behavior can be overridden using custom commands. ### Custom Executor Custom `Executor`s allow you to manipulate test `Data`, override important aspects of the command to execute (`Env`, `WorkingDir`), or otherwise give you full control on what the command does. You just need to implement an `Executor`: ```go package main import ( "errors" "testing" "gotest.tools/v3/assert" "github.com/containerd/errdefs" "github.com/containerd/nerdctl/mod/tigron/expect" "github.com/containerd/nerdctl/mod/tigron/require" "github.com/containerd/nerdctl/mod/tigron/test" "github.com/containerd/nerdctl/mod/tigron/tig" "github.com/containerd/nerdctl/v2/pkg/testutil/nerdtest" ) func TestMyThing(t *testing.T) { nerdtest.Setup() // Declare your test myTest := &test.Case{ Data: test.WithLabels(map[string]string{"sometestdata": "blah"}), Command: func(data test.Data, helpers test.Helpers) test.TestableCommand { return helpers.Command("run", "--name", data.Labels().Get("sometestdata")) }, Expected: func(data test.Data, helpers test.Helpers) *test.Expected { return &test.Expected{ ExitCode: 1, Errors: []error{ errors.New("foobla"), errdefs.ErrNotFound, }, Output: func(stdout string, t tig.T) { assert.Assert(t, stdout == data.Labels().Get("sometestdata")) }, } }, } myTest.Run(t) } ``` Note that inside your `Executor` you do have access to the full palette of command options, including: - `Background(timeout time.Duration)` which allows you to background a command execution - `WithWrapper(binary string, args ...string)` which allows you to "wrap" your command with another binary - `WithStdin(io.Reader)` which allows you to pass a reader to the command stdin - `WithCwd(string)` which allows you to specify the working directory (default to the test temp directory) - `Clone()` which returns a copy of the command, with env, cwd, etc and also `WithBinary` and `WithArgs`. ### On `helpers` Inside a custom `Executor`, `Manager`, or `Butler`, you have access to a collection of `helpers` to simplify command execution: - `helpers.Ensure(args ...string)` will run a command and ensure it exits successfully - `helpers.Fail(args ...string)` will run a command and ensure it fails - `helpers.Anyhow(args ...string)` will run a command but does not care if it succeeds or fails - `helpers.Capture(args ...string)` will run a command, ensure it is successful, and return the output - `helpers.Command(args ...string)` will return a command that can then be tested against expectations - `helpers.Custom(binary string, args ...string)` will do the same for any arbitrary command (not limited to nerdctl) - `helpers.T()` which returns the appropriate `*testing.T` for your context ## Setup and Cleanup Tests routinely require a set of actions to be performed _before_ you can run the command you want to test. A setup routine will get executed before your `Command`, and have access to and can manipulate your test `Data` and `Config`. Conversely, you very likely want to clean things up once your test is done. While temporary directories are cleaned for you with no action needed on your part, the app you are testing might have stateful data you may want to remove. Note that a `Cleanup` routine will get executed twice - after your `Command` has run its course evidently - but also, pre-emptively, before your `Setup`, so that possible leftovers from previous runs are taken care of. ```go package main import ( "errors" "testing" "gotest.tools/v3/assert" "github.com/containerd/errdefs" "github.com/containerd/nerdctl/mod/tigron/expect" "github.com/containerd/nerdctl/mod/tigron/require" "github.com/containerd/nerdctl/mod/tigron/test" "github.com/containerd/nerdctl/mod/tigron/tig" "github.com/containerd/nerdctl/v2/pkg/testutil/nerdtest" ) func TestMyThing(t *testing.T) { nerdtest.Setup() // Declare your test myTest := &test.Case{ Data: test.WithLabels(map[string]string{"sometestdata": "blah"}), Setup: func(data *test.Data, helpers test.Helpers){ helpers.Ensure("volume", "create", "foo") helpers.Ensure("volume", "create", "bar") }, Cleanup: func(data *test.Data, helpers test.Helpers){ helpers.Anyhow("volume", "rm", "foo") helpers.Anyhow("volume", "rm", "bar") }, Command: func(data test.Data, helpers test.Helpers) test.TestableCommand { return helpers.Command("run", "--name", data.Identifier()+data.Labels().Get("sometestdata")) }, Expected: func(data test.Data, helpers test.Helpers) *test.Expected { return &test.Expected{ ExitCode: 1, Errors: []error{ errors.New("foobla"), errdefs.ErrNotFound, }, Output: func(stdout string, t tig.T) { assert.Assert(t, stdout == data.Labels().Get("sometestdata")) }, } }, } myTest.Run(t) } ``` ## Subtests Subtests are just regular tests, attached to the `SubTests` slice of a test. Note that a subtest will inherit its parent `Data`, `Config` and `Env`, in the state they are at after the parent test has run its `Setup` and `Command` routines (but before `Cleanup`). This does _not_ apply to `Identifier()` and `TempDir()`, which are unique to the subtest. Also note that a test does not have to have a `Command`. This is a convenient pattern if you just need a common `Setup` for a bunch of subtests. ## Parallelism All tests (and subtests) are assumed to be parallelizable. You can force a specific `test.Case` to not be run in parallel though, by setting its `NoParallel` property to `true`. Note that if you want better isolation, it is usually better to use the requirement `nerdtest.Private` instead of `NoParallel` (see below). ## Requirements `test.Case` has a `Require` property that allow enforcing specific, per-test requirements. A `Requirement` is expected to make you `Skip` tests when the environment does not match expectations. Here are a few: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` ### About `nerdtest.Private` While all requirements above are self-descriptive or obvious, `nerdtest.Private` is a special case. If set, it will run tests inside a dedicated namespace that is private to the test. Note that subtests by default are going to be set in that same namespace, unless they ask for private as well, or they reset the `Namespace` config key. If the target is Docker - which does not support namespaces - asking for `Private` will disable parallelization. The purpose of private is to provide a truly clean-room environment for tests that are going to have side effects on others, or that do require an exclusive, pristine environment. Using private is generally preferable to disabling parallelization, as doing the latter would slow down the run and won't have the same isolation guarantees about the environment. ## Advanced command customization Testing any non-trivial binary likely assume a good amount of custom code to set up the right default behavior wrt environment, flags, etc. To do that, you can pass a `test.Testable` implementation to the `test.Customize` method. It basically lets you define your own `CustomizableCommand`, along with a hook to deal with ambient requirements that is run after `test.Require` and before `test.Setup`. `CustomizableCommand` are typically embedding a `test.GenericCommand` and overriding both the `Run` and `Clone` methods. Check the `nerdtest` implementation for details. ## Utilities TBD --- ## File: docs/build.md # Setting up `nerdctl build` with BuildKit `nerdctl build` (and `nerdctl compose build`) relies on [BuildKit](https://github.com/moby/buildkit). To use it, you need to set up BuildKit. BuildKit has 2 types of backends. - **containerd worker**: BuildKit relies on containerd to manage containers and images, etc. containerd needs to be up-and-running on the host. - **OCI worker**: BuildKit manages containers and images, etc. containerd isn't needed. This worker relies on runc for container execution. You need to set up BuildKit with either of the above workers. Note that OCI worker cannot access base images (`FROM` images in Dockerfiles) managed by containerd. Thus you cannot let `nerdctl build` use containerd-managed images as the base image. They include images previously built using `nerdctl build`. For example, the following build `bar` fails with OCI worker because it tries to use the previously built and containerd-managed image `foo`. ```console $ mkdir -p /tmp/ctx && cat < /tmp/ctx/Dockerfile FROM ghcr.io/stargz-containers/ubuntu:20.04-org RUN echo hello EOF $ nerdctl build -t foo /tmp/ctx $ cat < /tmp/ctx/Dockerfile FROM foo RUN echo bar EOF $ nerdctl build -t bar /tmp/ctx ``` This limitation can be avoided using containerd worker as mentioned later. ## Setting up BuildKit with containerd worker ### Rootless | :zap: Requirement | nerdctl >= 0.18, BuildKit >= 0.10 | |-------------------|-----------------------------------| ``` $ CONTAINERD_NAMESPACE=default containerd-rootless-setuptool.sh install-buildkit-containerd ``` `containerd-rootless-setuptool.sh` is aware of `CONTAINERD_NAMESPACE` and `CONTAINERD_SNAPSHOTTER` envvars. It installs buildkitd to the specified containerd namespace. This allows BuildKit using containerd-managed images in that namespace as the base image. Note that BuildKit can't use images in other namespaces as of now. If `CONTAINERD_NAMESPACE` envvar is not specified, this script configures buildkitd to use "buildkit" namespace (not "default" namespace). You can install an additional buildkitd process in a different namespace by executing this script with specifying the namespace with `CONTAINERD_NAMESPACE`. BuildKit will expose the socket at `$XDG_RUNTIME_DIR/buildkit-$CONTAINERD_NAMESPACE/buildkitd.sock` if `CONTAINERD_NAMESPACE` is specified. If `CONTAINERD_NAMESPACE` is not specified, that location will be `$XDG_RUNTIME_DIR/buildkit/buildkitd.sock`. ### Rootful ``` $ sudo systemctl enable --now buildkit ``` Then add the following configuration to `/etc/buildkit/buildkitd.toml` to enable containerd worker. ```toml [worker.oci] enabled = false [worker.containerd] enabled = true # namespace should be "k8s.io" for Kubernetes (including Rancher Desktop) namespace = "default" ``` ## Setting up BuildKit with OCI worker ### Rootless ``` $ containerd-rootless-setuptool.sh install-buildkit ``` As mentioned in the above, BuildKit with this configuration cannot use images managed by containerd. They include images previously built with `nerdctl build`. BuildKit will expose the socket at `$XDG_RUNTIME_DIR/buildkit/buildkitd.sock`. ### rootful ``` $ sudo systemctl enable --now buildkit ``` ## Which BuildKit socket will nerdctl use? You can specify BuildKit address for `nerdctl build` using `--buildkit-host` flag or `BUILDKIT_HOST` envvar. When BuildKit address isn't specified, nerdctl tries some default BuildKit addresses the following order and uses the first available one. - `/buildkit-/buildkitd.sock` - `/buildkit-default/buildkitd.sock` - `/buildkit/buildkitd.sock` For example, if you run rootless nerdctl with `test` containerd namespace, it tries to use `$XDG_RUNTIME_DIR/buildkit-test/buildkitd.sock` by default then try to fall back to `$XDG_RUNTIME_DIR/buildkit-default/buildkitd.sock` and `$XDG_RUNTIME_DIR/buildkit/buildkitd.sock` --- ## File: docs/builder-debug.md # Interactive debugging of Dockerfile (Experimental) nerdctl supports interactive debugging of Dockerfile as `nerdctl builder debug`. ``` $ nerdctl builder debug /path/to/context ``` This feature leverages [buildg](https://github.com/ktock/buildg), interactive debugger of Dockerfile. For command reference, please refer to the [Command reference doc in buildg repo](https://github.com/ktock/buildg#command-reference). :warning: This command currently doesn't use the host's `buildkitd` daemon but uses the patched version of BuildKit provided by buildg. This should be fixed to use the host's `buildkitd` in the future. ## Example Example Dockerfile: ```Dockerfile FROM busybox AS build1 RUN echo a > /a RUN echo b > /b RUN echo c > /c ``` Example debugging: ```console $ nerdctl builder debug --image=ubuntu:22.04 /tmp/ctx/ WARN[2022-05-17T10:15:48Z] using host network as the default#1 [internal] load .dockerignore #1 transferring context: 2B done #1 DONE 0.1s #2 [internal] load build definition from Dockerfile #2 transferring dockerfile: 108B done #2 DONE 0.1s #3 [internal] load metadata for docker.io/library/busybox:latest INFO[2022-05-17T10:15:51Z] debug session started. type "help" for command reference. Filename: "Dockerfile" => 1| FROM busybox AS build1 2| RUN echo a > /a 3| RUN echo b > /b 4| RUN echo c > /c (buildg) break 3 (buildg) breakpoints [0]: line: Dockerfile:3 [on-fail]: breaks on fail (buildg) continue #3 DONE 3.1s #4 [1/4] FROM docker.io/library/busybox@sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8 #4 resolve docker.io/library/busybox@sha256:d2b53584f580310186df7a2055ce3ff83cc0df6caacf1e3489bff8cf5d0af5d8 0.0s done #4 sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 0B / 772.81kB 0.2s #4 sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 0B / 772.81kB 5.3s #4 sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 0B / 772.81kB 10.4s #4 sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 772.81kB / 772.81kB 11.4s done #4 extracting sha256:50e8d59317eb665383b2ef4d9434aeaa394dcd6f54b96bb7810fdde583e9c2d1 0.1s done #4 DONE 20.2s #5 [2/4] RUN echo a > /a #5 DONE 0.1s Breakpoint[0]: reached line: Dockerfile:3 Filename: "Dockerfile" 1| FROM busybox AS build1 2| RUN echo a > /a *=> 3| RUN echo b > /b 4| RUN echo c > /c (buildg) exec --image sh # ls /debugroot/ a b bin dev etc home proc root tmp usr var # cat /debugroot/a /debugroot/b a b # (buildg) quit ``` --- ## File: docs/cni.md # Using CNI with nerdctl nerdctl uses CNI plugins for its container network, you can set network by either `--network` or `--net` option. ## Basic networks nerdctl support some basic types of CNI plugins without any configuration needed(you should have CNI plugin be installed), for Linux systems the basic CNI plugin types are `bridge`, `portmap`, `firewall`, `tuning`, for Windows system, the supported CNI plugin types are `nat` only. The default network `bridge` for Linux and `nat` for Windows if you don't set any network options. Configuration of the default network `bridge` of Linux: ```json { "cniVersion": "1.0.0", "name": "bridge", "plugins": [ { "type": "bridge", "bridge": "nerdctl0", "isGateway": true, "ipMasq": true, "hairpinMode": true, "ipam": { "type": "host-local", "routes": [{ "dst": "0.0.0.0/0" }], "ranges": [ [ { "subnet": "10.4.0.0/24", "gateway": "10.4.0.1" } ] ] } }, { "type": "portmap", "capabilities": { "portMappings": true } }, { "type": "firewall", "ingressPolicy": "same-bridge" }, { "type": "tuning" } ] } ``` ## Bridge isolation nerdctl >= 0.18 sets the `ingressPolicy` to `same-bridge` when `firewall` plugin >= 1.1.0 is installed. This `ingressPolicy` replaces the CNI `isolation` plugin used in nerdctl <= 0.17. When `firewall` plugin >= 1.1.0 is not found, nerdctl does not enable the bridge isolation. This means a container in `--net=foo` can connect to a container in `--net=bar`. ## macvlan/IPvlan networks nerdctl also support macvlan and IPvlan network driver. To create a `macvlan` network which bridges with a given physical network interface, use `--driver macvlan` with `nerdctl network create` command. ``` # nerdctl network create mac0 --driver macvlan \ --subnet=192.168.5.0/24 --gateway=192.168.5.2 -o parent=eth0 ``` You can specify the `parent`, which is the interface the traffic will physically go through on the host, defaults to default route interface. And the `subnet` should be under the same network as the network interface, an easier way is to use DHCP to assign the IP: ``` # nerdctl network create mac0 --driver macvlan --ipam-driver=dhcp ``` Using `--driver ipvlan` can create `ipvlan` network, the default mode for IPvlan is `l2`. ## DHCP host-name and other DHCP options Nerdctl automatically sets the DHCP host-name option to the hostname value of the container. Furthermore, on network creation, nerdctl supports the ability to set other DHCP options through `--ipam-options`. Currently, the following options are supported by the DHCP plugin: ``` dhcp-client-identifier subnet-mask routers user-class vendor-class-identifier ``` For example: ``` # nerdctl network create --driver macvlan \ --ipam-driver dhcp \ --ipam-opt 'vendor-class-identifier={"type": "provide", "value": "Hey! Its me!"}' \ my-dhcp-net ``` ## Custom networks You can also customize your CNI network by providing configuration files. When rootful, the expected root location is `/etc/cni/net.d`. For rootless, the expected root location is `~/.config/cni/net.d/` Configuration files (like `10-mynet.conf`) can be placed either in the root location, or under a subfolder. If in the root location, this network will be available to all nerdctl namespaces. If placed in a subfolder, it will be available only to the identically named namespace. For example, you have one configuration file(`/etc/cni/net.d/10-mynet.conf`) for `bridge` network: ```json { "cniVersion": "1.0.0", "name": "mynet", "type": "bridge", "bridge": "cni0", "isGateway": true, "ipMasq": true, "ipam": { "type": "host-local", "subnet": "172.19.0.0/24", "routes": [ { "dst": "0.0.0.0/0" } ] } } ``` This will configure a new CNI network with the name `mynet`, and you can use this network to create a container in any namespace: ```console # nerdctl run -it --net mynet --rm alpine ip addr show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if6120: mtu 1500 qdisc noqueue state UP link/ether 5e:5b:3f:0c:36:56 brd ff:ff:ff:ff:ff:ff inet 172.19.0.51/24 brd 172.19.0.255 scope global eth0 valid_lft forever preferred_lft forever inet6 fe80::5c5b:3fff:fe0c:3656/64 scope link tentative valid_lft forever preferred_lft forever ``` --- ## File: docs/command-reference.md # Command reference :whale: = Docker compatible :nerd_face: = nerdctl specific > [!NOTE] > - Unlisted `docker` CLI flags are unimplemented yet in `nerdctl` CLI. > It does not necessarily mean that the corresponding features are missing in containerd. > - Some commands and flags are only available on Linux. - [Container management](#container-management) - [:whale: nerdctl run](#whale-nerdctl-run) - [:whale: nerdctl exec](#whale-nerdctl-exec) - [:whale: nerdctl create](#whale-nerdctl-create) - [:whale: nerdctl cp](#whale-nerdctl-cp) - [:whale: nerdctl ps](#whale-nerdctl-ps) - [:whale: nerdctl inspect](#whale-nerdctl-inspect) - [:whale: nerdctl logs](#whale-nerdctl-logs) - [:whale: nerdctl port](#whale-nerdctl-port) - [:whale: nerdctl rm](#whale-nerdctl-rm) - [:whale: nerdctl stop](#whale-nerdctl-stop) - [:whale: nerdctl start](#whale-nerdctl-start) - [:whale: nerdctl restart](#whale-nerdctl-restart) - [:whale: nerdctl update](#whale-nerdctl-update) - [:whale: nerdctl wait](#whale-nerdctl-wait) - [:whale: nerdctl kill](#whale-nerdctl-kill) - [:whale: nerdctl pause](#whale-nerdctl-pause) - [:whale: nerdctl unpause](#whale-nerdctl-unpause) - [:whale: nerdctl rename](#whale-nerdctl-rename) - [:whale: nerdctl attach](#whale-nerdctl-attach) - [:whale: nerdctl container prune](#whale-nerdctl-container-prune) - [:whale: nerdctl diff](#whale-nerdctl-diff) - [:whale: nerdctl export](#whale-nerdctl-export) - [Build](#build) - [:whale: nerdctl build](#whale-nerdctl-build) - [:whale: nerdctl commit](#whale-nerdctl-commit) - [Image management](#image-management) - [:whale: nerdctl images](#whale-nerdctl-images) - [:whale: nerdctl pull](#whale-nerdctl-pull) - [:whale: nerdctl push](#whale-nerdctl-push) - [:whale: nerdctl load](#whale-nerdctl-load) - [:whale: nerdctl save](#whale-nerdctl-save) - [:whale: nerdctl import](#whale-nerdctl-import) - [:whale: nerdctl tag](#whale-nerdctl-tag) - [:whale: nerdctl rmi](#whale-nerdctl-rmi) - [:whale: nerdctl image inspect](#whale-nerdctl-image-inspect) - [:whale: nerdctl image history](#whale-nerdctl-image-history) - [:whale: nerdctl image prune](#whale-nerdctl-image-prune) - [:nerd_face: nerdctl image convert](#nerd_face-nerdctl-image-convert) - [:nerd_face: nerdctl image encrypt](#nerd_face-nerdctl-image-encrypt) - [:nerd_face: nerdctl image decrypt](#nerd_face-nerdctl-image-decrypt) - [Checkpoint management](#checkpoint-management) - [:whale: nerdctl checkpoint create](#whale-nerdctl-checkpoint-create) - [:whale: nerdctl checkpoint list](#whale-nerdctl-checkpoint-list) - [:whale: nerdctl checkpoint remove](#whale-nerdctl-checkpoint-remove) - [Manifest management](#manifest-management) - [:whale: nerdctl manifest annotate](#whale-nerdctl-manifest-annotate) - [:whale: nerdctl manifest create](#whale-nerdctl-manifest-create) - [:whale: nerdctl manifest inspect](#whale-nerdctl-manifest-inspect) - [:whale: nerdctl manifest push](#whale-nerdctl-manifest-push) - [:whale: nerdctl manifest rm](#whale-nerdctl-manifest-rm) - [Registry](#registry) - [:whale: nerdctl login](#whale-nerdctl-login) - [:whale: nerdctl logout](#whale-nerdctl-logout) - [:whale: nerdctl search](#whale-nerdctl-search) - [Network management](#network-management) - [:whale: nerdctl network create](#whale-nerdctl-network-create) - [:whale: nerdctl network ls](#whale-nerdctl-network-ls) - [:whale: nerdctl network inspect](#whale-nerdctl-network-inspect) - [:whale: nerdctl network rm](#whale-nerdctl-network-rm) - [:whale: nerdctl network prune](#whale-nerdctl-network-prune) - [Volume management](#volume-management) - [:whale: nerdctl volume create](#whale-nerdctl-volume-create) - [:whale: nerdctl volume ls](#whale-nerdctl-volume-ls) - [:whale: nerdctl volume inspect](#whale-nerdctl-volume-inspect) - [:whale: nerdctl volume rm](#whale-nerdctl-volume-rm) - [:whale: nerdctl volume prune](#whale-nerdctl-volume-prune) - [Namespace management](#namespace-management) - [:nerd_face: nerdctl namespace create](#nerd_face-nerdctl-namespace-create) - [:nerd_face: nerdctl namespace inspect](#nerd_face-nerdctl-namespace-inspect) - [:nerd_face: nerdctl namespace ls](#nerd_face-nerdctl-namespace-ls) - [:nerd_face: nerdctl namespace remove](#nerd_face-nerdctl-namespace-remove) - [:nerd_face: nerdctl namespace update](#nerd_face-nerdctl-namespace-update) - [AppArmor profile management](#apparmor-profile-management) - [:nerd_face: nerdctl apparmor inspect](#nerd_face-nerdctl-apparmor-inspect) - [:nerd_face: nerdctl apparmor load](#nerd_face-nerdctl-apparmor-load) - [:nerd_face: nerdctl apparmor ls](#nerd_face-nerdctl-apparmor-ls) - [:nerd_face: nerdctl apparmor unload](#nerd_face-nerdctl-apparmor-unload) - [Builder management](#builder-management) - [:whale: nerdctl builder prune](#whale-nerdctl-builder-prune) - [:nerd_face: nerdctl builder debug](#nerd_face-nerdctl-builder-debug) - [System](#system) - [:whale: nerdctl events](#whale-nerdctl-events) - [:whale: nerdctl info](#whale-nerdctl-info) - [:whale: nerdctl version](#whale-nerdctl-version) - [:whale: nerdctl system prune](#whale-nerdctl-system-prune) - [Stats](#stats) - [:whale: nerdctl stats](#whale-nerdctl-stats) - [:whale: nerdctl top](#whale-nerdctl-top) - [Shell completion](#shell-completion) - [:nerd_face: nerdctl completion bash](#nerd_face-nerdctl-completion-bash) - [:nerd_face: nerdctl completion zsh](#nerd_face-nerdctl-completion-zsh) - [:nerd_face: nerdctl completion fish](#nerd_face-nerdctl-completion-fish) - [:nerd_face: nerdctl completion powershell](#nerd_face-nerdctl-completion-powershell) - [Compose](#compose) - [:whale: nerdctl compose](#whale-nerdctl-compose) - [:whale: nerdctl compose up](#whale-nerdctl-compose-up) - [:whale: nerdctl compose logs](#whale-nerdctl-compose-logs) - [:whale: nerdctl compose build](#whale-nerdctl-compose-build) - [:whale: nerdctl compose create](#whale-nerdctl-compose-create) - [:whale: nerdctl compose exec](#whale-nerdctl-compose-exec) - [:whale: nerdctl compose down](#whale-nerdctl-compose-down) - [:whale: nerdctl compose images](#whale-nerdctl-compose-images) - [:whale: nerdctl compose start](#whale-nerdctl-compose-start) - [:whale: nerdctl compose stop](#whale-nerdctl-compose-stop) - [:whale: nerdctl compose port](#whale-nerdctl-compose-port) - [:whale: nerdctl compose ps](#whale-nerdctl-compose-ps) - [:whale: nerdctl compose pull](#whale-nerdctl-compose-pull) - [:whale: nerdctl compose push](#whale-nerdctl-compose-push) - [:whale: nerdctl compose pause](#whale-nerdctl-compose-pause) - [:whale: nerdctl compose unpause](#whale-nerdctl-compose-unpause) - [:whale: nerdctl compose config](#whale-nerdctl-compose-config) - [:whale: nerdctl compose cp](#whale-nerdctl-compose-cp) - [:whale: nerdctl compose kill](#whale-nerdctl-compose-kill) - [:whale: nerdctl compose restart](#whale-nerdctl-compose-restart) - [:whale: nerdctl compose rm](#whale-nerdctl-compose-rm) - [:whale: nerdctl compose run](#whale-nerdctl-compose-run) - [:whale: nerdctl compose top](#whale-nerdctl-compose-top) - [:whale: nerdctl compose version](#whale-nerdctl-compose-version) - [IPFS management](#ipfs-management) - [:nerd_face: nerdctl ipfs registry serve](#nerd_face-nerdctl-ipfs-registry-serve) - [Global flags](#global-flags) - [Unimplemented Docker commands](#unimplemented-docker-commands) ## Container management ### :whale: nerdctl run Run a command in a new container. Usage: `nerdctl run [OPTIONS] IMAGE [COMMAND] [ARG...]` :nerd_face: `ipfs://` prefix can be used for `IMAGE` to pull it from IPFS. See [`ipfs.md`](./ipfs.md) for details. :nerd_face: `oci-archive://` prefix can be used for `IMAGE` to specify a local file system path to an OCI formatted tarball. Basic flags: - :whale: `-a, --attach`: Attach STDIN, STDOUT, or STDERR - :whale: `-i, --interactive`: Keep STDIN open even if not attached - :whale: `-t, --tty`: Allocate a pseudo-TTY - :warning: WIP: currently `-t` conflicts with `-d` - :whale: `--sig-proxy`: Proxy received signals to the process (default true) - :whale: `-d, --detach`: Run container in background and print container ID - :whale: `--restart=(no|always|on-failure|unless-stopped)`: Restart policy to apply when a container exits - Default: "no" - always: Always restart the container if it stops. - on-failure[:max-retries]: Restart only if the container exits with a non-zero exit status. Optionally, limit the number of times attempts to restart the container using the :max-retries option. - unless-stopped: Always restart the container unless it is stopped. - :whale: `--rm`: Automatically remove the container when it exits - :whale: `--pull=(always|missing|never)`: Pull image before running - Default: "missing" - :whale: `-q, --quiet`: Suppress the pull output - :whale: `--pid=(host|container:)`: PID namespace to use - :whale: `--uts=(host)` : UTS namespace to use - :whale: `--stop-signal`: Signal to stop a container (default "SIGTERM") - :whale: `--stop-timeout`: Timeout (in seconds) to stop a container - :whale: `--detach-keys`: Override the default detach keys Platform flags: - :whale: `--platform=(amd64|arm64|...)`: Set platform Init process flags: - :whale: `--init`: Run an init inside the container that forwards signals and reaps processes. - :nerd_face: `--init-binary=`: The custom init binary to use. We suggest you use the [tini](https://github.com/krallin/tini) binary which is used in Docker project to get the same behavior. Please make sure the binary exists in your `PATH`. - Default: `tini` Isolation flags: - :whale: :nerd_face: `--isolation=(default|process|host|hyperv)`: Used on Windows to change process isolation level. `default` will use the runtime options configured in `default_runtime` in the [containerd configuration](https://github.com/containerd/containerd/blob/master/docs/cri/config.md#cri-plugin-config-guide) which is `process` in containerd by default. `process` runs process isolated containers. `host` runs [Host Process containers](https://kubernetes.io/docs/tasks/configure-pod-container/create-hostprocess-pod/). Host process containers inherit permissions from containerd process unless `--user` is specified then will start with user specified and the user specified must be present on the host. `host` requires Containerd 1.7+. `hyperv` runs Hyper-V hypervisor partition-based isolated containers. Not implemented for Linux. Network flags: - :whale: `--net, --network=(bridge|host|none|container:|ns:|)`: Connect a container to a network. - Default: "bridge" - `container:`: reuse another container's network stack, container has to be precreated. - :nerd_face: `ns:`: run inside an existing network namespace - :nerd_face: Unlike Docker, this flag can be specified multiple times (`--net foo --net bar`) - :whale: `-p, --publish`: Publish a container's port(s) to the host - :whale: `--dns`: Set custom DNS servers - :whale: `--dns-search`: Set custom DNS search domains - :whale: `--dns-opt, --dns-option`: Set DNS options - :whale: `-h, --hostname`: Container host name - :whale: `--domainname`: Container domain name - :whale: `--add-host`: Add a custom host-to-IP mapping (host:ip). `ip` could be a special string `host-gateway`, - which will be resolved to the `host-gateway-ip` in nerdctl.toml or global flag. - :whale: `--ip`: Specific static IP address(es) to use. Note that unlike docker, nerdctl allows specifying it with the default bridge network. - :whale: `--ip6`: Specific static IP6 address(es) to use. Should be used with user networks - :whale: `--mac-address`: Specific MAC address to use. Be aware that it does not check if manually specified MAC addresses are unique. Supports network type `bridge` and `macvlan` Resource flags: - :whale: `--cpus`: Number of CPUs - :whale: `--cpu-quota`: Limit the CPU CFS (Completely Fair Scheduler) quota - :whale: `--cpu-period`: Limit the CPU CFS (Completely Fair Scheduler) period - :whale: `--cpu-shares`: CPU shares (relative weight) - :whale: `--cpuset-cpus`: CPUs in which to allow execution (0-3, 0,1) - :whale: `--cpuset-mems`: Memory nodes (MEMs) in which to allow execution (0-3, 0,1). Only effective on NUMA systems - :whale: `--cpu-rt-period`: Limit CPU real-time period in microseconds. Only supported with cgroup v1. - :whale: `--cpu-rt-runtime`: Limit CPU real-time runtime in microseconds. Only supported with cgroup v1. - :whale: `--memory`: Memory limit - :whale: `--memory-reservation`: Memory soft limit - :whale: `--memory-swap`: Swap limit equal to memory plus swap: '-1' to enable unlimited swap - :whale: `--memory-swappiness`: Tune container memory swappiness (0 to 100) (default -1) - :whale: `--kernel-memory`: Kernel memory limit (deprecated) - :whale: `--oom-kill-disable`: Disable OOM Killer - :whale: `--oom-score-adj`: Tune container’s OOM preferences (-1000 to 1000, rootless: 100 to 1000) - :whale: `--pids-limit`: Tune container pids limit - :nerd_face: `--cgroup-conf`: Configure cgroup v2 (key=value) - :whale: `--blkio-weight`: Block IO (relative weight), between 10 and 1000, or 0 to disable (default 0) - :whale: `--blkio-weight-device`: Block IO weight (relative device weight) - :whale: `--device-read-bps`: Limit read rate (bytes per second) from a device - :whale: `--device-read-iops`: Limit read rate (IO per second) from a device - :whale: `--device-write-bps`: Limit write rate (bytes per second) to a device - :whale: `--device-write-iops`: Limit write rate (IO per second) to a device - :whale: `--cgroupns=(host|private)`: Cgroup namespace to use - Default: "private" on cgroup v2 hosts, "host" on cgroup v1 hosts - :whale: `--cgroup-parent`: Optional parent cgroup for the container - :whale: `--device`: Add a host device to the container Intel RDT flags: - :nerd_face: `--rdt-class=CLASS`: Name of the RDT class (or CLOS) to associate the container wit User flags: - :whale: `-u, --user`: Username or UID (format: [:]) - :nerd_face: `--umask`: Set the umask inside the container. Defaults to 0022. Corresponds to Podman CLI. - :whale: `--group-add`: Add additional groups to join - :whale: `--userns`: Set it to `host` to disable user namespacing set in nerdctl.toml or in cli. Security flags: - :whale: `--security-opt seccomp=`: specify custom seccomp profile - :whale: `--security-opt apparmor=`: specify custom AppArmor profile - :whale: `--security-opt label=`: specify custom selinux label - :whale: `--security-opt no-new-privileges`: disallow privilege escalation, e.g., setuid and file capabilities - :whale: `--security-opt systempaths=unconfined`: Turn off confinement for system paths (masked paths, read-only paths) for the container - :whale: `--security-opt writable-cgroups`: making the cgroups writeable - :nerd_face: `--security-opt privileged-without-host-devices`: Don't pass host devices to privileged containers - :whale: `--cap-add=`: Add Linux capabilities - :whale: `--cap-drop=`: Drop Linux capabilities - :whale: `--privileged`: Give extended privileges to this container - :nerd_face: `--systemd=(true|false|always)`: Enable systemd compatibility (default: false). - Default: "false" - true: Enable systemd compatibility is enabled if the entrypoint executable matches one of the following paths: - `/sbin/init` - `/usr/sbin/init` - `/usr/local/sbin/init` - always: Always enable systemd compatibility Corresponds to Podman CLI. Runtime flags: - :whale: `--runtime`: Runtime to use for this container, e.g. \"crun\", or \"io.containerd.runsc.v1\". - :whale: `--sysctl`: Sysctl options, e.g \"net.ipv4.ip_forward=1\" Volume flags: - :whale: `-v, --volume :[:]`: Bind mount a volume, e.g., `-v /mnt:/mnt:ro` - :whale: option `rw` : Read/Write (when writable) - :whale: option `ro` : Read-only. Recursively read-only (e.g., making children such as `/mnt/usb` read-only, too) when the kernel and the OCI runtime support it (kernel >= 5.12, and runc >= 1.1 or crun >= 1.8.6, as in Docker v25), otherwise non-recursive read-only. Use `--mount type=bind,...,readonly,bind-recursive=` to control the recursive read-only mode explicitly. - :nerd_face: option `rro`: **Deprecated** since the same feature was introduced in Docker v25 with a different syntax; use `--mount type=bind,...,readonly,bind-propagation=rprivate,bind-recursive=readonly` instead. Recursive read-only. Should be used in conjunction with `rprivate`. e.g., `-v /mnt:/mnt:rro,rprivate` makes children such as `/mnt/usb` to be read-only, too. Requires kernel >= 5.12, and runc >= 1.1 or crun >= 1.8.6; an error is raised when the recursive read-only mount is not supported. - :whale: option `shared`, `slave`, `private`: Non-recursive "shared" / "slave" / "private" propagation - :whale: option `rshared`, `rslave`, `rprivate`: Recursive "shared" / "slave" / "private" propagation - :nerd_face: option `bind`: Not-recursively bind-mounted - :nerd_face: option `rbind`: Recursively bind-mounted - :whale: option `z`: SELinux shared (multi-category) relabel of the volume content so it can be shared among containers - :whale: option `Z`: SELinux private unshared relabel of the volume content for this container only - Requires SELinux on the host and nerdctl started with `--selinux-enabled` (or `selinux_enabled = true` in `nerdctl.toml`). - Example: `nerdctl run --rm -v /var/data:/data:Z --selinux-enabled IMAGE` - :whale: `--tmpfs`: Mount a tmpfs directory, e.g. `--tmpfs /tmp:size=64m,exec`. - :whale: `--mount`: Attach a filesystem mount to the container. Consists of multiple key-value pairs, separated by commas and each consisting of a `=` tuple. e.g., `-- mount type=bind,source=/src,target=/app,bind-propagation=shared`. - :whale: `type`: Current supported mount types are `bind`, `volume`, `tmpfs`, `image`. The default type will be set to `volume` if not specified. i.e., `--mount src=vol-1,dst=/app,readonly` equals `--mount type=volume,src=vol-1,dst=/app,readonly` - Common Options: - :whale: `src`, `source`: Mount source spec for bind and volume. Mandatory for bind. - :whale: `dst`, `destination`, `target`: Mount destination spec. - :whale: `readonly`, `ro`: mount the filesystem read-only. Recursively read-only when the kernel and the OCI runtime support it (kernel >= 5.12, and runc >= 1.1 or crun >= 1.8.6, as in Docker v25). See the `bind-recursive` option below to control the recursive read-only mode explicitly. - :nerd_face: `rro`: **Deprecated** since the same feature was introduced in Docker v25 with a different syntax; use `readonly` with `bind-propagation=rprivate` and `bind-recursive=readonly` instead. Mount the filesystem recursively read-only. - Options specific to `bind`: - :whale: `bind-propagation`: `shared`, `slave`, `private`, `rshared`, `rslave`, or `rprivate`(default). - :whale: `bind-recursive`: `enabled`(default), `disabled`, `writable`, or `readonly`. - `enabled`: submounts are recursively bind-mounted, and a `readonly` mount is recursively read-only when the kernel and the OCI runtime support it. - `disabled`: submounts are not recursively bind-mounted. - `writable`: submounts of a `readonly` mount are kept writable (the default behavior of Docker until v24). - `readonly`: a `readonly` mount is forced to be recursively read-only; an error is raised when the kernel or the OCI runtime does not support it. Requires `bind-propagation=rprivate` to be specified in conjunction. Whether the OCI runtime supports recursive read-only mounts is detected by running `$RUNTIME features`, and the result is cached in the XDG cache directory (e.g., `~/.cache/nerdctl/oci-runtime-features`). - :whale: `bind-nonrecursive`: `true` or `false`(default). Deprecated alias for `bind-recursive=disabled` / `bind-recursive=enabled` (removed in Docker v29). If set to true, submounts are not recursively bind-mounted. - unimplemented options: `consistency` - Options specific to `tmpfs`: - :whale: `tmpfs-size`: Size of the tmpfs mount in bytes. Unlimited by default. - :whale: `tmpfs-mode`: File mode of the tmpfs in **octal**. Defaults to `1777` or world-writable. - Options specific to `volume`: - unimplemented options: `volume-nocopy`, `volume-label`, `volume-driver`, `volume-opt` - Options specific to `image`: - :whale: `src`, `source`: image reference (mandatory). - :whale: Currently, the image filesystem is mounted read-only. - unimplemented options: `image-subpath` - :whale: `--volumes-from`: Mount volumes from the specified container(s), e.g. "--volumes-from my-container". Rootfs flags: - :whale: `--read-only`: Mount the container's root filesystem as read only - :nerd_face: `--rootfs`: The first argument is not an image but the rootfs to the exploded container. Corresponds to Podman CLI. Env flags: - :whale: `--entrypoint`: Overwrite the default ENTRYPOINT of the image - :whale: `-w, --workdir`: Working directory inside the container - :whale: `-e, --env`: Set environment variables - :whale: `--env-file`: Set environment variables from file Metadata flags: - :whale: `--name`: Assign a name to the container - :whale: `-l, --label`: Set meta data on a container (Not passed through the OCI runtime since nerdctl v2.0, with an exception for `nerdctl/bypass4netns`) - :whale: `--label-file`: Read in a line delimited file of labels - :whale: `--annotation`: Add an annotation to the container (passed through to the OCI runtime) - :whale: `--cidfile`: Write the container ID to the file - :nerd_face: `--pidfile`: file path to write the task's pid. The CLI syntax conforms to Podman convention. Health check flags: - :whale: `--health-cmd`: Command to run to check container health - :whale: `--health-interval`: Time between running the check (e.g., 30s, 1m) - :whale: `--health-timeout`: Time to wait before considering the check failed (e.g., 5s) - :whale: `--health-retries`: Number of failures before container is considered unhealthy - :whale: `--health-start-period`: Start period for the container to initialize before starting health-retries countdown - :whale: `--no-healthcheck`: Disable any health checks defined by image or CLI Logging flags: - :whale: `--log-driver=(json-file|journald|fluentd|syslog|none)`: Logging driver for the container (default `json-file`). - :whale: `--log-driver=json-file`: The logs are formatted as JSON. The default logging driver for nerdctl. - The `json-file` logging driver supports the following logging options: - :whale: `--log-opt=max-size=`: The maximum size of the log before it is rolled. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to unlimited. - :whale: `--log-opt=max-file=`: The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. Only effective when `max-size` is also set. A positive integer. Defaults to 1. - :nerd_face: `--log-opt=log-path=`: The log path where the logs are written. The path will be created if it does not exist. If the log file exists, the old file will be renamed to `.1`. - Default: `////-json.log` - Example: `/var/lib/nerdctl/1935db59/containers/default//-json.log` - :whale: `--log-opt labels=production_status,geo`: A comma-separated list of logging-related labels this daemon accepts. - :whale: `--log-opt env=os,customer`: A comma-separated list of logging-related environment variables this daemon accepts. - :whale: `--log-driver=journald`: Writes log messages to `journald`. The `journald` daemon must be running on the host machine. - :whale: `--log-opt=tag=