### Index

# Overview



## What is k3d?

k3d is a lightweight wrapper to run [k3s](https://github.com/rancher/k3s) (Rancher Lab's minimal Kubernetes distribution) in docker.

k3d makes it very easy to create single- and multi-node [k3s](https://github.com/rancher/k3s) clusters in docker, e.g. for local development on Kubernetes.

**Note:** k3d is a **community-driven project** but it's not an official Rancher (SUSE) product.
**Sponsoring**: To spend any significant amount of time improving k3d, we rely on sponsorships:

- [**GitHub Sponsors**: ](https://github.com/sponsors/k3d-io)
- [**LiberaPay**: ](https://liberapay.com/k3d-io)
- [**IssueHunt**: ](https://issuehunt.io/r/k3d-io/k3d)

??? Tip "View a quick demo"
    <asciinema-player src="/static/asciicast/20210917_k3d_v5.0.0_01.cast" cols=200 rows=32></asciinema-player>

## Learning

!!! Tip "k3d demo repository: [k3d-io/k3d-demo](https://github.com/k3d-io/k3d-demo)"
    Featured use-cases include:

    - **hot-reloading** of code when developing on k3d (Python Flask App)
    - build-deploy-test cycle using **Tilt**
    - full cluster lifecycle for simple and **multi-server** clusters
    - Proof of Concept of using k3d as a service in **Drone CI**

- [Rancher Meetup - May 2020 - Simplifying Your Cloud-Native Development Workflow With K3s, K3c and K3d (YouTube)](https://www.youtube.com/watch?v=hMr3prm9gDM)

## Requirements

- [**docker**](https://docs.docker.com/install/) to be able to use k3d at all
  - Note: k3d v5.x.x requires at least Docker v20.10.5 (runc >= v1.0.0-rc93) to work properly (see [#807](https://github.com/k3d-io/k3d/issues/807))
- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) to interact with the Kubernetes cluster

## Releases

| Platform | Stage | Version | Release Date | Downloads so far |
|-----------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|---|
| [**GitHub Releases**](https://github.com/k3d-io/k3d/releases) | stable | [](https://github.com/k3d-io/k3d/releases/latest) | [](https://github.com/k3d-io/k3d/releases/latest) |  |
| [**GitHub Releases**](https://github.com/k3d-io/k3d/releases) | latest | [](https://github.com/k3d-io/k3d/releases) | [](https://github.com/k3d-io/k3d/releases) |  |
| [**Homebrew**](https://formulae.brew.sh/formula/k3d) | stable | [](https://formulae.brew.sh/formula/k3d) | - | - |
| [**Chocolatey**](https://chocolatey.org/packages/k3d/)| stable | [](https://chocolatey.org/packages/k3d/) | - | - |
| [**Scoop**](https://github.com/ScoopInstaller/Main/blob/master/bucket/k3d.json/)| stable | [](https://github.com/ScoopInstaller/Main/blob/master/bucket/k3d.json/) | - | - |

## Installation

You have several options there:

### [:fontawesome-regular-file-code: Install Script](https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh)

#### Install current latest release

- wget:

    ```bash
    wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
    ```

- curl:

    ```bash
    curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash
    ```

#### Install specific release

Use the install script to grab a specific release (via `TAG` environment variable):

- wget:

    ```bash
    wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG=v5.0.0 bash
    ```

- curl:

    ```bash
    curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG=v5.0.0 bash
    ```

### Other Installers

??? Tip "Other Installation Methods"

    - [:fontawesome-solid-beer: Homebrew (MacOS/Linux)](https://brew.sh):
        ```bash
        brew install k3d
        ```

        *Note*: The formula can be found in [homebrew/homebrew-core](https://github.com/Homebrew/homebrew-core/blob/master/Formula/k3d.rb) and is mirrored to [homebrew/linuxbrew-core](https://github.com/Homebrew/linuxbrew-core/blob/master/Formula/k3d.rb)

    - [:material-arch: AUR (Arch Linux User Repository)](https://aur.archlinux.org/):
        ```bash
        yay -S rancher-k3d-bin
        ```

        Package [rancher-k3d-bin](https://aur.archlinux.org/packages/rancher-k3d-bin/)

    - [:material-github: Download GitHub Release](https://github.com/k3d-io/k3d/releases)

      Grab a release binary from the [release tab](https://github.com/k3d-io/k3d/releases) and install it yourself

    - [:material-microsoft-windows: Chocolatey (Windows)](https://chocolatey.org/): `choco install k3d`

      *Note*: package source can be found in [erwinkersten/chocolatey-packages](https://github.com/erwinkersten/chocolatey-packages/tree/master/automatic/k3d)

    - [:material-microsoft-windows: Scoop (Windows)](https://scoop.sh//): `scoop install k3d`

      *Note*: package source can be found in [ScoopInstaller/Main](https://github.com/ScoopInstaller/Main/blob/master/bucket/k3d.json)

    - [arkade](https://github.com/alexellis/arkade): `arkade get k3d`

    - [asdf](https://asdf-vm.com): `asdf plugin-add k3d && asdf install k3d latest`

      *Note*: `asdf plugin-add k3d`, then `asdf install k3d <tag>` with `<tag> = latest` or `5.x.x` for a specific version (maintained by [spencergilbert/asdf-k3d](https://github.com/spencergilbert/asdf-k3d))

    - Others
      - install via go: `#!bash go install github.com/k3d-io/k3d/v5@latest` (**Note**: this will give you unreleased/bleeding-edge changes)

## Quick Start

Create a cluster named `mycluster` with just a single server node:

```bash
k3d cluster create mycluster
```

Use the new cluster with [`kubectl`](https://kubernetes.io/docs/tasks/tools/install-kubectl/), e.g.:

```bash
kubectl get nodes
```

??? Note "Getting the cluster's kubeconfig (included in `k3d cluster create`)"
    Get the new cluster's connection details merged into your default kubeconfig (usually specified using the `KUBECONFIG` environment variable or the default path `#!bash $HOME/.kube/config`) and directly switch to the new context:

    ```bash
    k3d kubeconfig merge mycluster --kubeconfig-switch-context
    ```

## Connect

1. Join the Rancher community on slack via [slack.rancher.io](https://slack.rancher.io/)
2. Go to [rancher-users.slack.com](https://rancher-users.slack.com) and join our channel #k3d
3. Start chatting

## Related Projects

- [vscode-k3d](https://github.com/inercia/vscode-k3d/): VSCode Extension to handle k3d clusters from within VSCode
- [k3x](https://github.com/inercia/k3x): a graphics interface (for Linux) to k3d.
- [AbsaOSS/k3d-action](https://github.com/AbsaOSS/k3d-action): fully customizable GitHub Action to run lightweight Kubernetes clusters.
- [AutoK3s](https://github.com/cnrancher/autok3s): a lightweight tool to help run K3s everywhere including k3d provider.
- [nolar/setup-k3d-k3s](https://github.com/nolar/setup-k3d-k3s): setup K3d/K3s for GitHub Actions.

---

### Design/Concepts

# Concepts

## Nodefilters

### About

Nodefilters are a concept in k3d to specify which nodes of a newly created cluster a condition or setting should apply to.

### Syntax

The overall syntax is `@<group>:<subset>[:<suffix]`.

- `@` denotes the start of a nodefilter in a k3d flag value
- `<group>` denotes the node group you want to filter in
  - one of `server`, `servers`, `agent`, `agents`, `loadbalancer`, `all`
    - note, that `all` also includes the cluster-external server loadbalancer (`k3d-proxy` container)
- `<subset>` denotes the subset of the chosen group you want to apply the flag to
  - wildcard `*`: all nodes in that group
  - index, e.g. `0`: only the first node of that group
  - list, e.g. `1,3,5`: nodes 1, 3 and 5 of that group
  - range, e.g. `2-4`: nodes 2 to 4 of that group
- `<suffix>` (optional) can hold some flag specific configuration
  - e.g. for the `--port` flag this could be `direct` or `proxy` (default) to configure the way of exposing ports

### Example

- Problem: You want to have Nginx as your ingress controller, but by default, K3s deploys Traefik.
- Solution: Disabling the default Traefik deployment using K3s' `--disable=traefik` flag.
- Note: It's enough to do this on the first (initializing) server node.
- How-To: `k3d cluster create notraefik --k3s-arg="--disable=traefik@server:0"`
  - Looking at `--k3s-arg="--disable=traefik@server:0"`, everything after the `@` sign is part of the nodefilter.
    - `server` is the node group: server nodes
    - after the `:` follows the subset, which in this case is the index `0`: the first server node to be created (`k3d-notraefik-server-0`)

---

### Design/Defaults

# Defaults

## k3d reserved settings

When you create a K3s cluster in Docker using k3d, we make use of some K3s configuration options, making them "reserved" for k3d.
This means, that overriding those options with your own may break the cluster setup.

### Environment Variables

The following K3s environment variables are used to configure the cluster:

| Variable | K3d Default | Configurable? |
|----------|-------------|---------------|
| `K3S_URL`| `https://$CLUSTERNAME-server-0:6443` | no |
| `K3S_TOKEN`| random | yes (`--token`) |
| `K3S_KUBECONFIG_OUTPUT`| `/output/kubeconfig.yaml` | no |

## k3d Loadbalancer

By default, k3d creates an Nginx loadbalancer alongside the clusters it creates to handle the port-forwarding.
The loadbalancer can partly be configured using k3d-defined settings.

| Nginx setting | k3d default | k3d setting |
|-------------|-------------|-------------|
| `proxy_timeout` (default for all server stanzas) | `600` (s) | `settings.defaultProxyTimeout` |  |
|`worker_connections` | `1024` | `settings.workerConnections` |

### Overrides

- Example via CLI: `k3d cluster create --lb-config-override settings.defaultProxyTimeout=900`
- Example via Config File:

  ```yaml
  # ... truncated ...
  k3d:
    loadbalancer:
      configOverrides:
        - settings.workerConnections=2048
  ```

## Multiple server nodes

- by default, when `--server` > 1 and no `--datastore-x` option is set, the first server node (server-0) will be the initializing server node
  - the initializing server node will have the `--cluster-init` flag appended
  - all other server nodes will refer to the initializing server node via `--server https://<init-node>:6443`

## API-Ports

- by default, we expose the API-Port (`6443`) by forwarding traffic from the default server loadbalancer (nginx container) to the server node(s)
- port `6443` of the loadbalancer is then mapped to a specific (`--api-port` flag) or a random (default) port on the host system

## Kubeconfig

- if `--kubeconfig-update-default` is set, we use the default loading rules to get the default kubeconfig:
  - First: kubeconfig specified via the KUBECONFIG environment variable (error out if multiple are specified)
  - Second: default kubeconfig in home directory (e.g. `$HOME/.kube/config`)

## Networking

- [by default, k3d creates a new (docker) network for every cluster](./networking.md)

---

### Design/Networking

# Networking

- Related issues:
  - [k3d-io/k3d #220](https://github.com/k3d-io/k3d/issues/220)

## Introduction

By default, k3d creates a new (docker) network for every new cluster.  
Using the `--network STRING` flag upon creation to connect to an existing network.  
Existing networks won't be managed by k3d together with the cluster lifecycle.

## Connecting to docker "internal"/pre-defined networks

### `host` network

When using the `--network` flag to connect to the host network (i.e. `k3d cluster create --network host`), you won't be able to create more than **one server node**.  
An edge case would be one server node (with agent disabled) and one agent node.

### `bridge` network

By default, every network that k3d creates is working in `bridge` mode.  
But when you try to use `--network bridge` to connect to docker's internal `bridge` network, you may run into issues with grabbing certificates from the API-Server.  
Single-Node clusters should work though.

### `none` "network"

Well.. this doesn't really make sense for k3d anyway ¯\\\_(ツ)\_/¯

---

### Design/Project

# Project Overview

## About This Page

On this page we'll try to give an overview of all the moving bits and pieces in k3d to ease contributions to the project.

## Directory Overview

- [`.github/`](https://github.com/k3d-io/k3d/tree/main/.github)
  - templates for issues and pull requests
  - GitHub Action workflow definitions
- [`cmd/`](https://github.com/k3d-io/k3d/tree/main/cmd)
  - everything related to the actual k3d CLI, like the whole command tree, config initialization, argument parsing, etc.
- [`docgen/`](https://github.com/k3d-io/k3d/tree/main/docgen)
  - sub-module used to auto-generate the documentation for the CLI commands, which ends up in [`docs/usage/commands/`](https://github.com/k3d-io/k3d/tree/main/docs/usage/commands)
- [`docs/`](https://github.com/k3d-io/k3d/tree/main/docs)
  - all the resources used to build [k3d.io](https://k3d.io) using mkdocs
- [`pkg/`](<https://github.com/k3d-io/k3d/tree/main/pkg>)
  - the place where the magic happens.. here you find all the main logic of k3d
  - all function calls within [`cmd/`](https://github.com/k3d-io/k3d/tree/main/cmd) that do non-trivial things are imported from here
  - this (or rather sub-packages) is what other projects would import as a module to work with k3d without using the CLI
- [`proxy/`](https://github.com/k3d-io/k3d/tree/main/proxy)
  - configuration to build the [`k3d-io/k3d-proxy`](https://github.com/k3d-io/k3d/pkgs/container/k3d-proxy) container image which is used as a loadbalancer/proxy in front of (almost) every k3d cluster
  - this is basically just a combination of NGINX with confd and some k3d-specific configuration details
- [`tests/`](https://github.com/k3d-io/k3d/tree/main/tests)
  - a set of bash scripts used for end-to-end (E2E) tests of k3d
  - mostly used for all the functionality of the k3d CLI which cannot be tested using Go unit tests
- [`tools/`](https://github.com/k3d-io/k3d/tree/main/tools)
  - sub-module used to build the [`k3d-io/k3d-tools`](https://github.com/k3d-io/k3d/pkgs/container/k3d-tools) container image which supports some k3d functionality like `k3d image import`
- [`vendor/`](https://github.com/k3d-io/k3d/tree/main/vendor)
  - result of `go mod vendor`, which contains all dependencies of k3d
- [`version/`](https://github.com/k3d-io/k3d/tree/main/version)
  - package used to code k3d/k3s versions into releases
  - this is where `go build` injects the version tags when building k3d
    - that's the output you see when issuing `k3d version`

## Packages Overview

- [`pkg/`](https://github.com/k3d-io/k3d/tree/main/pkg)
  - [`actions/`](https://github.com/k3d-io/k3d/tree/main/pkg/actions)
    - hook actions describing actions (commands, etc.) that run at specific stages of the node/cluster lifecycle
      - e.g. writing configuration files to the container filesystem just before the node (container) starts
  - [`client/`](https://github.com/k3d-io/k3d/tree/main/pkg/client)
    - all the top level functionality to work with k3d primitives
      - create/retrieve/update/delete/start/stop clusters, nodes, registries, etc. managed by k3d
  - [`config/`](https://github.com/k3d-io/k3d/tree/main/pkg/config)
    - everything related to the k3d configuration (files), like `SimpleConfig` and `ClusterConfig`
  - [`runtimes/`](https://github.com/k3d-io/k3d/tree/main/pkg/runtimes)
    - interface and implementations of runtimes that power k3d (currently, that's only Docker)
    - functions in [`client/`](https://github.com/k3d-io/k3d/tree/main/pkg/client) eventually call runtime functions to "materialize" nodes and clusters
  - [`tools/`](https://github.com/k3d-io/k3d/tree/main/pkg/tools)
    - functions eventually calling the [`k3d-tools`](https://github.com/k3d-io/k3d/pkgs/container/k3d-tools) container (see [`tools/`](https://github.com/k3d-io/k3d/tree/main/tools) in the repo root)
  - [`types/`](https://github.com/k3d-io/k3d/tree/main/pkg/types)
    - definition of all k3d primitives and many other details and defaults
    - e.g. contains the definition of a `Node` or a `Cluster` in k3d
  - [`util/`](https://github.com/k3d-io/k3d/tree/main/pkg/util)
    - some helper functions e.g. for string manipulation/generation, regexp or other re-usable usages

## Anatomy of a Cluster

By default, every k3d cluster consists of at least 2 containers (nodes):

1. (optional, but default and strongly recommended) loadbalancer

   - image: [`ghcr.io/k3d-io/k3d-proxy`](https://github.com/k3d-io/k3d/pkgs/container/k3d-proxy), built from [`proxy/`](https://github.com/k3d-io/k3d/tree/main/proxy)
   - purpose: proxy and load balance requests from the outside (i.e. most of the times your local host) to the cluster
     - by default, it e.g. proxies all the traffic for the Kubernetes API to port `6443` (default listening port of K3s) to all the server nodes in the cluster
     - can be used for multiple port-mappings to one or more nodes in your cluster
       - that way, port-mappings can also easily be added/removed after the cluster creation, as we can simply re-create the proxy without affecting cluster state

2. (required, always present) primary server node

   - image: [`rancher/k3s`](https://hub.docker.com/r/rancher/k3s/), built from [`github.com/k3s-io/k3s`](https://github.com/k3s-io/k3s)
   - purpose: (initializing) server (formerly: master) node of the cluster
     - runs the K3s executable (which runs containerd, the Kubernetes API Server, etcd/sqlite, etc.): `k3s server`
     - in a multi-server setup, it initializes the cluster with an embedded etcd database (using the K3s `--cluster-init` flag)

3. (optional) secondary server node(s)

   - image: [`rancher/k3s`](https://hub.docker.com/r/rancher/k3s/), built from [`github.com/k3s-io/k3s`](https://github.com/k3s-io/k3s)

4. (optional) agent node(s)

   - image: [`rancher/k3s`](https://hub.docker.com/r/rancher/k3s/), built from [`github.com/k3s-io/k3s`](https://github.com/k3s-io/k3s)
   - purpose: running the K3s agent process (kubelet, etc.): `k3s agent`

## Automation (CI)

The k3d repository mainly leverages the following two CI systems:

- GitHub Actions
  - 2 workflows in <https://github.com/k3d-io/k3d/tree/main/.github/workflows> to push the artifact to AUR (Arch Linux User Repository)
  - logs/history can be seen in the Actions tab: <https://github.com/k3d-io/k3d/actions>
  - static code analysis
  - build
  - tests
  - docker builds + pushes
  - render + push docs
  - (pre-) release to GitHub

## Documentation

The website [k3d.io](https://k3d.io) containing all the documentation for k3d is built using [`mkdocs`](https://www.mkdocs.org/), configured via the [`mkdocs.yml`](https://github.com/k3d-io/k3d/blob/main/mkdocs.yml) config file with all the content residing in the [`docs/`](https://github.com/k3d-io/k3d/tree/main/docs) directory (Markdown).  
Use `mkdocs serve` in the repository root to build and serve the webpage locally.  
Some parts of the documentation are being auto-generated, like [`docs/usage/commands/`](https://github.com/k3d-io/k3d/tree/main/docs/usage/commands) is auto-generated using Cobra's command docs generation functionality in [`docgen/`](https://github.com/k3d-io/k3d/tree/main/docgen).

---

### Faq/Compatibility

# Compatibility

With each release, we test if k3d works with specific versions of Docker and K3s, to ensure, that at least the most recent versions of Docker and the active releases (i.e. non-EOL release channels, similar to Kubernetes) work properly with it.
The tests happen automatically in GitHub Actions.
Some versions of Docker and K3s are expected to fail with specific versions of k3d due to e.g. incompatible dependencies or missing features.
We test a full cluster lifecycle with different [K3s channels](https://update.k3s.io/v1-release/channels), meaning that the following list refers to the current latest version released under the given channel.

## Releases

### v5.4.0 - 26.03.2022

Test Workflow: <https://github.com/k3d-io/k3d/actions/runs/2044325827>

#### Docker

* 20.10.5
* 20.10.12

**Expected to Fail** with the following versions:

* <= 20.10.4 (due to runc, see <https://github.com/k3d-io/k3d/issues/807>)

#### K3s

* Channel v1.23
* Channel v1.22

**Expected to Fail** with the following versions:

* <= v1.18 (due to not included, but expected CoreDNS in K3s)

### v5.3.0 - 03.02.2022

#### Docker

* 20.10.5
* 20.10.12

**Expected to Fail** with the following versions:

* <= 20.10.4 (due to runc, see <https://github.com/k3d-io/k3d/issues/807>)

#### K3s

* Channel v1.23
* Channel v1.22

**Expected to Fail** with the following versions:

* <= v1.18 (due to not included, but expected CoreDNS in K3s)

---

### Faq/Faq

# FAQ

## Issues with BTRFS

- As [@jaredallard](https://github.com/jaredallard) [pointed out](https://github.com/k3d-io/k3d/pull/48), people running `k3d` on a system with **btrfs**, may need to mount `/dev/mapper` into the nodes for the setup to work.
  - This will do: `#!bash k3d cluster create CLUSTER_NAME -v /dev/mapper:/dev/mapper`

## Issues with ZFS

- k3s currently has [no support for ZFS](https://github.com/rancher/k3s/issues/66) and thus, creating multi-server setups (e.g. `#!bash k3d cluster create multiserver --servers 3`) fails, because the initializing server node (server flag `--cluster-init`) errors out with the following log:

  ```bash
  starting kubernetes: preparing server: start cluster and https: raft_init(): io: create I/O capabilities probe file: posix_allocate: operation not supported on socket
  ```

  - This issue can be worked around by providing docker with a different filesystem (that's also better for docker-in-docker stuff).
  - A possible solution can be found here: [https://github.com/rancher/k3s/issues/1688#issuecomment-619570374](https://github.com/rancher/k3s/issues/1688#issuecomment-619570374)

## Pods evicted due to lack of disk space

- Pods go to evicted state after doing X
  - Related issues: [#133 - Pods evicted due to `NodeHasDiskPressure`](https://github.com/k3d-io/k3d/issues/133) (collection of #119 and #130)
  - Background: somehow docker runs out of space for the k3d node containers, which triggers a hard eviction in the kubelet
  - Possible [fix/workaround by @zer0def](https://github.com/k3d-io/k3d/issues/133#issuecomment-549065666):
    - cleanup your host file system: Yes, your host file system may actually be quite packed, triggering the eviction threshold.
      - on large disks, you may still have quite a few GB leftover, which is more than enough. In that case, lower the threshold as per below.
    - use a docker storage driver which cleans up properly (e.g. overlay2)
    - clean up or expand docker root filesystem
    - change the kubelet's eviction thresholds upon cluster creation:

      ```bash
      k3d cluster create \
        --k3s-arg '--kubelet-arg=eviction-hard=imagefs.available<1%,nodefs.available<1%@agent:*' \
        --k3s-arg '--kubelet-arg=eviction-minimum-reclaim=imagefs.available=1%,nodefs.available=1%@agent:*'
      ```

## Passing additional arguments/flags to k3s (and on to e.g. the kube-apiserver)

- The Problem: Passing a feature flag to the Kubernetes API Server running inside k3s.
- Example: you want to enable the EphemeralContainers feature flag in Kubernetes
- Solution:

  ```bash
    k3d cluster create \
    --k3s-arg '--kube-apiserver-arg=feature-gates=EphemeralContainers=true@server:*' \
    --k3s-arg '--kube-scheduler-arg=feature-gates=EphemeralContainers=true@server:*' \
    --k3s-arg '--kubelet-arg=feature-gates=EphemeralContainers=true@agent:*'
  ```

  - **Note**: Be aware of where the flags require dashes (`--`) and where not.
    - the k3s flag (`--kube-apiserver-arg`) has the dashes
    - the kube-apiserver flag `feature-gates` doesn't have them (k3s adds them internally)

- Second example:

  ```bash
  k3d cluster create k3d-one \
    --k3s-arg "--cluster-cidr=10.118.0.0/17@server:*" \
    --k3s-arg "--service-cidr=10.118.128.0/17@server:*" \
    --k3s-arg "--disable=servicelb@server:*" \
    --k3s-arg "--disable=traefik@server:*" \
    --verbose
  ```

  - **Note**: There are many ways to use the `"` and `'` quotes, just be aware, that sometimes shells also try to interpret/interpolate parts of the commands

## How to access services (like a database) running on my Docker Host Machine

- As of version v3.1.0, we're injecting the `host.k3d.internal` entry into the k3d containers (k3s nodes) and into the CoreDNS ConfigMap, enabling you to access your host system by referring to it as `host.k3d.internal`

## Running behind a corporate proxy

Running k3d behind a corporate proxy can lead to some issues with k3d that have already been reported in more than one issue.  
Some can be fixed by passing the `HTTP_PROXY` environment variables to k3d, some have to be fixed in docker's `daemon.json` file and some are as easy as adding a volume mount.

## Pods fail to start: `x509: certificate signed by unknown authority`

- Example Error Message:

  ```bash
  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image "docker.io/rancher/pause:3.1": failed to pull image "docker.io/rancher/pause:3.1": failed to pull and unpack image "docker.io/rancher/pause:3.1": failed to resolve reference "docker.io/rancher/pause:3.1": failed to do request: Head https://registry-1.docker.io/v2/rancher/pause/manifests/3.1: x509: certificate signed by unknown authority
  ```

- Problem: inside the container, the certificate of the corporate proxy cannot be validated
- Possible Solution: Mounting the CA Certificate from your host into the node containers at start time via `k3d cluster create --volume /path/to/your/certs.crt:/etc/ssl/certs/yourcert.crt`
- Issue: [k3d-io/k3d#535](https://github.com/k3d-io/k3d/discussions/535#discussioncomment-474982)

## Spurious PID entries in `/proc` after deleting `k3d` cluster with shared mounts

- When you perform cluster create and deletion operations multiple times with **same cluster name** and **shared volume mounts**, it was observed that `grep k3d /proc/*/mountinfo` shows many spurious entries
- Problem: Due to above, at times you'll see `no space left on device: unknown` when a pod is scheduled to the nodes
- If you observe anything of above sort you can check for inaccessible file systems and unmount them by using below command (note: please remove `xargs umount -l` and check for the diff o/p first)
- `diff <(df -ha | grep pods | awk '{print $NF}') <(df -h | grep pods | awk '{print $NF}') | awk '{print $2}' | xargs umount -l`
- As per the conversation on [k3d-io/k3d#594](https://github.com/k3d-io/k3d/issues/594#issuecomment-837900646) above issue wasn't reported/known earlier and so there are high chances that it's not universal.

## [SOLVED] Nodes fail to start or get stuck in `NotReady` state with log `nf_conntrack_max: permission denied`

### Problem

- When: This happens when running k3d on a Linux system with a kernel version >= 5.12.2 (and others like >= 5.11.19) when creating a new cluster
  - the node(s) stop or get stuck with a log line like this: `<TIMESTAMP>  F0516 05:05:31.782902       7 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied`
- Why: The issue was introduced by a change in the Linux kernel ([Changelog 5.12.2](https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.12.2): [Commit](https://github.com/torvalds/linux/commit/671c54ea8c7ff47bd88444f3fffb65bf9799ce43)), that changed the netfilter_conntrack behavior in a way that `kube-proxy` is not able to set the `nf_conntrack_max` value anymore

### Workaround

- Workaround: as a workaround, we can tell `kube-proxy` to not even try to set this value:

  ```bash
  k3d cluster create \
    --k3s-arg "--kube-proxy-arg=conntrack-max-per-core=0@server:*" \
    --k3s-arg "--kube-proxy-arg=conntrack-max-per-core=0@agent:*" \
    --image rancher/k3s:v1.20.6-k3s
  ```

### Fix

- **Note**: k3d v4.4.5 already uses rancher/k3s:v1.21.1-k3s1 as the new default k3s image, so no workarounds needed there!

This is going to be fixed "upstream" in k3s itself in [rancher/k3s#3337](https://github.com/k3s-io/k3s/pull/3337) and backported to k3s versions as low as v1.18.

- **The fix was released and backported in k3s, so you don't need to use the workaround when using one of the following k3s versions (or later ones)**
  - v1.18.19-k3s1 ([rancher/k3s#3344](https://github.com/k3s-io/k3s/pull/3344))
  - v1.19.11-k3s1 ([rancher/k3s#3343](https://github.com/k3s-io/k3s/pull/3343))
  - v1.20.7-k3s1 ([rancher/k3s#3342](https://github.com/k3s-io/k3s/pull/3342))
  - v1.21.1-k3s1 ([rancher/k3s#3341](https://github.com/k3s-io/k3s/pull/3341)))
- Issue Reference: [rancher/k3s#607](https://github.com/k3d-io/k3d/issues/607)

## DockerHub Pull Rate Limit

### Problem

You're deploying something to the cluster using an image from DockerHub and the image fails to be pulled, with a `429` response code and a message saying `You have reached your pull rate limit. You may increase the limit by authenticating and upgrading`.

### Cause

This is caused by DockerHub's pull rate limit (see <https://docs.docker.com/docker-hub/download-rate-limit/>), which limits pulls from unauthenticated/anonymous users to 100 pulls per hour and for authenticated users (not paying customers) to 200 pulls per hour (as of the time of writing).

### Solution

a) use images from a private registry, e.g. configured as a pull-through cache for DockerHub  
b) use a different public registry without such limitations, if the same image is stored there  
c) authenticate containerd inside k3s/k3d to use your DockerHub user  

#### (c) Authenticate Containerd against DockerHub

1. Create a registry configuration file for containerd:

  ```yaml
  # saved as e.g. $HOME/registries.yaml
  configs:
    "registry-1.docker.io":
      auth:
        username: "$USERNAME"
        password: "$PASSWORD"
  ```

2. Create a k3d cluster using that config:

  ```bash
  k3d cluster create --registry-config $HOME/registries.yaml
  ```

3. Profit. That's it. In the test for this, we pulled the same image 120 times in a row (confirmed, that pull numbers went up), without being rate limited (as a non-paying, normal user)

## Longhorn in k3d

### Problem

Longhorn is not working when deployed in a K3s cluster spawned with k3d.

### Cause

The container image of K3s is quite limited and doesn't contain the necessary libraries.  Also, additional volume mounts and more would be required to get Longhorn up and running properly.  
So basically Longhorn does rely too much on the host OS to work properly in the dockerized environment without quite some modifications.

### Solution

There are a few ways one can build a working image to use with k3d.  
See <https://github.com/k3d-io/k3d/discussions/478> for more info.

---

### Usage/Advanced/Calico

# Use Calico instead of Flannel

!!! info "Network Policies"
    k3s comes with a controller that enforces network policies by default.
    While You do not need to switch to any CNIs for Kubernetes network policies to be enforced, other CNIs such as Calico can help you to bridge the gap where Kubernetes network policies may lack some capabilities. See <https://github.com/k3s-io/k3s/issues/1308> for more information.  
    The docs below assume you want to switch to Calico's policy engine, thus setting `--disable-network-policy@server:*`.

## 1. Create the cluster without flannel
By default K3s deploys flannel CNI to take care of networking in your environment.
Since we want to use Calico in this example we have to disable the default CNI.
This can be done by using the `--k3s-arg` flag at the cluster creation time.  

Use the following command to create your cluster:
```bash
k3d cluster create "${clustername}" \
  --k3s-arg '--flannel-backend=none@server:*' \
  --k3s-arg '--disable-network-policy@server:*' \
  --k3s-arg '--cluster-cidr=192.168.0.0/16@server:*'
```

In this example :

- Change the `"${clustername}"` with the name of the cluster (or set a variable).
- Cluster will use the "192.168.0.0/16" CIDR, if you want to change the default CIDR make sure to change it in the `custom-resources.yaml` too.

## 2. Install Calico 
A simple way to install Calico is to use the Tigera Operator.
The operator helps us to configure, install and upgrade Calico in an environment.

Use the following command to install the operator:
```bash
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/tigera-operator.yaml
```

The operator periodically checks for the installation manifest.
This manifest is how we instruct the Tigera Operator to install Calico.

Use the following command to create the installation manifest:
```bash
kubectl create -f https://raw.githubusercontent.com/projectcalico/calico/v3.29.0/manifests/custom-resources.yaml
```

At this point, our installation is in progress, and we can verify it by using the following command:
```bash
kubectl get tigerastatus
```

After a minute, you should see a result similar to the following:
```
NAME        AVAILABLE   PROGRESSING   DEGRADED   SINCE
apiserver   True        False         False      30s
calico      True        False         False      10s
ippools     True        False         False      70s
```

Great Calico is up and running!

## 3. IP forwarding
By default, Calico disables IP forwarding inside the containers.
This can cause an issue in some cases where you are using load balancers. You can learn more about loadblanacers [here](https://docs.k3s.io/networking/networking-services#service-load-balancer).
To fix this issue we have to turn on the IP forwarding flag inside `calico-node` pods.

Use the following command to enable forwarding via the operator:
```bash
kubectl patch installation default --type=merge --patch='{"spec":{"calicoNetwork":{"containerIPForwarding":"Enabled"}}}'
```

## 4. What's next?
Check out our other guides, here some suggestions:
- Add an additional node to your setup. [see](../commands/k3d_node.md)
- Expose your services. [see](../exposing_services.md)

## References

- <https://rancher.com/docs/k3s/latest/en/installation/network-options/>  
- <https://docs.tigera.io/calico/latest/getting-started/kubernetes/k3s/quickstart>

---

### Usage/Advanced/Cuda

# Running CUDA workloads

If you want to run CUDA workloads on the K3s container you need to customize the container.  
CUDA workloads require the NVIDIA Container Runtime, so containerd needs to be configured to use this runtime.  
The K3s container itself also needs to run with this runtime.  
If you are using Docker you can install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html).

## Building a customized K3s image

To get the NVIDIA container runtime in the K3s image you need to build your own K3s image.  
The native K3s image is based on Alpine but the NVIDIA container runtime is not supported on Alpine yet.  
To get around this we need to build the image with a supported base image.

### Dockerfile

[Dockerfile](cuda/Dockerfile):  

```Dockerfile
{%
  include-markdown "./cuda/Dockerfile"
  comments=false
%}
```

This Dockerfile is based on the [K3s Dockerfile](https://github.com/rancher/k3s/blob/master/package/Dockerfile)
The following changes are applied:

1. Change the base images to nvidia/cuda:12.4.1-base-ubuntu22.04 so the NVIDIA Container Toolkit can be installed. The version of `cuda:xx.x.x` must match the one you're planning to use.
2. Add a manifest for the NVIDIA driver plugin for Kubernetes with an added RuntimeClass definition. See [k3s documentation](https://docs.k3s.io/advanced#nvidia-container-runtime-support).

### The NVIDIA device plugin

To enable NVIDIA GPU support on Kubernetes you also need to install the [NVIDIA device plugin](https://github.com/NVIDIA/k8s-device-plugin). The device plugin is a daemonset and allows you to automatically:

* Expose the number of GPUs on each nodes of your cluster
* Keep track of the health of your GPUs
* Run GPU enabled containers in your Kubernetes cluster.

```yaml
{%
  include-markdown "./cuda/device-plugin-daemonset.yaml"
  comments=false
%}
```

Two modifications have been made to the original NVIDIA daemonset:

1. Added RuntimeClass definition to the YAML frontmatter.

   ```yaml
   apiVersion: node.k8s.io/v1
   kind: RuntimeClass
   metadata:
     name: nvidia
   handler: nvidia
   ```

2. Added `runtimeClassName: nvidia` to the Pod spec.

Note: you must explicitly add `runtimeClassName: nvidia` to all your Pod specs to use the GPU. See [k3s documentation](https://docs.k3s.io/advanced#nvidia-container-runtime-support).

### Build the K3s image

To build the custom image we need to build K3s because we need the generated output.

Put the following files in a directory:

* [Dockerfile](cuda/Dockerfile)
* [device-plugin-daemonset.yaml](cuda/device-plugin-daemonset.yaml)
* [build.sh](cuda/build.sh)
* [cuda-vector-add.yaml](cuda/cuda-vector-add.yaml)

The `build.sh` script is configured using exports & defaults to `v1.28.8+k3s1`. Please set at least the `IMAGE_REGISTRY` variable! The script performs the following steps builds the custom K3s image including the nvidia drivers.

[build.sh](cuda/build.sh):

```bash
{%
  include-markdown "./cuda/build.sh"
  comments=false
%}
```

## Run and test the custom image with k3d

You can use the image with k3d:

```bash
k3d cluster create gputest --image=$IMAGE --gpus=1
```

Deploy a [test pod](cuda/cuda-vector-add.yaml):

```bash
kubectl apply -f cuda-vector-add.yaml
kubectl logs cuda-vector-add
```

This should output something like the following:

```bash
$ kubectl logs cuda-vector-add

[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
```

If the `cuda-vector-add` pod is stuck in `Pending` state, probably the device-driver daemonset didn't get deployed correctly from the auto-deploy manifests. In that case, you can apply it manually via `#!bash kubectl apply -f device-plugin-daemonset.yaml`.

## Acknowledgements

Most of the information in this article was obtained from various sources:

* [Add NVIDIA GPU support to k3s with containerd](https://dev.to/mweibel/add-nvidia-gpu-support-to-k3s-with-containerd-4j17)
* [microk8s](https://github.com/ubuntu/microk8s)
* [K3s](https://github.com/rancher/k3s)
* [k3s-gpu](https://gitlab.com/vainkop1/k3s-gpu)

## Authors

* [@markrexwinkel](https://github.com/markrexwinkel)
* [@vainkop](https://github.com/vainkop)
* [@iwilltry42](https://github.com/iwilltry42)
* [@dbreyfogle](https://github.com/dbreyfogle)

---

### Usage/Advanced/Podman

# Using Podman instead of Docker

Podman has an [Docker API compatibility layer](https://podman.io/blogs/2020/06/29/podman-v2-announce.html#restful-api). k3d uses the Docker API and is compatible with Podman v4 and higher.

!!! important "Podman support is experimental"
    k3d is not guaranteed to work with Podman. If you find a bug, do help by [filing an issue](https://github.com/k3d-io/k3d/issues/new?labels=bug&template=bug_report.md&title=%5BBUG%5D+Podman)

Tested with podman version:
```bash
Client:       Podman Engine
Version:      4.3.1
API Version:  4.3.1
```

## Using Podman

Ensure the Podman system socket is available:

```bash
sudo systemctl enable --now podman.socket
# or to start the socket daemonless
# sudo podman system service --time=0 &
```

Disable timeout for podman service:<br>
See the [podman-system-service (1)](https://docs.podman.io/en/latest/markdown/podman-system-service.1.html) man page for more information.
```bash
mkdir -p /etc/containers/containers.conf.d
echo 'service_timeout=0' > /etc/containers/containers.conf.d/timeout.conf
```

To point k3d at the right Docker socket, create a symbolic link:

```bash
sudo ln -s /run/podman/podman.sock /var/run/docker.sock
# or install your system podman-docker if available
sudo k3d cluster create
```

Alternatively, set `DOCKER_HOST` when running k3d:

```bash
export DOCKER_HOST=unix:///run/podman/podman.sock
export DOCKER_SOCK=/run/podman/podman.sock
sudo --preserve-env=DOCKER_HOST --preserve-env=DOCKER_SOCK k3d cluster create
```

### Using rootless Podman

Ensure the Podman user socket is available:

```bash
systemctl --user enable --now podman.socket
# or podman system service --time=0 &
```

Set `DOCKER_HOST` when running k3d:

```bash
XDG_RUNTIME_DIR=${XDG_RUNTIME_DIR:-/run/user/$(id -u)}
export DOCKER_HOST=unix://$XDG_RUNTIME_DIR/podman/podman.sock
export DOCKER_SOCK=$XDG_RUNTIME_DIR/podman/podman.sock
k3d cluster create
```

#### Using cgroup (v2)

By default, a non-root user can only get memory controller and pids controller to be delegated.

To run properly we need to enable CPU, CPUSET, and I/O delegation

!!! note "Make sure you're running cgroup v2"
    If `/sys/fs/cgroup/cgroup.controllers` is present on your system, you are using v2, otherwise you are using v1.

```bash
mkdir -p /etc/systemd/system/user@.service.d
cat > /etc/systemd/system/user@.service.d/delegate.conf <<EOF
[Service]
Delegate=cpu cpuset io memory pids
EOF
systemctl daemon-reload
```

Reference: [https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation](https://rootlesscontaine.rs/getting-started/common/cgroup2/#enabling-cpu-cpuset-and-io-delegation)

### Using remote Podman

[Start Podman on the remote host](https://github.com/containers/podman/blob/main/docs/tutorials/remote_client.md), and then set `DOCKER_HOST` when running k3d:

```
export DOCKER_HOST=ssh://username@hostname
export DOCKER_SOCK=/run/user/1000/podman/podman.sock
k3d cluster create
```

### macOS

Initialize a podman machine if not done already

```
podman machine init
```

Or start an already existing podman machine

```
podman machine start
```

Grab connection details 

```
podman system connection ls
Name                         URI                                                         Identity                                      Default
podman-machine-default       ssh://core@localhost:53685/run/user/501/podman/podman.sock  /Users/myusername/.ssh/podman-machine-default  true
podman-machine-default-root  ssh://root@localhost:53685/run/podman/podman.sock           /Users/myusername/.ssh/podman-machine-default  false
```

Edit your OpenSSH config file to specify the IdentityFile

```
vim ~/.ssh/config

Host localhost
	IdentityFile /Users/myusername/.ssh/podman-machine-default
```

#### Rootless mode

Delegate the `cpuset` cgroup controller to the user's systemd slice, export the docker environment variables referenced above for the non-root connection, and create the cluster:

```bash
podman machine ssh bash -e <<EOF
  printf '[Service]\nDelegate=cpuset\n' | sudo tee /etc/systemd/system/user@.service.d/k3d.conf
  sudo systemctl daemon-reload
  sudo systemctl restart "user@\${UID}"
EOF

export DOCKER_HOST=ssh://core@localhost:53685
export DOCKER_SOCKET=/run/user/501/podman/podman.sock
k3d cluster create --k3s-arg '--kubelet-arg=feature-gates=KubeletInUserNamespace=true@server:*'
```

#### Rootful mode

Export the docker environment variables referenced above for the root connection and create the cluster:

```bash
export DOCKER_HOST=ssh://root@localhost:53685
export DOCKER_SOCK=/run/podman/podman.sock
k3d cluster create
```

### Podman network

The default `podman` network has dns disabled. To allow k3d cluster nodes to communicate with dns a new network must be created.
```bash
podman network create k3d
podman network inspect k3d -f '{{ .DNSEnabled }}'
true
```

## Creating local registries

Because Podman does not have a default "bridge" network, you have to specify a network using the `--default-network` flag when creating a local registry:

```bash
k3d registry create --default-network podman mycluster-registry
```

To use this registry with a cluster, pass the `--registry-use` flag:

```bash
k3d cluster create --registry-use mycluster-registry mycluster
```

!!! note "Incompatibility with `--registry-create`"
    Because `--registry-create` assumes the default network to be "bridge", avoid `--registry-create` when using Podman. Instead, always create a registry before creating a cluster.

!!! note "Missing cpuset cgroup controller"
    If you experince an error regarding missing cpuset cgroup controller, ensure the user unit `xdg-document-portal.service` is disabled by running `systemctl --user stop xdg-document-portal.service`. See [this issue](https://github.com/systemd/systemd/issues/18293#issuecomment-831397578)

---

### Usage/Commands/K3d

## k3d

https://k3d.io/ -> Run k3s in Docker!

### Synopsis

https://k3d.io/
k3d is a wrapper CLI that helps you to easily create k3s clusters inside docker.
Nodes of a k3d cluster are docker containers running a k3s image.
All Nodes of a k3d cluster are part of the same docker network.

```
k3d [flags]
```

### Options

```
  -h, --help         help for k3d
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
      --version      Show k3d and default k3s version
```

### SEE ALSO

* [k3d cluster](k3d_cluster.md)	 - Manage cluster(s)
* [k3d completion](k3d_completion.md)	 - Generate completion scripts for [bash, zsh, fish, powershell | psh]
* [k3d config](k3d_config.md)	 - Work with config file(s)
* [k3d image](k3d_image.md)	 - Handle container images.
* [k3d kubeconfig](k3d_kubeconfig.md)	 - Manage kubeconfig(s)
* [k3d node](k3d_node.md)	 - Manage node(s)
* [k3d registry](k3d_registry.md)	 - Manage registry/registries
* [k3d version](k3d_version.md)	 - Show k3d and default k3s version

---

### Usage/Commands/K3d Cluster

## k3d cluster

Manage cluster(s)

### Synopsis

Manage cluster(s)

```
k3d cluster [flags]
```

### Options

```
  -h, --help   help for cluster
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d](k3d.md)	 - https://k3d.io/ -> Run k3s in Docker!
* [k3d cluster create](k3d_cluster_create.md)	 - Create a new cluster
* [k3d cluster delete](k3d_cluster_delete.md)	 - Delete cluster(s).
* [k3d cluster edit](k3d_cluster_edit.md)	 - [EXPERIMENTAL] Edit cluster(s).
* [k3d cluster list](k3d_cluster_list.md)	 - List cluster(s)
* [k3d cluster start](k3d_cluster_start.md)	 - Start existing k3d cluster(s)
* [k3d cluster stop](k3d_cluster_stop.md)	 - Stop existing k3d cluster(s)

---

### Usage/Commands/K3d Cluster Create

## k3d cluster create

Create a new cluster

### Synopsis


Create a new k3s cluster with containerized nodes (k3s in docker).
Every cluster will consist of one or more containers:

- 1 (or more) server node container (k3s)
- (optionally) 1 loadbalancer container as the entrypoint to the cluster (nginx)
- (optionally) 1 (or more) agent node containers (k3s)


```
k3d cluster create NAME [flags]
```

### Options

```
/* Detailed source-code truncated for AI context efficiency. */
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d cluster](k3d_cluster.md)	 - Manage cluster(s)

---

### Usage/Commands/K3d Cluster Delete

## k3d cluster delete

Delete cluster(s).

### Synopsis

Delete cluster(s).

```
k3d cluster delete [NAME [NAME ...] | --all] [flags]
```

### Options

```
  -a, --all             Delete all existing clusters
  -c, --config string   Path of a config file to use
  -h, --help            help for delete
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d cluster](k3d_cluster.md)	 - Manage cluster(s)

---

### Usage/Commands/K3d Cluster Edit

## k3d cluster edit

[EXPERIMENTAL] Edit cluster(s).

### Synopsis

[EXPERIMENTAL] Edit cluster(s).

```
k3d cluster edit CLUSTER [flags]
```

### Options

```
  -h, --help                                                               help for edit
      --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]   [EXPERIMENTAL] Map ports from the node containers (via the serverlb) to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER])
                                                                            - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80`
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d cluster](k3d_cluster.md)	 - Manage cluster(s)

---

### Usage/Commands/K3d Cluster List

## k3d cluster list

List cluster(s)

### Synopsis

List cluster(s).

```
k3d cluster list [NAME [NAME...]] [flags]
```

### Options

```
  -h, --help            help for list
      --no-headers      Disable headers
  -o, --output string   Output format. One of: json|yaml
      --token           Print k3s cluster token
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d cluster](k3d_cluster.md)	 - Manage cluster(s)

---

### Usage/Commands/K3d Cluster Start

## k3d cluster start

Start existing k3d cluster(s)

### Synopsis

Start existing k3d cluster(s)

```
k3d cluster start [NAME [NAME...] | --all] [flags]
```

### Options

```
  -a, --all                Start all existing clusters
  -h, --help               help for start
      --timeout duration   Maximum waiting time for '--wait' before canceling/returning.
      --wait               Wait for the server(s) (and loadbalancer) to be ready before returning. (default true)
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d cluster](k3d_cluster.md)	 - Manage cluster(s)

---

### Usage/Commands/K3d Cluster Stop

## k3d cluster stop

Stop existing k3d cluster(s)

### Synopsis

Stop existing k3d cluster(s).

```
k3d cluster stop [NAME [NAME...] | --all] [flags]
```

### Options

```
  -a, --all    Stop all existing clusters
  -h, --help   help for stop
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d cluster](k3d_cluster.md)	 - Manage cluster(s)

---

### Usage/Commands/K3d Completion

## k3d completion

Generate completion scripts for [bash, zsh, fish, powershell | psh]

### Synopsis

To load completions:

Bash:

	$ source <(k3d completion bash)

	# To load completions for each session, execute once:
	# Linux:
	$ k3d completion bash > /etc/bash_completion.d/k3d
	# macOS:
	$ k3d completion bash > /usr/local/etc/bash_completion.d/k3d

Zsh:

	# If shell completion is not already enabled in your environment,
	# you will need to enable it.  You can execute the following once:

	$ echo "autoload -U compinit; compinit" >> ~/.zshrc

	# To load completions for each session, execute once:
	$ k3d completion zsh > "${fpath[1]}/_k3d"

	# You will need to start a new shell for this setup to take effect.

fish:

	$ k3d completion fish | source

	# To load completions for each session, execute once:
	$ k3d completion fish > ~/.config/fish/completions/k3d.fish

PowerShell:

	PS> k3d completion powershell | Out-String | Invoke-Expression

	# To load completions for every new session, run:
	PS> k3d completion powershell > k3d.ps1
	# and source this file from your PowerShell profile.


```
k3d completion SHELL
```

### Options

```
  -h, --help   help for completion
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d](k3d.md)	 - https://k3d.io/ -> Run k3s in Docker!

---

### Usage/Commands/K3d Config

## k3d config

Work with config file(s)

### Synopsis

Work with config file(s)

```
k3d config [flags]
```

### Options

```
  -h, --help   help for config
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d](k3d.md)	 - https://k3d.io/ -> Run k3s in Docker!
* [k3d config init](k3d_config_init.md)	 - 
* [k3d config migrate](k3d_config_migrate.md)	 -

---

### Usage/Commands/K3d Config Init

## k3d config init



```
k3d config init [flags]
```

### Options

```
  -f, --force           Force overwrite of target file
  -h, --help            help for init
  -o, --output string   Write a default k3d config (default "k3d-default.yaml")
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d config](k3d_config.md)	 - Work with config file(s)

---

### Usage/Commands/K3d Config Migrate

## k3d config migrate



```
k3d config migrate INPUT [OUTPUT] [flags]
```

### Options

```
  -h, --help   help for migrate
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d config](k3d_config.md)	 - Work with config file(s)

---

### Usage/Commands/K3d Image

## k3d image

Handle container images.

### Synopsis

Handle container images.

```
k3d image [flags]
```

### Options

```
  -h, --help   help for image
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d](k3d.md)	 - https://k3d.io/ -> Run k3s in Docker!
* [k3d image import](k3d_image_import.md)	 - Import image(s) from docker into k3d cluster(s).

---

### Usage/Commands/K3d Image Import

## k3d image import

Import image(s) from docker into k3d cluster(s).

### Synopsis

Import image(s) from docker into k3d cluster(s).

If an IMAGE starts with the prefix 'docker.io/', then this prefix is stripped internally.
That is, 'docker.io/k3d-io/k3d-tools:latest' is treated as 'k3d-io/k3d-tools:latest'.

If an IMAGE starts with the prefix 'library/' (or 'docker.io/library/'), then this prefix is stripped internally.
That is, 'library/busybox:latest' (or 'docker.io/library/busybox:latest') are treated as 'busybox:latest'.

If an IMAGE does not have a version tag, then ':latest' is assumed.
That is, 'k3d-io/k3d-tools' is treated as 'k3d-io/k3d-tools:latest'.

A file ARCHIVE always takes precedence.
So if a file './k3d-io/k3d-tools' exists, k3d will try to import it instead of the IMAGE of the same name.

```
k3d image import [IMAGE | ARCHIVE [IMAGE | ARCHIVE...]] [flags]
```

### Options

```
  -c, --cluster stringArray   Select clusters to load the image to. (default [k3s-default])
  -h, --help                  help for import
  -k, --keep-tarball          Do not delete the tarball containing the saved images from the shared volume
  -t, --keep-tools            Do not delete the tools node after import
  -m, --mode string           Which method to use to import images into the cluster [auto, direct, tools]. See https://k3d.io/stable/usage/importing_images/ (default "tools-node")
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d image](k3d_image.md)	 - Handle container images.

---

### Usage/Commands/K3d Kubeconfig

## k3d kubeconfig

Manage kubeconfig(s)

### Synopsis

Manage kubeconfig(s)

```
k3d kubeconfig [flags]
```

### Options

```
  -h, --help   help for kubeconfig
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d](k3d.md)	 - https://k3d.io/ -> Run k3s in Docker!
* [k3d kubeconfig get](k3d_kubeconfig_get.md)	 - Print kubeconfig(s) from cluster(s).
* [k3d kubeconfig merge](k3d_kubeconfig_merge.md)	 - Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file.

---

### Usage/Commands/K3d Kubeconfig Get

## k3d kubeconfig get

Print kubeconfig(s) from cluster(s).

### Synopsis

Print kubeconfig(s) from cluster(s).

```
k3d kubeconfig get [CLUSTER [CLUSTER [...]] | --all] [flags]
```

### Options

```
  -a, --all    Output kubeconfigs from all existing clusters
  -h, --help   help for get
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d kubeconfig](k3d_kubeconfig.md)	 - Manage kubeconfig(s)

---

### Usage/Commands/K3d Kubeconfig Merge

## k3d kubeconfig merge

Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file.

### Synopsis

Write/Merge kubeconfig(s) from cluster(s) into new or existing kubeconfig/file.

```
k3d kubeconfig merge [CLUSTER [CLUSTER [...]] | --all] [flags]
```

### Options

```
  -a, --all                         Get kubeconfigs from all existing clusters
  -h, --help                        help for merge
  -d, --kubeconfig-merge-default    Merge into the default kubeconfig ($KUBECONFIG or /home/thklein/.kube/config)
  -s, --kubeconfig-switch-context   Switch to new context (default true)
  -o, --output string               Define output [ - | FILE ] (default from $KUBECONFIG or /home/thklein/.kube/config
      --overwrite                   [Careful!] Overwrite existing file, ignoring its contents
  -u, --update                      Update conflicting fields in existing kubeconfig (default true)
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d kubeconfig](k3d_kubeconfig.md)	 - Manage kubeconfig(s)

---

### Usage/Commands/K3d Node

## k3d node

Manage node(s)

### Synopsis

Manage node(s)

```
k3d node [flags]
```

### Options

```
  -h, --help   help for node
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d](k3d.md)	 - https://k3d.io/ -> Run k3s in Docker!
* [k3d node create](k3d_node_create.md)	 - Create a new k3s node in docker
* [k3d node delete](k3d_node_delete.md)	 - Delete node(s).
* [k3d node edit](k3d_node_edit.md)	 - [EXPERIMENTAL] Edit node(s).
* [k3d node list](k3d_node_list.md)	 - List node(s)
* [k3d node start](k3d_node_start.md)	 - Start an existing k3d node
* [k3d node stop](k3d_node_stop.md)	 - Stop an existing k3d node

---

### Usage/Commands/K3d Node Create

## k3d node create

Create a new k3s node in docker

### Synopsis

Create a new containerized k3s node (k3s in docker).

```
k3d node create NAME [flags]
```

### Options

```
  -c, --cluster string           Cluster URL or k3d cluster name to connect to. (default "k3s-default")
  -h, --help                     help for create
  -i, --image string             Specify k3s image used for the node(s) (default: copied from existing node)
      --k3s-arg stringArray      Additional args passed to k3d command
      --k3s-node-label strings   Specify k3s node labels in format "foo=bar"
      --memory string            Memory limit imposed on the node [From docker]
  -n, --network strings          Add node to (another) runtime network
      --replicas int             Number of replicas of this node specification. (default 1)
      --role string              Specify node role [server, agent] (default "agent")
      --runtime-label strings    Specify container runtime labels in format "foo=bar"
      --runtime-ulimit strings   Specify container runtime ulimit in format "ulimit=soft:hard"
      --timeout duration         Maximum waiting time for '--wait' before canceling/returning.
  -t, --token string             Override cluster token (required when connecting to an external cluster)
      --wait                     Wait for the node(s) to be ready before returning. (default true)
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d node](k3d_node.md)	 - Manage node(s)

---

### Usage/Commands/K3d Node Delete

## k3d node delete

Delete node(s).

### Synopsis

Delete node(s).

```
k3d node delete (NAME | --all) [flags]
```

### Options

```
  -a, --all          Delete all existing nodes
  -h, --help         help for delete
  -r, --registries   Also delete registries
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d node](k3d_node.md)	 - Manage node(s)

---

### Usage/Commands/K3d Node Edit

## k3d node edit

[EXPERIMENTAL] Edit node(s).

### Synopsis

[EXPERIMENTAL] Edit node(s).

```
k3d node edit NODE [flags]
```

### Options

```
  -h, --help                                                               help for edit
      --port-add [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER]   [EXPERIMENTAL] (serverlb only!) Map ports from the node container to the host (Format: [HOST:][HOSTPORT:]CONTAINERPORT[/PROTOCOL][@NODEFILTER])
                                                                            - Example: `k3d node edit k3d-mycluster-serverlb --port-add 8080:80`
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d node](k3d_node.md)	 - Manage node(s)

---

### Usage/Commands/K3d Node List

## k3d node list

List node(s)

### Synopsis

List node(s).

```
k3d node list [NODE [NODE...]] [flags]
```

### Options

```
  -h, --help            help for list
      --no-headers      Disable headers
  -o, --output string   Output format. One of: json|yaml
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d node](k3d_node.md)	 - Manage node(s)

---

### Usage/Commands/K3d Node Start

## k3d node start

Start an existing k3d node

### Synopsis

Start an existing k3d node.

```
k3d node start NODE [flags]
```

### Options

```
  -h, --help   help for start
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d node](k3d_node.md)	 - Manage node(s)

---

### Usage/Commands/K3d Node Stop

## k3d node stop

Stop an existing k3d node

### Synopsis

Stop an existing k3d node.

```
k3d node stop NAME [flags]
```

### Options

```
  -h, --help   help for stop
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d node](k3d_node.md)	 - Manage node(s)

---

### Usage/Commands/K3d Registry

## k3d registry

Manage registry/registries

### Synopsis

Manage registry/registries

```
k3d registry [flags]
```

### Options

```
  -h, --help   help for registry
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d](k3d.md)	 - https://k3d.io/ -> Run k3s in Docker!
* [k3d registry create](k3d_registry_create.md)	 - Create a new registry
* [k3d registry delete](k3d_registry_delete.md)	 - Delete registry/registries.
* [k3d registry list](k3d_registry_list.md)	 - List registries

---

### Usage/Commands/K3d Registry Create

## k3d registry create

Create a new registry

### Synopsis

Create a new registry.

```
k3d registry create NAME [flags]
```

### Options

```
      --default-network string    Specify the network connected to the registry (default "bridge")
  -h, --help                      help for create
  -i, --image string              Specify image used for the registry (default "docker.io/library/registry:2")
      --no-help                   Disable the help text (How-To use the registry)
  -p, --port [HOST:]HOSTPORT      Select which port the registry should be listening on on your machine (localhost) (Format: [HOST:]HOSTPORT)
                                   - Example: `k3d registry create --port 0.0.0.0:5111` (default "random")
      --proxy-password string     Specify the password of the proxied remote registry
      --proxy-remote-url string   Specify the url of the proxied remote registry
      --proxy-username string     Specify the username of the proxied remote registry
  -v, --volume [SOURCE:]DEST      Mount volumes into the registry node (Format: [SOURCE:]DEST
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d registry](k3d_registry.md)	 - Manage registry/registries

---

### Usage/Commands/K3d Registry Delete

## k3d registry delete

Delete registry/registries.

### Synopsis

Delete registry/registries.

```
k3d registry delete (NAME | --all) [flags]
```

### Options

```
  -a, --all    Delete all existing registries
  -h, --help   help for delete
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d registry](k3d_registry.md)	 - Manage registry/registries

---

### Usage/Commands/K3d Registry List

## k3d registry list

List registries

### Synopsis

List registries.

```
k3d registry list [NAME [NAME...]] [flags]
```

### Options

```
  -h, --help            help for list
      --no-headers      Disable headers
  -o, --output string   Output format. One of: json|yaml
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d registry](k3d_registry.md)	 - Manage registry/registries

---

### Usage/Commands/K3d Version

## k3d version

Show k3d and default k3s version

### Synopsis

Show k3d and default k3s version

```
k3d version [flags]
```

### Options

```
  -h, --help            help for version
  -o, --output string   This will return version information as a different format.  Only json is supported
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d](k3d.md)	 - https://k3d.io/ -> Run k3s in Docker!
* [k3d version list](k3d_version_list.md)	 - List k3d/K3s versions. Component can be one of 'k3d', 'k3s', 'k3d-proxy', 'k3d-tools'.

---

### Usage/Commands/K3d Version List

## k3d version list

List k3d/K3s versions. Component can be one of 'k3d', 'k3s', 'k3d-proxy', 'k3d-tools'.

```
k3d version list COMPONENT [flags]
```

### Options

```
  -e, --exclude string   Exclude Regexp (default excludes pre-releases, arch-specific tags and digests) (default "^sha-|.+(rc|engine|dind|alpha|beta|dev|test|arm|arm64|amd64|s390x).*")
  -f, --format string    [DEPRECATED] Use --output instead (default "raw")
  -h, --help             help for list
  -i, --include string   Include Regexp (default includes everything (default ".*")
  -l, --limit int        Limit number of tags in output (0 = unlimited)
  -o, --output string    Output Format [raw | repo] (default "raw")
  -s, --sort string      Sort Mode (asc | desc | off) (default "desc")
```

### Options inherited from parent commands

```
      --timestamps   Enable Log timestamps
      --trace        Enable super verbose output (trace logging)
      --verbose      Enable verbose output (debug logging)
```

### SEE ALSO

* [k3d version](k3d_version.md)	 - Show k3d and default k3s version

---

### Usage/Commands

# Command Tree

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Usage/Configfile

# Using Config Files

The config file feature is **available as of k3d v4.0.0**

## Introduction

!!! info "Syntax & Semantics"
    The options defined in the config file are not 100% the same as the CLI flags.  
    This concerns naming and style/usage/structure, e.g.

    - `--api-port` is split up into a field named `kubeAPI` that has 3 different "child fields" (`host`, `hostIP` and `hostPort`)
    - k3d options are bundled in a scope named `options.k3d`, where `--no-rollback` is defined as `options.k3d.disableRollback`
    - repeatable flags (like `--port`) are reflected as YAML lists

## Usage

Using a config file is as easy as putting it in a well-known place in your file system and then referencing it via flag:

- All options in config file: `k3d cluster create --config /home/me/my-awesome-config.yaml` (must be `.yaml`/`.yml`)
- With CLI override (name): `k3d cluster create somename --config /home/me/my-awesome-config.yaml`
- With CLI override (extra volume): `k3d cluster create --config /home/me/my-awesome-config.yaml --volume '/some/path:/some:path@server:0'`

## Required Fields

As of the time of writing this documentation, the config file only **requires** you to define two fields:

- `apiVersion` to match the version of the config file that you want to use (at this time it would be `apiVersion: k3d.io/v1alpha5`)
- `kind` to define the kind of config file that you want to use (currently we only have the `Simple` config)

So this would be the minimal config file, which configures absolutely nothing:

```yaml
apiVersion: k3d.io/v1alpha5
kind: Simple
```

## Config Options

The configuration options for k3d are continuously evolving and so is the config file (syntax) itself.  
Currently, the config file is still in an Alpha-State, meaning, that it is subject to change anytime (though we try to keep breaking changes low).

!!! info "Validation via JSON-Schema"
    k3d uses a [JSON-Schema](https://json-schema.org/) to describe the expected format and fields of the configuration file.  
    This schema is also used to [validate](https://github.com/xeipuuv/gojsonschema#validation) a user-given config file.  
    This JSON-Schema can be found in the specific config version sub-directory in the repository (e.g. [here for `v1alpha5`](https://github.com/k3d-io/k3d/blob/main/pkg/config/v1alpha5/schema.json)) and could be used to lookup supported fields or by linters to validate the config file, e.g. in your code editor.  

### All Options: Example

Since the config options and the config file are changing quite a bit, it's hard to keep track of all the supported config file settings, so here's an example showing all of them as of the time of writing:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Tips

- k3d [expands environment variables](https://pkg.go.dev/os#ExpandEnv) (`$VAR` or `${VAR}`) unconditionally in the config file, even before processing it in any way.  

## Config File vs. CLI Flags

k3d uses [`Cobra`](https://github.com/spf13/cobra) and [`Viper`](https://github.com/spf13/viper) for CLI and general config handling respectively.  
This automatically introduces a "config option order of priority" ([precedence order](https://github.com/spf13/viper#why-viper)):

!!! info "Config Precedence Order"
    Source: [spf13/viper#why-viper](https://github.com/spf13/viper#why-viper)  
    >Internal Setting > **CLI Flag** > Environment Variable > **Config File** > (k/v store >) Defaults

This means, that you can define e.g. a "base configuration file" with settings that you share across different clusters and override only the fields that differ between those clusters in your CLI flags/arguments.  
For example, you use the same config file to create three clusters which only have different names and `kubeAPI` (`--api-port`) settings.

## References

- k3d demo repository: <https://github.com/k3d-io/k3d-demo/blob/main/README.md#config-file-support>
- SUSE Blog: <https://www.suse.com/c/introduction-k3d-run-k3s-docker-src/> (Search for `The “Configuration as Code” Way`)

---

### Usage/Exposing Services

# Exposing Services

## 1. via Ingress (recommended)

In this example, we will deploy a simple nginx webserver deployment and make it accessible via ingress.  
Therefore, we have to create the cluster in a way, that the internal port 80 (where the `traefik` ingress controller is listening on) is exposed on the host system.

1. Create a cluster, mapping the ingress port 80 to localhost:8081

    `#!bash k3d cluster create --api-port 6550 -p "8081:80@loadbalancer" --agents 2`

  !!! info "Good to know"
      - `--api-port 6550` is not required for the example to work.  
        It's used to have `k3s`'s API-Server listening on port 6550 with that port mapped to the host system.
      - the port-mapping construct `8081:80@loadbalancer` means:  
        "map port `8081` from the host to port `80` on the container which matches the nodefilter `loadbalancer`"
        - the `loadbalancer` nodefilter matches only the `serverlb` that's deployed in front of a cluster's server nodes
          - all ports exposed on the `serverlb` will be proxied to the same ports on all server nodes in the cluster

2. Get the kubeconfig file (redundant, as `k3d cluster create` already merges it into your default kubeconfig file)

    `#!bash export KUBECONFIG="$(k3d kubeconfig write k3s-default)"`

3. Create a nginx deployment

    `#!bash kubectl create deployment nginx --image=nginx`

4. Create a ClusterIP service for it

    `#!bash kubectl create service clusterip nginx --tcp=80:80`

5. Create an ingress object for it by copying the following manifest to a file and applying with `#!bash kubectl apply -f thatfile.yaml`
  
  **Note**: `k3s` deploys [`traefik`](https://github.com/containous/traefik) as the default ingress controller

    ```YAML
    # apiVersion: networking.k8s.io/v1beta1 # for k3s < v1.19
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: nginx
      annotations:
        ingress.kubernetes.io/ssl-redirect: "false"
    spec:
      rules:
      - http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: nginx
                port:
                  number: 80
    ```

6. Curl it via localhost

    `#!bash curl localhost:8081/`

## 2. via NodePort

1. Create a cluster, mapping the port `30080` from `agent-0` to `localhost:8082`

  `#!bash k3d cluster create mycluster -p "8082:30080@agent:0" --agents 2`

  - **Note 1**: Kubernetes' default NodePort range is [`30000-32767`](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport)
  - **Note 2**: You may as well expose the whole NodePort range from the very beginning, e.g. via `k3d cluster create mycluster --agents 3 -p "30000-32767:30000-32767@server:0"` (See [this video from @portainer](https://www.youtube.com/watch?v=5HaU6338lAk))
    - **Warning**: Docker creates iptable entries and a new proxy process per port-mapping, so this may take a very long time or even freeze your system!

    ... (Steps 2 and 3 like above) ...

1. Create a NodePort service for it by copying the following manifest to a file and applying it with `#!bash kubectl apply -f`

    ```YAML
    apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: nginx
      name: nginx
    spec:
      ports:
      - name: 80-80
        nodePort: 30080
        port: 80
        protocol: TCP
        targetPort: 80
      selector:
        app: nginx
      type: NodePort
    ```

2. Curl it via localhost

    `#!bash curl localhost:8082/`

---

### Usage/Importing Images

# Importing modes

## Auto

Auto-determine whether to use `direct` or `tools-node`.

For remote container runtimes, `tools-node` is faster due to less network overhead, thus it is automatically selected for remote runtimes.

Otherwise direct is used.

## Direct

Directly load the given images to the k3s nodes. No separate container is spawned, no intermediate files are written.

## Tools Node

Start a `k3d-tools` container in the container runtime, copy images to that runtime, then load the images to k3s nodes from there.

---

### Usage/K3s

# K3s Features in k3d

K3s ships with lots of built-in features and services, some of which may only be used in "non-normal" ways in k3d due to the fact that K3s is running in containers.

## General: K3s documentation

- Automatically Deploying Manifests and Helm Charts: <https://rancher.com/docs/k3s/latest/en/helm/#automatically-deploying-manifests-and-helm-charts>
  - Note: `/var/lib/rancher/k3s/server/manifests` is also the path inside the K3s container filesystem, where all built-in component manifests are, so you can override them or provide your own variants by mounting files there, e.g. `--volume /path/to/my/custom/coredns.yaml:/var/lib/rancher/k3s/server/manifests/coredns.yaml` will override the packaged CoreDNS component.
- Customizing packaged Components with `HelmChartConfig`: <https://rancher.com/docs/k3s/latest/en/helm/#customizing-packaged-components-with-helmchartconfig>

## CoreDNS

> Cluster DNS service

### Resources

- Manifest embedded in K3s: <https://github.com/k3s-io/k3s/blob/master/manifests/coredns.yaml>
  - Note: it includes template variables (like `%{CLUSTER_DOMAIN}%`) that will be replaced by K3s before writing the file to the filesystem

### CoreDNS in k3d

Basically, CoreDNS works the same in k3d as it does in other clusters.
One thing to note though is, that the default `forward . /etc/resolv.conf` configured in the `Corefile` doesn't work the same, as the `/etc/resolv.conf` file inside the K3s node containers is not the same as the one on your local machine.

#### Modifications

As of k3d v5.x, k3d injects entries to the `NodeHosts` (basically a hosts file similar to `/etc/hosts` in Linux, which is managed by K3s) to enable Pods in the cluster to resolve the names of other containers in the same docker network (cluster network) and a special entry called `host.k3d.internal` which resolves to the IP of the network gateway (can be used to e.g. resolve DNS queries using your local resolver).
There's a PR in progress to make customizations easier (for k3d and for users): <https://github.com/k3s-io/k3s/pull/4397>

## local-path-provisioner

> Dynamically provisioning persistent local storage with Kubernetes

### Resources

- Source: <https://github.com/rancher/local-path-provisioner>
- Manifest embedded in K3s: <https://github.com/k3s-io/k3s/blob/master/manifests/local-storage.yaml>

### local-path-provisioner in k3d

In k3d, the local paths that the `local-path-provisioner` uses (default is `/var/lib/rancher/k3s/storage`) lies inside the container's filesystem, meaning that by default it's not mapped somewhere e.g. in your user home directory for you to use.
You'd need to map some local directory to that path to easily use the files inside this path: add `--volume $HOME/some/directory:/var/lib/rancher/k3s/storage@all` to your `k3d cluster create` command.

## Traefik

> Kubernetes Ingress Controller

### Resources

- Official Documentation: <https://doc.traefik.io/traefik/providers/kubernetes-ingress/>
- Manifest embedded in K3s: <https://github.com/k3s-io/k3s/blob/master/manifests/traefik.yaml>

### Traefik in k3d

k3d runs K3s in containers, so you'll need to expose the http/https ports on your host to easily access Ingress resources in your cluster. We have a guide over here explaining how to do this, [see](exposing_services.md)

## servicelb (klipper-lb)

> Embedded service load balancer in Klipper
> Allows you to use services with `type: LoadBalancer` in K3s by creating tiny proxies that use `hostPort`s

### Resources

- Source: <https://github.com/k3s-io/klipper-lb>

### servicelb in k3d

`klipper-lb` creates new pods that proxy traffic from `hostPort`s to the service ports of `type: LoadBalancer`.
The `hostPort` in this case is a port in a K3s container, not your local host, so you'd need to add the port-mapping via the `--port` flag when creating the cluster.

---

### Usage/Kubeconfig

# Handling Kubeconfigs

By default, k3d will update your default kubeconfig with your new cluster's details and set the current-context to it (can be disabled).  
To get a kubeconfig set up for you to connect to a k3d cluster without this automatism, you can go different ways.

??? question "What is the default kubeconfig?"
    We determine the path of the used or default kubeconfig in two ways:

    1. Using the `KUBECONFIG` environment variable, if it specifies *exactly one* file
    2. Using the default path (e.g. on Linux it's `#!bash $HOME/.kube/config`)

## Getting the kubeconfig for a newly created cluster

1. Create a new kubeconfig file **after** cluster creation

  - `#!bash k3d kubeconfig write mycluster`
    - *Note:* this will create (or update) the file `$HOME/.k3d/kubeconfig-mycluster.yaml`
    - *Tip:* Use it: `#!bash export KUBECONFIG=$(k3d kubeconfig write mycluster)`
    - *Note 2*: alternatively you can use `#!bash k3d kubeconfig get mycluster > some-file.yaml`

2. Update your default kubeconfig **upon** cluster creation (DEFAULT)

  - `#!bash k3d cluster create mycluster --kubeconfig-update-default`
    - *Note:* this won't switch the current-context (append `--kubeconfig-switch-context` to do so)

3. Update your default kubeconfig **after** cluster creation

  - `#!bash k3d kubeconfig merge mycluster --kubeconfig-merge-default`
    - *Note:* this won't switch the current-context (append `--kubeconfig-switch-context` to do so)

4. Update a different kubeconfig **after** cluster creation

  - `#!bash k3d kubeconfig merge mycluster --output some/other/file.yaml`
    - *Note:* this won't switch the current-context
  - The file will be created if it doesn't exist

!!! info "Switching the current context"
    None of the above options switch the current-context by default.  
    This is intended to be least intrusive, since the current-context has a global effect.  
    You can switch the current-context directly with the `kubeconfig merge` command by adding the `--kubeconfig-switch-context` flag.

## Removing cluster details from the kubeconfig

`#!bash k3d cluster delete mycluster` will always remove the details for `mycluster` from the default kubeconfig.
It will also delete the respective kubeconfig file in `$HOME/.k3d/` if it exists.

## Handling multiple clusters

`k3d kubeconfig merge` let's you specify one or more clusters via arguments _or_ all via `--all`.  
All kubeconfigs will then be merged into a single file if `--kubeconfig-merge-default` or `--output` is specified.  
If none of those two flags was specified, a new file will be created per cluster and the merged path (e.g. `$HOME/.k3d/kubeconfig-cluster1.yaml:$HOME/.k3d/cluster2.yaml`) will be returned.  
Note, that with multiple cluster specified, the `--kubeconfig-switch-context` flag will change the current context to the cluster which was last in the list.

---

### Usage/Multiserver

# Creating multi-server clusters

!!! info "Important note"
    For the best results (and less unexpected issues), choose 1, 3, 5, ... server nodes. (Read more on etcd quorum on [etcd.io](https://etcd.io/docs/v3.3/faq/#why-an-odd-number-of-cluster-members))
    At least 2 cores and 4GiB of RAM are recommended.

## Embedded etcd

Create a cluster with 3 server nodes using k3s' embedded etcd database.
The first server to be created will use the `--cluster-init` flag and k3d will wait for it to be up and running before creating (and connecting) the other server nodes.

```bash
k3d cluster create multiserver --servers 3
```

!!! info "Restarting cluster may fail"
    When you restart the cluster, each node's IP (meaning the underlying container's IP) could change. In this 
    situation, a node might fail to join the existing cluster and consequently fail to start. To address this, 
    you can use the experimental IPAM (IP Address Management) feature to assign each container a static IP. 
    To enable this, create the cluster with the `--subnet auto` or `--subnet 172.45.0.0/16` 
    (or whatever subnet you need) flags. With `--subnet auto`, k3d will create a fake docker network 
    to get an available subnet.
    
    See the relavent issue [#550](https://github.com/k3d-io/k3d/issues/550) for more details.

## Adding server nodes to a running cluster

In theory (and also in practice in most cases), this is as easy as executing the following command:

```bash
k3d node create newserver --cluster multiserver --role server
```

!!! important "There's a trap!"
    If your cluster was initially created with only a single server node, then this will fail.  
    That's because the initial server node was not started with the `--cluster-init` flag and thus is not using the etcd backend.

---

### Usage/Registries

# Using Image Registries

## Registries configuration file

You can add registries by specifying them in a `registries.yaml` and referencing it at creation time:
`#!bash k3d cluster create mycluster --registry-config "/home/YOU/my-registries.yaml"`.

This file is a regular [k3s registries configuration file](https://rancher.com/docs/k3s/latest/en/installation/private-registry/), and looks like this:

```yaml
mirrors:
  "my.company.registry:5000":
    endpoint:
      - http://my.company.registry:5000
```

In this example, an image with a name like `my.company.registry:5000/nginx:latest` would be _pulled_ from the registry running at `http://my.company.registry:5000`.

This file can also be used for providing additional information necessary for accessing some registries, like [authentication](#authenticated-registries) and [certificates](#secure-registries).

### Registries Configuration File embedded in k3d's SimpleConfig

If you're using a `SimpleConfig` file to configure your k3d cluster, you may as well embed the registries.yaml in there directly:

```yaml
apiVersion: k3d.io/v1alpha5
kind: Simple
metadata:
  name: test
servers: 1
agents: 2
registries:
  create: 
    name: myregistry
  config: |
    mirrors:
      "my.company.registry":
        endpoint:
          - http://my.company.registry:5000
```

Here, the config for the k3d-managed registry, created by the `create: {...}` option will be merged with the config specified under `config: |`.

### Authenticated registries

When using authenticated registries, we can add the _username_ and _password_ in a
`configs` section in the `registries.yaml`, like this:

```yaml
mirrors:
  my.company.registry:
    endpoint:
      - http://my.company.registry

configs:
  my.company.registry:
    auth:
      username: aladin
      password: abracadabra
```

### Secure registries

When using secure registries, the [`registries.yaml` file](#registries-file) must include information about the certificates. For example, if you want to use images from the secure registry running at `https://my.company.registry`, you must first download a CA file valid for that server and store it in some well-known directory like `${HOME}/.k3d/my-company-root.pem`.  

Then you have to mount the CA file in some directory in the nodes in the cluster and include that mounted file in a `configs` section in the [`registries.yaml` file](#registries-file).  
For example, if we mount the CA file in `/etc/ssl/certs/my-company-root.pem`, the `registries.yaml` will look like:

```yaml
mirrors:
  my.company.registry:
    endpoint:
      - https://my.company.registry

configs:
  my.company.registry:
    tls:
      # we will mount "my-company-root.pem" in the /etc/ssl/certs/ directory.
      ca_file: "/etc/ssl/certs/my-company-root.pem"
```

Finally, we can create the cluster, mounting the CA file in the path we specified in `ca_file`:

```bash
k3d cluster create \
  --volume "${HOME}/.k3d/my-registries.yaml:/etc/rancher/k3s/registries.yaml" \
  --volume "${HOME}/.k3d/my-company-root.pem:/etc/ssl/certs/my-company-root.pem"
```

## Using a local registry

### Preface: Referencing local registries

In the next sections, you're going to create a local registry (i.e. a container image registry running in a container in your docker host).  
That container will have a name, e.g. `mycluster-registry`.  
If you follow the guide closely (or definitely if you use the k3d-managed option), this name will be known to all the hosts (K3s containers) and workloads in your k3d cluster.  
However, you usually want to push images into that registry from your local machine, which **does not know** that name by default.  
Now you have a few options, including the following three:  

1. Use `localhost`: Since the container will have a port mapped to your local host, you can just directly reference it via e.g. `localhost:12345`, where `12345` is the mapped port
   - If you later pull the image from the registry, only the repository path (e.g. `myrepo/myimage:mytag` in `mycluster-registry:5000/myrepo/myimage:mytag`) matters to find your image in the targeted registry.
2. Get your machine to know the container name: For this you can use the plain old hosts file (`/etc/hosts` on Unix systems and `C:\windows\system32\drivers\etc\hosts` on Windows) by adding an entry like the following to the end of the file:  

  ```text
  127.0.0.1 mycluster-registry
  ```

3. Use some special resolving magic: Tools like `dnsmasq` or `nss-myhostname` (see info box below) and others can setup your local resolver to directly resolve the registry name to `127.0.0.1`.

!!! info "nss-myhostname to resolve `*.localhost`"
    Luckily (for Linux users), [NSS-myhostname](http://man7.org/linux/man-pages/man8/nss-myhostname.8.html) ships with many Linux distributions
    and should resolve `*.localhost` automatically to `127.0.0.1`.  
    Otherwise, it's installable using `sudo apt install libnss-myhostname`.

### Using k3d-managed registries

#### Create a dedicated registry together with your cluster

1. `#!bash k3d cluster create mycluster --registry-create mycluster-registry`: This creates your cluster `mycluster` together with a registry container called `mycluster-registry`

  - k3d sets everything up in the cluster for containerd to be able to pull images from that registry (using the `registries.yaml` file)
  - the port, which the registry is listening on will be mapped to a random port on your host system

2. Check the k3d command output or `#!bash docker ps -f name=mycluster-registry` to find the exposed port
3. [Test your registry](#testing-your-registry)

#### Create a customized k3d-managed registry

1. `#!bash k3d registry create myregistry.localhost --port 12345` creates a new registry called `k3d-myregistry.localhost` (could be used with automatic resolution of `*.localhost`, see next section - also, **note the `k3d-` prefix** that k3d adds to all resources it creates)
2. `#!bash k3d cluster create newcluster --registry-use k3d-myregistry.localhost:12345` (make sure you use the **`k3d-` prefix** here) creates a new cluster set up to use that registry
3. [Test your registry](#testing-your-registry)

### Using your own (not k3d-managed) local registry

_We recommend using a k3d-managed registry, as it plays nicely together with k3d clusters, but here's also a guide to create your own (not k3d-managed) registry, if you need features or customizations, that k3d does not provide:_

??? nonk3dregistry "Using your own (not k3d-managed) local registry"

    You can start your own local registry it with some `docker` commands, like:

    ```bash
    docker volume create local_registry
    docker container run -d --name registry.localhost -v local_registry:/var/lib/registry --restart always -p 12345:5000 registry:2
    ```

    These commands will start your registry container with name and port (on your host) `registry.localhost:12345`. In order to push to this registry, you will need to make it accessible as described in the next section.  
    Once your registry is up and running, we will need to add it to your `registries.yaml` configuration file.  
    Finally, you have to connect the registry network to the k3d cluster network: `#!bash docker network connect k3d-k3s-default registry.localhost`.  
    And then you can [test your local registry](#testing-your-registry).

### Pushing to your local registry address

!!! info "See Preface"
    The information below has been addressed in the [preface for this section](#preface-referencing-local-registries).

## Testing your registry

You should test that you can

- push to your registry from your local development machine.
- use images from that registry in `Deployments` in your k3d cluster.

We will verify these two things for a local registry (located at `k3d-registry.localhost:12345`) running in your development machine.  
Things would be basically the same for checking an external registry, but some additional configuration could be necessary in your local machine when using an authenticated or secure registry (please refer to Docker's documentation for this).

**Assumptions**: In the following test cases, we assume that the registry name `k3d-registry.localhost` resolves to `127.0.0.1` in your local machine (see [section preface for more details](#preface-referencing-local-registries)) and to the registry container IP for the k3d cluster nodes (K3s containers).

**Note**: as per the explanation in the [preface](#preface-referencing-local-registries), you could replace `k3d-registry.localhost:12345` with `localhost:12345` in the `docker tag` and `docker push` commands below (but not in the `kubectl` part!)

### Nginx Deployment

First, we can download some image (like `nginx`) and push it to our local registry with:

```bash
docker pull nginx:latest
docker tag nginx:latest k3d-registry.localhost:12345/nginx:latest
docker push k3d-registry.localhost:12345/nginx:latest
```

Then we can deploy a pod referencing this image to your cluster:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-test-registry
  labels:
    app: nginx-test-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-test-registry
  template:
    metadata:
      labels:
        app: nginx-test-registry
    spec:
      containers:
      - name: nginx-test-registry
        image: k3d-registry.localhost:12345/nginx:latest
        ports:
        - containerPort: 80
EOF
```

Then you should check that the pod is running with `kubectl get pods -l "app=nginx-test-registry"`.

### Alpine Pod

1. Pull the alpine image: `#!bash docker pull alpine:latest`
2. re-tag it to reference your newly created registry: `#!bash docker tag alpine:latest k3d-registry.localhost:12345/testimage:local`
3. push it: `#!bash docker push k3d-registry.localhost:12345/testimage:local`
4. Use kubectl to create a new pod in your cluster using that image to see, if the cluster can pull from the new registry: `#!bash kubectl run --image k3d-registry.localhost:12345/testimage:local testimage --command -- tail -f /dev/null`
   - (creates a container that will not do anything but keep on running)

## Creating a registry proxy / pull-through registry

1. Create a pull-through registry

    ```bash
    k3d registry create docker-io `# Create a registry named k3d-docker-io` \
      -p 5000 `# listening on local host port 5000` \ 
      --proxy-remote-url https://registry-1.docker.io `# let it mirror the Docker Hub registry` \
      -v ~/.local/share/docker-io-registry:/var/lib/registry `# also persist the downloaded images on the device outside the container`
    ```

2. Create `registry.yml`

    ```yaml
    mirrors:
      "docker.io":
        endpoint:
          - http://k3d-docker-io:5000
    ```

3. Create a cluster and using the pull-through cache

    ```bash
    k3d cluster create cluster01 --registry-use k3d-docker-io:5000 --registry-config registry.yml
    ```

4. After cluster01 ready, create another cluster with the same registry or rebuild the cluster, it will use the already locally cached images.

    ```bash
    k3d cluster create cluster02 --registry-use k3d-docker-io:5000 --registry-config registry.yml
    ```

### Creating a registry proxy / pull-through registry via configfile

1. Create a config file, e.g. `/home/me/test-regcache.yaml`

    ```yaml
    apiVersion: k3d.io/v1alpha5
    kind: Simple
    metadata:
      name: test-regcache
    registries:
      create:
        name: docker-io # name of the registry container
        proxy:
          remoteURL: https://registry-1.docker.io # proxy DockerHub
        volumes:
          - /tmp/reg:/var/lib/registry # persist data locally in /tmp/reg
      config: | # tell K3s to use this registry when pulling from DockerHub
        mirrors:
          "docker.io":
            endpoint:
              - http://docker-io:5000
    ```

2. Create cluster from config:

    ```bash
    k3d cluster create -c /home/me/test-regcache.yaml
    ```

---

### Requirements

mkdocs==1.6.1
mkdocs-material==9.5.46
pymdown-extensions==10.21.3
mkdocs-git-revision-date-localized-plugin==1.3.0
mkdocs-awesome-pages-plugin==2.9.3
mdx_truly_sane_lists==1.3 # https://github.com/radude/mdx_truly_sane_lists
mkdocs-include-markdown-plugin==7.1.8 # https://github.com/mondeja/mkdocs-include-markdown-plugin
mike==2.1.3 # versioned docs: https://github.com/jimporter/mike
mkdocs-material-extensions==1.3.1

---

### CHANGELOG

# Changelog

## v5.7.0 - 04.07.2024

### Added

- feat: support config embedded and external files (#1417)
- docs: add examples for config embedded and external files (#1432)
- feat: compatibility with docker userns-remap (#1442) 
- docs: mention ipam when creating multiserver cluster (#1451) 

### Changed

- docs: Update CUDA docs to use k3s suggested method (#1430)
- chore: upgrade go + dependencies + address all golangci-lint issues + fix deprecations (#1459)
- chore: upgrade docker dependency and adjust for deprecations (#1460) 

### Fixed

- fix: close output file (#1436)
- fix: Script exits fatally when resolv.conf is missing Docker nameserver (#1441)
- test: fix translate.go test following userns merge (#1444) 
- fix: respect ~/.kube/config as a symlink (#1455)
- fix: preserve coredns config during cluster restart (#1453)
  - **IMPORTANT** This makes use of the `coredns-custom` configmap, so please consider this in case you're using this configmap yourself!
- fix: make drain ignore DaemonSets & bypass PodDisruptionBudgets (#1414) 

## v5.6.3 - 10.04.2024

### Changed

- Dependency updates and related fixes

## v5.6.2 - 09.04.2024

### Added

-  [DOCS] Add scoop install option (#1390)
- feat: support writing kubeconfig to a stream (#1381)

### Changed

- Not using stdout directly for logging (#1339)
- change: enable fixes by default and consolidate lookup logic (#1349)
- Consistent logging during cluster creation flow (#1398)
- 

### Fixed

- change: fix docs link (#1343)
- 

## v5.6.0 - 21.08.2023

### Added

- add: iptables in DinD image (#1298)
- docs(podman): add usage for rootless mode on macOS (#1314)

### Changed

- **Potentially Breaking**: For people using k3d as a module: switch from netaddr.af to netipx + netip (changed some code around `host.k3d.internal` and the docker runtime)
- **Potentially Breaking**: K3d config directory may change for you: Adhere to XDG's configuration specification (#1320)

### Fixed

- docs: fix go install command (#1337)
- fix docs links in CONTRIBUTING.md
- chore: pkg imported more than once (#1313)

## v5.5.2 - 03.08.2023

### Fixed

- docs: fix list failing to render (#1300)
- bump dependencies to fix `Invalid Host Header` issue with [Docker/Moby#45935](https://github.com/moby/moby/issues/45935)

### Changed

- change: proxy - update nginx-alpine base image (#1309)
- change: add empty /tmp to binary-only image to make it work with config files

### Added

- add: workflow to label issues/prs by sponsors

## v5.5.1 - 19.05.2023

### Fixed

- fix/regression: custom registry config not parsed correctly (#1292)

## v5.5.0 - 17.05.2023

### Added

- Add support for ulimits (#1264)
  - new flag: `k3d cluster create --runtime-ulimit NAME[=SOFT]:[HARD]` (same for `k3d node create`)
- add: K3D_FIX_MOUNTS fix to make / rshared (e.g. to make Cilium work) (#1268)
  - new environment variable: `K3D_FIX_MOUNTS=1`
- add(docs): podman instructions for macOS (#1257)
- Adds json response of version info (#1262)
  - new flag: `k3d version -o json`

### Changed

- change: allow full K3s registry configuration (#1215)
- change: update deps (manual + dependabot)
- change: set e2e test ghaction timeout
- change: improved help text for k3d version ls
- change: deprecate 'k3d version ls --format' in favor of '--output'
- change: golangci-lint fix whitespaces
- change: udpate docs

### Fixed

- Fix panic when k3sURLEnvIndex is -1 (#1252)
- Fix spelling mistake in configfile.md (#1261)
- Correct typo: Inconsistent filename in registry documentation. (#1275)
- fix: k3d version ls (now via crane) (#1286)
- fix: registries.yaml file not marshalled correctly by k8s yaml package

### Deprecated

- change: deprecate 'k3d version ls --format' in favor of '--output'

## v5.4.9 - 16.03.2023 [BROKEN BUILD]

### Changed

- Updated docker dependency to v23.0.1
- change: replace deprecated set-output command with environment file in Github Actions (#1226)

### Fixed

- fix: go install was failing due to outdated docker dependency
- fix: handle colima host for host.k3d.internal lookup (#1228)

## v5.4.8 - 04.03.2023

### Changed

- Go 1.20 and updated dependencies
- change: Use loadbalancer or any *active* server as K3S_URL (#1190)
- change: graceful shutdown drains node before k3d container stops (#1119)
- change: update docs to use quotes around extra args (#1218)
- changed: update podman service documentation around network dns (#1210)
- change: no whitespace in goflags in makefile
- change: fix build with go 1.20 (#1216)

### Fixed

- fix: generate checksum for k3d binaries (#1209)
- fix: improved error handling when update.k3s.io returns a 5XX or invalid response (#1170)
- fix: install script on windows (#1168)
- fix: fix for link in doc (#1219)

## v5.4.7 - 02.02.2023

### Changed

- updated direct and transitive dependencies

### Fixed

- fix: avoid appending existing volumes (#1154)
- fix: indentation for CoreDNS doc (#1166)
- fix: logs error shadowing exec error (#1172)
- docs: Add missing backtick to k3s-arg example command (#1192)
- Support reading in registries-config via env (#1199)

## v5.4.6 - 29.08.2022

### Added

- add: ability to load configuration from stdin (#1126)

### Changed

- update dependencies
- introduce Go workspace mode
- updated docker/k3s version test-matrix
- Go 1.19
- More info on "node stopped returning log lines" error

### Fixed

- tests/e2e: failing e2e tests for parsing config file from stdin
- ci: "random" failing GitHub Actions due to "too many open files"
- docs: fix code highlighting
- docs: beautify bash commands (#1103)

## v5.4.5 - Broken/Unreleased

- This tag was reverted because of constant failures in GitHub Actions and the E2E Tests

## v5.4.4 - 11.07.2022

### Added

- Docs: Clarification of Network Policies in K3s (#1081)

### Changed

- Sponsorship information and updated issue templates
- Switch to `sigs.k8s.io/yaml` everywhere in the project to allow for consistent json/yaml output (#1094)

### Fixed

- Support running k3d with podman in rootless mode using cgroups v2 (#1084)
- `k3d config init` used the legacy config format (#1091)
- Properly handle image prefix "docker.io", etc during image import (#1096)

## v5.4.3 - 07.06.2022

### Added

- Support for pull-through registry (#1075)
  - In command `k3d registry create`
    - e.g. `k3d registry create --proxy-remote-url https://registry-1.docker.io -p 5000 -v /tmp/registry:/var/lib/registry`
  - In config file:

      ```yaml
      # ...
      registries:
        create:
          name: docker-io # name of the registry container
          proxy:
            remoteURL: https://registry-1.docker.io # proxy DockerHub
          volumes:
            - /tmp/reg:/var/lib/registry # persist data locally in /tmp/reg
        config: | # tell K3s to use this registry when pulling from DockerHub
          mirrors:
            "docker.io":
              endpoint:
                - http://docker-io:5000
      ```

  - See registry documentation

## v5.4.2 - 04.06.2022

### Added

- Docs: `hostAliases` in the config file
- New field `registries.create.image` (same as `k3d registry create --image`) in config `v1alpha4` (no version bump) (#1056)

### Changed

- Go 1.18

### Fixed

- docs: fix defaults-networking href (#1064)
- fix deleting of cluster by config file (#1054)
- fix: DOCKER_HOST handling of unix sockets (#1045)
- make: Use go install instead of go get for installing tools (#1038)
- fix: e2e tests safe git directory

## v5.4.1 - 29.03.2022

### Changed

- Updated dependencies (docker, containerd, etc.)

## v5.4.0 - 26.03.2022

**Note**: This is the **first independent release** of k3d

  - k3d moved from rancher/k3d to k3d-io/k3d
  - k3d is fully community-owned
  - k3d does not depend on any company's toolchain or accounts

**Note 2**: You can now fund the work on k3d using GitHub Sponsors ([@iwilltry42](https://github.com/sponsors/iwilltry42)) or IssueHunt ([k3d-io/k3d](https://issuehunt.io/r/k3d-io/k3d))

### Added

- GitHub Actions Release Workflow (#977 & #1024)
  - Replaces DroneCI
  - Now uses `buildx` & `buildx bake` for multiplatform builds (instead of VMs with the according architectures)
  - Now pushes to GHCR instead of DockerHub
- docs: added FAQ entry on using Longhorn in k3d
- docs: added config file tip that k3d expands environment variables
- docs: added section about using k3d with Podman (#987)
- docs: add connect section on homepage (#988)
- added `k3d node create --k3s-arg` flag (#1032)

### Changed

- references to rancher/k3d updated to k3d-io/k3d (#976)
- reference to rancher/k3s updated to k3s-io/k3s (#985)
- explicitly set `bridge` mode for k3d-created networks for Podman compatibility (#986)
- use secure defaults for curl in install script (#999)
- chore: update docs requirements and re-run docgen for commands (#1033)
- change: no default image for node creation in local cluster where image should be copied from existing nodes (#1034)

### Fixed

- fixed volume shortcuts not working because clusterconfig was not being processed
- fixed AUR Release pipeline with more relaxed version selection (#966)
- fixed ZSH completion output (#1014)
- Do not defer goroutine to delete tools node, as this leads to errors
- Hotfix: switch default for image import to original tools-node mode, as the new direct mode fails fairly often
- GetGatewayIP for host.k3d.internal should error out if there's no gateway defined (#1027)
- Store hostAliases in label to persist them across cluster stop/start (#1029)

### Deprecated

- DockerHub Images: k3d's images will now be pushed to GHCR under <https://github.com/orgs/k3d-io/packages?repo_name=k3d>

### Removed

- DroneCI Test & Release Pipeline

### Compatibility

This release was [automatically tested](https://github.com/k3d-io/k3d/actions/runs/2044325827) with the following setups:

#### Docker

- 20.10.5
- 20.10.12

**Expected to Fail** with the following versions:

- <= 20.10.4 (due to runc, see <https://github.com/rancher/k3d/issues/807>)

#### K3s

We test a full cluster lifecycle with different [K3s channels](https://update.k3s.io/v1-release/channels), meaning that the following list refers to the current latest version released under the given channel:

- Channel v1.23
- Channel v1.22

**Expected to Fail** with the following versions:

- <= v1.18 (due to not included, but expected CoreDNS in K3s)

## v5.3.0 - 03.02.2022

**Note:** Now trying to follow a standard scheme defined by <https://keepachangelog.com/en/1.0.0/>

### Added

- new config options to configure extra hosts by @iwilltry42 in <https://github.com/rancher/k3d/pull/938>
- host pid mode support for k3s-server and k3s-agent by @hlts2 in <https://github.com/rancher/k3d/pull/929>
- SimpleConfig v1alpha4 by @iwilltry42 in <https://github.com/rancher/k3d/pull/944>
- add env var LOG_COLORS=[1|true|0|false] to toggle colored log output (enabled by default) by @iwilltry42 in <https://github.com/rancher/k3d/pull/951>
- Compatibility Tests by @iwilltry42 in <https://github.com/rancher/k3d/pull/956>
- Volume Shortcuts and k3d-managed volumes by @iwilltry42 in <https://github.com/rancher/k3d/pull/916>
  - Use some destination shortcuts with the `--volume/-v` flag that k3d automatically expands
    - `k3s-storage` -> `/var/lib/rancher/k3s/storage`
    - `k3s-manifests` -> `/var/lib/rancher/k3s/server/manifests`
    - `k3s-manifests-custom` -> `/var/lib/rancher/k3s/server/manifests/custom` (not K3s default: this is just some sub-directory inside the auto-deploy manifests directory which will also be parsed)
    - `k3s-containerd` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml` (use with caution, K3s generates this file!)
    - `k3s-containerd-tmpl` -> `/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl` (used by K3s to generate the real config above)
    - `k3s-registry-config` -> `/etc/rancher/k3s/registries.yaml` (or just use `--registry-config`)
  - k3d-managed volumes
    - non-existing named volumes starting with a `k3d-` prefix will now be created and managed by `k3d`
- JSON schema versions in-repo to link to from schemastore.org by @iwilltry42 in <https://github.com/rancher/k3d/pull/942>

### Changed

- Config file compatible with Kustomize by @erikgb in <https://github.com/rancher/k3d/pull/945>
- chore: update direct dependencies by @iwilltry42 in <https://github.com/rancher/k3d/pull/935>

### Fixed

- serverlb should be created before using and restarted unless stopped by @wymli in <https://github.com/rancher/k3d/pull/948>
- fix typo in node.go by @eltociear in <https://github.com/rancher/k3d/pull/949>
- mutex on ensureToolsNode to avoid duplicate container name causing error by @iwilltry42 in <https://github.com/rancher/k3d/pull/952>
- detect '--disable=coredns' and conditionally disable injection by @iwilltry42 in <https://github.com/rancher/k3d/pull/955>
- invert logic for LOG_LEVEL parsing by @myitcv in <https://github.com/rancher/k3d/pull/958>

### Deprecated

- SimpleConfig API version `k3d.io/v1alpha3` is now deprecated in favor of `k3d.io/v1alpha4`

### Removed

- unused volume validation functionality in `cmd/util`, does not affect the CLI (#916)

### Compatibility

This release was automatically tested with the following setups:

#### Docker

- 20.10.5
- 20.10.12

**Expected to Fail** with the following versions:

- <= 20.10.4 (due to runc, see <https://github.com/rancher/k3d/issues/807>)

#### K3s

We test a full cluster lifecycle with different [K3s channels](https://update.k3s.io/v1-release/channels), meaning that the following list refers to the current latest version released under the given channel:

- Channel v1.23
- Channel v1.22

**Expected to Fail** with the following versions:

- <= v1.18 (due to not included, but expected CoreDNS in K3s)

## v5.2.2

### Fixes

- mitigate issue when importing images from multiple tars (#881, @sbaier1)
- fix: cluster delete should not fail if no cluster was found by config file (#886, @kuritka)

### Misc

- docs: new page about k3d concepts, incl. nodefilters (#888)
  - <https://k3d.io/v5.2.1/design/concepts/#nodefilters>

## v5.2.1

### Features & Enhancements

- improved Podman compatibility (#868, @serverwentdown)
  - last missing piece: release of <https://github.com/containers/podman/pull/12328>
- improved error handling and logs when waiting for container logs (ca47fac)

### Fixes

- fix: only replace default api host with docker host (#879)
- fix: use available hardcoded K3s version in version.go (0bbb5b9)

## v5.2.0

### Features & Enhancements

- Improve image import performance (#826, @sbaier1)
  - **New flag**: `k3d image import --mode [auto | direct | tools]`
    - `tools` is the old default, which spawns a `k3d-tools` container for importing
    - `auto` is the new default to automatically detect which mode should work best
    - `direct` directly streams the images into the node containers without the `k3d-tools` container
- Enhanced usability of nodefilters & error messages for wrong usage (#871)
- **New command**: `k3d version list [k3s | k3d | k3d-proxy | k3d-tools]` to get image tags that can be used with k3d (#870)
  - e.g. use `k3d version list k3s --format repo` to get the latest image available for K3s and use it via `k3d cluster create --image <image>`
  - Docs: [docs/usage/commands/k3d_version_list.md](./docs/usage/commands/k3d_version_list.md)

### Fixes

- cluster network: reserve IP extra IP for k3d-tools container in k3d-managed IPAM to avoid conflicts
- process the SimpleConfig before validating it to avoid early exit in hostnetwork mode (#860)
- error out if `K3D_FIX_DNS=1` is set and user tries to mount a file to `/etc/resolv.conf` (conflict)
- clusterStart: only run actions which are necessary given the start reason (e.g. `cluster start` vs. `cluster create`)
- fix injection of `host.k3d.internal` based on resolving `host.docker.internal` (#872)
  - also now uses `host.docker.internal` in kubeconfig based on certain conditions (see PR)

### Misc

- tests/e2e: parellelize and cleanup tests -> cut execution speed in half (#848 & #849)
  - also run some make targets in parallel
  - new env var `E2E_PARALLEL=<int>` to configure parallelism
  - test output is now redirected to files inside the runner and only the logs of failed tests will later be output
- Update dependencies, including docker, containerd & k8s
- docs: clarify usage of local registries with k3d
- docs: fix port numbers in registry usage guide

### Notes

- k3d v5.x.x requires at least docker version 20.10.4

## v5.1.0

### Features

- clusterCreate: `--image` option (also in config file) magic words to follow K3s channels (#841)
  - `latest`/`stable` to follow latest/stable channels of K3s
  - `+<channel>` (prefix `+`) where `<channel>` can as well be `latest` or `stable`, but also e.g. `v1.21`
  - k3d will then check the K3s channel server to get the latest image for that channel

### Enhancements

- nodeHooks: add descriptions and log them for more verbosity (#843)
- `node create`: inject `host.k3d.internal` into `/etc/hosts` similar to the `cluster create` command (#843)

### Fix

- `--network host`: do not do any network magic (like `host.k3d.internal` injection, etc.) when `host` network is used (#844)

### Misc

- CI/Makefile: build with `-mod vendor`
- docs: document using some K3s features in k3d, including `servicelb`, `traefik`, `local-storage-provisioner` and `coredns` (#845)

## v5.0.3

### Enhancements & Fixes

- simplified way of getting a Docker API Client that works with Docker Contexts and `DOCKER_*` environment variable configuration (#829, @dragonflylee)
  - fix: didn't honor `DOCKER_TLS` environment variables before

## v5.0.2

### Enhancements

- CoreDNS Configmap is now edited in the auto-deploy manifest on disk instead of relying on `kubectl patch` command (#814)
- refactor: add cmd subcommands in a single function call (#819, @moeryomenko)
- handle ready-log-messages by type and intent & check them in single log streams instead of checking whole chunks every time (#818)

### Fixes

- fix: config file check failing with env var expansion because unexpanded input file was checked

### Misc

- cleanup: ensure that connections/streams are closed once unused (#818)
- cleanup: split type definitions across multiple files to increase readability (#818)
- docs: clarify `node create` help text about cluster reference (#808, @losinggeneration)
- refactor: move from io/ioutil (deprecated) to io and os packages (#827, @Juneezee)

## v5.0.1

### Enhancement

- add `HostFromClusterNetwork` field to `LocalRegistryHosting` configmap as per KEP-1755 (#754)

### Fixes

- fix: nilpointer exception on failed exec process with no returned logreader
- make post-create cluster preparation (DNS stuff mostly) more resilient (#780)
- fix v1alpha2 -> v1alpha3 config migration (and other related issues) (#799)

### Misc

- docs: fix typo (#784)
- docs: fix usage of legacy `--k3s-agent/server-arg` flag

## v5.0.0

This release contains a whole lot of new features, breaking changes as well as smaller fixes and improvements.
The changelog shown here is likely not complete but gives a broad overview over the changes.
For more details, please check the v5 milestone (<https://github.com/rancher/k3d/milestone/27>) or even the commit history.
The docs have been updated, so you should also find the information you need there, with more to come!

The demo repository has also been updated to work with k3d v5: <https://github.com/iwilltry42/k3d-demo>.

**Info**: <https://k3d.io> is now versioned, so you can checkout different versions of the documentation by using the dropdown menu in the page title bar!

**Feedback welcome!**

### Breaking Changes

- new syntax for nodefilters
  - dropped the usage of square brackets `[]` for indexing, as it caused problems with some shells trying to interpret them
  - new syntax: `@identifier[:index][:opt]` (see <https://github.com/rancher/k3d/discussions/652>)
    - example for a port-mapping: `--port 8080:80@server:0:proxy`
      - identifier = `server`, index = `0`, opt = `proxy`
      - `opt` is an extra optional argument used for different purposes depending on the flag
        - currently, only the `--port` flag has `opt`s, namely `proxy` and `direct` (see other breaking change)
- port-mapping now go via the loadbalancer (serverlb) by default
  - the `--port` flag has the `proxy` opt (see new nodefilter syntax above) set by default
  - to leverage the old behavior of direct port-mappings, use the `direct` opt on the port flag
  - the nodefilter `loadbalancer` will now do the same as `servers:*;agents:*` (proxied via the loadbalancer)
- flag `--registries-create` transformed from bool flag to string flag: let's you define the name and port-binding of the newly created registry, e.g. `--registry-create myregistry.localhost:5001`

### Fixes

- cleaned up and properly sorted the sanitization of existing resources used to create new nodes (#638)

### Features & Enhancements

- new command: `k3d node edit` to edit existing nodes (#615)
  - currently only allows `k3d node edit NODE --port-add HOSTPORT:CONTAINERPORT` for the serverlb/loadbalancer to add new ports
  - pkg: new `NodeEdit` function
- new (hidden) command: `k3d debug` with some options for debugging k3d resources (#638)
  - e.g. `k3d debug loadbalancer get-config` to get the current loadbalancer configuration
- loadbalancer / k3d-proxy (#638)
  - updated fork of `confd` to make usage of the file backend including a file watcher for auto-reloads
    - this also checks the config before applying it, so the lb doesn't crash on a faulty config
  - updating the loadbalancer writes the new config file and also checks if everything's going fine afterwards
  - some settings of the loadbalancer can now be configured using `--lb-config-override`, see docs at <https://k3d.io/v5.0.0/design/defaults/#k3d-loadbalancer>
- helper images can now be set explicitly via environment variables: `K3D_IMAGE_LOADBALANCER` & `K3D_IMAGE_TOOLS` (#638)
- concurrently add new nodes to an existing cluster (remove some dumb code) (#640)
  - `--wait` is now the default for `k3d node create`
- normalized flag usage for k3s and runtime (#598, @ejose19)
  - rename `k3d cluster create --label` to `k3d cluster create --runtime-label` (as it's labelling the node on runtime level, e.g. docker)
    - config option moved to `options.runtime.labels`
  - add `k3d cluster create --k3s-node-label` to add Kubernetes node labels via k3s flag (#584, @developer-guy, @ejose, @dentrax)
    - new config option `options.k3s.nodeLabels`
  - the same for `k3d node create`
- improved config file handling (#605)
  - new version `v1alpha3`
    - warning when using outdated version
    - validation dynamically based on provided config apiVersion
    - new default for `k3d config init`
  - new command `k3d config migrate INPUT [OUTPUT]` to migrate config files between versions
    - currently supported migration `v1alpha2` -> `v1alpha3`
  - pkg: new `Config` interface type to support new generic `FromViper` config file parsing
- changed flags `--k3s-server-arg` & `--k3s-agent-arg` into `--k3s-arg` with nodefilter support (#605)
  - new config path `options.k3s.extraArgs`
- config file: environment variables (`$VAR`, `${VAR}` will be expanded unconditionally) (#643)
- docker context support (#601, @developer-guy & #674)
- Feature flag using the environment variable `K3D_FIX_DNS` and setting it to a true value (e.g. `export K3D_FIX_DNS=1`) to forward DNS queries to your local machine, e.g. to use your local company DNS

### Misc

- tests/e2e: timeouts everywhere to avoid killing DroneCI (#638)
- logs: really final output when creating/deleting nodes (so far, we were not outputting a final success message and the process was still doing stuff) (#640)
- tests/e2e: add tests for v1alpha2 to v1alpha3 migration
- docs: use v1alpha3 config version
- docs: update general appearance and cleanup

## v4.4.8

## Enhancements

- Improved DroneCI Pipeline for Multiarch Images and SemVer Tags (#712)
  - **Important**: New images will not have the `v` prefix in the tag anymore!
    - but now real releases will use the "hierarchical" SemVer tags, so you could e.g. subscribe to rancher/k3d-proxy:4 to get v4.x.x images for the proxy container

## Fixes

- clusterCreate: do not override hostIP if hostPort is missing (#693, @lukaszo)
- imageImport: import all listed images, not only the first one (#701, @mszostok)
- clusterCreate: when memory constraints are set, only pull the image used for checking the edac folder, if it's not present on the machine
- fix: update k3d-tools dependencies and use API Version Negotiation, so it still works with older versions of the Docker Engine (#679)

### Misc

- install script: add darwin/arm64 support (#676, @colelawrence)
- docs: fix go install command (#677, @Rots)
- docs: add project overview (<https://k3d.io/internals/project/>) (#680)

## v4.4.7

### Features / Enhancements

- new flag: `k3d image import --keep-tools` to not delete the tools node container after importing the image(s) (#672)
- improve image name handling when importing images (#653, @cimnine)
  - normalize image names internally, e.g. strip prefixes that docker adds, but that break the process
  - see <https://k3d.io/usage/commands/k3d_image_import/> for more info

### Fixes

- Use default gateway, when bridge network doesn't have it (#666, @kuritka)
- Start an existing, but not running tools node to re-use it when importing an image (#672)

### Misc

- deps: switching back to upstream viper including the StringArray fix
- docs: reference to "nolar/setup-k3d-k3s" step for GitHub Actions (#668, @nolar)
- docs: updated and simplified CUDA guide (#662, @vainkop) (#669)

## v4.4.6

### Fixes

- fix an issue where the cluster creation would stall waiting for the `starting worker processes` log message from the loadbalancer/serverlb
  - this was likely caused by a rounding issue when asking docker to get the container logs starting at a specific timestamp
  - we now drop subsecond precision for this to avoid the rounding issue, which was confirmed to work
  - see issues #592 & #621

### Misc

- to debug the issue mentioned above, we introduced a new environment variable `K3D_LOG_NODE_WAIT_LOGS`, which can be set to a list of node roles (e.g. `K3D_LOG_NODE_WAIT_LOGS=loadbalancer,agent`) to output the container logs that k3d inspects

## v4.4.5

### Fixes

- overall: use the getDockerClient helper function everywhere to e.g. support docker via ssh everywhere
- nodeCreate: do not copy meminfo/edac volume mounts from existing nodes, to avoid conflicts with generated mounts
- kubeconfig: fix file handling on windows (#626 + #628, @dragonflylee)

### Misc

- docs: add [FAQ entry](https://k3d.io/faq/faq/#nodes-fail-to-start-or-get-stuck-in-notready-state-with-log-nf_conntrack_max-permission-denied) on nf_conntrack_max: permission denied issue from kube-proxy (#607)
- docs: cleanup, fix formatting, etc.
- license: update to include 2021 in time range
- docs: link to AutoK3s (#614, @JacieChao)
- tests/e2e: update the list of tested k3s versions

## v4.4.4

### Enhancements

- nodes created via `k3d node create` now inherit the registry config from existing nodes (if there is any) (#597)
- the cgroupv2 hotfix (custom entrypoint script) is now enabled by default (#603)
  - disable by setting the environment variable `K3D_FIX_CGROUPV2=false`

### Fixes

- fix using networks without IPAM config (e.g. `host`)

### Misc

- docs: edit links on k3d.io now point to the correct branch (`main`)
- docs: new FAQ entry on spurious PID entries when using shared mounts (#609, @leelavg)

## v4.4.3

### Highlights

- cgroupv2 support: to properly work on cgroupv2 systems, k3s has to move all the processes from the root cgroup to a new /init cgroup and enable subtree_control
  - this is going to be included in the k3s agent code directly (<https://github.com/k3s-io/k3s/pull/3242>)
  - for now we're overriding the container entrypoint with a script that does this (#579, compare <https://github.com/k3s-io/k3s/pull/3237>)
  - thanks a lot for all the input and support @AkihiroSuda
  - **Usage**: set the environment variable `K3D_FIX_CGROUPV2` to a `true` value before/when creating a cluster with k3d
    - e.g. `export K3D_FIX_CGROUPV2=1`

### Fixes

- fix: docker volume not mountable due to validation failure
  - was not able to mount named volume on windows as we're checking for `:` meant for drive-letters and k3d separators

### Misc

- fix create command's flags typo (#568, @Jason-ZW)

## v4.4.2

### Fixes

- k3d-proxy: rename udp upstreams to avoid collisions/duplicates (#564)

### Features

- add *hidden* command `k3d runtime-info` used for debugging (#553)
  - this comes with some additions on package/runtime level
- add *experimental* `--subnet` flag to get some k3d IPAM to ensure that server nodes keep static IPs across restarts (#560)

### Misc

- docs: fix typo (#556, @gcalmettes)
- docs: fix typo (#561, @alechartung)
- ci/drone: pre-release on `-dev.X` tags
- ci/drone: always build no matter the branch name (just not release)
- docs: add automatic command tree generation via cobra (#562)
- makefile: use `go env gopath` as install target for tools (as per #445)
- JSONSchema: add some examples and defaults (now also available via <https://raw.githubusercontent.com/rancher/k3d/main/pkg/config/v1alpha2/schema.json> in your IDE)

## v4.4.1

### Fixes

- use viper fork that contains a fix to make cobra's `StringArray` flags work properly
  - this fixes the issue, that flag values containing commas got split (because we had to use `StringSlice` type flags)
  - this is to be changed back to upstream viper as soon as <https://github.com/spf13/viper/pull/398> (or a similar fix) got merged

## v4.4.0

### Features / Enhancements

- Support for Memory Limits using e.g. `--servers-memory 1g` or `--agents-memory 1.5g` (#494, @konradmalik)
  - enabled by providing fake `meminfo` files

### Fixes

- fix absolute paths in volume mounts on Windows (#510, @markrexwinkel)

### Documentation

- clarify registry names in docs and help text
- add usage section about config file (#534)
- add FAQ entry on certificate error when running behind corporate proxy
- add MacPorts install instructions (#539, @herbygillot)
- Heal Shruggie: Replace amputated arm (#540, @claycooper)

## v4.3.0

### Features / Enhancements

- Use Go 1.16
  - update dependencies, including kubernetes, docker, containerd and more
  - add `darwin/arm64` (Apple Silicon, M1) build target (#530)
  - use the new `//go:embed` feature to directly embed the jsonschema in the binary (#529)
- Add a status column to `k3d registry list` output (#496, @ebr)
- Allow non-prefixed (i.e. without `k3d-` prefix) user input when fetching resources (e.g. `k3d node get mycluster-server-0` would return successfully)

### Fixes

- Allow absolute paths for volumes on Windows (#510, @markrexwinkel)
- fix nil-pointer exception in case of non-existent IPAM network config
- Properly handle combinations of host/hostIP in kubeAPI settings reflected in the kubeconfig (#500, @fabricev)

### Misc

- docs: fix typo in stop command help text (#513, @searsaw)
- ci/ghaction: AUR (pre-)release now on Ubuntu 20.04 and latest archlinux image
- REMOVE incomplete and unused `containerd` runtime from codebase, as it was causing issues to build for windows and hasn't made any progress in quite some time now

## v4.2.0

### Features / Enhancements

- add processing step for cluster config, to configure it e.g. for hostnetwork mode (#477, @konradmalik)
- allow proxying UDP ports via the load balancer (#488, @k0da)

### Fixes

- fix usage of `DOCKER_HOST` env var for Kubeconfig server ref (trim port)
- fix error when trying to attach the same node (e.g. registry) to the same network twice (#486, @kuritka)
- fix Kube-API settings in configg file got overwritten (#490, @dtomasi)

### Misc

- add `k3d.version` label to created resources
- add Pull-Request template
- docs: add hint on minimal requirements for multi-server clusters (#481, @Filius-Patris)

## v4.1.1

### Fixes

- fix: `--k3s-server-arg` and `--k3s-agent-arg` didn't work (Viper StringArray incompatibility) (#482)

## v4.1.0

### Highlights

#### :scroll: Configuration Enhancements

- :snake: use [viper](https://github.com/spf13/viper) for configuration management
  - takes over the job of properly fetching and merging config options from
    - CLI arguments/flags
    - environment variables
    - config file
  - this also fixes some issues with using the config file (like cobra defaults overriding config file values)
- :heavy_check_mark: add JSON-Schema validation for the `Simple` config file schema
- :new: config version `k3d.io/v1alpha2` (some naming changes)
  - `exposeAPI` -> `kubeAPI`
  - `options.k3d.noRollback` -> `options.k3d.disableRollback`
  - `options.k3d.prepDisableHostIPInjection` -> `options.k3d.disableHostIPInjection`

#### :computer: Docker over SSH

- Support Docker over SSH (#324, @ekristen & @inercia)

### Features & Enhancements

- add root flag `--timestamps` to enable timestamped logs
- improved multi-server cluster support (#467)
  - log a warning, if one tries to create a cluster with only 2 nodes (no majority possible, no fault tolerance)
  - revamped cluster start procedure: init-node, sorted servers, agents, helpers
  - different log messages per role and start-place (that we wait for to consider a node to be ready)
  - module: `NodeStartOpts` now accept a `ReadyLogMessage` and `NodeState` now takes a `Started` timestamp string

### Fixes

- do not ignore `--no-hostip` flag and don't inject hostip if `--network=host` (#471, @konradmalik)
- fix: `--no-lb` ignored
- fix: print error cause when serverlb fails to start

### Misc

- tests/e2e: add config override test
- tests/e2e: add multi server start-stop cycle test
- tests/e2e: improved logs with stage and test details.
- builds&tests: use Docker 20.10 and BuildKit everywhere
- :memo: docs: add <https://github.com/AbsaOSS/k3d-action> (GitHub Action) as a related project (#476, @kuritka)

### Tested with

- E2E Tests ran with k3s versions
  - v1.17.17-k3s1 (see Known Issues below)
  - v1.18.15-k3s1 (see Known Issues below)
  - v1.19.7-k3s1
  - v1.20.2-k3s1

### Known Issues

- automatic multi-server cluster restarts tend to fail with k3s versions v1.17.x & v1.18.x and probably earlier versions (using dqlite)
- Using Viper brings us lots of nice features, but also one problem:
  - We had to switch StringArray flags to StringSlice flags, which
    - allow to use multiple flag values comma-separated in a single flag, but also
    - split flag values that contain a comma into separate parts (and we cannot handle issues that arise due to this)
      - so if you rely on commas in your flag values (e.g. for `--env X=a,b,c`), please consider filing an issue or supporting <https://github.com/spf13/viper/issues/246> and <https://github.com/spf13/viper/pull/398>
      - `--env X=a,b,c` would be treated the same as `--env X=a`, `--env b`, `--env c`

## v4.0.0

### Breaking Changes

#### Module

**If you're using k3d as a Go module, please have a look into the code to see all the changes!**

- We're open for chats via Slack or GitHub discussions

- Module is now on `github.com/rancher/k3d/v4` due to lots of breaking changes
- `pkg/cluster` is now `pkg/client`
- `ClusterCreate` and `NodeCreate` don't start the entities (containers) anymore
  - `ClusterRun` and `NodeRun` orchestrate the new Create and Start functionality
- `NodeDelete`/`ClusterDelete` now take an additional `NodeDeleteOpts`/`ClusterDeleteOpts` struct to toggle specific steps
- NodeSpec now features a list of networks (required for registries)
- New config flow: CLIConfig (SimpleConfig) -> ClusterConfig -> Cluster + Opts

#### CLI

- Some flags changed to also use `noun-action` syntax
  - e.g. `--switch-context --update-default-kubeconfig` -> `--kubeconfig-switch-context --kubeconfig-update-default`
  - this eases grouping and visibility

### Changes

#### Features

- **Registry Support**
  - k3d-managed registry like we had it in k3d v1.x
  - Option 1: default settings, paired with cluster creation
    - `k3d cluster create --registry-create` -> New registry for that cluster
    - `k3d cluster create --registry-use` -> Re-use existing registry
  - Option 2: customized, managed stand-alone
    - `k3d registry [create/start/stop/delete]`
    - Check the documentation, help text and tutorials for more details
  - Communicate managed registry using the LocalRegistryHostingV1 spec from [KEP-1755](https://github.com/kubernetes/enhancements/blob/0d69f7cea6fbe73a7d70fab569c6898f5ccb7be0/keps/sig-cluster-lifecycle/generic/1755-communicating-a-local-registry/README.md)
    - interesting especially for tools that reload images, like Tilt or Skaffold

- **Config File Support**
  - Put all your CLI-Arguments/Flags into a more readable config file and re-use it everywhere (keep it in your repo)
    - Note: this is not always a 1:1 matching in naming/syntax/semantics
  - `k3d cluster create --config myconfig.yaml`

    ```yaml
    apiVersion: k3d.io/v1alpha1
    kind: Simple
    name: mycluster
    servers: 3
    agents: 2
    ports:
      - port: 8080:80
        nodeFilters:
          - loadbalancer
    ```

  - Check out our test cases in [pkg/config/test_assets/](./pkg/config/test_assets/) for more config file examples
  - **Note**: The config file format (& feature) might still be a little rough around the edges and it's prone to change quickly until we hit a stable release of the config

- [WIP] Support for Lifecycle Hooks
  - Run any executable at specific stages during the cluster and node lifecycles
    - e.g. we modify the `registries.yaml` in the `preStart` stage of nodes
    - Guides will follow

- Print container creation time (#431, @inercia)
- add output formats for `cluster ls` and `node ls` (#439, @inercia)

#### Fixes

- import image: avoid nil pointer exception in specific cases
- cluster delete: properly handle node and network (#437)
- --port: fix bnil-pointer exception when exposing port on non-existent loadbalancer
- completion/zsh: source completion file

#### Misc

- Now building with Go 1.15
  - same for the k3d-tools code
- updated dependencies (including Docker v20.10)
- tests/e2e: add `E2E_INCLUDE` and rename `E2E_SKIP` to `E2E_EXCLUDE`
- tests/e2e: allow overriding the Helper Image Tag via `E2E_HELPER_IMAGE_TAG`
- docs: spell checking (#434, @jsoref)
- docs: add Chocolatey install option (#443, @erwinkersten)

---

### CONTRIBUTING

# Contributing Guidelines

Hi there! Welcome to the k3d and Rancher Community!
We welcome everyone who likes to use and improve our software.

## Getting Started

Before starting to work with and on k3d, please read and understand our [**Code of Conduct**](./CODE_OF_CONDUCT.md).

Get an Overview of the k3d project in the documentation: [k3d.io/stable/design/project/](https://k3d.io/stable/design/project/)

Before opening an issue or a Pull-Request, please use GitHub's search function to check whether something similar is already in process and hook in there instead.

## Get Recognized

We want to foster a collaborative environment, where every contribution is welcomed and recognized.

If you want to show up in our Contributors section, please make use of the @all-contributors bot integrated with this repository.

Here's a full guide on using the bot: <https://allcontributors.org/docs/en/bot/usage>.
The simplest way to use it is (in a comment on an issue or a pull-request): `@all-contributors please add <username> for <contributions>`, where `<contributions>` is a [list of contributions](https://allcontributors.org/docs/en/emoji-key).

Here's an [example comment on a PR](https://github.com/k3d-io/k3d/pull/368#issuecomment-704320376) to tell the bot to add @zeerorg (who had the initial idea for k3s in docker) to the list of contributors for the ideas and code he added:

> @all-contributors please add @zeerorg for ideas and code

The bot will open a PR to add the user to the list and posts the link as a follow-up comment on the issue/PR.

---

### README

# [](https://k3d.io/)

[](./LICENSE.md)


[](https://pkg.go.dev/github.com/k3d-io/k3d/v5)
[](./go.mod)
[](https://goreportcard.com/report/github.com/k3d-io/k3d)


[](#contributors-)

[](code_of_conduct.md)

## [k3s in docker](https://k3d.io)

k3s is the lightweight Kubernetes distribution by Rancher: [k3s-io/k3s](https://github.com/k3s-io/k3s)

k3d creates containerized k3s clusters. This means, that you can spin up a multi-node k3s cluster on a single machine using docker.

[](https://asciinema.org/a/436420)

**Note:** k3d is a **community-driven project** but it's not an official Rancher (SUSE) product.
**Sponsoring**: To spend any significant amount of time improving k3d, we rely on sponsorships:

  - [**GitHub Sponsors**: ](https://github.com/sponsors/k3d-io)
  - [**LiberaPay**: ](https://liberapay.com/k3d-io)
  - [**IssueHunt**: ](https://issuehunt.io/r/k3d-io/k3d)
## Learning

- Website with documentation: [k3d.io](https://k3d.io/)
- [Rancher Meetup - May 2020 - Simplifying Your Cloud-Native Development Workflow With K3s, K3c and K3d (YouTube)](https://www.youtube.com/watch?v=hMr3prm9gDM)
  - k3d demo repository: [k3d-io/k3d-demo](https://github.com/k3d-io/k3d-demo)

## Requirements

- [docker](https://docs.docker.com/install/)
  - Note: k3d v5.x.x requires at least Docker v20.10.5 (runc >= v1.0.0-rc93) to work properly (see [#807](https://github.com/k3d-io/k3d/issues/807))

## Releases

- May 2020: v1.7.x -> **v3.0.0** (rewrite)
- January 2021: v3.x.x -> **v4.0.0** (breaking changes)
- September 2021: v4.4.8 -> **v5.0.0** (breaking changes)

| Platform | Stage | Version | Release Date | Downloads so far |
|-----------------|--------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------|---|
| [**GitHub Releases**](https://github.com/k3d-io/k3d/releases) | stable | [](https://github.com/k3d-io/k3d/releases/latest) | [](https://github.com/k3d-io/k3d/releases/latest) |  |
| [**GitHub Releases**](https://github.com/k3d-io/k3d/releases) | latest | [](https://github.com/k3d-io/k3d/releases) | [](https://github.com/k3d-io/k3d/releases) |  |
| [**Homebrew**](https://formulae.brew.sh/formula/k3d) | stable | [](https://formulae.brew.sh/formula/k3d) | - | - |
| [**Chocolatey**](https://chocolatey.org/packages/k3d/)| stable | [](https://chocolatey.org/packages/k3d/) | - | - |
| [**Scoop**](https://github.com/ScoopInstaller/Main/blob/master/bucket/k3d.json/)| stable | [](https://github.com/ScoopInstaller/Main/blob/master/bucket/k3d.json/) | - | - |

## Get

You have several options there:

- use the install script to grab the latest release:
  - wget: `wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash`
  - curl: `curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | bash`
- use the install script to grab a specific release (via `TAG` environment variable):
  - wget: `wget -q -O - https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG=v5.0.0 bash`
  - curl: `curl -s https://raw.githubusercontent.com/k3d-io/k3d/main/install.sh | TAG=v5.0.0 bash`

- use [Homebrew](https://brew.sh): `brew install k3d` (Homebrew is available for MacOS and Linux)
  - Formula can be found in [homebrew/homebrew-core](https://github.com/Homebrew/homebrew-core/blob/master/Formula/k3d.rb) and is mirrored to [homebrew/linuxbrew-core](https://github.com/Homebrew/linuxbrew-core/blob/master/Formula/k3d.rb)
- install via [MacPorts](https://www.macports.org): `sudo port selfupdate && sudo port install k3d` (MacPorts is available for MacOS)
- install via [AUR](https://aur.archlinux.org/) package [rancher-k3d-bin](https://aur.archlinux.org/packages/rancher-k3d-bin/): `yay -S rancher-k3d-bin`
- grab a release from the [release tab](https://github.com/k3d-io/k3d/releases) and install it yourself.
- install via go: `go install github.com/k3d-io/k3d/v5@latest` (**Note**: this will give you unreleased/bleeding-edge changes)
- use [Chocolatey](https://chocolatey.org/): `choco install k3d` (Chocolatey package manager is available for Windows)
  - package source can be found in [erwinkersten/chocolatey-packages](https://github.com/erwinkersten/chocolatey-packages/tree/master/automatic/k3d)
- use [Scoop](https://scoop.sh/): `scoop install k3d` (Scoop package manager is available for Windows)
  - package source can be found in [ScoopInstaller/Main](https://github.com/ScoopInstaller/Main/blob/master/bucket/k3d.json)

or...

## Build

1. Clone this repo, e.g. via `git clone git@github.com:k3d-io/k3d.git` or `go get github.com/k3d-io/k3d/v5@main`
2. Inside the repo run
   - 'make install-tools' to make sure required go packages are installed
3. Inside the repo run one of the following commands
   - `make build` to build for your current system
   - `go install` to install it to your `GOPATH` (**Note**: this will give you unreleased/bleeding-edge changes)
   - `make build-cross` to build for all systems

## Usage

Check out what you can do via `k3d help` or check the docs @ [k3d.io](https://k3d.io)

Example Workflow: Create a new cluster and use it with `kubectl`

1. `k3d cluster create CLUSTER_NAME` to create a new single-node cluster (= 1 container running k3s + 1 loadbalancer container)
2. [Optional, included in cluster create] `k3d kubeconfig merge CLUSTER_NAME --kubeconfig-switch-context` to update your default kubeconfig and switch the current-context to the new one
3. execute some commands like `kubectl get pods --all-namespaces`
4. `k3d cluster delete CLUSTER_NAME` to delete the default cluster

## Connect

1. Join the Rancher community on slack via [slack.rancher.io](https://slack.rancher.io/)
2. Go to [rancher-users.slack.com](https://rancher-users.slack.com) and join our channel #k3d
3. Start chatting

## History

This repository is based on [@zeerorg](https://github.com/zeerorg/)'s [zeerorg/k3s-in-docker](https://github.com/zeerorg/k3s-in-docker), reimplemented in Go by [@iwilltry42](https://github.com/iwilltry42/) in [iwilltry42/k3d](https://github.com/iwilltry42/k3d), which got adopted by Rancher in [rancher/k3d](https://github.com/rancher/k3d) and was now moved into its own GitHub organization at [k3d-io/k3d](https://github.com/k3d-io/k3d).

## Related Projects

- [k3x](https://github.com/inercia/k3x): GUI (Linux) to k3d
- [vscode-k3d](https://github.com/inercia/vscode-k3d): vscode plugin for k3d
- [AbsaOSS/k3d-action](https://github.com/AbsaOSS/k3d-action): fully customizable GitHub Action to run lightweight Kubernetes clusters.
- [AutoK3s](https://github.com/cnrancher/autok3s): a lightweight tool to help run K3s everywhere including k3d provider.
- [nolar/setup-k3d-k3s](https://github.com/nolar/setup-k3d-k3s): setup K3d/K3s for GitHub Actions.

## Contributing

k3d is a community-driven project and so we welcome contributions of any form, be it code, logic, documentation, examples, requests, bug reports, ideas or anything else that pushes this project forward.

Please read our [**Contributing Guidelines**](./CONTRIBUTING.md) and the related [**Code of Conduct**](./CODE_OF_CONDUCT.md).

You can find an overview of the k3d project (e.g. explanations and a repository guide) in the documentation: [k3d.io/stable/design/project/](https://k3d.io/stable/design/project/)

[](code_of_conduct.md)

## Contributors ✨

Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)):




<table>
  <tbody>
    <tr>
      <td align="center" valign="top" width="14.28%"><a href="https://twitter.com/iwilltry42"><img src="https://avatars3.githubusercontent.com/u/25345277?v=4?s=100" width="100px;" alt="Thorsten Klein"/><br /><sub><b>Thorsten Klein</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=iwilltry42" title="Code">💻</a> <a href="https://github.com/k3d-io/k3d/commits?author=iwilltry42" title="Documentation">📖</a> <a href="#ideas-iwilltry42" title="Ideas, Planning, & Feedback">🤔</a> <a href="#maintenance-iwilltry42" title="Maintenance">🚧</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://blog.zeerorg.site/"><img src="https://avatars0.githubusercontent.com/u/13547997?v=4?s=100" width="100px;" alt="Rishabh Gupta"/><br /><sub><b>Rishabh Gupta</b></sub></a><br /><a href="#ideas-zeerorg" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/k3d-io/k3d/commits?author=zeerorg" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="http://www.zenika.com"><img src="https://avatars3.githubusercontent.com/u/25585516?v=4?s=100" width="100px;" alt="Louis Tournayre"/><br /><sub><b>Louis Tournayre</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=louiznk" title="Documentation">📖</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/lionelnicolas"><img src="https://avatars3.githubusercontent.com/u/6538664?v=4?s=100" width="100px;" alt="Lionel Nicolas"/><br /><sub><b>Lionel Nicolas</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=lionelnicolas" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/toonsevrin.keys"><img src="https://avatars1.githubusercontent.com/u/5507199?v=4?s=100" width="100px;" alt="Toon Sevrin"/><br /><sub><b>Toon Sevrin</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=toonsevrin" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="http://debian-solutions.de"><img src="https://avatars3.githubusercontent.com/u/1111056?v=4?s=100" width="100px;" alt="Dennis Hoppe"/><br /><sub><b>Dennis Hoppe</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=dhoppe" title="Documentation">📖</a> <a href="#example-dhoppe" title="Examples">💡</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://dellinger.dev"><img src="https://avatars0.githubusercontent.com/u/3109892?v=4?s=100" width="100px;" alt="Jonas Dellinger"/><br /><sub><b>Jonas Dellinger</b></sub></a><br /><a href="#infra-JohnnyCrazy" title="Infrastructure (Hosting, Build-Tools, etc)">🚇</a></td>
    </tr>
    <tr>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/markrexwinkel"><img src="https://avatars2.githubusercontent.com/u/10704814?v=4?s=100" width="100px;" alt="markrexwinkel"/><br /><sub><b>markrexwinkel</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=markrexwinkel" title="Documentation">📖</a></td>
      <td align="center" valign="top" width="14.28%"><a href="http://inerciatech.com/"><img src="https://avatars2.githubusercontent.com/u/1841612?v=4?s=100" width="100px;" alt="Alvaro"/><br /><sub><b>Alvaro</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=inercia" title="Code">💻</a> <a href="#ideas-inercia" title="Ideas, Planning, & Feedback">🤔</a> <a href="#plugin-inercia" title="Plugin/utility libraries">🔌</a></td>
      <td align="center" valign="top" width="14.28%"><a href="http://wsl.dev"><img src="https://avatars2.githubusercontent.com/u/905874?v=4?s=100" width="100px;" alt="Nuno do Carmo"/><br /><sub><b>Nuno do Carmo</b></sub></a><br /><a href="#content-nunix" title="Content">🖋</a> <a href="#tutorial-nunix" title="Tutorials">✅</a> <a href="#question-nunix" title="Answering Questions">💬</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/erwinkersten"><img src="https://avatars0.githubusercontent.com/u/4391121?v=4?s=100" width="100px;" alt="Erwin Kersten"/><br /><sub><b>Erwin Kersten</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=erwinkersten" title="Documentation">📖</a></td>
      <td align="center" valign="top" width="14.28%"><a href="http://www.alexsears.com"><img src="https://avatars.githubusercontent.com/u/3712883?v=4?s=100" width="100px;" alt="Alex Sears"/><br /><sub><b>Alex Sears</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=searsaw" title="Documentation">📖</a></td>
      <td align="center" valign="top" width="14.28%"><a href="http://shanduur.github.io"><img src="https://avatars.githubusercontent.com/u/32583062?v=4?s=100" width="100px;" alt="Mateusz Urbanek"/><br /><sub><b>Mateusz Urbanek</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=Shanduur" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/benjaminjb"><img src="https://avatars.githubusercontent.com/u/4651855?v=4?s=100" width="100px;" alt="Benjamin Blattberg"/><br /><sub><b>Benjamin Blattberg</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=benjaminjb" title="Code">💻</a></td>
    </tr>
    <tr>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/sbaier1"><img src="https://avatars.githubusercontent.com/u/20339314?v=4?s=100" width="100px;" alt="Simon Baier"/><br /><sub><b>Simon Baier</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=sbaier1" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://ambrose.website"><img src="https://avatars.githubusercontent.com/u/1705906?v=4?s=100" width="100px;" alt="Ambrose Chua"/><br /><sub><b>Ambrose Chua</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=serverwentdown" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://www.zenior.no/"><img src="https://avatars.githubusercontent.com/u/1142578?v=4?s=100" width="100px;" alt="Erik Godding Boye"/><br /><sub><b>Erik Godding Boye</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=erikgb" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/ethinx"><img src="https://avatars.githubusercontent.com/u/965612?v=4?s=100" width="100px;" alt="York Wong"/><br /><sub><b>York Wong</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=ethinx" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/gonzalesraul"><img src="https://avatars.githubusercontent.com/u/7568984?v=4?s=100" width="100px;" alt="Raul Gonzales"/><br /><sub><b>Raul Gonzales</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=gonzalesraul" title="Code">💻</a> <a href="https://github.com/k3d-io/k3d/commits?author=gonzalesraul" title="Documentation">📖</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://hoon.dev"><img src="https://avatars.githubusercontent.com/u/6194958?v=4?s=100" width="100px;" alt="Sunghoon Kang"/><br /><sub><b>Sunghoon Kang</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=devholic" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://blogs.kameshs.dev"><img src="https://avatars.githubusercontent.com/u/947745?v=4?s=100" width="100px;" alt="Kamesh Sampath"/><br /><sub><b>Kamesh Sampath</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=kameshsampath" title="Code">💻</a></td>
    </tr>
    <tr>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/arikmaor"><img src="https://avatars.githubusercontent.com/u/1926243?v=4?s=100" width="100px;" alt="Arik Maor"/><br /><sub><b>Arik Maor</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=arikmaor" title="Code">💻</a> <a href="#tutorial-arikmaor" title="Tutorials">✅</a> <a href="#example-arikmaor" title="Examples">💡</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://radiusmethod.com"><img src="https://avatars.githubusercontent.com/u/1850811?v=4?s=100" width="100px;" alt="Danny Gershman"/><br /><sub><b>Danny Gershman</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=dgershman" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/stopanko"><img src="https://avatars.githubusercontent.com/u/3759349?v=4?s=100" width="100px;" alt="stopanko"/><br /><sub><b>stopanko</b></sub></a><br /><a href="#financial-stopanko" title="Financial">💵</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://github.com/dbreyfogle"><img src="https://avatars.githubusercontent.com/u/27653146?v=4?s=100" width="100px;" alt="Danny Breyfogle"/><br /><sub><b>Danny Breyfogle</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=dbreyfogle" title="Documentation">📖</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://tech.aabouzaid.com/"><img src="https://avatars.githubusercontent.com/u/6760103?v=4?s=100" width="100px;" alt="Ahmed AbouZaid"/><br /><sub><b>Ahmed AbouZaid</b></sub></a><br /><a href="#ideas-aabouzaid" title="Ideas, Planning, & Feedback">🤔</a> <a href="https://github.com/k3d-io/k3d/commits?author=aabouzaid" title="Code">💻</a> <a href="https://github.com/k3d-io/k3d/commits?author=aabouzaid" title="Documentation">📖</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://roudier.io"><img src="https://avatars.githubusercontent.com/u/1056127?v=4?s=100" width="100px;" alt="Pierre Roudier"/><br /><sub><b>Pierre Roudier</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=proudier" title="Code">💻</a></td>
      <td align="center" valign="top" width="14.28%"><a href="https://www.fita.dev"><img src="https://avatars.githubusercontent.com/u/7291317?v=4?s=100" width="100px;" alt="Adrien Horgnies"/><br /><sub><b>Adrien Horgnies</b></sub></a><br /><a href="https://github.com/k3d-io/k3d/commits?author=AdrienHorgnies" title="Documentation">📖</a></td>
    </tr>
  </tbody>
</table>






This project follows the [all-contributors](https://github.com/all-contributors/all-contributors) specification. Contributions of any kind welcome!

## Sponsors

**Thanks to our amazing sponsors!** 🙏

<style>
  .grid {
    display: grid;
    grid-template-columns: repeat(auto-fill, minmax(150px, 1fr));
    gap: 8px;
  }

  .grid-item {
    padding: 20px;
    text-align: center;
  }
</style>
<div class="grid">
  <div class="grid-item"><a href="https://github.com/j7nw4r"><img src="https:&#x2F;&#x2F;github.com&#x2F;j7nw4r.png" width="100px" alt="User avatar: Johnathan W" /><br /><sub><b>Johnathan W</b></sub></a></div>
  <div class="grid-item"><a href="https://github.com/stackabletech"><img src="https:&#x2F;&#x2F;github.com&#x2F;stackabletech.png" width="100px" alt="User avatar: Stackable" /><br /><sub><b>Stackable</b></sub></a></div>
</div>

---