# Technical Documentation: kubernetes-sigs/kind
> โน๏ธ **Provenance:** Multi-Tier Ingestion ยท [CodeWiki Reference](https://codewiki.google/github.com/kubernetes-sigs/kind)
### Overview (`README.md`)
# Please see [Our Documentation](https://kind.sigs.k8s.io/docs/user/quick-start/) for more in-depth installation etc.
kind is a tool for running local Kubernetes clusters using Docker container "nodes".
kind was primarily designed for testing Kubernetes itself, but may be used for local development or CI.
If you have [go] and [docker], [podman] or [nerdctl] installed `go install sigs.k8s.io/kind@v0.33.0 && kind create cluster` is all you need!
kind consists of:
- Go [packages][packages] implementing [cluster creation][cluster package], [image build][build package], etc.
- A command line interface ([`kind`][kind cli]) built on these packages.
- Docker [image(s)][images] written to run systemd, Kubernetes, etc.
- [`kubetest`][kubetest] integration also built on these packages (WIP)
kind bootstraps each "node" with [kubeadm][kubeadm]. For more details see [the design documentation][design doc].
**NOTE**: kind is still a work in progress, see the [1.0 roadmap].
## Installation and usage
For a complete [install guide] see [the documentation here][install guide].
You can install kind with `go install sigs.k8s.io/kind@v0.33.0`.
**NOTE**: please use the latest go to do this. KIND is developed with the latest stable go, see [`.go-version`](./.go-version) for the exact version we're using.
This will put `kind` in `$(go env GOPATH)/bin`. If you encounter the error
`kind: command not found` after installation then you may need to either add that directory to your `$PATH` as
shown [here](https://golang.org/doc/code.html#GOPATH) or do a manual installation by cloning the repo and run
`make build` from the repository.
Without installing go, kind can be built reproducibly with docker using `make build`.
Stable binaries are also available on the [releases] page. Stable releases are
generally recommended for CI usage in particular.
To install, download the binary for your platform from "Assets" and place this
into your `$PATH`:
On Linux:
```console
# For AMD64 / x86_64
[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-$(uname)-amd64
# For ARM64
[ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-$(uname)-arm64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
```
On macOS via Homebrew:
```console
brew install kind
```
On macOS via MacPorts:
```console
sudo port selfupdate && sudo port install kind
```
On macOS via Bash:
```console
# For Intel Macs
[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-darwin-amd64
# For M1 / ARM Macs
[ $(uname -m) = arm64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.33.0/kind-darwin-arm64
chmod +x ./kind
mv ./kind /some-dir-in-your-PATH/kind
```
On Windows:
```powershell
curl.exe -Lo kind-windows-amd64.exe https://kind.sigs.k8s.io/dl/v0.33.0/kind-windows-amd64
Move-Item .\kind-windows-amd64.exe c:\some-dir-in-your-PATH\kind.exe
# OR via Chocolatey (https://chocolatey.org/packages/kind)
choco install kind
```
To use kind, you will need to [install docker].
Once you have docker running you can create a cluster with:
```console
kind create cluster
```
To delete your cluster use:
```console
kind delete cluster
```
To create a cluster from Kubernetes source:
- ensure that Kubernetes is cloned in `$(go env GOPATH)/src/k8s.io/kubernetes`
- build a node image and create a cluster with:
```console
kind build node-image
kind create cluster --image kindest/node:latest
```
Multi-node clusters and other advanced features may be configured with a config
file, for more usage see [the docs][user guide] or run `kind [command] --help`
## Community
Please reach out for bugs, feature requests, and other issues!
The maintainers of this project are reachable via:
- [Kubernetes Slack] in the [#kind] channel
- [filing an issue] against this repo
- The Kubernetes [SIG-Testing Mailing List]
Current maintainers are [@aojea], [@BenTheElder], and [@stmcginnis] - feel free to
reach out if you have any questions!
Pull Requests are very welcome!
If you're planning a new feature, please file an issue to discuss first.
Check the [issue tracker] for `help wanted` issues if you're unsure where to
start, or feel free to reach out to discuss. ๐
See also: our own [contributor guide] and the Kubernetes [community page].
## Why kind?
- kind supports multi-node (including HA) clusters
- kind supports building Kubernetes release builds from source
- support for make / bash or docker, in addition to pre-published builds
- kind supports Linux, macOS and Windows
- kind is a [CNCF certified conformant Kubernetes installer](https://landscape.cncf.io/?selected=kind)
### Code of conduct
Participation in the Kubernetes community is governed by the [Kubernetes Code of Conduct].
[go]: https://golang.org/
[go-supported]: https://golang.org/doc/devel/release.html#policy
[docker]: https://www.docker.com/
[podman]: https://podman.io/
[nerdctl]: https://github.com/containerd/nerdctl
[community page]: https://kubernetes.io/community/
[Kubernetes Code of Conduct]: code-of-conduct.md
[Go Report Card Badge]: https://goreportcard.com/badge/sigs.k8s.io/kind
[Go Report Card]: https://goreportcard.com/report/sigs.k8s.io/kind
[conformance tests]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md
[packages]: ./pkg
[cluster package]: ./pkg/cluster
[build package]: ./pkg/build
[kind cli]: ./main.go
[images]: ./images
[kubetest]: https://github.com/kubernetes/test-infra/tree/master/kubetest
[kubeadm]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
[design doc]: https://kind.sigs.k8s.io/docs/design/initial
[user guide]: https://kind.sigs.k8s.io/docs/user/quick-start
[SIG-Testing Mailing List]: https://groups.google.com/forum/#!forum/kubernetes-sig-testing
[issue tracker]: https://github.com/kubernetes-sigs/kind/issues
[filing an issue]: https://github.com/kubernetes-sigs/kind/issues/new
[Kubernetes Slack]: http://slack.k8s.io/
[#kind]: https://kubernetes.slack.com/messages/CEKK1KTN2/
[1.0 roadmap]: https://kind.sigs.k8s.io/docs/contributing/1.0-roadmap
[install docker]: https://docs.docker.com/install/
[@BenTheElder]: https://github.com/BenTheElder
[@munnerz]: https://github.com/munnerz
[@aojea]: https://github.com/aojea
[@amwat]: https://github.com/amwat
[@stmcginnis]: https://github.com/stmcginnis
[contributor guide]: https://kind.sigs.k8s.io/docs/contributing/getting-started
[releases]: https://github.com/kubernetes-sigs/kind/releases
[install guide]: https://kind.sigs.k8s.io/docs/user/quick-start/#installation
[modules]: https://github.com/golang/go/wiki/Modules
---
### Overview (`site/README.md`)
# kind's docs
kind's docs are built with [hugo], and can be browsed live at https://kind.sigs.k8s.io/
To browse them locally, install hugo and run `make serve` from this directory.
[hugo]: https://gohugo.io
---
### Overview (`hack/ci/README.md`)
This directory contains glue used to test the kind repo in CI.
We don't recommend reusing anything from these scripts, and intend to replace
them at some point.
---
### Overview (`hack/tools/README.md`)
This directory contains a stub go module used to track version of development
tools like the Kubernetes code generators.
---
### Overview (`images/base/README.md`)
# images/base
This directory contains sources for building the `kind` base "node" image.
The image can be built with `make quick`.
## Maintenance
This image needs to do a number of unusual things to support running systemd,
nested containers, and Kubernetes. All of what we do and why we do it
is documented inline in the [Dockerfile](./Dockerfile).
If you make any changes to this image, please continue to document exactly
why we do what we do, citing upstream documentation where possible.
See also [`pkg/cluster`](./../../pkg/cluster) for logic that interacts with this image.
## Alternate Sources
Kind frequently picks up new releases of dependent projects including
containerd, runc, cni, and crictl. If you choose to use the provided Dockerfile
but use build arguments to specify a different base image or application version
for dependencies, be aware that you may possibly encounter bugs and undesired
behavior.
## Design
See [base-image](https://kind.sigs.k8s.io/docs/design/base-image/) for more design details.
---
### Overview (`images/kindnetd/README.md`)
# kindnetd
`kindnetd` is a simple networking daemon with the following responsibilities:
- IP masquerade (of traffic leaving the nodes that is headed out of the cluster)
- Ensuring netlink routes to pod CIDRs via the host node IP for each
- Ensuring a simple CNI config based on the standard [ptp] / [host-local] [plugins] and the node's pod CIDR
kindnetd is based on [aojea/kindnet] which is in turn based on [leblancd/kube-v6-test].
We use this to implement KIND's standard CNI / cluster networking configuration.
## Building
cd to this directory on mac / linux with docker installed and run `make quick`.
To push an image run `make push`.
[ptp]: https://www.cni.dev/plugins/current/main/ptp/
[host-local]: https://www.cni.dev/plugins/current/ipam/host-local/
[plugins]: https://github.com/containernetworking/plugins
[aojea/kindnet]: https://github.com/aojea/kindnet
[leblancd/kube-v6-test]: https://github.com/leblancd/kube-v6-test/tree/master
---
### Overview (`images/local-path-helper/README.md`)
# local-path-helper
This image is the image used for the https://github.com/rancher/local-path-provisioner helper pod.
## Building
You can `make quick` in this directory to build a test image.
To push an actual image use `make push`.
---
### Overview (`images/local-path-provisioner/README.md`)
# local-path-provisioner
This image packages https://github.com/rancher/local-path-provisioner to meet
our requirements.
- Not based on alpine (see: https://github.com/kubernetes/kubernetes/issues/109406#issuecomment-1103479928)
- Control over building with current patched go version
## Building
You can `make quick` in this directory to build a test image.
To push an actual image use `make push`.
---
### Overview (`images/node/README.md`)
## images/node
See: [`pkg/build/nodeimage/build.go`][pkg/build/nodeimage/build.go]
and [`pkg/build/nodeimage/buildcontext.go`][pkg/build/nodeimage/buildcontext.go], this
image is built programmatically with docker run / exec / commit for performance
reasons with large artifacts.
Roughly this image is [the base image](./../base), with the addition of:
- installing the Kubernetes packages / binaries
- placing the Kubernetes docker images in `/kind/images/*.tar`
- placing a file in `/kind/version` containing the Kubernetes semver
See [`node-image`][node-image.md] for more design details.
[pkg/build/nodeimage/build.go]: ./../../pkg/build/nodeimage/build.go
[pkg/build/nodeimage/buildcontext.go]: ./../../pkg/build/nodeimage/buildcontext.go
[node-image.md]: https://kind.sigs.k8s.io/docs/design/node-image
---
### Overview (`hack/ci/push-latest-cli/README.md`)
This tooling is used for our automated builds.
These are meant to be consumed *only* by The Kubernetes Project's CI for
testing Kubernetes.
These builds are currently not supported for other purposes.
See github.com/kubernetes/test-infra for the automation that invokes these.
---
### Overview (`hack/make-rules/update/README.md`)
This directory contains tools used to update dependencies, generated files, etc.
---
### Overview (`hack/make-rules/verify/README.md`)
This directory contains tools to verify the state and quality of the repo.
---
### Overview (`hack/third_party/gimme/README.md`)
# gimme
This is an unmodified copy of [gimme], so we don't have to download it
from the internet.
[gimme]: https://github.com/travis-ci/gimme
---
### Overview (`images/base/scripts/third_party/gimme/README.md`)
# gimme
This is an unmodified copy of [gimme], so we don't have to download it
from the internet.
[gimme]: https://github.com/travis-ci/gimme
---
### Overview (`images/kindnetd/scripts/third_party/gimme/README.md`)
# gimme
This is an unmodified copy of [gimme], so we don't have to download it
from the internet.
[gimme]: https://github.com/travis-ci/gimme
---
### Overview (`images/local-path-provisioner/scripts/third_party/gimme/README.md`)
# gimme
This is an unmodified copy of [gimme], so we don't have to download it
from the internet.
[gimme]: https://github.com/travis-ci/gimme
---
### Site Content Docs Contributing Getting Started (`site/content/docs/contributing/getting-started.md`)
---
title: "Getting Started"
menu:
main:
parent: "contributing"
identifier: "getting started"
weight: 1
toc: true
description: |-
Welcome! ๐
This guide covers how to start contributing to kind ๐
---
## 1. Familiarize Yourself With Contributing to Kubernetes Projects
### Read the Kubernetes Community Guidelines
Make sure you read the [Kubernetes community guidelines][community].
In specific, read through the [Kubernetes contributor guidelines][contributor].
### Setup GitHub Account
Kubernetes and kind are developed on [GitHub][github] and will require
an account to contribute.
### Sign CNCF CLA
The Kubernetes project requires the [CNCF][CNCF] [CLA][CNCF-cla] be signed against
your GitHub account for all contributions in all subprojects.
You'll need to get the CLA signed to contribute.
### Check The Kubernetes Contributor Guides
You may come back to this later, but we highly recommend reading these:
- [Kubernetes Contributor Guide](https://git.k8s.io/community/contributors/guide)
- Main contributor documentation, or you can just jump directly to the [contributing section](https://git.k8s.io/community/contributors/guide#contributing)
- [Contributor Cheat Sheet](https://git.k8s.io/community/contributors/guide/contributor-cheatsheet)
- Common resources for existing developers
## 2. Install Tools
### Install Git
Our source code is managed with [`git`][git], to develop locally you
will need to install `git`.
You can check if `git` is already on your system and properly installed with
the following command:
```
git --version
```
### Install Docker
Currently, to create clusters you will need to install [Docker][docker].
If you haven't already, [install Docker][install docker], following the
[official instructions][install docker].
If you have an existing installation, check your version and make sure you have
the latest Docker.
To check if `docker` has been installed:
```
docker --version
```
This documentation is written using Docker version 18.09.2.
### Install Go (optional)
KIND is written in [Go][golang], however our makefiles automatically ensure the
correct version of go when building or testing.
You may still wish to install go on your machine to make it easier to integrate
into your editor etc. You can find the version of go we're currently using to develop kind in the [`.go-version`][go-version] file in the kind repo.
Install or upgrade [Go using the instructions for your operating system][golang].
You can check if Go is in your system with the following command:
```
go version
```
## 3. Read The Docs
The [design principles], [1.0 roadmap], and [initial design]
may be helpful to review before contributing. These docs cover some of the project
philosophy and direction.
## 4. Reaching Out
Issues are tracked on GitHub. Please check [the issue tracker][issues] to see
if there is any existing discussion or work related to your interests.
In particular, if you're just getting started, you may want to look for issues
labeled good first issue or help wanted which are standard labels in the Kubernetes
project.
The help wanted label marks issues we're actively seeking help with while good first issue is additionally applied to a subset of issues we think will be particularly good for newcomers.
If you're interested in working on any of these, leave a comment to let us know!
If you do not see anything, please [file a new issue][file an issue].
> **NOTE**: _Please_ file an enhancement / [feature request issue][fr-issue] to discuss features before filing a PR (ideally even before writing any code), we have a lot to consider with respect to our
> existing users and future support when accepting any new feature.
>
> To streamline the process, please reach out and discuss the concept and design
> / approach ASAP so the maintainers and community can get involved early.
Also -- Please reach out in general for bugs, feature requests, and other issues!
The maintainers of this project are reachable via:
- [Kubernetes Slack] in the [#kind] channel (most active, along with the community)
- The issue tracker by [filing an issue][file an issue]
- The Kubernetes [SIG-Testing][SIG-Testing] [Mailing List][SIG-Testing Mailing List]
Current maintainers are [@aojea] and [@BenTheElder] -- feel free to
reach out directly if you have any questions!
See also: the Kubernetes [community page].
## 5. Next Steps
Okay, so you've gotten your development environment setup, you've read all the
contributor guides, signed the CLA ... now what?
If you're planning to contribute code changes, you'll want to read the [development guide] next.
If you're looking to contribute documentation improvements, first: Thank you! ๐๐ค
You'll specifically want to see the [documentation section] of the development guide.
[git]: https://git-scm.com/
[hugo]: https://gohugo.io
[issues]: https://github.com/kubernetes-sigs/kind/issues
[file an issue]: https://github.com/kubernetes-sigs/kind/issues/new/choose
[design principles]: /docs/design/principles
[1.0 roadmap]: /docs/contributing/1.0-roadmap
[project scope]: /docs/contributing/project-scope
[project structure]: /docs/contributing/project-structure
[initial design]: /docs/design/initial
[github]: https://github.com/
[golang]: https://golang.org/doc/install
[docker]: https://www.docker.com/
[install docker]: https://docs.docker.com/install/#supported-platforms
[community]: https://github.com/kubernetes/community
[contributor]: https://github.com/kubernetes/community/blob/master/contributors/guide/README.md
[Kubernetes Slack]: https://slack.k8s.io/
[#kind]: https://kubernetes.slack.com/messages/CEKK1KTN2/
[@BenTheElder]: https://github.com/BenTheElder
[@aojea]: https://github.com/aojea
[community page]: https://kubernetes.io/community/
[modules]: https://github.com/golang/go/wiki/Modules
[SIG-Testing Mailing List]: https://groups.google.com/forum/#!forum/kubernetes-sig-testing
[CNCF]: https://www.cncf.io/
[CNCF-cla]: https://git.k8s.io/community/CLA.md
[fr-issue]: https://github.com/kubernetes-sigs/kind/issues/new?labels=kind%2Ffeature&template=enhancement.md
[SIG-Testing]: https://github.com/kubernetes/community/blob/master/sig-testing/README.md
[go-version]: https://sigs.k8s.io/kind/.go-version
[development guide]: /docs/contributing/development
[documentation section]: /docs/contributing/development#documentation
---
### Site Content Docs Contributing 1.0 Roadmap (`site/content/docs/contributing/1.0-roadmap.md`)
---
title: "1.0 Roadmap ๐บ"
menu:
main:
parent: "contributing"
identifier: "1.0-roadmap"
weight: 4
toc: true
description: |-
New year, new roadmap ๐
This document outlines some goals, non-goals, and future aspirations for kind
as a project.
---
## Beta Requirements
To reach "beta" [grade][deprecation-policy] kind needs to at minimum:
- [x] Improve documentation (Though this will eternally be "In Progress" !)
- [X] create a documentation site - [#268]
- [x] expand examples of using kind (We can always use more, but we have more of this now)
- [x] cover known issues, debugging, work-arounds, etc.
- [x] Reliably [pass][kind-conformance-dashboard] the Kubernetes [conformance tests]
- [x] [Certify][certified] Conformance
- [x] Support multi-node clusters - [#117]
- [x] Support offline / air-gapped clusters
- [x] pre-loaded / offline CNI - [#200]
- [x] Support mounting host directories - [#62]
- [x] Improve Windows support
- [x] add Windows binaries to releases - [#155]
- [x] improve instructions for KUBECONFIG in particular
- [ ] Support usage as a properly versioned, supported, and documented library. This includes following semver without every release being a major / breaking change to the API (which must be extensible without breakage), ensuring the CLI only uses a suitable public library surface, documentation and examples for the library, versioned types for public APIs (E.G. config format), etc.
- TODO: what exactly do we want here? Should this really be beta blocking?
- [x] should be possible to troubleshoot kind without needing to modify kind ~or use external debugging tools~ (this should be possible now, if not perfect!)
- [x] consistent logging (what is logged, when should it be logged, what levels are used) (this is consistent-ish now, if not perfect)
- [x] errors have appropriate context (this is debatable and never perfect, but improved a lot, especially if you use `-v 1` or greater)
for managing clusters in test harnesses
- [x] move API types / labels from `*.k8s.io` into [`*.x-k8s.io`][api-review-voluntary]
- [x] Support all currently [upstream-supported Kubernetes versions][version-skew-policy]
- [ ] come up with a plan for stable node image <-> KIND compatibility
## GA Requirements
To reach "GA" [grade][deprecation-policy] kind needs to at minimum:
- [x] Support non-AMD64 architectures (namely ARM) - [#166]
- [ ] Automated publishing of Kubernetes release based kind "node" images - [#197]
- [x] Support for runtimes other than docker/default including podman, ignite etc.
- [x] First class support for skewed node (Kubernetes) versions (I believe this is relatively first-class now, things should work fine if you specify different node images)
- ... TBD, more will be added here ...
## Non-Goals
- Supporting every possible Kubernetes configuration
- In order to best support offline / hermetic clusters, we will likely not
offer many options for CNI etc. out of the box. We may revisit this later.
- Being "production workload ready" - kind is meant to be used:
- for testing Kubernetes itself
- for testing against Kubernetes (EG in CI on Travis, Circle, etc.)
- for "local" clusters on developer machines
- NOT to host workloads serving user traffic etc.
- Replacing [Phippy] ๐ฆ -- kind isn't trying to replace all the things
and Phippy is awesome โค๏ธ
Longer Term goals include:
- Enabling a suitable local storage provider for testing applications that need
persistent storage
## Misc
- [x] setup a regular Zoom meeting for the project [#244]
- [x] achieve certified Kubernetes conformance [#245]
Other goals / tasks not listed here can be found both in [the 1.0 project] and
more generally triaged for rough-priority in the [GitHub issues].
[kind-conformance-dashboard]: https://testgrid.k8s.io/conformance-kind
[version-skew-policy]: https://kubernetes.io/docs/setup/release/version-skew-policy/
[deprecation-policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/
[certified]: https://github.com/cncf/k8s-conformance/pull/451#issuecomment-457663982
[conformance tests]: https://github.com/cncf/k8s-conformance
[api-review-voluntary]: https://github.com/kubernetes/community/blob/master/sig-architecture/api-review-process.md#voluntary
[1.0]: https://github.com/kubernetes-sigs/kind/projects/1
[the 1.0 project]: https://github.com/kubernetes-sigs/kind/projects/1
[GitHub issues]: https://github.com/kubernetes-sigs/kind/issues
[#62]: https://github.com/kubernetes-sigs/kind/issues/62
[#117]: https://github.com/kubernetes-sigs/kind/issues/117
[#166]: https://github.com/kubernetes-sigs/kind/issues/166
[#155]: https://github.com/kubernetes-sigs/kind/issues/155
[#197]: https://github.com/kubernetes-sigs/kind/issues/197
[#200]: https://github.com/kubernetes-sigs/kind/issues/200
[#244]: https://github.com/kubernetes-sigs/kind/issues/244
[#245]: https://github.com/kubernetes-sigs/kind/issues/245
[#268]: https://github.com/kubernetes-sigs/kind/pull/268
[Phippy]: https://phippy.io/
---
### Site Content Docs Contributing Development (`site/content/docs/contributing/development.md`)
---
title: "Development"
menu:
main:
parent: "contributing"
identifier: "development"
weight: 3
toc: true
description: |-
๐ง This is a work-in-progress ๐ง
This page is intended to provide contributors with an introduction to developing the kind project.
---
## Overview
KIND provides various utilities for development wrapped in `make`.
Most scripts require more or less only `make` + `bash` on the host, and generally
stick to POSIX utilities. Some scripts use `docker` e.g. for image building or
to use docker images containing special tools.
This guide will introduce you to important make targets and their usage.
## Building
### Building `kind`
Invoke `make build` and `bin/kind` will contain the freshly built `kind` binary
upon a successful build.
Like other targets, this target automatically manages the correct [`.go-version`][go-version] and doesn't require that you install any tooling (just `make` / `bash`). We accomplish this using a copy of [gimme] in `hack/third_party/gimme`.
### Building The Base Image
> **NOTE**: Most development should not require changes to the base image, however if your changes do, here's how to build and test it.
To build the "base image" for development use the `make quick` command in `images/base` directory: `make -C images/base quick`
By default, the base image will be tagged as `kindest/base:$(date +v%Y%m%d)-$(git describe --always --dirty)` format.
If you want to change this, you can set `TAG` environment variable.
`TAG=v0.1.0 make -C images/base quick`
For "production" base images one of the maintainers will run `make -C images/base push` which cross-compiles for all architectures and pushes to the registry.
You generally don't need to cross build during development, and currently the cross
build *must* be pushed instead of loaded locally, due to limitations in `docker buildx` (TODO: link to upstream issue).
To test out your changes take the image you built with `make quick` and use it
as the `--base-image` flag when running `kind build node-image` / building node images. You can then create a cluster with this node image (`kind create cluster --image=kindest/node:latest`)
For "Production" base image updates one of the maintainers will bump `DefaultBaseImage` in `pkg/build/nodeimage/defaults.go` to point to the newly pushed image.
### Building Node Images
See: ["building images"](/docs/user/quick-start/#building-images) in the user [quick start].
## Updating Generated Code
You can regenerate checked-in generated sources with `make generate`.
KIND does not use much generated code, but it does use a little.
Namely kind uses [`deepcopy-gen`] to generate `DeepCopy` methods for API types.
There is also a `make update` target meant to cover all automated code generation
+ formatting (`gofmt`).
## Testing
- Run `make test` to run all tests (unit + integration).
- Run `make unit` to run only unit tests.
- Run `make integration` to run only integration tests.
Like other targets, these targets automatically manage the correct [`.go-version`][go-version] and doesn't require that you install any tooling (just `make` / `bash`).
### E2E Testing
๐ง More here coming soon ... ๐ง
TLDR: `hack/ci/e2e.sh` will run e2e tests against your local Kubernetes checkout.
Depending on your changes, you may want to e2e tests.
In the future we plan to have e2e smoke tests that are cheaper / don't require
building Kubernetes.
## Linting
You can run all of our lints at once with `make verify`.
Lints include:
- checking that generated code is up to date
- you can run just this one with `hack/make-rules/verify/generated.sh`
- [golangci-lint] with a custom config (`hack/tools/.golangci.yml`) to lint Go sources
- you can run just this one with `make lint`
- This linter is essentially an optimized combination of _many_ Go linters
- [shellcheck] to lint our shell scripts (invoked via docker so you don't need to install it)
- you can run just this one with `make shellcheck`
## Documentation
Our docs are built with [hugo] just like [kubernetes.io](https://kubernetes.io).
We provide a Makefile for development that automatically manages a go toolchain
and uses that to build and run hugo. You only need `make`, `bash`,
and `curl` or `wget` installed.
Markdown content is under `site/content/` with a structure mirroring this site.
Static files are under `site/static` (e.g. images are under `site/static/images/`).
For simple content changes you can also just edit the markdown sources and send a
pull request. A build preview will be created by netlify which you can browse by
clicking the "details" link next to the `deploy/netlify` status at the bottom of
your pull request on GitHub.
These are also predictable as `https://deploy-preview-$PR_NUMBER--k8s-kind.netlify.app/`, just replace `$PR_NUMBER` with the number of your Pull Request.
For more involved site / documentation development, you can run `make -C site serve` from the kind repo to run a local instance of the documentation, browsable at [http://localhost:1313](http://localhost:1313).
This site has a custom hugo theme under `site/layouts` & `site/assets`. It's
mostly relatively simple but it has a few extra features:
- The theme layout takes a `description` parameter in page [front matter]
- This renders the blockquote you see just below the page title, on this page with the text `This page is intended to provide contributors with an introduction to developing the kind project.`
- We have a few useful but simple custom shortcodes
### Shortcodes
[Shortcodes](https://gohugo.io/content-management/shortcodes/) are a hugo feature, the kind docs use the following custom shortcodes:
1. `absURL` -- When you need an absolute URL e.g. for `kubectl apply -f $URL`, this
shortcode converts an input URL to absolute. Usage: `{{}}`
1. `securitygoose` -- This is a special shortcode for fun security notices. It wraps
inner markdown content. Usage: `{{}} Notice markdown content here {{}}`
1. `codeFromFile` -- Creates a nice codeblock with a copy button from a file. Usage: `{{}}`
1. `codeFromInline` -- Creates a nice codeblock with a copy button from an inline code snippet. Usage: `{{}} func main() {{}}`
1. `readFile` -- is used to inline file contents. Usage: `{{%/* readFile "static/examples/ingress/contour/patch.json" */%}}`
## CI
The KIND project runs in / on Kubernetes' Custom CI, "[prow]" (prow.k8s.io).
This is both true for CI in the KIND repo, and in the Kubernetes repo where kind
is used to test Kubernetes.
It's important to note that we run larger, slower Kubernetes e2e tests in the kind
repo to mimic what we run in the Kubernetes repo. This is because Kubernetes tests
with kind at HEAD to have the latest fixes for running bleeding edge Kubernetes.
We ensure that the tests continue to work in the kind repo before merging any code changes.
We also have some limited usage experiments with [GitHub Actions], currently
only for testing `podman` or `nerdctl` support in kind.
### Configuration
Our repo's prow configuration is at https://git.k8s.io/test-infra/config/jobs/kubernetes-sigs/kind
GitHub actions are configured in `.github/workflows` in the kind repo.
[gimme]: https://github.com/travis-ci/gimme
[shellcheck]: https://shellcheck.net
[golangci-lint]: https://github.com/golangci/golangci-lint
[go-version]: https://sigs.k8s.io/kind/.go-version
[quick start]: /docs/user/quick-start/
[hugo]: https://gohugo.io
[prow]: https://git.k8s.io/test-infra/
[GitHub Actions]: https://github.com/features/actions
[front matter]: https://gohugo.io/content-management/front-matter/
---
### Site Content Docs Contributing Project Scope (`site/content/docs/contributing/project-scope.md`)
---
title: "Project Scope"
menu:
main:
parent: "contributing"
identifier: "project-scope"
weight: 2
toc: true
description: |-
This document outlines some scoping and major priorities for kind.
See also: the [1.0 roadmap], and the [1.0 tracking milestone].
[1.0 roadmap]: /docs/contributing/1.0-roadmap
[1.0 tracking milestone]: https://github.com/kubernetes-sigs/kind/milestone/2
---
## Priorities (from greatest to least)
### P-1: Bootstrapping the kind Project Itself
---
**Stakeholders**:
- kind maintainers
- kind contributors
**Covered Work**:
- Releases & tooling
- Automated image publishing
- Documentation bootstrapping (IE this site)
- Enough Kubernetes testing to test kind itself (Kubernetes Conformance tests)
- Setting up linters and other tools to verify quality
- Setting up a recurring subproject meeting
### P0: Support Testing Kubernetes
---
**Stakeholders**:
- [SIG Testing][sigs]
- [SIG Cluster-Lifecycle][sigs]
- the [kubeadm] subproject
- Possibly [SIG Release][sigs] (mainly to provide easy access to alpha and beta tags)
**Covered Work**:
- Limited workloads / [e2e testing][e2e]
- Cluster bring-up (IE [kubeadm])
- Kubernetes build (and currently install, but that may be problematic for cross-platform [#166])
- Node skew, client skew (kubectl / e2e versions)
- Image publishing
- Kubernetes CI tooling and [jobs][kubeadm-kind-job]
- Most everything in the [1.0 roadmap]
- ...
### P1: Support Testing Kubernetes Applications
---
**Stakeholders**: Various projects both inside & outside the Kubernetes Org.
- [cert-manager]
- [cluster-api-provider-aws]
- [cluster-api-provider-azure]
- ...
**Covered Work**:
Most of the necessary work should be covered under
[P1: Support Testing Kubernetes Applications](#p1-support-testing-kubernetes-applications),
however there is some additional work.
- Improve "kind as a library"
- better and more controllable logging
- generally more control over output
- example usage & documentation
- better / tighter API contracts
- Most of the rest should be covered by improving "kind the binary" outlined above
- ...
### P2: Provide Cheap Bootstrap Clusters for the Cluster-API
---
**Stakeholders**:
- various [cluster-api][cluster-api] [provider implementation][cluster-api provider implementations]
developers
- various [cluster-api][cluster-api] users
### P3: Extended Testing Not Covered Above
---
**Stakeholders**:
- Indeterminate / many
Possibly supporting various things that we cannot reasonably test today including:
- "node" tests, e.g. reboot
- Upgrades, downgrades
- Anything depending on ingress
- Anything depending on persistent storage / PVs
- Testing the cluster-api proper with some sort of machine provisioning
- Device plugin, e.g. GPU
- ...
Several of these make sense but are not possible with the current tooling and will require a reasonable amount of design and thought to do well. Some of them may not be solve-able in a good way, but are at least technologically feasible to explore.
## Out of Scope
---
Some things we can likely never cover in a reasonable way:
- Cloud provider / [CCM]
- Some of the node testing (which portions exactly is currently unclear)
- Being an alternative to "docker compose" etc.
- Replacing [Phippy][phippy] โค๏ธ ๐ฆ โค๏ธ
- ...
[#166]: https://github.com/kubernetes-sigs/kind/issues/166
[1.0 roadmap]: /docs/contributing/1.0-roadmap
[1.0 tracking milestone]: https://github.com/kubernetes-sigs/kind/milestone/2
[phippy]: https://phippy.io
[kubeadm]: https://github.com/kubernetes/kubeadm
[sigs]: https://github.com/kubernetes/community/blob/master/sig-list.md
[e2e]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests.md
[kubeadm-kind-job]: https://testgrid.k8s.io/sig-cluster-lifecycle-all#kubeadm-kind-master
[cert-manager]: https://github.com/jetstack/cert-manager
[cluster-api-provider-aws]: https://github.com/kubernetes-sigs/cluster-api-provider-aws
[cluster-api-provider-azure]: https://github.com/kubernetes-sigs/cluster-api-provider-azure
[cluster-api]: https://github.com/kubernetes-sigs/cluster-api
[cluster-api provider implementations]: https://github.com/kubernetes-sigs/cluster-api#provider-implementations
[CCM]: https://github.com/kubernetes/kubernetes/tree/master/cmd/cloud-controller-manager
---
### Site Content Docs Design Base Image (`site/content/docs/design/base-image.md`)
---
title: "Base Image"
menu:
main:
parent: "design"
identifier: "base-image"
---
This page used to host a doc about the initial design, this has been found confusing
so we've updated it to clarify the current expectations. While the sources of the project
are fully open, depending on the specifics of the node image internals is not supported.
We only support that base images will create a working node image with `kind build node-image` at the kind release they were shipped with.
The contents and implemlentation of the images are subject to change at any time
to fix bugs, improve reliability, performance, or maintainability.
DO NOT DEPEND ON THE INTERNALS OF THE BASE IMAGES.
KIND provides [conformant][conformance] Kubernetes, anything else is an implementation detail.
We will not accept bugs about "breaking changes" to base images and you depend on the implementation details at your own peril.
[conformance]: https://www.cncf.io/training/certification/software-conformance/
---
### Site Content Docs Design Initial (`site/content/docs/design/initial.md`)
---
title: "Initial design"
menu:
main:
parent: "design"
identifier: "design-initial"
weight: 2
description: |-
This document covers some of the initial design for `kind`.
**NOTE**: Some of this is out of date relative to what is currently implemented.
This mostly exists for historical purposes, the [the original proposal][original proposal]
covers some more details.
Going forward the [design principles] may be more relevant.
[original proposal]: https://docs.google.com/document/d/1VL0shYfKl7goy5Zj4Rghpixbye4M8zs_N2gWoQTSKh0/
[design principles]: /docs/design/principles
---
## Overview
`kind` or **k**ubernetes **in** **d**ocker is a suite of tooling for local
Kubernetes "clusters" where each "node" is a Docker container.
`kind` is targeted at testing Kubernetes.
`kind` is divided into go packages implementing most of the functionality, a
command line for users, and a "node" base image. The intent is that the `kind`
suite of packages should eventually be importable and reusable by other
tools (e.g. [kubetest][kubetest])
while the CLI provides a quick way to use and debug these packages.
For [the original proposal][original proposal] by [Q-Lee][q-lee] see [the kubernetes-sig-testing post][sig-testing-post] (NOTE: this document is shared with [kubernetes-sig-testing][kubernetes-sig-testing]).
In short `kind` targets local clusters for testing purposes. While not all
testing can be performed without "real" clusters in "the cloud" with provider
enabled CCMs, enough can that we want something that:
- runs very cheap clusters that any developer can locally replicate
- integrates with our tooling
- is thoroughly documented and maintainable
- is very stable, and has extensive error handling and sanity checking
- passes all conformance tests
In practice kind looks something like this:
## Clusters
Clusters are managed by logic in [`pkg/cluster`][pkg/cluster], which the
`kind` cli wraps.
Each "cluster" is identified by an internal but well-known [docker object label](https://docs.docker.com/config/labels-custom-metadata/) key, with the cluster
name / ID as the value on each "node" container.
We initially offload this type of state into the containers and Docker.
Similarly the container names are automatically managed by `kind`, though
we will select over labels instead of names because these are less brittle and
are properly namespaced. Doing this also avoids us needing to manage anything
on the host filesystem, but should not degrade usage.
The `KUBECONFIG` will be bind-mounted to a temp directory, with the tooling
capable of detecting this from the containers and providing helpers to use it.
## Images
To run Kubernetes in a container, we first need suitable container image(s).
A single standard [base layer][base-image.md] is used, containing basic
utilities like systemd, certificates, mount, etc.
Installing Kubernetes etc. is performed on top of this image, and may be cached
in pre-built images. We expect to provide images with releases already installed
for use in integrating against Kubernetes.
For more see [node-image.md][node-image.md].
## Cluster Lifecycle
### Cluster Creation
Each "node" runs as a docker container. Each container initially boots to a
pseudo "paused" state, with [the entrypoint][entrypoint] waiting for `SIGUSR1`.
This allows us to manipulate and inspect the container with `docker exec ...`
and other tools prior to starting systemd and all of the components.
This setup includes fixing mounts and pre-loading saved docker images.
Once the nodes are sufficiently prepared, we signal the entrypoint to actually
"boot" the node.
We then wait for the Docker service to be ready on the node before running
`kubeadm` to initialize the node.
Once kubeadm has booted, we export the [KUBECONFIG][kubeconfig], then apply
an [overlay network][overlay network].
At this point users can test Kubernetes by using the exported kubeconfig.
### Cluster Deletion
All "node" containers in the cluster are tagged with docker labels identifying
the cluster by the chosen cluster name (default is "kind"), to delete a cluster
we can simply list and delete containers with this label.
[kubetest]: https://github.com/kubernetes/test-infra/tree/master/kubetest
[original proposal]: https://docs.google.com/document/d/1VL0shYfKl7goy5Zj4Rghpixbye4M8zs_N2gWoQTSKh0/
[q-lee]: https://github.com/q-lee
[sig-testing-post]: https://groups.google.com/d/msg/kubernetes-sig-testing/uVkosorBnVc/8DDC3qvMAwAJ
[kubernetes-sig-testing]: https://groups.google.com/forum/#!forum/kubernetes-sig-testing
[pkg/cluster]: https://github.com/kubernetes-sigs/kind/tree/main/pkg/cluster
[base-image.md]: /docs/design/base-image
[node-image.md]: /docs/design/node-image
[entrypoint]: https://github.com/kubernetes-sigs/kind/blob/main/images/base/files/usr/local/bin/entrypoint
[kubeconfig]: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/
[overlay network]: https://kubernetes.io/docs/setup/independent/create-cluster-kubeadm/#pod-network
[design principles]: /docs/design/principles
---
### Site Content Docs Design Node Image (`site/content/docs/design/node-image.md`)
---
title: "Node Image"
menu:
main:
parent: "design"
identifier: "node-image"
---
This page used to host a doc about the initial design, this has been found confusing
so we've updated it to clarify the current expectations. While the sources of the project
are fully open, depending on the specifics of the node image internals is not supported.
We only support that node images will create a working Kubernetes node at the advertised version with the kind version they
were released with (and best effort with other releases), see the release notes.
The contents and implemlentation of the images are subject to change at any time
to fix bugs, improve reliability, performance, or maintainability.
DO NOT DEPEND ON THE INTERNALS OF THE NODE IMAGES.
KIND provides [conformant][conformance] Kubernetes, anything else is an implementation detail.
We will not accept bugs about "breaking changes" to node images and you depend on the implementation details at your own peril.
[conformance]: https://www.cncf.io/training/certification/software-conformance/
---
### Site Content Docs Design Principles (`site/content/docs/design/principles.md`)
---
title: "Principles"
menu:
main:
parent: "design"
identifier: "design-principles"
weight: 1
toc: true
description: |-
While developing kind the following principles should be considered.
---
## Degrade Gracefully
As much as possible kind should not fail, because it is to be used for testing.
Partially degraded states can still be useful and still be debugged.
As a concrete example: We "pre-load" images that the cluster depends on by
packing them into the "[node image][node image]". If these images fail to
load or are not present in the node image, kind will fall back to letting the
"node"s container runtime attempt to pull them.
Similarly we must at least support all officially supported Kubernetes releases,
which may mean gracefully degrading functionality for older releases.
## Target CRI Functionality
Currently kind only supports [containerd], with experimental support for [podman].
It uses these container runtimes directly to create "node" containers.
With the long term goal of [supporting multiple container runtimes] and
avoid unnecessary coupling, we try to target functionality covered by the
Kubernetes [CRI][CRI] (Container Runtime Interface).
## Leverage Existing Tooling
Where possible we should _not_ reinvent the wheel.
Examples include:
- [kubeadm] is used to handle node configuration, certificates, etc.
- [kustomize] is used to handle merging user provided config patches with our
generated kubeadm configs
- [k8s.io/apimachinery] is used to build our own configuration functionality
- In general we re-use k8s.io [utility libraries][k8s.io/utils] and [generators][k8s.io/code-generator]
Re-implementing some amount of functionality is expected, particularly
between languages and for internal / insufficiently-generic components, but in general
we should collaborate where possible.
## Avoid Breaking Users
Going forward kind will avoid breaking changes to the command line interface
and configuration.
Next we will extend this to a documented set of re-usable
packages (To be determined, but likely IE [pkg/cluster]).
While we are alpha grade currently, we will move to beta and respect
the [Kubernetes Deprecation Policy].
Externally facing features should consider long-term supportability and
extensibility.
## Follow Kubernetes API Conventions
As a general rule of thumb kind prefers to implement configuration using
Kubernetes style configuration files.
While doing this we should respect the Kubernetes [API Conventions].
Additionally we should minimize the number of flags used and avoid structured
values in flags as these cannot be versioned.
## Minimize Assumptions
Avoid making any unnecessary assumptions. Currently we assume:
- [docker] is installed on the host and the current user has permission to talk to dockerd
- Unless using experimental support for [podman] or [nerdctl].
- In the future we may instead only assume that a CRI is available. See [above](#target-cri-functionality).
- "node" images follow our format
- However whenever we make changes we do not assume the updated contents definitely exist
- Metadata in the images is assumed to be correct
- When building Kubernetes, we make the same assumptions & requirements as upstream
## Be Hermetic
As an extension of minimizing assumptions, kind should be as hermetic as possible.
In other words:
- Strive for reproducibility of operations
- Avoid depending on external services, vendor / pre-pull dependencies
## No External State
State is offloaded into the "node" containers in the form of labels, files in
the container filesystem, and processes in the container. The cluster itself
stores all state. No external state stores are used and the only stateful
process is the container runtime. kind does not itself store or manage state.
This simplifies a lot of problems and eases portability, while forcing cluster
interactions to be consistent.
## Consider Automation
While kind strives to present a pleasant UX to users on their local machines,
**automation for end to end testing is the original & primary use case**.
Automated usage should be considered for all features.
[containerd]: https://containerd.io/
[docker]: https://www.docker.com/
[podman]: https://www.podman.io/
[nerdctl]: https://github.com/containerd/nerdctl
[node image]: /docs/design/node-image
[supporting multiple container runtimes]: https://github.com/kubernetes-sigs/kind/issues/154
[CRI]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/container-runtime-interface.md
[kubeadm]: https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm/
[kustomize]: https://github.com/kubernetes-sigs/kustomize
[k8s.io/apimachinery]: https://github.com/kubernetes/apimachinery
[Kubernetes Deprecation Policy]: https://kubernetes.io/docs/reference/using-api/deprecation-policy/
[API Conventions]: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md
[pkg/cluster]: https://github.com/kubernetes-sigs/kind/tree/main/pkg/cluster
[k8s.io/utils]: https://github.com/kubernetes/utils
[k8s.io/code-generator]: https://github.com/kubernetes/code-generator
---
### Site Content Docs User Auditing (`site/content/docs/user/auditing.md`)
---
title: "Auditing"
menu:
main:
parent: "user"
identifier: "user-auditing"
weight: 4
description: |-
This guide covers how to enable Kubernetes API [auditing] on a kind cluster.
[auditing]: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/
---
## Overview
Kubernetes auditing provides a security-relevant, chronological set of records documenting the sequence of actions in a cluster. Auditing requires a file to define the [audit policy] and a backend configuration to store the logged events. Auditing supports two types of backends: log (file) & webhook. The following exercise uses the log backend.
Steps:
- Create the local audit-policy file
- Mount the local audit-policy file into the kind control plane
- Expose the control plane mounts to the API server
- Enable the auditing API flags
- Create a cluster
## Setup
### Create an `audit-policy.yaml` file
The [audit policy] defines the level of granularity outputted by the Kubernetes API server. The example below logs all requests at the "Metadata" level. See the [audit policy] docs for more examples.
{{< codeFromInline lang="bash" >}}
cat < audit-policy.yaml
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
EOF
{{< /codeFromInline >}}
### Create a `kind-config.yaml` file.
To enable audit logging, use kind's [configuration file] to pass additional setup instructions. Kind uses `kubeadm` to provision the cluster and the configuration file has the ability to pass `kubeadmConfigPatches` for further customization.
{{< codeFromInline lang="bash" >}}
cat < kind-config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
apiServer:
# enable auditing flags on the API server
extraArgs:
audit-log-path: /var/log/kubernetes/kube-apiserver-audit.log
audit-policy-file: /etc/kubernetes/policies/audit-policy.yaml
# mount new files / directories on the control plane
extraVolumes:
- name: audit-policies
hostPath: /etc/kubernetes/policies
mountPath: /etc/kubernetes/policies
readOnly: true
pathType: "DirectoryOrCreate"
- name: "audit-logs"
hostPath: "/var/log/kubernetes"
mountPath: "/var/log/kubernetes"
readOnly: false
pathType: DirectoryOrCreate
# mount the local file on the control plane
extraMounts:
- hostPath: ./audit-policy.yaml
containerPath: /etc/kubernetes/policies/audit-policy.yaml
readOnly: true
EOF
{{< /codeFromInline >}}
## Launch a new cluster
{{< codeFromInline lang="bash" >}}
kind create cluster --config kind-config.yaml
{{< /codeFromInline >}}
## View audit logs
Once the cluster is running, view the log files on the control plane in `/var/log/kubernetes/kube-apiserver-audit.log`.
{{< codeFromInline lang="bash" >}}
docker exec kind-control-plane cat /var/log/kubernetes/kube-apiserver-audit.log
{{< /codeFromInline >}}
## Troubleshooting
If logs are not present, let's ensure a few things are in place.
### Is the local audit-policy file mounted in the control-plane?
{{< codeFromInline lang="bash" >}}
docker exec kind-control-plane ls /etc/kubernetes/policies
{{< /codeFromInline >}}
Expected output:
```bash
audit-policy.yaml
```
### Does the API server contain the mounts and arguments?
{{< codeFromInline lang="bash" >}}
docker exec kind-control-plane cat /etc/kubernetes/manifests/kube-apiserver.yaml | grep audit
{{< /codeFromInline >}}
Expected output:
```bash
- --audit-log-path=/var/log/kubernetes/kube-apiserver-audit.log
- --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
name: audit-logs
name: audit-policies
name: audit-logs
name: audit-policies
```
If the control plane requires further debugging use `docker exec -it kind-control-plane bash` to start an interactive terminal session with the container.
[audit policy]: https://kubernetes.io/docs/tasks/debug-application-cluster/audit/#audit-policy
[configuration file]: /docs/user/configuration
---
### Site Content Docs User Configuration (`site/content/docs/user/configuration.md`)
---
title: "Configuration"
menu:
main:
parent: "user"
identifier: "user-configuration"
weight: 3
toc: true
description: |-
This guide covers how to configure KIND cluster creation.
We know this is currently a bit lacking and will expand it over time - PRs welcome!
---
## Getting Started
To configure kind cluster creation, you will need to create a [YAML] config file.
This file follows Kubernetes conventions for versioning etc.
A minimal valid config is:
```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
```
This config merely specifies that we are configuring a KIND cluster (`kind: Cluster`)
and that the version of KIND's config we are using is `v1alpha4` (`apiVersion: kind.x-k8s.io/v1alpha4`).
Any given version of kind may support different versions which will have different
options and behavior. This is why we must always specify the version.
This mechanism is inspired by Kubernetes resources and component config.
To use this config, place the contents in a file `config.yaml` and then run
`kind create cluster --config=config.yaml` from the same directory.
You can also include a full file path like `kind create cluster --config=/foo/bar/config.yaml`.
The structure of the `Cluster` type is defined by a Go struct, which is described
[here](https://pkg.go.dev/sigs.k8s.io/kind/pkg/apis/config/v1alpha4#Cluster).
### A Note On CLI Parameters and Configuration Files
Unless otherwise noted, parameters passed to the CLI take precedence over their
equivalents in a config file. For example, if you invoke:
{{< codeFromInline lang="bash" >}}
kind create cluster --name my-cluster
{{< /codeFromInline >}}
The name `my-cluster` will be used regardless of the presence of that value in
your config file.
## Cluster-Wide Options
The following high level options are available.
NOTE: not all options are documented yet! We will fix this with time, PRs welcome!
### Name Your Cluster
You can give your cluster a name by specifying it in your config:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: app-1-cluster
{{< /codeFromInline >}}
### Feature Gates
Kubernetes [feature gates] can be enabled cluster-wide across all Kubernetes
components with the following config:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
# any feature gate can be enabled here with "Name": true
# or disabled here with "Name": false
# not all feature gates are tested, however
"CSIMigration": true
{{< /codeFromInline >}}
### Runtime Config
Kubernetes API server runtime-config can be toggled using the `runtimeConfig`
key, which maps to the `--runtime-config` [kube-apiserver flag](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/).
This may be used to e.g. disable beta / alpha APIs, or even enable deprecated APIs.
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
runtimeConfig:
"api/alpha": "false"
"apps/v1beta2": "true"
{{< /codeFromInline >}}
### Networking
Multiple details of the cluster's networking can be customized under the
`networking` field.
#### IP Family
KIND has support for IPv4, IPv6 and dual-stack clusters, with the default being `ipv4`. You can change this by setting `ipFamily` under `networking` to `ipv6` or `dual`, see below for more requirements.
##### IPv6 clusters
You can run IPv6 single-stack clusters using `kind`, if the host that runs the docker containers support IPv6.
Most operating systems / distros have IPv6 enabled by default, but you can check on Linux with the following command:
```sh
sudo sysctl net.ipv6.conf.all.disable_ipv6
```
You should see:
```sh
net.ipv6.conf.all.disable_ipv6 = 0
```
If you are using Docker on Windows or Mac, you will need to use an IPv4 port
forward for the API Server from the host because IPv6 port forwards don't work
on these platforms, you can do this with the following config:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
ipFamily: ipv6
apiServerAddress: 127.0.0.1
{{< /codeFromInline >}}
On Linux all you need is:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
ipFamily: ipv6
{{< /codeFromInline >}}
##### Dual Stack clusters
You can run dual stack clusters using `kind` 0.11+, on kubernetes versions 1.20+.
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
ipFamily: dual
{{< /codeFromInline >}}
#### API Server
The API Server listen address and port can be customized with:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
# WARNING: It is _strongly_ recommended that you keep this the default
# (127.0.0.1) for security reasons. However it is possible to change this.
apiServerAddress: "127.0.0.1"
# By default the API server listens on a random open port.
# You may choose a specific port but probably don't need to in most cases.
# Using a random port makes it easier to spin up multiple clusters.
apiServerPort: 6443
{{< /codeFromInline >}}
{{< securitygoose >}}**NOTE**: You should really think thrice before exposing your kind cluster publicly!
kind does not ship with state of the art security or any update strategy (other than
disposing your cluster and creating a new one)! We strongly discourage exposing kind
to anything other than loopback.{{}}
#### Pod Subnet
You can configure the subnet used for pod IPs by setting
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
podSubnet: "10.244.0.0/16"
{{< /codeFromInline >}}
By default, kind uses ```10.244.0.0/16``` pod subnet for IPv4 and ```fd00:10:244::/56``` pod subnet for IPv6.
#### Service Subnet
You can configure the Kubernetes service subnet used for service IPs by setting
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
serviceSubnet: "10.96.0.0/12"
{{< /codeFromInline >}}
By default, kind uses ```10.96.0.0/16``` service subnet for IPv4 and ```fd00:10:96::/112``` service subnet for IPv6.
#### Disable Default CNI
KIND ships with a simple networking implementation ("kindnetd") based around
standard CNI plugins (`ptp`, `host-local`, ...) and simple netlink routes.
This CNI also handles IP masquerade.
You may disable the default to install a different CNI. This is a power user
feature with limited support, but many common CNI manifests are known to work,
e.g. Calico.
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
# the default CNI will not be installed
disableDefaultCNI: true
{{< /codeFromInline >}}
#### kube-proxy mode
You can configure the kube-proxy mode that will be used, between iptables, nftables (Kubernetes v1.31+), and ipvs.
By default iptables is used
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
kubeProxyMode: "nftables"
{{< /codeFromInline >}}
To disable kube-proxy, set the mode to `"none"`.
### Nodes
The `kind: Cluster` object has a `nodes` field containing a list of `node`
objects. If unset this defaults to:
```yaml
nodes:
# one node hosting a control plane
- role: control-plane
```
You can create a multi node cluster with the following config:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
# One control plane node and three "workers".
#
# While these will not add more real compute capacity and
# have limited isolation, this can be useful for testing
# rolling updates etc.
#
# The API-server and other control plane components will be
# on the control-plane node.
#
# You probably don't need this unless you are testing Kubernetes itself.
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
{{< /codeFromInline >}}
Multiple `control-plane` nodes may be specified in order to test a "high availability"
control plane.
## Per-Node Options
The following options are available for setting on each entry in `nodes`.
NOTE: not all options are documented yet! We will fix this with time, PRs welcome!
### Kubernetes Version
You can set a specific Kubernetes version by setting the `node`'s container image. You can find available image tags on the [releases page](https://github.com/kubernetes-sigs/kind/releases). Please include the `@sha256:` [image digest](https://docs.docker.com/engine/reference/commandline/pull/#pull-an-image-by-digest-immutable-identifier) from the image in the release notes, as seen in this example:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.16.4@sha256:b91a2c2317a000f3a783489dfb755064177dbc3a0b2f4147d50f04825d016f55
- role: worker
image: kindest/node:v1.16.4@sha256:b91a2c2317a000f3a783489dfb755064177dbc3a0b2f4147d50f04825d016f55
{{< /codeFromInline >}}
[Reference](https://kind.sigs.k8s.io/docs/user/quick-start/#creating-a-cluster)
**Note**: Kubernetes versions are expressed as x.y.z, where x is the major version, y is the minor version, and z is the patch version, following [Semantic Versioning](https://semver.org/) terminology. For more information, see [Kubernetes Release Versioning.](https://github.com/kubernetes/sig-release/blob/master/release-engineering/versioning.md#kubernetes-release-versioning)
### Extra Mounts
Extra mounts can be used to pass through storage on the host to a kind node
for persisting data, mounting through code etc.
{{< codeFromFile file="static/examples/config-with-mounts.yaml" lang="yaml" >}}
**NOTE**: If you are using Docker for Mac or Windows check that the hostPath is
included in the Preferences -> Resources -> File Sharing.
For more information see the [Docker file sharing guide.](https://docs.docker.com/docker-for-mac/#file-sharing)
### Extra Port Mappings
Extra port mappings can be used to port forward to the kind nodes. This is a
cross-platform option to get traffic into your kind cluster.
If you are running Docker without the Docker Desktop Application on Linux, you can simply send traffic to the node IPs from the host without extra port mappings.
With the installation of the Docker Desktop Application, whether it is on macOs, Windows or Linux, you'll want to use these.
You may also want to see the [Ingress Guide].
> **NOTE**: If you're running Kind on a remote host and need to send traffic to Kind node
> IPs from a different host than where kind is running, you need to configure port-mapping.
{{< codeFromFile file="static/examples/config-with-port-mapping.yaml" lang="yaml" >}}
An example http pod mapping host ports to a container port.
{{< codeFromInline lang="yaml">}}
kind: Pod
apiVersion: v1
metadata:
name: foo
spec:
containers:
- name: foo
image: hashicorp/http-echo:0.2.3
args:
- "-text=foo"
ports:
- containerPort: 5678
hostPort: 80
{{< /codeFromInline >}}
#### NodePort with Port Mappings
To use port mappings with `NodePort`, the kind node `containerPort` and the service `nodePort` needs to be equal.
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30950
hostPort: 80
{{< /codeFromInline >}}
And then set `nodePort` to be 30950.
{{< codeFromInline lang="yaml">}}
kind: Pod
apiVersion: v1
metadata:
name: foo
labels:
app: foo
spec:
containers:
- name: foo
image: hashicorp/http-echo:0.2.3
args:
- "-text=foo"
ports:
- containerPort: 5678
---
apiVersion: v1
kind: Service
metadata:
name: foo
spec:
type: NodePort
ports:
- name: http
nodePort: 30950
port: 5678
selector:
app: foo
{{< /codeFromInline >}}
[Ingress Guide]: /docs/user/ingress
### Extra Labels
Extra labels might be useful for working with
[nodeSelectors](https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/).
An example label for specifying a `tier` label:
{{< codeFromInline lang="yaml">}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
extraPortMappings:
- containerPort: 30950
hostPort: 80
labels:
tier: frontend
- role: worker
labels:
tier: backend
{{< /codeFromInline >}}
### Kubeadm Config Patches
KIND uses [`kubeadm`](/docs/design/principles/#leverage-existing-tooling)
to configure cluster nodes.
Formally KIND runs `kubeadm init` on the first control-plane node, we can customize the flags by using the kubeadm
[InitConfiguration](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-file)
([spec](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3#InitConfiguration))
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: InitConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "my-label=true"
{{< /codeFromInline >}}
If you want to do more customization, there are four configuration types available during `kubeadm init`: `InitConfiguration`, `ClusterConfiguration`, `KubeProxyConfiguration`, `KubeletConfiguration`. For example, we could override the apiserver flags by using the kubeadm [ClusterConfiguration](https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/control-plane-flags/) ([spec](https://pkg.go.dev/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3#ClusterConfiguration)):
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
kubeadmConfigPatches:
- |
kind: ClusterConfiguration
apiServer:
extraArgs:
enable-admission-plugins: NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
{{< /codeFromInline >}}
> **NOTE**: When using `KubeletConfiguration`, kubeadm only reads the kubelet configuration from the **first** node, which will apply to all nodes.
> This is a current [limitation](https://github.com/kubernetes-sigs/kind/issues/3849).
As a result, if you want to change the kubelet's configuration for any additional node, such as applying a taint, you must use `JoinConfiguration`:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: test
nodes:
- role: control-plane
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
register-with-taints: "my-taint=presence:NoSchedule"
{{< /codeFromInline >}}
On every additional node configured in the KIND cluster,
worker or control-plane (in HA mode),
KIND runs `kubeadm join` which can be configured using the
[JoinConfiguration](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/#config-file)
([spec](https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta3#JoinConfiguration))
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "my-label2=true"
- role: control-plane
kubeadmConfigPatches:
- |
kind: JoinConfiguration
nodeRegistration:
kubeletExtraArgs:
node-labels: "my-label3=true"
{{< /codeFromInline >}}
If you need more control over patching, strategic merge and JSON6092 patches can
be used as well. These are specified using files in a directory, for example
`./patches/kube-controller-manager.yaml` could be the following.
{{< codeFromInline lang="yaml" >}}
apiVersion: v1
kind: Pod
metadata:
name: kube-controller-manager
namespace: kube-system
spec:
containers:
- name: kube-controller-manager
env:
- name: KUBE_CACHE_MUTATION_DETECTOR
value: "true"
{{< /codeFromInline >}}
Then in your kind YAML configuration use the following.
{{< codeFromInline lang="yaml" >}}
nodes:
- role: control-plane
extraMounts:
- hostPath: ./patches
containerPath: /patches
kubeadmConfigPatches:
- |
kind: InitConfiguration
patches:
directory: /patches
{{< /codeFromInline >}}
Note the `extraMounts` stanza. The node is a container created by
`kind`. `kubeadm` is run inside this node container, and the local directory
that contains the patches has to be accessible to `kubeadm`. `extraMounts`
plumbs a local directory through to this node container.
This example was for changing the manager in the control plane. To use a patch
for a worker node, use a `JoinConfiguration` patch and an `extraMounts` stanza
for the `worker` role.
[YAML]: https://yaml.org/
[feature gates]: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
---
### Site Content Docs User Ingress (`site/content/docs/user/ingress.md`)
---
title: "Ingress"
menu:
main:
parent: "user"
identifier: "user-ingress"
weight: 3
description: |-
This guide covers setting up [ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) on a kind cluster.
---
## Compatibility๏ผ
This guide applies to [cloud-provider-kind](https://github.com/kubernetes-sigs/cloud-provider-kind) v0.9.0+. For older versions, refer to historical docs.
## Setting Up Ingress
Ingress exposes HTTP and HTTPS routes from outside the cluster to services within the cluster.
Since cloud-provider-kind v0.9.0, it natively supports Ingress. No third-party ingress controllers are required by default.
For third-party ingress solutions (e.g., Ingress NGINX, Contour), please follow their official documentation.
> **NOTE**: Gateway API is also natively supported (along with Ingress). See the official [Ingress migration guide](https://gateway-api.sigs.k8s.io/guides/getting-started/migrating-from-ingress/) for details.
## Create Cluster
> **WARNING**: If you are using a [rootless container runtime], ensure your host is
> properly configured before creating the KIND cluster. Most Ingress and Gateway controllers will
> not work if these steps are skipped.
Create a kind cluster and run [Cloud Provider KIND] that automatically enables LoadBalancer support for Ingress. Create a cluster as follows.
{{< codeFromInline lang="bash" >}}
kind create cluster
{{< /codeFromInline >}}
## Using Ingress
The following example creates simple http-echo services and an Ingress object to route to these services.
```yaml
{{% readFile "static/examples/ingress/usage.yaml" %}}
```
Apply the configuration:
{{< codeFromInline lang="bash" >}}
kubectl apply -f {{< absURL "examples/ingress/usage.yaml" >}}
{{< /codeFromInline >}}
### Verify Ingress Works
Check the External IP assigned to the Ingress by the built-in LoadBalancer.
{{< codeFromInline lang="bash" >}}
kubectl get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
example-ingress example.com 172.18.0.5 80 10m
{{< /codeFromInline >}}
{{< codeFromInline lang="bash" >}}
# get the Ingress IP
INGRESS_IP=$(kubectl get ingress example-ingress -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
# should output "foo-app"
curl ${INGRESS_IP}/foo
# should output "bar-app"
curl ${INGRESS_IP}/bar
{{< /codeFromInline >}}
[LoadBalancer]: /docs/user/loadbalancer/
[Cloud Provider KIND]: /docs/user/loadbalancer/
[rootless container runtime]: /docs/user/rootless/
---
### Site Content Docs User Kind Example Config.Yaml (`site/content/docs/user/kind-example-config.yaml`)
# this config file contains all config fields with comments
# NOTE: this is not a particularly useful config file
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
# patch the generated kubeadm config with some extra settings
kubeadmConfigPatches:
- |
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
evictionHard:
nodefs.available: "0%"
# patch it further using a JSON 6902 patch
kubeadmConfigPatchesJSON6902:
- group: kubeadm.k8s.io
version: v1beta3
kind: ClusterConfiguration
patch: |
- op: add
path: /apiServer/certSANs/-
value: my-hostname
# 1 control plane node and 3 workers
nodes:
# the control plane node config
- role: control-plane
# the three workers
- role: worker
- role: worker
- role: worker
---
### Site Content Docs User Known Issues (`site/content/docs/user/known-issues.md`)
---
title: "Known Issues"
menu:
main:
parent: "user"
identifier: "known-issues"
weight: 2
description: |-
Having problems with kind? This guide covers some known problems and solutions / workarounds.
It may additionally be helpful to:
- check our [issue tracker]
- [file an issue][file an issue] (if there isn't one already)
- reach out and ask for help in [#kind] on the [kubernetes slack]
[issue tracker]: https://github.com/kubernetes-sigs/kind/issues
[file an issue]: https://github.com/kubernetes-sigs/kind/issues/new
[#kind]: https://kubernetes.slack.com/messages/CEKK1KTN2/
[kubernetes slack]: https://slack.k8s.io/
---
## Contents
* [Troubleshooting Kind](#troubleshooting-kind)
* [Kubectl Version Skew](#kubectl-version-skew) (Kubernetes limits supported version skew)
* [Docker Installed With Snap](#docker-installed-with-snap) (snap filesystem restrictions problematic)
* [Failure to Build Node Image](#failure-to-build-node-image) (usually need to increase resources)
* [Failing to Properly Start Cluster](#failing-to-properly-start-cluster) (various causes)
* [Pod Errors Due to "too many open files"](#pod-errors-due-to-too-many-open-files) (likely [inotify] limits which are not namespaced)
* [Docker Permission Denied](#docker-permission-denied) (ensure you have permission to use docker)
* [Windows Containers](#windows-containers) (unsupported / infeasible)
* [Unsupported Architectures](#unsupported-architectures) (images not pre-built yet)
* [Unable to Pull Images](#unable-to-pull-images) (various)
* [Chrome OS](#chrome-os) (needs KubeletInUserNamespace)
* [AppArmor](#apparmor) (may break things, consider disabling)
* [IPv6 Port Forwarding](#ipv6-port-forwarding) (docker doesn't seem to implement this correctly)
* [Couldn't find an alternative telinit implementation to spawn](#docker-init-daemon-config)
* [Fedora](#fedora) (various)
* [Failed to get rootfs info](#failed-to-get-rootfs-info--stat-failed-on-dev)
* [Docker Desktop for macOS and Windows](#docker-desktop-for-macos-and-windows)
* [Older Linux Distributions](#older-linux-distributions)
* [Failure to Create Cluster on WSL2](#failure-to-create-cluster-on-wsl2)
* [Local Subnet Clashes](#local-subnet-clashes)
* [Unable to `kind load` Docker images](#unable-to-kind-load-docker-images) (`content digest ... not found`)
## Troubleshooting Kind
If the cluster fails to create, try again with the `--retain` option (preserving the failed container),
then run `kind export logs` to export the logs from the container to a temporary directory on the host.
## Kubectl Version Skew
You may have problems interacting with your kind cluster if your client(s) are
skewed too far from the kind node version. Kubernetes [only supports limited skew][version skew]
between clients and the API server.
This is a issue that frequently occurs when running `kind` alongside Docker For Mac.
This problem is related to a bug in [docker on macOS][for-mac#3663]
If you see something like the following error message:
```bash
$ kubectl edit deploy -n kube-system kubernetes-dashboard
error: SchemaError(io.k8s.api.autoscaling.v2beta1.ExternalMetricStatus): invalid object doesn't have additional properties
```
You can check your client and server versions by running:
{{< codeFromInline lang="bash" >}}
kubectl version
{{< /codeFromInline >}}
If there is a mismatch between the server and client versions, you should install a newer client version.
If you are using Mac, you can install kubectl via homebrew by running:
{{< codeFromInline lang="bash" >}}
brew install kubernetes-cli
{{< /codeFromInline >}}
And overwrite the symlinks created by Docker For Mac by running:
{{< codeFromInline lang="bash" >}}
brew link --overwrite kubernetes-cli
{{< /codeFromInline >}}
[for-mac#3663]: https://github.com/docker/for-mac/issues/3663
## Docker Installed with Snap
If you installed Docker with [snap], it is likely that `docker` commands do not
have access to `$TMPDIR`. This may break some kind commands which depend
on using temp directories (`kind build ...`).
Currently a workaround for this is setting the `TMPDIR` environment variable to
a directory snap does have access to when working with kind.
This can for example be some directory under `$HOME`.
## Failure to build node image
Building kind's node image may fail due to running out of memory on Docker for Mac or Docker for Windows.
See [kind#229][kind#229].
If you see something like this:
```
/* Detailed source-code truncated for AI context efficiency. */
```
Then you may try increasing the resource limits for the Docker engine on Mac or Windows.
It is recommended that you allocate at least 8GB of RAM to build Kubernetes.
Open the **Preferences** (macOS) or **Settings** (Windows) menu.
On macOS:
On Windows:
Go to the **Advanced** settings page, and change the settings there, see
[changing Docker's resource limits][Docker resource lims].
On macOS:
On Windows:
## Failing to properly start cluster
This issue is similar to a
[failure while building the node image](#failure-to-build-node-image).
If the cluster creation process was successful but you are unable to see any
Kubernetes resources running, for example:
```txt
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0261f7512fd kindest/node:v1.12.2 "/usr/local/bin/entrโฆ" About a minute ago Up About a minute 0.0.0.0:64907->64907/tcp kind-1-control-plane
$ docker exec -it c0261f7512fd /bin/sh
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
#
```
or `kubectl` being unable to connect to the cluster,
```txt
$ kind export kubeconfig
$ kubectl cluster-info
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Unable to connect to the server: EOF
```
Then as in [kind#156][kind#156], you may solve this issue by claiming back some
space on your machine by removing unused data or images left by the Docker
engine by running:
{{< codeFromInline lang="bash" >}}
docker system prune
{{< /codeFromInline >}}
And / or:
{{< codeFromInline lang="bash" >}}
docker image prune
{{< /codeFromInline >}}
You can verify the issue by exporting the logs (`kind export logs`) and looking
at the kubelet logs, which may have something like the following:
```txt
Dec 07 00:37:53 kind-1-control-plane kubelet[688]: I1207 00:37:53.229561 688 eviction_manager.go:340] eviction manager: must evict pod(s) to reclaim ephemeral-storage
Dec 07 00:37:53 kind-1-control-plane kubelet[688]: E1207 00:37:53.229638 688 eviction_manager.go:351] eviction manager: eviction thresholds have been met, but no pods are active to evict
```
## Pod errors due to "too many open files"
This may be caused by running out of [inotify](https://linux.die.net/man/7/inotify) resources. Resource limits are defined by `fs.inotify.max_user_watches` and `fs.inotify.max_user_instances` system variables. For example, in Ubuntu these default to 8192 and 128 respectively, which is not enough to create a cluster with many nodes.
To increase these limits temporarily run the following commands on the host:
{{< codeFromInline lang="bash" >}}
sudo sysctl fs.inotify.max_user_watches=524288
sudo sysctl fs.inotify.max_user_instances=512
{{< /codeFromInline >}}
To make the changes persistent, edit the file `/etc/sysctl.conf` and add these lines:
{{< codeFromInline lang="bash" >}}
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512
{{< /codeFromInline >}}
## Docker permission denied
When using `kind`, we assume that the user you are executing kind as has permission to use docker.
If you initially ran Docker CLI commands using `sudo`, you may see the following error, which indicates that your `~/.docker/` directory was created with incorrect permissions due to the `sudo` commands.
```txt
WARNING: Error loading config file: /home/user/.docker/config.json
open /home/user/.docker/config.json: permission denied
```
To fix this problem, either follow the docker's docs [manage docker as a non root user][manage docker as a non root user],
or try to use `sudo` before your commands (if you get `command not found` please check [this comment about sudo with kind][sudo with kind]).
## Docker init daemon config
Please make sure that when you use `kind`, you can't have `"init": true` in your `/etc/docker/daemon.json` because that will
cause `/sbin/init` to show the following cryptic message *Couldn't find an alternative telinit implementation to spawn*.
This has to to with `/sbin/init` not running as process id 1.
## Windows Containers
[Docker Desktop for Windows][docker desktop for windows] supports running both Linux (the default) and Windows Docker containers.
`kind` for Windows requires Linux containers. To switch between Linux and Windows containers see [this page][switch between windows and linux containers].
Windows containers are not like Linux containers and do not support running docker in docker and therefore cannot support kind.
## Unsupported Architectures
KIND currently ships pre-built images for AMD64 and ARM64 architectures.
In the future we may support others, but currently demand has been low and the cost to build
has been high.
To use kind on other architectures, you need to first build a base image
and then build a node image.
Run `images/base/build.sh` and then taking note of the built image name use `kind build node-image --base-image=kindest/base:tag-i-built`.
There are more details about how to do this in the [Quick Start] guide.
## Unable to pull images
When using named KIND instances you may sometimes see your images failing to pull correctly on pods. This will usually manifest itself with the following output when doing a `kubectl describe pod my-pod`
```txt
Failed to pull image "docker.io/my-custom-image:tag": rpc error: code = Unknown desc = failed to resolve image "docker.io/library/my-custom-image:tag": no available registry endpoint: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
```
If this image has been loaded onto your kind cluster using the command `kind load docker-image my-custom-image` then you have likely not provided the name parameter.
Re-run the command this time adding the `--name my-cluster-name` param:
`kind load docker-image my-custom-image --name my-cluster-name`
## Chrome OS
To run Kubernetes inside Chrome OS the LXC container must allow nesting. In Crosh session (ctrl+alt+t):
```txt
crosh> vmc launch termina
(termina) chronos@localhost ~ $ lxc config set penguin security.nesting true
(termina) chronos@localhost ~ $ lxc restart penguin
```
Then KIND cluster must use KubeletInUserNamespace feature gate (available since Kubernetes 1.22):
```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
KubeletInUserNamespace: true
```
## AppArmor
If your host has [AppArmor] enabled you may run into [moby/moby/issues/7512](https://github.com/moby/moby/issues/7512#issuecomment-51845976).
You will likely need to disable apparmor on your host or at least any profile(s)
related to applications you are trying to run in KIND.
See Previous Discussion: [kind#1179]
## IPv6 Port Forwarding
Docker assumes that all the IPv6 addresses should be reachable, hence doesn't implement
port mapping using NAT [moby#17666].
You will likely need to use Kubernetes services like NodePort or LoadBalancer to access
your workloads inside the cluster via the nodes IPv6 addresses.
See Previous Discussion: [kind#1326]
## Failed to get rootfs info / "stat failed on /dev/..."
On some systems, creating a cluster times out with these errors in kubelet.log (device varies):
```txt
stat failed on /dev/nvme0n1p3 with error: no such file or directory
"Failed to start ContainerManager" err="failed to get rootfs info: failed to get device for dir \"/var/lib/kubelet\": could not find device with major: 0, minor: 40 in cached partitions map"
```
Kubernetes needs access to storage device nodes in order to do some stuff, e.g. tracking free disk space. Therefore, Kind needs to mount the necessary device nodes from the host into the control-plane container โ however, it cannot always determine which device Kubernetes requires, since this varies with the host OS and filesystem. For example, the error above occurred with a BTRFS filesystem on Fedora Desktop 35.
This can be worked around by including the necessary device as an extra mount in the cluster configuration file.
```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraMounts:
- hostPath: /dev/nvme0n1p3
containerPath: /dev/nvme0n1p3
propagation: HostToContainer
```
To identify the device that must be listed, two variations have been observed.
* The device reported in the error message is a symlink (e.g. `/dev/mapper/luks-903aad3d-...`) โ in this case, the config file should refer to the target of that symlink (e.g. `/dev/dm-0`).
* The device reported in the error message is a regular block device (e.g. `/dev/nvme0n1p3`) โ in this case, use the device reported.
See Previous Discussion: [kind#2411]
## Fedora
### Firewalld
On Fedora 32 [firewalld] moved to nftables backend by default.
This seems to be incompatible with Docker, leading to KIND cluster nodes not
being able to reach each other.
You can work around this by changing the `FirewallBackend` in the `/etc/firewalld/firewalld.conf` file from `nftables` to `iptables` and restarting firewalld.
```console
sed -i /etc/firewalld/firewalld.conf 's/FirewallBackend=.*/FirewallBackend=iptables/'
systemctl restart firewalld
```
See [#1547 (comment)](https://github.com/kubernetes-sigs/kind/issues/1547#issuecomment-623756313)
and [Docker and Fedora 32 article](https://fedoramagazine.org/docker-and-fedora-32/)
### SELinux
On Fedora 33 an update to the SELinux policy causes `kind create cluster` to fail with an error like
```sh
docker: Error response from daemon: open /dev/dma_heap: permission denied.
```
Although the policy has been fixed in Fedora 34, the fix has not been backported to Fedora 33 as of June 28, 2021. Putting SELinux in permissive mode (`setenforce 0`) is one known workaround. This disables SELinux until the next boot. For more details, see [kind#2296].
## Docker Desktop for macOS and Windows
Docker containers cannot be executed natively on macOS and Windows, therefore
Docker Desktop runs them in a Linux VM. As a consequence, the container networks
are not exposed to the host and you cannot reach the kind nodes via IP.
You may be able to work around this limitation by configuring [extra port
mappings](https://kind.sigs.k8s.io/docs/user/configuration/#extra-port-mappings),
leveraging [cloud-provider-kind](https://github.com/kubernetes-sigs/cloud-provider-kind),
using a network proxy, or other solution specific to your environment.
## Older Linux Distributions
KIND uses a cgroup setting of `cgroupns=private`. The cgroup namespace functionality was added in 2016, so some of the
older Linux distributions, using older kernels, do not have the required functionality for KIND to work. Notably, distros
like Red Hat Enterprise Linux 7 and its clones.
Attempting to create a KIND cluster on a system with an older kernel will result in a failure, with an error message similar to:
```txt
Command Output: WARNING: Your kernel does not support cgroup namespaces. Cgroup namespace setting discarded.
```
Using KIND in these environments will require upgrading your OS to a more recent version that supports cgroup namespaces.
Another option is to run a virtual machine using a newer kernel.
## Failure to Create Cluster on WSL2
Some Linux kernel options for WSL2 do not have cgroup configured in a way that
KIND and other Linux-focused tools may expect. This may result in a failure
message when attempting to create a cluster, similar to:
```txt
unable to start container process: error adding pid 655569 to cgroups
```
The KIND development team is not able to provide support with Windows and WSL, so
the project relies on community support and feedback. It has been noted that the
steps detailed in [https://github.com/spurin/wsl-cgroupsv2](https://github.com/spurin/wsl-cgroupsv2)
have been necessary to resolve this issue.
## Local Subnet Clashes
KIND creates a separate docker network named `kind` that will be configured with default IPAM settings. If you are using the default IPAM configuration in your `daemon.json` you
may have conflicts with existing networks (like VPNs, labs, etc) that route the 172.17.x.x networks. To resolve this you can reconfigure the daemon-wide IPAM so that all
networks will be created in subnets that do not have these conflicts.
An example configuration that you can add to your `daemon.json` is below. This would configure `10.253.0.0/16` as the defauld CIDR with each individual network receiving a /24
subnet to use for allocation.
```json
"default-address-pools": [
{
"base": "10.253.0.0/16",
"size": 24
}
]
```
For more information on the Docker Engine config file check out [these docs](https://docs.docker.com/engine/daemon/).
## Unable to `kind load` Docker Images
When Docker is using the containerd image store, `kind load docker-image` may fail with an error similar to:
```text
ERROR: failed to load image: command "docker exec --privileged -i kind-control-plane ctr --namespace=k8s.io images import --all-platforms --digests --snapshotter=overlayfs -" failed with error: exit status 1
Command Output: ctr: content digest sha256:...: not found
```
See [kind#3795] for additional details.
A targeted workaround is to export only the platform required by the kind nodes and load the resulting image archive:
```sh
docker image save \
--platform linux/amd64 \
--output /tmp/image.tar \
my-image:tag
kind load image-archive /tmp/image.tar --name my-cluster
```
Alternatively, disabling Docker's containerd image store may avoid this issue.
> **NOTE**: Disabling the containerd image store changes Docker's host-wide image storage behavior.
> The single-platform image archive workaround is more targeted when only loading images into kind is affected.
You can disable Docker's containerd image store adding the following to `/etc/docker/daemon.json`:
```json
{
"features": {
"containerd-snapshotter": false
}
}
```
Restart Docker after changing this configuration.
Docker Desktop also exposes a setting for enabling or disabling the containerd image store. See [kind#3795] for discussion of this workaround for different Docker environments.
[kind#156]: https://github.com/kubernetes-sigs/kind/issues/156
[kind#229]: https://github.com/kubernetes-sigs/kind/issues/229
[kind#1179]: https://github.com/kubernetes-sigs/kind/issues/1179
[kind#1326]: https://github.com/kubernetes-sigs/kind/issues/1326
[kind#2296]: https://github.com/kubernetes-sigs/kind/issues/2296
[kind#2411]: https://github.com/kubernetes-sigs/kind/issues/2411
[kind#3795]: https://github.com/kubernetes-sigs/kind/issues/3795
[moby#17666]: https://github.com/moby/moby/issues/17666
[Docker resource lims]: https://docs.docker.com/docker-for-mac/#advanced
[snap]: https://snapcraft.io/
[manage docker as a non root user]: https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user
[sudo with kind]: https://github.com/kubernetes-sigs/kind/issues/713#issuecomment-512665315
[docker desktop for windows]: https://hub.docker.com/editions/community/docker-ce-desktop-windows
[switch between windows and linux containers]: https://docs.docker.com/docker-for-windows/#switch-between-windows-and-linux-containers
[version skew]: https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-version-skew
[Quick Start]: /docs/user/quick-start
[AppArmor]: https://en.wikipedia.org/wiki/AppArmor
[firewalld]: https://firewalld.org/
[inotify]: https://en.wikipedia.org/wiki/Inotify
---
### Site Content Docs User Loadbalancer (`site/content/docs/user/loadbalancer.md`)
---
title: "LoadBalancer"
menu:
main:
parent: "user"
identifier: "user-loadbalancer"
weight: 3
description: |-
This guide covers how to get service of type LoadBalancer working in a kind cluster using [Cloud Provider KIND].
This guide complements Cloud Provider KIND [installation docs].
[Cloud Provider KIND]: https://github.com/kubernetes-sigs/cloud-provider-kind
[installation docs]: https://github.com/kubernetes-sigs/cloud-provider-kind?tab=readme-ov-file#install
[Ingress Guide]: /docs/user/ingress
[Configuration Guide]: /docs/user/configuration#extra-port-mappings
---
## Installing Cloud Provider KIND
Cloud Provider KIND can be installed using golang
{{< codeFromInline lang="bash" >}}
go install sigs.k8s.io/cloud-provider-kind@latest
{{< /codeFromInline >}}
or downloading one of the [released binaries](https://github.com/kubernetes-sigs/cloud-provider-kind/releases).
Cloud Provider KIND runs as a standalone binary in your host and connects to your KIND cluster and provisions new Load Balancer containers for your Services. It requires privileges to open ports on the system and to connect to the container runtime.
## Using LoadBalancer
The following example creates a loadbalancer service that routes to two http-echo pods, one that outputs foo and the other outputs bar.
```yaml
{{% readFile "static/examples/loadbalancer/usage.yaml" %}}
```
Apply the contents
{{< codeFromInline lang="yaml" >}}
kubectl apply -f https://kind.sigs.k8s.io/examples/loadbalancer/usage.yaml
{{< /codeFromInline>}}
Now verify that the loadbalancer works by sending traffic to it's external IP and port.
{{< codeFromInline lang="bash" >}}
LB_IP=$(kubectl get svc/foo-service -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
{{< /codeFromInline >}}
```bash
# should output foo and bar on separate lines
for _ in {1..10}; do
curl ${LB_IP}:5678
done
```
---
### Site Content Docs User Local Registry (`site/content/docs/user/local-registry.md`)
---
title: "Local Registry"
menu:
main:
parent: "user"
identifier: "user-local-registry"
weight: 3
description: |-
This guide covers how to configure KIND with a local container image registry.
In the future this will be replaced by [a built-in feature](https://github.com/kubernetes-sigs/kind/issues/1213), and this guide will
cover usage instead.
---
## Create A Cluster And Registry
The following shell script will create a local docker registry and a kind cluster
with it enabled.
{{< codeFromFile file="static/examples/kind-with-registry.sh" >}}
## Using The Registry
The registry can be used like this.
1. First we'll pull an image `docker pull gcr.io/google-samples/hello-app:1.0`
2. Then we'll tag the image to use the local registry `docker tag gcr.io/google-samples/hello-app:1.0 localhost:5001/hello-app:1.0`
3. Then we'll push it to the registry `docker push localhost:5001/hello-app:1.0`
4. And now we can use the image `kubectl create deployment hello-server --image=localhost:5001/hello-app:1.0`
> **NOTE**: The example commands above use Docker. If you are using Podman
> and see `http: server gave HTTP response to HTTPS client` when pushing to
> `localhost:5001`, configure Podman to treat the local registry as insecure,
> or push with `podman push --tls-verify=false localhost:5001/hello-app:1.0`.
If you build your own image and tag it like `localhost:5001/image:foo` and then use
it in kubernetes as `localhost:5001/image:foo`.
If for some reason you have code running *inside* of a pod within the cluster that
needs to use this registry directly (e.g. to build and push an image) then that
code will need to use the `kind-registry:5000` HTTP endpoint directly, as the
code running inside your pod will not see the containerd config.
Pod manifests / pod specs / pod YAML should use `localhost:5001`,
which will be rerouted to match the same name as the host via the containerd config.
> **NOTE**: A bit about "localhost" and containers ...
> "localhost" resolves to a loopback IP, which are network-namespace local.
> Network-namespace local means that the `127.0.0.1` / `localhost` / `::1`
> inside your container is NOT the same as the one on your host
> (unless using `hostNetwork: true` / `--net=host`).
>
> KIND nodes are *not* `--net=host` and most pods are not, so really these
> "`localhost`s" are local to your pod container, to your kind node, and to your
> host machine in most cases.
>
> However we can tell containerd that `localhost:5001` should route to the
> registry container such that the same name works on both the host and your machine.
>
> We do this so you can conveniently push and pull from a "local"
> address on your host and in your pod YAML.
---
### Site Content Docs User Private Registries (`site/content/docs/user/private-registries.md`)
---
title: "Private Registries"
menu:
main:
parent: "user"
identifier: "user-private-registries"
weight: 3
toc: true
description: |-
This guide discusses how to use kind with image registries that
require authentication.
There are multiple ways to do this, which we try to cover here.
---
## Use ImagePullSecrets
Kubernetes supports configuring pods to use `imagePullSecrets` for pulling
images. If possible, this is the preferable and most portable route.
See [the upstream kubernetes docs for this][imagePullSecrets],
kind does not require any special handling to use this.
If you already have the config file locally but would still like to use secrets,
read through kubernetes' docs for [creating a secret from a file][imagePullFileSecrets].
## Pull to the Host and Side-Load
kind can [load an image][loading an image] from the host with the `kind load ...`
commands. If you configure your host with credentials to pull the desired
image(s) and then load them to the nodes you can avoid needing to authenticate
on the nodes.
## Add Credentials to the Nodes
Generally the upstream docs for [using a private registry] apply, with kind
there are two options for this.
### Mount a Config File to Each Node
If you pre-create a docker config.json containing credential(s) on the host
you can mount it to each kind node.
Assuming your file is at `/path/to/my/secret.json`, the kind config would be:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraMounts:
- containerPath: /var/lib/kubelet/config.json
hostPath: /path/to/my/secret.json
{{< /codeFromInline >}}
**Important**: The mounted file must contain the actual base64-encoded
credentials in the `auths` field. It cannot reference an external credential
store or helper.
Many systems store docker credentials outside of `config.json` using a
credential helper (e.g. `"credsStore": "desktop"` on Docker Desktop,
`"credsStore": "osxkeychain"` on macOS, or `"credsStore": "secretservice"` on
Linux). If your `~/.docker/config.json` contains a `credsStore` or `credHelpers`
key, the credentials themselves are **not** in the file and mounting it into a
kind node will not work because the credential helper binary is not present
inside the node.
To check whether your config contains plain credentials or a credential store
reference:
```sh
cat ~/.docker/config.json
```
A config with plain credentials looks like:
```json
{
"auths": {
"myregistry.example.com": {
"auth": "dXNlcjpwYXNzd29yZA=="
}
}
}
```
A config that uses an external credential store (which will **not** work when
mounted) looks like:
```json
{
"auths": {},
"credsStore": "desktop"
}
```
To generate a config file with plain credentials, create a temporary config that
bypasses the credential store and log in with it:
```sh
# Create a temp dir to hold the plain-credential config
DOCKER_CONFIG=$(mktemp -d)
export DOCKER_CONFIG
# Seed an empty auths entry to disable the credential store
cat <"${DOCKER_CONFIG}/config.json"
{
"auths": { "myregistry.example.com": {} }
}
EOF
# Log in โ credentials will be written as plain base64 auth in the file
docker login myregistry.example.com
# Use ${DOCKER_CONFIG}/config.json as your hostPath
```
For registries that use short-lived or token-based authentication (such as Azure
Container Registry with OAuth tokens), the static mount approach may not be
reliable once the token expires. For ACR, prefer logging in with a
[service principal][acrServicePrincipal] whose credentials are long-lived:
```sh
docker login myregistry.azurecr.io \
--username \
--password
```
[acrServicePrincipal]: https://learn.microsoft.com/en-us/azure/container-registry/container-registry-auth-service-principal
#### Use an Access Token
A credential can be programmatically added to the nodes at runtime.
If you do this then kubelet must be restarted on each node to pick up the new credentials.
An example shell snippet for generating a [gcr.io][GCR] cred file on your host machine
using Access Tokens:
{{< codeFromFile file="static/examples/kind-gcr.sh" >}}
#### Use a Service Account
Access tokens are short lived, so you may prefer to use a Service Account and keyfile instead.
First, either download the key from the console or generate one with gcloud:
```
gcloud iam service-accounts keys create --iam-account
```
Then, replace the `gcloud auth print-access-token | ...` line from the [access token snippet](#use-an-access-token) with:
```
cat | docker login -u _json_key --password-stdin https://gcr.io
```
See Google's [upstream docs][keyFileAuthentication] on key file authentication for more details.
[keyFileAuthentication]: https://cloud.google.com/container-registry/docs/advanced-authentication#json_key_file
[imagePullSecrets]: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod
[imagePullFileSecrets]: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#registry-secret-existing-credentials
[loading an image]: /docs/user/quick-start/#loading-an-image-into-your-cluster
[using a private registry]: https://kubernetes.io/docs/concepts/containers/images/#using-a-private-registry
[GCR]: https://cloud.google.com/container-registry/
#### Use a Certificate
If you have a registry authenticated with certificates, and both certificates and keys
reside on your host folder, it is possible to mount and use them into the `containerd` plugin
patching the default configuration, like in the example:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
# This option mounts the host docker registry folder into
# the control-plane node, allowing containerd to access them.
extraMounts:
- containerPath: /etc/docker/certs.d/registry.dev.example.com
hostPath: /etc/docker/certs.d/registry.dev.example.com
# NOTE: the following patch is not necessary with images from kind v0.27.0+
# It may enable some older images to work similarly
containerdConfigPatches:
- |-
[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d"
{{< /codeFromInline >}}
---
### Site Content Docs User Quick Start (`site/content/docs/user/quick-start.md`)
---
title: "Quick Start"
menu:
main:
parent: "user"
identifier: "user-quick-start"
weight: 1
toc: true
description: |-
This guide covers getting started with the `kind` command.
**If you are having problems please see the [known issues] guide.**
[known issues]: /docs/user/known-issues
---
## Installation
> **NOTE**: `kind` does not require [`kubectl`](https://kubernetes.io/docs/reference/kubectl/overview/),
> but you will not be able to perform some of the examples in our docs without it.
> To install `kubectl` see the upstream [kubectl installation docs](https://kubernetes.io/docs/tasks/tools/install-kubectl/).
If you are a go developer you may find the [go install option](#installing-with-go-install) convenient.
Otherwise we supply downloadable [release binaries](#installing-from-release-binaries), community-managed [packages](#installing-with-a-package-manager), and a [source installation guide](#installing-from-source).
Stable tagged releases (currently {{< stableVersion >}}) are generally strongly recommended for CI usage in particular.
You may need to install the latest code from source at HEAD if you are developing Kubernetes itself at HEAD / the latest sources.
### Installing From Release Binaries
Pre-built binaries are available on our [releases page](https://github.com/kubernetes-sigs/kind/releases).
To install, download the binary for your platform from "Assets", then rename it to `kind` (or perhaps `kind.exe` on Windows) and place this
into your `$PATH` at your preferred binary installation directory.
On Linux:
{{< codeFromInline lang="bash" >}}
# For AMD64 / x86_64
[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/{{< stableVersion >}}/kind-linux-amd64
# For ARM64
[ $(uname -m) = aarch64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/{{< stableVersion >}}/kind-linux-arm64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
{{< /codeFromInline >}}
On macOS:
{{< codeFromInline lang="bash" >}}
# For Intel Macs
[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/{{< stableVersion >}}/kind-darwin-amd64
# For M1 / ARM Macs
[ $(uname -m) = arm64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/{{< stableVersion >}}/kind-darwin-arm64
chmod +x ./kind
mv ./kind /some-dir-in-your-PATH/kind
{{< /codeFromInline >}}
On Windows in [PowerShell](https://en.wikipedia.org/wiki/PowerShell):
{{< codeFromInline lang="powershell" >}}
curl.exe -Lo kind-windows-amd64.exe https://kind.sigs.k8s.io/dl/{{< stableVersion >}}/kind-windows-amd64
Move-Item .\kind-windows-amd64.exe c:\some-dir-in-your-PATH\kind.exe
{{< /codeFromInline >}}
### Installing From Source
In addition to the pre-built binary + package manager installation options listed
above you can install kind from source with `go install sigs.k8s.io/kind@{{< stableVersion >}}` or clone this repo
and run `make build` from the repository.
#### Installing With `make`
Using `make build` does not require installing Go and will build kind reproducibly,
the binary will be in `bin/kind` inside your clone of the repo.
You should only need `make` and standard userspace utilities to run this build,
it will automatically obtain the correct go version with our vendored copy of [`gimme`](https://github.com/travis-ci/gimme).
You can then call `./bin/kind` to use it, or copy `bin/kind` into some directory in your system `PATH` to
use it as `kind` from the command line.
`make install` will attempt to mimic `go install` and has the same path requirements as `go install` below.
#### Installing with `go install`
When installing with [Go](https://golang.org/) please use the latest stable Go release.
To install use: `go install sigs.k8s.io/kind@{{< stableVersion >}}`.
If you are building from a local source clone, use `go install .` from the top-level directory of the clone.
`go install` will typically put the `kind` binary inside the `bin` directory under `go env GOPATH`, see
Go's ["Compile and install packages and dependencies"](https://golang.org/cmd/go/#hdr-Compile_and_install_packages_and_dependencies)
for more on this.
You may need to add that directory to your `$PATH` if you encounter the error
`kind: command not found` after installation, you can find a guide for adding a directory to your `PATH` at https://gist.github.com/nex3/c395b2f8fd4b02068be37c961301caa7#file-path-md.
### Installing With A Package Manager
The kind community has enabled installation via the following package managers.
> **NOTE**: The following are community supported efforts. The `kind` maintainers are not involved in the creation
> of these packages, and the upstream community makes no claims on the validity, safety, or content of them.
On macOS via Homebrew:
{{< codeFromInline lang="bash" >}}
brew install kind
{{< /codeFromInline >}}
On macOS via MacPorts:
{{< codeFromInline lang="bash" >}}
sudo port selfupdate && sudo port install kind
{{< /codeFromInline >}}
On Windows via Chocolatey (https://chocolatey.org/packages/kind)
{{< codeFromInline lang="powershell" >}}
choco install kind
{{< /codeFromInline >}}
On Windows via Scoop (https://scoop.sh/#/apps?q=kind&id=faec311bb7c6b4a174169c8c02358c74a78a10c2)
{{< codeFromInline lang="powershell" >}}
scoop bucket add main
scoop install main/kind
{{< /codeFromInline >}}
On Windows via Winget (https://github.com/microsoft/winget-pkgs/tree/master/manifests/k/Kubernetes/kind)
{{< codeFromInline lang="powershell" >}}
winget install Kubernetes.kind
{{< /codeFromInline >}}
On Arch Linux via pacman (https://archlinux.org/packages/extra/x86_64/kind/)
{{< codeFromInline lang="bash" >}}
sudo pacman -Syu kind
{{< /codeFromInline >}}
## Discovering Additional Command Options
kind provides built-in help for all commands and subcommands.
You can explore available flags and usage details by running:
kind help
kind --help
kind --help
This applies to commands such as `kind load image-archive`
## Creating a Cluster
Creating a Kubernetes cluster is as simple as `kind create cluster`.
This will bootstrap a Kubernetes cluster using a pre-built
[node image][node image]. Prebuilt images are hosted at[`kindest/node`][kindest/node], but to find images suitable for a given release currently you should check the [release notes] for your given kind version (check with `kind version`) where
you'll find a complete listing of images created for a kind release.
To specify another image use the `--image` flag -- `kind create cluster --image=...`.
Using a different image allows you to change the Kubernetes version of the created
cluster.
If you desire to build the node image yourself with a custom version see the
[building images](#building-images) section.
By default, the cluster will be given the name `kind`.
Use the `--name` flag to assign the cluster a different context name.
If you want the `create cluster` command to block until the control plane
reaches a ready status, you can use the `--wait` flag and specify a timeout.
To use `--wait` you must specify the units of the time to wait. For example, to
wait for 30 seconds, do `--wait 30s`, for 5 minutes do `--wait 5m`, etc.
More usage can be discovered with `kind create cluster --help`.
kind can auto-detect the [docker], [podman], or [nerdctl] installed and choose the available one. If you want to turn off the auto-detect, use the environment variable `KIND_EXPERIMENTAL_PROVIDER=docker`, `KIND_EXPERIMENTAL_PROVIDER=podman` or `KIND_EXPERIMENTAL_PROVIDER=nerdctl` to
select the runtime.
> **NOTE**: podman and nerdctl operate in [rootless mode](/docs/user/rootless) by default. Extra
> setup is needed for KIND clusters to be fully functional.
## Interacting With Your Cluster
After [creating a cluster](#creating-a-cluster), you can use [kubectl][kubectl]
to interact with it by using the [configuration file generated by kind][access multiple clusters].
By default, the cluster access configuration is stored in ${HOME}/.kube/config
if $KUBECONFIG environment variable is not set.
If $KUBECONFIG environment variable is set, then it is used as a list of paths
(normal path delimiting rules for your system). These paths are merged. When a value
is modified, it is modified in the file that defines the stanza. When a value is created,
it is created in the first file that exists. If no files in the chain exist,
then it creates the last file in the list.
You can use the `--kubeconfig` flag when creating the cluster, then only that file is loaded.
The flag may only be set once and no merging takes place.
To see all the clusters you have created, you can use the `get clusters`
command.
For example, let's say you create two clusters:
```
kind create cluster # Default cluster context name is `kind`.
...
kind create cluster --name kind-2
```
When you list your kind clusters, you will see something like the following:
```
kind get clusters
kind
kind-2
```
In order to interact with a specific cluster, you only need to specify the
cluster name as a context in kubectl:
```
kubectl cluster-info --context kind-kind
kubectl cluster-info --context kind-kind-2
```
## Deleting a Cluster
If you created a cluster with `kind create cluster` then deleting is equally
simple:
```
kind delete cluster
```
If the flag `--name` is not specified, kind will use the default cluster
context name `kind` and delete that cluster.
> **Note**: By design, requesting to delete a cluster that does not exist
> will not return an error. This is intentional and is a means to have an
> idempotent way of cleaning up resources.
## Loading an Image Into Your Cluster
You can load one or more images into your kind cluster:
```bash
kind load docker-image my-app:latest
```
```bash
kind load docker-image my-app:latest my-db:latest my-cache:latest
```
Note: If using a named cluster you will need to specify the name of the cluster:
```bash
kind load docker-image my-app:latest --name test-cluster
```
Additionally, image archives can be loaded with:
`kind load image-archive /my-image-archive.tar`
This allows a workflow like:
```
docker build -t my-custom-image:unique-tag ./my-image-dir
kind load docker-image my-custom-image:unique-tag
kubectl apply -f my-manifest-using-my-image.yaml
```
> **NOTE**: You can get a list of images present on a cluster node by
using `docker exec`:
> ```
> docker exec -it my-node-name crictl images
> ```
> Where `my-node-name` is the name of the Docker container (e.g. `kind-control-plane`).
> **NOTE**: The Kubernetes default pull policy is `IfNotPresent` unless
the image tag is `:latest` or omitted (and implicitly `:latest`) in which case the default policy is `Always`.
`IfNotPresent` causes the Kubelet to skip pulling an image if it already exists.
> If you want those images loaded into node to work as expected, please:
>
> - don't use a `:latest` tag
>
> and / or:
>
> - specify `imagePullPolicy: IfNotPresent` or `imagePullPolicy: Never` on your container(s).
>
> See [Kubernetes imagePullPolicy][Kubernetes imagePullPolicy] for more information.
See also: [Using kind with Private Registries][Private Registries].
## Building Images
> **NOTE**: If you're using Docker Desktop, be sure to read [Settings for Docker Desktop](#settings-for-docker-desktop) first.
kind runs a local Kubernetes cluster by using Docker containers as "nodes".
kind uses the [`node-image`][node image] to run Kubernetes artifacts, such
as `kubeadm` or `kubelet`.
The `node-image` in turn is built off the [`base-image`][base image], which
installs all the dependencies needed for Docker and Kubernetes to run in a
container.
Currently, kind supports one default way to build a `node-image`
if you have the [Kubernetes][kubernetes] source in your host machine
(`$GOPATH/src/k8s.io/kubernetes`), by using `source`.
You can also specify a different path to kubernetes source using
```
kind build node-image /path/to/kubernetes/source
```
> **NOTE**: Building Kubernetes node-images requires everything building upstream
> Kubernetes requires, we wrap the upstream build. This includes Docker with buildx.
> See: https://git.k8s.io/community/contributors/devel/development.md#building-kubernetes-with-docker
One shortcut to use a kubernetes release is to specify the version
directly to pick up the official tar-gzipped files:
```
kind build node-image v1.30.0
```
If you prefer to use existing tar-gzipped files like the ones from the kubernetes
release, you can specify those as well from a URL or local directory, for example:
```
kind build node-image https://dl.k8s.io/v1.30.0/kubernetes-server-linux-arm64.tar.gz
kind build node-image $HOME/Downloads/kubernetes-server-linux-amd64.tar.gz
```
To clear any confusion, you can specify the type of build explicitly
using `--type` parameter, please see the following examples:
```
kind build node-image --type url https://dl.k8s.io/v1.30.0/kubernetes-server-linux-arm64.tar.gz
kind build node-image --type file $HOME/Downloads/kubernetes-server-linux-amd64.tar.gz
kind build node-image --type release v1.30.0
kind build node-image --type source $HOME/go/src/k8s.io/kubernetes/
```
> **NOTE**: modes other than source directory namely `url`, `file` and `release` are only
> available in kind v0.24 and above.
### Settings for Docker Desktop
If you are building Kubernetes (for example - `kind build node-image`) on MacOS or Windows then you need a minimum of 6GB of RAM
dedicated to the virtual machine (VM) running the Docker engine. 8GB is recommended.
> **Note**: Resource requirements vary depending on usage.
> The values listed here are minimums for basic clusters.
> Workloads such as building node images, multi-node clusters,
> or running additional components may require more CPU and memory.
To change the resource limits for the Docker on Mac, you'll need to open the
**Preferences** menu.
Now, go to the **Advanced** settings page, and change the
settings there, see [changing Docker's resource limits][Docker resource lims].
To change the resource limits for the Docker on Windows, you'll need to right-click the Moby
icon on the taskbar, and choose "Settings". If you see "Switch to Linux Containers", then you'll need
to do that first before opening "Settings"
Now, go to the **Advanced** settings page, and change the
settings there, see [changing Docker's resource limits][Docker resource lims].
You may also try removing any unused data left by the Docker engine - e.g.,
`docker system prune`.
## Advanced
### Configuring Your kind Cluster
For a sample kind configuration file see [kind-example-config][kind-example-config].
To specify a configuration file when creating a cluster, use the `--config`
flag:
```
kind create cluster --config kind-example-config.yaml
```
#### Multi-node clusters
In particular, many users may be interested in multi-node clusters. A simple
configuration for this can be achieved with the following config file contents:
```yaml
# three node (two workers) cluster config
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
```
#### Control-plane HA
You can also have a cluster with multiple control-plane nodes:
```yaml
# a cluster with 3 control-plane nodes and 3 workers
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
- role: worker
- role: worker
```
#### Mapping ports to the host machine
You can map extra ports from the nodes to the host machine with `extraPortMappings`:
```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 80
hostPort: 80
listenAddress: "0.0.0.0" # Optional, defaults to "0.0.0.0"
protocol: udp # Optional, defaults to tcp
```
This can be useful if using `NodePort` services or daemonsets exposing host ports.
Note: binding the `listenAddress` to `127.0.0.1` may affect your ability to access the service.
You may want to see the [Ingress Guide] and [LoadBalancer Guide].
[Ingress Guide]: /docs/user/ingress
[LoadBalancer Guide]: /docs/user/loadbalancer
#### Setting Kubernetes version
You can also set a specific Kubernetes version by setting the `node`'s container image. You can find available image tags on the [releases page](https://github.com/kubernetes-sigs/kind/releases). Please use the `sha256` shasum for your desired kubernetes version, as seen in this example:
```yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.16.4@sha256:b91a2c2317a000f3a783489dfb755064177dbc3a0b2f4147d50f04825d016f55
- role: worker
image: kindest/node:v1.16.4@sha256:b91a2c2317a000f3a783489dfb755064177dbc3a0b2f4147d50f04825d016f55
```
### Enable Feature Gates in Your Cluster
Feature gates are a set of key=value pairs that describe alpha or experimental features. In order to enable a gate you have to [customize your kubeadm configuration][customize control plane with kubeadm], and it will depend on what gate and component you want to enable. An example kind config can be:
{{< codeFromInline lang="yaml" >}}
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
FeatureGateName: true
{{< /codeFromInline >}}
### Configure kind to use a proxy
If you are running kind in an environment that requires a proxy, you may need to configure kind to use it.
You can configure kind to use a proxy using one or more of the following [environment variables][proxy environment variables] (uppercase takes precedence):
* `HTTP_PROXY` or `http_proxy`
* `HTTPS_PROXY` or `https_proxy`
* `NO_PROXY` or `no_proxy`
> **NOTE**: If you set a proxy it would be passed along to everything in the kind nodes. `kind` will automatically append certain addresses into `NO_PROXY` before passing it to the nodes so that Kubernetes components connect to each other directly, but you may need to configure
> additional addresses depending on your usage.
### Exporting Cluster Logs
kind has the ability to export all kind related logs for you to explore.
To export all logs from the default cluster (context name `kind`):
```
kind export logs
Exported logs to: /tmp/396758314
```
Like all other commands, if you want to perform the action on a cluster with a
different context name use the `--name` flag.
As you can see, kind placed all the logs for the cluster `kind` in a
temporary directory. If you want to specify a location then simply add the path
to the directory after the command:
```
kind export logs ./somedir
Exported logs to: ./somedir
```
The structure of the logs will look more or less like this:
```
.
โโโ docker-info.txt
โโโ kind-control-plane/
โโโ containers
โโโ docker.log
โโโ inspect.json
โโโ journal.log
โโโ kubelet.log
โโโ kubernetes-version.txt
โโโ pods/
```
The logs contain information about the Docker host, the containers running
kind, the Kubernetes cluster itself, etc.
[modules]: https://github.com/golang/go/wiki/Modules
[go-supported]: https://golang.org/doc/devel/release.html#policy
[docker]: https://www.docker.com/
[podman]: https://podman.io/
[nerdctl]: https://github.com/containerd/nerdctl
[known issues]: /docs/user/known-issues
[releases]: https://github.com/kubernetes-sigs/kind/releases
[node image]: /docs/design/node-image
[base image]: /docs/design/base-image
[kind-example-config]: https://raw.githubusercontent.com/kubernetes-sigs/kind/main/site/content/docs/user/kind-example-config.yaml
[kubernetes]: https://github.com/kubernetes/kubernetes
[kindest/node]: https://hub.docker.com/r/kindest/node/
[kubectl]: https://kubernetes.io/docs/reference/kubectl/overview/
[Docker resource lims]: https://docs.docker.com/docker-for-mac/#advanced
[install docker]: https://docs.docker.com/install/
[proxy environment variables]: https://docs.docker.com/network/proxy/#use-environment-variables
[CGO]: https://golang.org/cmd/cgo/
[Kubernetes imagePullPolicy]: https://kubernetes.io/docs/concepts/containers/images/#updating-images
[Private Registries]: /docs/user/private-registries
[customize control plane with kubeadm]: https://kubernetes.io/docs/setup/independent/control-plane-flags/
[access multiple clusters]: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/
[release notes]: https://github.com/kubernetes-sigs/kind/releases
---
### Site Content Docs User Resources (`site/content/docs/user/resources.md`)
---
title: "Resources"
menu:
main:
parent: "user"
identifier: "resources"
weight: 4
toc: true
description: |-
This page contains references to additional external resources for learning about KIND and how to use it.
---
## Using KIND in CI
The [kind-ci/examples] project is a work-in-progress project to give working
examples of using KIND in various continuous integration environments / platforms.
## Tutorials and Guides
Here are a useful external guides / tutorials covering things not yet covered in our docs:
### How to use KIND with MetallLB
> **NOTE**: We now have a kind integrated loadbalancer solution, see [the loadbalancer page](/docs/user/loadbalancer/).
https://mauilion.dev/posts/kind-metallb/
### How to Test a Kubernetes PR with KIND
https://mauilion.dev/posts/kind-k8s-testing/
### Using Contour Ingress with KIND
https://projectcontour.io/kindly-running-contour/
### Local Ingress Domains for your Kind Cluster
https://mjpitz.com/blog/2020/10/21/local-ingress-domains-kind/
### Connect directly to Docker-for-Mac containers via IP address
https://golangexample.com/connect-directly-to-docker-for-mac-containers-via-ip-address/
### Developing for Kubernetes with KinD
https://docs.gitlab.com/charts/development/kind/
### Using CRI-O with KIND
> **NOTE**: Depending on implementation details of the node image is not supported, only that the node images contain what kind needs to run Kubernetes at a given version.
>
> Installing CRI-O is possible but not supported.
https://github.com/cri-o/cri-o/blob/release-1.31/tutorials/crio-in-kind.md#cri-o-in-kind
## KubeCon Talks
The authors have given the following talks relating to KIND:
### Keep Calm and Load Balance on KIND - Benjamin Elder & Antonio Ojea
At KubeCon EU 2024 we spoke about [Cloud Provider KIND and how to use Load Balancers Services with KIND](https://sched.co/1YhhY)
{{< youtube id="U6_-y24rJnI" class="video-wrapper" >}}
### Deep Dive: KIND - Benjamin Elder & Antonio Ojea
At KubeCon US 2019 we spoke about [KIND internals and the challenges ahead on the road to 1.0][kind-deep-dive].
{{< youtube id="tT-GiZAr6eQ" class="video-wrapper" >}}
### A Kind Workflow for Contributing to Kubernetes - Benjamin Elder & Duffie Cooley & James Munnelly & Patrick Lang
At KubeCon US 2019 we provided a hands on tutorial [for contributing and testing your Kubernetes code with KIND][kind-workflow-for-contributing-to-kubernetes].
{{< youtube id="BPVO2mcfjJk" class="video-wrapper" >}}
### Testing your K8s apps with KIND - Benjamin Elder & James Munnelly
At KubeCon EU 2019 we spoke about [KIND and testing your Kubernetes Applications][testing-k8s-apps-with-kind].
{{< youtube id="8KtmevMFfxA" class="video-wrapper" >}}
### Deep Dive: Testing SIG - Benjamin Elder & James Munnelly
At KubeCon EU 2019 we spoke about KIND and how we use it to test Kubernetes for the [SIG Testing Deep Dive][sig-testing-deep-dive-kind].
{{< youtube id="6m9frvTxK0o" class="video-wrapper" >}}
### Behind Your PR: How Kubernetes Uses Kubernetes to Run Kubernetes CI - Sen Lu & Benjamin Elder
At KubeCon NA 2018 we spoke with [Sen Lu][@krzyzacy] about The Kubernetes Project's
testing tools and infrastructure, including a brief discussion of KIND and running
it on Kubernetes's Kubernetes-based CI infrastructure.
{{< youtube id="pz0lpl6h-Gc" class="video-wrapper" >}}
[@krzyzacy]: https://github.com/krzyzacy
[kind-ci/examples]: https://github.com/kind-ci/examples
[testing-k8s-apps-with-kind]: https://kccnceu19.sched.com/event/MPYy/testing-your-k8s-apps-with-kind-benjamin-elder-google-james-munnelly-jetstackio
[sig-testing-deep-dive-kind]: https://kccnceu19.sched.com/event/MPkC/deep-dive-testing-sig-benjamin-elder-google-james-munnelly-jetstack
[kind-deep-dive]: https://kccncna19.sched.com/event/Uah7/deep-dive-kind-benjamin-elder-google-antonio-ojea-garcia-suse
[kind-workflow-for-contributing-to-kubernetes]: https://kccncna19.sched.com/event/Uaek/tutorial-a-kind-workflow-for-contributing-to-kubernetes-benjamin-elder-google-duffie-cooley-vmware-james-munnelly-jetstack-patrick-lang-microsoft-limited-available-seating-first-come-first-served-basis
---
### Site Content Docs User Rootless (`site/content/docs/user/rootless.md`)
---
title: "Rootless"
menu:
main:
parent: "user"
identifier: "rootless"
weight: 3
---
Starting with kind 0.11.0, [Rootless Docker](https://docs.docker.com/go/rootless/), [Rootless Podman](https://github.com/containers/podman/blob/master/docs/tutorials/rootless_tutorial.md) and [Rootless nerdctl](https://github.com/containerd/nerdctl/blob/main/docs/rootless.md) can be used as the node provider of kind.
## Provider requirements
- Docker: 20.10 or later
- Podman: 3.0 or later
- nerdctl: 1.7 or later
## Host requirements
### cgroup v2
The host needs to be running with cgroup v2, which is the default for many Linux disributions:
- Ubuntu: 21.10 and later.
- Fedora: 31 and later.
- Arch: April 2021 release and later.
You can verify the cgroup version used by your controller runtime with the following procedure:
- `docker`: Run `docker info` and look for `Cgroup Version: 2` in the output.
- `podman`: Run `podman info` and look for `cgroupVersion: v2` in the output.
- `nerdctl`: Run `nerdctl info` and look for `Cgroup Version: 2` in the output.
If the `info` output prints `Cgroup Version: 1` or equivalent, try the following to enable cgroup v2:
1. In `/etc/default/grub`, add the line `GRUB_CMDLINE_LINUX="systemd.unified_cgroup_hierarchy=1"`
2. Run `sudo update-grub` to enable cgroup v2.
Your host will also need to enable [cgroup delegation](https://systemd.io/CGROUP_DELEGATION/) of the `cpu` controller for
user services. This is enabled by default for distributions running `systemd` version 252 and higher.
To enable cgroup delegation for all the controllers, do the following:
1. Check your version of `systemd` by running `systemctl --version`. If the output prints
`systemd 252` or higher, no further action is needed. Example output below from a Fedora host:
```sh
$ systemctl --version
systemd 257 (257.9-2.fc42)
```
2. For systems with older versions of `systemd`, first create the directory
`/etc/systemd/system/user@.service.d/` if it is not present.
```sh
sudo mkdir -p /etc/systemd/system/user@.service.d/
```
3. Next, create the file `/etc/systemd/system/user@.service.d/delegate.conf` with the following content:
```ini
[Service]
Delegate=yes
```
4. Reload systemd for these changes to take effect:
```sh
sudo systemctl daemon-reload
```
5. If using docker, reload the user docker daemon:
```sh
systemctl --user restart docker
```
### Networking
Containers running in rootless mode may not loaded with host-level iptable modules.
This breaks the behavior of most networking components, such as Ingress and Gateway controllers.
To load the iptable modules, do the following:
1. First, use `lsmod` to check which kernel modules are loaded by default for user processes on
your system. Use `grep` to find which iptable modules are loaded:
```sh
lsmod | grep "ip.*table"
```
2. Check the output for the following kernel modules:
- `ip6_tables`
- `ip6table_nat`
- `ip_tables`
- `iptable_nat`
3. If one or more of the kernel modules above are not present, your system needs to load these at
startup for each process. First, run the following command to add these missing modules:
```sh
sudo tee /etc/modules-load.d/iptables.conf > /dev/null <<'EOF'
ip6_tables
ip6table_nat
ip_tables
iptable_nat
EOF
```
4. Check that the new module loading configuration is correct. You should see the following output:
```sh
$ cat /etc/modules-load.d/iptables.conf
ip6_tables
ip6table_nat
ip_tables
iptable_nat
```
5. Next, restart the `systemd-modules-load` service to make these changes effective immediately:
```sh
sudo systemctl restart systemd-modules-load.service
```
Alternatively, restart your system to ensure these changes take effect.
### Increase PID Limits
KIND nodes are represented as individual containers on their hosts. Runtimes such as podman set
default [process id limits](https://docs.podman.io/en/v4.3/markdown/options/pids-limit.html#pids-limit-limit)
that may be too low for the node or for a pod running on the node. The Ingress NGINX Controller is
[particularly susceptible](https://github.com/kubernetes-sigs/kind/issues/3451) to this issue.
To increase the PID limit, do the following:
1. If using podman, edit your `containers.conf` file (generally located in
`/etc/containers/containers.conf` or `~/.config/containers/containers.conf`) to increase the PIDs
limit to a desired value (default 4096 on most systems):
```ini
[containers]
pids_limit = 65536
```
2. Re-recreate the KIND cluster for these changes to take effect:
```sh
kind delete cluster && kind create cluster
```
### Increase inotify Limits
As documented in [known issues](/docs/user/known-issues/#pod-errors-due-to-too-many-open-files), pods may
fail by reaching inotify watch and instance limits. Ingress controllers such as NGINX and Contour
are particularly susceptible to this issue.
To increase the inotify limits, do the following:
1. As root, create a `.conf` file in `/etc/sysctl.d` that increases the `fs.inotify` max user settings:
```
fs.inotify.max_user_watches = 524288
fs.inotify.max_user_instances = 512
```
2. Reload `sysctl` for these changes to take effect:
```sh
sudo sysctl --system
```
Alternatively, restart your system for these changes to take effect.
### Allow Binding to Privileged Ports
If you use the `extraPortMappings` method to provide ingress to your KIND cluster, you can allow
the KIND node container to bind to ports 80 and 443 on the host. User containers cannot bind to
ports below 1024 by default as they are considered privileged.
You can avoid this issue by binding the node to a non-privileged host port, such as 8080 or 8443:
```yaml
# kind config.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 80
hostPort: 8080
protocol: TCP
- containerPort: 443
hostPort: 8443
protocol: TCP
```
Note that with this configuration, requests to your cluster ingress will need to add the
appropriate port number. In the example above, HTTP requests must use `localhost:8080` in the URL.
To allow a KIND node to bind to ports 80 and/or 443 on the host, do the following:
1. As root, create a `.conf` file in `/etc/sysctl.d` that lowers the privileged port start number:
```
# Allow unprivileged binding to HTTP port 80
# Use 443 if you only need binding to the default HTTPS port
net.ipv4.ip_unprivileged_port_start=80
```
2. Reload `sysctl` for these changes to take effect:
```sh
sudo sysctl --system
```
Alternatively, restart your system for these changes to take effect.
## Restrictions
The restrictions of Rootless Docker apply to kind clusters as well.
e.g.
- OverlayFS cannot be used unless the host is using kernel >= 5.11, or Ubuntu/Debian kernel
- Cannot mount block storage
- Cannot mount NFS
## Creating a kind cluster with Rootless Docker
To create a kind cluster with Rootless Docker, just run:
```console
$ export DOCKER_HOST=unix://${XDG_RUNTIME_DIR}/docker.sock
$ kind create cluster
```
## Creating a kind cluster with Rootless Podman
To create a kind cluster with Rootless Podman, just run:
```console
$ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster
```
On some distributions, you might need to use systemd-run to start kind into its own cgroup scope:
```console
$ systemd-run --scope --user kind create cluster
```
or
```console
$ systemd-run --scope --user -p "Delegate=yes" kind create cluster
```
If you still get the error `running kind with rootless provider requires setting systemd property "Delegate=yes"` even with [host requirements](#host-requirements) configured.
### Podman Log Driver
When using rootless Podman, kind may fail with the following error even though the cluster appears
to have started successfully:
```
ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"
```
This failure can have many different root causes, but if it looks like the node container has initialized,
it may be caused by an architectural issue with rootless Podman's log handling. kind monitors cluster
startup by tailing `podman logs`, but in rootless mode, by default Podman uses a userspace log relay
that may not connect until after systemd has already emitted the expected boot message. As a result,
kind's log watcher never sees the line and times out.
A workaround to this is to configure Podman to use a different log driver so kind can reliably
tail the logs.
One option is to use the `k8s-file` log driver:
```sh
mkdir -p ~/.config/containers
cat >> ~/.config/containers/containers.conf <}}
Enable-WindowsOptionalFeature -Online -FeatureName VirtualMachinePlatform, Microsoft-Windows-Subsystem-Linux
{{< /codeFromInline >}}
1. Reboot when prompted.
1. After the reboot, set WSL to default to WSL2. Open an admin PowerShell window and run
{{< codeFromInline lang="powershell" >}}
wsl --set-default-version 2
{{< /codeFromInline >}}
1. Now, you can install your Linux distro of choice by searching the Windows Store. If you don't want to use the Windows Store, then follow the steps in the WSL docs for [manual install](https://docs.microsoft.com/en-us/windows/wsl/install-manual).
1. Start up your distro with the shortcut added to the start menu
## Setting up Docker in WSL2 with Docker Desktop
Install Docker with WSL2 backend here: https://docs.docker.com/docker-for-windows/wsl/
## Setting up Docker in WSL2 without Docker Desktop
Alternatively, docker can be installed in WSL2 without using Docker Desktop.
See for example: https://dev.to/bowmanjd/install-docker-on-windows-wsl-without-docker-desktop-34m9
Now, move on to the [Quick Start](/docs/user/quick-start) to set up your cluster with kind.
## Accessing a Kubernetes Service running in WSL2
1. prepare cluster config with exported node port
{{< codeFromInline lang="yaml" >}}
# cluster-config.yml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraPortMappings:
- containerPort: 30000
hostPort: 30000
protocol: TCP
{{< /codeFromInline >}}
1. create cluster `kind create cluster --config=cluster-config.yml`
1. create deployment `kubectl create deployment nginx --image=nginx --port=80`
1. create service `kubectl create service nodeport nginx --tcp=80:80 --node-port=30000`
1. access service `curl localhost:30000`
Alternatively, see [Helpful Tips for WSL2](#helpful-tips-for-wsl2)
## Kubernetes Service with Session Affinity
If you want to create a Kubernetes Service with `sessionAffinity: ClientIP` it will not be accessible (and neither will any Service created afterwards).
WSL2 kernel is missing `xt_recent` kernel module, which is used by Kube Proxy to implement session affinity. You need to compile a custom kernel to enable this feature.
1. Build a kernel with `xt_recent` kernel module enabled
{{< codeFromInline lang="bash" >}}
docker run --name wsl-kernel-builder --rm -it ubuntu:latest bash
WSL_COMMIT_REF=linux-msft-wsl-6.6.87.2-1 # change this line to the version you want to build
# Install dependencies
apt update
apt install -y git build-essential flex bison libssl-dev libelf-dev bc dwarves python3 cpio
# Checkout WSL2 Kernel repo
mkdir src
cd src
git init
git remote add origin https://github.com/microsoft/WSL2-Linux-Kernel.git
git config --local gc.auto 0
git -c protocol.version=2 fetch --no-tags --prune --progress --no-recurse-submodules --depth=1 origin +${WSL_COMMIT_REF}:refs/remotes/origin/build/linux-msft-wsl-5.15.y
git checkout --progress --force -B build/linux-msft-wsl-5.15.y refs/remotes/origin/build/linux-msft-wsl-5.15.y
# Enable xt_recent kernel module
sed -i 's/# CONFIG_NETFILTER_XT_MATCH_RECENT is not set/CONFIG_NETFILTER_XT_MATCH_RECENT=y/' Microsoft/config-wsl
# Compile the kernel
make -j2 KCONFIG_CONFIG=Microsoft/config-wsl
# From the host terminal copy the newly built kernel
docker cp wsl-kernel-builder:/src/arch/x86/boot/bzImage .
{{< /codeFromInline >}}
1. Configure WSL to use newly built kernel: https://docs.microsoft.com/en-us/windows/wsl/wsl-config#configure-global-options-with-wslconfig
Create a `.wslconfig` file in `C:\Users\\`:
{{< codeFromInline lang="toml" >}}
[wsl2]
kernel=c:\\path\\to\\your\\kernel\\bzImage
{{< /codeFromInline >}}
## Helpful Tips for WSL2
- If you want to terminate the WSL2 instance to save memory or "reboot", open an admin PowerShell prompt and run `wsl --terminate `. Closing a WSL2 window doesn't shut it down automatically.
- You can check the status of all installed distros with `wsl --list --verbose`.
- If you had a distro installed with WSL1, you can convert it to WSL2 with `wsl --set-version 2`
- Alternative of [Accessing a Kubernetes Service running in WSL2](#accessing-a-kubernetes-service-running-in-wsl2) or [Setting Up An Ingress Controller](/docs/user/ingress/#setting-up-an-ingress-controller) for accessing workloads is using `kubectl port-forward --address=0.0.0.0`.
- See the [Known Issues](/docs/user/known-issues/) page for additional Windows-related issues and concerns.
---
### Site Content Docs User Working Offline (`site/content/docs/user/working-offline.md`)
---
title: "Working Offline"
menu:
main:
parent: "user"
identifier: "working-offline"
weight: 3
description: |-
This guide covers how to work with KIND in an offline / airgapped environment.
You should first [install kind][installation documentation] before continuing.
[installation documentation]: https://kind.sigs.k8s.io/docs/user/quick-start#installation
---
## Using a pre-built [node image][node image]
KIND provides some pre-built images,
these images contain everything necessary to create a cluster and can be used in an offline environment.
You can find available image tags on the [releases page][releases page].
Please include the `@sha256:` [image digest][image digest] from the image in the release notes.
You can pull it when you have network access,
or pull it on another machine and then transfer it to the target machine.
```
โ ~ docker pull kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62
sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62: Pulling from kindest/node
cc5a81c29aab: Pull complete
81c62728355f: Pull complete
ed9cffdd962a: Pull complete
6a46f000fce2: Pull complete
6bd890da28be: Pull complete
0d88bd219ffe: Pull complete
af5240f230f0: Pull complete
Digest: sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62
Status: Downloaded newer image for kindest/node@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62
docker.io/kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62
```
You can [save node image][docker save] to a tarball.
```
โ ~ docker save -o kind.v1.17.0.tar kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62
# or
โ ~ docker save kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62 | gzip > kind.v1.17.0.tar.gz
```
When you transport image tarball to the machine,
you can load the node image by [`docker load`][docker load] command.
```
โ ~ docker load -i kind.v1.17.0.tar
Loaded image ID: sha256:ec6ab22d89efc045f4da4fc862f6a13c64c0670fa7656fbecdec5307380f9cb0
# or
โ ~ docker load -i kind.v1.17.0.tar.gz
Loaded image ID: sha256:ec6ab22d89efc045f4da4fc862f6a13c64c0670fa7656fbecdec5307380f9cb0
```
And [create a tag][docker tag] for it.
```
โ ~ docker image tag kindest/node:v1.17.0@sha256:9512edae126da271b66b990b6fff768fbb7cd786c7d39e86bdf55906352fdf62 kindest/node:v1.17.0
โ ~ docker image ls kindest/node
REPOSITORY TAG IMAGE ID CREATED SIZE
kindest/node v1.17.0 ec6ab22d89ef 3 weeks ago 1.23GB
```
Finally, you can create a cluster by specifying the `--image` flag.
```
โ ~ kind create cluster --image kindest/node:v1.17.0
Creating cluster "kind" ...
โ Ensuring node image (kindest/node:v1.17.0) ๐ผ
โ Preparing nodes ๐ฆ
โ Writing configuration ๐
โ Starting control-plane ๐น๏ธ
โ Installing CNI ๐
โ Installing StorageClass ๐พ
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community ๐
```
## Building the [node image][node image]
In addition to using pre-built node image,
KIND also provides the ability to build [node image][node image] from Kubernetes source code.
Please note that during the image building process, you need to download many dependencies.
It is recommended that you build at least once online to ensure that these dependencies are downloaded to your local.
See [building the node image][building the node image] for more detail.
The node-image in turn is built off the [base image][base image].
### Prepare Kubernetes source code
You can clone Kubernetes source code.
```sh
โ ~ mkdir -p $GOPATH/src/k8s.io
โ ~ cd $GOPATH/src/k8s.io
โ ~ git clone https://github.com/kubernetes/kubernetes
```
### Building image
```sh
โ ~ kind build node-image --image kindest/node:main $GOPATH/src/k8s.io/kubernetes
Starting to build Kubernetes
...
Image build completed.
```
When the image build is complete, you can create a cluster by passing the `--image` flag.
```sh
โ ~ kind create cluster --image kindest/node:main
Creating cluster "kind" ...
โ Ensuring node image (kindest/node:main) ๐ผ
โ Preparing nodes ๐ฆ
โ Writing configuration ๐
โ Starting control-plane ๐น๏ธ
โ Installing CNI ๐
โ Installing StorageClass ๐พ
Set kubectl context to "kind-kind"
You can now use your cluster with:
kubectl cluster-info --context kind-kind
Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community ๐
```
## HA cluster
If you want to create a control-plane HA cluster
then you need to create a config file and use this file to start the cluster.
```sh
โ ~ cat << EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
# 3 control plane node and 1 workers
nodes:
- role: control-plane
- role: control-plane
- role: control-plane
- role: worker
EOF
```
Note that in an offline environment, in addition to preparing the node image,
you also need to prepare HAProxy image in advance.
You can find the specific tag currently in use at [loadbalancer source code][loadbalancer source code].
[installation documentation]: https://kind.sigs.k8s.io/docs/user/quick-start#installation
[node image]: https://kind.sigs.k8s.io/docs/design/node-image
[releases page]: https://github.com/kubernetes-sigs/kind/releases
[image digest]: https://docs.docker.com/engine/reference/commandline/pull/#pull-an-image-by-digest-immutable-identifier
[docker save]: https://docs.docker.com/engine/reference/commandline/save/
[docker load]: https://docs.docker.com/engine/reference/commandline/load/
[docker tag]: https://docs.docker.com/engine/reference/commandline/tag/
[base image]: https://kind.sigs.k8s.io/docs/design/base-image/
[building the node image]: https://kind.sigs.k8s.io/docs/user/quick-start/#building-images
[loadbalancer source code]: https://github.com/kubernetes-sigs/kind/blob/main/pkg/cluster/internal/loadbalancer/const.go#L20
---
--- METRICS ---
- Files Extracted: 37
- Estimated Token Budget: ~38598 tokens
- Canonical Reference: https://codewiki.google/github.com/kubernetes-sigs/kind