# Technical Documentation: k0sproject/k0s

> ℹ️ **Provenance:** Multi-Tier Ingestion · [CodeWiki Reference](https://codewiki.google/github.com/k0sproject/k0s)

### Overview (`README.md`)

# k0s - The Zero Friction Kubernetes



[](https://www.bestpractices.dev/projects/9994)
[](https://app.fossa.com/projects/git%2Bgithub.com%2Fk0sproject%2Fk0s?ref=badge_shield)
[](https://github.com/k0sproject/k0s/actions/workflows/go.yml?query=branch%3Amain)
[](https://github.com/k0sproject/k0s/actions/workflows/ostests-nightly.yaml)

[](https://github.com/k0sproject/k0s/tags?label=Downloads)


 





## Overview

k0s is an open source, all-inclusive Kubernetes distribution, which is configured with all of the features needed to build a Kubernetes cluster and packaged as a single binary for ease of use. Due to its simple design, flexible deployment options and modest system requirements, k0s is well suited for

- Any cloud
- Bare metal
- Edge and IoT

k0s drastically reduces the complexity of installing and running a CNCF certified Kubernetes distribution. With k0s new clusters can be bootstrapped in minutes and developer friction is reduced to zero. This allows anyone with no special skills or expertise in Kubernetes to easily get started.

k0s is distributed as a single binary with zero host OS dependencies besides the host OS kernel. It works with any Linux without additional software packages or configuration. Any security vulnerabilities or performance issues can be fixed directly in the k0s distribution that makes it extremely straightforward to keep the clusters up-to-date and secure.



## Key Features

- Certified and 100% upstream Kubernetes
- Multiple installation methods: [single-node](docs/install.md), [multi-node](docs/k0sctl-install.md), [airgap](docs/airgap-install.md) and [Docker](docs/k0s-in-docker.md)
- Automatic lifecycle management with k0sctl: [upgrade](docs/upgrade.md), [backup and restore](docs/backup.md)
- Modest [system requirements](docs/system-requirements.md) (1 vCPU, 1 GB RAM)
- Available as a single binary with no [external runtime dependencies](docs/external-runtime-deps.md) besides the kernel
- Flexible deployment options with [control plane isolation](docs/networking.md#controller-worker-communication) as default
- Scalable from a single node to large, [high-available](docs/high-availability.md) clusters
- Supports custom [Container Network Interface (CNI)](docs/networking.md) plugins (Kube-Router is the default, Calico is offered as a preconfigured alternative)
- Supports custom [Container Runtime Interface (CRI)](docs/runtime.md) plugins (containerd is the default)
- Supports all Kubernetes storage options with [Container Storage Interface (CSI)](docs/storage.md)
- Supports a variety of [datastore backends](docs/configuration.md#specstorage): etcd (default for multi-node clusters), SQLite (default for single node clusters), MySQL, and PostgreSQL
- Supports x86-64, ARM64, ARMv7 and RISC-V
- Includes [Konnectivity service](docs/networking.md#controller-worker-communication), CoreDNS and Metrics Server


## Getting Started

If you'd like to try k0s, please jump in to our:

- [Quick Start Guide](https://docs.k0sproject.io/stable/install/) - Create a full Kubernetes cluster with a single node that includes both the controller and the worker.
- [Install using k0sctl](https://docs.k0sproject.io/stable/k0sctl-install/) - Deploy and upgrade multi-node clusters with one command.
- [NanoDemo](https://docs.k0sproject.io/stable/#demo) - Watch a .gif recording on how to create a k0s instance.
- [Run k0s in Docker](https://docs.k0sproject.io/stable/k0s-in-docker/) - Run k0s controllers and workers in containers.
- For docs, tutorials, and other k0s resources, see [docs main page](https://docs.k0sproject.io).


## Join the Community

The main community communication channel is [#k0s-users] on Kubernetes Slack.
There you'll find like-minded people and reach out for assistance and
discussions.

We welcome your help in building k0s! If you are interested, we invite you to
join the [#k0s-dev] channel on Kubernetes Slack and check out the [contributing
guidelines].

We also run a [community blog] that's open to posts from anyone. Subscribe to
its [RSS feed][blog-rss] to stay up to date.

[community blog]: https://blog.k0sproject.io/
[blog-rss]: https://blog.k0sproject.io/index.xml
[#k0s-users]: https://kubernetes.slack.com/archives/C0809EA06QZ
[contributing guidelines]: https://docs.k0sproject.io/stable/contributors/
[#k0s-dev]: https://kubernetes.slack.com/archives/C07VAPJUECS

### Community calls

We hold regular community calls. Everyone is welcome to drop by to ask
questions, discuss ideas or planned contributions, or just chat:

- Community office hours, Eastern Hemisphere (EMEA-friendly time)
- Community office hours, Western Hemisphere (Americas-friendly time)
- Maintainers call, open to everyone but meant for maintainers to sync up on
  plans and work

All events are listed in the [k0sproject meeting calendar], and the
[meeting minutes] are public.

[k0sproject meeting calendar]: https://zoom-lfx.platform.linuxfoundation.org/meetings/k0s?view=month
[meeting minutes]: https://notes.cncf.io/s/_5FVgysYK


### Adopters

k0s is used across diverse environments, from small-scale far-edge deployments
to large data centers. Share your use case and add yourself to the list of
[adopters].

[adopters]: ADOPTERS.md


## Motivation

_We have seen a gap between the host OS and Kubernetes that runs on top of it: How to ensure they work together as they are upgraded independent from each other? Who is responsible for vulnerabilities or performance issues originating from the host OS that affect the K8S on top?_

**→** k0s is fully self contained. It’s distributed as a single binary with no host OS deps besides the kernel. Any vulnerability or perf issues may be fixed in k0s Kubernetes.

_We have seen Kubernetes with cumbersome lifecycle management, high minimum system requirements, weird host OS and infra restrictions, and/or need to use different distros to meet different use cases._

**→** k0s is designed to be lightweight at its core. It comes with a tool to automate cluster lifecycle management. It works on any host OS and infrastructure, and may be extended to work with any use cases such as edge, IoT, telco, public clouds, private data centers, and hybrid & hyper converged cloud applications without sacrificing the pure Kubernetes compliance or amazing developer experience.


## Status

k0s is ready for production (starting from v1.21.0+k0s.0). Since the initial release of k0s back in November 2020, we have made numerous releases, improved stability, added new features, and most importantly, listened to our users and community in an effort to create the most modern Kubernetes product out there. The active development continues to make k0s even better.


## Scope

While some Kubernetes distros package everything and the kitchen sink, k0s tries to minimize the amount of "add-ons" to bundle in. Instead, we aim to provide a robust and versatile "base" for running Kubernetes in various setups. Of course we will provide some ways to easily control and setup various "add-ons", but we will not bundle many of those into k0s itself. There are a couple of reasons why we think this is the correct way:

- Many of the addons such as ingresses, service meshes, storage etc. are VERY opinionated. We try to build this base with fewer opinions. :D
- Keeping up with the upstream releases with many external addons is very maintenance heavy. Shipping with old versions does not make much sense either.

With strong enough arguments we might take in new addons, but in general those should be something that are essential for the "core" of k0s.


## Build

The requirements for building k0s from source are as follows:

- GNU Make (v3.81 or newer)
- A POSIX shell
- coreutils
- findutils
- Docker

All of the compilation steps are performed inside Docker containers, no
installation of Go is required.

The k0s binary can be built in different ways:

The "k0s" way, self-contained, all binaries compiled from source, statically
linked, including embedded binaries:

```shell
make
```

The "package maintainer" way, without building and embedding the required
binaries. This assumes necessary binaries are provided separately at runtime:

```shell
make EMBEDDED_BINS_BUILDMODE=none
```

Docker build integration is enabled by default. However, in environments without
Docker, you can use the Go toolchain installed on the host system to build k0s
without embedding binaries. Note that static linking is not possible with
glibc-based toolchains:

```shell
make DOCKER='' EMBEDDED_BINS_BUILDMODE=none BUILD_GO_LDFLAGS_EXTRA=''
```

Note that the k0s build system does not currently support building the embedded
binaries without Docker. However, the embedded binaries can be built
independently using Docker:

```shell
make -C embedded-bins
```

Builds can be done in parallel:

```shell
make -j$(nproc)
```

## Smoke test

Additionally to the requirements for building k0s, the smoke tests _do_ require
a local Go installation. you can run `./vars.sh go_version` in a terminal to
find out the version that's being used to build k0s. It will print the
corresponding Go version to stdout.

To run a basic smoke test after build:

```shell
make check-basic
```

---

### Overview (`docs/README.md`)

# k0s - The Zero Friction Kubernetes






{%
    include-markdown "../README.md"
    start=""
    end=""
%}

{%
    include-markdown "../README.md"
    start=""
    end=""
%}

## Getting Started

- [Quick Start Guide](install.md) for creating a full Kubernetes cluster with a single node.
- [Install using k0sctl](k0sctl-install.md) - Deploy and upgrade multi-node clusters with one command.
- [Run k0s in Docker](k0s-in-docker.md) - Run k0s controllers and workers in containers.

## Demo



{%
    include-markdown "../README.md"
    start=""
    end=""
%}


### Adopters

k0s is used across diverse environments, from small-scale far-edge deployments
to large data centers. Share your use case and add yourself to the list of
[adopters].

[adopters]: adopters.md

## Commercial Support

[Mirantis](https://www.mirantis.com/software/k0s/) offers technical support, professional services and training for k0s. The support subscriptions include, for example, prioritized support (Phone, Web, Email) and access to verified extensions on top of your k0s cluster.

For any k0s inquiries, please contact us via email [info@k0sproject.io](mailto:info@k0sproject.io).

{%
    include-markdown "../README.md"
    start=""
    end=""
%}

{%
    include-markdown "../README.md"
    start=""
    end=""
%}

---

### Overview (`inttest/README.md`)

# Integration tests a.k.a e2e testing

This folder is the root of k0s integration tests. These tests are such that run the actual k0s clusters, currently using [bootloose](https://github.com/k0sproject/bootloose) as the target environment.

## Running the tests

There's a [`Makefile`](Makefile) defining the tests as a set of make targets.

## Test design

We're currently building the tests as Golang tests with the help of [testify](https://github.com/stretchr/testify/) test suite concept. The suite concept allows us to have suite level setup and teardown functionality so we can bootstrap and delete the test environment properly during testing. The suite setup phase creates the "infrastructure" for the tests and the teardown, as the name implies, deletes the infra.

## Keeping the test env after tests

Sometimes, especially when debugging some test failures, it's good to leave the environment running after the tests have ran. To control that behavior there's an env variable called `K0S_KEEP_AFTER_TESTS`. The value given to that has the following logic:

- no value or `K0S_KEEP_AFTER_TESTS="never"`: The test env is NOT left running regardless of the test results
- `K0S_KEEP_AFTER_TESTS="always"`: The test env is left running regardless of the test results
- `K0S_KEEP_AFTER_TESTS="failure"`: The test env is left running only if the tests have failed

The test output show how to run manual cleanup for the environment, something like:

```shell
TestNetworkSuite: bootloosesuite.go:138: bootloose cluster left intact for debugging. Needs to be manually cleaned with: bootloose delete --config /tmp/afghzzvp-bootloose.yaml
```

This allows you to run manual cleanup after you've done the needed debugging.

## Which tests to run when?

The plan is to run only the basic (and quick) smoke tests on each PR commit. We should build some bot like functionality to run longer and more expensive tests using some trigger before the final merge of a PR. Naturally we should be able to run any of the tests locally, or at least triggered locally, to ensure we can actually debug what is happening.

---

### Overview (`hack/ostests/README.md`)

# OpenTofu modules for k0s OS testing

Provisioning of k0s test clusters using different operating system stacks and
network providers based on OpenTofu and k0sctl.

By default, a test cluster will have the following properties:

* 3 controller nodes
* 2 worker nodes
* podCIDR is set to 10.244.0.0/16
* Node-local load balancing is enabled.
* All the rest of the configuration is left at its defaults.

Two workers are a minimum requirement for the Kubernetes conformance tests.

## Requirements

* [OpenTofu] >= 1.8

For the local plumbing:

* A POSIXish environment (`env`, `sh`, `echo`, `printf`)
* [k0sctl] (tested with 0.17.1)
* [jq] (tested with ~= 1.6)

For the AWS infra:

* Have the CLI credentials setup, in the usual AWS CLI way.
* Have a configured default region. That region will be targeted by OpenTofu.

[OpenTofu]: https://opentofu.org
[k0sctl]: https://github.com/k0sproject/k0sctl/
[jq]: https://jqlang.github.io/jq/

## Deploying a cluster

Be sure to meet the requisites listed above. Configure AWS credentials and
region. This is just an example, there are [other ways][aws-config] to do it.

```shell
export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...
export AWS_REGION=...
```

Select the desired cluster configuration for OpenTofu. This is just one example;
other methods of passing configuration options are covered in the [official
OpenTofu documentation][tf-config].

```shell
export TF_VAR_os=alpine_3_24
export TF_VAR_arch=x86_64
export TF_VAR_k0s_version=stable
export TF_VAR_k0s_network_provider=calico
export TF_VAR_k0s_kube_proxy_mode=ipvs
```

[aws-config]: https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html
[tf-config]: https://opentofu.org/docs/language/values/variables/#assigning-values-to-root-module-variables

Apply the configuration:

```shell
tofu apply
```

## OpenTofu configuration variables

### `os`: Operating system stack

* `al2023`: Amazon Linux 2023
* `alpine_3_21`: Alpine Linux 3.21
* `alpine_3_24`: Alpine Linux 3.24
* `centos_9`: CentOS Stream 9
* `centos_10`: CentOS Stream 10 (Coughlan)
* `debian_11`: Debian GNU/Linux 11 (bullseye) ([supported until 2026-08-31][debian-lts])
* `debian_12`: Debian GNU/Linux 12 (bookworm) ([supported until 2028-06-30][debian-lts])
* `fcos_stable`: [Fedora CoreOS stable stream](https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/#_streams)
* `fedora_41`: Fedora Linux 41 (Cloud Edition)
* `flatcar`: Flatcar Container Linux by Kinvolk
* `oracle_8_9`: Oracle Linux Server 8.9
* `oracle_9_3`: Oracle Linux Server 9.3
* `rhel_7`: Red Hat Enterprise Linux Server 7.9 (Maipo)
* `rhel_8`: Red Hat Enterprise Linux 8.10 (Ootpa)
* `rhel_9`: Red Hat Enterprise Linux 9.7 (Plow)
* `rocky_8`: Rocky Linux 8.10 (Green Obsidian)
* `rocky_9`: Rocky Linux 9.5 (Blue Onyx)
* `sles_15`: SUSE Linux Enterprise Server 15 SP6
* `ubuntu_2004`: Ubuntu 20.04 LTS
* `ubuntu_2204`: Ubuntu 22.04 LTS
* `ubuntu_2404`: Ubuntu 24.04
* `windows_server_2022`: Windows Server 2022 (runs the control plane on Alpine 3.23)

[debian-lts]: https://wiki.debian.org/LTS

### `arch`: Node architecture

The underlying processor architecture for the to-be-provisioned cluster. Note
that not all operating systems support all architectures.

* `x86_64`
* `arm64`

### `k0sctl_skip`: Skip k0s provisioning altogether

Just provision the infrastructure, but no k0s cluster. This may be used for
development and testing purposes.

Assuming the AWS credentials are available, it can be used like this:

```sh
tofu init
export TF_VAR_os=alpine_3_24
export TF_VAR_k0sctl_skip=true
tofu apply
tofu output -json | jq -r '
  (.ssh_private_key_filename.value | @sh) as $keyFile
    | .hosts.value[]
    | select(.connection.type = "ssh")
    | . as {name: $host, ipv4: $ip, connection: {username: $user}}
    | "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i \($keyFile) \($user)@\($ip) # \($host)"
'
```

### `k0s_version`: The k0s version to deploy

This may be a fixed version number, "stable" or "latest".

### `k0s_executable_path`: The k0s version to deploy

Path to the k0s executable to use, or null if it should be downloaded. Note that
for Windows, the `.exe` suffix is appended automatically.

### `k0s_network_provider`: Network provider

* `kuberouter`
* `calico`

### `k0s_kube_proxy_mode`: Mode of operation for kube-proxy

* `iptables`
* `ipvs`

See Kubernetes's [IPVS README] for details.

[IPVS README]: https://github.com/kubernetes/kubernetes/blob/master/pkg/proxy/ipvs/README.md

### Adding a new operating system

* Navigate to [modules/os/](modules/os/) and add a new file `os_<the-os-id>.tf`.
  Have a look at the other `os_*.tf` files for how it should look like.
* Add a new OS entry to [modules/os/main.tf](modules/os/main.tf).
* Update this README.
* Test it: Be sure to have the requisites ready, as described at the top of this
  README, then do `tofu apply -var=os=<the-os-id>`. When done, don't
  forget to clean up: `tofu destroy -var=os=<the-os-id>`.
* Update the [nightly trigger] and [matrix workflow] with the new OS ID.

### Updating the existing OS versions

New OS version usually mean just updating the OS AMI filtering criteria.
To test the AMI filtering locally, you can do the following:

```sh
tofu apply -refresh-only -target=module.os.data.aws_ami.<the-os-id>
tofu state show "module.os.data.aws_ami.<the-os-id>[0]"
```

## GitHub Actions workflows

There's a reusable GitHub Actions workflow available in [ostests-e2e.yaml]. It
will deploy the OpenTofu resources and perform Kubernetes conformance tests
against the provisioned test cluster.

[ostests-e2e.yaml]: ../../.github/workflows/ostests-e2e.yaml

### Launch a workflow run

There's a [nightly trigger] for the OS testing workflow. It will select and run
a single testing parameter combination each day. There's also a [matrix
workflow] that exposes more knobs and can be triggered manually, e.g. via [gh]:

```console
$ gh workflow run ostests-matrix.yaml --ref some/experimental/branch \
  -f oses='["alpine_3_24"]' \
  -f network-providers='["calico"]'
✓ Created workflow_dispatch event for ostests-matrix.yaml at some/experimental/branch

To see runs for this workflow, try: gh run list --workflow=ostests-matrix.yaml
```

Add `-f fail-fast=true` to the above command line to stop the e2e test suite
after the first test failure has been recorded.

[gh]: https://github.com/cli/cli

## TODO

* Figure out the best/canonical way to change host names of the AWS instances

[nightly trigger]: ../../.github/workflows/ostests-nightly.yaml
[matrix workflow]: ../../.github/workflows/ostests-matrix.yaml

---

### Architecture Index (`docs/architecture/index.md`)

# Architecture

**Note:** As k0s is a dynamic project, the product architecture may occasionally outpace the documentation. The high level concepts and patterns, however, should always apply.

## Packaging

The k0s package is a single, self-extracting binary that embeds Kubernetes binaries, the benefits of which include:

- Statically compiled
- No OS-level dependencies
- Requires no RPMs, dependencies, snaps, or any other OS-specific packaging
- Provides a single package for all operating systems
- Allows full version control for each dependency



## Control plane

As a single binary, k0s acts as the process supervisor for all other control plane components. As such, there is no container engine or kubelet running on controllers by default, which thus means that a cluster user cannot schedule workloads onto controller nodes.



Using k0s you can create, manage, and configure each of the components, running each as a "naked" process. Thus, there is no container engine running on the controller node.

## Storage

Kubernetes control plane typically supports only etcd as the datastore. k0s, however, supports many other datastore options in addition to etcd, which it achieves by including [kine](https://github.com/k3s-io/kine). Kine allows the use of a wide variety of backend data stores, such as MySQL, PostgreSQL, SQLite, and dqlite (refer to the [`spec.storage` documentation](../configuration.md#specstorage)).

In the case of k0s managed etcd, k0s manages the full lifecycle of the etcd cluster. For example, by joining a new controller node with `k0s controller "long-join-token"` k0s  automatically adjusts the etcd cluster membership info to allow the new member to join the cluster.

**Note**: k0s cannot shrink the etcd cluster. As such, to shut down the k0s controller on a node that node must first be manually removed from the etcd cluster.

## Worker node



As with the control plane, with k0s you can create and manage the core worker components as naked processes on the worker node.

By default, k0s workers use [containerd](https://containerd.io) as a high-level runtime and [runc](https://github.com/opencontainers/runc) as a low-level runtime. Custom runtimes are also supported, refer to [Using custom CRI runtimes](../runtime.md#using-custom-cri-runtimes).

---

### Contributors Index (`docs/contributors/index.md`)

# Contributing to k0s

Thank you for taking the time to contribute to k0s! This page gives you an
overview of how to contribute, and where to find the details.

Please follow the **[k0sproject community's code of conduct][coc]** and
the **[k0sproject AI policy][ai policy]** in all your interactions with the project.

If you have found a security vulnerability, don't disclose it publicly. Report
it confidentially, as described in the **[security policy]**.

[coc]: https://github.com/k0sproject/community/blob/main/CODE_OF_CONDUCT.md
[ai policy]: https://github.com/k0sproject/community/blob/main/AI-POLICY.md
[security policy]: https://github.com/k0sproject/k0s/blob/main/SECURITY.md

## Contributing to the repository

K0s is more than its code. Contributions to the documentation are as welcome and
as valuable as contributions to the code base.

### Seek early feedback

If there's more than one way to approach your change, reach out via [GitHub
issues] or other [community channels](../README.md#join-the-community) before
you start building it. Alternatives to a feature and different implementation
paths are easier to compare in the abstract than in a finished diff. Discussing
early also shows whether a change fits the project's direction, and whether
someone is already working on it.

### Code

The [code guidelines](code_guidelines.md) collect the conventions of the k0s
code base beyond what the linters enforce, such as comments, error handling,
logging, and how to maintain compatibility across versions.

If your change affects how k0s is used or configured, update the documentation
along with it.

The [testing](testing.md) page explains how to run the linters, unit tests, and
integration tests locally, how to debug CI failures, and how to run CI on your
own fork.

### Documentation

The documentation lives under the `docs` directory and is published to
<https://docs.k0sproject.io>. Fixing a typo, clarifying a confusing section, or
writing a guide for something that isn't covered yet are all great
contributions.

The [documentation](docs.md) page explains how the docs are built, how to add
a page to the navigation, and how to preview your changes locally.

### Pull requests

All changes, to the documentation and to the code alike, arrive as pull
requests. The [GitHub workflow](github_workflow.md) page walks you through the
process of

- forking the repository,
- writing commits that serve one purpose each and are signed off to certify
  the [Developer Certificate of Origin][DCO],
- opening the pull request,
- and reworking your commits after a review round.

Before opening a pull request, run the linters and related tests locally, as
described on the [testing](testing.md) page. CI runs on a pull request only
after a maintainer has approved it, so catching problems locally saves a round
trip.

[DCO]: https://developercertificate.org/

## Contributing beyond the repository

Not every contribution is a change to this repository. There are many ways to
help k0s and the k0sproject community as a whole:

- **Report bugs** via [GitHub issues].  
  A clear report with steps to reproduce, the k0s version, and relevant logs is
  a valuable contribution on its own.
- **Help others** in [GitHub issues] and on [#k0s-users].  
  Answering a question, sharing how you solved a problem, or adding what you
  know to an issue someone else reported helps the community.
- **Help the maintainers** by testing and reviewing [pull requests].  
  Trying out a change in your environment, or reading it with a fresh pair of
  eyes, is valuable feedback that speeds up the review.
- **Share your experience.**  
  Write a [blog post][k0s-blog], give a talk, or tell us about your use case.
  Add yourself to the list of [adopters], and let us know on Slack, so we can
  spread the word.
- **Join the [community calls](../README.md#community-calls).**  
  Discuss ideas, ask questions, or just get to know the people behind k0s.

[GitHub issues]: https://github.com/k0sproject/k0s/issues
[#k0s-users]: https://kubernetes.slack.com/archives/C0809EA06QZ
[pull requests]: https://github.com/k0sproject/k0s/pulls
[k0s-blog]: https://github.com/k0sproject/blog
[adopters]: https://github.com/k0sproject/k0s/blob/main/ADOPTERS.md

## License

By contributing, you agree that your contributions will be licensed as follows:

- All content residing under the "docs/" directory of this repository is licensed under "Creative Commons Attribution Share Alike 4.0 International" (CC-BY-SA-4.0). See docs/LICENSE for details.
- Content outside of the above mentioned directories or restrictions above is available under the "Apache License 2.0".

---

### Airgap Install (`docs/airgap-install.md`)

# Air gapped Installation

You can install k0s in environments without Internet access. Air gapped
installations require an image bundle that contains all the container images
that would normally be pulled over the network. K0s uses so-called OCI archives
for this: Tarball representations of an [OCI Image Layout]. They allow for
multiple images to be packed into a single file. K0s will watch for image
bundles in the `<data-dir>/images` folder will automatically import them into
the container runtime.

There are several ways to obtain an image bundle:

- Use the pre-built image bundles for different target platforms that are
  created for each k0s release. They contain all the images for the default k0s
  [image configuration](configuration.md#specimages) and can be downloaded from
  the [GitHub releases page].
- Create your own image bundle. In this case, you can easily customize the
  bundle to include container images that are not used by default in k0s.

**Note:** When importing image bundles, k0s uses ["loose" platform
matching](https://pkg.go.dev/github.com/containerd/platforms@v0.2.1#Only). For
example, on arm/v8, k0s will also import arm/v7, arm/v6, and arm/v5 images. This
means that your bundle can contain multi-arch images, and the import will be
done using platform compatibility.

[OCI Image Layout]: https://github.com/opencontainers/image-spec/blob/v1.0/image-layout.md
[GitHub releases page]: https://github.com/k0sproject/k0s/releases/{{{ k0s_version }}}

## Creating image bundles

### Using k0s built-in tooling

k0s ships with the [`k0s airgap`](cli/k0s_airgap.md) sub-command, which is
dedicated for tooling for airgapped environments. It allows for listing the
required images for a given configuration, as well as bundling them into an OCI
Image Layout archive.

1. Create the list of images required by k0s.

   ```shell
   k0s airgap list-images --all >airgap-images.txt
   ```

2. Review this list and edit it according to your needs.

3. Create the image bundle.

   ```shell
   k0s airgap bundle-artifacts -v -o image-bundle.tar <airgap-images.txt
   ```

### From a running worker node

As containerd pulls all the images during the k0s worker normal bootstrap, you
can use it to build the OCI bundle with images.

Use the following commands on a machine with an installed k0s worker:

```shell
k0s ctr images export image-bundle.tar $(k0s airgap list-images | xargs)
```

### Using third-party tools

There are several CLI tools that can help you fetch OCI artifacts and manage OCI
Image Layouts, such as [skopeo], [oras], or [crane]. The following is an example
uses Docker:

[skopeo]: https://github.com/containers/skopeo
[oras]: https://oras.land/
[crane]: https://github.com/google/go-containerregistry/blob/main/cmd/crane/README.md

#### Docker

1. Create the list of images required by k0s.

   ```shell
   k0s airgap list-images --all >airgap-images.txt
   ```

2. Review this list and edit it according to your needs.

3. Pull the images.

   ```shell
   xargs -I{} docker pull {} <airgap-images.txt
   ```

4. Create the bundle.

   ```shell
   docker image save -o image-bundle.tar $(xargs <airgap-images.txt)
   ```

## Placing image bundles on worker nodes

### By hand

Copy the `image-bundle.tar` to the target machine into the `images` directory in
the k0s data directory. Copy the bundle only to the worker nodes. Controller
nodes don't use it.

```console
# mkdir -p /var/lib/k0s/images
# cp image-bundle.tar /var/lib/k0s/images/image-bundle.tar
```

### Via k0sctl

As an alternative to the previous step, you can use `k0sctl` to upload image
bundles to worker nodes. `k0sctl` can also be used to upload the k0s binary file
to all nodes. Take a look at this example configuration with one controller and
one worker node to upload k0s binary and an image bundle:

```yaml
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  k0s:
    version: {{{ k0s_version }}}
  hosts:
    - role: controller
      ssh:
        address: <controller-ip-address>
        user: ubuntu
        keyPath: /path/to/.ssh/id_rsa

      #  uploadBinary: <boolean>
      #    When true the k0s binaries are cached and uploaded
      #    from the host running k0sctl instead of downloading
      #    directly to the target host.
      uploadBinary: true

      #  k0sBinaryPath: <local filepath>
      #    Upload a custom or manually downloaded k0s binary
      #    from a local path on the host running k0sctl to the
      #    target host.
      # k0sBinaryPath: path/to/k0s_binary/k0s

    - role: worker
      ssh:
        address: <worker-ip-address>
        user: ubuntu
        keyPath: /path/to/.ssh/id_rsa
      uploadBinary: true
      files:
        # This airgap bundle file will be uploaded from the k0sctl
        # host to the specified directory on the target host
        - src: /path/to/airgap-bundle-amd64.tar
          dstDir: /var/lib/k0s/images
          perm: 0755
```

## Disable image pulling (optional)

Use the following k0s configuration to ensure that all pods and pod templates
managed by k0s contain an `imagePullPolicy` of `Never`, ensuring that no images
are pulled from the Internet at any time.

```yaml
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  images:
    default_pull_policy: Never
```

---

### Autopilot Multicommand (`docs/autopilot-multicommand.md`)

# **Multi-Command Plans**

**Autopilot** relies on a **Plan** for defining the **Commands** that should be
executed, the **Signal Nodes** that each should be run on, and the status of
each **Command**.

A **Plan**:

* Defines one or many **Commands** that specify what actions should be performed.
* Specifies how **Signal Nodes** should be discovered per-**Command**.
* Saves the status of the **Plan** execution by resolved **Signal Nodes**

A **Command**:

* An instructional step inside of a **Plan** that is applied against a **Signal Node**

A **Signal Node**:

* Any node (controller or worker) that can receive updates with Autopilot.

## **Execution**

The execution of a **Plan** is the result of processing **Commands** through
a number of **Processing States**.

When a **Plan** is executed, each of the **Commands** are executed in the order
of their appearance in the **Plan**.

* A **Plan** transitions to the next **Command** only when the current **Command** posts
a state of **Completed**.
* Any **Command** that posts one of the recognized **Error States** will result in the
current **Command** and **Plan** to abort processing.
  * The status of the **Command** and **Plan** will reflect this.
* A **Plan** is considered finished only when all of its defined **Commands** post a
**Completed** state.

## **Status**

The progress and state of each **Command** is recorded in the **Plan** status.

* Every **Command** in the **Plan** has an associated status entry with the same index
as the **Command**
  * i.e. The second **Command** in a **Plan** has an index of `1`, and so does its status.
* The status of all **Commands** is taken into consideration when determining if the
**Plan** is finished.

---

## **Example**

The following is an example of a **Plan** that has been applied as is currently being
processed by **autopilot**.

(line numbers added for commentary below)

```
/* Detailed source-code truncated for AI context efficiency. */
```

* Lines **7-33** are the two **Commands** that make up this plan - an `airgapupdate` and `k0supdate`.
* Lines **38-55** are the associated status entries for the two **Commands**.

The state of this **Plan** excerpt is that **autopilot** has successfully processed the **Plan**, and
has begun processing the `airgapupdate` **Command**. Its status indicates **SignalSent** which means
that the **Signal Node** has been sent signaling information to perform an air gapped update.

---

## **Processing States**

The following are the various states that both `Plan`s and `Command`s
adhere to.

```mermaid
stateDiagram-v2
    [*]-->NewPlan
    NewPlan-->SchedulableWait
    NewPlan-->Errors***
    
    SchedulableWait-->Schedulable
    SchedulableWait-->Completed
    Schedulable-->SchedulableWait

    Errors***-->[*]
    Completed-->[*]
```

Note that the **Errors*** state is elaborated in detail below in **Error States**.

### **NewPlan**

When a **Plan** is created with the name `autopilot`, the **NewPlan** state
processing takes effect.

It is the responsibility of **NewPlan** to ensure that the status of **all**
the **Commands** are represented in the **Plan** status. This **Plan** status
is needed at later points in **Plan** processing to determine if the entire
**Plan** is completed.

The main difference between **NewPlan** and all the other states is that
**NewPlan** will iterate over all commands; the other states deal with the
active command.

### **SchedulableWait**

Used to evaluate a **Command** to determine if it can be scheduled for processing.
If the **Command** is determined that it can be processed, the state is set to
**Schedulable**.

### **Schedulable**

The **Schedulable** state is set by **SchedulableWait** to indicate that this
command should execute. The execution of a **Command** in this state will be
whichever logic is defined by the **Command**.

The ending of this state should either transition to **SchedulableWait** for
further processing + completion detection, or transition to an error.

### **Completed**

The **Completed** state indicates that the command has finished processing.
Once a plan/command are in the **Completed** state, no further processing
will occur on this plan/command.

---

## **Error States**

When a plan or command processing goes into one of the designated error
states, this is considered fatal and the plan/command processing will
terminate.

Error states are generally defined by the **Command** implementation.
The core **autopilot** functionality is only interested when in the
4 core states (**NewPlan**, **SchedulableWait**, **Schedulable**, **Completed**),
and treats all other states as an error.

```mermaid
flowchart TD
    Errors --> IncompleteTargets
    Errors --> Restricted
    Errors -. no longer in use .-> MissingSignalNode
```

| Error State | Command | States | Description |
| ----------- | ------- | ------ | ----------- |
| **IncompleteTargets** | `airgapupdate`, `k0supdate` | **NewPlan**, **Schedulable** | Indicates that a **Signal Node** that existed during the discover phase in **NewPlan** no longer exists (ie. no `ControlNode` or `Node` object) |
| **Restricted** | `airgapupdate`, `k0supdate` | **NewPlan** | Indicates that a **Plan** has requested an update of a **Signal Node** type that contradicts the startup exclusions (the `--exclude-from-plans` argument) |
| **~~MissingSignalNode~~** | `airgapupdate`, `k0supdate` | **Schedulable** | Legacy state used by older k0s releases to indicate **IncompleteTargets**; no longer in use. |

---

## **Sequence: Example**

Using the example above as a reference, this outlines the basic sequence of events of
state transitions to the operations performed on each object.

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Autopilot (`docs/autopilot.md`)

# Autopilot

A tool for updating your `k0s` controller and worker nodes using specialized plans.
There is a public update-server hosted on the same domain as the documentation site. See the example below on how to use it. There is only a single channel `edge_release` available. The channel exposes the latest released version.

## How it works

* You create a `Plan` YAML
  * Defining the update payload (new version of `k0s`, URLs for platforms, etc.)
  * Add definitions for all the nodes that should receive the update.
    * Either statically, or dynamically using label/field selectors
* Apply the `Plan`
  * Applying a `Plan` is a simple `kubectl apply` operation.
* Monitor the progress
  * The applied `Plan` provides a status that details the progress.

## Automatic updates

To enable automatic updates, create an `UpdateConfig` object:

```yaml
apiVersion: autopilot.k0sproject.io/v1beta2
kind: UpdateConfig
metadata:
  name: example
  namespace: default
spec:
  channel: edge_release
  updateServer: https://updates.k0sproject.io/
  upgradeStrategy:
    type: periodic
    periodic:
      # The following fields configures updates to happen only on Tue or Wed at 13:00-15:00
      days: [Tuesdays,Wednesday]
      startTime: "13:00"
      length: 2h
  planSpec: # This defines the plan to be created IF there are updates available
    ...
```

## Safeguards

There are a number of safeguards in place to avoid breaking a cluster.

### Stateless Component

* The **autopilot** component were designed to not require any heavy state, or
massive synchronization. Controllers can disappear, and backup controllers can
resume the **autopilot** operations.

### Workers Update Only After Controllers

* The versioning that Kubelet and the Kubernetes API server adhere to requires that Kubelets
should **not** be of a newer version than the API server.

* How **autopilot** handles this is that when a `Plan` is applied that has both controller
and worker nodes, **all** the controller nodes will be updated first. It is only when
**all** controllers have updated **successfully** that worker nodes will receive their
update instructions.

### Plans are Immutable

* When you apply a `Plan`, **autopilot** evaluates all of the controllers and workers that
should be included into the `Plan`, and tracks them in the status. After this point, no
additional changes to the plan (other than status) will be recognized.
  * This helps in largely dynamic worker node environments where nodes that may have been
    matched by the `selector` discovery method no longer exist by the time the update
    is ready to be scheduled.

### Controller Quorum Safety

* Prior to scheduling a controller update, **autopilot** queries the API server of **all**
  controllers to ensure that they report a successful `/ready`.
* Only once all controllers are `/ready` will the current controller get sent update signaling.
* If a previously discovered controller target can no longer be resolved, the `Plan` transitions
  into an `IncompleteTargets` state and execution ends. Transient readiness failures requeue.

### Controllers Update Sequentially

* Despite having the configuration options for controllers to set concurrency, only **one**
  controller will be updated at a time.

### Update Payload Verification

* Each `update` object payload can provide an optional `sha256` hash of the update content
  (specified in `url`), which is compared against the update content after it downloads.

## Configuration

**Autopilot** relies on a `Plan` object on its instructions on what to update.

Here is an arbitrary **Autopilot** plan:

```yaml
apiVersion: autopilot.k0sproject.io/v1beta2
kind: Plan
metadata:
  name: autopilot

spec:
  id: id1234
  timestamp: now

  commands:
    - k0supdate:
        version: {{{ k0s_version }}}
        platforms:
          linux-amd64:
            url: https://github.com/k0sproject/k0s/releases/download/{{{ k0s_version }}}/k0s-{{{ k0s_version }}}-amd64
            sha256: '0000000000000000000000000000000000000000000000000000000000000000'
        targets:
          controllers:
            discovery:
              static:
                nodes:
                  - ip-172-31-44-131
                  - ip-172-31-42-134
                  - ip-172-31-39-65
          workers:
            limits:
              concurrent: 5
            discovery:
              selector:
                labels: environment=staging
                fields: metadata.name=worker2
```

### Core Fields

#### `apiVersion <string> (required)`

* The current version of the Autopilot API is `v1beta2`, with a full group-version of `autopilot.k0sproject.io/v1beta2`

#### `metadata.name <string> (required)`

* The name of the plan should always be `autopilot`
  * **Note:** Plans will not execute if they don't follow this convention.

### Spec Fields

#### `spec.id <string> (optional)`

* An identifier that can be provided by the creator for informational and tracking purposes.

#### `spec.timestamp <string> (optional)`

* A timestamp value that can be provided by the creator for informational purposes. **Autopilot**
does nothing with this information.

#### `spec.commands[] (required)`

* The `commands` contains the commands that should be performed as a part of the plan.

### **`k0supdate`** Command

#### `spec.commands[].k0supdate.version <string> (required)`

* The version of the binary being updated. This version is used to compare against the installed
version before and after update to ensure success.

#### `spec.commands[].k0supdate.platforms.*.url <string> (required)`

* A URL providing where the updated binary should be downloaded from, for this specific platform.
  * The naming of platforms is a combination of `$GOOS` and `$GOARCH`, separated by a hyphen (`-`)
    * e.g.: `linux-amd64`, `linux-arm64`, `linux-arm`
  * **Note:** The main supported platform is `linux`. **Autopilot** may work on other platforms, however
this has not been tested.

#### `spec.commands[].k0supdate.platforms.*.sha256 <string> (optional)`

* If a SHA256 hash is provided for the binary, the completed downloaded will be verified against it.

#### `spec.commands[].k0supdate.targets.controllers <object> (optional)`

* This object provides the details of how `controllers` should be updated.

#### `spec.commands[].k0supdate.targets.controllers.limits.concurrent <int> (fixed as 1)`

* The configuration allows for specifying the number of concurrent controller updates
through the plan spec, however for controller targets this is fixed always to `1`.
* By ensuring that only one controller updates at a time, we aim to avoid scenarios
where quorom may be disrupted.

#### `spec.commands[].k0supdate.targets.workers <object> (optional)`

* This object provides the details of how `workers` should be updated.

#### `spec.commands[].k0supdate.targets.workers.limits.concurrent <int> (optional, default = 1)`

* Specifying a `concurrent` value for worker targets will allow for that number of workers
to be updated at a time. If no value is provided, `1` is assumed.

### **`airgapupdate`** Command

#### `spec.commands[].airgapupdate.version <string> (required)`

* The version of the airgap bundle being updated.

#### `spec.commands[].airgapupdate.platforms.*.url <string> (required)`

* An URL providing where the updated binary should be downloaded from, for this specific platform.
  * The naming of platforms is a combination of `$GOOS` and `$GOARCH`, separated by a hyphen (`-`)
    * eg: `linux-amd64`, `linux-arm64`, `linux-arm`
  * **Note:** The main supported platform is `linux`. **Autopilot** may work on other platforms, however
this has not been tested.

#### `spec.commands[].airgapupdate.platforms.*.sha256 <string> (optional)`

* If a SHA256 hash is provided for the binary, the completed downloaded will be verified against it.

#### `spec.commands[].airgapupdate.workers <object> (optional)`

* This object provides the details of how `workers` should be updated.

#### `spec.commands[].airgapupdate.workers.limits.concurrent <int> (optional, default = 1)`

* Specifying a `concurrent` value for worker targets will allow for that number of workers
to be updated at a time. If no value is provided, `1` is assumed.

### Static Discovery

This defines the `static` discovery method used for this set of targets (`controllers`, `workers`). The `static` discovery method relies on a fixed set of hostnames defined
in `.nodes`.

It is expected that a `Node` (workers) or `ControlNode` (controllers) object exists with
the same name.

```yaml
  static:
    nodes:
      - ip-172-31-44-131
      - ip-172-31-42-134
      - ip-172-31-39-65
```

#### `spec.commands[].k0supdate.targets.*.discovery.static.nodes[] <string> (required for static)`

* A list of hostnames that should be included in target set (`controllers`, `workers`).

### Selector Target Discovery

The `selector` target discovery method relies on a dynamic query to the Kubernetes API
using labels and fields to produce a set of hosts that should be updated.

Providing both `labels` and `fields` in the `selector` definition will result in a logical `AND` of both operands.

```yaml
  selector:
    labels: environment=staging
    fields: metadata.name=worker2
```

Specifying an empty selector will result in *all* nodes being selected for this target set.

```yaml
  selector: {}
```

#### `spec.commands[].k0supdate.targets.*.discovery.selector.labels <string> (optional)`

* A collection of name/value labels that should be used for finding the appropriate nodes
for the update of this target set.

#### `spec.commands[].k0supdate.targets.*.discovery.selector.fields <string> (optional)`

* A collection of name/value fields that should be used for finding the appropriate nodes
for the update of this target set.
  * **Note:** Currently only the field `metadata.name` is available as a query field.

## Status Reporting

After a `Plan` has been applied, its progress can be viewed in the `.status` of the
`autopilot` Plan.

```shell
    kubectl get plan autopilot -oyaml
```

An example of a `Plan` status:

```yaml
  status:
    state: SchedulableWait
    commands:
    - state: SchedulableWait
      k0supdate:
        controllers:
        - lastUpdatedTimestamp: "2022-04-07T15:52:44Z"
          name: controller0
          state: SignalCompleted
        - lastUpdatedTimestamp: "2022-04-07T15:52:24Z"
          name: controller1
          state: SignalCompleted
        - lastUpdatedTimestamp: "2022-04-07T15:52:24Z"
          name: controller2
          state: SignalPending
        workers:
        - lastUpdatedTimestamp: "2022-04-07T15:52:24Z"
          name: worker0
          state: SignalPending
        - lastUpdatedTimestamp: "2022-04-07T15:52:24Z"
          name: worker1
          state: SignalPending
        - lastUpdatedTimestamp: "2022-04-07T15:52:24Z"
          name: worker2
          state: SignalPending
```

To read this status, this indicates that:

* The overall status of the update is `SchedulableWait`, meaning that **autopilot** is
  waiting for the next opportunity to process a command.
* There are three controller nodes
  * Two controllers have `SignalCompleted` successfully
  * One is waiting to be signaled (`SignalPending`)
* There are also three worker nodes
  * All are awaiting signaling updates (`SignalPending`)

### Plan Status

The `Plan` status at `.status.status` represents the overall status of the **autopilot**
update operation. There are a number of statuses available:

| Status | Description | Ends Plan? |
| ------ | ----------- | ---------- |
| `IncompleteTargets` | There are nodes in the resolved `Plan` that do not have associated `Node` (worker) or `ControlNode` (controller) objects. | Yes |
| `Schedulable` | Indicates that the `Plan` can be re-evaluated to determine which next node to update. | No |
| `SchedulableWait` | Scheduling operations are in progress, and no further update scheduling should occur. | No |
| `Completed` | The `Plan` has run successfully to completion. | Yes |
| `Restricted` | The `Plan` included node types (controller or worker) that violates the `--exclude-from-plans` restrictions. | Yes |

### Node Status

Similar to the **Plan Status**, the individual nodes can have their own statuses:

| Status | Description |
| ------ | ----------- |
| `SignalPending` | The node is available and awaiting an update signal |
| `SignalSent` | Update signaling has been successfully applied to this node. |
| `SignalMissingPlatform` | This node is a platform that an update has not been provided for. |
| `SignalMissingNode` | This node does have an associated `Node` (worker) or `ControlNode` (controller) object. |

## UpdateConfig

### UpdateConfig Core Fields

#### `apiVersion <string> (required field)`

* API version. The current version of the Autopilot API is `v1beta2`, with a full group-version of `autopilot.k0sproject.io/v1beta2`

#### `metadata.name <string> (required field)`

* Name of the config.

### Spec

#### `spec.channel <string> (optional)`

* Update channel to use. Supported values: `stable`(default), `unstable`.

#### `spec.updateServer <string> (optional)`

* Update server url. Defaults to `https://updates.k0sproject.io`

#### `spec.upgradeStrategy.type <enum:cron|periodic>`

* Select which update strategy to use.

#### `spec.upgradeStrategy.cron <string> (optional)` **DEPRECATED**

* Schedule to check for updates in crontab format.

#### `spec.upgradeStrategy.cron <object>`

Fields:

* `days`: On which weekdays to check for updates
* `startTime`: At which time of day to check updates
* `length`: The length of the update window

#### `spec.planSpec <string> (optional)`

* Describes the behavior of the autopilot generated `Plan`

### Example

```yaml
apiVersion: autopilot.k0sproject.io/v1beta2
kind: UpdateConfig
metadata:
  name: example
spec:
  channel: stable
  updateServer: https://updates.k0sproject.io/
  upgradeStrategy:
    type: periodic
    periodic:
      # The following fields configures updates to happen only on Tue or Wed at 13:00-15:00
      days: [Tuesdays,Wednesday]
      startTime: "13:00"
      length: 2h
  # Optional. Specifies a created Plan object
  planSpec:
    commands:
      - k0supdate: # optional
          forceupdate: true # optional
          targets:
            controllers:
              discovery:
                static:
                  nodes:
                    - ip-172-31-44-131
                    - ip-172-31-42-134
                    - ip-172-31-39-65
            workers:
              limits:
                concurrent: 5
              discovery:
                selector:
                  labels: environment=staging
                  fields: metadata.name=worker2
        airgapupdate: # optional
          workers:
            limits:
              concurrent: 5
            discovery:
              selector:
                labels: environment=staging
                fields: metadata.name=worker2
```

## FAQ

### Q: How do I apply the `Plan` and `ControlNode` CRDs?

A: These CRD definitions are embedded in the **k0s** binary and applied on startup.
No additional action is needed.

### Q: How will `ControlNode` instances get removed?

A: `ControlNode` instances are created by **autopilot** controllers as they startup. When
controllers disappear, they will **not** remove their associated `ControlNode` instance. It
is the responsibility of the operator/administrator to ensure their maintenance.

### Q: I upgraded my workers, and now Kubelets are no longer reporting

You probably upgraded your workers to an API version greater than what is
available on the API server.

https://kubernetes.io/releases/version-skew-policy/

> Make sure that your controllers are at the desired version **first** before
> upgrading workers.

---

### Backup (`docs/backup.md`)

# Backup/Restore overview

k0s has integrated support for backing up cluster state and configuration. The k0s backup utility is aiming to back up and restore k0s managed parts of the cluster.

The backups created by `k0s backup` command have following pieces of your cluster:

- certificates (the content of the `<data-dir>/pki` directory)
- etcd snapshot, if the etcd data store is used
- Kine/SQLite snapshot, if the Kine/SQLite data store is used
- k0s.yaml
- any custom defined manifests under the `<data-dir>/manifests`
- any image bundles located under the `<data-dir>/images`
- any helm configuration

Parts **NOT** covered by the backup utility:

- `PersistentVolumes` of any running application
- data store, in case something else than etcd or Kine/SQLite is used
- any configuration to the cluster introduced by manual changes (e.g. changes that weren't saved under the `<data-dir>/manifests`)

Any of the backup/restore related operations MUST be performed on the controller node.

## Backup/restore a k0s node locally

### Backup (local)

To create backup run the following command on the controller node:

```shell
k0s backup --save-path=<directory>
```

The directory used for the `save-path` value must exist and be writable. The default value is the current working directory.
The command provides backup archive using following naming convention: `k0s_backup_<ISODatetimeString>.tar.gz`

Because of the date/time usage, it is guaranteed that none of the previously created archives would be overwritten.

To output the backup archive to standard output, use `-` as the save path.

### Restore (local)

To restore cluster state from the archive use the following command on the controller node:

```shell
k0s restore /tmp/k0s_backup_2021-04-26T19_51_57_000Z.tar.gz
```

The command would fail if the data directory for the current controller has overlapping data with the backup archive content.

The command would use the archived `k0s.yaml` as the cluster configuration description.
It is written to the current working directory as `k0s_<archive timestamp>.yaml`
(use `--config-out` to change this).

If the backup is restored onto a node with a different IP address than the one
it was taken on, and the archived configuration pins the etcd peer address via
`spec.storage.etcd.peerAddress`, adjust it before starting the controller:
either set it to the new node's address or remove it to have it detected
automatically. Then start the controller with `--config` pointing at the
adjusted file. k0s will update the peer URL of the restored etcd member
accordingly on startup. This only works as long as the restored controller is
the sole etcd member, i.e. before any additional controllers are joined.

In case if your cluster is HA, after restoring single controller node, join the rest of the controller nodes to the cluster.
E.g. steps for N nodes cluster would be:

- Restore backup on fresh machine
- Run controller there
- Join N-1 new machines to the cluster the same way as for the first setup.

To read the backup archive from standard input, use `-` as the file path.

### Encrypting backups (local)

By using `-` as the save or restore path, it is possible to pipe the backup archive through an encryption utility such as [GnuPG](https://gnupg.org/) or [OpenSSL](https://www.openssl.org/).

Note that unencrypted data will still briefly exist as temporary files on the local file system during the backup archive generation.

#### Encrypting backups using GnuPG

Follow the instructions for your operating system to install the `gpg` command if it is not already installed.

This tutorial only covers the bare minimum for example purposes. For secure key management practices and advanced usage refer to the GnuPG user manual.

To generate a new key-pair, use:

```shell
gpg --gen-key
```

The key will be stored in your key ring.

```shell
gpg --list-keys
```

This will output a list of keys:

```shell
/home/user/.gnupg/pubring.gpg
------------------------------
pub   4096R/BD33228F 2022-01-13
uid                  Example User <user@example.com>
sub   4096R/2F78C251 2022-01-13
```

To export the private key for decrypting the backup on another host, note the key ID (`BD33228F` in this example) in the list and use:

```shell
gpg --export-secret-keys --armor BD33228F > k0s.key
```

To create an encrypted k0s backup:

```shell
k0s backup --save-path - | gpg --encrypt --recipient user@example.com > backup.tar.gz.gpg
```

#### Restoring encrypted backups using GnuPG

You must have the private key in your GnuPG keychain. To import the key that was exported in the previous example, use:

```shell
gpg --import k0s.key
```

To restore the encrypted backup, use:

```shell
gpg --decrypt backup.tar.gz.gpg | k0s restore -
```

## Backup/restore a k0s cluster using k0sctl

With k0sctl you can perform cluster level backup and restore remotely with one command.

### Backup (remote)

To create backup run the following command:

```shell
k0sctl backup
```

k0sctl connects to the cluster nodes to create a backup. The backup file is stored in the current working directory.

### Restore (remote)

To restore cluster state from the archive use the following command:

```shell
k0sctl apply --restore-from /path/to/backup_file.tar.gz
```

The control plane load balancer address (`externalAddress`) needs to remain the same between backup and restore. This is caused by the fact that all worker node components connect to this address and cannot currently be re-configured.

---

### Cis Benchmark (`docs/cis_benchmark.md`)

# Kube-bench Security Benchmark

[Kube-bench](https://github.com/aquasecurity/kube-bench) is an open source tool which can be used to verify security best practices as defined in CIS Kubernetes Benchmark. It provides a number of tests to help harden your k0s clusters. By default, k0s will pass Kube-bench benchmarks with some exceptions, which are shown below.

## Run

Follow the Kube-bench [quick start instructions](https://github.com/aquasecurity/kube-bench/#quick-start).

After installing the Kube-bench on the host that is running `k0s` cluster run the following command:

```shell
kube-bench run --config-dir docs/kube-bench/cfg/ --benchmark k0s-1.0
```

## Summary of disabled checks

### Master Node Security Configuration

The current configuration has in total 8 master checks disabled:

1. **id: 1.2.10** - EventRateLimit requires external yaml config. It is left for the users to configure it

    ```yaml
    type: skip
    text: "Ensure that the admission control plugin EventRateLimit is set (Manual)"
    ```

2. **id: 1.2.12** - By default this isn't passed to the apiserver for air-gap functionality

    ```yaml
    type: skip
    text: "Ensure that the admission control plugin AlwaysPullImages is set (Manual)"
    ```

3. **id: 1.2.22** - For sake of simplicity of k0s all audit configurations are skipped. It is left for the users to configure it

    ```yaml
    type: skip
    text: "Ensure that the --audit-log-path argument is set (Automated)"
    ```

4. **id: 1.2.23** - For sake of simplicity of k0s all audit configuration are skipped. It is left for the users to configure it

    ```yaml
    type: skip
    text: "Ensure that the --audit-log-maxage argument is set to 30 or as appropriate (Automated)"
    ```

5. **id: 1.2.24** - For sake of simplicity of k0s all audit configurations are skipped. It is left for the users to configure it

    ```yaml
    type: skip
    text: "Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate (Automated)"
    ```

6. **id: 1.2.25** - For sake of simplicity of k0s all audit configurations are skipped. It is left for the users to configure it

    ```yaml
    type: skip
    text: "Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate (Automated)"
    ```

7. **id: 1.2.33** - By default it is not enabled. Left for the users to decide

    ```yaml
    type: skip
    text: "Ensure that the --encryption-provider-config argument is set as appropriate (Manual)"
    ```

8. **id: 1.2.34** - By default it is not enabled. Left for the users to decide

    ```yaml
    type: skip
    text: "Ensure that encryption providers are appropriately configured (Manual)"
    ```

### Worker Node Security Configuration

and 4 node checks disabled:

1. **id: 4.1.1** - not applicable since k0s does not use kubelet service file

    ```yaml
    type: skip
    text: "Ensure that the kubelet service file permissions are set to 644 or more restrictive (Automated)"
    ```

2. **id: 4.1.2** - not applicable since k0s does not use kubelet service file

    ```yaml
    type: skip
    text: "Ensure that the kubelet service file ownership is set to root:root (Automated)"
    ```

3. **id: 4.2.6** - k0s does not set this. See https://github.com/kubernetes/kubernetes/issues/66693

    ```yaml
    type: skip
    text: "Ensure that the --protect-kernel-defaults argument is set to true (Automated)"
    ```

4. **id: 4.2.10** - k0s doesn't set this up because certs get auto rotated

    ```yaml
    type: skip
    text: "Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate (Manual)"
    ```

### Control Plane Configuration

3 checks for the control plane:

1. **id: 3.1.1** - For purpose of being fully automated k0s is skipping this check

    ```yaml
    type: skip
    text: "Client certificate authentication should not be used for users (Manual)"
    ```

2. **id: 3.2.1** - out-of-the box configuration does not have any audit policy configuration but users can customize it in spec.api.extraArgs section of the config

    ```yaml
    type: skip
    text: "Ensure that a minimal audit policy is created (Manual)"
    ```

3. **id: 3.2.2** - Same as previous

    ```yaml
    type: skip
    text: "Ensure that the audit policy covers key security concerns (Manual)"
    ```

### Kubernetes Policies

Policy checks are also disabled. The checks are manual and are up to the end user to decide on them.

---

### Cloud Providers (`docs/cloud-providers.md`)

# Cloud providers

K0s supports all [Kubernetes cloud controllers]. However, those must be installed as separate cluster add-ons since k0s builds Kubernetes components in *providerless* mode.

[Kubernetes cloud controllers]: https://kubernetes.io/docs/concepts/architecture/cloud-controller/

## Enable cloud provider support in kubelet

You must enable cloud provider mode for kubelet. To do this, run the workers with `--enable-cloud-provider=true`.

When deploying with [k0sctl](k0sctl-install.md), you can add this into the `installFlags` of worker hosts.

```yaml
spec:
  hosts:
  - ssh:
      address: 10.0.0.1
      user: root
      keyPath: ~/.ssh/id_rsa
    installFlags:
      - --enable-cloud-provider
      - --kubelet-extra-args="--cloud-provider=external"
    role: worker
```

## Deploy the cloud provider

You can use any means to deploy your cloud controller into the cluster. Most providers support [Helm charts](helm-charts.md) to deploy them.

**Note**: The prerequisites for the various cloud providers can vary (for example, several require that configuration files be present on all of the nodes). Refer to your chosen cloud provider's documentation as necessary.

## k0s Cloud Provider

Alternatively, k0s provides its own lightweight cloud provider that can be used to statically assign `ExternalIP` values to worker nodes via Kubernetes annotations.  This is beneficial for those who need to expose worker nodes externally via static IP assignments.

To enable this functionality, add the parameter `--enable-k0s-cloud-provider=true` to all controllers, and `--enable-cloud-provider=true` to all workers.

Adding a static IP address to a node using `kubectl`:

```shell
kubectl annotate \
    node <node> \
    k0sproject.io/node-ip-external=<external IP>[,<external IP 2>][,<external IP 3>]
```

Both IPv4 and IPv6 addresses and multiple comma-separated values are supported.

### Defaults

The default node refresh interval is `2m`, which can be overridden using the `--k0s-cloud-provider-update-frequency=<duration>` parameter when launching the controller(s).

The default port that the cloud provider binds to can be overridden using the `--k0s-cloud-provider-port=<int>` parameter when launching the controller(s).

---

### Commercial Support (`docs/commercial-support.md`)

# Commercial support

Commercial support for k0s if offered by [Mirantis Inc.](https://mirantis.com).

Mirantis can provide various different levels of support starting from DevCare (9-to-5) all the way to OpsCare+ with fully managed service.

On top of our normal [release and support](releases.md) model our commercial customers have access to critical security patches even for released versions that fall outside of the Open Source maintained releases.[^1] Commercial support also includes support for k0s related tooling such as [k0sctl](https://github.com/k0sproject/k0sctl).

If you are interested in commercial support for k0s check out our [support description](https://www.mirantis.com/support/enterprise-support-options/) and please [contact us](https://www.mirantis.com/contact/) for further details.

[^1]: This is assuming there is a compatible release of upstream project with the fix

---

### Compose Cluster.Yaml (`docs/compose-cluster.yaml`)

name: compose-cluster

x-k0s-image: &k0s-image docker.io/k0sproject/k0s:{{{ k0s_docker_version }}}

x-k0s-controller: &k0s-controller
  image: *k0s-image
  networks:
    - k0s-net
  tmpfs:
    - /run # this is where k0s stores runtime data
    - /tmp
  configs:
    - source: k0s.yaml
      target: /etc/k0s/k0s.yaml
  labels:
    - traefik.enable=true
    - traefik.tcp.routers.kube-api.service=kube-api
    - traefik.tcp.routers.kube-api.rule=HostSNI(`*`)
    - traefik.tcp.routers.kube-api.entrypoints=kube-api
    - traefik.tcp.services.kube-api.loadbalancer.server.port=6443
    - traefik.tcp.routers.k0s-api.service=k0s-api
    - traefik.tcp.routers.k0s-api.rule=HostSNI(`*`)
    - traefik.tcp.routers.k0s-api.entrypoints=k0s-api
    - traefik.tcp.services.k0s-api.loadbalancer.server.port=9443
    - traefik.tcp.routers.konnectivity.service=konnectivity
    - traefik.tcp.routers.konnectivity.rule=HostSNI(`*`)
    - traefik.tcp.routers.konnectivity.entrypoints=konnectivity
    - traefik.tcp.services.konnectivity.loadbalancer.server.port=8132
  restart: on-failure

x-k0s-worker: &k0s-worker
  image: *k0s-image
  networks:
    - k0s-net
  depends_on:
    - k0s-controller-1
  command: [k0s, worker, --token-file, /run/secrets/k0sproject.io/tokens/worker]
  volumes:
    - /var/lib/k0s # this is where k0s stores its data
    - /var/log/pods # this is where k0s stores pod logs
    - /dev/kmsg:/dev/kmsg:ro # required by kubelets OOM watcher
    - k0s-worker-token:/run/secrets/k0sproject.io/tokens:ro
  tmpfs:
    - /run # this is where k0s stores runtime data
    - /tmp
  devices:
    - /dev/kmsg # required by kubelet's OOM watcher
  cap_add:
    - sys_admin
    - net_admin
    - sys_ptrace
    - sys_resource
    - syslog
  security_opt:
    - seccomp:unconfined # allow access to the session keyring
  restart: on-failure

networks:
  k0s-net:
    driver: bridge

services:
  k0s-lb:
    image: docker.io/library/traefik:v3.7.14
    container_name: k0s-lb
    hostname: k0s-lb
    networks:
      - k0s-net
    command:
      - --api.insecure=true
      - --providers.docker=true
      - --providers.docker.exposedbydefault=false
      - --entryPoints.kube-api.address=:6443
      - --entryPoints.k0s-api.address=:9443
      - --entryPoints.konnectivity.address=:8132
    ports:
      - 6443:6443 # publish the Kubernetes API server port
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

  k0s-controller-1:
    <<: *k0s-controller
    container_name: k0s-controller-1
    hostname: k0s-controller-1
    depends_on:
      - k0s-lb
    command: [k0s, controller]
    post_start:
      - command:
          - /bin/sh
          - -euc
          - |
            bootstrap() {
              # this works even if etcd isn't up

              find /run/secrets/k0sproject.io/controller-token ! -path /run/secrets/k0sproject.io/controller-token -prune -exec rm -rf {} +
              k0s token pre-shared --role controller \
                --cert /var/lib/k0s/pki/ca.crt \
                --url https://k0s-lb:9443 \
                --out /run/secrets/k0sproject.io/controller-token/

              find /run/secrets/k0sproject.io/worker-token ! -path /run/secrets/k0sproject.io/worker-token -prune -exec rm -rf {} +
              k0s token pre-shared --role worker \
                --cert /var/lib/k0s/pki/ca.crt \
                --url https://k0s-lb:6443 \
                --out /run/secrets/k0sproject.io/worker-token/

              mv /run/secrets/k0sproject.io/controller-token/*.yaml /var/lib/k0s/manifests/k0s-token-secrets/controller.yaml
              mv /run/secrets/k0sproject.io/controller-token/token_* /run/secrets/k0sproject.io/controller-token/controller
              mv /run/secrets/k0sproject.io/worker-token/*.yaml /var/lib/k0s/manifests/k0s-token-secrets/worker.yaml
              mv /run/secrets/k0sproject.io/worker-token/token_* /run/secrets/k0sproject.io/worker-token/worker
            }

            while [ ! -f /var/lib/k0s/pki/ca.crt ] || ! bootstrap; do
              sleep 1
            done
            sleep 10 # give this controller a bit of a head start

    volumes:
      - /var/lib/k0s # this is where k0s stores its data
      - k0s-token-secrets:/var/lib/k0s/manifests/k0s-token-secrets
      - k0s-controller-token:/run/secrets/k0sproject.io/controller-token
      - k0s-worker-token:/run/secrets/k0sproject.io/worker-token

  k0s-controller-2: &k0s-additional-controller
    <<: *k0s-controller
    container_name: k0s-controller-2
    hostname: k0s-controller-2
    depends_on:
      - k0s-controller-1
    command:
      [
        k0s,
        controller,
        --token-file,
        /run/secrets/k0sproject.io/tokens/controller,
      ]
    volumes:
      - /var/lib/k0s # this is where k0s stores its data
      - k0s-token-secrets:/var/lib/k0s/manifests/k0s-token-secrets:ro
      - k0s-controller-token:/run/secrets/k0sproject.io/tokens:ro

  k0s-controller-3:
    <<: *k0s-additional-controller
    container_name: k0s-controller-3
    hostname: k0s-controller-3

  k0s-worker-1:
    <<: *k0s-worker
    container_name: k0s-worker-1
    hostname: k0s-worker-1

  k0s-worker-2:
    <<: *k0s-worker
    container_name: k0s-worker-2
    hostname: k0s-worker-2

  k0s-worker-3:
    <<: *k0s-worker
    container_name: k0s-worker-3
    hostname: k0s-worker-3

volumes:
  k0s-token-secrets:
    driver: local
    driver_opts:
      type: tmpfs
      device: tmpfs
  k0s-controller-token:
    driver: local
    driver_opts:
      type: tmpfs
      device: tmpfs
  k0s-worker-token:
    driver: local
    driver_opts:
      type: tmpfs
      device: tmpfs

configs:
  k0s.yaml:
    content: |
      spec:
        api:
          externalAddress: k0s-lb

---

### Compose.Yaml (`docs/compose.yaml`)

services:
  k0s-controller:
    image: docker.io/k0sproject/k0s:{{{ k0s_docker_version }}}
    container_name: k0s-controller
    hostname: k0s-controller
    network_mode: bridge # other modes are unsupported
    ports:
      - 6443:6443 # publish the Kubernetes API server port
    volumes:
      - /var/lib/k0s # this is where k0s stores its data
      - /var/log/pods # this is where k0s stores pod logs
      - /dev/kmsg:/dev/kmsg:ro # required by kubelets OOM watcher
    tmpfs:
      - /run # this is where k0s stores runtime data
    devices:
      - /dev/kmsg # required by kubelet's OOM watcher
    cap_add:
      - sys_admin
      - net_admin
      - sys_ptrace
      - sys_resource
      - syslog
    security_opt:
      - seccomp:unconfined # allow access to the session keyring
    configs:
      - source: k0s.yaml
        target: /etc/k0s/k0s.yaml

configs:
  k0s.yaml:
    content: |
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: ClusterConfig
      metadata:
        name: k0s
      spec:
        storage:
          type: kine
      # Any additional configuration goes here ...

---

### Configuration Validation (`docs/configuration-validation.md`)

# Configuration validation

k0s command-line interface has the ability to validate config syntax:

```shell
k0s config validate --config path/to/config/file
```

`config validate` sub-command can validate the following:

1. YAML formatting
2. [SAN addresses](configuration.md#specapi)
3. [Network providers](configuration.md#specnetwork)
4. [Worker profiles](configuration.md#specworkerprofiles)

---

### Configuration (`docs/configuration.md`)

# Configuration options

## Using a configuration file

k0s can be installed without a config file. In that case the default configuration will be used. You can, though, create and run your own non-default configuration (used by the k0s controller nodes).

k0s supports providing only partial configurations. In case of partial configuration is provided, k0s will use the defaults for any missing values.

1. Generate a YAML config file that uses the default settings.

   ```shell
   mkdir -p /etc/k0s
   k0s config create > /etc/k0s/k0s.yaml
   ```

2. Modify the new YAML config file according to your needs, refer to [Configuration file reference](#configuration-file-reference) below. You can remove the default values if wanted as k0s supports partial configs too.

3. Install k0s with your new config file.

   ```shell
   sudo k0s install controller -c /etc/k0s/k0s.yaml
   ```

4. If you need to modify your existing configuration later on, you can change your config file also when k0s is running, but remember to restart k0s to apply your configuration changes.

   ```shell
   sudo k0s stop
   sudo k0s start
   ```

## Configuring k0s via k0sctl

k0sctl can deploy your configuration options at cluster creation time. Your
options should be placed in the `spec.k0s.config` section of the k0sctl
configuration file. See the section on how to install [k0s via
k0sctl][k0sctl-install] and the [k0sctl README] for more information.

[k0sctl-install]: k0sctl-install.md
[k0sctl README]: https://github.com/k0sproject/k0sctl/blob/main/README.md

## Configuration file reference

**CAUTION**: As many of the available options affect items deep in the stack, you should fully understand the correlation between the configuration file components and your specific environment before making any changes.

A YAML config file follows, with defaults as generated by the `k0s config create` command:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## `spec` Key Detail

### `spec.api`

| Element                      | Description                                                                                                                                                                                                                                                                   |
| ---------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `address`                    | IP Address used by cluster components to talk to the API server. Also serves as one of the addresses pushed on the k0s create service certificate on the API. Defaults to first non-local address found on the node.                                                          |
| `onlyBindToAddress`          | The API server binds to all interfaces by default. With this option set to `true`, the API server will only listen on the IP address configured by the `address` option (first non-local address by default). This can be necessary with multi-homed control plane nodes.     |
| `externalAddress`            | The load balancer address (for k0s controllers running behind a load balancer). Configures all cluster components to connect to this address and configures this address for use when joining new nodes to the cluster.                                                       |
| `sans`                       | List of additional addresses to push to API servers serving the certificate.                                                                                                                                                                                                  |
| `ca.expiresAfter`            | The expiration duration of the CA certificate (default: 87600h)                                                                                                                                                                                                               |
| `ca.certificatesExpireAfter` | The expiration duration of the server certificate (default: 8760h)                                                                                                                                                                                                            |
| `extraArgs`                  | Map of key-values (strings) for any extra arguments to pass down to Kubernetes API server process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support. (default: empty)                  |
| `rawArgs`                    | Slice of strings for any raw arguments to pass down to the kube-apiserver process. These are appended after `extraArgs`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty) |
| `port`¹                      | Custom port for the Kubernetes API server to listen on (default: 6443). When set to a privileged port (< 1024), k0s automatically grants the `CAP_NET_BIND_SERVICE` capability to the kube-apiserver process to allow binding to the port.                                    |
| `k0sApiPort`¹                | Custom port for k0s API server to listen on (default: 9443)                                                                                                                                                                                                                   |

¹ If `port` and `k0sApiPort` are used with the `externalAddress` element, the load balancer serving at `externalAddress` must listen on the same ports.

### `spec.storage`

| Element                           | Description                                                                                                                                                                                                                                        |
| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `type`                            | Type of the data store (valid values:`etcd` or `kine`). **Note**: Type `etcd` will cause k0s to create and manage an elastic etcd cluster within the controller nodes.                                                                             |
| `etcd.peerAddress`                | Node address used for etcd cluster peering.                                                                                                                                                                                                        |
| `etcd.extraArgs`                  | Map of key-values (strings) for any extra arguments to pass down to etcd process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support.                         |
| `etcd.rawArgs`                    | Slice of strings for any raw arguments to pass down to the etcd process. These are appeneded after `extraArg`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. |
| `etcd.ca.expiresAfter`            | The expiration duration of the CA certificate (default: 87600h)                                                                                                                                                                                    |
| `etcd.ca.certificatesExpireAfter` | The expiration duration of the server certificate (default: 8760h)                                                                                                                                                                                 |
| `etcd.externalCluster`            | Configuration when etcd is externally managed, i.e. running on dedicated nodes. See [`spec.storage.etcd.externalCluster`](#specstorageetcdexternalcluster)                                                                                         |
| `kine.dataSource`                 | [kine](https://github.com/k3s-io/kine) data source URL.                                                                                                                                                                                            |
| `kine.extraArgs`                  | Map of key-values (strings) for any extra arguments to pass down to kine process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support.                         |
| `kine.rawArgs`                    | Slice of strings for any raw arguments to pass down to the kine process. These are appended after `extraArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty)                                                 |

#### `spec.storage.etcd.externalCluster`

k0s can also work with an externally managed Etcd cluster. If this is configured, k0s will NOT set up etcd, it has to be managed manually.

| Element          | Description                                                                                                                                                 |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `endpoints`      | Array of Etcd endpoints to use.                                                                                                                             |
| `etcdPrefix`     | Prefix to use for this cluster. The same external Etcd cluster can be used for several k0s clusters, each prefixed with a unique prefix to store data with. |
| `caFile`         | CaFile is the host path to a file with the Etcd cluster CA certificate.                                                                                     |
| `clientCertFile` | ClientCertFile is the host path to a file with the TLS certificate for etcd client.                                                                         |
| `clientKeyFile`  | ClientKeyFile is the host path to a file with the TLS key for etcd client.                                                                                  |

### `spec.network`

| Element                | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `provider`             | Network provider (valid values: `calico`, `kuberouter`, or `custom`). For `custom`, you can push any network provider (default: `kuberouter`). Be aware that it is your responsibility to configure all the CNI-related setups, including the CNI provider itself and all necessary host levels setups (for example, CNI binaries). **Note:** Once you initialize the cluster with a network provider the only way to change providers is through a full cluster redeployment. The CNI files are applied as static manifests. |
| `podCIDR`              | Pod network CIDR to use in the cluster by the network provider. Defaults to `10.244.0.0/16`.                                                                                                                                                                                                                                                                                                                                                                                                                                  |
| `serviceCIDR`          | CNI-independent Network CIDR to use for cluster VIP services. Defaults to `10.96.0.0/12`.                                                                                                                                                                                                                                                                                                                                                                                                                                     |
| `primaryAddressFamily` | Defines the primary family for the cluster. Valid values are empty, `IPv4`, `IPv6`. If empty, K0s determines it based on `.spec.API.ExternalAddress`, if this isn't present it will use `.spec.API.Address.`. If both addresses are empty or the chosen address is a host name, defaults to `IPv4`.                                                                                                                                                                                                                           |
| `clusterDomain`        | Cluster domain to be passed to the [kubelet](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/#kubelet-config-k8s-io-v1beta1-KubeletConfiguration) and the CoreDNS configuration. Defaults to `cluster.local`.                                                                                                                                                                                                                                                                                          |

#### `spec.network.calico`

| Element                 | Description                                                                                                                                                                                                                                                                                                                                                                                               |
| ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `mode`                  | Indicates the Calico backend to use. Either `bird` or `vxlan` (default). The deprecated legacy value `ipip` is also accepted. Defaults to `vxlan` with encapsulated `overlay`.                                                                                                                                                                                                                            |
| `overlay`               | Overlay mode: `Always` (default), `CrossSubnet` or `Never` (requires `mode=vxlan` to disable Calico overlay-network, leaving pod CIDRs to be routed by the underlying network).                                                                                                                                                                                                                           |
| `vxlanPort`             | The UDP port for VXLAN (default: `4789`).                                                                                                                                                                                                                                                                                                                                                                 |
| `vxlanVNI`              | The virtual network ID for VXLAN (default: `4096`).                                                                                                                                                                                                                                                                                                                                                       |
| `mtu`                   | MTU for overlay network (default: `0`, which causes Calico to detect optimal MTU during bootstrap).                                                                                                                                                                                                                                                                                                       |
| `wireguard`             | Enable WireGuard-based encryption (default: `false`). Your host system must be WireGuard ready (refer to the [Calico documentation](https://docs.projectcalico.org/security/encrypt-cluster-pod-traffic) for details).                                                                                                                                                                                    |
| `flexVolumeDriverPath`  | The host path for Calico's flex-volume-driver(default: `/usr/libexec/k0s/kubelet-plugins/volume/exec/nodeagent~uds`). Change this path only if the default path cannot be used (refer to [Project Calico Issue #2712](https://github.com/projectcalico/calico/issues/2712) for details). Ideally, you will pair this option with a custom `volumePluginDir` in the profile you use for your worker nodes. |
| `ipAutodetectionMethod` | Used to force Calico to pick up the interface for pod network inter-node routing (default: `""`, meaning not set, so that Calico will instead use its defaults). For more information, refer to the [Calico documentation](https://docs.projectcalico.org/reference/node/configuration#ip-autodetection-methods).                                                                                         |
| `envVars`               | Map of key-values (strings) for any calico-node [environment variable](https://docs.projectcalico.org/reference/node/configuration#ip-autodetection-methods).                                                                                                                                                                                                                                             |

#### `spec.network.calico.envVars`

Environment variable's value must be string, e.g.:

```yaml
spec:
  network:
    provider: calico
    calico:
      envVars:
        TEST_BOOL_VAR: "true"
        TEST_INT_VAR: "42"
        TEST_STRING_VAR: test
```

K0s runs Calico with some predefined vars, which can be overwritten by setting new value in `spec.network.calico.envVars`:

```shell
CALICO_IPV4POOL_CIDR: "{{ spec.network.podCIDR }}"
CALICO_DISABLE_FILE_LOGGING: "true"
FELIX_DEFAULTENDPOINTTOHOSTACTION: "ACCEPT"
FELIX_LOGSEVERITYSCREEN: "info"
FELIX_HEALTHENABLED: "true"
FELIX_PROMETHEUSMETRICSENABLED: "true"
FELIX_FEATUREDETECTOVERRIDE: "ChecksumOffloadBroken=true"
```

`FELIX_FEATUREDETECTOVERRIDE: ChecksumOffloadBroken=true` disables VXLAN offloading because of [projectcalico/calico#4727](https://github.com/projectcalico/calico/issues/4727).

In single-stack mode, there are additional vars:

```shell
FELIX_IPV6SUPPORT: "false"
```

In dual-stack mode, there are additional vars:

```shell
CALICO_IPV6POOL_NAT_OUTGOING: "true"
FELIX_IPV6SUPPORT: "true"
IP6: "autodetect"
CALICO_IPV6POOL_CIDR: "{{ spec.network.dualStack.IPv6podCIDR }}"
```

#### `spec.network.kuberouter`

| Element          | Description                                                                                                                                                                                                                                                                                                                                                                                                   |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `autoMTU`        | Auto-detection of used MTU (default: `true`).                                                                                                                                                                                                                                                                                                                                                                 |
| `mtu`            | Override MTU setting, if `autoMTU` must be set to `false`.                                                                                                                                                                                                                                                                                                                                                    |
| `metricsPort`    | Kube-router metrics server port. Set to 0 to disable metrics (default: `8080`).                                                                                                                                                                                                                                                                                                                               |
| `peerRouterIPs`  | DEPRECATED: Use extraArgs with peerRouterIPs instead. Comma-separated list of [global peer addresses](https://github.com/cloudnativelabs/kube-router/blob/master/docs/bgp.md#global-external-bgp-peers).                                                                                                                                                                                                      |
| `peerRouterASNs` | DEPRECATED: Use extraArgs with peerRouterASNs instead. Comma-separated list of [global peer ASNs](https://github.com/cloudnativelabs/kube-router/blob/master/docs/bgp.md#global-external-bgp-peers).                                                                                                                                                                                                          |
| `hairpin`        | Configure [hairpin mode](https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#hairpin-mode). Supported modes are `Enabled`: enabled cluster wide, `Allowed`: must be allowed per service using annotations, `Disabled`: doesn't work at all (default: `Enabled`)                                                                                                                     |
| `hairpinMode`    | **Deprecated** Use `hairpin` instead. If both `hairpin` and `hairpinMode` are defined, this is ignored. If only hairpinMode is configured explicitly activates hairpinMode.                                                                                                                                                                                                                                   |
| `ipMasq`         | IP masquerade for traffic originating from the pod network, and destined outside of it (default: false)                                                                                                                                                                                                                                                                                                       |
| `extraArgs`      | Extra arguments to pass to Kube-router. `extraArgs` are recommended over `rawArgs` if the use case allows it. Can be also used to override any k0s managed arguments. For reference, see Kube-router [documentation](https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md#command-line-options). Any behavior triggered by these parameters is outside k0s support. (default: empty) |
| `rawArgs`        | Slice of strings for any raw arguments to pass down to the kube-router process. These are appended after `extraArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty)                                                                                                                                                                                                     |

**Note**: Kube-router allows many networking aspects to be configured per node, service, and pod (for more information, refer to the [Kube-router user guide](https://github.com/cloudnativelabs/kube-router/blob/master/docs/user-guide.md)).

#### `spec.network.kubeProxy`

| Element              | Description                                                                                                                                                                                                                                                               |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `disabled`           | Disable kube-proxy altogether (default: `false`).                                                                                                                                                                                                                         |
| `mode`               | Kube proxy operating mode, supported modes `iptables`, `ipvs`, `nftables`¹, `userspace` (default: `iptables`)                                                                                                                                                             |
| `iptables`           | Kube proxy iptables settings                                                                                                                                                                                                                                              |
| `ipvs`               | Kube proxy IPVS settings                                                                                                                                                                                                                                                  |
| `nftables`           | Kube proxy nftables settings                                                                                                                                                                                                                                              |
| `nodePortAddresses`  | Kube proxy [nodePortAddresses](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/)                                                                                                                                                             |
| `healthzBindAddress` | Kube proxy [healthzBindAddress](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) (default: 0.0.0.0:10256)                                                                                                                                   |
| `metricsBindAddress` | Kube proxy [metricsBindAddress](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/) (default: 0.0.0.0:10249)                                                                                                                                   |
| `extraArgs`          | Map of key-values (strings) for any extra arguments to pass down to kube-proxy process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support. (default: empty)                         |
| `rawArgs`            | Slice of strings for any raw arguments to pass down to the kube-proxy process. These are appended after `extraArgs`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty) |

¹ For nftables, the kube-proxy container's nftables version needs to be less
than or equal to the host OS's in order to avoid segmentation faults. For
example, a host OS with nftables v1.1.1 requires
`quay.io/k0sproject/kube-proxy:{# no auto-bumps-here #}{{{ "v1.33.2" }}}` or
lower.

Default kube-proxy iptables settings:

```yaml
iptables:
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
```

Default kube-proxy IPVS settings:

```yaml
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: false
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s
```

Default kube-proxy nftables settings:

```yaml
nftables:
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
```

#### `spec.network.nodeLocalLoadBalancing`

Configuration options related to k0s's [node-local load balancing] feature.

| Element      | Description                                                                                                                           |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| `enabled`    | Indicates if node-local load balancing should be used to access Kubernetes API servers from worker nodes. Default: `false`.           |
| `type`       | The type of the node-local load balancer to deploy on worker nodes. Default: `EnvoyProxy`. Supported values: `EnvoyProxy`, `Traefik`. |
| `envoyProxy` | Configuration options related to the "EnvoyProxy" type of load balancing.                                                             |
| `traefik`    | Configuration options related to the "Traefik" type of load balancing.                                                                |

[node-local load balancing]: nllb.md

##### `spec.network.nodeLocalLoadBalancing.envoyProxy`

Configuration options required for using Envoy as the backing implementation for
node-local load balancing.

**Note:** This type of load balancing is not supported on ARMv7, RISC-V and
Windows.

| Element                      | Description                                                                                                                              |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| `image`                      | The OCI image that's being used for the Envoy Pod.                                                                                       |
| `imagePullPolicy`            | The pull policy being used used for the Envoy Pod. Defaults to `spec.images.default_pull_policy` if omitted.                             |
| `apiServerBindPort`          | Port number on which to bind the Envoy load balancer for the Kubernetes API server to on a worker's loopback interface. Default: `7443`. |
| `konnectivityServerBindPort` | Port number on which to bind the Envoy load balancer for the konnectivity server to on a worker's loopback interface. Default: `7132`.   |

##### `spec.network.nodeLocalLoadBalancing.traefik`

Configuration options required for using Traefik as the backing implementation
for node-local load balancing.

| Element                      | Description                                                                                                                                |
| ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `image`                      | The OCI image that's being used for the Traefik Pod.                                                                                       |
| `imagePullPolicy`            | The pull policy being used used for the Traefik Pod. Defaults to `spec.images.default_pull_policy` if omitted.                             |
| `apiServerBindPort`          | Port number on which to bind the Traefik load balancer for the Kubernetes API server to on a worker's loopback interface. Default: `7443`. |
| `konnectivityServerBindPort` | Port number on which to bind the Traefik load balancer for the konnectivity server to on a worker's loopback interface. Default: `7132`.   |

##### `spec.network.controlPlaneLoadBalancing`

Configuration options related to k0s's [control plane load balancing] feature

| Element      | Description                                                                                                                    |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------ |
| `enabled`    | Indicates if control plane load balancing should be enabled. Default: `false`.                                                 |
| `type`       | The type of the control plane load balancer to deploy on controller nodes. Currently, the only supported type is `Keepalived`. |
| `keepalived` | Contains the keepalived configuration.                                                                                         |

[control plane load balancing]: cplb.md

##### `spec.network.controlPlaneLoadBalancing.Keepalived`

Configuration options related to keepalived in [control plane load balancing]

| Element                  | Description                                                                                                                     |
| ------------------------ | ------------------------------------------------------------------------------------------------------------------------------- |
| `vrrpInstances`          | Configuration options related to the VRRP. This is an array which allows to configure multiple virtual IPs.                     |
| `virtualServers`         | Configuration options related to load balancing. This is an array which allows to configure multiple load balancers.            |
| `userSpaceProxyBindPort` | The port the user space proxy will bind to. This port is for internal use only, but listens on every interface. Default: `6444` |
| `disableLoadBalancer`    | Disables the load balancer. Default: `false`                                                                                    |
| `configTemplateVRRP`     | Path to a custom Keepalived configuration template for VRRP. Default: empty .                                                   |
| `configTemplateVS`       | Path to a custom Keepalived configuration template for Virtual Servers. Default: empty                                          |

##### `spec.network.controlPlaneLoadBalancing.keepalived.vrrpInstances`

Configuration options required for using VRRP to configure VIPs in control plane load balancing.

| Element                 | Description                                                                                                                                                                                                                                                                                          |
| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `virtualIPs`            | List of virtual IP address used by the VRRP instance. Each virtual IP must be a CIDR as defined in RFC 4632 and RFC 4291.A list of the CIDRs handled by the VRRP instance.                                                                                                                           |
| `interface`             | The NIC used by the virtual router. K0s allows to use either a device name or a MAC address. If not specified, k0s will use the interface that owns the default route.                                                                                                                               |
| `virtualRouterID`       | The VRRP router ID. If not specified, k0s will automatically number the IDs for each VRRP instance, starting with 51. It must be in the range of 1-255, all the control plane nodes must use the same `virtualRouterID`. Other clusters in the same network must not use the same `virtualRouterID`. |
| `advertIntervalSeconds` | Advertisement interval in seconds. Defaults to 1 second.                                                                                                                                                                                                                                             |
| `authPass`              | The password for accessing VRRPD. This is not a security feature but a way to prevent accidental misconfigurations. It must be in the range of 1-8 characters                                                                                                                                        |
| `unicastPeers`          | A list of IP addresses to connect using unicast. If this field is specified, `unicastSourceIP` is mandatory, and this list must not contain the IP address specified in `unicastSourceIP`.                                                                                                           |
| `unicastSourceIP`       | The source IP address when using unicast. If `unicastPeers` isn't defined this field is ignored.                                                                                                                                                                                                     |
| `addressLabel`          | The label for the VRRP instance for IPv6 VIPs. This value is ignored for IPv4 VIPs. This is used to set the routing preference as per RFC 6724. If set to `0`, uses default value. Default: `10000`                                                                                                  |

##### `spec.network.controlPlaneLoadBalancing.keepalived.virtualServers`

Configuration options required for using VRRP to configure VIPs in control plane load balancing.

| Element                     | Description                                                                                                                                |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| `ipAddress`                 | The load balancer's listen address.                                                                                                        |
| `delayLoop`                 | Delay timer for check polling. Accepts microsecond precision. Further precision will be truncated without warnings. Defaults to `1m`.      |
| `lbAlgo`                    | Algorithm used by keepalived. Supported algorithms: `rr`, `wrr`, `lc`, `wlc`, `lblc`, `dh`, `sh`, `sed`, `nq`. Default: `rr`.              |
| `lbKind`                    | Kind of IPVS load balancer. Supported values: `NAT`, `DR`, `TUN`. Default: `DR`.                                                           |
| `persistenceTimeoutSeconds` | Timeout for persistent connections in seconds. Must be in the range of 1-2678400 (31 days). If not specified, defaults to 360 (6 minutes). |

### `spec.controllerManager`

| Element     | Description                                                                                                                                                                                                                                                                            |
| ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `extraArgs` | Map of key-values (strings) for any extra arguments you want to pass down to the Kubernetes controller manager process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support. (default: empty)      |
| `rawArgs`   | Slice of strings for any raw arguments to pass down to the kube-controller-manager process. These are appended after `extraArgs`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty) |

### `spec.scheduler`

| Element     | Description                                                                                                                                                                                                                                                                   |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `extraArgs` | Map of key-values (strings) for any extra arguments you want to pass down to Kubernetes scheduler process. `extraArgs` are recommended over `rawArgs` if the use case allows it. Any behavior triggered by these parameters is outside k0s support. (default: empty)          |
| `rawArgs`   | Slice of strings for any raw arguments to pass down to the kube-scheduler process. These are appended after `extraArgs`. If possible, it's recommended to use `extraArgs` over `rawArgs`. Any behavior triggered by these parameters is outside k0s support. (default: empty) |

### `spec.workerProfiles`

Worker profiles are used to manage worker-specific configuration in a
centralized manner. A ConfigMap is generated for each worker profile. Based on
the `--profile` argument given to the `k0s worker`, the configuration in the
corresponding ConfigMap is picked up during start-up.

The worker profiles are defined as an array. Each element has following
properties:

| Property | Description                                                                      |
| -------- | -------------------------------------------------------------------------------- |
| `name`   | String; name to use as profile selector for the worker process                   |
| `values` | Object; [Kubelet configuration][kubelet-config] overrides, see below for details |

#### `spec.workerProfiles[].values` (Kubelet configuration overrides)

The Kubelet configuration overrides of a profile override the defaults defined
by k0s.

Note that there are several fields that cannot be overridden:

- `clusterDNS`
- `clusterDomain`
- `apiVersion`
- `kind`
- `staticPodURL`

[kubelet-config]: https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/

#### Configuration examples

##### Custom volumePluginDir

```yaml
spec:
  workerProfiles:
    - name: custom-pluginDir
      values:
        volumePluginDir: /var/libexec/k0s/kubelet-plugins/volume/exec
```

##### Eviction Policy

```yaml
spec:
  workerProfiles:
    - name: custom-eviction
      values:
        evictionHard:
          memory.available: "500Mi"
          nodefs.available: "1Gi"
          imagefs.available: "100Gi"
        evictionMinimumReclaim:
          memory.available: "0Mi"
          nodefs.available: "500Mi"
          imagefs.available: "2Gi"
```

##### Unsafe Sysctls

```yaml
spec:
  workerProfiles:
    - name: custom-eviction
      values:
        allowedUnsafeSysctls:
          - fs.inotify.max_user_instances
```

### `spec.featureGates`

Available components are:

- `kube-apiserver`
- `kube-controller-manager`
- `kubelet`
- `kube-scheduler`
- `kube-proxy`

If `components` is omitted, propagates to all Kubernetes components.

Modifies `extraArgs`.

#### Examples

##### Generic feature gates example

```yaml
spec:
  featureGates:
    - name: feature-gate-0
      enabled: true
      components:
        [
          "kube-apiserver",
          "kube-controller-manager",
          "kubelet",
          "kube-scheduler",
        ]
    - name: feature-gate-1
      enabled: true
    - name: feature-gate-2
      enabled: false
```

##### Kubelet feature gates example

The below is an example of a k0s config with feature gates enabled:

```yaml
spec:
  featureGates:
    - name: DevicePlugins
      enabled: true
      components: ["kubelet"]
    - name: Accelerators
      enabled: true
      components: ["kubelet"]
    - name: AllowExtTrafficLocalEndpoints
      enabled: false
```

### `spec.images`

Nodes under the `images` key all have the same basic structure:

```yaml
spec:
  images:
    coredns:
      image: {{{ src_var('CoreDNSImage') }}}
      version: {{{ src_var('CoreDNSImageVersion') }}}
```

If you want the list of default images and their versions to be included, use `k0s config create --include-images`.

#### Available keys

- `spec.images.konnectivity`
- `spec.images.metricsserver`
- `spec.images.kubeproxy`
- `spec.images.coredns`
- `spec.images.pause`
- `spec.images.calico.cni`
- `spec.images.calico.node`
- `spec.images.calico.kubecontrollers`
- `spec.images.kuberouter.cni`
- `spec.images.kuberouter.cniInstaller`
- `spec.images.repository`¹
- `spec.images.default_pull_policy`²

¹ If `spec.images.repository` is set and not empty, every image will be pulled from there.

² If `spec.images.default_pull_policy` is set and not empty, it will be used as a pull policy for each bundled image.

#### Image example

{% set cali_ver = src_var('CalicoKubeControllersImageVersion') -%}
{% set metrics_ver = src_var('MetricsImageVersion') -%}

```yaml
images:
  repository: airgap-repo.local
  calico:
    kubecontrollers:
      image: repo.acme.corp/k0sproject/calico-kube-controllers
      version: {{{ cali_ver }}}
  metricsserver:
    image: repo.acme.corp/k0sproject/metrics-server
    version: {{{ metrics_ver }}}
```

In the runtime the image names are calculated as
`airgap-repo.local/k0sproject/calico-kube-controllers:{{{ cali_ver }}}` and
`airgap-repo.local/k0sproject/metrics-server:{{{ metrics_ver }}}`. This only
affects the image pull location, and thus omitting an image specification here
will not disable component deployment.

### `spec.extensions.helm`

`spec.extensions.helm` is the config file key in which you configure the list of [Helm](https://helm.sh) repositories and charts to deploy during cluster bootstrap (for more information, refer to [Helm Charts](helm-charts.md)).

### `spec.konnectivity`

The `spec.konnectivity` key is the config file key in which you configure Konnectivity-related settings.

- `agentPort` agent port to listen on (default 8132)
- `adminPort` admin port to listen on (default 8133)

### `spec.telemetry`

To improve the end-user experience k0s is configured by default to collect telemetry data from clusters and send it to the k0s development team. To disable the telemetry function, change the `enabled` setting to `false`.

The telemetry interval is ten minutes.

```yaml
spec:
  telemetry:
    enabled: true
```

### Component patches

!!! warning "Experimental feature"
     Patching is an experimental feature, which means the behavior of this feature might be unstable and may change without a major version bump.

Several k0s-managed components let you customize the Kubernetes resources they
generate before those resources are written and applied. This is an escape
hatch for adjustments k0s does not expose as dedicated config options. Patches
live under the component they belong to:

| Component      | Config key                        |
| -------------- | --------------------------------- |
| CoreDNS        | `spec.network.coreDNS.patches`    |
| kube-proxy     | `spec.network.kubeProxy.patches`  |
| kube-router    | `spec.network.kuberouter.patches` |
| Calico         | `spec.network.calico.patches`     |
| metrics-server | `spec.metricsServer.patches`      |

Each entry selects a target resource by `kind` and `name` (optionally narrowed
by `namespace`) and provides the patch `type` and `content`. The `content` may
be written as JSON or YAML.

| Element            | Description                                                                                             |
| ------------------ | ------------------------------------------------------------------------------------------------------- |
| `target.kind`      | The Kubernetes `kind` of the generated resource to patch (e.g. `Deployment`, `Service`, `ConfigMap`).   |
| `target.name`      | The `metadata.name` of the generated resource to patch.                                                 |
| `target.namespace` | Optional. Narrows the match to a single namespace.                                                      |
| `patch.type`       | One of `JSON` (RFC 6902 JSON Patch), `MergePatch` (RFC 7386 JSON Merge Patch) or `StrategicMergePatch`. |
| `patch.content`    | The patch body, as JSON or YAML.                                                                        |

Multiple patches matching the same resource are applied in the order listed.
Note that `StrategicMergePatch` patches are only supported for built-in
Kubernetes resource kinds.

#### Example

```yaml
spec:
  network:
    coreDNS:
      patches:
        - target:
            kind: Deployment
            name: coredns
            namespace: kube-system
          patch:
            type: StrategicMergePatch
            content: |
              spec:
                replicas: 3
                template:
                  spec:
                    containers:
                      - name: coredns
                        resources:
                          limits:
                            memory: 256Mi
```

## Disabling controller components

k0s allows to completely disable some of the system components. This allows
users to build a minimal Kubernetes control plane and use what ever components
they need to fulfill their need for the control plane. Disabling the system
components happens through a command line flag for the controller process:

```text
--disable-components strings                     disable components (valid items: applier-manager,autopilot,control-api,coredns,csr-approver,endpoint-reconciler,helm,konnectivity-server,kube-controller-manager,kube-proxy,kube-scheduler,metrics-server,network-provider,node-role,system-rbac,update-prober,windows-node)
```

If you use k0sctl, just add the flag when installing the cluster for the first
controller at `spec.hosts.installFlags` in the config file like e.g.:

```yaml
spec:
  hosts:
    - role: controller
      installFlags:
        - --disable-components=metrics-server
```

As seen from the component list, the only always-on component is the Kubernetes
API server, without that k0s serves no purpose.

## Kubelet root directory

Unlike vanilla Kubernetes, k0s by default deploys kubelet's root directory
inside `--data-dir`, which defaults to `/var/lib/k0s/` and kubelet ultimately
runs in `/var/lib/k0s/kubelet`. This can result in incompatibilities on external
software that mounts the kubelet working directory, such as CSI plugins.

This can be changed using the flag `--kubelet-root-dir`, for instance, to use
Kubernetes' default value it can be set up as:

```shell
sudo k0s install worker --token-file /var/lib/k0s/join-token --kubelet-root-dir=/var/lib/kubelet
```

Keep in mind that changing the flag in an existing node will not remove existing directories.
Equally importantly, some pods such as CSI plugins are likely to mount this directory,
having inconsistent values across nodes is very likely to cause problems on Deployments and
DaemonSets.

---

### Cplb (`docs/cplb.md`)

# Control plane load balancing

For clusters that don't have an [externally managed load balancer](high-availability.md#load-balancer) for the k0s
control plane, there is another option to get a highly available control plane called control plane load balancing (CPLB).

CPLB provides clusters a highly available VIP (virtual IP) and load balancing for **accessing the cluster externally**.
For internal traffic (nodes to control plane) k0s provides [NLLB](nllb.md). Both features are fully compatible and it's
recommended to use both together if you don't have an external load balancer.

Load balancing means that an IP address will forward the traffic to every control plane node, Virtual IPs mean that
this IP address will be present on at least one node at a time.

CPLB relies on [keepalived](https://www.keepalived.org) for highly available VIPs. Internally, Keepalived uses the
[VRRP protocol](https://datatracker.ietf.org/doc/html/rfc3768). Load Balancing can be done through either userspace
reverse proxy implemented in k0s (recommended for simplicity), or it can use Keepalived's virtual servers feature,
which ultimately relies on IPVS.

## Compatibility

CPLB depends on multiple technologies to work together as a whole, making it difficult to work on every single scenario.

### Single node

CPLB is incompatible with running as a [single node](k0s-single-node.md). This means k0s must not be started using the `--single` flag.

### Controller + worker

K0s only supports the userspace reverse proxy load balancer. Keepalived's VirtualServers are not supported with controller + worker.

Both Kube-Router and Calico managed by k0s are supported with the userspace reverse proxy load balancer, however, k0s creates iptables
rules in the control plane nodes which may be incompatible with custom CNI plugins.

### External address

If [`spec.api.externalAddress`](configuration.md#specapi) is defined, control
plane load balancing implicitly
[disables](configuration.md#disabling-controller-components) k0s's endpoint
reconciler component, just as if the `--disable-components=endpoint-reconciler`
flag had been specified.

### Node Local load balancing

CPLB is fully compatible with [NLLB](nllb.md), however NLLB is incompatible with [`spec.api.externalAddress`](configuration.md#specapi).

## Virtual IPs - High availability

### What is a VIP (virtual IP)

A virtual IP is an IP address that isn't tied to a single network interface,
instead it floats between multiple servers. This is a failover mechanism that
grants that there is always at least a functioning server and removes a single
point of failure.

### Configuring VIPs

CPLB relies internally on Keepalived's VRRP Instances. A VRRP Instance is a
server that will manage one or more VIPs. Most users will need exactly one
VRRP instance with exactly one VIP, however k0s allows multiple VRRP servers
with multiple VIPs for more advanced use cases such as network segmentation.

A virtualIP requires:

1. A user-defined CIDR address which must be routable in the network. For most installations, this will be in the same CIDR as the physical interface.
**WARNING:** K0s is not aware of external IP address management and the administrator is responsible for ensuring that IP addresses aren't colliding.
2. A user-defined password which should be unique for each cluster. This password is a mechanism to prevent accidental conflicts. It's not encrypted
and doesn't prevent malicious attacks in any way.
3. A virtual router ID, which defaults to 51. This virtual router ID **must be unique** in the broadcast domain.
4. A network interface, if not defined, k0s will chose the network interface that owns the default route.

Except the network interface, all the other fields must be equal on every control plane node.

This is a minimal example:

```yaml
spec:
  network:
    controlPlaneLoadBalancing:
      enabled: true
      type: Keepalived
      keepalived:
        vrrpInstances:
        - virtualIPs: ["<VIP address>/<netmask>"] # for instance ["172.16.0.100/16"]
          authPass: "<my password>"
```

By default, VRRP Instances use multicast as per [RFC 3768]. It's possible to configure VRRP
instances to use unicast:

```yaml
spec:
  network:
    controlPlaneLoadBalancing:
      enabled: true
      type: Keepalived
      keepalived:
        vrrpInstances:
        - virtualIPs: ["<VIP address>/<netmask>"] # for instance ["172.16.0.100/16"]
          authPass: "<my password>"
          unicastSourceIP: <ip address of this controller>
          unicastPeers: [<ip address of other controllers>, ...]
```

When using unicast, k0st does not attempt to detect `unicastSourceIP` and it must be defined explicitly and
`unicastPeers` must include the IP address of the other controllers' `unicastSourceIP`.

[RFC 3768]: https://datatracker.ietf.org/doc/html/rfc3768#section-5.2.2

#### IPv6 VIPs egress routing preference

In IPv6 there aren't primary addresses, instead the routing preference is determined by the operating system using
either IP labels or IP rules. K0s Virtual Addresses uses [RFC 6724] IP labels. By default, k0s sets the label to
10000 so that it still uses the main IP address as a source of egress connections.

This value may be replaced per vrrPInstance:

```yaml
spec:
  network:
    controlPlaneLoadBalancing:
      enabled: true
      type: Keepalived
      keepalived:
        vrrpInstances:
        - virtualIPs: ["<VIP address>/<netmask>"] # for instance ["2001:db8:2::1/64"]
          authPass: "<my password>"
          addressLabel: 30000
```

The label value can be verified using iproute2:

```shell
$ ip addrlabel | grep 2001:db8:2::1
prefix 2001:db8:2::1/128 label 30000
```

The prefix always uses netmask 128.

K0s doesn't attempt to modify labels that do not belong to the VIP.

[RFC 6724]: https://datatracker.ietf.org/doc/html/rfc6724

## Load Balancing

Currently k0s allows to chose one of two load balancing mechanism:

1. A userspace reverse proxy running in the k0s process. This is the default and recommended setting.
2. For users who may need extra performance or more flexible algorithms, k0s can use the keepalived virtual servers load balancer feature.

All control plane nodes must use the same load balancing mechanism. Different Load balancing mechanism
is not supported and has undefined behavior.

### Load Balancing - Userspace Reverse Proxy

This is the default behavior, in order to enable it simple configure a VIP
using a VRRP instance.

```yaml
spec:
  network:
    controlPlaneLoadBalancing:
      enabled: true
      type: Keepalived
      keepalived:
        vrrpInstances:
        - virtualIPs: ["<VIP address>/<netmask>"] # for instance ["172.16.0.100/16"]
          authPass: "<my password>"
```

### Keepalived Virtual Servers Load Balancing

The Keepalived virtual servers Load Balancing is more performant than the userspace reverse proxy load balancer. However, it's
 not recommended because it has some drawbacks:

1. It's incompatible with controller+worker.
2. May not work on every infrastructure.
3. Troubleshooting is significantly more complex.
4. When there is more than one VRRPInstance, we must do the load balancing in all the servers
which in some rare circumstances can provoke temporary routing loops.

```yaml
spec:
  network:
    controlPlaneLoadBalancing:
      enabled: true
      type: Keepalived
      keepalived:
        vrrpInstances:
        - virtualIPs: ["<VIP address>/<netmask>"] # for instance ["172.16.0.100/16"]
          authPass: "<my password>"
        virtualServers:
        - ipAddress: "<VIP address without netmask>" # for instance 172.16.0.100
```

## Full example using `k0sctl`

The following example shows a full `k0sctl` configuration file featuring three
controllers and three workers with control plane load balancing enabled.

```
/* Detailed source-code truncated for AI context efficiency. */
```

Save the above configuration into a file called `k0sctl.yaml` and apply it in
order to bootstrap the cluster:

```
/* Detailed source-code truncated for AI context efficiency. */
```

The cluster with the two nodes should be available by now. Setup the kubeconfig
file in order to interact with it:

```console
k0sctl kubeconfig > k0s-kubeconfig
export KUBECONFIG=$(pwd)/k0s-kubeconfig
```

All three worker nodes are ready:

```console
$ kubectl get nodes
NAME                   STATUS   ROLES           AGE     VERSION
worker-0.k0s.lab       Ready    <none>          8m51s   {{{ k8s_version }}}+k0s
worker-1.k0s.lab       Ready    <none>          8m51s   {{{ k8s_version }}}+k0s
worker-2.k0s.lab       Ready    <none>          8m51s   {{{ k8s_version }}}+k0s
```

Only one controller has the VIP:

```console
$ for i in controller-{0..2} ; do echo $i ; ssh $i -- ip -4 --oneline addr show | grep eth0; done
controller-0
2: eth0    inet 192.168.122.37/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0\       valid_lft 2381sec preferred_lft 2381sec
2: eth0    inet 192.168.122.200/24 scope global secondary eth0\       valid_lft forever preferred_lft forever
controller-1
2: eth0    inet 192.168.122.185/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0\       valid_lft 2390sec preferred_lft 2390sec
controller-2
2: eth0    inet 192.168.122.87/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0\       valid_lft 2399sec preferred_lft 2399sec
```

The cluster is using control plane load balancing and is able to tolerate the
outage of one controller node. Shutdown the first controller to simulate a
failure condition:

```console
$ ssh controller-0 'sudo poweroff'
Connection to 192.168.122.37 closed by remote host.
```

Control plane load balancing provides high availability, the VIP will have moved to a different node:

```console
$ for i in controller-{1..2} ; do echo $i ; ssh $i -- ip -4 --oneline addr show | grep eth0; done
controller-1
2: eth0    inet 192.168.122.185/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0\       valid_lft 2173sec preferred_lft 2173sec
2: eth0    inet 192.168.122.200/24 scope global secondary eth0\       valid_lft forever preferred_lft forever
controller-2
2: eth0    inet 192.168.122.87/24 brd 192.168.122.255 scope global dynamic noprefixroute eth0\       valid_lft 2182sec preferred_lft 2182sec
````

And the cluster will be working normally:

```console
$ kubectl get nodes
NAME                   STATUS   ROLES           AGE     VERSION
worker-0.k0s.lab       Ready    <none>          8m51s   {{{ k8s_version }}}+k0s
worker-1.k0s.lab       Ready    <none>          8m51s   {{{ k8s_version }}}+k0s
worker-2.k0s.lab       Ready    <none>          8m51s   {{{ k8s_version }}}+k0s
```

## Troubleshooting

Although Virtual IP addresses and load balancing work together and are closely
related, these are two independent processes and must be troubleshooting as two
independent features.

### Troubleshooting Virtual IPs

The first thing to check is that the VIP is present in exactly one node at a time,
for instance if a cluster has an `172.17.0.102/16` address and the interface is `eth0`,
the expected output is similar to:

```console
controller0:/# ip a s eth0
53: eth0@if54: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet 172.17.0.102/16 scope global secondary eth0
       valid_lft forever preferred_lft forever
```

```console
controller1:/# ip a s eth0
55: eth0@if56: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.3/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever
```

If the `virtualServers` feature is used, there must be a dummy interface on the
node called `dummyvip0` which has the VIP, but with a `/32` bits network mask.
This isn't the VIP and has to be there even if the VIP is held by another node.

```console
controller0:/# ip a s dummyvip0 | grep 172.17.0.102
    inet 172.17.0.102/32 scope global dummyvip0
```

```console
controller1:/# ip a s dummyvip0 | grep 172.17.0.102
    inet 172.17.0.102/32 scope global dummyvip0
```

If this isn't present in the nodes, keepalived logs can be seen in the k0s-logs, and
can be filtered with `component=keepalived`.

```console
controller0:/# journalctl -u k0scontroller | grep component=keepalived
time="2024-11-19 12:56:11" level=info msg="Starting to supervise" component=keepalived
time="2024-11-19 12:56:11" level=info msg="Started successfully, go nuts pid 409" component=keepalived
time="2024-11-19 12:56:11" level=info msg="Tue Nov 19 12:56:11 2024: Starting Keepalived v{{{ build_var('keepalived_version') }}}" component=keepalived stream=stderr
[...]
```

The Keepalived configuration is stored in a file called `keepalived.conf` in the
k0s run directory, by default `/run/k0s/keepalived.conf`, in this file there
should be a `vrrp_instance` section for each `vrrpInstance`.

Finally, k0s should have two keepalived processes running.

### Troubleshooting the Load Balancer's Endpoint List

Both the user space reverse proxy load balancer and Keepalived's virtual servers need an endpoint list to
do the load balancing. They share a component called `cplb-reconciler` which responsible for setting the
load balancer's endpoint list. This component monitors constantly the endpoint `kubernetes` in the
`default`namespace:

```console
controller0:/# kubectl get ep kubernetes -n default
NAME         ENDPOINTS                                         AGE
kubernetes   172.17.0.6:6443,172.17.0.7:6443,172.17.0.8:6443   9m14s
```

You can see the `cplb-reconciler` updates by running:

```console
controller0:/# journalctl -u k0scontroller | grep component=cplb-reconciler
time="2024-11-20 20:29:28" level=error msg="Failed to watch API server endpoints, last observed version is \"\", starting over in 10s ..." component=cplb-reconciler error="Get \"https://172.17.0.6:6443/api/v1/namespaces/default/endpoints?fieldSelector=metadata.name%3Dkubernetes&timeout=30s&timeoutSeconds=30\": dial tcp 172.17.0.6:6443: connect: connection refused"
time="2024-11-20 20:29:38" level=info msg="Updated the list of IPs: [172.17.0.6]" component=cplb-reconciler
time="2024-11-20 20:29:55" level=info msg="Updated the list of IPs: [172.17.0.6 172.17.0.7]" component=cplb-reconciler
time="2024-11-20 20:29:59" level=info msg="Updated the list of IPs: [172.17.0.6 172.17.0.7 172.17.0.8]" component=cplb-reconciler
```

### Troubleshooting the user space reverse proxy load balancer

The user space reverse proxy load balancer runs in the k0s process. It listens a separate socket, by default on port 6444:

```console
controller0:/# netstat -tlpn | grep 6444
tcp        0      0 :::6444                 :::*                    LISTEN      345/k0s
```

Then the requests to the VIP on the apiserver port are forwarded to this socket using one iptables rule:

```console
-A PREROUTING -d <VIP>/32 -p tcp -m tcp --dport <apiserver port> -j REDIRECT --to-ports <userspace proxy port>
```

A real life example of a cluster using the VIP `17.177.0.102` looks like:

```console
controller0:/# /var/lib/k0s/bin/iptables-save | grep 6444
-A PREROUTING -d 172.17.0.102/32 -p tcp -m tcp --dport 6443 -j REDIRECT --to-ports 6444
```

Keep in mind that clusters using IPv6 as a primary address family, should use ip6tables or ip6tables-save.

It the load balancer is not load balancing for whatever reason, you can establish connections to it directly. A good way to see if it's actually load balancing is checking the serving certificate:

```console
controller0:/# ip -o addr s eth0
43: eth0    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0\       valid_lft forever preferred_lft forever
43: eth0    inet 172.17.0.102/16 scope global secondary eth0\       valid_lft forever preferred_lft forever

controller0:/# openssl s_client -connect 172.17.0.102:6444  </dev/null 2>/dev/null | openssl x509 -noout -fingerprint
SHA1 Fingerprint=B7:90:E6:E4:E1:EE:5B:19:72:99:02:28:54:36:D9:84:D5:39:67:8B
controller0:/# openssl s_client -connect 172.17.0.102:6444  </dev/null 2>/dev/null | openssl x509 -noout -fingerprint
SHA1 Fingerprint=89:94:5C:E5:50:7E:40:B2:E5:20:E7:70:E8:58:91:ED:63:B0:EC:65
controller0:/# openssl s_client -connect 172.17.0.102:6444  </dev/null 2>/dev/null | openssl x509 -noout -fingerprint
SHA1 Fingerprint=49:0D:79:FD:79:6F:A0:E4:9D:BA:A1:65:9C:C5:54:CF:E5:20:BF:A8
controller0:/# openssl s_client -connect 172.17.0.102:6444  </dev/null 2>/dev/null | openssl x509 -noout -fingerprint
SHA1 Fingerprint=B7:90:E6:E4:E1:EE:5B:19:72:99:02:28:54:36:D9:84:D5:39:67:8B
```

Note: You can't query the port 6444 on the localhost address, there is an iptables conflict. You are expected to
be able to reach the port 6443 on any address and the port 6444 on any address except localhost.

### Troubleshooting Keepalived Virtual Servers

You can verify the Keepalived logs and configuration file using the steps described in the section
[troubleshooting virtual IPs](#troubleshooting-virtual-ips) above.

When virtual servers are enabled K0s generates two additional files:

* `keepalived-virtualservers-generated.conf`: This file contains the list of control plane nodes that should be balanced to.
* `keepalived-virtualservers-consumed.conf`: This is a symbolic link which points to `keepalived-virtualservers-generated.conf`
if the Keepalived VRRP instance's current state is `master` or to `/dev/null` if it's `backup`. This file is only generated if
there is exactly one VRRP instance.

Additionally, you can check the actual IPVS configuration using `ipvsadm`:

```console
controller0:/# ipvsadm --save -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.122.200:6443 rr persistent 360
  -> 192.168.122.185:6443              Route   1      0          0
  -> 192.168.122.87:6443               Route   1      0          0
  -> 192.168.122.122:6443              Route   1      0          0
  ```

  In this example `192.168.122.200` is the virtual IP, and `192.168.122.185`, `192.168.122.87`
  and `192.168.122.122` are the control plane nodes.

If there is only one VRRP instance, only the current master should be load balancing.

## Custom Keepalived Templates

**Warning:** Any customization to Keepalived templates is outside the scope of k0s support.
Template variables are not considered a stable API and templates can break during upgrades. Use at your own risk.

K0s allows you to customize the Keepalived configuration by providing custom Go templates for advanced users
who need an extra layer of customization. A minimal example would be:

```yaml
spec:
  network:
    controlPlaneLoadBalancing:
      enabled: true
      type: Keepalived
      keepalived:
        vrrpInstances:
        - virtualIPs: ["<VIP address>/<netmask>"]
          authPass: "<my password>"
        configTemplateVRRP: /path/to/custom-vrrp-template.conf
        configTemplateVS: /path/to/custom-virtualservers-template.conf
```

It is recommended to create custom templates based on the default templates, these can be accessed using the following commands:

```shell
# View the default VRRP template
k0s keepalived-config vrrp

# View the default Virtual Servers template
k0s keepalived-config virtualservers
```

The templates are only read during the k0s bootstrap. If a template is changed it's required to restart k0s.

### Keepalived template variables

#### VRRP Variables (configTemplateVRRP)

| Variable                                 | Description                                              |
|------------------------------------------|----------------------------------------------------------|
| `.IPVSLoadBalancer`                      | Determines whether IPVS load balancer is enabled or not. |
| `.K0sBin`                                | K0s binary path.                                         |
| `.RunDir`                                | K0s run directory.                                       |
| `.VRRPInstances`                         | Struct holding all the VRRP instances.                   |
| `.VRRPInstances[].AdvertIntervalSeconds` | Advert interval of the VRRP instance.                    |
| `.VRRPInstances[].AuthPass`              | Password for accessing VRRPD.                            |
| `.VRRPInstances[].Interface`             | Network interface used by the virtual router.            |
| `.VRRPInstances[].UnicastPeers`          | List of unicast peer IP addresses.                       |
| `.VRRPInstances[].UnicastSourceIP`       | Source IP address for unicast communication.             |
| `.VRRPInstances[].VirtualIPs`            | List of virtual IP addresses with CIDR notation.         |
| `.VRRPInstances[].VirtualRouterID`       | VRRP router ID.                                          |

#### Virtual Servers Variables (configTemplateVS)

| Variable                                      | Description                                             |
|-----------------------------------------------|---------------------------------------------------------|
| `.APIServerPort`                              | Kubernetes API server port.                             |
| `.RealServers`                                | List of real server IP addresses (control plane nodes). |
| `.VirtualServers`                             | Array of virtual server configurations.                 |
| `.VirtualServers[].DelayLoop`                 | Delay timer for check polling.                          |
| `.VirtualServers[].IPAddress`                 | Virtual IP address used by the virtual server.          |
| `.VirtualServers[].LBAlgo`                    | Load balancing algorithm.                               |
| `.VirtualServers[].LBKind`                    | Load balancing forwarding method.                       |
| `.VirtualServers[].PersistenceTimeoutSeconds` | Timeout value for persistent connections in seconds.    |

### Important Considerations

* **No Support**: Custom template configurations are not officially supported. If you encounter issues with custom templates, you may be asked to reproduce the issue with default templates.
* **Updates**: When upgrading k0s, review the default templates for any changes that may need to be incorporated into your custom templates.
* **Testing**: Always test custom templates thoroughly in a non-production environment before deploying to production.
* **Syntax Validation**: k0s performs minimal validation of custom templates. Invalid Keepalived configuration will cause the CPLB feature to fail.

---

### Custom Ca (`docs/custom-ca.md`)

# Install using custom CA certificates and SA key pair

k0s generates all needed certificates automatically in the `<data-dir>/pki` directory (`/var/lib/k0s/pki`, by default).

But sometimes there is a need to have the CA certificates and SA key pair in advance.
To make it work, just put files to the `<data-dir>/pki` and `<data-dir>/pki/etcd`:

```shell
export LIFETIME=365
mkdir -p /var/lib/k0s/pki/etcd
cd /var/lib/k0s/pki
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days $LIFETIME -out ca.crt -subj "/CN=Custom CA"
openssl genrsa -out sa.key 2048
openssl rsa -in sa.key -outform PEM -pubout -out sa.pub
cd ./etcd
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days $LIFETIME -out ca.crt -subj "/CN=Custom CA"
```

Then you can [install k0s as usual](./install.md).

## Pre-generated tokens

It's possible to get join in advance without having a running cluster.

```shell
k0s token pre-shared --role worker --cert /var/lib/k0s/pki/ca.crt --url https://<controller-ip>:6443/
```

The command above generates a join token and a Secret. A Secret should be deployed to the cluster to authorize the token.
For example, you can put the Secret under the [manifest](manifests.md) directory and it will be deployed automatically.

Please note that if you are generating a join token for a controller, the port number needs to be 9443 instead of 6443.
Controller bootstrapping requires talking to the k0s API server instead of the Kubernetes API server.
Here's an example of a command for pre-generating a token for a controller.

```shell
k0s token pre-shared --role controller --cert /var/lib/k0s/pki/ca.crt --url https://<controller-ip>:9443/
```

## See also

- [Certificate Authorities](troubleshooting/certificate-authorities.md)

---

### Directories (`docs/directories.md`)

# k0s directories

This page describes the directories k0s reads and writes on the host, what they
are used for, and the permissions k0s expects on them.

**Please note that the directory layouts described here are considered an
implementation detail and may change without further notice, even between patch
releases, unless explicitly stated otherwise, e.g., by other parts of the
documentation referencing those paths or files within them.**

The two parent directories are configurable via CLI flags:

- `--data-dir` (default `/var/lib/k0s` on Linux, `C:\var\lib\k0s` on Windows)<br>
  persistent state
- `--run-dir` (default `/run/k0s` on Linux)<br>
  runtime state such as PID files and unix sockets

Additionally, on k0s nodes with workloads enabled, you can set the following directories:

- `--kubelet-root-dir` (default `<data-dir>/kubelet`)<br>
  Kubelet root directory

## Data directory layout

Inside `<data-dir>` (default `/var/lib/k0s`):

| Path                           | Purpose                                                                                | Mode   |
| ------------------------------ | -------------------------------------------------------------------------------------- | ------ |
| `<data-dir>/`                  | Top-level data directory.                                                              | `0755` |
| `<data-dir>/bin/`              | Embedded binaries (kubelet, etcd, containerd, …) extracted on first start.             | `0755` |
| `<data-dir>/pki/`              | Cluster CA and component certificates managed by k0s.                                  | `0751` |
| `<data-dir>/pki/etcd/`         | etcd-specific certificates.                                                            | `0711` |
| `<data-dir>/etcd/`             | etcd database files if k0s is configured to use the embedded etcd.                     | `0700` |
| `<data-dir>/manifests/`        | Manifest Deployer drop-in directory. Files placed here are applied to the cluster.     | `0755` |
| `<data-dir>/images/`           | Image bundles imported into the managed containerd instance                            | `0755` |
| `<data-dir>/kine/`             | Kine database files when k0s is configured to use Kine instead of etcd.                | `0750` |

Individual certificate files inside `pki/` are written with mode `0644`
(public certs) or `0640` (private keys), and admin/kubelet config files are
written with mode `0600`.

## Run directory layout

Inside `<run-dir>` (default `/run/k0s`):

| Path                    | Purpose                                                                 | Mode            |
| ----------------------- | ----------------------------------------------------------------------- | --------------- |
| `<run-dir>/`            | Top-level run directory.                                                | `0755`          |
| `<run-dir>/*.pid`       | PID files for supervised processes (kubelet, etcd, containerd, …).      | `0644`          |
| `<run-dir>/status.sock` | Unix socket exposing local status to `k0s status` and similar commands. | (process umask) |

The run directory is intentionally outside the data directory so that on
systems where `/run` is a tmpfs, runtime state is cleared on reboot while
persistent state under `<data-dir>` survives.

## Notes

- All paths above are derived from the same `--data-dir` / `--run-dir` flags
  passed to `k0s` and propagated to its supervised components, so changing
  the flags moves the whole tree consistently.
- The CSI guide in the [Storage](storage.md) section documents the kubelet root
  path caveat, since the default differs from upstream Kubernetes
  (`<data-dir>/kubelet` rather than `/var/lib/kubelet`).
- Backup/restore covers `pki/`, `manifests/`, and `images/`, along with the etcd
  or k0s database files, ik k0s is configured to use them. See
  [Backup/Restore](backup.md) for details.

---

### Dual Stack (`docs/dual-stack.md`)

# IPv4/IPv6 dual-stack networking

Enabling dual-stack networking in k0s allows your cluster to handle both IPv4 and
IPv6 addresses. Follow the configuration examples below to set up dual-stack mode.

## Enabling dual-stack using the default CNI (Kube-router)

In order to enable dual-stack networking using the default CNI provider, use the
following example configuration:

```yaml
spec:
  network:
    # Kube-router is the default CNI provider
    # provider: kube-router
    podCIDR: 10.244.0.0/16
    serviceCIDR: 10.96.0.0/12
    dualStack:
      enabled: true
      IPv6podCIDR: fd00::/108
      IPv6serviceCIDR: fd01::/108
```

This configuration will set up all Kubernetes components and Kube-router
accordingly for dual-stack networking.

### Configuring the node CIDR mask size

By default, k0s uses a `/117` node CIDR mask size for IPv6, which provides 2048
IP addresses per node and a `/24` for IPv4 which provides 256 addresses per node.

For IPv6, using the example configuration `IPv6PodCIDR: fd00::/108`, there
are 9 bits available for node allocation (117 - 108 = 9) and 11 bits available for
pod allocation (128 - 117 = 11). This allows for 512 nodes per cluster and 2048
IPs per node.

For IPv4, using the default `PodCIDR: 10.244.0.0/16`, there are 8 bits available
for node allocation and 8 bits available for pod allocation. This allows for 256
nodes per cluster and 256 IPs per node.
per cluster and 256 IPs per node.

You can customize the node CIDR mask size using the controller manager's extra arguments:

```yaml
spec:
  controllerManager:
    extraArgs:
      node-cidr-mask-size-ipv6: "120"
      node-cidr-mask-size-ipv4: "21"
  network:
    dualStack:
      enabled: true
      IPv6podCIDR: fd00::/108
      IPv6serviceCIDR: fd01::/108
```

## Using Calico as the CNI provider

Calico does not support IPv6 tunneling in the default `vxlan` mode, so if you
prefer to use Calico as your CNI provider, make sure to select `bird` mode. Use
the following example configuration:

```yaml
spec:
  network:
    provider: calico
    calico:
      mode: bird
    podCIDR: 10.244.0.0/16
    serviceCIDR: 10.96.0.0/12
    dualStack:
      enabled: true
      IPv6podCIDR: fd00::/108
      IPv6serviceCIDR: fd01::/108
```

## Specifying the default IP family

In Kubernetes dual stack clusters, by default all the services are single stack,
including `kubernetes.default.svc`, which is used to communicate with the
Kubernetes API servers.

This is specially important when specifying explicitly
`spec.api.externalAddress` or `spec.api.address`.

To explicitly define the family which will be used by default use the following
configuration:

```yaml
spec:
  network:
    # primaryAddressFamily is optional
    primaryAddressFamily: <IPv4|IPv6>
```

If not defined explicitly, k0s will determine it based on `spec.api.externalAddress`,
if this field is not defined, k0s will use `spec.api.address`. If the field used is
a host name or both are empty, k0s will use IPv4.

## Custom CNI providers

While the dual-stack configuration section configures all components managed by
k0s for dual-stack operation, the custom CNI provider must also be configured
accordingly. Refer to the documentation for your specific CNI provider to ensure
a proper dual-stack setup that matches that of k0s.

## Additional Resources

For more detailed information and troubleshooting, refer to the following resources:

* <https://kubernetes.io/docs/concepts/services-networking/dual-stack/>
* <https://kubernetes.io/docs/tasks/network/validate-dual-stack/>
* <https://www.tigera.io/blog/dual-stack-operation-with-calico-on-kubernetes/>
* <https://docs.tigera.io/calico/3.27/networking/ipam/ipv6#enable-dual-stack>

---

### Dynamic Configuration (`docs/dynamic-configuration.md`)

# Dynamic configuration

k0s comes with the option to enable dynamic configuration for cluster level components. This covers all the components other than etcd (or SQLite) and the Kubernetes API server. This option enables k0s configuration directly via Kubernetes API as opposed to using a configuration file for all cluster configuration.

This feature has to be enabled on every controller in the cluster using the `--enable-dynamic-config` flag in `k0s controller` or `k0s install controller` commands. Mixing controllers with and without dynamic configuration enabled in the same cluster will cause a conflict.

## Dynamic vs. static configuration

The existing and enabled-by-default method is what we call static configuration. That's the way where the k0s process reads the config from the given YAML file (or uses the default config if no config is given by user) and configures every component accordingly. This means that for any configuration change the cluster admin has to restart all controllers on the cluster and have matching configs on each controller node.

In dynamic configuration mode, the first controller to start when the cluster is created uses the provided YAML as the bootstrap configuration and stores it in the Kubernetes API. All other controllers retrieve the existing configuration from the Kubernetes API and use it to configure all components except etcd and the Kubernetes API server. After the initial cluster bootstrap, the configuration object in the Kubernetes API becomes the source of truth for all controllers.

## Cluster configuration vs. controller node configuration

In the [k0s configuration options](configuration.md) there are some options that are cluster-wide and some that are specific to each controller node in the cluster. The following list outlines which options are controller node specific and have to be configured only via the local file:

- `spec.api` - these options configure how the local Kubernetes API server is setup
- `spec.storage` - these options configure how the local storage (etcd or SQLite) is setup
- `spec.network.controlPlaneLoadBalancing` - these options configure how to setup [Control Plane Load Balancing](cplb.md).

In case of an HA control plane, all the controllers will need this part of the configuration as otherwise they will not be able to get the storage and Kubernetes API server running.

## Configuration location

The cluster wide configuration is stored in the Kubernetes API as a custom resource called `clusterconfig`. There's currently only one instance named `k0s`. You can edit the configuration with what ever means possible, for example with:

```shell
k0s config edit
```

This will open the configuration object for editing in your system's default editor.

## Configuration reconciliation

The dynamic configuration uses the typical operator pattern for operation. k0s controller will detect when the object changes and will reconcile the configuration changes to be reflected to how different components are configured. So say you want to change the MTU setting for Kube-router CNI networking you'd change the config to contain e.g.:

```yaml
    kuberouter:
      mtu: 1350
      autoMTU: false
```

This will change the Kube-router related ConfigMap and thus make Kube-router to use different MTU settings for new pods.

## Configuration options

The configuration object is a 1-to-1 mapping with the existing [configuration YAML](configuration.md). All the configuration options EXCEPT options under `spec.api` and `spec.storage` are dynamically reconciled.

As with any Kubernetes cluster there are certain things that just cannot be changed on-the-fly, this is the list of non-changeable options:


- `network.podCIDR`
- `network.serviceCIDR`
- `network.provider`
- `network.controlPlaneLoadBalancing`
- `network.primaryAddressFamily`

During the manual installation of control plane nodes with `k0s install`, all these
non-changeable options must be defined in the configuration file. This is necessary
because these fields can be used before the dynamic configuration reconciler is
initialized. Both k0sctl and k0smotron handle this without user intervention.

## Configuration status

The dynamic configuration reconciler operator will write status events for all the changes it detects. To see all dynamic config related events, use:

```shell
k0s config status
```

```shell
LAST SEEN   TYPE      REASON                OBJECT              MESSAGE
64s         Warning   FailedReconciling     clusterconfig/k0s   failed to validate config: [invalid pod CIDR invalid ip address]
59s         Normal    SuccessfulReconcile   clusterconfig/k0s   Successfully reconciler cluster config
69s         Warning   FailedReconciling     clusterconfig/k0s   cannot change CNI provider from kuberouter to calico
```

---

### Environment Variables (`docs/environment-variables.md`)

# Environment variables

`k0s install` does not support environment variables.

Setting environment variables for components used by k0s depends on the used init system. The environment variables set in `k0scontroller` or `k0sworker` service will be inherited by k0s components, such as `etcd`, `containerd`, `konnectivity`, etc.

Component specific environment variables can be set in `k0scontroller` or `k0sworker` service. For example: for `CONTAINERD_HTTPS_PROXY`, the prefix `CONTAINERD_` will be stripped and converted to `HTTPS_PROXY` in the `containerd` process.

For those components having environment prefix convention such as `ETCD_xxx`,
they are handled specially, i.e. the prefix will not be stripped. For example,
`ETCD_MAX_WALS` will still be `ETCD_MAX_WALS` in etcd process.

The proxy environment variables `HTTP_PROXY`, `HTTPS_PROXY`, `NO_PROXY` are
always overridden by component specific environment variables, so
`ETCD_HTTPS_PROXY` will still be converted to `HTTPS_PROXY` in etcd process.

## systemd

Create a drop-in directory and add config file with a desired environment variable:

```shell
mkdir -p /etc/systemd/system/k0scontroller.service.d
tee -a /etc/systemd/system/k0scontroller.service.d/http-proxy.conf <<EOT
[Service]
Environment=HTTP_PROXY=192.168.33.10:3128
EOT
```

## OpenRC

Export the desired environment variable overriding service configuration in the
`/etc/conf.d` directory:

```shell
echo 'export HTTP_PROXY="192.168.33.10:3128"' > /etc/conf.d/k0scontroller
```

---

### Experimental Windows (`docs/experimental-windows.md`)

# Run k0s worker nodes in Windows

**IMPORTANT**: Windows support for k0s is under active development and **must be** considered experimental.

## Prerequisites

- The cluster must be running at least one worker node and control plane on Linux. You can use Windows to run additional worker nodes. **Supported Windows versions**: Windows Server 2019 and Windows Server 2022.

## Installation

### 1. Generate worker token from control plane

On your Linux control plane node, generate a worker token:

```shell
k0s token create --role=worker > worker-token.txt
```

Transfer this `worker-token.txt` file to your Windows node at `C:\k0s-token.txt`.

### 2. Enable Windows Containers feature

Open PowerShell as Administrator on the Windows node and run:

```powershell
Install-WindowsFeature -Name Containers
```

**Note**: A system reboot is required after enabling this feature.

### 3. Download and install k0s

After rebooting, open PowerShell as Administrator and run:

```powershell
# Set variables
$K0S_VERSION = "v1.34.3+k0s.0"
$TOKEN_FILE = "C:\k0s-token.txt"

# Download k0s
$encodedVersion = $K0S_VERSION.Replace('+', '%2B')
$url = "https://github.com/k0sproject/k0s/releases/download/$encodedVersion/k0s-$K0S_VERSION-amd64.exe"
Invoke-WebRequest -Uri $url -OutFile "$env:LOCALAPPDATA\Microsoft\WindowsApps\k0s.exe"

# Install and start k0s worker
k0s install worker --token-file $TOKEN_FILE
k0s start
```

**Note**: k0s supervises `kubelet.exe` and `kube-proxy.exe`.

### Network connectivity in AWS

Disable the `Change Source/Dest. Check` option for the network interface attached to your EC2 instance. In AWS, the console option for the network interface is in the **Actions** menu.

## Useful commands

### Run pod with cmd.exe shell

```shell
kubectl run win --image=hello-world:nanoserver --command=true -i --attach=true -- cmd.exe
```

### Manifest for pod with IIS web-server

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: iis
spec:
  containers:
  - name: iis
    image: mcr.microsoft.com/windows/servercore/iis
    imagePullPolicy: IfNotPresent
```

---

### Extensions (`docs/extensions.md`)

# Cluster extensions

k0s allows users to use extensions to extend cluster functionality.

At the moment the only supported type of extensions is helm based charts.

The default configuration has no extensions.

## Helm based extensions

### Configuration example

```
/* Detailed source-code truncated for AI context efficiency. */
```

By using the configuration above, the cluster would:

- add stable and prometheus-community chart repositories
- install the `prometheus-community/prometheus` chart of the specified version to the `default` namespace.

The chart installation is implemented by using the `helm.k0sproject.io/Chart` custom resource. k0s automatically converts charts defined in the k0s configuration into Chart resources. You can also create Chart resources directly for more flexible management.

The cluster has a controller which monitors Chart resources and supports the following operations:

- install
- upgrade
- delete

For security reasons, the cluster operates only on Chart resources in the `kube-system` namespace, however, the target namespace for releases can be any namespace.

#### Chart Resource Specification

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: prometheus
  namespace: kube-system
spec:
  chartName: prometheus-community/prometheus
  version: "14.6.1"
  namespace: default
  timeout: 20m
  values: |
    alertmanager:
      persistentVolume:
        enabled: false
  repository:
    url: https://prometheus-community.github.io/helm-charts
    # Or use secret-based authentication:
    # configFrom:
    #   secretRef:
    #     name: helm-creds
status:
  releaseName: prometheus
  version: "14.6.1"
  appVersion: "2.31.1"
  revision: 1
  namespace: default
  updated: "2024-01-15T10:30:45Z"
```

The `spec.repository` field allows each Chart to include its own repository configuration, making Charts self-contained and portable. When using k0s configuration file, k0s automatically embeds the repository configuration into the generated Chart resources.

For more details on using Chart resources directly, see the [Helm Charts](helm-charts.md) documentation.

---

### External Runtime Deps (`docs/external-runtime-deps.md`)

# External runtime dependencies

k0s is packaged as a single binary, which includes all the needed components.
All the binaries are statically linked which means that in typical use cases
there's an absolute minimum of external runtime dependencies.

However, depending on the node role and cluster configuration, some of the
underlying components _may_ have specific dependencies, like OS level tools,
packages and libraries. This page aims to provide a comprehensive overview.

The following command checks for known requirements on a host (currently only
available on Linux):

```shell
k0s sysinfo
```

## Linux specific



### Linux kernel configuration

Needless to say, as k0s operates Kubernetes worker nodes, a certain number of
Linux kernel modules and configurations are required on the host. This
essentially stems from the requirement to run containers and set up networking
for them.

The needed kernel configuration items are listed below. All of them are
available in Kernel versions 4.3 and above. If running on older kernels, check
if the distro in use has backported some features; nevertheless, it might meet
the requirements. k0s will check the Linux kernel release as part of its
pre-flight checks and issue a warning if it's below 3.10.

The list covers ONLY the k0s/Kubernetes components' needs on worker nodes. Your
own workloads may require more.



- [`CONFIG_CGROUPS`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L927):
  Control Group support
  - [`CONFIG_CGROUP_SCHED`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L1081):
    Group CPU scheduler
    - [`CONFIG_FAIR_GROUP_SCHED`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L1090):
      Group scheduling for SCHED_OTHER<br>
      [kubernetes/kubeadm#2335 (comment)](https://github.com/kubernetes/kubeadm/issues/2335#issuecomment-717996215)
      - _(optional)_ [`CONFIG_CFS_BANDWIDTH`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L1095):
        CPU bandwidth provisioning for FAIR_GROUP_SCHED<br>
        Required if CPU CFS quota enforcement is enabled for containers that
        specify CPU limits (`--cpu-cfs-quota`).
  - _(optional)_ [`CONFIG_BLK_CGROUP`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L1119):
    Block IO controller<br>
    [kubernetes/kubernetes#92287 (comment)](https://github.com/kubernetes/kubernetes/issues/92287#issuecomment-1010723587)
- [`CONFIG_NAMESPACES`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L1168):
  Namespaces support
  - [`CONFIG_UTS_NS`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L1180):
    UTS namespace
  - [`CONFIG_IPC_NS`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L1187):
    IPC namespace
  - [`CONFIG_PID_NS`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L1210):
    PID namespace
  - [`CONFIG_NET_NS`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L1218):
    Network namespace
- [`CONFIG_NET`](https://github.com/torvalds/linux/blob/v4.3/net/Kconfig#L5):
  Networking support
  - [`CONFIG_INET`](https://github.com/torvalds/linux/blob/v4.3/net/Kconfig#L58):
    TCP/IP networking
  - [`CONFIG_NETFILTER`](https://github.com/torvalds/linux/blob/v4.3/net/Kconfig#L109):
    Network packet filtering framework (Netfilter)
    - _(optional)_ [`CONFIG_NETFILTER_ADVANCED`](https://github.com/torvalds/linux/blob/v4.3/net/Kconfig#L171):
      Advanced netfilter configuration
    - [`CONFIG_NETFILTER_XTABLES`](https://github.com/torvalds/linux/blob/v4.3/net/netfilter/Kconfig#L567):
      Netfilter Xtables support
      - [`CONFIG_NETFILTER_XT_TARGET_REDIRECT`](https://github.com/torvalds/linux/blob/v4.3/net/netfilter/Kconfig#L853):
        REDIRECT target support
      - [`CONFIG_NETFILTER_XT_MATCH_COMMENT`](https://github.com/torvalds/linux/blob/v4.3/net/netfilter/Kconfig#L1002):
        "comment" match support
- [`CONFIG_EXT4_FS`](https://github.com/torvalds/linux/blob/v4.3/fs/ext4/Kconfig#L33):
  The Extended 4 (ext4) filesystem
- [`CONFIG_PROC_FS`](https://github.com/torvalds/linux/blob/v4.3/fs/proc/Kconfig#L1):
  /proc file system support

**Note:** As part of its pre-flight checks, k0s will try to inspect and validate
the kernel configuration. In order for that to succeed, the configuration needs
to be accessible at runtime. There are some typical places that k0s will check.
A bullet-proof way to ensure the accessibility is to enable
[`CONFIG_IKCONFIG_PROC`](https://github.com/torvalds/linux/blob/v4.3/init/Kconfig#L807),
and, if enabled as a module, to load the `configs` module: `modprobe configs`.

### Control Groups (cgroups)

[cgroup v2] is required by default, k0s will fail its pre-flight checks if it's
unavailable on the node.

As of Kubernetes 1.31, [cgroup v1] is in [maintenance mode] ([KEP-4569]). As of
Kubernetes 1.35, the kubelet will fail to start by default if cgroup v2 is not
detected on the node. In order to run Kubernetes with cgroup v1, bypass the k0s
pre-flight checks via `--ignore-pre-flight-checks` and set `failCgroupV1: false`
in the kubelet configuration via the k0s [worker profiles]:

```yaml
...
spec:
  workerProfiles:
    # By default, workers will pick up the default profile.
    - name: default
      values:
        failCgroupV1: false
```

!!! warning "Migrate to cgroup v2 now!"
    The above escape hatch will not be available indefinitely, so plan
    accordingly. [KEP-5573] states:
    
    > We will commit to removing cgroup v1 code but there is not yet a timeline for
    > removal. The removal will be done no earlier than 1.38 to maintain the k8s
    > deprecation policy.

Required [cgroup] controllers:

- `cpu`
- `cpuacct`
- `cpuset`
- `memory`
- `devices`
- `freezer`
- `pids`

Optional cgroup controllers:

- `hugetlb` ([kubernetes/kubeadm#2335 (comment)])
- `blkio` ([kubernetes/kubernetes#92287 (comment)])<br>
  containerd and cri-o will use blkio to track disk I/O and throttling in both
  cgroup v1 and v2.

[cgroup]: https://man7.org/linux/man-pages/man7/cgroups.7.html
[cgroup v1]: https://www.kernel.org/doc/html/v5.16/admin-guide/cgroup-v1/
[cgroup v2]: https://www.kernel.org/doc/html/v5.16/admin-guide/cgroup-v2.html
[kubernetes/kubeadm#2335 (comment)]: https://github.com/kubernetes/kubeadm/issues/2335#issuecomment-722405527
[kubernetes/kubernetes#92287 (comment)]: https://github.com/kubernetes/kubernetes/issues/92287#issuecomment-1010723587
[maintenance mode]: https://kubernetes.io/blog/2024/08/14/kubernetes-1-31-moving-cgroup-v1-support-maintenance-mode/
[KEP-4569]: https://www.kubernetes.dev/resources/keps/4569/
[KEP-5573]: https://www.kubernetes.dev/resources/keps/5573/
[worker profiles]: worker-node-config.md#kubelet-configuration

### No integration with Name Service Switch (NSS) APIs

The k0s Linux binaries, including the ones distributed on the GitHub releases
pages, are by default statically linked and do not depend on the host's
C&nbsp;library. This ensures that k0s can run seamlessly across a wide range of
Linux environments. The `k0s` executable and most of the embedded binaries are
built without [cgo] and use the [pure Go resolver][netdns-go], which only
consults the `files` and `dns` sources of `/etc/nsswitch.conf`. The Go
executables that do require cgo, such as `containerd` and `runc`, as well as all
non-Go executables, are statically linked against [musl libc], which doesn't
implement NSS at all. In all cases, k0s cannot use [glibc's NSS APIs], which
require dynamic linking.

This limitation is particularly relevant when a system uses NSS plugins, such as
[nss-myhostname], for resolving network names like `localhost`. Systems lacking
a dedicated stub resolver capable of handling `localhost` DNS queries
specifically will encounter issues running k0s. To mitigate this, users are
advised to either activate a stub DNS resolver, such as `systemd-resolved`, or
to manually add `localhost` entries to the `/etc/hosts` file as shown below:

```text
127.0.0.1 localhost
::1 localhost
```

The same limitation applies to systems that resolve all names via NSS, such as
through [nss-resolve] on hosts running `systemd-resolved`. Programs linked
against glibc can resolve names just fine on such systems, even if
`/etc/resolv.conf` doesn't list any nameservers. In contrast, k0s and its
embedded components only ever consult `/etc/resolv.conf`. If that file has no
nameserver entries, they fall back to querying a nameserver on localhost, which
usually doesn't exist. This leaves k0s without working name resolution, although
the host itself appears fine. Make sure that `/etc/resolv.conf` lists reachable
nameservers. When using `systemd-resolved`, this typically means making
`/etc/resolv.conf` a symbolic link to `/run/systemd/resolve/stub-resolv.conf`,
as described in the [systemd-resolved documentation]. Pods can't reach that stub
resolver on the host's loopback interface, which is why k0s configures the
kubelet to use systemd-resolved's uplink file `/run/systemd/resolve/resolv.conf`
for pods instead. The k0s pre-flight checks detect this situation by resolving a
name under the reserved `invalid` top-level domain, which any working resolver
answers with NXDOMAIN. If that lookup fails, k0s issues a warning. If
`/etc/resolv.conf` doesn't list any nameservers on top of that, worker nodes
refuse to start. Controllers only warn in that case.

[cgo]: https://pkg.go.dev/cmd/cgo
[netdns-go]: https://pkg.go.dev/net#hdr-Name_Resolution
[musl libc]: https://musl.libc.org/
[glibc's NSS APIs]: https://www.gnu.org/software/libc/manual/html_node/Name-Service-Switch.html
[nss-myhostname]: https://www.freedesktop.org/software/systemd/man/latest/nss-myhostname.html
[nss-resolve]: https://www.freedesktop.org/software/systemd/man/latest/nss-resolve.html
[systemd-resolved documentation]: https://www.freedesktop.org/software/systemd/man/latest/systemd-resolved.service.html#/etc/resolv.conf

### Kernel Parameters

#### Inotify Instance Limit

[Inotify API](https://www.man7.org/linux/man-pages/man7/inotify.7.html) allows
applications to monitor filesystem changes.

Workloads that run many pods or watch large numbers of files can exhaust the
node's default inotify limits, particularly `fs.inotify.max_user_instances`,
which limits how many inotify instances one user (UID) can have.

A value of at least 1024 is recommended; 8192 is commonly used in Kubernetes
environments:

```text
fs.inotify.max_user_instances = 8192
```

1. Check the current value:

   ```bash
   cat /proc/sys/fs/inotify/max_user_instances
   ```

2. To apply the new value persistently, create a `sysctl` configuration file,
   for example, `/etc/sysctl.d/99-inotify.conf`, and place the parameter
   setting from above in it.

3. Load the new kernel values:

   ```bash
   sudo sysctl --system
   ```

### External hard dependencies

There are very few external tools that are needed or used.

#### mount/umount

When setting up pods, kubelet will call `mount` binary on the host. Similarly when destroying pods it will call `umount`. `mount` and `umount` are only needed on worker nodes where kubelet runs.

### External soft dependencies

There are a few external tools that may be needed or used under specific
circumstances:

#### containerd and AppArmor

In order to use containerd in conjunction with [AppArmor], it must be enabled in
the kernel and the `/sbin/apparmor_parser` executable must be installed on the
host, otherwise containerd will [disable][cd-aa] AppArmor support.

[cd-aa]: https://github.com/containerd/containerd/blob/v2.3.5/pkg/apparmor/apparmor_linux.go#L34-L45
[AppArmor]: https://wiki.ubuntu.com/AppArmor

#### iptables

iptables may be executed to detect if there are any existing iptables rules and
if those are in legacy or nft mode. If iptables is not found, k0s will assume
that there are no pre-existing iptables rules.

#### useradd / adduser

During `k0s install` the external tool `useradd` will be used on the controllers
to create system user accounts for k0s. If this doesn't exist it will fall-back to
busybox's `adduser`.

#### userdel / deluser

`k0s reset` will execute either `userdel` or `deluser` to clean up system user
accounts.

#### modprobe

On k0s worker `modprobe` will be executed to load missing kernel modules if they
are not detected.

#### id

External `id` will be executed as a fallback if local user lookup fails, in case
NSS is used.

## Windows specific



TBD.

---

### Feature Gates (`docs/feature-gates.md`)

# Feature Gates

Feature gates are a mechanism to enable or disable experimental features in k0s.
They allow you to control which features are active in your cluster, providing
a way to safely test new functionality before it becomes generally available.

## Overview

Feature gates in k0s follow a lifecycle model similar to Kubernetes:

- **Alpha**: Experimental features that are disabled by default and may be unstable or removed
- **Beta**: Features that are more stable but still being tested, enabled by default
- **GA (Generally Available)**: Stable features that cannot be disabled. If explicitly disabled
k0s will fail to start

Once a feature reaches GA status, the feature gate cannot be disabled anymore and it will be
eventually removed.

## Using Feature Gates

Feature gates can be configured using the `--feature-gates` flag when starting the k0s controllers:

```shell
k0s controller --feature-gates="ExampleFeature=true,AnotherFeature=false"
```

The feature gates configuration must be consistent across all controllers in the cluster.

**Important notes:**

- Currently feature gates apply only to the controllers. They may be added to workers in future releases
- Setting a GA feature gate to false will cause k0s startup to fail
- Unknown feature gates will cause k0s start up to fail
- Feature gates use case-sensitive CamelCase names

## Available Features

The following table is a summary of the feature gates that you can set. Each feature is designed for a specific stage of stability:

| Feature           | Default | Stage | Since | Until |
|-------------------|---------|-------|-------|-------|
| `IPv6SingleStack` | `false` | Alpha | 1.34  |       |

### Feature Details

- `IPv6SingleStack`: Enables single-stack IPv6 support in k0s clusters.

---

### Helm Charts (`docs/helm-charts.md`)

# Helm Charts

k0s provides built-in support for deploying applications using Helm charts. You can deploy charts using:

- **Chart custom resources** (recommended) - Self-contained Kubernetes resources that include repository configuration
- **k0s configuration file** - Centralized configuration for bootstrap and cluster-level management
- **Helm CLI** - Standard Helm commands work with k0s clusters

## Using Chart Custom Resources (Recommended)

The recommended way to deploy Helm charts in k0s is by creating `Chart` custom resources directly. This approach provides:

- **Self-contained configuration** - Each Chart includes its repository configuration
- **GitOps-friendly** - Manage charts as standard Kubernetes resources
- **Flexible credentials** - Different charts can use different repository credentials
- **Portable** - Charts can be moved between clusters easily
- **No k0s restart needed** - Apply changes instantly with `kubectl`

### Quick Start

Create a Chart resource to deploy Prometheus:

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: prometheus
  namespace: kube-system
spec:
  chartName: prometheus-community/prometheus
  version: "14.6.1"
  namespace: default
  values: |
    alertmanager:
      persistentVolume:
        enabled: false
  repository:
    url: https://prometheus-community.github.io/helm-charts
```

Apply it: `kubectl apply -f prometheus-chart.yaml`

### Chart Resource Configuration

#### Spec Fields

| Field          | Default value | Description                                                                    |
|----------------|---------------|--------------------------------------------------------------------------------|
| `chartName`    | _(required)_  | Chart reference: `repo/chart`, `oci://registry/chart`, or `/path/to/chart.tgz` |
| `version`      | _(required)_  | Chart version to install                                                       |
| `namespace`    | _(required)_  | Target namespace for the release                                               |
| `releaseName`  | -             | Helm release name (defaults to Chart resource name)                            |
| `values`       | -             | Custom chart values as YAML formatted string                                   |
| `timeout`      | `10m`         | Timeout to wait for release install                                            |
| `forceUpgrade` | `true`        | When set to `false`, disables the use of the `--force` flag when upgrading     |
| `repository`   | -             | Repository configuration (see below)                                           |

#### Repository Configuration

| Field      | Description                                                   |
|------------|---------------------------------------------------------------|
| `url`      | Repository URL (required for traditional repos, omit for OCI) |
| `username` | Username for Basic HTTP authentication                        |
| `password` | Password for Basic HTTP authentication                        |
| `caFile`   | CA bundle file path for HTTPS verification                    |
| `certFile` | TLS certificate file path for mTLS                            |
| `keyFile`  | TLS key file path for mTLS                                    |
| `insecure` | Skip TLS certificate checks (default: `false`)                |

**Note:** For traditional Helm repositories, only repositories providing a valid `index.yaml` are supported. See the [Helm repository documentation](https://helm.sh/docs/topics/chart_repository/) for details.

### Examples

#### Traditional Helm Repository

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: nginx-ingress
  namespace: kube-system
spec:
  chartName: ingress-nginx/ingress-nginx
  version: "4.0.0"
  namespace: ingress-nginx
  repository:
    url: https://kubernetes.github.io/ingress-nginx
```

#### With Authentication

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: private-chart
  namespace: kube-system
spec:
  chartName: myrepo/mychart
  version: "1.0.0"
  namespace: default
  repository:
    url: https://charts.example.com
    username: myuser
    password: mytoken
```

#### OCI Registry (Public)

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: oci-chart
  namespace: kube-system
spec:
  chartName: oci://ghcr.io/org/chart
  version: "0.5.0"
  namespace: default
  # No repository field needed for public OCI registries
```

#### OCI Registry with Authentication

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: private-oci-chart
  namespace: kube-system
spec:
  chartName: oci://registry.example.com:5000/charts/app
  version: "1.0.0"
  namespace: default
  repository:
    # URL not needed for OCI - it's in chartName
    username: robot-account
    password: secret-token
```

#### OCI Registry with Custom CA and mTLS

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: registry-with-tls
  namespace: kube-system
spec:
  chartName: oci://registry.internal.com:8080/charts/app
  version: "1.0.0"
  namespace: default
  repository:
    caFile: /etc/k0s/pki/registry-ca.crt
    certFile: /etc/k0s/pki/client.crt
    keyFile: /etc/k0s/pki/client.key
```

#### Local Chart File

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: local-chart
  namespace: kube-system
spec:
  chartName: /var/lib/k0s/charts/mychart-1.0.0.tgz
  version: "1.0.0"
  namespace: default
  # No repository field needed for local files
```

#### Secret-Based Authentication

For sensitive credentials, use Kubernetes Secrets instead of embedding them in Chart resources:

```bash
# Create a secret with repository credentials
kubectl create secret generic helm-registry-creds \
  --from-literal=username=myuser \
  --from-literal=password=mypassword \
  -n kube-system
```

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: app-from-private-registry
  namespace: kube-system
spec:
  chartName: oci://registry.example.com/charts/app
  version: "1.0.0"
  namespace: default
  repository:
    configFrom:
      secretRef:
        name: helm-registry-creds
        # namespace defaults to Chart's namespace if omitted
```

**Secret Keys:**

- `url` - Repository URL
- `username` - HTTP Basic auth username  
- `password` - HTTP Basic auth password
- `ca.crt` - CA certificate bundle (PEM format)
- `tls.crt` - Client TLS certificate (PEM format)
- `tls.key` - Client TLS private key (PEM format)
- `insecure` - Set to "true" to skip TLS verification

**Precedence:** Secret values override inline repository fields when present. Inline fields provide defaults when corresponding secret keys are missing or empty.

**Example with defaults:**

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: app-with-defaults
  namespace: kube-system
spec:
  chartName: oci://registry.example.com/charts/app
  version: "1.0.0"
  namespace: default
  repository:
    url: oci://registry-fallback.local/charts/app  # Used only if secret doesn't provide 'url'
    username: default-user                         # Used only if secret doesn't provide 'username'
    configFrom:
      secretRef:
        name: helm-registry-creds  # Secret values always override inline fields
```

### Chart Lifecycle

Charts are automatically managed by k0s:

- **Install**: When a Chart resource is created, k0s installs the Helm release
- **Upgrade**: When Chart spec changes (version, values), k0s upgrades the release
- **Uninstall**: When Chart resource is deleted, k0s uninstalls the release

Monitor chart status:

```bash
kubectl get charts -n kube-system
kubectl describe chart prometheus -n kube-system
```

**Note:** Chart resources must be created in the `kube-system` namespace for security reasons, but can deploy releases to any target namespace.

## Using k0s Configuration (Alternative Method)

For cluster-level Helm chart management during bootstrap or centralized configuration, you can define charts in the k0s configuration file. k0s automatically converts these into Chart custom resources.

This approach:

- Requires modifying `k0s.yaml` and restarting k0s controllers
- Useful for bootstrap and initial cluster setup
- Centralized configuration for all charts

### Configuration Structure

### Chart install and upgrade options

Charts are processed with the following Helm options by default:

- `--create-namespace`
- `--atomic`
- `--force` (only for the `upgrade` command)
- `--wait`
- `--wait-for-jobs`

See the configuration tables below for how to customize these options.

### Repository configuration

| Field      | Default value | Description                                                                                       |
|------------|---------------|---------------------------------------------------------------------------------------------------|
| `name`     | _(required)_  | The repository name                                                                               |
| `url`      | _(required)_  | The repository URL                                                                                |
| `insecure` | `true`        | Whether to skip TLS certificate checks when connecting to the repository                          |
| `caFile`   | -             | CA bundle file to use when verifying HTTPS-enabled servers                                        |
| `certFile` | -             | The TLS certificate file to use for HTTPS client authentication (mTLS)                            |
| `keyFile`  | -             | The TLS key file to use for HTTPS client authentication (mTLS)                                    |
| `username` | -             | Username for Basic HTTP authentication                                                            |
| `password` | -             | Password for Basic HTTP authentication                                                            |

**Note:** k0s supports only classic Helm chart repositories that provide a valid `index.yaml`. Direct links to chart folders or files (for example raw GitHub URLs) are not recognized as Helm repositories and will not work unless they follow the full Helm repository structure. For details on how a valid Helm chart repository must be structured, see: https://helm.sh/docs/topics/chart_repository/#create-a-chart-repository

### Chart configuration

| Field          | Default value | Description                                                                            |
|----------------|---------------|----------------------------------------------------------------------------------------|
| `name`         | -             | Release name                                                                           |
| `chartname`    | -             | Chart name in form `repository/chartname` or path to tgz file                          |
| `version`      | -             | Chart version to install                                                               |
| `timeout`      | -             | Timeout to wait for release install                                                    |
| `values`       | -             | Custom chart values as YAML formatted string                                           |
| `namespace`    | -             | Namespace to install the chart into                                                    |
| `forceUpgrade` | `true`        | When set to `false`, disables the use of the `--force` flag when upgrading the chart   |
| `order`        | `0`           | Order in which to apply the manifest. For equal values, alphanumeric ordering is used. |

### Example k0s.yaml Configuration

In this example, Prometheus is configured from the prometheus-community Helm chart repository:

```yaml
spec:
  extensions:
    helm:
      concurrencyLevel: 5
      repositories:
      - name: stable
        url: https://charts.helm.sh/stable
      - name: prometheus-community
        url: https://prometheus-community.github.io/helm-charts
      - name: helm-repo-with-auth
        url: https://charts.example.com
        username: myuser
        password: mytoken
      charts:
      - name: prometheus-stack
        chartname: prometheus-community/prometheus
        version: "14.6.1"
        timeout: 20m
        order: 1
        values: |
          alertmanager:
            persistentVolume:
              enabled: false
          server:
            persistentVolume:
              enabled: false
        namespace: default
```

Add this to your `k0s.yaml` and restart k0s controllers. The charts will deploy automatically.

## Migrating from k0s Config to Chart Resources

To migrate from k0s.yaml configuration to Chart custom resources:

1. For each chart in your `spec.extensions.helm.charts` section:
   - Find the matching repository from `spec.extensions.helm.repositories`
   - Create a Chart resource with the repository configuration embedded
   - Apply the Chart resource: `kubectl apply -f chart.yaml`

2. Optionally remove charts from k0s.yaml (both methods can coexist)

**Example migration:**

**Before (k0s.yaml):**

```yaml
spec:
  extensions:
    helm:
      repositories:
      - name: prometheus-community
        url: https://prometheus-community.github.io/helm-charts
      charts:
      - name: prometheus
        chartname: prometheus-community/prometheus
        version: "14.6.1"
        namespace: default
```

**After (Chart resource):**

```yaml
apiVersion: helm.k0sproject.io/v1beta1
kind: Chart
metadata:
  name: prometheus
  namespace: kube-system
spec:
  chartName: prometheus-community/prometheus
  version: "14.6.1"
  namespace: default
  repository:
    url: https://prometheus-community.github.io/helm-charts
```

## Popular Chart Examples

Example extensions that work well with k0s:

- Ingress controllers: [nginx ingress](https://github.com/helm/charts/tree/master/stable/nginx-ingress), [Traefik ingress](https://github.com/traefik/traefik-helm-chart) (refer to the k0s documentation for [Installing the Traefik Ingress Controller](examples/traefik-ingress.md))
- Volume storage providers: [OpenEBS](https://openebs.github.io/openebs/), [Rook](https://github.com/rook/rook/blob/master/Documentation/helm-operator.md), [Longhorn](https://longhorn.io/docs/0.8.1/deploy/install/install-with-helm/)
- Monitoring: [Prometheus](https://github.com/prometheus-community/helm-charts/), [Grafana](https://github.com/grafana/helm-charts)

## Helm debug logging

Running k0s controller with `--debug=true` enables helm debug logging.

---

### High Availability (`docs/high-availability.md`)

# Control Plane High Availability

You can create high availability for the control plane by distributing the control plane across multiple nodes and installing a load balancer on top. Etcd can be co-located with the controller nodes (default in k0s) to achieve highly available data store at the same time.



**Note:** In this context, a two-node control plane is considered highly
available, even though it is not HA from etcd's perspective. The same
requirement for the [load balancer](#load-balancer) still applies.

## Network considerations

You should plan to allocate the control plane nodes into different zones. This will avoid failures in case one zone fails.

For etcd high availability it's recommended to configure 3 or 5 controller nodes. For more information, refer to the [etcd documentation](https://etcd.io/docs/latest/faq/#why-an-odd-number-of-cluster-members).

## Load Balancer

Control plane high availability requires a tcp load balancer, which acts as a single point of contact to access the controllers. The load balancer needs to allow and route traffic to each controller through the following ports:

- 6443 (for Kubernetes API)
- 8132 (for Konnectivity)
- 9443 (for controller join API)

The load balancer can be implemented in many different ways and k0s doesn't have any additional requirements. You can use for example HAProxy, nginx or your cloud provider's load balancer.

### Example configuration: HAProxy

Add the following lines at the end of `haproxy.cfg`:

```txt
frontend kubeAPI
    bind :6443
    mode tcp
    default_backend kubeAPI_backend
frontend konnectivity
    bind :8132
    mode tcp
    default_backend konnectivity_backend
frontend controllerJoinAPI
    bind :9443
    mode tcp
    default_backend controllerJoinAPI_backend

backend kubeAPI_backend
    mode tcp
    server k0s-controller1 <ip-address1>:6443 check check-ssl verify none
    server k0s-controller2 <ip-address2>:6443 check check-ssl verify none
    server k0s-controller3 <ip-address3>:6443 check check-ssl verify none
backend konnectivity_backend
    mode tcp
    server k0s-controller1 <ip-address1>:8132 check check-ssl verify none
    server k0s-controller2 <ip-address2>:8132 check check-ssl verify none
    server k0s-controller3 <ip-address3>:8132 check check-ssl verify none
backend controllerJoinAPI_backend
    mode tcp
    server k0s-controller1 <ip-address1>:9443 check check-ssl verify none
    server k0s-controller2 <ip-address2>:9443 check check-ssl verify none
    server k0s-controller3 <ip-address3>:9443 check check-ssl verify none

listen stats
   bind *:9000
   mode http
   stats enable
   stats uri /
```

The last block "listen stats" is optional, but can be helpful. It enables HAProxy statistics with a separate dashboard to monitor for example the health of each backend server. You can access it using a web browser:

```txt
http://<ip-addr>:9000
```

Restart HAProxy to apply the configuration changes.

## k0s configuration

First and foremost, all controllers should utilize the same CA certificates and SA key pair:

```txt
/var/lib/k0s/pki/ca.key
/var/lib/k0s/pki/ca.crt
/var/lib/k0s/pki/sa.key
/var/lib/k0s/pki/sa.pub
/var/lib/k0s/pki/etcd/ca.key
/var/lib/k0s/pki/etcd/ca.crt
```

To generate these certificates, you have two options: either generate them
manually using the instructions for [installing custom CA certificates], and
then share them between controller nodes, or use k0sctl to generate and share
them automatically.

The second important aspect is: the load balancer address must be configured to k0s either by using `k0s.yaml` or by using k0sctl to automatically deploy all controllers with the same configuration:

[installing custom CA certificates]: custom-ca.md

### Configuration using k0s.yaml (for each controller)

Note to update your load balancer's public IP address into two places.

```yaml
spec:
  api:
    externalAddress: <load balancer public ip address>
```

### Configuration using k0sctl.yaml (for k0sctl)

Add the following lines at the end of `k0sctl.yaml`. Note to update your load
balancer's public IP address into two places.

```yaml
  k0s:
    config:
      spec:
        api:
          externalAddress: <load balancer public ip address>
```

For greater detail about k0s configuration, refer to the [Full configuration file reference](configuration.md).

---

### Install (`docs/install.md`)

# Quick Start Guide

Completing this Quick Start results in a single-node Kubernetes cluster that
incorporates both the controller and worker roles. This setup is ideal for
environments that don't require high availability or multiple nodes.

## Prerequisites

**Note**: Before proceeding, make sure to review the [System Requirements](system-requirements.md).

The following steps work on every typical Linux distribution that uses either
systemd or OpenRC as its init system.

## Install k0s

1. Download k0s

    Run the k0s download script to download the latest stable version of k0s and make it executable from /usr/local/bin/k0s.

    ```shell
    curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh
    ```

    Alternatively, download it from the [k0s releases page]. This approach is
    required in air gapped environments.

2. Install k0s as a service

    The `k0s install` sub-command installs k0s as a system service on a host that is running one of the supported init systems: systemd or OpenRC. You can execute the install for workers, controllers or single node (controller+worker) instances.

    Run the following command to install a single node k0s that includes the controller and worker functions with the default configuration:

    ```shell
    sudo k0s install controller --single
    ```

    **Note**: The `--single` option disables features needed for multi-node
    clusters, so the cluster cannot be extended. To retain the ability to expand
    the cluster in the future, use:

    ``` shell
    sudo k0s install controller --enable-worker --no-taints
    ```

    The `k0s install controller` sub-command accepts the same flags and parameters as the `k0s controller`. Refer to [manual install](k0s-multi-node.md#install-k0s) for a custom config file example.

    It is possible to set environment variables with the install command:

    ```shell
    sudo k0s install controller -e ETCD_UNSUPPORTED_ARCH=arm
    ```

    The system service can be reinstalled with the `--force` flag:

    ```shell
    sudo k0s install controller --single --force
    sudo systemctl daemon-reload
    ```

    If you need to reinstall and immediately start the service, combine `--force` with `--start`:

    ```shell
    sudo k0s install controller --single --force --start
    ```

    When used with `--start`, the `--force` flag will also restart an already running service.

3. Start k0s as a service

    To start the k0s service, run:

    ```shell
    sudo k0s start
    ```

    Alternatively, both steps can be combined by using the `--start` flag with the install command:

    ```shell
    sudo k0s install controller --single --start
    ```

    The k0s service will start automatically after the node restart.

    A minute or two typically passes before the node is ready to deploy applications.

4. Check service, logs and k0s status

    Run the following command to check the k0s instance status:

    ```shell
    $ sudo k0s status
    Version: {{{ k0s_version }}}
    Process ID: 436
    Role: controller
    Workloads: true
    Init System: linux-systemd
    ```

5. Access the cluster using kubectl

    **Note**: k0s embeds the Kubernetes command-line tool *kubectl*.

    Use kubectl to deploy applications or check node status:

    ```shell
    $ sudo k0s kubectl get nodes
    NAME   STATUS   ROLES    AGE    VERSION
    k0s    Ready    <none>   4m6s   {{{ k8s_version }}}+k0s
    ```

[k0s releases page]: https://github.com/k0sproject/k0s/releases/latest

## Uninstall k0s

The removal of k0s is a two-step process.

1. Stop the service.

    ```shell
    sudo k0s stop
    ```

2. Execute the `k0s reset` command.

     The `k0s reset` command cleans up the installed system service, data directories, containers, mounts and network namespaces.

    ```shell
    sudo k0s reset
    ```

3. Reboot the system.

    A few small k0s fragments persist even after the reset, such as iptables
    rules. Reboot the machine after running `k0s reset`.

## Next Steps

- [Install using k0sctl](k0sctl-install.md): Deploy multi-node clusters using just one command
- [Manual Install](k0s-multi-node.md): (Advanced) Manually deploy multi-node clusters
- [Control plane configuration options](configuration.md): Networking and data store configuration
- [Worker node configuration options](worker-node-config.md): Node labels and kubelet arguments
- [Support for cloud providers](cloud-providers.md): Load balancer or storage configuration
- [Installing the Traefik Ingress Controller](examples/traefik-ingress.md):
  Ingress deployment information
- [Air gapped/offline installation](airgap-install.md): Air gaped deployment

---

### K0s In Docker (`docs/k0s-in-docker.md`)

# Run k0s in Docker

You can create a k0s cluster on top of Docker.

## Prerequisites

You will require a [Docker environment](https://docs.docker.com/get-docker/) running on a Mac, Windows, or Linux system.

## Container images

The k0s OCI images are published to both Docker Hub and GitHub Container
registry. For simplicity, the examples given here use Docker Hub (GitHub
requires separate authentication, which is not covered here). The image names
are as follows:

- `docker.io/k0sproject/k0s:{{{ k0s_docker_version }}}`
- `ghcr.io/k0sproject/k0s:{{{ k0s_docker_version }}}`

**Note:** Due to Docker's tag validation scheme, `-` is used as the k0s version
separator instead of the usual `+`. For example, k0s version `{{{ k0s_version
}}}` is tagged as `docker.io/k0sproject/k0s:{{{ k0s_docker_version }}}`.

## Start k0s

### 1. Run a controller

By default, running the k0s OCI image will launch a controller with workloads
enabled (i.e. a controller with the `--enable-worker` flag) to provide an easy
local testing "cluster":

```sh
docker run -d --name k0s-controller --hostname k0s-controller \
  -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --privileged `# this is the easiest way to enable container-in-container workloads` \
  -p 6443:6443 `# publish the Kubernetes API server port` \
  docker.io/k0sproject/k0s:{{{ k0s_docker_version }}}
```

Explanation of command line arguments:

- `-d` runs the container in detached mode, i.e. in the background.
- `--name k0s-controller` names the container "k0s-controller".
- `--hostname k0s-controller` sets the host name of the container to
  "k0s-controller".
- `-v /var/lib/k0s -v /var/log/pods` creates two Docker volumes and mounts them
  to `/var/lib/k0s` and `/var/log/pods` respectively inside the container,
  ensuring that cluster data persists across container restarts.
- `--tmpfs /run` **TODO**
- `--privileged` gives the container the elevated privileges that k0s needs to
  function properly within Docker. See the section on [adding additional
  workers](#2-optional-add-additional-workers) for a more detailed discussion of
  privileges.
- `-p 6443:6443` exposes the container's Kubernetes API server port 6443 to the
  host, allowing you to interact with the cluster externally.
- `docker.io/k0sproject/k0s:{{{ k0s_docker_version }}}` is the name of the
  k0s image to run.

By default, the k0s image starts a k0s controller with worker components enabled
within the same container, creating a cluster with a single
controller-and-worker node using the following command:

```Dockerfile
{% include "../Dockerfile" start="# Start CMD" end="# End CMD" %}
```

Alternatively, a controller-only node can be run like this:

```sh
docker run -d --name k0s-controller --hostname k0s-controller \
  --read-only `# k0s won't write any data outside the below paths` \
  -v /var/lib/k0s `# this is where k0s stores its data` \
  --tmpfs /run `# this is where k0s stores runtime data` \
  --tmpfs /tmp `# allow writing temporary files` \
  -p 6443:6443 `# publish the Kubernetes API server port` \
  docker.io/k0sproject/k0s:{{{ k0s_docker_version }}} \
  k0s controller
```

Note the addition of `k0s controller` to override the image's default command.
Also note that a controller-only node requires fewer privileges.

### 2. (Optional) Add additional workers

You can add multiple worker nodes to the cluster and then distribute your
application containers to separate workers.

1. Acquire a [join token](k0s-multi-node.md#3-create-a-join-token) for the
   worker:

   ```sh
   token=$(docker exec k0s-controller k0s token create --role=worker)
   ```

2. Run the container to create and join the new worker:

   ```sh
   docker run -d --name k0s-worker1 --hostname k0s-worker1 \
     -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \
     --tmpfs /run `# this is where k0s stores runtime data` \
     --privileged `# this is the easiest way to enable container-in-container workloads` \
     docker.io/k0sproject/k0s:{{{ k0s_docker_version }}} \
     k0s worker $token
   ```

   Alternatively, with fine-grained privileges:
   

   ```sh
   docker run -d --name k0s-worker1 --hostname k0s-worker1 \
     -v /var/lib/k0s -v /var/log/pods `# this is where k0s stores its data` \
     --tmpfs /run `# this is where k0s stores runtime data` \
     --security-opt seccomp=unconfined \
     --device /dev/kmsg \
     --cap-add sys_admin \
     --cap-add net_admin \
     --cap-add sys_ptrace \
     --cap-add sys_resource \
     --cap-add syslog \
     docker.io/k0sproject/k0s:{{{ k0s_docker_version }}} \
     k0s worker "$token"
   ```

   Notes on the security-related flags:

   - `--security-opt seccomp=unconfined` is required for `runc` to access the
     [session keyring].
   - `--device /dev/kmsg` makes `/dev/kmsg` visible from inside the container.
     The kubelet's OOM watcher uses this.
     

   Notes on [Linux capabilities]:

   - `CAP_SYS_ADMIN` allows for a variety of administrative tasks, including
     mounting file systems and managing namespaces, which are necessary for
     creating and configuring nested containers.
   - `CAP_NET_ADMIN` allows manipulation of network settings such as interfaces
     and routes, allowing containers to create isolated or bridged networks, and
     so on.
   - `CAP_SYS_PTRACE` allows to inspect and modify processes, used to monitor
     other containers in a nested environment.
     
   - `CAP_SYS_RESOURCE` allows containers to override resource limits for things
     like memory or file descriptors, used to manage and adjust resource
     allocation in nested container environments.
     
   - `CAP_SYSLOG` allows containers to perform privileged syslog operations.
     This is required in order to read `/dev/kmsg`.

   Note that more privileges may be required depending on your cluster
   configuration and workloads.

   Repeat this step for each additional worker node and adjust the container and
   host names accordingly. Make sure that the workers can reach the controller
   on the [required ports]. If you are using Docker's default bridged network,
   this should be the case.

[session keyring]: https://www.man7.org/linux/man-pages/man7/session-keyring.7.html
[Linux capabilities]: https://www.man7.org/linux/man-pages/man7/capabilities.7.html
[required ports]: networking.md#controller-worker-communication

### 3. Access your cluster

#### a) Using kubectl within the container

To check cluster status and list nodes, use:

```sh
docker exec k0s-controller k0s kubectl get nodes
```

#### b) Using kubectl locally

To configure local access to your k0s cluster, follow these steps:

1. Generate the kubeconfig:

    ```sh
    docker exec k0s-controller k0s kubeconfig admin > ~/.kube/k0s.config
    ```

2. Update kubeconfig with Localhost Access:

    To automatically replace the server IP with localhost dynamically in `~/.kube/k0s.config`, use the following command:

    ```sh
    sed -i '' -e "$(awk '/server:/ {print NR; exit}' ~/.kube/k0s.config)s|https://.*:6443|https://localhost:6443|" ~/.kube/k0s.config
    ```

    This command updates the kubeconfig to point to localhost, allowing access to the API server from your host machine

3. Set the KUBECONFIG Environment Variable:

    ```sh
    export KUBECONFIG=~/.kube/k0s.config
    ```

4. Verify Cluster Access:

    ```sh
    kubectl get nodes
    ```

#### c) Use [Lens]

Access the k0s cluster using Lens by following the instructions on [how to add a
cluster].

[Lens]: https://k8slens.dev/
[how to add a cluster]: https://docs.k8slens.dev/getting-started/add-cluster/

## Use Docker Compose (alternative)

As an alternative you can run k0s using Docker Compose:


```yaml
{% include "compose.yaml" %}
```

Below is a more complex example, using Traefik as a load balancer, along with
three controller and three worker nodes:


```yaml
{% include "compose-cluster.yaml" %}
```

Running the above:

```console
 ❯ docker compose up -d
[+] Running 11/11
 ✔ Network compose-cluster_k0s-net                Created                0.1s
 ✔ Volume "compose-cluster_k0s-token-secrets"     Created                0.0s
 ✔ Volume "compose-cluster_k0s-controller-token"  Created                0.0s
 ✔ Volume "compose-cluster_k0s-worker-token"      Created                0.0s
 ✔ Container k0s-lb                               Started                0.5s
 ✔ Container k0s-controller-1                     Started                11.8s
 ✔ Container k0s-controller-2                     Started                12.2s
 ✔ Container k0s-worker-1                         Started                12.4s
 ✔ Container k0s-worker-2                         Started                12.3s
 ✔ Container k0s-worker-3                         Started                12.1s
 ✔ Container k0s-controller-3                     Started                12.5s
```

After a short while:

```console
$ docker exec k0s-controller-1 k0s kc get node,po -A
NAME                STATUS   ROLES    AGE     VERSION
node/k0s-worker-1   Ready    <none>   1m36s   {{{ k8s_version }}}+k0s
node/k0s-worker-2   Ready    <none>   1m36s   {{{ k8s_version }}}+k0s
node/k0s-worker-3   Ready    <none>   1m36s   {{{ k8s_version }}}+k0s

NAMESPACE     NAME                                  READY   STATUS    RESTARTS   AGE
kube-system   pod/coredns-7d4f7fbd5c-54lxp          1/1     Running   0          1m27s
kube-system   pod/coredns-7d4f7fbd5c-pwbck          1/1     Running   0          1m27s
kube-system   pod/konnectivity-agent-5g8pn          1/1     Running   0          1m22s
kube-system   pod/konnectivity-agent-6rp7r          1/1     Running   0          1m22s
kube-system   pod/konnectivity-agent-zx9fn          1/1     Running   0          1m22s
kube-system   pod/kube-proxy-9m77t                  1/1     Running   0          1m36s
kube-system   pod/kube-proxy-v5vs6                  1/1     Running   0          1m36s
kube-system   pod/kube-proxy-xfw2h                  1/1     Running   0          1m36s
kube-system   pod/kube-router-6c62v                 1/1     Running   0          1m36s
kube-system   pod/kube-router-98ss8                 1/1     Running   0          1m36s
kube-system   pod/kube-router-lr46f                 1/1     Running   0          1m36s
kube-system   pod/metrics-server-7778865875-fzhx6   1/1     Running   0          1m37s
```

## Known limitations

### No custom Docker networks

Currently, k0s nodes cannot be run if the containers are configured to use custom networks (for example, with `--net my-net`). This is because Docker sets up a custom DNS service within the network which creates issues with CoreDNS. No completely reliable workarounds are available, however no issues should arise from running k0s cluster(s) on a bridge network.

## Next Steps

- [Install using k0sctl](k0sctl-install.md): Deploy multi-node clusters using just one command
- [Control plane configuration options](configuration.md): Networking and data store configuration
- [Worker node configuration options](worker-node-config.md): Node labels and kubelet arguments
- [Support for cloud providers](cloud-providers.md): Load balancer or storage configuration
- [Installing the Traefik Ingress Controller](examples/traefik-ingress.md): Ingress deployment information

---

### K0s Multi Node (`docs/k0s-multi-node.md`)

# Manual Install (Advanced)

You can manually set up k0s nodes by creating a multi-node cluster that is locally managed on each node. This involves several steps, to first install each node separately, and to then connect the node together using access tokens.

## Prerequisites

**Note**: Before proceeding, make sure to review the [System Requirements](system-requirements.md).

The following steps work on every typical Linux distribution that uses either
systemd or OpenRC as its init system.

You can speed up the use of the `k0s` command by enabling [shell completion](shell-completion.md).

## Install k0s

### 1. Download k0s

Run the k0s download script to download the latest stable version of k0s and make it executable from `/usr/bin/k0s`.

```shell
curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh
```

The download script accepts the following environment variables:

| Variable                          | Purpose                                           |
|:----------------------------------|:--------------------------------------------------|
| `K0S_VERSION={{{ k0s_version }}}` | Select the version of k0s to be installed         |
| `DEBUG=true`                      | Output commands and their arguments at execution. |

**Note**: If you require environment variables and use sudo, you can do:

```shell
curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo K0S_VERSION={{{ k0s_version }}} sh
```

### 2. Bootstrap a controller node

Create a configuration file:

```shell
mkdir -p /etc/k0s
k0s config create > /etc/k0s/k0s.yaml
```

**Note**: For information on settings modification, refer to the [configuration](configuration.md) documentation.

```shell
sudo k0s install controller -c /etc/k0s/k0s.yaml
```

```shell
sudo k0s start
```

Alternatively, both steps can be combined with the `--start` flag:

```shell
sudo k0s install controller -c /etc/k0s/k0s.yaml --start
```

The system service can be reinstalled with the `--force` flag. When combined with `--start`, it will also restart an already running service:

```shell
sudo k0s install controller -c /etc/k0s/k0s.yaml --force --start
```

k0s process acts as a "supervisor" for the control plane components. In moments the control plane will be up and running.

### 3. Create a join token

You need a token to join workers to the cluster. The token embeds information that enables mutual trust between the worker and controller(s) and which allows the node to join the cluster as worker.

To get a token, run the following command on one of the existing controller nodes:

```shell
sudo k0s token create --role=worker
```

The resulting output is a long [token](#about-join-tokens) string, which you can use to add a worker to the cluster.

For enhanced security, run the following command to set an expiration time for the token:

```shell
sudo k0s token create --role=worker --expiry=100h > token-file
```

### 4. Add workers to the cluster

To join the worker, run k0s in the worker mode with the join token you created:

```shell
sudo k0s install worker --token-file /path/to/token/file
```

```shell
sudo k0s start
```

Alternatively, both steps can be combined with the `--start` flag:

```shell
sudo k0s install worker --token-file /path/to/token/file --start
```

The system service can be reinstalled with the `--force` flag. When combined with `--start`, it will also restart an already running service:

```shell
sudo k0s install worker --token-file /path/to/token/file --force --start
```

#### About join tokens

The join tokens are base64-encoded [kubeconfigs](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/) for several reasons:

- Well-defined structure
- Capable of direct use as bootstrap auth configs for kubelet
- Embedding of CA info for mutual trust

The bearer token embedded in the kubeconfig is a [bootstrap token](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/). For controller join tokens and worker join tokens k0s uses different usage attributes to ensure that k0s can validate the token role on the controller side.

### 5. Add controllers to the cluster

**Note**: Either etcd or an external data store (MySQL or PostgreSQL) via kine must be in use to add new controller nodes to the cluster. Pay strict attention to the [high availability configuration](high-availability.md) and make sure the configuration is identical for all controller nodes.

To create a join token for the new controller, run the following command on an existing controller:

```shell
sudo k0s token create --role=controller --expiry=1h > token-file
```

On the new controller, run:

```shell
sudo k0s install controller --token-file /path/to/token/file -c /etc/k0s/k0s.yaml
```

Important notice here is that each controller in the cluster must have k0s.yaml otherwise some cluster nodes will use default config values which will lead to inconsistency behavior.
If your configuration file includes IP addresses (node address, SANs, etcd peer address), remember to update them accordingly for this specific controller node.

```shell
sudo k0s start
```

Alternatively, both steps can be combined with the `--start` flag:

```shell
sudo k0s install controller --token-file /path/to/token/file -c /etc/k0s/k0s.yaml --start
```

The system service can be reinstalled with the `--force` flag. When combined with `--start`, it will also restart an already running service:

```shell
sudo k0s install controller --token-file /path/to/token/file -c /etc/k0s/k0s.yaml --force --start
```

### 6. Check k0s status

Run the following command to check the k0s instance status:

```shell
 sudo k0s status
```

```shell
Version: {{{ k0s_version }}}
Process ID: 2769
Parent Process ID: 1
Role: controller
Init System: linux-systemd
Service file: /etc/systemd/system/k0scontroller.service
```

### 7. Access your cluster

Use the Kubernetes 'kubectl' command-line tool that comes with k0s binary to deploy your application or check your node status:

```shell
sudo k0s kubectl get nodes
```

```shell
NAME   STATUS   ROLES    AGE    VERSION
k0s    Ready    <none>   4m6s   {{{ k8s_version }}}+k0s
```

You can also access your cluster easily with [Lens](https://k8slens.dev/), simply by copying the kubeconfig and pasting it to Lens:

```shell
sudo cat /var/lib/k0s/pki/admin.conf
```

**Note**: To access the cluster from an external network you must replace `localhost` in the kubeconfig with the host ip address for your controller.

## Next Steps

- [Install using k0sctl](k0sctl-install.md): Deploy multi-node clusters using just one command
- [Control plane configuration options](configuration.md): Networking and data store configuration
- [Worker node configuration options](worker-node-config.md): Node labels and kubelet arguments
- [Support for cloud providers](cloud-providers.md): Load balancer or storage configuration
- [Installing the Traefik Ingress Controller](examples/traefik-ingress.md): Ingress deployment information

---

### K0sctl Install (`docs/k0sctl-install.md`)

# Install using k0sctl

k0sctl is a command-line tool for bootstrapping and managing k0s clusters. k0sctl connects to the provided hosts using SSH and gathers information on the hosts, with which it forms a cluster by configuring the hosts, deploying k0s, and then connecting the k0s nodes together.



With k0sctl, you can create multi-node clusters in a manner that is automatic and easily repeatable. This method is recommended for production cluster installation.

**Note**: The k0sctl install method is necessary for automatic upgrade.

## Prerequisites

You can execute k0sctl on any system that supports the Go language. Pre-compiled k0sctl binaries are available on the [k0sctl releases page](https://github.com/k0sproject/k0sctl)).

**Note**: For target host prerequisites information, refer to the [k0s System Requirements](system-requirements.md).

## Install k0s

### 1. Install k0sctl tool

k0sctl is a single binary, the instructions for downloading and installing of which are available in the [k0sctl github repository](https://github.com/k0sproject/k0sctl#installation).

### 2. Configure the cluster

1. Run the following command to create a k0sctl configuration file:

    ```shell
    k0sctl init > k0sctl.yaml
    ```

    This action creates a `k0sctl.yaml` file in the current directory:

    ```yaml
    apiVersion: k0sctl.k0sproject.io/v1beta1
    kind: Cluster
    metadata:
      name: k0s-cluster
    spec:
      hosts:
      - role: controller
        ssh:
          address: 10.0.0.1 # replace with the controller's IP address
          user: root
          keyPath: ~/.ssh/id_rsa
      - role: worker
        ssh:
          address: 10.0.0.2 # replace with the worker's IP address
          user: root
          keyPath: ~/.ssh/id_rsa
    ```

2. Provide each host with a valid IP address that is reachable by k0sctl, and the connection details for an SSH connection.

 **Note**: Refer to the [k0sctl documentation](https://github.com/k0sproject/k0sctl#configuration-file-spec-fields) for k0sctl configuration specifications.

### 3. Deploy the cluster

Run `k0sctl apply` to perform the cluster deployment:

```
/* Detailed source-code truncated for AI context efficiency. */
```

### 4. Access the cluster

To access your k0s cluster, use k0sctl to generate a `kubeconfig` for the purpose.

```shell
k0sctl kubeconfig > kubeconfig
```

With the `kubeconfig`, you can access your cluster using either kubectl or [Lens](https://k8slens.dev/).

```shell
kubectl get pods --kubeconfig kubeconfig -A
```

```shell
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5f6546844f-w8x27   1/1     Running   0          3m50s
kube-system   calico-node-vd7lx                          1/1     Running   0          3m44s
kube-system   coredns-5c98d7d4d8-tmrwv                   1/1     Running   0          4m10s
kube-system   konnectivity-agent-d9xv2                   1/1     Running   0          3m31s
kube-system   kube-proxy-xp9r9                           1/1     Running   0          4m4s
kube-system   metrics-server-6fbcd86f7b-5frtn            1/1     Running   0          3m51s
```

## Known limitations

* k0sctl does not perform any discovery of hosts, and thus it only operates on the hosts listed in the provided configuration.
* k0sctl can only add more nodes to the cluster. It cannot remove existing nodes.

## Next Steps

* [Control plane configuration options](configuration.md): Networking and datastore configuration
* [Worker node configuration options](worker-node-config.md): Node labels and kubelet arguments
* [Support for cloud providers](cloud-providers.md): Load balancer or storage configuration
* [Installing the Traefik Ingress Controller](examples/traefik-ingress.md): Ingress deployment information

---

### Manifests (`docs/manifests.md`)

# Manifest Deployer

Included with k0s, Manifest Deployer is one of two methods you can use to run k0s with your preferred extensions (the other being by defining your extensions as [Helm charts](helm-charts.md)).

## Overview

Manifest Deployer runs on the controller nodes and provides an easy way to automatically deploy manifests at runtime.

By default, k0s reads all manifests under `/var/lib/k0s/manifests` that end with `.yaml` and ensures that their state matches the cluster state. It does not read files with other extensions, such as `.yml`. Moreover, on removal of a manifest file, k0s will automatically prune all of its associated resources.

The use of Manifest Deployer is quite similar to the use the `kubectl apply` command. The main difference between the two is that Manifest Deployer constantly monitors the directory for changes, and thus you do not need to manually apply changes that are made to the manifest files.

### Note

- Each directory that is a direct descendant of `/var/lib/k0s/manifests` is considered to be its own "stack". Nested directories (further subfolders), however, are excluded from the stack mechanism and thus are not automatically deployed by the Manifest Deployer.

- k0s uses the independent stack mechanism for some of its internal in-cluster components, as well as for other resources. Be sure to only touch the manifests that are not managed by k0s.

- Explicitly define the namespace in the manifests (Manifest Deployer does not have a default namespace).

## Example

To try Manifest Deployer, create a new folder under `/var/lib/k0s/manifests` and then create a manifest file (such as `nginx.yaml`) with the following content:

```yaml
apiVersion: v1
kind: Namespace
metadata:
  name: nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
```

New pods will appear soon thereafter.

```shell
sudo k0s kubectl get pods --namespace nginx
```

```shell
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-66b6c48dd5-8zq7d   1/1     Running   0          10m
nginx-deployment-66b6c48dd5-br4jv   1/1     Running   0          10m
nginx-deployment-66b6c48dd5-sqvhb   1/1     Running   0          10m
```

---

### Networking (`docs/networking.md`)

# Networking

## In-cluster networking

k0s supports any standard [CNI] network provider. For convenience, k0s does come bundled with two built-in providers, [Kube-router] and [Calico].

[CNI]: https://github.com/containernetworking/cni
[Kube-router]: https://github.com/cloudnativelabs/kube-router
[Calico]: https://www.projectcalico.org/

### Custom CNI configuration

You can opt-out of having k0s manage the network setup and choose instead to use any network plugin that adheres to the CNI specification. To do so, configure `custom` as the [network provider] in the k0s configuration file (`k0s.yaml`). You can deploy the CNI provider you want to use, either with Helm, plain Kubernetes manifests or any other way.

[network provider]: configuration.md#specnetwork

### Notes

- When deploying k0s with the default settings, all pods on a node can communicate with all pods on all nodes. No configuration changes are needed to get started.
- Once you initialize the cluster with a network provider the only way to change providers is through a full cluster redeployment.

### Kube-router

Kube-router is built into k0s, and so by default the distribution uses it for network provisioning. Kube-router uses the standard Linux networking stack and tools, and you can set up CNI networking without any overlays by using BGP as the main mechanism for in-cluster networking.

- Uses a bit less resources (~15%)
- Does NOT support Windows nodes

### Calico

In addition to Kube-router, k0s also offers Calico as an alternative,
integrated network provider. Calico is a layer 3 container networking solution
that routes packets to pods. For example, it supports pod-specific network
policies that help secure Kubernetes clusters in demanding use cases. Calico in
k0s uses VXLAN by default. However, IP in IP is also supported via the `bird`
mode.

- Uses a bit more resources
- Supports Windows nodes

## Controller-Worker communication

One goal of k0s is to allow for the deployment of an isolated control plane, which may prevent the establishment of an IP route between controller nodes and the pod network. Thus, to enable this communication path (which is mandated by conformance tests), k0s deploys [Konnectivity service](https://kubernetes.io/docs/tasks/extend-kubernetes/setup-konnectivity/) to proxy traffic from the API server (control plane) into the worker nodes. This ensures that we can always fulfill all the Kubernetes API functionalities, but still operate the control plane in total isolation from the workers.

**Note**: To allow Konnectivity agents running on the worker nodes to establish the connection, configure your firewalls for outbound access, port 8132. Moreover, configure your firewalls for outbound access, port 6443, in order to access Kube-API from the worker nodes.



## Required ports and protocols

| Protocol | Port  | Service        | Direction                     | Notes                                                                                                                                                                                                        |
|----------|-------|----------------|-------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| TCP      | 2380  | etcd           | controller ⟷ controller       |                                                                                                                                                                                                              |
| TCP      | 6443  | kube-apiserver | worker, CLI ⟶ controller      | Authenticated Kubernetes API using mTLS, ServiceAccount tokens with RBAC                                                                                                                                     |
| TCP      | 179   | kube-router    | worker ⟷ worker               | BGP routing sessions between peers                                                                                                                                                                           |
| UDP      | 4789  | calico         | worker ⟷ worker               | Calico VXLAN overlay                                                                                                                                                                                         |
| TCP      | 10250 | kubelet        | controller, worker ⟶ host `*` | Authenticated kubelet API for the controller node `kube-apiserver` (and `metrics-server` add-ons) using mTLS                                                                                                 |
| TCP      | 9443  | k0s api        | controller ⟷ controller       | k0s controller join API, TLS with token auth                                                                                                                                                                 |
| TCP      | 8132  | konnectivity   | worker ⟷ controller           | Konnectivity is used as "reverse" tunnel between kube-apiserver and worker kubelets                                                                                                                          |
| TCP      | 112   | keepalived     | controller ⟷ controller       | Only required for control plane load balancing VRRPInstances. Unless unicast is explicitly enabled, port 122 works on the ip address 224.0.0.18. 224.0.0.18 is a multicast IP address defined in [RFC 3768]. |

You also need enable all traffic to and from the [podCIDR and serviceCIDR] subnets on nodes with a worker role.

[podCIDR and serviceCIDR]: configuration.md#specnetwork
[RFC 3768]: https://datatracker.ietf.org/doc/html/rfc3768#section-5.2.2

## iptables

`iptables` can work in two distinct modes, `legacy` and `nftables`. k0s auto-detects the mode and prefers `nftables`. To check which mode k0s is configured with check `ls -lah /var/lib/k0s/bin/`. The target of the symbolic link `iptables` reveals the mode which k0s selected. k0s has the same logic as other k8s components, but to ensure all component have picked up the same mode you can check via:

- kube-proxy: `nsenter -t $(pidof kube-proxy) -m iptables -V`
- kube-router: `nsenter -t $(pidof kube-router) -m /sbin/iptables -V`
- calico: `nsenter -t $(pidof -s calico-node) -m iptables -V`

There are [known](https://bugzilla.netfilter.org/show_bug.cgi?id=1632) version incompatibility issues in iptables versions. k0s ships (in `/var/lib/k0s/bin`) a version of iptables that is tested to interoperate with all other Kubernetes components it ships with. However if you have other tooling (firewalls etc.) on your hosts that uses iptables and the host iptables version is different from the one that k0s (and other Kubernetes components) ship, it may cause networking issues. This is due to the fact that iptables, being user-space tooling, does not provide any strong version compatibility guarantees.

## Firewalld & k0s

If you are using [firewalld] on your hosts, make sure that it runs with the same
backend (`nftables` or `iptables`) that k0s and the bundled Kubernetes
components use. Otherwise, kube-proxy, Kube-router, or Calico will program rules
in the wrong backend, which will result in a networking failure. You can
configure the active backend via the `FirewallBackend` option in
`/etc/firewalld/firewalld.conf`.

Firewalld is known to be enabled by default in Oracle Linux.

Instead of adding individual ports, create dedicated firewalld services for k0s.
The k0s automated test suite uses the following XML snippets, so using them will
ensure that your configuration is aligned with our automated validation process:

Create the service file for controller nodes in
`/etc/firewalld/services/k0s-controller.xml` as follows:

```xml
{% include "../hack/ostests/modules/os/k0s-controller.firewalld-service.xml" %}
```

Create the service file for worker nodes in
`/etc/firewalld/services/k0s-worker.xml` as follows:

```xml
{% include "../hack/ostests/modules/os/k0s-worker.firewalld-service.xml" %}
```

With the services in place, enable them on the nodes. For nodes running
controller components:

```console
$ sudo firewall-cmd --permanent --add-service=k0s-controller
success
```

For nodes running worker components:

```console
$ sudo firewall-cmd --permanent --add-service=k0s-worker
success
$ sudo firewall-cmd --permanent --add-masquerade
success
```

For nodes that run both controller and worker components, i.e. nodes running
with the `controller --enable-worker` flag, you need to enable both.

Allow traffic from your pod and service networks so that the host accepts
traffic arriving via the overlay interfaces:

```console
$ sudo firewall-cmd --permanent --add-source=<podCIDR>
success
$ sudo firewall-cmd --permanent --add-source=<serviceCIDR>
success
```

Replace `<podCIDR>` and `<serviceCIDR>` with the values configured in
`k0s.yaml`. When you schedule workloads on controller nodes, add both
`k0s-controller` and `k0s-worker` services to the machine. Add `--zone=<zone>`
to the commands if you are configuring a zone other than the default.

Reload firewalld once after applying the permanent changes:

```console
$ sudo firewall-cmd --reload
success
```

For automated image builds where firewalld is not yet running (for example
during cloud-init), use the offline variant:

```console
$ sudo firewall-offline-cmd --add-service=k0s-controller
success
$ sudo firewall-offline-cmd --add-service=k0s-worker
success
$ sudo firewall-offline-cmd --add-masquerade
success
$ sudo systemctl reload firewalld
```

After reloading, you should see the k0s services applied:

```console
$ sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 10.244.0.0/16 10.96.0.0/12
  services: cockpit dhcpv6-client k0s-controller k0s-worker ssh
  ports:
  protocols:
  forward: yes
  masquerade: yes
  forward-ports:
  source-ports:
  icmp-blocks:
  rich rules:
```

[firewalld]: https://firewalld.org/

---

### Nllb (`docs/nllb.md`)

# Node-local load balancing

For clusters that don't have an [externally managed load balancer] for the k0s
control plane, there is another option to get a highly available control plane,
at least from within the cluster. K0s calls this "node-local load balancing". In
contrast to an externally managed load balancer, node-local load balancing takes
place exclusively on the worker nodes. It does not contribute to making the
control plane highly available to the outside world (e.g. humans interacting
with the cluster using management tools such as [Lens](https://k8slens.dev/) or
`kubectl`), but rather makes the cluster itself internally resilient to
controller node outages.

[externally managed load balancer]: high-availability.md#load-balancer

## Technical functionality

The k0s worker process manages a load balancer on each worker node's loopback
interface and configures the relevant components to use that load balancer. This
allows for requests from worker components to the control plane to be
distributed among all currently available controller nodes, rather than being
directed to the controller node that has been used to join a particular worker
into the cluster. This improves the reliability and fault tolerance of the
cluster in case a controller node becomes unhealthy.

Both [Envoy] and [Traefik] are supported as backing load balancers, with Envoy
being the default. Please note that Envoy is unavailable for ARMv7, RISC-V and
Windows. If you plan to run k0s on these platforms, use Traefik instead.

[Envoy]: https://www.envoyproxy.io/
[Traefik]: https://traefik.io/

## Enabling in a cluster

In order to use node-local load balancing, the cluster needs to comply with the
following:

* The cluster doesn't use an externally managed load balancer, i.e. the cluster
  configuration doesn't specify a non-empty
  [`spec.api.externalAddress`][specapi].
* K0s isn't running as a [single node](k0s-single-node.md), i.e. it isn't
  started using the `--single` flag.
* The cluster should have multiple controller nodes. Node-local load balancing
  also works with a single controller node, but is only useful in conjunction
  with a highly available control plane.

Add the following to the cluster configuration (`k0s.yaml`):

```yaml
spec:
  network:
    nodeLocalLoadBalancing:
      enabled: true
      type: EnvoyProxy
```

Or alternatively, if using [`k0sctl`](k0sctl-install.md), add the following to
the k0sctl configuration (`k0sctl.yaml`):

```yaml
spec:
  k0s:
    config:
      spec:
        network:
          nodeLocalLoadBalancing:
            enabled: true
            type: EnvoyProxy
```

All newly added worker nodes will then use node-local load balancing. The k0s
worker process on worker nodes that are already running must be restarted for
the new configuration to take effect.

[specapi]: configuration.md#specapi

## Full example using `k0sctl`

The following example shows a full `k0sctl` configuration file featuring three
controllers and two workers with node-local load balancing enabled:

```yaml
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  k0s:
    version: {{{ k0s_version }}}
    config:
      spec:
        network:
          nodeLocalLoadBalancing:
            enabled: true
            type: EnvoyProxy
  hosts:
    - role: controller
      ssh:
        address: 10.81.146.254
        keyPath: k0s-ssh-private-key.pem
        port: 22
        user: k0s
    - role: controller
      ssh:
        address: 10.81.146.184
        keyPath: k0s-ssh-private-key.pem
        port: 22
        user: k0s
    - role: controller
      ssh:
        address: 10.81.146.113
        keyPath: k0s-ssh-private-key.pem
        port: 22
        user: k0s
    - role: worker
      ssh:
        address: 10.81.146.198
        keyPath: k0s-ssh-private-key.pem
        port: 22
        user: k0s
    - role: worker
      ssh:
        address: 10.81.146.51
        keyPath: k0s-ssh-private-key.pem
        port: 22
        user: k0s
```

Save the above configuration into a file called `k0sctl.yaml` and apply it in
order to bootstrap the cluster:

```
/* Detailed source-code truncated for AI context efficiency. */
```

The cluster with the two nodes should be available by now. Setup the kubeconfig
file in order to interact with it:

```shell
k0sctl kubeconfig > k0s-kubeconfig
export KUBECONFIG=$(pwd)/k0s-kubeconfig
```

The three controllers are available and provide API Server endpoints:

```console
$ kubectl -n kube-node-lease get \
    lease/k0s-ctrl-k0s-controller-0 \
    lease/k0s-ctrl-k0s-controller-1 \
    lease/k0s-ctrl-k0s-controller-2 \
    lease/k0s-endpoint-reconciler
NAME                        HOLDER                                                             AGE
k0s-ctrl-k0s-controller-0   9ec2b221890e5ed6f4cc70377bfe809fef5be541a2774dc5de81db7acb2786f1   2m37s
k0s-ctrl-k0s-controller-1   fe45284924abb1bfce674e5a9aa8d647f17c81e53bbab17cf28288f13d5e8f97   2m18s
k0s-ctrl-k0s-controller-2   5ab43278e63fc863b2a7f0fe1aab37316a6db40c5a3d8a17b9d35b5346e23b3d   2m9s
k0s-endpoint-reconciler     9ec2b221890e5ed6f4cc70377bfe809fef5be541a2774dc5de81db7acb2786f1   2m37s

$ kubectl -n default get endpoints
NAME         ENDPOINTS                                                  AGE
kubernetes   10.81.146.113:6443,10.81.146.184:6443,10.81.146.254:6443   2m49s
```

The first controller is the current k0s leader. The two worker nodes can be
listed, too:

{% set kubelet_ver = k8s_version + '+k0s' -%}
{% set kubelet_ver_len = kubelet_ver | length -%}

```console
$ kubectl get nodes -owide
NAME           STATUS   ROLES    AGE     {{{ 'VERSION'   | ljust(kubelet_ver_len) }}}   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
k0s-worker-0   Ready    <none>   2m16s   {{{ kubelet_ver | ljust(kubelet_ver_len) }}}   10.81.146.198   <none>        Alpine Linux v3.17   5.15.83-0-virt   containerd://{{{ build_var('containerd_version') }}}
k0s-worker-1   Ready    <none>   2m15s   {{{ kubelet_ver | ljust(kubelet_ver_len) }}}   10.81.146.51    <none>        Alpine Linux v3.17   5.15.83-0-virt   containerd://{{{ build_var('containerd_version') }}}
```

There is one node-local load balancer pod running for each worker node:

```console
$ kubectl -n kube-system get pod -owide -l app.kubernetes.io/managed-by=k0s,app.kubernetes.io/component=nllb
NAME                READY   STATUS    RESTARTS   AGE   IP              NODE           NOMINATED NODE   READINESS GATES
nllb-k0s-worker-0   1/1     Running   0          81s   10.81.146.198   k0s-worker-0   <none>           <none>
nllb-k0s-worker-1   1/1     Running   0          85s   10.81.146.51    k0s-worker-1   <none>           <none>
```

The cluster is using node-local load balancing and is able to tolerate the
outage of one controller node. Shutdown the first controller to simulate a
failure condition:

```console
$ ssh -i k0s-ssh-private-key.pem k0s@10.81.146.254 'echo "Powering off $(hostname) ..." && sudo poweroff'
Powering off k0s-controller-0 ...
```

Node-local load balancing provides high availability from within the cluster,
not from the outside. The generated kubeconfig file lists the first controller's
IP as the Kubernetes API server address by default. As this controller is gone
by now, a subsequent call to `kubectl` will fail:

```console
$ kubectl get nodes
Unable to connect to the server: dial tcp 10.81.146.254:6443: connect: no route to host
```

Changing the server address in `k0s-kubeconfig` from the first controller to
another one makes the cluster accessible again. Pick one of the other controller
IP addresses and put that into the kubeconfig file. The addresses are listed
both in `k0sctl.yaml` as well as in the output of `kubectl -n default get
endpoints` above.

```console
$ ssh -i k0s-ssh-private-key.pem k0s@10.81.146.184 hostname
k0s-controller-1

$ sed -i s#https://10\\.81\\.146\\.254:6443#https://10.81.146.184:6443#g k0s-kubeconfig

$ kubectl get nodes -owide
NAME           STATUS   ROLES    AGE     {{{ 'VERSION'   | ljust(kubelet_ver_len) }}}   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION   CONTAINER-RUNTIME
k0s-worker-0   Ready    <none>   3m35s   {{{ kubelet_ver | ljust(kubelet_ver_len) }}}   10.81.146.198   <none>        Alpine Linux v3.17   5.15.83-0-virt   containerd://{{{ build_var('containerd_version') }}}
k0s-worker-1   Ready    <none>   3m34s   {{{ kubelet_ver | ljust(kubelet_ver_len) }}}   10.81.146.51    <none>        Alpine Linux v3.17   5.15.83-0-virt   containerd://{{{ build_var('containerd_version') }}}

$ kubectl -n kube-system get pods -owide -l app.kubernetes.io/managed-by=k0s,app.kubernetes.io/component=nllb
NAME                READY   STATUS    RESTARTS   AGE     IP              NODE           NOMINATED NODE   READINESS GATES
nllb-k0s-worker-0   1/1     Running   0          2m31s   10.81.146.198   k0s-worker-0   <none>           <none>
nllb-k0s-worker-1   1/1     Running   0          2m35s   10.81.146.51    k0s-worker-1   <none>           <none>
```

The first controller is no longer active. Its IP address is not listed in the
`default/kubernetes` Endpoints resource and its k0s controller lease is orphaned:

```console
$ kubectl -n default get endpoints
NAME         ENDPOINTS                               AGE
kubernetes   10.81.146.113:6443,10.81.146.184:6443   3m56s

$ kubectl -n kube-node-lease get \
    lease/k0s-ctrl-k0s-controller-0 \
    lease/k0s-ctrl-k0s-controller-1 \
    lease/k0s-ctrl-k0s-controller-2 \
    lease/k0s-endpoint-reconciler
NAME                        HOLDER                                                             AGE
k0s-ctrl-k0s-controller-0                                                                      4m47s
k0s-ctrl-k0s-controller-1   fe45284924abb1bfce674e5a9aa8d647f17c81e53bbab17cf28288f13d5e8f97   4m28s
k0s-ctrl-k0s-controller-2   5ab43278e63fc863b2a7f0fe1aab37316a6db40c5a3d8a17b9d35b5346e23b3d   4m19s
k0s-endpoint-reconciler     5ab43278e63fc863b2a7f0fe1aab37316a6db40c5a3d8a17b9d35b5346e23b3d   4m47s
```

Despite that controller being unavailable, the cluster remains operational. The
third controller has become the new k0s leader. Workloads will run just fine:

```console
$ kubectl -n default run nginx --image=nginx
pod/nginx created

$ kubectl -n default get pods -owide
NAME    READY   STATUS    RESTARTS   AGE   IP           NODE           NOMINATED NODE   READINESS GATES
nginx   1/1     Running   0          16s   10.244.0.5   k0s-worker-1   <none>           <none>

$ kubectl -n default logs nginx
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
[notice] 1#1: using the "epoll" event method
[notice] 1#1: nginx/1.23.3
[notice] 1#1: built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
[notice] 1#1: OS: Linux 5.15.83-0-virt
[notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
[notice] 1#1: start worker processes
[notice] 1#1: start worker process 28
```

---

### Podsecurity (`docs/podsecurity.md`)

# Pod Security Standards

Since Pod Security Policies have been removed in Kubernetes v1.25, Kubernetes
offers [Pod Security Standards] – a new way to enhance cluster security.

To enable Pod Security Standards in k0s, create an admission controller
configuration file:

    ```yaml
    apiVersion: apiserver.config.k8s.io/v1
    kind: AdmissionConfiguration
    plugins:
    - name: PodSecurity
      configuration:
        apiVersion: pod-security.admission.config.k8s.io/v1beta1
        kind: PodSecurityConfiguration
        # Defaults applied when a mode label is not set.
        defaults:
          enforce: "privileged"
          enforce-version: "latest"
        exemptions:
          # Don't forget to exempt namespaces or users that are responsible for deploying
          # cluster components, because they need to run privileged containers
          usernames: ["admin"]
          namespaces: ["kube-system"]
    ```

Add these extra arguments to the k0s configuration:

    ```yaml
    apiVersion: k0s.k0sproject.io/v1beta1
    kind: ClusterConfig
    spec:
      api:
        extraArgs:
          admission-control-config-file: /path/to/admission/control/config.yaml
    ```

[Pod Security Standards]: https://kubernetes.io/docs/concepts/security/pod-security-standards/

---

### Raspberry Pi4 (`docs/raspberry-pi4.md`)

# Create a Raspberry Pi 4 cluster

## Prerequisites

This guide assumes that you use a [Raspberry Pi 4 Model B] computer and a
sufficiently large SD card of at least 32 GB. We will be using [Ubuntu Linux]
for this guide, although k0s should run quite fine on other 64-bit Linux
distributions for the Raspberry Pi as well. Please [file a Bug] if you encounter
any obstacles.

[Raspberry Pi 4 Model B]: https://www.raspberrypi.org/products/raspberry-pi-4-model-b/
[Ubuntu Linux]: https://ubuntu.com/
[file a Bug]: https://github.com/k0sproject/k0s/issues/new?assignees=&labels=bug&template=BUG_REPORT.yml

## Set up the system

### Prepare SD card and boot up the Raspberry Pi

Install [Ubuntu Server 22.04.1 LTS 64-bit for Raspberry Pi][ubuntu-dl]. Ubuntu
provides a [step by step guide][ubuntu-pi] for the installation process. They
use [Raspberry Pi Imager], a specialized imaging utility that you can use to
write the Ubuntu image, amongst others, to your SD cards. Follow that
[guide][ubuntu-pi] to get a working installation. (You can skip part 5 of the
guide, since we won't need a Desktop Environment to run k0s.)

Alternatively, you can also opt to [download][2204-dl] the Ubuntu server image
for Raspberry Pi manually and write it to an SD card using a tool like `dd`:

```console
wget https://cdimage.ubuntu.com/releases/22.04.1/release/ubuntu-22.04.1-preinstalled-server-arm64+raspi.img.xz
unxz ubuntu-22.04.1-preinstalled-server-arm64+raspi.img.xz
dd if=ubuntu-22.04.1-preinstalled-server-arm64+raspi.img of=/dev/mmcblk0 bs=4M status=progress
```

**Note**: The manual process is more prone to accidental data loss than the
guided one via Raspberry Pi Imager. Be sure to choose the correct device names.
The previous content of the SD card will be wiped. Moreover, the partition
written to the SD card needs to be resized to make the full capacity of the card
available to Ubuntu. This can be achieved, for example, in this way:

```console
growpart /dev/mmcblk0 2
resize2fs /dev/mmcblk0p2
```

Ubuntu uses [cloud-init] to allow for automated customizations of the system
configuration. The cloud-init configuration files are located on the boot
partition of the SD card. You can mount that partition and modify those, e.g. to
provision network configuration, users, authorized SSH keys, additional packages
and also an automatic installation of k0s.

After you have prepared the SD card, plug it into the Raspberry Pi and boot it
up. Once cloud-init finished bootstrapping the system, the default login
credentials are set to user `ubuntu` with password `ubuntu` (which you will be
prompted to change on first login).

[ubuntu-dl]: https://ubuntu.com/download/raspberry-pi
[ubuntu-pi]: https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi
[Raspberry Pi Imager]: https://www.raspberrypi.org/blog/raspberry-pi-imager-imaging-utility/
[2204-dl]: https://ubuntu.com/download/raspberry-pi/thank-you?version=22.04.1&architecture=server-arm64+raspi
[cloud-init]: https://cloud-init.io/

### Review network configurations

**Note**: For network configuration purposes, this documentation assumes that all
of your computers are connected on the same subnet.

Review k0s's [required ports and protocols] to ensure that your network and
firewall configurations allow necessary traffic for the cluster.

Review the [Ubuntu Server Networking Configuration] documentation to ensure that
all systems have a static IP address on the network, or that the network is
providing a static DHCP lease for the nodes. If the network should be managed
via cloud-init, please refer to [their documentation][cloud-init-network].

[required ports and protocols]: networking.md#required-ports-and-protocols
[Ubuntu Server Networking Configuration]: https://ubuntu.com/server/docs/network-configuration
[cloud-init-network]: https://cloudinit.readthedocs.io/en/latest/topics/network-config-format-v2.html#examples

### (Optional) Provision SSH keys

Ubuntu Server deploys and enables [OpenSSH](https://www.openssh.com/) via
cloud-init by default. Confirm, though, that for whichever user you will deploy
the cluster with on the build system, their SSH Key is [copied to each node's
root user][copy-ssh-key]. Before you start, the configuration should be such
that the current user can run:

```shell
ssh root@${HOST}
```

Where `${HOST}` is any node and the login can succeed with no further prompts.

[copy-ssh-key]: https://www.cyberciti.biz/faq/use-ssh-copy-id-with-an-openssh-server-listing-on-a-different-port/

### (Optional) Create a swap file

While having a swap file is _technically optional_, it can help to ease memory
pressure when running memory intensive workloads or on Raspberry Pis with less
than 8 GB of RAM.

1. To create a swap file:

    ```shell
    fallocate -l 2G /swapfile && \
    chmod 0600 /swapfile && \
    mkswap /swapfile && \
    swapon -a
    ```

2. Ensure that the usage of swap is not too aggressive by setting the `sudo
   sysctl vm.swappiness=10` (the default is generally higher) and configuring it
   to be persistent in `/etc/sysctl.d/*`.

3. Ensure that your swap is mounted after reboots by confirming that the
   following line exists in your `/etc/fstab` configuration:

    ```shell
    /swapfile         none           swap sw       0 0
    ```

## Download k0s

Download a [k0s release](https://github.com/k0sproject/k0s/releases/latest). For
example:

```shell
wget -O /tmp/k0s https://github.com/k0sproject/k0s/releases/download/{{{ k0s_version }}}/k0s-{{{ k0s_version }}}-arm64 # replace version number!
sudo install /tmp/k0s /usr/local/bin/k0s
```

― or ―

Use the k0s download script (as one command) to download the latest stable k0s
and make it executable in `/usr/bin/k0s`.

```shell
curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh
```

At this point you can run `k0s`:

```console
ubuntu@ubuntu:~$ k0s version
{{{ k0s_version }}}
```

To check if k0s's [system requirements](system-requirements.md) and [external
runtime dependencies](external-runtime-deps.md) are fulfilled by your current
setup, you can invoke `k0s sysinfo`:

```
/* Detailed source-code truncated for AI context efficiency. */
```

## Deploy a node

Each node can now serve as a control plane node or worker node or both.

### As single node

This is a self-contained single node setup which runs both control plane
components and worker components. If you don't plan join any more nodes into the
cluster, this is for you.

Install the `k0scontroller` service:

```console
ubuntu@ubuntu:~$ sudo k0s install controller --single
ubuntu@ubuntu:~$ sudo systemctl status k0scontroller.service
○ k0scontroller.service - k0s - Zero Friction Kubernetes
     Loaded: loaded (/etc/systemd/system/k0scontroller.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: https://docs.k0sproject.io
```

Start it:

```
/* Detailed source-code truncated for AI context efficiency. */
```

{% set kubelet_ver = k8s_version + '+k0s' -%}
{% set kubelet_ver_len = kubelet_ver | length -%}

When the cluster is up, try to have a look:

```console
ubuntu@ubuntu:~$ sudo k0s kc get nodes -owide
NAME     STATUS   ROLES           AGE     {{{ 'VERSION'   | ljust(kubelet_ver_len) }}}   INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
ubuntu   Ready    control-plane   4m41s   {{{ kubelet_ver | ljust(kubelet_ver_len) }}}   10.152.56.54   <none>        Ubuntu 22.04.1 LTS   5.15.0-1013-raspi   containerd://{{{ build_var('containerd_version') }}}
ubuntu@ubuntu:~$ sudo k0s kc get pod -owide -A
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE     IP             NODE     NOMINATED NODE   READINESS GATES
kube-system   kube-proxy-kkv2l                  1/1     Running   0          4m44s   10.152.56.54   ubuntu   <none>           <none>
kube-system   kube-router-vf2pv                 1/1     Running   0          4m44s   10.152.56.54   ubuntu   <none>           <none>
kube-system   coredns-88b745646-wd4mp           1/1     Running   0          5m10s   10.244.0.2     ubuntu   <none>           <none>
kube-system   metrics-server-7d7c4887f4-ssk49   1/1     Running   0          5m6s    10.244.0.3     ubuntu   <none>           <none>
```

Overall, the single k0s node uses less than 1 GiB of RAM:

```console
ubuntu@ubuntu:~$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.7Gi       715Mi       1.3Gi       3.0Mi       1.7Gi       2.8Gi
Swap:             0B          0B          0B
```

### As a controller node

This will install k0s as a single non-HA controller. It won't be able to run any
workloads, so you need to connect more workers to it.

Install the `k0scontroller` service. Note that we're not specifying any flags:

```console
ubuntu@ubuntu:~$ sudo k0s install controller
ubuntu@ubuntu:~$ systemctl status k0scontroller.service
○ k0scontroller.service - k0s - Zero Friction Kubernetes
     Loaded: loaded (/etc/systemd/system/k0scontroller.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: https://docs.k0sproject.io
```

Start it:

```
/* Detailed source-code truncated for AI context efficiency. */
```

As soon as the controller is up, we can try to inspect the API as we did for the
single node:

```console
ubuntu@ubuntu:~$ sudo k0s kc get nodes -owide
No resources found
ubuntu@ubuntu:~$ sudo k0s kc get pod -owide -A
NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
kube-system   coredns-88b745646-6tpwm           0/1     Pending   0          29s   <none>   <none>   <none>           <none>
kube-system   metrics-server-7d7c4887f4-9k5k5   0/1     Pending   0          24s   <none>   <none>   <none>           <none>
```

As we see, there are no nodes and two pending pods. A control plane without
workers. The memory consumption is below the single node controller, but not
much:

```console
ubuntu@ubuntu:~$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.7Gi       678Mi       2.3Gi       3.0Mi       758Mi       2.9Gi
Swap:             0B          0B          0B
```

This controller runs a full-fledged control plane, backed by [etcd], as opposed
to the lightweight [kine] based one from the single node example. For the
latter, k0s doesn't support joining new nodes.

More nodes can be added by creating join tokens. To add a worker node, create a
token for it:

```
/* Detailed source-code truncated for AI context efficiency. */
```

Save the join token for subsequent steps.

[etcd]: https://etcd.io
[kine]: https://github.com/k3s-io/kine

### As a worker node

To join an existing k0s cluster, create the join token file for the worker
(where `$TOKEN_CONTENT` is one of the join tokens created in the control plane
setup):

```console
sudo sh -c 'mkdir -p /var/lib/k0s/ && umask 077 && echo "$TOKEN_CONTENT" > /var/lib/k0s/join-token'
```

After that, install the `k0sworker` service:

```console
ubuntu@ubuntu:~$ sudo k0s install worker --token-file /var/lib/k0s/join-token
ubuntu@ubuntu:~$ systemctl status k0sworker.service
○ k0sworker.service - k0s - Zero Friction Kubernetes
     Loaded: loaded (/etc/systemd/system/k0sworker.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
       Docs: https://docs.k0sproject.io
```

Start the service:

```
/* Detailed source-code truncated for AI context efficiency. */
```

As this is a worker node, we cannot access the Kubernetes API via the builtin
`k0s kc` subcommand, but we can check the k0s API instead:

```console
ubuntu@ubuntu:~$ sudo k0s status
Version: {{{ k0s_version }}}
Process ID: 1631
Role: worker
Workloads: true
SingleNode: false
```

The memory requirements are also pretty low:

```console
ubuntu@ubuntu:~$ free -h
               total        used        free      shared  buff/cache   available
Mem:           3.7Gi       336Mi       2.1Gi       3.0Mi       1.2Gi       3.2Gi
Swap:             0B          0B          0B
```

## Connect to the cluster

On a controller node, generate a new `raspi-cluster-master` user with admin
rights and get a [kubeconfig] for it:

```
/* Detailed source-code truncated for AI context efficiency. */
```

Using the above kubeconfig, you can now access and use the cluster:

```
/* Detailed source-code truncated for AI context efficiency. */
```

[kubeconfig]: https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/

---

### Raspberry Pi5 (`docs/raspberry-pi5.md`)

# Create a Raspberry Pi 5 cluster

## Prerequisites

This guide assumes that you have a [Raspberry Pi 5] and a
sufficiently large SD card of at least 32 GB. We will be using [Raspberry Pi OS]
for this guide, though k0s should run just fine on other 64-bit Linux
distributions for the Raspberry Pi 5 as well. Please [file a Bug] if you encounter
any issues.

[Raspberry Pi 5]: https://www.raspberrypi.com/products/raspberry-pi-5/
[Raspberry Pi OS]: https://www.raspberrypi.com/software/
[file a Bug]: https://github.com/k0sproject/k0s/issues/new?assignees=&labels=bug&template=BUG_REPORT.yml

## Set up the system

Follow the official Raspberry Pi instructions on [booting Pi OS from the USB drive](https://projects.raspberrypi.org/en/projects/install-raspberry-pi-desktop/3) or another method depending on your needs.

### SSH Access

1. Typically, you can enable SSH while creating the bootable drive for Raspberry Pi OS. If not, you can enable it later

2. Add your public key to the server.
    From your host machine, run [ssh-copy-id] to copy your public SSH key to your Pi 5:

    ```bash
    ssh-copy-id -i ~/.ssh/id_rsa.pub <YOUR_USER_NAME>@<IP_ADDRESS_OF_THE_SERVER>
    ```

    When prompted, enter the password for your user account for the Pi. Your public key should be copied at the appropriate folder on the remote Pi automatically.

    Note: `~/.ssh/id_rsa.pub` is the default location for the public ssh key. If your key is elsewhere, adjust accordingly.

3. Verify SSH access

    ```shell
    ssh <YOUR_USER_NAME>@<IP_ADDRESS_OF_THE_SERVER>
    ```

    If your key has a paraphrase, you’ll be prompted for it.

[ssh-copy-id]: https://www.cyberciti.biz/faq/use-ssh-copy-id-with-an-openssh-server-listing-on-a-different-port/

### Enable the memory cgroup controller

Raspberry Pi OS does not enable the memory cgroup controller by default. However, it is required to run containerized workloads, so enable it:

1. Edit /boot/cmdline.txt:

    ```bash
    sudo nano /boot/cmdline.txt
    ```

    Append (on the same single line):

    ```bash
    cgroup_enable=memory cgroup_memory=1
    ```

2. Reboot

    ```bash
    sudo reboot
    ```

## Install k0s

### Download k0s

Download a [k0s release](https://github.com/k0sproject/k0s/releases/latest). For
example:

  ```shell
  wget -O /tmp/k0s https://github.com/k0sproject/k0s/releases/download/{{{ k0s_version }}}/k0s-{{{ k0s_version }}}-arm64 # replace version number!
  sudo install /tmp/k0s /usr/local/bin/k0s
  ```

― or ―

Use the k0s download script (as one command) to download the latest stable k0s
and make it executable in `/usr/bin/k0s`.

  ```shell
  curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh
  ```

At this point you can run `k0s`:

  ```console
  $ sudo k0s version
  {{{ k0s_version }}}
  ```

To check if k0s's [system requirements](system-requirements.md) and [external
runtime dependencies](external-runtime-deps.md) are fulfilled by your current
setup, you can invoke `k0s sysinfo`:

  ```
/* Detailed source-code truncated for AI context efficiency. */
```

### Deploy a Node using k0s

If you want a more hands-on process for setting up your Pi 5 devices as Kubernetes nodes with the k0s binary, refer to [the guide for Pi devices](./raspberry-pi4.md).

## Deploy a Node using k0sctl

### Install k0sctl on Your Host

Follow [the k0sctl installation guide](https://github.com/k0sproject/k0sctl#installation) and install k0sctl on your host machine.

### Single node K0s cluster

For this example, we'll create a cluster.yaml that describes your known Pi 5 device and use it as a single node (controller & worker) cluster, for example:

  ```yaml
  apiVersion: k0sctl.k0sproject.io/v1beta1
  kind: Cluster
  metadata:
    name: k0s-cluster
    user: admin
  spec:
    hosts:
    - ssh:
        address: <IP_ADDRESS_OF_THE_SERVER>
        user: <YOUR_USER_NAME>
        port: 22
        keyPath: ~/.ssh/id_rsa
      role: controller+worker
  ```

### SSH agent

  By default, k0sctl doesn’t prompt you for passphrases, so the easiest solution is to load your key into an SSH agent before running k0sctl. Here’s how you can do it:

  1. Start the SSH agent (if not already running)

      ```bash
      eval "$(ssh-agent -s)"
      ```

  2. Add your private key (you’ll be prompted for the passphrase)

      ```bash
      ssh-add ~/.ssh/id_rsa
      ```

  3. Verify the key is loaded

      ```bash
      ssh-add -l
      ```

### Deploy cluster

  1. Apply the cluster.yaml using k0sctl on your local machine.

      ```
/* Detailed source-code truncated for AI context efficiency. */
```

  2. Fetch the kubeconfig use k0sctl.

      ```bash
        k0sctl kubeconfig --config cluster.yaml > pi_cluster.kubeconfig
      ```

  3. Export KUBECONFIG and verify

      ```console
      $ export KUBECONFIG=pi_cluster.kubeconfig
      $ kubectl get nodes
      NAME   STATUS   ROLES           AGE     VERSION
      pi     Ready    control-plane   2m54s   {{{ k8s_version }}}+k0s
      ```

## Tear down k0s on Pi 5

If you need to remove k0s entirely (for example, if you run into conflicts or just want a clean slate):

  1. Stop existing processes

      ```bash
      sudo k0s stop
      ```

  2. Reset k0s

      ```bash
      sudo k0s reset
      ```

  3. Remove k0s binaries

      ```bash
      sudo rm -rf /usr/local/bin/k0s
      ```

  4. Reboot Pi 5.

      ```bash
      sudo reboot
      ```

---

### Reinstall K0sctl (`docs/reinstall-k0sctl.md`)

# Reinstall a node

`k0sctl` currently does not support changing all the configuration of containerd (`state`, `root`) on the fly.

For example, in order to move containerd's `root` directory to a new partition/drive, you have to provide `--data-dir /new/drive` in your k0sctl `installFlags` for each (worker) node. `--data-dir` is an option of `k0s` and then added to the service unit.

The following is an example of that:

```yaml
# spec.hosts[*].installFlags
  - role: worker
    installFlags:
      - --profile flatcar
      - --enable-cloud-provider
      - --data-dir /new/drive
      - --kubelet-extra-args="--cloud-provider=external"
```

However, the `installFlags` are only used when the node is installed.

## Steps

Drain the node:

```shell
kubectl drain node.hostname
```

Access your node (e.g. via ssh) to stop and reset k0s:

```shell
sudo k0s stop
sudo k0s reset
```

Reboot the node (for good measure):

```shell
sudo systemctl reboot
```

Once the node is available again, run `k0sctl apply` to integrate it into your cluster and uncordon the node to allow pods to be scheduled:

```shell
k0sctl apply -c config.yaml
kubectl uncordon node.hostname
```

---

### Releases (`docs/releases.md`)

# Releases

This page describes how we release and support the k0s project. [Mirantis Inc.](https://mirantis.com) can also provide [commercial support](commercial-support.md) for k0s.

## Upstream Kubernetes release & support cycle

{% set version_parts = k0s_version.split('.', maxsplit=2) -%}
{% set k0s_minor = version_parts[0] + '.' + version_parts[1] -%}
{% set k0s_prev_minor = version_parts[0] + '.' ~ ((version_parts[1] | int) - 1) -%}

This [release and support cycle](https://kubernetes.io/releases/) is followed
for ALL new minor releases. A minor release can be e.g. {{{ k0s_prev_minor }}},
{{{ k0s_minor }}} and so on. What this means in practice is that every 4 months
there is a new minor release published.

After a minor release is published, the upstream community is maintaining it for 14 months. Maintenance in this case means that upstream Kubernetes provides bug fixes, CVE mitigations and such for 14 months per minor release.



## k0s release and support model

Starting from the k0s 1.21, k0s started following the Kubernetes project's [release and support model](https://kubernetes.io/releases/).

k0s project follows closely the upstream Kubernetes release cycle. The only difference to upstream Kubernetes release / maintenance schedule is that our initial release date is always a few weeks behind the upstream Kubernetes version release date as we are building our version of k0s from the officially released version of Kubernetes and need time for testing the final version before shipping.



Given the fact that upstream Kubernetes provides support and patch releases for a minor version for roughly 14 months, it means that k0s will follow this same model. Each minor release is maintained for roughly 14 months since its initial release.

k0s project will typically include patches and fixes included in a Kubernetes
upstream patch release for the fixes needed in k0s own codebase. For example, if
a bug is identified in {{{ k0s_minor }}} series k0s project will create and ship
a fix for it with the next upstream Kubernetes {{{ k0s_minor }}}.x release. In
rare cases where a critical bug is identified we may also ship "out of band"
patches. Such out-of-band release would be identified in the version string
suffix. For example a normal release following Kubernetes upstream would be {{{
k0s_minor }}}.0+k0s.0 whereas a critical out-of-band patch would be identified as
{{{ k0s_minor }}}.0+k0s.1.

## New features and enhancements

The biggest new k0s features will typically only be delivered on top of the latest Kubernetes version, but smaller enhancements can be included in older release tracks as well.

## Version string

{% set version_parts = k0s_version.split('+', maxsplit=1) -%}

The k0s version string consists of the Kubernetes version and the k0s version. For example:

```text
{{{ version_parts | join('+') }}}
```

The Kubernetes version (`{{{ version_parts[0] }}}`) is the first part, and the
last part (`{{{ version_parts[1] }}}`) reflects the k0s version, which is built
on top of the certain Kubernetes version.

---

### Remove Controller (`docs/remove_controller.md`)

# Remove or replace a controller

You can manually remove or replace a controller from a multi-node k0s cluster (>=3 controllers) without downtime.
However, you have to [maintain quorum on Etcd](https://etcd.io/docs/v3.3/faq/#why-an-odd-number-of-cluster-members) while doing so.

## Remove a controller

If your controller is also a worker (`k0s controller --enable-worker`), you first have to delete the controller from Kubernetes itself.
To do so, run the following commands from the controller:

```shell
# Remove the containers from the node and cordon it
k0s kubectl drain --ignore-daemonsets --delete-emptydir-data <controller>
# Delete the node from the cluster
k0s kubectl delete node <controller>
```

Delete Autopilot's `ControlNode` object for the controller node:

```console
k0s kubectl delete controlnode.autopilot.k0sproject.io <controller>
```

Then you need to remove it from the Etcd cluster.
For example, if you want to remove `controller01` from a cluster with 3 controllers:

```shell
# First, list the Etcd members
k0s etcd member-list
{"members":{"controller01":"<PEER_ADDRESS1>", "controller02": "<PEER_ADDRESS2>", "controller03": "<PEER_ADDRESS3>"}}
# Then, remove the controller01 using its peer address
k0s etcd leave --peer-address "<PEER_ADDRESS1>"
```

The controller is now removed from the cluster.
To [reset k0s on the machine](reset.md), run the following commands:

```shell
k0s stop
k0s reset
reboot
```

### Declarative Etcd member management

Starting from version 1.30, k0s also supports a declarative way to remove an
etcd member. Since in k0s the etcd cluster is set up so that the etcd API is
**NOT** exposed outside the nodes, it makes it difficult for external automation
like Cluster API, Terraform, etc. to handle controller node replacements.

Each controller manages their own `EtcdMember` object.

```shell
k0s kubectl get etcdmember
NAME          PEER ADDRESS   MEMBER ID           JOINED   RECONCILE STATUS
controller0   172.17.0.2     b8e14bda2255bc24    True
controller1   172.17.0.3     cb242476916c8a58    True
controller2   172.17.0.4     9c90504b1bc867bb    True
```

By marking an `EtcdMember` object to leave the etcd cluster, k0s will handle the
interaction with etcd. For example, in a 3 controller HA setup, you can
remove a member by flagging it to leave:

```console
$ kubectl patch etcdmember controller2 -p '{"spec":{"leave":true}}' --type merge
etcdmember.etcd.k0sproject.io/controller2 patched
```

The join/leave status is tracked in the object's conditions. This allows you to
wait for the leave to actually happen:

```console
$ kubectl wait etcdmember controller2 --for condition=Joined=False
etcdmember.etcd.k0sproject.io/controller2 condition met
```

You'll see the node left etcd cluster:

```console
$ k0s kc get etcdmember
NAME          PEER ADDRESS   MEMBER ID           JOINED   RECONCILE STATUS
controller0   172.17.0.2     b8e14bda2255bc24    True
controller1   172.17.0.3     cb242476916c8a58    True
controller2   172.17.0.4     9c90504b1bc867bb    False    Success
```

```console
$ k0s etcd member-list
{"members":{"controller0":"https://172.17.0.2:2380","controller1":"https://172.17.0.3:2380"}}
```

The objects for members that have already left the etcd cluster are kept
available for tracking purposes. Once the member has left the cluster, the
object status will reflect that it is safe to remove it.

**Note:** If you re-join same node without removing the corresponding `etcdmember` object the desired state will be updated back to `spec.leave: false` automatically. This is since currently in k0s there's no easy way to prevent a node joining etcd cluster.

## Replace a controller

To replace a controller, you first remove the old controller (like described above) then follow the [manual installation procedure](k0s-multi-node.md) to add the new one.

---

### Requirements (`docs/requirements.txt`)

babel==2.18.0
backrefs==8.1
bracex==3.1
certifi==2026.7.22
charset-normalizer==3.4.7
click==8.5.0
colorama==0.4.6
ghp-import==2.1.0
hjson==3.1.0
idna==3.20
Jinja2==3.1.6
Markdown==3.11
MarkupSafe==3.0.4
mdx-truly-sane-lists==1.3
mergedeep==1.3.4
mike==2.2.0
mkdocs==1.6.1
mkdocs-exclude==1.0.2
mkdocs-get-deps==0.2.2
mkdocs-include-markdown-plugin==7.3.0
mkdocs-macros-plugin==1.5.0
mkdocs-material==9.7.7
mkdocs-material-extensions==1.3.1
packaging==26.3
paginate==0.5.7
pathspec==1.1.1
platformdirs==4.13.0
Pygments==2.21.0
pymdown-extensions==12.2
pyparsing==3.3.3
python-dateutil==2.9.0.post0
PyYAML==6.0.3
pyyaml_env_tag==1.1
requests==2.34.2
six==1.17.0
super_collections==0.6.2
termcolor==3.3.0
urllib3==2.7.0
verspec==0.1.0
watchdog==6.0.0
wcmatch==11.1

---

### Reset (`docs/reset.md`)

# Reset (Uninstall)

Resetting k0s nodes essentially reverts them to a pre-k0s state. This operation
is particularly useful in scenarios where you need to cleanly remove k0s from
nodes, either for troubleshooting purposes or as a precursor to a fresh
installation. It is designed to remove all components, configurations, and data
associated with k0s from a node. This includes, but is not limited to, the
following:

* Processes and containers: Terminates all running k0s processes to ensure that
  there are no active components left. This includes all container processes
  managed by the Container Runtime.
* Mounts under k0s data directory: In order to prevent persistent data to be
  deleted, all mount points under k0s' data directory will be unmounted. If an
  unmount fails, it will be unmounted lazy.
* Data stored on the node: Deletes the whole k0s data directory, which includes
  * all k0s-related configuration files, including those used for cluster setup
    and node-specific settings,
  * all data stores managed by k0s, including etcd or kine data for cluster
    state and any temporary files created during operation,
  * and all certificates and other key material generated by k0s for node
    authentication and communication within the cluster.
* Network settings: Reverts any network configurations made by k0s, such as
  network interfaces or iptables rules set up specifically for cluster
  communication. This is done on a best effort basis. It's recommended that you
  reboot the host after a reset to ensure that there are no k0s remnants in the
  host's network configuration. Custom CNI plugins are not cleaned up.
* Registration with the host's init system: Reverts the registration done by
  `k0s install`. After a reset, k0s won't be automatically started when the
  host boots.

After a successful reset, the k0s binary itself remains. It can then be used to
join another cluster or create a new one.

It's important to understand that performing a reset is a destructive operation
that will result in the loss of all data and configurations related to k0s on
the affected nodes. Therefore, it should be performed with care, ideally after
ensuring that all necessary data has been backed up or that the cluster can be
safely disassembled without data loss.

K0s can be reset locally on a host using the `k0s reset` command. Whole clusters
can be reset remotely with the `k0sctl reset` command.

## Reset a k0s node locally

To ensure operational safety, `k0s reset` includes a safeguard that prevents it
from being executed while k0s is running, so it must be stopped first:

1. Stop the service:

    ```console
    sudo k0s stop
    ```

2. Invoke the `reset` command:

    ```console
    $ sudo k0s reset
    WARN[2024-03-28 09:15:36] To ensure a full reset, a node reboot is recommended.
    ```

## Reset a k0s cluster remotely using k0sctl

K0sctl can be used to connect and reset all cluster nodes in a single command.

1. Invoke `k0sctl reset` command:

    ```console
    $ k0sctl reset --config k0sctl.yaml
    k0sctl v0.21.0 Copyright 2023, k0sctl authors.
    ? Going to reset all of the hosts, which will destroy all configuration and data, Are you sure? Yes
    INFO ==> Running phase: Connect to hosts
    INFO [ssh] 13.53.43.63:22: connected
    INFO [ssh] 13.53.218.149:22: connected
    INFO ==> Running phase: Detect host operating systems
    INFO [ssh] 13.53.43.63:22: is running Ubuntu 22.04.4 LTS
    INFO [ssh] 13.53.218.149:22: is running Ubuntu 22.04.4 LTS
    INFO ==> Running phase: Acquire exclusive host lock
    INFO ==> Running phase: Prepare hosts
    INFO ==> Running phase: Gather k0s facts
    INFO [ssh] 13.53.43.63:22: found existing configuration
    INFO [ssh] 13.53.43.63:22: is running k0s controller version {{{ k0s_version }}}
    INFO [ssh] 13.53.218.149:22: is running k0s worker version {{{ k0s_version }}}
    INFO [ssh] 13.53.43.63:22: checking if worker  has joined
    INFO ==> Running phase: Reset workers
    INFO [ssh] 13.53.218.149:22: reset
    INFO ==> Running phase: Reset controllers
    INFO [ssh] 13.53.43.63:22: reset
    INFO ==> Running phase: Reset leader
    INFO [ssh] 13.53.43.63:22: reset
    INFO ==> Running phase: Release exclusive host lock
    INFO ==> Running phase: Disconnect from hosts
    INFO ==> Finished in 8s
    ```

---

### Runtime (`docs/runtime.md`)

# Runtime

k0s supports any container runtime that implements the [CRI] specification.

k0s comes bundled with [containerd] as the default Container Runtime Interface (CRI) and [runc] as the default low-level runtime. In most cases they don't require any configuration changes. However, if custom configuration is needed, this page provides some examples.



[CRI]: https://github.com/kubernetes/cri-api
[containerd]: https://github.com/containerd/containerd
[runc]: https://github.com/opencontainers/runc

## containerd configuration

By default k0s manages the full containerd configuration. You have the option of fully overriding, and thus also managing, the configuration yourself.

### User managed containerd configuration

In the default k0s generated configuration there's a "magic" comment telling containerd it is managed by k0s:

```toml
# k0s_managed=true
```

If you wish to take over the configuration management, remove this line.

To make changes to the containerd configuration you must first generate a default containerd configuration, with the default values set to `/etc/k0s/containerd.toml`:

```shell
containerd config default > /etc/k0s/containerd.toml
```

`k0s` runs containerd with the following default values:

```shell
/var/lib/k0s/bin/containerd \
    --root=/var/lib/k0s/containerd \
    --state=/run/k0s/containerd \
    --address=/run/k0s/containerd.sock \
    --config=/etc/k0s/containerd.toml
```

Next, add the following default values to the configuration file:

```toml
version = 3
root = "/var/lib/k0s/containerd"
state = "/run/k0s/containerd"
...

[grpc]
  address = "/run/k0s/containerd.sock"
```

## k0s managed dynamic runtime configuration

As of 1.27.1, k0s allows dynamic configuration of containerd CRI runtimes. This
works by k0s creating a special directory in `/etc/k0s/containerd.d/`, where
users can place partial containerd configuration TOML files (i.e. files with a
`.toml` extension). k0s will automatically pick up these files and add them as containerd
configuration `imports`. When k0s sees changes in this directory, it will automatically reload containerd with the new configuration.

Starting with k0s 1.36, k0s uses containerd 2 which modifies the configuration
and all drop-in files must use the containerd v3 configuration format. Either
omit the `version` field entirely or set it to `version = 3`. Any file that sets
`version = 2` or that references the legacy `io.containerd.grpc.v1.cri` plugin
path will cause k0s to exit with an error on pre-flight checks.  If you have drop-in files from previous versions,
update them before restarting k0s.

Please note, that in order for drop-ins in `/etc/k0s/containerd.d` to take effect on running configuration, `/etc/k0s/containerd.toml` needs to be k0s managed.

If you change the first magic line (`# k0s_managed=true`) in the `/etc/k0s/containerd.toml` (by accident or on purpose), it automatically becomes "not k0s managed". To make it "k0s managed" again, remove `/etc/k0s/containerd.toml` and restart k0s service on the node, it'll be recreated by k0s.

### Examples

The following sections provide some examples of how to use k0s managed drop-in
configurations for common configuration tasks.

#### Configuring OCI registry mirrors

Say you host an internal registry mirror of Docker Hub as
`docker-mirror.internal.acme.corp`. Then you could use the following drop-in
configuration:

`/etc/k0s/containerd.d/cri-registry.toml`:

```toml
version = 3

[plugins."io.containerd.cri.v1.images".registry]
config_path = "/etc/k0s/containerd.d/certs.d"
```

Create the file `/etc/k0s/containerd.d/certs.d/docker.io/hosts.toml` with the following content:

```toml
# Always resolve tags against Docker Hub as the "server"
server = "https://registry-1.docker.io"

[host."https://docker-mirror.internal.acme.corp:5000"]
capabilities = ["pull", "resolve"]
# Trust the internal mirror's certificate. Paths will be resolved relative to
# the containing file's directory, e.g. the below path will be resolved to
# /etc/k0s/containerd.d/certs.d/docker.io/internal-mirror.crt
ca = "internal-mirror.crt"
# Or, alternatively, disable TLS verification, if you know what you're doing:
# skip_verify = true

# Use Google's public pull-through cache
[host."https://mirror.gcr.io"]
capabilities = ["pull", "resolve"]
```

For more details on how to configure registry hosts, please refer to the
[official containerd configuration][containerd-hosts].

[containerd-hosts]: https://github.com/containerd/containerd/blob/release/2.2/docs/hosts.md

#### Using gVisor

[gVisor](https://gvisor.dev/docs/) is an application kernel, written in Go, that implements a substantial portion of the Linux system call interface. It provides an additional layer of isolation between running applications and the host operating system.

1. Install the needed gVisor binaries into the host.

    ```shell
    (
      set -e
      ARCH=$(uname -m)
      URL=https://storage.googleapis.com/gvisor/releases/release/latest/${ARCH}
      wget ${URL}/runsc ${URL}/runsc.sha512 \
        ${URL}/containerd-shim-runsc-v1 ${URL}/containerd-shim-runsc-v1.sha512
      sha512sum -c runsc.sha512 \
        -c containerd-shim-runsc-v1.sha512
      rm -f *.sha512
      chmod a+rx runsc containerd-shim-runsc-v1
      sudo mv runsc containerd-shim-runsc-v1 /usr/local/bin
    )
    ```

    Refer to the [gVisor install docs](https://gvisor.dev/docs/user_guide/install/) for more information.

2. Prepare the config for `k0s` managed containerd, to utilize gVisor as additional runtime:

    ```shell
    cat <<EOF | sudo tee /etc/k0s/containerd.d/gvisor.toml
    version = 3

    [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.runsc]
      runtime_type = "io.containerd.runsc.v1"
    EOF
    ```

3. Start and join the worker into the cluster, as normal:

    ```shell
    k0s worker $token
    ```

4. Register containerd to the Kubernetes side to make gVisor runtime usable for workloads (by default, containerd uses normal runc as the runtime):

    ```shell
    cat <<EOF | kubectl apply -f -
    apiVersion: node.k8s.io/v1
    kind: RuntimeClass
    metadata:
      name: gvisor
    handler: runsc
    EOF
    ```

    At this point, you can use gVisor runtime for your workloads:

    ```yaml
    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx-gvisor
    spec:
      runtimeClassName: gvisor
      containers:
      - name: nginx
        image: nginx
    ```

5. (Optional) Verify that the created nginx pod is running under gVisor runtime:

    ```shell
    # kubectl exec nginx-gvisor -- dmesg | grep -i gvisor
    [    0.000000] Starting gVisor...
    ```

#### Using nvidia-container-runtime

First, deploy the NVIDIA GPU operator Helm chart with the following commands on top of your k0s cluster:

```shell
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install nvidia-gpu-operator -n nvidia-gpu-operator \
  --create-namespace \
  --set operator.defaultRuntime=containerd \
  --set toolkit.env[0].name=CONTAINERD_CONFIG \
  --set toolkit.env[0].value=/etc/k0s/containerd.d/nvidia.toml \
  --set toolkit.env[1].name=CONTAINERD_SOCKET \
  --set toolkit.env[1].value=/run/k0s/containerd.sock \
  --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
  --set toolkit.env[2].value=nvidia \
  nvidia/gpu-operator
```

With this Helm chart values, NVIDIA GPU operator will deploy both driver and toolkit to the GPU nodes and additionally will configure containerd with NVIDIA specific runtime.

**Note**: Detailed installation instructions can be found in the [official
NVIDIA GPU Operator documentation][install-nvidia-gpu-operator].

[install-nvidia-gpu-operator]: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

## Using custom CRI runtimes

**Warning**: You can use your own CRI runtime with k0s (for example, `docker`). However, k0s will not start or manage the runtime, and configuration is solely your responsibility.

Use the option `--cri-socket` to run a k0s worker with a custom CRI runtime. The option takes input in the form of `<type>:<url>` (the only supported type is `remote`).

### Using Docker as the container runtime

As of Kubernetes 1.24, the use of Docker as a container runtime is [no longer
supported][dockershim deprecation] out of the box. However, Mirantis provides
[cri-dockerd], a shim that allows Docker to be controlled via CRI. It's [based
on the dockershim][future of dockershim] that was previously part of upstream
Kubernetes.

#### Configuration

In order to use Docker as the container runtime for k0s, the following steps
need to be taken:

1. Manually install required components.
  On each `k0s worker` and `k0s controller --enable-worker` node, both
  Docker Engine and cri-dockerd need to be installed manually. Follow the
  official [Docker Engine installation guide][install docker] and [cri-dockerd
  installation instructions][install cri-dockerd].

2. Configure and restart affected k0s nodes.
  Once installations are complete, the nodes needs to be restarted with the
  `--cri-socket` flag pointing to cri-dockerd's socket, which is typically
  located at `/var/run/cri-dockerd.sock`. For instance, the commands to start a
  node would be as follows:

      k0s worker --cri-socket=remote:unix:///var/run/cri-dockerd.sock

  or, respectively

  ```console
  k0s controller --enable-worker --cri-socket=remote:unix:///var/run/cri-dockerd.sock
  ```

  When running k0s [as a service](cli/k0s_install.md), consider reinstalling the
  service with the appropriate flags:

  ```console
  sudo k0s install --force worker --cri-socket=remote:unix:///var/run/cri-dockerd.sock
  ```

  or, respectively

  ```console
  sudo k0s install --force controller --enable-worker --cri-socket=remote:unix:///var/run/cri-dockerd.sock
  ```

In scenarios where Docker is managed via systemd, it is crucial that the
`cgroupDriver: systemd` setting is included in the Kubelet configuration. It can
be added to the `workerProfiles` section of the k0s configuration. An example of
how the k0s configuration might look:

```yaml
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  workerProfiles:
    - name: systemd-docker-cri
      values:
        cgroupDriver: systemd
```

Note that this is a cluster-wide configuration setting that must be added to
the k0s controller's configuration rather than directly to the workers, or to
the cluster configuration if using [dynamic configuration]. See the [worker
profiles] section of the documentation for more details. When starting workers,
both the `--profile=systemd-docker-cri` and `--cri-socket` flags are required.
The profile name, such as `systemd-docker-cri`, is flexible. Alternatively,
this setting can be applied to the `default` profile, which will apply to all
nodes started without a specific profile. In this case, the `--profile` flag is
not needed.

Please note that there are currently some [pitfalls around container
metrics][cadvisor-metrics] when using cri-dockerd.

[dockershim deprecation]: https://kubernetes.io/blog/2020/12/02/dockershim-faq/
[cri-dockerd]: https://github.com/Mirantis/cri-dockerd
[future of dockershim]: https://www.mirantis.com/blog/the-future-of-dockershim-is-cri-dockerd/
[install docker]: https://docs.docker.com/engine/install/
[install cri-dockerd]: https://github.com/Mirantis/cri-dockerd#using-cri-dockerd
[worker profiles]: worker-node-config.md#worker-profiles
[dynamic configuration]: dynamic-configuration.md
[cadvisor-metrics]: ./troubleshooting/troubleshooting.md#using-a-custom-container-runtime-and-missing-labels-in-prometheus-metrics

#### Verification

The successful configuration can be verified by executing the following command:

{% set kubelet_ver = k8s_version + '+k0s' -%}
{% set kubelet_ver_len = kubelet_ver | length -%}

```console
$ kubectl get nodes -o wide
NAME              STATUS   ROLES    AGE   {{{ 'VERSION'   | ljust(kubelet_ver_len) }}}   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
docker-worker-0   Ready    <none>   15m   {{{ kubelet_ver | ljust(kubelet_ver_len) }}}   172.27.77.155   <none>        Ubuntu 22.04.3 LTS   5.15.0-82-generic   docker://24.0.7
```

On the worker nodes, the Kubernetes containers should be listed as regular
Docker containers:

```
/* Detailed source-code truncated for AI context efficiency. */
```

---

### Security (`docs/security.md`)

# Security policy

For the security policy, please refer to the [upstream k0s documentation](https://github.com/k0sproject/k0s/blob/main/SECURITY.md).

---

### Selinux (`docs/selinux.md`)

# SELinux Overview

SELinux enforces mandatory access control policies that confine user programs and system services, as well as access to files and network resources. Limiting privilege to the minimum required to work reduces or eliminates the ability of these programs and daemons to cause harm if faulty or compromised.

Enabling SELinux in container runtime provides an additional security control to help further enforce isolation among deployed containers and the host.

This guide describes how to enable SELinux in Kubernetes environment provided by k0s on CentOS and Red Hat Enterprise Linux (RHEL).

## Requirements

- SELinux is enabled on host OS of the worker nodes.
- SELinux has the container-selinux policy installed.
- SELinux labels are correctly set for k0s installation files of the worker nodes.
- SELinux is enabled in container runtime such as containerd on the worker nodes.

### Check whether SELinux is enabled

SELinux is enabled on CentOS and RHEL by default. Below command output indicates SELinux is enabled.

```shell
$ getenforce
Enforcing
```

### Install container-selinux

It is required to have [container-selinux](https://github.com/containers/container-selinux) installed.
In most Fedora based distributions including Fedora 37, Red Hat Enterprise Linux 7, 8 and 8, CentOS
7 and 8 and Rocky Linux 9 this can be achieved by installing the package container-selinux.

In RHEL 7 and CentOS 7 this is achieved by running:

```shell
yum install -y container-selinux
```

In the rest of the metnioned distributions run:

```shell
dnf install -y container-selinux
```

### Set SELinux labels for k0s installation files

Run below commands on the host OS of the worker nodes.

```shell
DATA_DIR="/var/lib/k0s"
sudo semanage fcontext -a -t container_runtime_exec_t "${DATA_DIR}/bin/containerd.*"
sudo semanage fcontext -a -t container_runtime_exec_t "${DATA_DIR}/bin/runc"
sudo restorecon -R -v ${DATA_DIR}/bin
sudo semanage fcontext -a -t container_var_lib_t "${DATA_DIR}/containerd(/.*)?"
sudo semanage fcontext -a -t container_ro_file_t "${DATA_DIR}/containerd/io.containerd.snapshotter.*/snapshots(/.*)?"
sudo restorecon -R -v ${DATA_DIR}/containerd
sudo semanage fcontext -a -t container_file_t "/etc/cni/.*"
sudo restorecon -R -v /etc/cni
sudo semanage fcontext -a -t container_file_t "/opt/cni/.*"
sudo restorecon -R -v /opt/cni
```

### Enable SELinux in containerd of k0s

Add below lines to `/etc/k0s/containerd.d/selinux.toml` of the worker nodes. k0s will automatically restart containerd with the new configuration if needed.

```toml
[plugins.'io.containerd.cri.v1.runtime']
  enable_selinux = true
```

## Verify SELinux works in Kubernetes environment

By following the example [Assign SELinux labels to a Container](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#assign-selinux-labels-to-a-container), deploy a testing pod using below YAML file:

```yaml
apiVersion: v1
kind: Pod
metadata:
  name: test-selinux
spec:
  containers:
  - image: busybox
    name: test-selinux
    command: ["sleep", "infinity"]
    securityContext:
      seLinuxOptions:
        level: "s0:c123,c456"
```

After the pod starts, ssh to the worker node on which the pod is running and check the pod process. It should display the label `s0:c123,c456` that you specified in YAML file:

```shell
$ ps -efZ | grep -F 'sleep infinity'
system_u:system_r:container_t:s0:c123,c456 root 3346 3288  0 16:39 ?       00:00:00 sleep infinity
```

## Node-Local Load Balancing (NLLB) SELinux Support

When using k0s's Node-Local Load Balancing (NLLB) feature with SELinux enabled, k0s automatically sets the correct SELinux labels on NLLB configuration files. The Envoy proxy configuration files in `/run/k0s/nllb/envoy/` are labeled with `container_file_t` context, allowing the Envoy container to read them.

No manual configuration is required for NLLB to work with SELinux. The SELinux labeling is handled automatically when:

- The NLLB feature is enabled in the worker configuration
- SELinux is enabled on the system
- Containerd is configured with SELinux support (`enable_selinux = true`)

You can verify the correct labels are set:

```shell
$ ls -lZ /run/k0s/nllb/envoy/
-r--r--r--. 1 root root system_u:object_r:container_file_t:s0 1122 Jan 19 22:26 cds.yaml
-r--r--r--. 1 root root system_u:object_r:container_file_t:s0  850 Jan 19 22:26 envoy.yaml
```

---

### Shell Completion (`docs/shell-completion.md`)

# Enabling Shell Completion

## Introduction

Shell completion enhances the user experience by providing auto-completion for
commands in the terminal. K0s supports shell completion for the following
shells:

- `bash`, [GNU Bash](https://www.gnu.org/software/bash/)
- `zsh`, [the Z-shell](https://www.zsh.org/)
- `fish`, [the friendly interactive shell](https://fishshell.com/)
- `powershell`, [Microsoft PowerShell](https://learn.microsoft.com/powershell/)

## General Usage

To generate a completion script for your shell, use the following command: `k0s
completion <shell_name>`. Sourcing the completion script in your shell enables
k0s autocompletion.

## bash

One-shot usage: `source <(k0s completion bash)`.

This is a recipe to load completions for each new shell. Adjust to your personal
needs:

```bash
mkdir ~/.bash_completion.d
k0s completion bash >~/.bash_completion.d/k0s

cat <<'EOF' >~/.bashrc
for compFile in ~/.bash_completion.d/*; do
  [ ! -f "$compFile" ] || source -- "$compFile"
done
unset compFile
EOF
```

Then restart the shell or source `~/.bashrc`.

## zsh

One-shot usage: `source <(k0s completion bash)`.

Following a recipe to load completions for each new shell. Adjust to your
personal needs. If shell completion is not already enabled in your zsh
environment you will need to enable it:

```zsh
echo "autoload -Uz compinit; compinit" >>~/.zshrc
```

Place the completion script in a custom `site-functions` folder:

```zsh
mkdir -p -- ~/.local/share/zsh/site-functions
k0s completion zsh >~/.local/share/zsh/site-functions/_k0s
```

Edit `~/.zshrc` and add the line `fpath+=(~/.local/share/zsh/site-functions)`
somewhere before `compinit` is called. After that, restart the shell.

When using [Oh My ZSH!], you can create a [custom plugin]:

```zsh
mkdir -- "$ZSH_CUSTOM/plugins/k0s"
cat <<'EOF' >"$ZSH_CUSTOM/plugins/k0s/k0s.plugin.zsh"
k0s completion zsh >| "$ZSH_CACHE_DIR/completions/_k0s" &|
EOF
omz plugin enable k0s
```

Then restart the shell.

[Oh My ZSH!]: https://ohmyz.sh/
[custom plugin]: https://github.com/ohmyzsh/ohmyzsh/wiki/Customization#overriding-and-adding-plugins

## fish

One-shot usage: `k0s completion fish | source`.

This is a recipe to load completions for each new shell. Adjust to your personal
needs:

```shell
mkdir -p -- "${XDG_CONFIG_HOME:-$HOME/.config}/fish/completions"
k0s completion fish >"${XDG_CONFIG_HOME:-$HOME/.config}/fish/completions/k0s.fish"
```

Then restart the shell.

## powershell

Save the completion script into a file:

```powershell
k0s completion powershell > C:\path\to\k0s.ps1
```

You can import it like so:

```powershell
Import-Module C:\path\to\k0s.ps1
```

To automatically load the module for each new shell session, add the above line
to your shell profile. You can find the path to your profile via `Write-Output
$profile`.

---

### Single Stack Ipv6 (`docs/single-stack-ipv6.md`)

# IPv6 single-stack networking

K0s introduced experimental IPv6 single-stack networking, allowing clusters to operate
exclusively with IPv6 addresses. IPv6 single stack is currently in alpha state and requires
the [IPv6SingleStack feature gate](feature-gates.md) to be enabled on every controller.

IPv6 single-stack is fully compatible with both [node-local load balancing](nllb.md) and
[control plane load balancing](cplb.md).

## Enabling IPv6 single-stack using the default CNI (Kube-router)

In order to enable IPv6 single-stack networking using the default CNI provider, both an IPv6
`podCIDR` and `serviceCIDR` must be provided. Dual stack must not be enabled. Migrations
to dual stack are possible as long as the primary address family remains IPv6.

```yaml
spec:
  network:
    podCIDR: fd00::/108
    serviceCIDR: fd01::/108
```

Ensure that every controller in the cluster is executed with the
[IPv6SingleStack feature gate](feature-gates.md) enabled:

```shell
k0s controller --feature-gates="IPv6SingleStack=true"
```

This configuration will set up all Kubernetes components and Kube-router for IPv6
single-stack networking.

### Configuring the node CIDR mask size

By default, k0s uses a `/117` node CIDR mask size for IPv6, which provides
2048 IP addresses per node and and a `/24` for IPv4 which provides 256
addresses per node.

Using the example configuration `IPv6PodCIDR: fd00::/108`, there are 9 bits
available for node allocation (117 - 108 = 9) and 11 bits available for
pod allocation (128 - 117 = 11). This allows 512 nodes with 2048 IPs per node.

You can customize the node CIDR mask size using the controller manager's extra arguments:

```yaml
spec:
  controllerManager:
    extraArgs:
      node-cidr-mask-size: "120"
  network:
    podCIDR: fd00::/108
    serviceCIDR: fd01::/108
```

With a `/120` node CIDR mask size, the cluster would support more nodes (4096 nodes maximum
with the `/108` pod CIDR) but each node would receive 256 IP addresses.

### Important Notes for Kube-router

Currently kube-router on IPv6 doesn't pass the kubernetes network conformance tests.

---

### Storage (`docs/storage.md`)

# Storage

## CSI

k0s supports a wide range of different storage options by utilizing Container Storage Interface (CSI). All Kubernetes storage solutions are supported and users can easily select the storage that fits best for their needs.

When the storage solution implements CSI, Kubernetes can communicate with the storage to create and configure persistent volumes. This makes it easy to dynamically provision the requested volumes. It also expands the supported storage solutions from the previous generation, in-tree volume plugins. More information about the CSI concept is described on the [Kubernetes Blog](https://kubernetes.io/blog/2019/01/15/container-storage-interface-ga/).



### Installing 3rd party storage solutions

Follow your storage driver's installation instructions. Note that by default the Kubelet installed by k0s uses a slightly different path for its working directory (`/var/lib/k0s/kubelet` instead of `/var/lib/kubelet`). Consult the CSI driver's configuration documentation on how to customize this path. The actual path can differ if you defined the flag `--data-dir`.

## Example storage solutions

Different Kubernetes storage solutions are explained in the [official Kubernetes storage documentation](https://kubernetes.io/docs/concepts/storage/volumes/). All of them can be used with k0s. Here are some popular ones:

- Rook-Ceph (Open Source)
- GlusterFS (Open Source)
- Longhorn (Open Source)
- Amazon EBS
- Google Persistent Disk
- Azure Disk
- Portworx

If you are looking for a fault-tolerant storage with data replication, you can follow our [tutorial for configuring Ceph storage with Rook](examples/rook-ceph.md).

---

### Support Bundle Controller.Yaml (`docs/support-bundle-controller.yaml`)

apiVersion: troubleshoot.sh/v1beta2
kind: SupportBundle
metadata:
  name: controller
spec:
  uri: # TODO
  collectors:
    - clusterInfo: {}
    - clusterResources:
        namespaces:
        - kube-system
        - k0s-autopilot
        - kube-node-lease
        - default # so we get kubernetes svc endpoints
    - nodeMetrics: {}
  hostCollectors:
    # System Info Collectors
    - cpu: {}
    - hostOS: {}
    - hostServices: {}
    - ipv4Interfaces: {}
    - memory: {}
    - time: {}
    # Certificate Info for ETCD and K8s API
    - certificate:
        collectorName: k8s-api-keypair
        certificatePath: /var/lib/k0s/pki/server.crt
        keyPath: /var/lib/k0s/pki/server.key
    - certificate:
        collectorName: etcd-keypair
        certificatePath: /var/lib/k0s/pki/etcd/server.crt
        keyPath: /var/lib/k0s/pki/etcd/server.key
    # Disk usage for commonly used directories in kURL installs
    - diskUsage:
        collectorName: root
        path: /
    # Run collectors for system information
    - run:
        collectorName: k8s-api-healthz-6443
        command: "curl"
        args: ["-k", "--cert", "/var/lib/k0s/pki/admin.crt", "--key", "/var/lib/k0s/pki/admin.key", "https://localhost:6443/healthz?verbose"]
    - run:
        collectorName: curl-etcd-health-2379
        command: "curl"
        args: ["-ki", "https://localhost:2379/health", "--cert", "/var/lib/k0s/pki/apiserver-etcd-client.crt", "--key", "/var/lib/k0s/pki/apiserver-etcd-client.key"]
    - run:
        collectorName: etcd-members
        command: "k0s"
        args: ["etcd", "member-list"]
    - run:
        collectorName: "free"
        command: "free"
        args: ["-m"]
    - run:
        collectorName: "top"
        command: "top"
        args: ["-b", "-n", "1"]
    - run:
        collectorName: "uptime"
        command: "uptime"
        args: []
    - run:
        collectorName: "uname"
        command: "uname"
        args: ["-a"]
    - run:
        collectorName: "df"
        command: "df"
        args: ["-h"]
    - run:
        collectorName: "iostat"
        command: "iostat"
        args: ["-x"]
    # Systemctl service statuses
    - run:
        collectorName: "systemctl-firewalld-status"
        command: "systemctl"
        args: ["status", "firewalld"]
    - run:
        collectorName: "systemctl-ufw-status"
        command: "systemctl"
        args: ["status", "ufw"]
    - run:
        collectorName: "systemctl-k0s-status"
        command: "systemctl"
        args: ["status", "k0s*"]
    # systemd Service Configurations
    - run:
        collectorName: "systemctl-cat-journald"
        command: "systemctl"
        args: ["cat", "systemd-journald"]
    - run:
        collectorName: "systemctl-cat-k0s"
        command: "systemctl"
        args: ["cat", "k0s*"]
    # TODO Add same checks for rc-service
    # Logs for k0s
    - run:
        collectorName: "journalctl-k0s"
        command: "journalctl"
        args: ["-u", "k0s*", "--no-pager", "-S", "7 days ago"]
    - run:
        collectorName: "journalctl-dmesg"
        command: "journalctl"
        args: ["--dmesg", "--no-pager", "-S", "7 days ago"]
    # k0s status
    - run:
        collectorName: k0s-status
        command: "k0s"
        args: ["status", "-o", "yaml"]
    # Gathering hostname info to help troubleshoot scenarios where the hostname mismatch
    - run:
        collectorName: "hostnames"
        command: "sh"
        args:
          - -c
          - |
            echo "hostname = $(hostname)"
            echo "/proc/sys/kernel/hostname = $(cat /proc/sys/kernel/hostname)"
            echo "uname -n = $(uname -n)"
    # System Info Collectors
    - run:
        collectorName: "vmstat"
        command: "vmstat"
        args: ["-w"]
    - run:
        collectorName: "ps-high-load"
        command: "sh"
        args: ["-c", "ps -eo s,user,cmd | grep ^[RD] | sort | uniq -c | sort -nbr | head -20"]
    - filesystemPerformance:
        collectorName: filesystem-latency-two-minute-benchmark
        timeout: 2m
        directory: /var/lib/k0s/etcd
        fileSize: 22Mi
        operationSizeBytes: 2300
        datasync: true
        enableBackgroundIOPS: true
        backgroundIOPSWarmupSeconds: 10
        backgroundWriteIOPS: 300
        backgroundWriteIOPSJobs: 6
        backgroundReadIOPS: 50
        backgroundReadIOPSJobs: 1
        exclude: true
    - run:
        collectorName: "localhost-ips"
        command: "sh"
        args: ["-c", "host localhost"]
  hostAnalyzers:
    - time:
        checkName: "ntp-status"
        outcomes:
          - fail:
              when: "ntp == unsynchronized+inactive"
              message: "System clock is not synchronized"
          - warn:
              when: "ntp == unsynchronized+active"
              message: System clock not yet synchronized
          - pass:
              when: "ntp == synchronized+active"
              message: "System clock is synchronized"
          - warn:
              when: "timezone != UTC"
              message: "Non UTC timezone can interfere with system function"
          - pass:
              when: "timezone == UTC"
              message: "Timezone is set to UTC"
    - diskUsage:
        checkName: "root"
        collectorName: "root"
        outcomes:
          - fail:
              when: "total < 40Gi"
              message: The disk containing directory / has less than 40Gi of total space
          - warn:
              when: "used/total > 80%"
              message: The disk containing directory / is more than 80% full
          - warn:
              when: "available < 10Gi"
              message: The disk containing directory / has less than 10Gi of disk space available
          - pass:
              message: The disk containing directory / has sufficient space
    - diskUsage:
        checkName: "tmp"
        collectorName: "tmp"
        outcomes:
          - warn:
              when: "total < 8Gi"
              message: The disk containing directory /tmp has less than 8Gi of total space
          - warn:
              when: "used/total > 80%"
              message: The disk containing directory /tmp is more than 80% full
          - warn:
              when: "available < 2Gi"
              message: The disk containing directory /tmp has less than 2Gi of disk space available
          - pass:
              message: The disk containing directory /tmp has sufficient space
    - filesystemPerformance:
        collectorName: filesystem-latency-two-minute-benchmark
        outcomes:
          - pass:
              when: "p99 < 10ms"
              message: "Write latency is ok (p99 target < 10ms)"
          - warn:
              message: "Write latency is high. p99 target >= 10ms)"
        exclude: true
  analyzers:
    - textAnalyze:
        checkName: Kubernetes API health check
        fileName: host-collectors/run-host/k8s-api-healthz-6443.txt
        regex: ".*healthz check passed*"
        outcomes:
          - fail:
              when: "false"
              message: "Kubernetes API health check did not pass. One or more components are not working."
          - pass:
              when: "true"
              message: "Kubernetes API health check passed"
    - textAnalyze:
        checkName: ETCD API Health
        fileName: host-collectors/run-host/curl-etcd-health-2379.txt
        regex: ".*\"health\":\"true\"*"
        outcomes:
          - fail:
              when: "false"
              message: "ETCD status returned: unhealthy"
          - pass:
              when: "true"
              message: "ETCD status returned: healthy"
    - textAnalyze:
        checkName: Check if localhost resolves to 127.0.0.1
        fileName: host-collectors/run-host/localhost-ips.txt
        regex: 'localhost has address 127.0.0.1'
        outcomes:
          - fail:
              when: "false"
              message: "'localhost' does not resolve to 127.0.0.1 ip address"
          - pass:
              when: "true"
              message: "'localhost' resolves to 127.0.0.1 ip address"

---

### Support Bundle Worker.Yaml (`docs/support-bundle-worker.yaml`)

# Spec to run when a kURL cluster is down and in-cluster specs can't be run
apiVersion: troubleshoot.sh/v1beta2
kind: SupportBundle
metadata:
  name: worker
spec:
  uri: #TODO
  hostCollectors:
    # System Info Collectors
    - blockDevices: {}
    - cpu: {}
    - hostOS: {}
    - hostServices: {}
    - ipv4Interfaces: {}
    - memory: {}
    - time: {}
    - ipv4Interfaces: {}
    # Disk usage for commonly used directories in kURL installs
    - diskUsage:
        collectorName: root
        path: /
    - diskUsage:
        collectorName: tmp
        path: /tmp
    - diskUsage:
        collectorName: var-lib-k0s
        path: /var/lib/k0s
    # Run collectors for system information
    - run:
        collectorName: k8s-api-healthz-6443
        command: "k0s"
        args: ["kubectl", "--kubeconfig", "/var/lib/k0s/kubelet.conf", "get", "--raw", "/healthz?verbose"]
    - run:
        collectorName: "free"
        command: "free"
        args: ["-m"]
    - run:
        collectorName: "top"
        command: "top"
        args: ["-b", "-n", "1"]
    - run:
        collectorName: "uptime"
        command: "uptime"
        args: []
    - run:
        collectorName: "uname"
        command: "uname"
        args: ["-a"]
    - run:
        collectorName: "df"
        command: "df"
        args: ["-h"]
    - run:
        collectorName: "iostat"
        command: "iostat"
        args: ["-x"]
    # SELinux status
    - run:
        collectorName: "sestatus"
        command: "sestatus"
        args: []
    - run:
        collectorName: "apparmor-status"
        command: "apparmor_status"
        args: []
    - run:
        collectorName: "crictl-ps"
        command: "k0s"
        args: ["ctr", "c", "ls"]
    - run:
        collectorName: "iptables"
        command: "/var/lib/k0s/bin/iptables"
        args: ["-L", "-v"]
    - run:
        collectorName: "iptables-version"
        command: "/var/lib/k0s/bin/iptables"
        args: ["-V"]
    - run:
        collectorName: "lsblk"
        command: "lsblk"
        args: ["--fs"]
    - run:
        collectorName: "netstat-ports"
        command: "netstat"
        args: ["-t", "-u", "-l", "-p", "-n"]
    - run:
        collectorName: "netstat-route-table"
        command: "netstat"
        args: ["-r", "-n"]
    - run:
        collectorName: "ip-route-table"
        command: "ip"
        args: ["route"]
    - run:
        collectorName: "sysctl"
        command: "sysctl"
        args: ["-a"]
    # Systemctl service statuses for CRI, Kubelet, and Firewall
    - run:
        collectorName: "systemctl-k0s-status"
        command: "systemctl"
        args: ["status", "k0s*"]
    # systemd Service Configurations for CRI, Kubelet
    - run:
        collectorName: "systemctl-cat-journald"
        command: "systemctl"
        args: ["cat", "systemd-journald"]
    - run:
        collectorName: "systemctl-cat-k0s"
        command: "systemctl"
        args: ["cat", "k0s*"]
    # Logs for CRI, Kubelet, Kernel
    - run:
        collectorName: "journalctl-k0s"
        command: "journalctl"
        args: ["-u", "k0s*", "--no-pager", "-S", "7 days ago"]
    - run:
        collectorName: "journalctl-dmesg"
        command: "journalctl"
        args: ["--dmesg", "--no-pager", "-S", "7 days ago"]
    
    # sysctl parameters
    - run:
        collectorName: "sysctl-all"
        command: "sh"
        args: ["-c", "sysctl --all 2>/dev/null"]
    - run:
        collectorName: "k0s-sysinfo"
        command: "k0s"
        args: ["sysinfo"]
    # Gathering hostname info to help troubleshoot scenarios where the hostname mismatch
    - run:
        collectorName: "hostnames"
        command: "sh"
        args:
          - -c
          - |
            echo "hostname = $(hostname)"
            echo "/proc/sys/kernel/hostname = $(cat /proc/sys/kernel/hostname)"
            echo "uname -n = $(uname -n)"
    
    # System Info Collectors
    - run:
        collectorName: "du-root"
        command: "sh"
        args: ["-c", "du -Shax / --exclude /proc | sort -rh | head -20"]
    - run:
        collectorName: "mount"
        command: "mount"
        args: ["-l"]
    - run:
        collectorName: "vmstat"
        command: "vmstat"
        args: ["-w"]
    - run:
        collectorName: "ps-high-load"
        command: "sh"
        args: ["-c", "ps -eo s,user,cmd | grep ^[RD] | sort | uniq -c | sort -nbr | head -20"]
    - run:
        collectorName: "ps-detect-antivirus-and-security-tools"
        command: "sh"
        args: [-c, "ps -ef | grep -E 'clamav|sophos|esets_daemon|fsav|symantec|mfend|ds_agent|kav|bdagent|s1agent|falcon|illumio' | grep -v grep"]
    - run:
        collectorName: "localhost-ips"
        command: "sh"
        args: ["-c", "host localhost"]
  hostAnalyzers:
    - time:
        checkName: "ntp-status"
        outcomes:
          - fail:
              when: "ntp == unsynchronized+inactive"
              message: "System clock is not synchronized"
          - warn:
              when: "ntp == unsynchronized+active"
              message: System clock not yet synchronized
          - pass:
              when: "ntp == synchronized+active"
              message: "System clock is synchronized"
          - warn:
              when: "timezone != UTC"
              message: "Non UTC timezone can interfere with system function"
          - pass:
              when: "timezone == UTC"
              message: "Timezone is set to UTC"
    - diskUsage:
        checkName: "root"
        collectorName: "root"
        outcomes:
          - fail:
              when: "total < 40Gi"
              message: The disk containing directory / has less than 40Gi of total space
          - warn:
              when: "used/total > 80%"
              message: The disk containing directory / is more than 80% full
          - warn:
              when: "available < 10Gi"
              message: The disk containing directory / has less than 10Gi of disk space available
          - pass:
              message: The disk containing directory / has sufficient space
    - diskUsage:
        checkName: "tmp"
        collectorName: "tmp"
        outcomes:
          - warn:
              when: "total < 8Gi"
              message: The disk containing directory /tmp has less than 8Gi of total space
          - warn:
              when: "used/total > 80%"
              message: The disk containing directory /tmp is more than 80% full
          - warn:
              when: "available < 2Gi"
              message: The disk containing directory /tmp has less than 2Gi of disk space available
          - pass:
              message: The disk containing directory /tmp has sufficient space
    - diskUsage:
        checkName: "var-lib-k0s"
        collectorName: "var-lib-k0s"
        outcomes:
          - warn:
              when: "used/total > 80%"
              message: The disk containing directory /var/lib/k0s is more than 80% full
          - warn:
              when: "available < 10Gi"
              message: The disk containing directory /var/lib/k0s has less than 10Gi of disk space available
          - pass:
              message: The disk containing directory /var/lib/k0s has sufficient space
    
    - filesystemPerformance:
        collectorName: filesystem-latency-two-minute-benchmark
        outcomes:
          - pass:
              when: "p99 < 10ms"
              message: "Write latency is ok (p99 target < 10ms)"
          - warn:
              message: "Write latency is high. p99 target >= 10ms)"
        exclude: true
  analyzers:
    - textAnalyze:
        checkName: Hostname Mismatch
        fileName: host-collectors/run-host/journalctl-k0s.txt
        regex: ".*can only access node lease with the same name as the requesting node.*"
        outcomes:
          - fail:
              when: "true"
              message: "Possible hostname change. Verify that the current hostname matches what's expected by the k8s control plane"
          - pass:
              when: "false"
              message: "No signs of hostname changes found"
    - textAnalyze:
        checkName: Kubernetes API health check
        fileName: host-collectors/run-host/k8s-api-healthz-6443.txt
        regex: ".*healthz check passed*"
        outcomes:
          - fail:
              when: "false"
              message: "Kubernetes API health check did not pass. One or more components are not working."
          - pass:
              when: "true"
              message: "Kubernetes API health check passed"
    - textAnalyze:
        checkName: Check if localhost resolves to 127.0.0.1
        fileName: host-collectors/run-host/localhost-ips.txt
        regex: 'localhost has address 127.0.0.1'
        outcomes:
          - fail:
              when: "false"
              message: "'localhost' does not resolve to 127.0.0.1 ip address"
          - pass:
              when: "true"
              message: "'localhost' resolves to 127.0.0.1 ip address"
    - textAnalyze:
        checkName: Check if SELinux is enabled
        fileName: host-collectors/run-host/sestatus.txt
        regex: '(?m)^Current mode:\s+enforcing'
        ignoreIfNoFiles: true
        outcomes:
          - fail:
              when: "true"
              message: "SELinux is enabled when it should be disabled for kubernetes to work properly"
          - pass:
              when: "false"
              message: "SELinux is disabled as expected"
    - textAnalyze:
        checkName: "Detect Threat Management and Network Security Tools"
        fileName: host-collectors/run-host/ps-detect-antivirus-and-security-tools.txt
        regex: '\b(clamav|sophos|esets_daemon|fsav|symantec|mfend|ds_agent|kav|bdagent|s1agent|falcon|illumio)\b'
        ignoreIfNoFiles: true
        outcomes:
          - fail:
              when: "true"
              message: "Antivirus or Network Security tools detected. These tools can interfere with kubernetes operation."
          - pass:
              when: "false"
              message: "No Antivirus or Network Security tools detected."

---

### System Monitoring (`docs/system-monitoring.md`)

# System components monitoring

Controller nodes [are isolated](architecture/index.md#control-plane) by default, which thus means that a cluster user cannot schedule workloads onto controller nodes.

k0s provides a mechanism to expose system components for monitoring. System component metrics can give a better look into what is happening inside them. Metrics are particularly useful for building dashboards and alerts.
You can read more about metrics for Kubernetes system components in the [Kubernetes documentation](https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/).

**Note:** the mechanism is an opt-in feature, you can enable it on installation:

```shell
sudo k0s install controller --enable-metrics-scraper
```

Once enabled, a new set of objects will appear in the cluster:

```shell
❯ ~ kubectl get all -n k0s-system
NAME                                   READY   STATUS    RESTARTS   AGE
pod/k0s-pushgateway-6c5d8c54cf-bh8sb   1/1     Running   0          43h

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/k0s-pushgateway   ClusterIP   10.100.11.116   <none>        9091/TCP   43h

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/k0s-pushgateway   1/1     1            1           43h

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/k0s-pushgateway-6c5d8c54cf   1         1         1       43h
```

That's not enough to start scraping these additional metrics. For [Prometheus
Operator](https://prometheus-operator.dev/) based solutions, you can create a
`ServiceMonitor` for it like this:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: k0s
  namespace: k0s-system
spec:
  endpoints:
  - port: http
    honorLabels: true 
  selector:
    matchLabels:
      app: k0s-observability
      component: pushgateway
      k0s.k0sproject.io/stack: metrics
```

Note that it won't clear alerts like `KubeControllerManagerDown` or
`KubeSchedulerDown` as they are based on Prometheus' internal `up` metrics. But
you can get rid of these alerts by modifying them to detect a working component
like this:

```promql
absent(apiserver_audit_event_total{job="kube-scheduler"})
```

## Jobs

The list of components which is scrapped by k0s:

- kube-scheduler
- kube-controller-manager
- etcd
- kine

**Note:** kube-apiserver metrics are not scrapped since they are accessible via `kubernetes` endpoint within the cluster.

## Architecture



k0s uses a pushgateway with a TTL to make it possible to detect issues with the metrics delivery. The default TTL is 2 minutes.

---

### System Requirements (`docs/system-requirements.md`)

# System requirements

This page describes the system requirements for k0s.

## Minimum memory and CPU requirements

The minimum requirements for k0s detailed below are approximations, and thus your results may vary.

| Role                | Memory (RAM in GB) | Virtual CPUs (vCPU) |
|---------------------|--------------------|---------------------|
| Controller node     |                  1 |                   1 |
| Worker node         |                0.5 |                   1 |
| Controller + worker |                  1 |                   1 |

## Controller node recommendations

| # of worker nodes | # of pods | Recommended RAM (in GB) | Recommended vCPU |
|-------------------|-----------|-------------------------|------------------|
|                10 |      1000 |                     1-2 |              1-2 |
|                50 |      5000 |                     2-4 |              2-4 |
|               100 |     10000 |                     4-8 |              2-4 |
|               500 |     50000 |                    8-16 |              4-8 |
|              1000 |    100000 |                   16-32 |             8-16 |
|              5000 |    150000 |                   32-64 |            16-32 |

k0s has the standard Kubernetes limits for the maximum number of nodes, pods, etc. For more details, see [the Kubernetes considerations for large clusters](https://kubernetes.io/docs/setup/best-practices/cluster-large/).

k0s controller node measured memory consumption can be found below on this page.

## Storage

It's recommended to use an SSD for [optimal storage performance](https://etcd.io/docs/current/op-guide/performance/) (cluster latency and throughput are sensitive to storage).

For worker nodes it is required that there is at least 15% relative disk space free.

The specific storage consumption for k0s is as follows:

| Role                 | Usage (k0s part) | Minimum required |
|----------------------|------------------|------------------|
| Controller node      | ~0.5 GB          | ~0.5 GB          |
| Worker node          | ~1.3 GB          | ~1.6 GB          |
| Controller + worker  | ~1.7 GB          | ~2.0 GB          |

**Note**: The operating system and application requirements must be considered in addition to the k0s part.

## Host operating system

k0s runs on Linux and Windows operating systems.

The following operating systems are automatically tested as part of our CI:



| Operating System              | Version/Notes                                                                           |
|-------------------------------|-----------------------------------------------------------------------------------------|
| Amazon Linux                  | 2023                                                                                    |
| Alpine Linux                  | 3.21, 3.23                                                                              |
| CentOS Stream                 | 9, 10 (Coughlan)                                                                        |
| Debian GNU/Linux              | 11 (bullseye) (supported until 2026-08-31), 12 (bookworm) (supported until 2028-06-30)  |
| Fedora CoreOS                 | stable stream                                                                           |
| Fedora Linux                  | 41 (Cloud Edition)                                                                      |
| Flatcar Container Linux       | by Kinvolk                                                                              |
| Red Hat Enterprise Linux      | 7.9 (Maipo), 8.10 (Ootpa), 9.7 (Plow)                                                   |
| Rocky Linux                   | 8.10 (Green Obsidian), 9.5 (Blue Onyx)                                                  |
| SUSE Linux Enterprise Server  | 15 SP6                                                                                  |
| Ubuntu                        | 20.04 LTS, 22.04 LTS, 24.04                                                             |

**Note:** For detailed Linux-specific requirements, please refer to the [Linux specific requirements].

[Linux specific requirements]: external-runtime-deps.md#linux-specific

## Architecture

- `x86_64`
- `aarch64`
- `armv7l`
- `riscv64` (No pre-compiled binaries, no CI coverage)

## Networking

For information on the required ports and protocols, refer to [networking](networking.md).

## External runtime dependencies

k0s strives to be as independent from the operating system as possible. See the
dedicated section on [external runtime dependencies](external-runtime-deps.md)
for details on current and past requirements.

To run some automated compatiblility checks on your system, use
[`k0s sysinfo`](cli/k0s_sysinfo.md).

## Controller node measured memory consumption

The following table shows the measured memory consumption in the cluster of one controller node.

| # of Worker nodes | # of Pods (besides default) | Memory consumption (in MB) |
|-------------------|-----------------------------|----------------------------|
|                 1 |                           0 |                        510 |
|                 1 |                         100 |                        600 |
|                20 |                           0 |                        660 |
|                20 |                        2000 |                       1000 |
|                50 |                           0 |                        790 |
|                50 |                        5000 |                       1400 |
|               100 |                           0 |                       1000 |
|               100 |                       10000 |                       2300 |
|               200 |                           0 |                       1500 |
|               200 |                       20000 |                       3300 |

Measurement details:

- k0s v1.22.4+k0s.2 (default configuration with etcd)
- Ubuntu Server 20.04.3 LTS, OS part of the used memory was around 180 MB
- Hardware: AWS t3.xlarge (4 vCPUs, 16 GB RAM)
- Pod image: nginx:1.21.4

---

### Upgrade (`docs/upgrade.md`)

# Upgrade

The k0s upgrade is a simple process due to its single binary distribution. The k0s single binary file includes all the necessary parts for the upgrade and essentially the upgrade process is to replace that file and restart the service.

This tutorial explains two different approaches for k0s upgrade:

- [Upgrade a k0s node locally](#upgrade-a-k0s-node-locally)
- [Upgrade a k0s cluster using k0sctl](#upgrade-a-k0s-cluster-using-k0sctl)

## Upgrade a k0s node locally

If your k0s cluster has been deployed with k0sctl, then k0sctl provides the easiest upgrade method. In that case jump to the next chapter. However, if you have deployed k0s without k0sctl, then follow the upgrade method explained in this chapter.

Before starting the upgrade, consider moving your applications to another node if you want to avoid downtime. This can be done by [draining a worker node](https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/). Remember to uncordon the worker node afterwards to tell Kubernetes that it can resume scheduling new pods onto the node.

The upgrade process is started by stopping the currently running k0s service.

```shell
sudo k0s stop
```

Now you can replace the old k0s binary file. The easiest way is to use the download script. It will download the latest k0s binary and replace the old binary with it. You can also do this manually without the download script.

```shell
curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh
```

Then you can start the service (with the upgraded k0s) and your upgrade is done.

```shell
sudo k0s start
```

## Upgrade a k0s cluster using k0sctl

The upgrading of k0s clusters using k0sctl occurs not through a particular
command (there is no `upgrade` sub-command in k0sctl) but by way of the
configuration file. The configuration file describes the desired state of the
cluster. When you pass the description to the `k0sctl apply` command, the system
discovers the current state and performs whatever actions are necessary to bring
the cluster to the desired state. For example, the system might perform an
upgrade.

### k0sctl cluster upgrade process

The following operations occur during a k0sctl upgrade:

1. Upgrade of each controller, one at a time. There is no downtime if multiple controllers are configured.

2. Upgrade of workers, in batches of 10%.

3. Draining of workers, which allows the workload to move to other nodes prior to the actual upgrade of the worker node components. (To skip the drain process, use the ``--no-drain`` option.)

4. The upgrade process continues once the upgraded nodes return to **Ready** state.

You can configure the desired cluster version in the k0sctl configuration by setting the value of `spec.k0s.version`:

```yaml
spec:
  k0s:
    version: {{{ k0s_version }}}
```

If you do not specify a version, k0sctl checks online for the latest version and defaults to it.

```shell
k0sctl apply
```

```shell
...
...
INFO[0001] ==> Running phase: Upgrade controllers
INFO[0001] [ssh] 10.0.0.23:22: starting upgrade
INFO[0001] [ssh] 10.0.0.23:22: Running with legacy service name, migrating...
INFO[0011] [ssh] 10.0.0.23:22: waiting for the k0s service to start
INFO[0016] ==> Running phase: Upgrade workers
INFO[0016] Upgrading 1 workers in parallel
INFO[0016] [ssh] 10.0.0.17:22: upgrade starting
INFO[0027] [ssh] 10.0.0.17:22: waiting for node to become ready again
INFO[0027] [ssh] 10.0.0.17:22: upgrade successful
INFO[0027] ==> Running phase: Disconnect from hosts
INFO[0027] ==> Finished in 27s
INFO[0027] k0s cluster version {{{ k0s_version }}} is now installed
INFO[0027] Tip: To access the cluster you can now fetch the admin kubeconfig using:
INFO[0027]      k0sctl kubeconfig
```

---

### User Management (`docs/user-management.md`)

# User Management

Kubernetes, and thus k0s, does not have any built-in functionality to manage users. Kubernetes relies solely on external sources for user identification and authentication. A client certificate is considered an external source in this case as Kubernetes API server "just" validates that the certificate is signed by a trusted CA. This means that it is recommended to use e.g. [OpenID Connect](./examples/oidc/oidc-cluster-configuration.md) to configure the API server to trust tokens issued by an external Identity Provider.

k0s comes with some helper commands to create kubeconfig with client certificates for users. There are few caveats that one needs to take into consideration when using client certificates:

* Client certificates have long expiration time, they're valid for one year
* Client certificates cannot be revoked (general Kubernetes challenge)

## Adding a Cluster User

Run the [kubeconfig create](cli/k0s_kubeconfig_create.md) command on the controller to add a user to the cluster. The command outputs a kubeconfig for the user, to use for authentication.

```shell
k0s kubeconfig create [username]
```

## Enabling Access to Cluster Resources

Create the user with the `system:masters` group to grant the user access to the cluster:

```shell
k0s kubeconfig create --groups "system:masters" testUser > k0s.config
```

Create a `roleBinding` to grant the user access to the resources:

```shell
k0s kubectl create clusterrolebinding --kubeconfig k0s.config testUser-admin-binding --clusterrole=admin --user=testUser
```

---

### Verifying Signs (`docs/verifying-signs.md`)

# Verifying Signed Binaries

K0smotron team provides signed binaries for k0s. The signatures are created using [cosign](https://docs.sigstore.dev/signing/quickstart/).
Public key and signature files are available for download from the [releases page](https://github.com/k0sproject/k0s/releases/latest).
Binaries can be verified using the `cosign` tool, for example:

{% set k0s_url_ver = k0s_version | urlencode -%}

```shell
cosign verify-blob \
  --key https://github.com/k0sproject/k0s/releases/download/{{{ k0s_url_ver }}}/cosign.pub \
  --signature https://github.com/k0sproject/k0s/releases/download/{{{ k0s_url_ver }}}/k0s-{{{ k0s_url_ver }}}-amd64.sig \
  k0s-{{{ k0s_version }}}-amd64
```

---

### Version Skew Policy (`docs/version-skew-policy.md`)

# Version skew policy

k0s ships a curated set of Kubernetes components (kube-apiserver,
kube-controller-manager, kube-scheduler, kubelet, kube-proxy, …) for each
release, so the version skew between those components inside a single k0s
release is always supported by upstream Kubernetes by construction.

What you still control as an operator is the skew **between releases of
k0s** that you mix in one cluster — for example, controllers running k0s
`v1.31.x+k0s.0` while workers still run `v1.30.x+k0s.0` during a rolling
upgrade.

## Supported skew

k0s is designed and tested for clusters in which all nodes run either the same
minor version or two adjacent ones, as happens during a rolling upgrade. In
practice, that means:

- **Controllers** must all be within **one minor** of each other.
- **Workers** may be **one minor older** than the controllers, but **must never
  be newer**. This only applies to cluster upgrades. When **bootstrapping a new
  cluster**, workers must run the **same minor** as the controllers.

Patch-version differences inside the same minor are always supported and are the
expected state during a phased upgrade.

Note that the upstream Kubernetes [version skew policy] permits wider skews,
such as kubelets up to three minors older than the API server. k0s doesn't
support those. Each k0s version is only expected to interoperate with the
previous one, and upgrades proceed one minor at a time.

[version skew policy]: https://kubernetes.io/releases/version-skew-policy/

## Upgrades

When upgrading k0s in place:

1. Upgrade controllers first, **one minor at a time** (e.g. v1.30 → v1.31,
   not v1.30 → v1.32).
2. Upgrade workers after the control plane is fully on the new minor.
3. Consider moving to the latest patch release of the old minor first. This
   isn't required, but it takes the old minor's latest fixes along into the
   upgrade.

[Autopilot], [k0sctl], and [k0smotron] all upgrade controllers before workers.
When upgrading manually, keep to the same order.

[Autopilot]: autopilot.md
[k0sctl]: k0sctl-install.md
[k0smotron]: https://github.com/k0sproject/k0smotron

## Downgrades

Downgrades are **not** supported by upstream Kubernetes and are not tested
by k0s. If a release needs to be rolled back, restore from a
[backup](backup.md) taken before the upgrade rather than running an older
k0s binary against a newer cluster state.

## Major versions

There is currently no Kubernetes v2 and no k0s v2, so major-version skew is
not a configuration that exists in practice.

---

### Worker Node Config (`docs/worker-node-config.md`)

# Configuration options for worker nodes

Although the `k0s worker` command does not take in any special YAML configuration, there are still methods for configuring the workers to run various components.

## Node labels

The `k0s worker` command accepts the `--labels` flag, with which you can make the newly joined worker node register itself, in the Kubernetes API, with the given set of labels.

For example, running the worker with `k0s worker --token-file k0s.token --labels="k0sproject.io/foo=bar,k0sproject.io/other=xyz"` results in:

{% set kubelet_ver = k8s_version + '+k0s' -%}
{% set kubelet_ver_len = kubelet_ver | length -%}

```console
$ kubectl get node --show-labels
NAME      STATUS     ROLES    AGE   {{{ 'VERSION'   | ljust(kubelet_ver_len) }}}   LABELS
worker0   NotReady   <none>   10s   {{{ kubelet_ver | ljust(kubelet_ver_len) }}}   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,k0sproject.io/foo=bar,k0sproject.io/other=xyz,kubernetes.io/arch=amd64,kubernetes.io/hostname=worker0,kubernetes.io/os=linux
```

Controller worker nodes are assigned `node.k0sproject.io/role=control-plane` and `node-role.kubernetes.io/control-plane=true` labels:

```console
$ kubectl get node --show-labels
NAME          STATUS     ROLES           AGE   {{{ 'VERSION'   | ljust(kubelet_ver_len) }}}   LABELS
controller0   NotReady   control-plane   10s   {{{ kubelet_ver | ljust(kubelet_ver_len) }}}   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/hostname=worker0,kubernetes.io/os=linux,node.k0sproject.io/role=control-plane,node-role.kubernetes.io/control-plane=true
```

**Note:** Setting the labels is only effective on the first registration of the node. Changing the labels thereafter has no effect.

## Taints

The `k0s worker` command accepts the `--taints` flag, with which you can make the newly joined worker node register itself with the given set of taints.

**Note:** Controller nodes running with `--enable-worker` are assigned `node-role.kubernetes.io/control-plane:NoExecute` taint automatically. You can disable default taints using `--no-taints` parameter.

```shell
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints
```

```shell
NAME          TAINTS
controller0   [map[effect:NoSchedule key:node-role.kubernetes.io/control-plane]]
worker0       <none>
```

## Kubelet configuration

The `k0s worker` command accepts a generic flag to pass in any set of arguments
for the kubelet process.

For example, running `k0s worker --token-file=k0s.token
--kubelet-extra-args="--node-ip=1.2.3.4 --address=0.0.0.0"` passes in the given
flags to Kubelet as-is. As such, you must confirm that any flags you are passing
in are properly formatted and valued as k0s will not validate those flags.

### Worker Profiles

Kubelet configuration fields can also be set via worker profiles. Worker
profiles are defined in the main k0s.yaml and are used to generate ConfigMaps
containing a custom `kubelet.config.k8s.io/v1beta1/KubeletConfiguration`
object.
See also the [examples of k0s.yaml containing worker
profiles](./configuration.md#configuration-examples) and the [list of possible
Kubelet configuration
fields](https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/).

### Kubelet serving certificates

By default, k0s configures kubelets to request their serving certificates from
the Kubernetes API via [Certificate Signing Requests] (CSRs), by enabling
`serverTLSBootstrap` in the generated kubelet configuration. The kubelet then
requests a certificate from the `kubernetes.io/kubelet-serving` signer and uses
it to serve its own API. Clients of that API verify the certificate against the
cluster CA. The Kubernetes API server verifies it, for example, when serving
`kubectl logs` and `kubectl exec` requests. Similarly, k0s's `metrics-server`
component, which bundles the [Kubernetes Metrics Server], verifies it when
scraping the kubelet's resource metrics. Other properly configured monitoring
systems do the same.

Kubernetes doesn't approve these CSRs automatically. k0s ships a controller
component called `csr-approver` that does so, provided the request meets all of
the following conditions:

- The PEM-encoded certificate request size doesn't exceed 512 KiB.
- It is a well-formed kubelet serving certificate request, according to
  Kubernetes' validation rules.
- The certificate request has no more than 4096 SANs, counting DNS names and IP
  addresses together.
- It was created by the very node it requests the certificate for. The
  requesting user must have the `system:node:<nodeName>` name and be a member of
  the `system:nodes` group, and the certificate's common name must be identical
  to the requesting user name.
- The requested DNS names and IP addresses don't identify the cluster's control
  plane or other in-cluster services: no IP address may be inside the service
  CIDR(s), and no DNS name may be `kubernetes`, `kubernetes.default`, or a name
  within the `svc` or cluster domains.
- The node exists in the cluster.
- The node's `status.addresses` contains no more than 4096 entries.
- The requested DNS names and IP addresses are all listed in the node's
  `status.addresses`.

Requests that don't meet these conditions are left pending, and the reason is
logged by the k0s controller. Note that a node's addresses are reported by its
kubelet, so flags like `--node-ip` directly influence which addresses a
certificate may be issued for.

The CSR approver checks for pending requests periodically. It only considers the
newest pending request of each node per pass, so that a node can't hold up
others by creating large numbers of requests. It can be turned off via `k0s
controller --disable-components csr-approver`. In that case, kubelet serving
certificates need to be approved by some other means, otherwise the affected
kubelet APIs remain unavailable.

[Certificate Signing Requests]: https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/
[Kubernetes Metrics Server]: https://github.com/kubernetes-sigs/metrics-server

## IPTables Mode

k0s detects the iptables backend automatically based on the existing records. On a brand-new setup, `iptables-nft` will be used.
There is an `--iptables-mode` flag to specify the mode explicitly. Valid values: `nft`, `legacy` and `auto` (default).

```shell
k0s worker --iptables-mode=nft
```

---

### Architecture Adr 001 Autopilot Oci Basic Auth Support (`docs/architecture/adr-001-autopilot-oci-basic-auth-support.md`)

# ADR 1: Add Support for OCI Registry and HTTP Authentication

## Context

Registries are increasingly being used as generic artifact stores, expanding
beyond their traditional role of hosting container images. To align with this
trend, it is beneficial for Autopilot to support pulling artifacts directly
from registries. Currently, Autopilot's capabilities are limited to downloading
artifacts via the HTTP\[S\] protocols.

Enhancing Autopilot to pull artifacts directly from registries will streamline
workflows and improve efficiency by allowing integration and deployment of
diverse artifacts without relying solely on HTTP\[S\] endpoints. This update
will enable Autopilot to handle registry-specific protocols and authentication
mechanisms, aligning it with modern deployment practices.

Currently, Autopilot does not support the retrieval of artifacts via the HTTP
protocol when authentication is required. Implementing this feature to
accommodate such authentication methods would be a valuable enhancement.

## Decision

Implement support in Autopilot for pulling artifacts, such as k0s binaries and
image bundles, directly from a registry using the
[ORAS](https://oras.land/docs/) client. Additionally, add support for HTTP
authentication to ensure secure access to artifacts.

## Solution

Starting with the current `PlanResourceURL` struct:

```go
type PlanResourceURL struct {
	// URL is the URL of a downloadable resource.
	URL string `json:"url"`

	// Sha256 provides an optional SHA256 hash of the URL's content for
	// verification.
	Sha256 string `json:"sha256,omitempty"`
}
```

We must specify to Autopilot where to access credentials for remote artifact
pulls. This will be achieved by adjusting the struct as follows:

```go
type PlanResourceURL struct {
	// URL is the URL of a downloadable resource.
	URL string `json:"url"`

	// Sha256 provides an optional SHA256 hash of the URL's content for
	// verification.
	Sha256 string `json:"sha256,omitempty"`

	// SecretRef holds a reference to a secret where the credentials are
	// stored. We use these credentials when pulling the artifacts from the
	// provided URL using
	// any of the supported protocols (http, https, and oci).
	SecretRef *corev1.SecretReference `json:"secretRef,omitempty"`

	// InsecureSkipTLSVerify indicates whether certificates in the remote
	// URL (if using TLS) can be ignored.
	InsecureSkipTLSVerify bool `json:"insecureSkipTLSVerify,omitempty"`
}
```

`SecretRef` property is of type `SecretReference` as defined by
`k8s.io/api/core/v1` package. The secret pointed by the provided `SecretRef`
will be used for pulling artifacts using either HTTP\[S\] or OCI protocols.

### Example Configurations

#### Configuration for OCI

```yaml
url: oci://my.registry/binaries/k0s:v1.30.1+k0s.0
sha256: e95603f167cce6e3cffef5594ef06785b3c1c00d3e27d8e4fc33824fe6c38a99
secretRef:
  namespace: kube-system
  name: artifacts-registry
```

#### Configuration for OCI using plain HTTP transport

```yaml
url: oci+http://my.registry/binaries/k0s:v1.30.1+k0s.0
sha256: e95603f167cce6e3cffef5594ef06785b3c1c00d3e27d8e4fc33824fe6c38a99
secretRef:
  namespace: kube-system
  name: artifacts-registry
```

#### Configuration for HTTPS

```yaml
url: https://my.file.server/binaries/k0s-v1.30.1+k0s.0
sha256: e95603f167cce6e3cffef5594ef06785b3c1c00d3e27d8e4fc33824fe6c38a99
secretRef:
  namespace: kube-system
  name: artifacts-basic-auth
```

#### Configuration for HTTP

```yaml
url: http://my.file.server/binaries/k0s-v1.30.1+k0s.0
sha256: e95603f167cce6e3cffef5594ef06785b3c1c00d3e27d8e4fc33824fe6c38a99
secretRef:
  namespace: kube-system
  name: artifacts-token-based-auth
```

### Secrets Layout

The following standard Kubernetes secret types are supported:

- [`kubernetes.io/basic-auth`](https://kubernetes.io/docs/concepts/configuration/secret/#basic-authentication-secret)<br>
  The username and password are used according to the protocol's standard
  procedure for password-based authentication.

- [`kubernetes.io/dockerconfigjson`](https://kubernetes.io/docs/concepts/configuration/secret/#docker-config-secrets)<br>
   It works in the same way as a Pod's [image pull secret]. Only supported for
  the `oci://` protocol. (Might be supported for other protocols in the future,
  as well).

[image pull secret]: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

Potentially supported in the future:

- [`kubernetes.io/tls`](https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets)<br>
  For TLS client authentication.

Moreover, k0s supports the following custom secret type:

- `k0sproject.io/http-authorization-header`<br>
  Sets a custom value for the HTTP Authorization header:

  ```yaml
  apiVersion: v1
  kind: Secret
  metadata:
    name: creds
    namespace: kube-system
  data:
    authorization: "Bearer abc123def456ghi789jkl0"
  ```

  The `authorization` entry is used as is, with its content placed directly into
  the `Authorization` header. A secret like the above will make Autopilot set
  the `Authorization` header to `Bearer abc123def456ghi789jkl0`.

### Additional Details

- The `InsecureSkipTLSVerify` property is equivalent to defining
  `InsecureSkipTLSVerify` on a Go HTTP client.
- The `InsecureSkipTLSVerify` property will be valid for both `oci://` and
  `https://` protocols. It has no effect for the `oci+http://` and `http://`
  protocols.
- If a protocol is not specified or an incorrect one is provided, an error
  state should be activated.
- If no `SecretRef` is defined, access will be anonymous (no authentication).

## Status

Proposed

## Consequences

- Users will have an additional protocol to be aware of.
- If the Secret referenced by `SecretRef` does not exist, the download will
  fail.
- Users need to be notified about different failure types (e.g., unreadable
  secret, invalid secret).
- Additional configuration is required to handle authentication, ensuring
  secure access to resources.
- We will allow downloads from remote places using self-signed certificates if
  requested to.

---

### Contributors Code Guidelines (`docs/contributors/code_guidelines.md`)

# Code guidelines

This page collects guidelines for the k0s code base. Many of them stem from
things that come up frequently during code reviews, or from k0s design
constraints, such as being a per-node agent. These guidelines complement
linters, which enforce general formatting and Go coding style. However, linters
perform static code analysis and can't judge broader concepts, such as whether a
change is the right size or if a comment is appropriate and accurate.

None of this is set in stone. The right trade-off depends on what a particular
change does, and reviewers may weigh things differently from case to case. The
guidelines also evolve with the project: if one of them stops serving its
purpose, change it rather than working around it.

Existing code doesn't follow all of this. When touching such code, prefer these
guidelines over consistency with the surrounding code, but don't churn unrelated
lines just to bring them in line.



## Size the code to the problem

Match the amount of structure to what the change needs. A few lines at the call
site don't need a helper, a helper doesn't need to be exported, and neither
needs its own abstraction until there's a second user. Extra structure isn't
free; it creates more surface area to read, review, and maintain.

Remove dead code when you come across it, but keep such cleanups separate from
functional changes, at least in the [commit message].

## Comments

Comments cost readers time every time they read the code. Keep them crisp. Don't
comment on obvious things or things that readers can quickly figure out. Comment
on what they can't, such as ordering requirements, timing assumptions, or
compatibility with earlier versions. Judge a comment based on what it adds, not
by its length. Restating what the code already says is unhelpful even in a
single line, whereas a comment that explains an important, broader concept which
can't be better documented elsewhere may span several sentences or even
paragraphs.

Comments describe the present, not the past. Version control holds the history.
A comment that explains what the code used to do, which bug motivated a test, or
how a change came about belongs in the [commit message], not in the code.

Avoid referencing file names or line numbers in comments. They become outdated
with the next refactoring, and nothing will tell you. Refer to Go identifiers
instead, using [doc links] where possible, so that the reference stays navigable
and tooling catches renames.

[doc links]: https://go.dev/doc/comment#doclinks

## Errors

An error message is a fragment that gets chained with others through wrapping.
Keep it lowercase and without trailing punctuation, as per the [Go
conventions][error-strings], so that `failed to apply stack: cannot create
resource: connection refused` reads as one line.

Add context to an error only if it's neither part of the error already nor known
to the caller anyway. Many errors, such as those returned by the standard
library's I/O functions, are self-describing, and the caller knows which
function it called. Wrapping them with a message that restates the operation
adds noise, not information.

[error-strings]: https://go.dev/wiki/CodeReviewComments#error-strings

## Logging

Logs are the primary tool users have to understand what k0s is doing on their
systems. They're often read out of context, in between lots of other output, by
someone who doesn't know the code. A good log message tells them what happened
and, ideally, what they can do about it, such as which file, resource, or
configuration to examine. That helps both sides. Users who can troubleshoot
their own systems don't need to ask for help, and the questions that do reach
the developers arrive with the right information attached.

Log what helps readers understand what's going on and why. What's noise in one
situation is a decisive hint in another, and the log level is what separates the
two. It decides who sees a message, and when.

For instance, a periodic task that finds nothing is noise at `Info` level, but
useful at `Debug` level. Pick the level based on what the reader is expected to
do with the message. As a rule of thumb, use `Debug` for details that only
matter when troubleshooting, `Info` for the normal course of events, `Warn` for
problems that k0s handles on its own, but that may need attention if they
persist, and `Error` for failures that require someone to act.

While retries are ongoing, log each failed attempt at `Debug` if failures are
expected or at `Warn` if they are not. Escalate to `Error` once retries are
exhausted or if the operation has not succeeded within a reasonable time. If the
consequences aren't obvious, explain what will happen, e.g. if an operation will
be retried, skipped, or abandoned.

Prefer putting variable data, especially error values, into fields rather than
into the message text, so that messages stay stable and searchable. Fields are
useful when the same name is used across messages because a filter on that name
finds every message about the same thing. Messages without placeholders work the
other way around. They read the same wherever they're logged, so filtering on
them finds all things affected by the same event. Values that no other message
would share are fine in the message text. Keep messages in sync with the
conditions they describe. When a condition changes, its message usually has to
change, too.

Unlike an error message, a log message stands on its own. Write it as a
sentence, starting with a capital letter and without a trailing period.

Log errors where they're handled, not where they're passed on. An error that's
both logged and returned will usually end up in the logs once for every layer it
passes through. When logging an error, avoid repeating what the error message
already says. The message provides context, while the error provides the cause.

## Compatibility across versions

k0s is a per-node agent. Each node reads its own configuration at startup and
acts locally. Cluster-wide coordination is the responsibility of the users or of
additional tooling, such as [Autopilot], [k0sctl] or [k0smotron]. Nodes are
upgraded individually rather than all at once, so controllers and workers of
adjacent versions run side by side for a while, within the limits of the
[version skew policy].

Any change has to work under those conditions: a new version must interoperate
with the previous one, and it must keep reading whatever an earlier version has
persisted, on disk or in the cluster. The burden is on the newer version, since
the older one can't know anything about behavior that didn't exist yet. Whatever
it persists itself must either stay usable by the previous version, or be kept
apart from what the previous version reads, for example by versioning its name
or location.

Prefer designs that converge node by node to those that require every node to
flip simultaneously. If the old and new behaviors can't coexist, don't hide the
switch in the upgrade. Keep the old behavior until users explicitly enable the
new one so that they can choose when the disruption happens, and document what
the switch entails.

Changed configuration defaults must not alter the behavior of clusters that were
created with a previous version. New clusters get the new defaults, while
existing clusters keep behaving as before unless they opt in. This applies to
defaults that stand in for a choice users could have made in the configuration,
not to properties of the release itself, such as the versions of the components
that k0s ships.

### Transitional code

Code that only exists to migrate from an earlier version or clean up leftovers
will be removed eventually. Keep it small and local, and mark it with a `TODO`
note indicating the version in which it can be removed. Don't build abstractions
or test suites around something scheduled for deletion unless the logic is
complex enough to justify the test. Verify it once and explain how in the pull
request.

[Autopilot]: ../autopilot.md
[k0sctl]: ../k0sctl-install.md
[k0smotron]: https://github.com/k0sproject/k0smotron
[version skew policy]: ../version-skew-policy.md

## Tests

Use [testify] for assertions, and add assertion messages that state the
expectation, so that a failure reads as a sentence. Test names describe the
behavior under test, not the reason the test was added. Use obviously made-up
values for test inputs that aren't the subject of the test, so that it's clear
which values matter and which don't. See the [testing guide] for how to run the
different kinds of tests, and what's expected before a pull request is merged.

[testify]: https://github.com/stretchr/testify
[testing guide]: testing.md

[commit message]: github_workflow.md#commit-push

---

### Contributors Docs (`docs/contributors/docs.md`)

# Contributing to the k0s documentation

We use [mkdocs](https://www.mkdocs.org) and [mike](https://github.com/jimporter/mike) for publishing docs to [docs.k0sproject.io](https://docs.k0sproject.io).
This guide will provide a simple how-to on how to configure and deploy newly added docs to our website.

## Requirements

Install mike: https://github.com/jimporter/mike#installation

## Adding A New link to the Navigation

- All docs must live under the `docs` directory (I.E., changes to the main `README.md` are not reflected in the website).
- Add a new link under `nav` in the main [mkdocs.yml](https://github.com/k0sproject/k0s/blob/main/mkdocs.yml) file:

```yaml
# ... other directives
{%
    include "../../mkdocs.yml"
    start="# ~~~ START NAV SNIPPET ~~~\n"
    end="\n# ~~~ END NAV SNIPPET ~~~\n"
%}
  # more navigation links ...
```

- Once your changes are pushed to `main`, the "Publish Docs" job will start running: https://github.com/k0sproject/k0s/actions?query=workflow%3A%22Publish+docs+via+GitHub+Pages%22
- You should see the deployment outcome in the `gh-pages` deployment page: https://github.com/k0sproject/k0s/deployments/activity_log?environment=github-pages

## Testing docs locally

We've got a dockerized setup for easily testing docs locally. Simply run
`make docs-serve-dev`. The docs will be available on http://localhost:8000.

**Note** If you have something already running locally on port `8000` you can
choose another port like so: `make docs-serve-dev DOCS_DEV_PORT=9999`. The docs
will then be available on http://localhost:9999.

---

### Contributors Github Workflow (`docs/contributors/github_workflow.md`)

# GitHub Workflow

This guide assumes you have already cloned the upstream repo to your system via `git clone`, or via `go get github.com/k0sproject/k0s`.

## Fork The Project

1. Go to http://github.com/k0sproject/k0s
2. On the top, right-hand side, click on "fork" and select your username for the fork destination.

## Adding the Forked Remote

```shell
export GITHUB_USER={ your github username }
```

```shell
cd $WORKDIR/k0s
git remote add $GITHUB_USER git@github.com:${GITHUB_USER}/k0s.git

# Prevent push to Upstream
git remote set-url --push origin no_push

# Set your fork remote as a default push target
git push --set-upstream $GITHUB_USER main
```

Your remotes should look something like this:

```shell
git remote -v
```

```shell
origin  https://github.com/k0sproject/k0s (fetch)
origin  no_push (push)
my_fork git@github.com:{ github_username }/k0s.git (fetch)
my_fork git@github.com:{ github_username }/k0s.git (push)
```

## Create & Rebase Your Feature Branch

Create a feature branch and switch to it:

```shell
git checkout -b my_feature_branch
```

Rebase your branch:

```shell
git fetch origin && \
  git rebase origin/main
```

```shell
Current branch my_feature_branch is up to date.
```

Please don't use `git pull` instead of the above `fetch` / `rebase`. `git pull` does a merge, which leaves merge commits. These make the commit history messy and violate the principle that commits ought to be individually understandable and useful.

## Commit & Push

Each commit should serve one purpose. Keep unrelated changes, such as
refactorings or cleanups you make along the way, in separate commits. Commit
**and sign** your changes:

```shell
git commit --signoff
```

Commit messages should have a [short](https://cbea.ms/git-commit/#limit-50),
[capitalized](https://cbea.ms/git-commit/#capitalize) subject [without trailing
period](https://cbea.ms/git-commit/#end) as first line. After the subject a
[blank line](https://cbea.ms/git-commit/#separate) and then a longer description
that [explains why](https://cbea.ms/git-commit/#why-not-how) the change was
made, unless it is obvious.

Use [imperative mood](https://cbea.ms/git-commit/#imperative) in the commit message.

For example:

```text
Summarize changes in around 50 characters or less

More detailed explanatory text, if necessary. Wrap it to about 72
characters or so. In some contexts, the first line is treated as the
subject of the commit and the rest of the text as the body. The
blank line separating the summary from the body is critical (unless
you omit the body entirely); various tools like `log`, `shortlog`
and `rebase` can get confused if you run the two together.

Explain the problem that this commit is solving. Focus on why you
are making this change as opposed to how (the code explains that).
Are there side effects or other unintuitive consequences of this
change? Here's the place to explain them.

Further paragraphs come after blank lines.

 - Bullet points are okay, too

 - Typically a hyphen or asterisk is used for the bullet, preceded
   by a single space, with blank lines in between.

Put references to earlier commits and other relevant material at the
bottom, like this:

Fixes: a50271db8 ("Implement k0s feature gates")
See: f9773809e ("Encapsulate debug flag handling")
See: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/

Signed-off-by: Name Lastname <user@example.com>
```

The commit message is read by whoever looks at this change years from now via
`git blame` or `git log`. Write it for them:

- State what was wrong, what the change does, and why. If the change fixes a
  regression, say what the effect was in released versions.
- Mention non-obvious constraints the code depends on, such as ordering,
  timing, or compatibility with data written by earlier versions. These are
  the things a future reader can't reconstruct from the diff.
- If you remove dead code or do unrelated cleanup while you're there, say so.
  Don't present it as part of the fix.
- Leave out how you found the bug, what you tried, what the change does *not*
  fix, and anything addressed to the reviewer. That material belongs in the
  pull request description or the discussion.
- Don't reference source file paths unless it aids understanding. The diff
  already lists them.

Use `Fixes:` for the commit that introduced the problem this change corrects,
if it can be identified. The issue a change resolves is referenced from the
pull request description instead, where GitHub's closing keywords act on it.
Use `See:` for anything else that helps to understand the change: related
commits, external documentation, specifications, and the like.

When referencing other commits, use the abbreviated hash, e.g. as printed by
`git log --oneline`, along with the subject line: `a50271db8 ("Implement k0s
feature gates")`. That way, the reference is meaningful in `git log`, whether or
not the hash resolves. In general, prefer what works locally over what only
works on GitHub.

Note that GitHub adds a timeline event to an issue or pull request whenever a
commit referencing it is pushed to any public repository, including your own
fork. Since a rebase creates new commits, every force-push repeats those events,
and frequent rebases leave a trail of identical entries in the timeline. That's
another reason to reference issues and pull requests from the pull request
description rather than from commit messages.

When ready, push your changes to your fork's repository:

```shell
git push --set-upstream my_fork my_feature_branch
```

You can go back and edit/build/test some more, then `commit --amend` in a few
cycles.

After review rounds, fold any fixup commits into the commits they belong to and
rewrite the messages so that each commit describes its final change, not its
history. Commits titled "Address review comments" are fine during reviews, but
clutter the history once merged. See [Squashing Commits](#squashing-commits) for
how to do that.

## Open a Pull Request

See GitHub's docs on how to [create a pull request from a fork][pr-from-fork].

Just as with commits, a pull request should serve one purpose. While a pull
request may have a broader goal than a single commit, each commit within it
should be a self-contained step toward that goal. Keep unrelated changes in
separate pull requests. Focused pull requests are easier to understand, review,
and manage. Contested but unrelated parts of a pull request might prevent other
acceptable parts from being merged, and backports become more manageable when
they don't have to be untangled from unrelated changes first.

The purpose of the pull request description is to facilitate discussion. It
should include the context, the alternatives considered, what was verified and
how it was verified, and what the change deliberately leaves out. It must not
contradict the commit messages. When you rewrite commits after a review round,
re-read the description and update it, too.

You may open a pull request in [draft mode]. Automated tests run against a draft
just like against any other pull request, but it will not be assigned for
review. Once a PR is ready for review, mark it as "Ready for review", and code
owners will be notified.

[pr-from-fork]: https://docs.github.com/pull-requests/how-tos/create-pull-requests/creating-a-pull-request-from-a-fork
[draft mode]: https://github.blog/2019-02-14-introducing-draft-pull-requests

### Get a code review

Once your pull request has been opened it will be assigned to one or more
reviewers, and will go through a series of [smoke tests]. It can be merged once
all automated checks pass and a code owner has approved it.

Commit changes made in response to review comments should be added to the same
branch on your fork. Reviews usually take a few rounds. Address feedback by
reworking the commits themselves, rather than by appending fixup commits, so
that each commit is still individually understandable. It's good practice to
rebase them on the current `main` branch while you're at it.

Very small PRs are easy to review. Very large PRs are very difficult to review.

[smoke tests]: testing.md#integration-tests-aka-inttests-or-smoketests

### Squashing Commits

Commits on your branch should represent meaningful milestones or units of work.
Small commits that contain typo fixes, rebases, review feedbacks, etc should be squashed.

To do that, it's best to perform an [interactive rebase](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History):

#### Example

Rebase your feature branch against upstream main branch:

```shell
git rebase -i origin/main
```

If your PR has 3 commits, output would be similar to this:

```shell
pick f7f3f6d Changed some code
pick 310154e fixed some typos
pick a5f4a0d made some review changes

# Rebase 710f0f8..a5f4a0d onto 710f0f8
#
# Commands:
# p, pick <commit> = use commit
# r, reword <commit> = use commit, but edit the commit message
# e, edit <commit> = use commit, but stop for amending
# s, squash <commit> = use commit, but meld into previous commit
# f, fixup <commit> = like "squash", but discard this commit's log message
# x, exec <command> = run command (the rest of the line) using shell
# b, break = stop here (continue rebase later with 'git rebase --continue')
# d, drop <commit> = remove commit
# l, label <label> = label current HEAD with a name
# t, reset <label> = reset HEAD to a label
# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]
# .       create a merge commit using the original merge commit's
# .       message (or the oneline, if no original merge commit was
# .       specified). Use -c <commit> to reword the commit message.
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out
```

Use a command line text editor to change the word `pick` to `f` of `fixup` for the commits you want to squash, then save your changes and continue the rebase:

Per the output above, you can see that:

```shell
fixup <commit> = like "squash", but discard this commit's log message
```

Which means that when rebased, the commit message "fixed some typos" will be removed, and squashed with the parent commit.

### Push Your Final Changes

Once done, you can push the final commits to your branch:

```shell
git push --force
```

You can run multiple iteration of `rebase`/`push -f`, if needed.

---

### Contributors Testing (`docs/contributors/testing.md`)

# Testing Your Code

k0s uses GitHub Actions to run automated tests on pull requests. The workflows
only run after a k0s maintainer has approved them, and a pull request won't be
merged until all tests are green. To save time and prevent your PR from going
stale, run the tests locally before submitting it.

## Run Local Verifications

Please run the following style and formatting commands, and then fix and check
in any changes. These commands catch formatting and common mistakes. For
anything they can't catch, see the [code guidelines](code_guidelines.md).

1. Linting

   We use [`golangci-lint`](https://golangci-lint.run/) for style verification.
   In the repository's root directory, simply run:

   ```shell
   make lint
   ```

   There's no need to install `golangci-lint` manually. The build system will
   take care of that.

2. Go fmt

   ```shell
   go fmt ./...
   ```

3. Checking the documentation

   Follow the [instructions for testing the documentation
   locally][testing-docs-locally] to verify any changes.

4. Pre-submit Flight Checks

   In the repository root directory, make sure that:

   * `make build && git diff --exit-code` runs successfully.  
     Verifies that the build is working and that the generated source code
     matches the one that's checked into source control.
   * `make check-unit` runs successfully.  
     Verifies that all the unit tests pass.
   * `make check-basic` runs successfully.  
     Verifies basic cluster functionality using one controller and two workers.
   * `make check-hacontrolplane` runs successfully.  
     Verifies that joining of controllers works.

   Please note that this last test is prone to "flakiness", so it might fail on
   occasion. If it fails constantly, take a deeper look at your code to find the
   source of the problem.

   If you find that all tests passed, you may open a pull request upstream.

[testing-docs-locally]: docs.md#testing-docs-locally

## Integration tests (a.k.a. inttests or smoketests)

The integration tests are located in the inttest directory and are run in CI as
"smoketests". These tests use Docker and [bootloose-based][bootloose] nodes to
launch k0s clusters locally. See [README.md][inttest-readme].

[bootloose]: https://github.com/k0sproject/bootloose
[inttest-readme]: https://github.com/k0sproject/k0s/blob/main/inttest/README.md

### Requirements

* Docker (Linux) with the ability to run privileged containers.
* Sufficient disk space for the bootloose images and test artifacts.

### Running locally

You can run a single integration test by calling `make` from the repository
root:

```shell
make check-<name>
```

Examples:

```shell
make check-basic
make check-ap-airgap
```

Some tests, such as those related to airgap and IPv6, require image bundles. The
Makefile automatically builds and integrates those when you run `make
check-...`. The underlying environment variables are:

* `K0S_IMAGES_BUNDLE` (airgap bundle)
* `K0S_EXTRA_IMAGES_BUNDLE` (IPv6 bundle)

### Debugging local failures

When a test fails, the suite writes logs and a [support bundle] to the temporary
directory (as determined by [os.TempDir]), e.g.:

* `/tmp/controller*.out.log`, `/tmp/controller*.err.log`
* `/tmp/worker*.out.log`, `/tmp/worker*.err.log`
* `/tmp/support-bundle.tar.gz`

Start by inspecting the logs. You can also use [sbctl] with the support bundle
to inspect the state of the test cluster at the end of the test execution.

[support bundle]: https://github.com/replicatedhq/troubleshoot?tab=readme-ov-file#support-bundle
[os.TempDir]: https://pkg.go.dev/os#TempDir
[sbctl]: https://github.com/replicatedhq/sbctl?tab=readme-ov-file#command-line-tool-for-examining-k8s-resources-in-troubleshoots-support-bundles

### Debugging CI failures

When a CI run fails, the aforementioned files are uploaded and attached as job
artifacts. In the GitHub Actions UI, open the failed job, download and extract
the artifact for that test (named `<smoketest-name>-<arch>-files` or similar)
and extract it. You can then use the extracted files in the same way described
in the previous section.

### Running CI on your fork

The k0s GitHub Action workflows use GitHub's hosted runners whenever possible.
This allows you to run the same CI workflows on your fork by pushing to the main
branch or opening a pull request on your fork. Ensure that GitHub Actions are
enabled for the forked repository. Note that forked repositories do not have
access to k0s's ARMv7 runners; therefore, all ARMv7-related workflow runs will
be skipped.

Running the CI on your fork may provide faster feedback than running it on a
pull request against the k0s repository because running the GitHub workflows for
pull requests requires approval from a k0s maintainer. Additionally, it allows
you to tinker with the tests and workflows however you like to debug things.
This could be useful if you cannot run the smoketests locally.

## Cleanup the local workspace

In order to clean up the local workspace, run `make clean`. It will clean up all
of the intermediate files and directories created during the k0s build. Note
that you can't just use `git clean -X` or even `rm -rf`, since the Go modules
cache sets all of its subdirectories to read-only. If you get in trouble while
trying to delete your local workspace, try `chmod -R u+w /path/to/workspace &&
rm -rf /path/to/workspace`.

---

### Examples Ansible Playbook (`docs/examples/ansible-playbook.md`)

# Creating a cluster with an Ansible Playbook

Ansible is a popular infrastructure-as-code tool that can use to automate tasks for the purpose of achieving the desired state in a system. With Ansible (and the k0s-Ansible playbook) you can quickly install a multi-node Kubernetes Cluster.

**Note**: Before using Ansible to create a cluster, you should have a general understanding of Ansible (refer to the official [Ansible User Guide](https://docs.ansible.com/ansible/latest/user_guide/index.html).

## Prerequisites

You will require the following tools to install k0s on local virtual machines:

| Tool        | Detail                                                                                                                                                                 |
| ----------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `multipass` | A lightweight VM manager that uses KVM on Linux, Hyper-V on Windows, and hypervisor.framework on macOS. [Installation information](https://multipass.run/docs)         |
| `ansible`   | An infrastructure as code tool. [Installation Guide](https://docs.ansible.com/ansible/latest/installation_guide/index.html)                                            |
| `kubectl`   | Command line tool for running commands against Kubernetes clusters.  [Kubernetes Install Tools](https://docs.ansible.com/ansible/latest/installation_guide/index.html) |

## Create the cluster

1. Download k0s-ansible

    Clone the k0s-ansible repository on your local machine:

    ```shell
    git clone https://github.com/movd/k0s-ansible.git
    cd k0s-ansible
    ```

2. Create virtual machines

    **Note**: Though multipass is the VM manager in use here, there is no interdependence.

    Create a number of virtual machines. For the automation to work, each instance must have passwordless SSH access. To achieve this, provision each instance with a cloud-init manifest that imports your current users' public SSH key and into a user `k0s` (refer to the bash script below).

    This creates 7 virtual machines:

    ```shell
    ./tools/multipass_create_instances.sh 7
    ```

    ```shell
    Create cloud-init to import ssh key...
    [1/7] Creating instance k0s-1 with multipass...
    Launched: k0s-1
    [2/7] Creating instance k0s-2 with multipass...
    Launched: k0s-2
    [3/7] Creating instance k0s-3 with multipass...
    Launched: k0s-3
    [4/7] Creating instance k0s-4 with multipass...
    Launched: k0s-4
    [5/7] Creating instance k0s-5 with multipass...
    Launched: k0s-5
    [6/7] Creating instance k0s-6 with multipass...
    Launched: k0s-6
    [7/7] Creating instance k0s-7 with multipass...
    Launched: k0s-7
    Name State IPv4 Image
    k0s-1 Running 192.168.64.32 Ubuntu 20.04 LTS
    k0s-2 Running 192.168.64.33 Ubuntu 20.04 LTS
    k0s-3 Running 192.168.64.56 Ubuntu 20.04 LTS
    k0s-4 Running 192.168.64.57 Ubuntu 20.04 LTS
    k0s-5 Running 192.168.64.58 Ubuntu 20.04 LTS
    k0s-6 Running 192.168.64.60 Ubuntu 20.04 LTS
    k0s-7 Running 192.168.64.61 Ubuntu 20.04 LTS
    ```

3. Create Ansible inventory

    1. Copy the sample to create the inventory directory:

        ```shell
        cp -rfp inventory/sample inventory/multipass
        ```

    2. Create the inventory.

        Assign the virtual machines to the different host groups, as required by the playbook logic.

        | Host group             | Detail                                                                                                                                                    |
        | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
        | `initial_controller`   | Must contain a single node that creates the worker and controller tokens needed by the other nodes                                                        |
        | `controller`           | Can contain nodes that, together with the host from `initial_controller`, form a highly available isolated control plane                                  |
        | `worker`               | Must contain at least one node, to allow for the deployment of Kubernetes objects                                                                         |

    3. Fill in `inventory/multipass/inventory.yml`. This can be done by direct entry using the metadata provided by `multipass list,`, or you can use the following Python script `multipass_generate_inventory.py`:

        ```shell
        ./tools/multipass_generate_inventory.py
        ```

        ```shell
        Designate first three instances as control plane
        Created Ansible Inventory at: /Users/dev/k0s-ansible/tools/inventory.yml
        $ cp tools/inventory.yml inventory/multipass/inventory.yml
        ```

        Your `inventory/multipass/inventory.yml` should resemble the example below:

        ```yaml
        ---
        all:
          children:
            initial_controller:
              hosts:
                k0s-1:
            controller:
              hosts:
                k0s-2:
                k0s-3:
            worker:
              hosts:
                k0s-4:
                k0s-5:
                k0s-6:
                k0s-7:
          hosts:
            k0s-1:
              ansible_host: 192.168.64.32
            k0s-2:
              ansible_host: 192.168.64.33
            k0s-3:
              ansible_host: 192.168.64.56
            k0s-4:
              ansible_host: 192.168.64.57
            k0s-5:
              ansible_host: 192.168.64.58
            k0s-6:
              ansible_host: 192.168.64.60
            k0s-7:
              ansible_host: 192.168.64.61
          vars:
            ansible_user: k0s
        ```

4. Test the virtual machine connections

    Run the following command to test the connection to your hosts:

    ```shell
    ansible -i inventory/multipass/inventory.yml -m ping
    ```

    ```shell
    k0s-4 | SUCCESS => {
        "ansible_facts": {
            "discovered_interpreter_python": "/usr/bin/python3"
        },
        "changed": false,
        "ping": "pong"
    }
    ...
    ```

    If the test result indicates success, you can proceed.

5. Provision the cluster with Ansible

    Applying the playbook, k0s download and be set up on all nodes, tokens will be exchanged, and a kubeconfig will be dumped to your local deployment environment.

    ```shell
    ansible-playbook site.yml -i inventory/multipass/inventory.yml
    ```

    ```
/* Detailed source-code truncated for AI context efficiency. */
```

## Use the cluster with kubectl

A kubeconfig was copied to your local machine while the playbook was running which you can use to gain access to your new Kubernetes cluster:

```shell
export KUBECONFIG=/Users/dev/k0s-ansible/inventory/multipass/artifacts/k0s-kubeconfig.yml
kubectl cluster-info
```

```shell
Kubernetes control plane is running at https://192.168.64.32:6443
CoreDNS is running at https://192.168.64.32:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://192.168.64.32:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

$ kubectl get nodes -o wide
NAME    STATUS     ROLES    AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
k0s-4   Ready      <none>   21s   v1.20.1-k0s1   192.168.64.57   <none>        Ubuntu 20.04.1 LTS   5.4.0-54-generic   containerd://1.4.3
k0s-5   Ready      <none>   21s   v1.20.1-k0s1   192.168.64.58   <none>        Ubuntu 20.04.1 LTS   5.4.0-54-generic   containerd://1.4.3
k0s-6   NotReady   <none>   21s   v1.20.1-k0s1   192.168.64.60   <none>        Ubuntu 20.04.1 LTS   5.4.0-54-generic   containerd://1.4.3
k0s-7   NotReady   <none>   21s   v1.20.1-k0s1   192.168.64.61   <none>        Ubuntu 20.04.1 LTS   5.4.0-54-generic   containerd://1.4.3
```

**Note**: The first three control plane nodes will not display, as the control plane is fully isolated. To check on the distributed etcd cluster, you can use ssh to securely log a controller node, or you can run the following ad-hoc command:

```shell
ansible k0s-1 -a "k0s etcd member-list -c /etc/k0s/k0s.yaml" -i inventory/multipass/inventory.yml | tail -1 | jq
```

```json
{
  "level": "info",
  "members": {
    "k0s-1": "https://192.168.64.32:2380",
    "k0s-2": "https://192.168.64.33:2380",
    "k0s-3": "https://192.168.64.56:2380"
  },
  "msg": "done",
  "time": "2020-12-23T00:21:22+01:00"
}
```

Once all worker nodes are at `Ready` state you can use the cluster. You can test the cluster state by creating a simple nginx deployment.

```shell
kubectl create deployment nginx --image=gcr.io/google-containers/nginx --replicas=5
```

```shell
deployment.apps/nginx created
```

```shell
kubectl expose deployment nginx --target-port=80 --port=8100
```

```shell
service/nginx exposed
```

```shell
kubectl run hello-k0s --image=quay.io/prometheus/busybox --rm -it --restart=Never --command -- wget -qO- nginx:8100
```

```shell
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx on Debian!</title>
...
pod "hello-k0s" deleted
```

**Note**: k0s users are the developers of k0s-ansible. Please send your feedback, bug reports, and pull requests to [github.com/movd/k0s-ansible](https://github.com/movd/k0s-ansible)._

---

### Examples Gitops Flux (`docs/examples/gitops-flux.md`)

# Using GitOps with Flux

This tutorial describes the benefits of using GitOps with k0s and provides an example of deploying an application with Flux v2.

GitOps is a practice where you leverage Git as the single source of truth. It offers a declarative way to do Kubernetes cluster management and application delivery. The desired states, using Kubernetes manifests and helm packages, are pulled from a git repository and automatically deployed to the cluster. This also makes it quick to re-deploy and recover applications whenever needed.



## Why GitOps with k0s

k0s doesn't come with a lot of different extensions and add-ons that some users might find useful (and some not). Instead, k0s comes with 100% upstream Kubernetes and is compatible with all Kubernetes extensions. This makes it easy for k0s users to freely select the needed extensions that their applications and infrastructure need, without conflicting to any predefined options. Now, GitOps is a perfect practice to deploy these extensions automatically with applications by defining and configuring them directly in Git. This will also help with cluster security as the cluster doesn't need to be accessed directly when application changes are needed. However, this puts more stress on the Git access control, because changes in Git are propagated automatically to the cluster.

## Install k0s

Let's start by installing k0s. Any k0s deployment option will do, but to keep things simple, this [Quick Start Guide](../install.md) gets you started with a single node k0s cluster.

Run these three commands to download k0s, install and start it:

  ```shell
  curl --proto '=https' --tlsv1.2 -sSf https://get.k0s.sh | sudo sh
  sudo k0s install controller --single
  sudo k0s start
  ```

## Set kubeconfig

Next, you need to set the KUBECONFIG variable, which is needed by Flux CLI later on.

  ```shell
  sudo k0s kubeconfig admin > kubeconfig
  export KUBECONFIG=$PWD/kubeconfig
  ```

## Install Flux

To proceed with Flux, install the Flux CLI, which is used for configuring Flux to your Kubernetes cluster. For macOS and Linux, this can be done either with brew or bash script. Use one of them:

  ```shell
  brew install fluxcd/tap/flux
  ```

or

  ```shell
  curl --proto '=https' --tlsv1.2 -sSf https://fluxcd.io/install.sh | sudo bash
  ```

For more details of the Flux installation, check the [Flux documentation](https://fluxcd.io/docs/get-started/).

## Configure Flux for a GitHub repository

Export your GitHub personal access token ([instructions how to get it](https://docs.github.com/en/enterprise-server@3.4/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)) and username:

  ```shell
  export GITHUB_TOKEN=<your-token>
  export GITHUB_USER=<your-username>
  ```

Come up with a GitHub repo name (e.g. flux-demo), which will be used by Flux to store (and sync) the config files.

  ```shell
  export GITHUB_REPO_NAME=<select-repo-name-to-be-created>
  ```

Bootstrap flux to your cluster. The GitHub repo will be created automatically by Flux:

  ```shell
  flux bootstrap github \
  --owner=$GITHUB_USER \
  --repository=$GITHUB_REPO_NAME \
  --branch=main \
  --path=./clusters/my-cluster \
  --personal
  ```

Now you are all set with Flux and can proceed to deploy your first application.

## Deploy example application

Next, we'll deploy a simple web application and expose it using a NodePort service. In the previous step, we configured Flux to track the path /clusters/my-cluster/ in your repository. Now clone the repo to your local machine:

  ```shell
  git clone git@github.com:$GITHUB_USER/$GITHUB_REPO_NAME.git
  cd $GITHUB_REPO_NAME/clusters/my-cluster/
  ```

Create the following YAML file (simple-web-server-with-nodeport.yaml) into the same directory:

  ```YAML
  apiVersion: v1
  kind: Namespace
  metadata:
    name: web
  ---
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: web-server
    namespace: web
  spec:
    selector:
      matchLabels:
        app: web
    template:
      metadata:
        labels:
          app: web
      spec:
        containers:
        - name: httpd
          image: httpd:2.4.53-alpine
          ports:
          - containerPort: 80
  ---
  apiVersion: v1
  kind: Service
  metadata:
    name: web-server-service
    namespace: web
  spec:
    type: NodePort
    selector:
      app: web
    ports:
      - port: 80
        targetPort: 80
        nodePort: 30003
  ```

Then push the new file to the repository:

  ```shell
  git add .
  git commit -m "Add web server manifest"
  git push
  ```

Check that Flux detects your changes and the web server gets applied (by default this should happen within 1 min):

  ```shell
  flux get kustomizations
  ```

If the deployment went successfully, you should see the newly added objects:

  ```shell
  sudo k0s kubectl get all -n web
  ```

You can try to access the web application using

  ```shell
  curl localhost:30003
  ```

or by using a web browser [http://localhost:30003](http://localhost:30003).

**Voilà!** You have now installed the example application using the GitOps method with Flux. As a next step you can try to modify the web app YAML file or add another application directly in to the Git repo and see how Flux will automatically pick up the changes without accessing the cluster with kubectl.

## Uninstall Flux

If you want to uninstall Flux from the cluster, run:

  ```shell
  flux uninstall --namespace=flux-system
  ```

Your applications, which were installed by Flux, will remain in the cluster, but you don't have the Flux processes anymore to sync up the desired state from Git.

---

### Examples Longhorn (`docs/examples/longhorn.md`)

# Longhorn

This tutorial covers installing Longhorn as a Helm extension on k0s. Longhorn is a distributed block storage system for Kubernetes that provides enterprise-grade features including:

- **Distributed storage**: Provides persistent storage across your cluster nodes
- **Snapshots and backups**: Built-in snapshot and backup capabilities for data protection
- **Live upgrades**: Zero-downtime upgrades of storage volumes
- **High availability**: Automatic failover and replication for data durability
- **Cross-cluster disaster recovery**: Backup to external storage systems

Longhorn is particularly well-suited for stateful applications that require persistent storage, such as databases, message queues, and content management systems.

For more information on Longhorn, refer to the [official documentation](https://longhorn.io/docs/).

## Prerequisites

Before installing Longhorn, ensure your k0s cluster meets these requirements:

### System Requirements

Please refer to the [official Longhorn v1.10.x Support Matrix](https://www.suse.com/suse-longhorn/support-matrix/all-supported-versions/longhorn-v1-10-x/) for detailed system requirements.

### Required Software

Longhorn requires the iSCSI initiator utilities to be installed on every node in your cluster:

**Why iSCSI is required**: Longhorn uses iSCSI protocol to provide block storage to pods. The iSCSI initiator allows nodes to connect to Longhorn volumes as if they were local block devices.

Ubuntu/Debian:

```sh
sudo apt-get install -y open-iscsi
```

RHEL/CentOS/Rocky:

```sh
sudo yum install -y iscsi-initiator-utils
```

SUSE:

```sh
sudo zypper install -y open-iscsi
```

Ensure the iSCSI daemon is enabled and running:

```sh
sudo systemctl enable --now iscsid
```

## Install via k0s Helm extension

Add Longhorn as a Helm chart in your k0s ClusterConfig. Apply this configuration through your normal k0s configuration workflow.

### Basic Installation

The simplest way to install Longhorn is to add it to your k0s ClusterConfig:

```yaml
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  extensions:
    helm:
      repositories:
        - name: longhorn
          url: https://charts.longhorn.io
      charts:
        - name: longhorn
          chartname: longhorn/longhorn
          namespace: longhorn-system
          version: 1.10.1
```

### Verification Steps

After applying the configuration, verify that Longhorn is properly installed:

**Check pod status**: Wait for all Longhorn components to become ready:

```sh
k0s kubectl -n longhorn-system get pods
```

Expected output should show all pods in `Running` status. This may take 2-3 minutes.

**Verify StorageClass**: Confirm that the `longhorn` StorageClass was created:

```sh
$ k0s kubectl get storageclass
NAME                 PROVISIONER          RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
longhorn (default)   driver.longhorn.io   Delete          Immediate           true                   16m
longhorn-static      driver.longhorn.io   Delete          Immediate           true                   16m
```

You should see `longhorn` listed as a storage class.

### Single-Node Cluster Configuration

Single-node clusters require setting the default replica count to 1 to prevent scheduling failures, as Longhorn's default of 3 replicas cannot be satisfied on a single node. This setting can be adjusted in the values or post-installation via the UI (Settings > General > Default Replica Count).

```yaml
spec:
  extensions:
    helm:
      charts:
        - name: longhorn
          chartname: longhorn/longhorn
          namespace: longhorn-system
          version: "latest"
          values: |
            defaultSettings:
              defaultReplicaCount: 1
```

## Access the Longhorn UI

You can port-forward the frontend service:

```sh
k0s kubectl -n longhorn-system port-forward svc/longhorn-frontend 8080:80
```

Then open <http://localhost:8080> in your browser.

## Example: PVC and Deployment

Use this example to create a PVC and mount it in a Deployment using Longhorn.

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: app-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: nginx
          name: nginx
          volumeMounts:
            - name: persistent-storage
              mountPath: /var/lib/nginx
      volumes:
        - name: persistent-storage
          persistentVolumeClaim:
            claimName: app-pvc
```

Apply the manifest:

```sh
k0s kubectl apply -f nginx.yaml
```

Check the PVC and pod status:

```sh
k0s kubectl get pvc,pods
```

## Next Steps

Longhorn supports backing up volumes to external storage systems such as S3, NFS, or CIFS. To enable backups:

1. Create a backup target secret with your storage credentials
2. Configure the backup target URL in Longhorn settings
3. Set up recurring backup jobs for automatic protection

For detailed instructions on configuring backup targets and setting up automated backups, refer to the [Longhorn Backup Documentation](https://longhorn.io/docs/1.10.1/snapshots-and-backups/backup-and-restore/set-backup-target/).

## Additional Resources

- [Longhorn Official Documentation](https://longhorn.io/docs/)
- [Longhorn Architecture Guide](https://longhorn.io/docs/1.10.1/concepts/)
- [Best Practices for Production Deployments](https://longhorn.io/docs/1.10.1/best-practices/)
- [Troubleshooting Guide](https://longhorn.io/docs/1.10.1/troubleshoot/troubleshooting/)

---

### Examples Metallb Loadbalancer (`docs/examples/metallb-loadbalancer.md`)

# Installing MetalLB Load Balancer

This tutorial covers the installation of MetalLB load balancer on k0s. k0s doesn't come with an in-built load balancer, but it's easy to deploy MetalLB as shown in this document.

## About Load Balancers

Load balancers can be used for exposing applications to the external network. Load balancer provides a single IP address to route incoming requests to your app. In order to successfully create Kubernetes services of type LoadBalancer, you need to have the load balancer (implementation) available for Kubernetes.

Load balancer can be implemented by a [cloud provider](../cloud-providers.md) as an external service (with additional cost). This can also be implemented internally in the Kubernetes cluster (pure SW solution) with MetalLB.

## MetalLB

MetalLB implements the Kubernetes service of type LoadBalancer. When a LoadBalancer service is requested, MetalLB allocates an IP address from the configured range and makes the network aware that the IP “lives” in the cluster.



One of the benefits of MetalLB is that you avoid all cloud provider dependencies. That's why MetalLB is typically used for bare-metal deployments.

See the MetalLB requirements in the [MetalLB's official documentation](https://metallb.universe.tf/#requirements). By default, k0s runs with Kube-Router CNI, which is compatible with MetalLB as long as you don't use MetalLB’s BGP mode. If you are not using Kube-Router and you are using kube-proxy in IPVS mode, you need to enable strict ARP mode in kube-proxy (see [MetalLB preparations](https://metallb.universe.tf/installation/#preparation)):

```yaml
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig

metadata:
  name: k0s

spec:
  network:
    kubeProxy:
      mode: ipvs
      ipvs:
        strictARP: true
```

Port 7946 (TCP & UDP) must be allowed between the nodes. In addition, before installing MetalLB, make sure there is no other software running on port 7946 on the nodes, such as docker daemon.

### Install MetalLB

1. Install MetalLB using the official Helm chart and k0s Helm [extension manager](../helm-charts.md):

   ```yaml
   apiVersion: k0s.k0sproject.io/v1beta1
   kind: ClusterConfig

   metadata:
     name: k0s

   spec:
     extensions:
       helm:
         repositories:
           - name: metallb
             url: https://metallb.github.io/metallb

         charts:
           - name: metallb
             chartname: metallb/metallb
             version: "0.15.3"
             namespace: metallb-system
   ```

   Other installation methods are available in the [MetalLB's official documentation](https://metallb.org/installation/).

2. Create ConfigMap for MetalLB

   Next you need to create ConfigMap, which includes an IP address range for the load balancer. The pool of IPs must be dedicated to MetalLB's use. You can't reuse for example the Kubernetes node IPs or IPs controlled by other services. You can, however, use private IP addresses, for example 192.168.1.180-192.168.1.199, but then you need to take care of the routing from the external network if you need external access. In this example, we don't need it.

   Create a YAML file accordingly, and deploy it: `kubectl apply -f metallb-l2-pool.yaml`

   ```YAML
   ---
   apiVersion: metallb.io/v1beta1
   kind: IPAddressPool

   metadata:
     name: first-pool
     namespace: metallb-system

   spec:
     addresses:
       - <ip-address-range-start>-<ip-address-range-stop>
       # or
       - <ip-address>/<cidr>
   ---
   apiVersion: metallb.io/v1beta1
   kind: L2Advertisement

   metadata:
     name: example
     namespace: metallb-system
   ```

3. Deploy an example application (web server) with a load balancer

   ```YAML
   apiVersion: v1
   kind: Namespace

   metadata:
     name: web
   ---
   apiVersion: apps/v1
   kind: Deployment

   metadata:
     name: web-server
     namespace: web

   spec:
     selector:
       matchLabels:
         app: web
     template:
       metadata:
         labels:
           app: web
       spec:
         containers:
         - name: httpd
           image: httpd:2.4.53-alpine
           ports:
           - containerPort: 80
   ---
   apiVersion: v1
   kind: Service

   metadata:
     name: web-server-service
     namespace: web

   spec:
     selector:
       app: web
     ports:
       - protocol: TCP
         port: 80
         targetPort: 80
     type: LoadBalancer
   ```

4. Check your LoadBalancer

   Run the following command to see your LoadBalancer with the external-ip and port.

   ```shell
   kubectl get service -n web
   ```

5. Access your example application

   If you used private IP addresses for MetalLB in the ConfigMap (in step 2), you should run the following command from the local network. Use the IP address from the previous step.

   ```shell
   curl <EXTERNAL-IP>
   ```

   If you are successful, you should see `<html><body><h1>It works!</h1></body></html>`.

## Additional information

For more information about MetalLB installation, take a look at [the official MetalLB documentation](https://metallb.universe.tf/installation/).

## Alternative examples

Get load balancer using [cloud provider](../cloud-providers.md).

---

### Examples Nginx Ingress (`docs/examples/nginx-ingress.md`)

# Installing NGINX Ingress Controller

This tutorial covers the installation of [Ingress NGINX Controller], which is an
open source project made by the Kubernetes community. k0s doesn't come with a
built-in ingress controller, but it's easy to deploy Ingress NGINX Controller as
shown in this document. Other ingress solutions can be used as well (see the
links at the end of the page).

[Ingress NGINX Controller]: https://github.com/kubernetes/ingress-nginx

## `NodePort` vs. `LoadBalancer` vs. `Ingress`

Kubernetes offers multiple options for exposing services to external networks. The main options are `NodePort`, `LoadBalancer` and `Ingress`.

**NodePort**, as the name says, means that a port on a node is configured to route incoming requests to a certain service. The port range is limited to 30000-32767, so you cannot expose commonly used ports like 80 or 443 via `NodePort`.

**LoadBalancer** is a service, which is typically implemented by the cloud provider as an external service (with additional cost). Load balancers can also be installed internally in the Kubernetes cluster with MetalLB, which is typically used for bare-metal deployments. Load balancer provides a single IP address to access your services, which can run on multiple nodes.

**Ingress controller** helps to consolidate routing rules of multiple applications into one entity. Ingress controller is exposed to an external network with the help of `NodePort`, `LoadBalancer` or host network. You can also use Ingress controller to terminate TLS for your domain in one place, instead of terminating TLS for each application separately.

## Ingress NGINX Controller

Ingress NGINX Controller is a very popular ingress for Kubernetes. In many cloud environments, it can be exposed to an external network by using the load balancer offered by the cloud provider. However, cloud load balancers are not necessary. Load balancer can also be implemented with MetalLB, which can be deployed in the same Kubernetes cluster. Another option to expose the Ingress controller to an external network is to use `NodePort`. The third option is to use host network. All of these alternatives are described in more detail on below, with separate examples.



### Installation using `NodePort`

Installing using `NodePort` is the most simple example for Ingress NGINX Controller as we can avoid the load balancer dependency. `NodePort` is used for exposing the controller to the external network.

1. Install Ingress NGINX Controller (using the official manifests from the official `ingress-nginx` Kubernetes git repository)

    ```shell
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.3/deploy/static/provider/baremetal/deploy.yaml
    ```

2. Check that the Ingress controller pods have started

    ```shell
    kubectl get pods -n ingress-nginx
    ```

3. Check that you can see the NodePort service

    ```shell
    kubectl get services -n ingress-nginx
    ```

4. From version `v1.0.0` of the Ingress NGINX Controller, [a ingressclass object is required](https://kubernetes.github.io/ingress-nginx/#what-is-an-ingressclass-and-why-is-it-important-for-users-of-ingress-nginx-controller-now).

    In the default installation, an ingressclass object named `nginx` has already been created.

    ```shell
    $ kubectl -n ingress-nginx get ingressclasses
    NAME    CONTROLLER             PARAMETERS   AGE
    nginx   k8s.io/ingress-nginx   <none>       162m
    ```

    If this is only instance of the Ingress NGINX Controller, you should [add the annotation](https://kubernetes.github.io/ingress-nginx/#i-have-only-one-instance-of-the-ingresss-nginx-controller-in-my-cluster-what-should-i-do) `ingressclass.kubernetes.io/is-default-class` in your ingress class:

    ```shell
    kubectl -n ingress-nginx annotate ingressclasses nginx ingressclass.kubernetes.io/is-default-class="true"
    ```

5. Try connecting the controller using the `NodePort` from the previous step (in the range of 30000-32767)

    ```shell
    curl <worker-external-ip>:<node-port>
    ```

    If you don't yet have any backend service configured, you should see "404 Not Found" from nginx. This is okay for now. If you see a response from nginx, the ingress controller is running and you can reach it.

6. Deploy a small test application (Apache HTTP Server) to verify your ingress controller.

    Create the following YAML file and name it `simple-web-server-with-ingress.yaml`:

    ```yaml
    apiVersion: v1
    kind: Namespace
    metadata:
      name: web
    ---
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: web-server
      namespace: web
    spec:
      selector:
        matchLabels:
          app: web
      template:
        metadata:
          labels:
            app: web
        spec:
          containers:
          - name: httpd
            image: httpd:2.4.53-alpine
            ports:
            - containerPort: 80
    ---
    apiVersion: v1
    kind: Service
    metadata:
      name: web-server-service
      namespace: web
    spec:
      selector:
        app: web
      ports:
        - protocol: TCP
          port: 5000
          targetPort: 80
    ---
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: web-server-ingress
      namespace: web
    spec:
      ingressClassName: nginx
      rules:
      - host: web.example.com
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-server-service
                port:
                  number: 5000
    ```

    Deploy the app:

    ```shell
    kubectl apply -f simple-web-server-with-ingress.yaml
    ```

7. Verify that you can access your application using the `NodePort` from step 3.

    ```shell
    curl <worker-external-ip>:<node-port> -H 'Host: web.example.com'
    ```

    If you are successful, you should see ```<html><body><h1>It works!</h1></body></html>```.

### Install using `LoadBalancer`

In this example you'll install Ingress NGINX Controller using `LoadBalancer` on k0s.

1. Install a `LoadBalancer`

    There are two alternatives to install `LoadBalancer` on k0s. Follow the links in order to install `LoadBalancer`.

    - [MetalLB](metallb-loadbalancer.md) as a pure software solution running internally in the k0s cluster
    - [Cloud provider's](../cloud-providers.md) load balancer running outside of the k0s cluster

2. Verify `LoadBalancer`

    In order to proceed you need to have a load balancer available for the Kubernetes cluster. To verify that it's available, deploy a simple Service of type `LoadBalancer`.

    ```yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: example-load-balancer
    spec:
      selector:
        app: web
      ports:
        - protocol: TCP
          port: 80
          targetPort: 80
      type: LoadBalancer
    ```

    ```shell
    kubectl apply -f example-load-balancer.yaml
    ```

    Then run the following command to see your `LoadBalancer` with an external IP address.

    ```shell
    kubectl get service example-load-balancer
    ```

    If the `LoadBalancer` is not available, you won't get an IP address for EXTERNAL-IP. Instead, it's ```<pending>```. In this case you should go back to the previous step and check your load balancer availability.

    If you are successful, you'll see a real IP address and you can proceed further.

    You can delete the example-load-balancer:

    ```shell
    kubectl delete -f example-load-balancer.yaml
    ```

3. Install Ingress NGINX Controller by following the steps in the previous chapter (step 1 to step 4).

4. Edit the Ingress NGINX Controller to use `LoadBalancer` instead of `NodePort`

    ```shell
    kubectl edit service ingress-nginx-controller -n ingress-nginx
    ```

    Find the spec.type field and change it from `NodePort` to `LoadBalancer`.

5. Check that you can see the `ingress-nginx` service with type `LoadBalancer`.

    ```shell
    kubectl get services -n ingress-nginx
    ```

6. Try connecting to the ingress controller

    If you used private IP addresses for MetalLB in step 2, you should run the following command from the local network. Use the IP address from the previous step, column EXTERNAL-IP.

    ```shell
    curl <EXTERNAL-IP>
    ```

    If you don't yet have any backend service configured, you should see "404 Not Found" from nginx. This is okay for now. If you see a response from nginx, the Ingress Controller is running and you can reach it using `LoadBalancer`.

7. Deploy a small test application (Apache HTTP Server) to verify your `Ingress`.

    Create the YAML file `simple-web-server-with-ingress.yaml` as described in the previous chapter (step 6) and deploy it.

    ```shell
    kubectl apply -f simple-web-server-with-ingress.yaml
    ```

8. Verify that you can access your application through the `LoadBalancer` and ingress controller.

    ```shell
    curl <worker-external-ip> -H 'Host: web.example.com'
    ```

    If you are successful, you should see ```<html><body><h1>It works!</h1></body></html>```.

### Install using host network

The host network option exposes Ingress directly using the worker nodes' IP addresses. It also allows you to use ports 80 and 443. This option doesn't use any Service objects (`ClusterIP`, `NodePort`, `LoadBalancer`) and it has the limitation that only one Ingress controller Pod may be scheduled on each cluster node.

1. Install Ingress NGINX Controller manifests:

    ```shell
    wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.3/deploy/static/provider/baremetal/deploy.yaml
    ```

2. Edit `deploy.yaml`. Find the Deployment ingress-nginx-controller and enable the host network option by adding the `hostNetwork` line:

    ```shell
    spec:
      template:
        spec:
          hostNetwork: true
    ```

    You can also remove the Service ingress-nginx-controller completely, because it won't be needed.

3. Install Ingress

    ```shell
    kubectl apply -f deploy.yaml
    ```

4. Try to connect to the ingress controller, deploy a test application and verify the access. These steps are similar to the previous install methods.

## Additional information

For more information about Ingress NGINX Controller installation, take a look at
the official [installation guide] and [bare-metal considerations].

[installation guide]: https://kubernetes.github.io/ingress-nginx/deploy/
[bare-metal considerations]: https://kubernetes.github.io/ingress-nginx/deploy/baremetal/

## Alternative examples for ingress controllers on k0s

[Traefik Ingress](traefik-ingress.md)

---

### Examples Openebs (`docs/examples/openebs.md`)

# OpenEBS

This tutorial covers the installation of OpenEBS as a Helm extension. OpenEBS
can be installed as a helm chart by adding it as an extension to your
configuration.

## Configuration

Most options can be directly set via the `values` section of the chart config.

[Chart in Artifacthub](https://artifacthub.io/packages/helm/openebs/openebs)

### Kubelet path

Be sure to set the correct kubelet path as the default one from the chart is `/var/lib/kubelet`, but k0s uses `/var/lib/k0s/kubelet`.

You can skip this if you set a [custom directory](https://docs.k0sproject.io/stable/configuration/?h=data+dir#kubelet-root-directory) via `--kubelet-root-dir`.

More infos in the [OpenEBS Quickstart Guide](https://openebs.io/docs/quickstart-guide/installation).

```yaml
lvm-localpv:
  lvmNode:
    kubeletDir: /var/lib/k0s/kubelet
zfs-localpv:
  zfsNode:
    kubeletDir: /var/lib/k0s/kubelet
mayastor:
  csi:
    node:
      kubeletDir: /var/lib/k0s/kubelet
```

### Storage engines

Each storage engine can be individually enabled or disabled.

The default of the chart is to enable all engines except "rawfile".
If you want to disable replicated storage via mayastor, check the example below.

You can always check the default values here: https://github.com/openebs/openebs/blob/helm-testing/release/4.4/charts/values.yaml

```yaml
engines:
  local:
    lvm:
      enabled: true
    rawfile:
      enabled: true
    zfs:
      enabled: true
    rawfile:
        enabled: false
  replicated:
    mayastor:
      enabled: true
```

### Monitoring

Per default the chart also installs Loki, Minio and Alloy for logging purposes.
If you do not need these you can disable them.

```yaml
alloy:
  enabled: false
loki:
  enabled: false
  minio:
    enabled: false
```

### Local storage only variant

```yaml
extensions:
  helm:
    repositories:
      - name: openebs-internal
        url: https://openebs.github.io/openebs
    charts:
      - name: openebs
        chartname: openebs-internal/openebs
        version: "4.4.0"
        namespace: openebs
        order: 1
        values: |
          engines:
            replicated:
              mayastor:
                enabled: false
          lvm-localpv:
            lvmNode:
              kubeletDir: /var/lib/k0s/kubelet
          zfs-localpv:
            zfsNode:
              kubeletDir: /var/lib/k0s/kubelet
          localpv-provisioner:
            hostpathClass:
              isDefaultClass: false
```

### Local + replicated storage variant

Be sure to read the [prerequisites](https://openebs.io/docs/quickstart-guide/prerequisites) of OpenEBS for distributed storage, as some host dependencies are needed.

```yaml
extensions:
  helm:
    repositories:
      - name: openebs-internal
        url: https://openebs.github.io/openebs
    charts:
      - name: openebs
        chartname: openebs-internal/openebs
        version: "4.4.0"
        namespace: openebs
        order: 1
        values: |
          lvm-localpv:
            lvmNode:
              kubeletDir: /var/lib/k0s/kubelet
          zfs-localpv:
            zfsNode:
              kubeletDir: /var/lib/k0s/kubelet
          mayastor:
            csi:
              node:
                kubeletDir: /var/lib/k0s/kubelet
          # uncomment this to only use 1 cpu per io-engine (for smaller workloads)
          # io_engine:
          #   cpuCount: "1"  
          localpv-provisioner:
            hostpathClass:
              isDefaultClass: false
```

### Set default storage class

If you want OpenEBS "hostpath" to be your default storage class, set `isDefaultClass` to `true`.

## Usage

Once installed, the cluster will have multiple storage classes available for you to use (depending on the enabled engines).

```shell
k0s kubectl get storageclass
```

```shell
NAME                         PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
mayastor-etcd-localpv        openebs.io/local                    Delete          WaitForFirstConsumer   false                  76m
openebs-hostpath (default)   openebs.io/local                    Delete          WaitForFirstConsumer   false                  76m

# and depending on the enabled storage providers e.g.:
openebs-single-replica       io.openebs.csi-mayastor             Delete          Immediate              true                   76m
```

The `openebs-hostpath` is the storage class that maps to `/var/openebs/local`.

The `openebs-device` is not configured and could be configured by [manifest deployer](../manifests.md) accordingly to the [OpenEBS documentation](https://docs.openebs.io/)

### Example

Use following manifests as an example of pod with mounted volume:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: openebs-hostpath
  resources:
    requests:
      storage: 5Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - image: nginx
          name: nginx
          volumeMounts:
            - name: persistent-storage
              mountPath: /var/lib/nginx
      volumes:
        - name: persistent-storage
          persistentVolumeClaim:
            claimName: nginx-pvc
```

```shell
k0s kubectl apply -f nginx.yaml
```

```shell
persistentvolumeclaim/nginx-pvc created
deployment.apps/nginx created
bash-5.1# k0s kc get pods
NAME                    READY   STATUS    RESTARTS   AGE
nginx-d95bcb7db-gzsdt   1/1     Running   0          30s
```

```shell
k0s kubectl get pv
```

```shell
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS       REASON   AGE
pvc-9a7fae2d-eb03-42c3-aaa9-1a807d5df12f   5Gi        RWO            Delete           Bound    default/nginx-pvc   openebs-hostpath            30s
```

---

### Examples Rook Ceph (`docs/examples/rook-ceph.md`)

# Installing Ceph Storage with Rook

In this tutorial you'll create a Ceph storage for k0s. Ceph is a highly scalable, distributed storage solution. It offers object, block, and file storage, and it's designed to run on any common hardware. Ceph implements data replication into multiple volumes that makes it fault-tolerant. Another clear advantage of Ceph in Kubernetes is the dynamic provisioning. This means that applications just need to request the storage (persistent volume claim) and Ceph will automatically provision the requested storage without a manual creation of the persistent volume each time.

Unfortunately, the Ceph deployment as such can be considered a bit complex. To make the deployment easier, we'll use Rook operator. Rook is a CNCF project and it's dedicated to storage orchestration. Rook supports several storage solutions, but in this tutorial we will use it to manage Ceph.

This tutorial uses three worker nodes and one controller. It's possible to use less nodes, but using three worker nodes makes it a good example for deploying a high-available storage cluster. We use external storage partitions, which are assigned to the worker nodes to be used by Ceph.

After the Ceph deployment we'll deploy a sample application (MongoDB) to use the storage in practice.



## Prerequisites

- Linux
- GitHub access
- AWS account
- Terraform

## Deployment steps

### 1. Preparations

In this example we'll use Terraform to create four Ubuntu virtual machines on AWS. Using Terraform makes the VM deployment fast and repeatable. You can avoid manually setting up everything in the AWS GUI. Moreover, when you have finished with the tutorial, it's very easy to tear down the virtual machines with Terraform (with one command). However, you can set up the nodes in many different ways and it doesn't make a difference in the following steps.

We will use k0sctl to create the k0s cluster. The k0sctl repository also includes a ready-made Terraform configuration to create the virtual machines on AWS. We'll use that. Let's start be cloning the k0sctl repo.

```sh
git clone git@github.com:k0sproject/k0sctl.git
```

Take a look at the Terraform files

```sh
cd k0sctl/examples/aws-tf
ls -l
```

Open `variables.tf` and set the number of controller and worker nodes like this:

```terraform
variable "cluster_name" {
  type    = string
  default = "k0sctl"
}

variable "controller_count" {
  type    = number
  default = 1
}

variable "worker_count" {
  type    = number
  default = 3
}

variable "cluster_flavor" {
  type    = string
  default = "t3.small"
}
```

Open `main.tf` to check or modify k0s version near the end of the file.

You can also configure a different name to your cluster and change the default VM type. `t3.small` (2 vCPUs, 2 GB RAM) runs just fine for this tutorial.

### 2. Create the virtual machines

For AWS, you need an account. Terraform will use the following environment variable: `AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN`. You can easily copy-paste them from the AWS portal. For more information, see [the AWS documentation](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html).



When the environment variables are set, you can proceed with Terraform and deploy the virtual machines.

```shell
terraform init
terraform apply
```

If you decide to create the virtual machines manually using AWS GUI, you need to disable source / destination checking. This needs to be disabled always for multi-node Kubernetes clusters in order to get the node-to-node communication working due to Network Address Translation. For Terraform this is already taken care of in the default configuration.

### 3. Create and attach the volumes

Ceph requires one of the following storage options for storing the data:

- Raw devices (no partitions or formatted filesystems)
- Raw partitions (no formatted filesystem)
- PVs available from a storage class in block mode

We will be using raw partitions (AWS EBS volumes), which can be easily attached to the worker node virtual machines. They are automatically detected by Ceph with its default configuration.

Deploy AWS EBS volumes, one for each worker node. You can manually create three EBS volumes (for example 10 GB each) using the AWS GUI and attach those to your worker nodes. Formatting shouldn't be done. Instead, Ceph handles that part automatically.

After you have attached the EBS volumes to the worker nodes, log in into one of the workers and check the available block devices:

```shell
lsblk -f
```

```shell
NAME        FSTYPE   LABEL           UUID                                 FSAVAIL FSUSE% MOUNTPOINT
loop0       squashfs                                                            0   100% /snap/amazon-ssm-agent/3552
loop1       squashfs                                                            0   100% /snap/core18/1997
loop2       squashfs                                                            0   100% /snap/snapd/11588
loop3       squashfs                                                            0   100% /snap/lxd/19647
nvme0n1
└─nvme0n1p1 ext4     cloudimg-rootfs e8070c31-bfee-4314-a151-d1332dc23486    5.1G    33% /
nvme1n1
```

The last line (`nvme1n1`) in this example printout corresponds to the attached EBS volume. Note that it doesn't have any filesystem (`FSTYPE` is empty). This meets the Ceph storage requirements and you are good to proceed.

### 4. Install k0s using k0sctl

You can use Terraform to automatically output a config file for k0sctl with the IP addresses and access details.

```shell
terraform output -raw k0s_cluster > k0sctl.yaml
```

After that deploying k0s becomes very easy with the ready-made configuration.

```shell
k0sctl apply --config k0sctl.yaml
```

It might take around 2-3 minutes for k0sctl to connect each node, install k0s and connect the nodes together to form a cluster.

### 5. Access k0s cluster

To access your new cluster remotely, you can use k0sctl to fetch kubeconfig and use that with kubectl or Lens.

```shell
k0sctl kubeconfig --config k0sctl.yaml > kubeconfig
export KUBECONFIG=$PWD/kubeconfig
kubectl get nodes
```

The other option is to login to your controller node and use the k0s in-built kubectl to access the cluster. Then you don't need to worry about kubeconfig (k0s takes care of that automatically).

```shell
ssh -i aws.pem <username>@<ip-address>
sudo k0s kubectl get nodes
```

### 6. Deploy Rook

To get started with Rook, let's first clone the Rook GitHub repo:

```shell
git clone --single-branch --branch release-1.7 https://github.com/rook/rook.git
cd rook/cluster/examples/kubernetes/ceph
```

We will use mostly the default Rook configuration. However, k0s kubelet directory must be configured in `operator.yaml` like this

```yaml
ROOK_CSI_KUBELET_DIR_PATH: "/var/lib/k0s/kubelet"
```

To create the resources, which are needed by the Rook’s Ceph operator, run

```shell
kubectl apply -f crds.yaml -f common.yaml -f operator.yaml
```

Now you should see the operator running. Check them with

```shell
kubectl get pods -n rook-ceph
```

### 7. Deploy Ceph Cluster

Then you can proceed to create a Ceph cluster. Ceph will use the three EBS volumes attached to the worker nodes:

```shell
kubectl apply -f cluster.yaml
```

It takes some minutes to prepare the volumes and create the cluster. Once this is completed you should see the following output:

```shell
kubectl get pods -n rook-ceph
```

```
/* Detailed source-code truncated for AI context efficiency. */
```

### 8. Configure Ceph block storage

Before Ceph can provide storage to your cluster, you need to create a `ReplicaPool` and a `StorageClass`. In this example, we use the default configuration to create the block storage.

```shell
kubectl apply -f ./csi/rbd/storageclass.yaml
```

### 9. Request storage

Create a new manifest file `mongo-pvc.yaml` with the following content:

```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mongo-pvc
spec:
  storageClassName: rook-ceph-block
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi
```

This will create Persistent Volume Claim (PVC) to request a 2 GB block storage from Ceph. Provisioning will be done dynamically. You can define the block size freely as long as it fits to the available storage size.

```shell
kubectl apply -f mongo-pvc.yaml
```

You can now check the status of your PVC:

```shell
kubectl get pvc
```

When the PVC gets the requested volume reserved (bound), it should look like this:

```shell
kubectl get pvc
```

```shell
NAME        STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      AGE
mongo-pvc   Bound    pvc-08337736-65dd-49d2-938c-8197a8871739   2Gi        RWO            rook-ceph-block   6s
```

### 10. Deploy an example application

Let's deploy a MongoDB database to verify the Ceph storage. Create a new file `mongo.yaml` with the following content:

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongo
spec:
  selector:
    matchLabels:
      app: mongo
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - image: mongo:4.0
        name: mongo
        ports:
        - containerPort: 27017
          name: mongo
        volumeMounts:
        - name: mongo-persistent-storage
          mountPath: /data/db
      volumes:
      - name: mongo-persistent-storage
        persistentVolumeClaim:
          claimName: mongo-pvc
```

Deploy the database:

```shell
kubectl apply -f mongo.yaml
```

### 11. Access the application

Open the MongoDB shell using the MongoDB pod:

```shell
kubectl get pods
```

```shell
NAME                    READY   STATUS    RESTARTS   AGE
mongo-b87cbd5cc-4wx8t   1/1     Running   0          76s
```

```shell
kubectl exec -it mongo-b87cbd5cc-4wx8t -- mongo
```

Create a DB and insert some data:

```mongodb
> use testDB
switched to db testDB
> db.testDB.insertOne( {name: "abc", number: 123  })
{
  "acknowledged" : true,
  "insertedId" : ObjectId("60815690a709d344f83b651d")
}
> db.testDB.insertOne( {name: "bcd", number: 234  })
{
  "acknowledged" : true,
  "insertedId" : ObjectId("6081569da709d344f83b651e")
}
```

Read the data:

```mongodb
> db.getCollection("testDB").find()
{ "_id" : ObjectId("60815690a709d344f83b651d"), "name" : "abc", "number" : 123 }
{ "_id" : ObjectId("6081569da709d344f83b651e"), "name" : "bcd", "number" : 234 }
>
```

You can also try to restart the MongoDB pod or restart the worker nodes to verity that the storage is persistent.

### 12. Clean-up

You can use Terraform to take down the virtual machines:

```shell
terraform destroy
```

Remember to delete the EBS volumes separately.

### Conclusions

You have now created a replicated Ceph storage for k0s. All your data is stored to multiple disks at the same time so you have a fault-tolerant solution. You also have enabled dynamic provisioning. Your applications can request the available storage without a manual creation of the persistent volumes each time.

This was just one example to deploy distributed storage to k0s cluster using an operator. You can easily use different Kubernetes storage solutions with k0s.

---

### Examples Traefik Ingress (`docs/examples/traefik-ingress.md`)

# Installing Traefik Ingress Controller

You can configure k0s with the [Traefik Ingress Controller], a [MetalLB service load balancer], and deploy the Traefik Dashboard using a service sample. To do this you leverage Helm's extensible bootstrapping functionality to add the correct extensions to the `k0s.yaml` file during cluster configuration.

[Traefik Ingress Controller]: https://doc.traefik.io/traefik/getting-started/quick-start-with-kubernetes/
[MetalLB service load balancer]: https://metallb.io/

## Install Traefik and MetalLB

Configure k0s to install Traefik and MetalLB during cluster bootstrapping by adding their [Helm charts](../helm-charts.md) as extensions in the k0s configuration file (`k0s.yaml`).

**Note:**
A good practice is to have a small range of IP addresses that are addressable on your network, preferably outside the assignment pool your DHCP server allocates (though any valid IP range should work locally on your machine). Providing an addressable range allows you to access your load balancer and Ingress services from anywhere on your local network.

```yaml
extensions:
  helm:
    repositories:
      - name: traefik
        url: https://traefik.github.io/charts

      - name: metallb
        url: https://metallb.github.io/metallb

    charts:
      - name: traefik
        chartname: traefik/traefik
        version: "39.0.1"
        namespace: traefik-system

      - name: metallb
        chart: metallb/metallb
        version: "0.15.3"
        namespace: metallb-system
```

## Configure traefik

If you want to configure Traefik via the Helm Chart, you can do this by providing values in the `extensions.helm.charts` section. For more infos check out the [Helm Chart docs](../helm-charts.md).

### Enable automatic http to https redirect

To automatically redirect http requests to https, set the following config via the `values` section:

```yaml
extensions:
  helm:
    repositories: ...

    charts:
      - name: traefik
        chartname: traefik/traefik
        version: "39.0.1"
        namespace: traefik-system
        values: |
          ports:
            web:
              http:
                redirections:
                  entryPoint:
                    to: websecure
                    scheme: https
                    permanent: true

      - name: metallb
        ...
```

### Add new entrypoints

Traefik can be configured to add new entrypoints which serve to accept traffic via a specified port. This can either be via the [http(s) load-balancer](https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/crd/http/ingressroute/), or the [tcp/udp load-balancer](https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/crd/tcp/ingressroutetcp/).
For detailled infos check out the [official docs](https://doc.traefik.io/traefik/reference/install-configuration/entrypoints/).

To add a new entrypoint which can be used by `IngressRoute` or `IngressRouteTCP/IngressRouteUDP` CRD, set the following config via the `values` section:

```yaml
extensions:
  helm:
    repositories: ...

    charts:
      - name: traefik
        chartname: traefik/traefik
        version: "39.0.1"
        namespace: traefik-system
        values: |
          ports:
            ssh: # name of the entrypoint
              port: 2222 # port inside the container
              expose:
                default: true
              exposedPort: 22 # port of the load-balancer which is accessed externally
              protocol: TCP # protocol

      - name: metallb
        ...
```

## Create ConfigMap for MetalLB

Next you need to create ConfigMap, which includes an IP address range for the load balancer. The pool of IPs must be dedicated to MetalLB's use. You can't reuse for example the Kubernetes node IPs or IPs controlled by other services.

> For detailed infos about the installation and configuration of MetalLB see the [Extension docs](metallb-loadbalancer.md)

Create a YAML file accordingly, and deploy it: `kubectl apply -f metallb-l2-pool.yaml`

```YAML
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool

metadata:
  name: first-pool
  namespace: metallb-system

spec:
  addresses:
  - <ip-address-range-start>-<ip-address-range-stop>
  - <ip-address>/<cidr>
  # example for a range
  - 192.168.0.1-192.168.0.5
  # example for a single address with cidr
  - 192.168.0.5/32
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement

metadata:
  name: example
  namespace: metallb-system
```

## Retrieve the Load Balancer IP

After you start your cluster, run `kubectl get all` to confirm the deployment of Traefik and MetalLB. The command should return a response with the `metallb` and `traefik` resources, along with a service load balancer that has an assigned `EXTERNAL-IP`.

```shell
kubectl get all
```

_Output_:

```shell
NAME                                                 READY   STATUS    RESTARTS   AGE
pod/metallb-1607085578-controller-864c9757f6-bpx6r   1/1     Running   0          81s
pod/metallb-1607085578-speaker-245c2                 1/1     Running   0          60s
pod/traefik-1607085579-77bbc57699-b2f2t              1/1     Running   0          81s

NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE
service/kubernetes           ClusterIP      10.96.0.1        <none>           443/TCP                      96s
service/traefik-1607085579   LoadBalancer   10.105.119.102   192.168.0.5      80:32153/TCP,443:30791/TCP   84s

NAME                                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/metallb-1607085578-speaker   1         1         1       1            1           kubernetes.io/os=linux   87s

NAME                                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/metallb-1607085578-controller   1/1     1            1           87s
deployment.apps/traefik-1607085579              1/1     1            1           84s

NAME                                                       DESIRED   CURRENT   READY   AGE
replicaset.apps/metallb-1607085578-controller-864c9757f6   1         1         1       81s
replicaset.apps/traefik-1607085579-77bbc57699              1         1         1       81s
```

Take note of the `EXTERNAL-IP` given to the `service/traefik-xxx` load balancer. In this example, `192.168.0.5` has been assigned and can be used to access services via the Ingress proxy:

```shell
NAME                         TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)                      AGE
service/traefik-1607085579   LoadBalancer   10.105.119.102   192.168.0.5      80:32153/TCP,443:30791/TCP   84s
```

Receiving a 404 response here is normal, as you've not configured any Ingress resources to respond yet:

```shell
curl http://192.168.0.5
```

```shell
404 page not found
```

## Deploy and access the Traefik Dashboard

With an available and addressable load balancer present on your cluster, now you can quickly deploy the Traefik dashboard and access it from anywhere on your LAN (assuming that MetalLB is configured with an addressable range).

1. Create the Traefik Dashboard [IngressRoute](https://doc.traefik.io/traefik/reference/routing-configuration/kubernetes/crd/http/ingressroute/) in a YAML file:

   ```yaml
   apiVersion: traefik.io/v1alpha1
   kind: IngressRoute

   metadata:
     name: traefik-dashboard
     namespace: traefik-system

   spec:
     entryPoints:
       - web
       - websecure
     routes:
       - match: PathPrefix(`/dashboard`) || PathPrefix(`/api`)
         kind: Rule
         services:
           - name: api@internal
             kind: TraefikService
   ```

2. Deploy the resource:

   ```shell
   kubectl apply -f traefik-dashboard.yaml
   ```

   _Output_:

   ```shell
   ingressroute.traefik.io/v1alpha1/traefik-dashboard created
   ```

   At this point you should be able to access the dashboard using the `EXTERNAL-IP` that you noted above by visiting `http://192.168.0.5/dashboard/` in your browser:

   

3. Create a simple `whoami` Deployment, Service, and [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) manifest:

   ```yaml
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: whoami-deployment
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: whoami
     template:
       metadata:
         labels:
           app: whoami
       spec:
         containers:
           - name: whoami-container
             image: containous/whoami
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: whoami-service
   spec:
     ports:
       - name: http
         targetPort: 80
         port: 80
     selector:
       app: whoami
   ---
   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: whoami-ingress
   spec:
     rules:
       - http:
           paths:
             - path: /whoami
               pathType: Exact
               backend:
                 service:
                   name: whoami-service
                   port:
                     number: 80
   ```

4. Apply the manifests:

   ```shell
   kubectl apply -f whoami.yaml
   ```

   _Output_:

   ```shell
   deployment.apps/whoami-deployment created
   service/whoami-service created
   ingress.networking.k8s.io/whoami-ingress created
   ```

5. Test the ingress and service:

   ```shell
   curl http://192.168.0.5/whoami
   ```

   _Output_:

   ```shell
   Hostname: whoami-deployment-85bfbd48f-7l77c
   IP: 127.0.0.1
   IP: ::1
   IP: 10.244.214.198
   IP: fe80::b049:f8ff:fe77:3e64
   RemoteAddr: 10.244.214.196:34858
   GET /whoami HTTP/1.1
   Host: 192.168.0.5
   User-Agent: curl/7.68.0
   Accept: */*
   Accept-Encoding: gzip
   X-Forwarded-For: 192.168.0.82
   X-Forwarded-Host: 192.168.0.5
   X-Forwarded-Port: 80
   X-Forwarded-Proto: http
   X-Forwarded-Server: traefik-1607085579-77bbc57699-b2f2t
   X-Real-Ip: 192.168.0.82
   ```

## Further details

With the Traefik Ingress Controller it is possible to use third party tools,
such as [ngrok], to go further and expose your load balancer to the world. In
doing this you enable dynamic certificate provisioning through [Let's Encrypt],
using either [cert-manager] or Traefik's own built-in [ACME provider].

[ngrok]: https://ngrok.com/
[Let's Encrypt]: https://letsencrypt.org/
[cert-manager]: https://cert-manager.io/docs/
[ACME provider]: https://doc.traefik.io/traefik/v2.0/user-guides/crd-acme/

---

### Internal Upgrading Calico (`docs/internal/upgrading-calico.md`)

# Upgrading Calico

K0s bundles Kubernetes manifests for Calico. The manifests are retrieved from
the [official Calico repo].

As fetching and modifying the entire multi-thousand line file is error-prone,
you may follow these steps to upgrade Calico to the latest version:

1. run `./hack/get-calico.sh <version>`
2. check the git diff to see if it looks sensible
3. re-apply our manual adjustments (documented below)
4. compile, pray, and test
5. commit and create a PR

[official Calico repo]: https://github.com/projectcalico/calico/blob/master/manifests/calico.yaml

## Manual Adjustments

All manual adjustments should be visible in the git diff. This section provides
a checklist to ensure that these changes are still applied correctly. The code
blocks in this section are k0s-specific modifications. Any such modification
should be accompanied by Go template comments to distinguish them from the
original code.

To see the diff **without** CRDs, you can do something like:

```sh
git diff ':!static/manifests/calico/CustomResourceDefinition'
```

That'll make it easier to spot any needed changes.

### Inlining the `calico-config` ConfigMap

Instead of using a ConfigMap to manage the Calico backend and MTU settings,
incorporate them directly into the places where they're used. This will trigger
rolling updates whenever something changes.

This includes the template for the node's CNI plugin configuration (the
`cni_network_config` key in the ConfigMap, the `CNI_NETWORK_CONFIG` environment
variable of the init container for the `calico-node` DaemonSet). Calico's CNI
network plugin installer executable ships with a default template, but k0s needs
to tweak this a bit for IPv6 support.

### Support for switching Calico backends

Search for `.Mode` to find:

```yaml
# Enable IPIP
- name: CALICO_IPV4POOL_IPIP
  value: "{{ if eq .Mode "bird" }}{{ .Overlay }}{{ else }}Never{{ end }}"
# Enable or Disable VXLAN on the default IP pool.
- name: CALICO_IPV4POOL_VXLAN
  value: "{{ if and (eq .Mode "vxlan") .EnableIPv4 }}{{ .Overlay }}{{ else }}Never{{ end }}"
# Enable or Disable VXLAN on the default IPv6 IP pool.
- name: CALICO_IPV6POOL_VXLAN
  value: "{{ if and (eq .Mode "vxlan") .EnableIPv6 }}{{ .Overlay }}{{ else }}Never{{ end }}"
{{- if eq .Mode "vxlan" }}
- name: FELIX_VXLANPORT
  value: "{{ .VxlanPort }}"
- name: FELIX_VXLANVNI
  value: "{{ .VxlanVNI }}"
{{- end }}
```

### iptables auto-detection

```yaml
# Auto detect the iptables backend
- name: FELIX_IPTABLESBACKEND
  value: "auto"
```

### Support for enabling WireGuard

```yaml
{{- if .EnableWireguard }}
- name: FELIX_WIREGUARDENABLED
  value: "true"
{{- end }}
```

### Support for changing the cluster CIDR

```yaml
- name: CALICO_IPV4POOL_CIDR
  value: "{{ .ClusterCIDR }}"
```

### Support for changing the MTU

Search for `.MTU` to find:

```yaml
# Set MTU for tunnel device used if ipip is enabled
- name: FELIX_IPINIPMTU
  value: "{{ .MTU }}"
# Set MTU for the VXLAN tunnel device.
- name: FELIX_VXLANMTU
  value: "{{ .MTU }}"
# Set MTU for the Wireguard tunnel device.
- name: FELIX_WIREGUARDMTU
  value: "{{ .MTU }}"
```

### Remove bgp support

Remove `bgp` from `CLUSTER_TYPE`:

```yaml
- name: CLUSTER_TYPE
  value: "k8s"
```

Disable BIRD checks on liveness and readiness probes when not running the BIRD
backend:

```yaml
{{- if eq .Mode "bird" }}
- -bird-live
{{- end }}
```

```yaml
{{- if eq .Mode "bird" }}
- -bird-ready
{{- end }}
```

### Remove eBPF support

Comment out the parts of the manifests related to eBPF support. They are usually
commented as such by upstream.

### Disable VXLAN offloading

See <https://github.com/k0sproject/k0s/issues/3024> for details.

```yaml
- name: FELIX_FEATUREDETECTOVERRIDE
  value: ChecksumOffloadBroken=true
```

### Template container image names

Instead of hardcoded image names and versions use placeholders to support
configuration level settings. Following placeholders are used:

- `CalicoCNIImage` for calico/cni
- `CalicoNodeImage` for calico/node
- `CalicoKubeControllersImage` for calico/kube-controllers

Also, all containers in manifests were modified to have the `imagePullPolicy`
field:

```yaml
imagePullPolicy: {{ .PullPolicy }}
```

---

### Troubleshooting FAQ (`docs/troubleshooting/FAQ.md`)

# Frequently asked questions

## How is k0s pronounced?

kay-zero-ess

## How do I run a single node cluster?

The cluster can be started with:

```shell
k0s controller --single
```

See also the [Getting Started](https://docs.k0sproject.io/stable/install/) tutorial.

## How do I connect to the cluster?

You find the config in `${DATADIR}/pki/admin.conf` (default: `/var/lib/k0s/pki/admin.conf`). Copy this file, and change the `localhost` entry to the public IP address of the controller. Use the modified config to connect with kubectl:

```shell
export KUBECONFIG=/path/to/admin.conf
kubectl ...
```

## Why doesn't `kubectl get nodes` list the k0s controllers?

As a default, the control plane does not run kubelet at all, and will not accept any workloads, so the controller will not show up on the node list in kubectl. If you want your controller to accept workloads and run pods, you do so with:
`k0s controller --enable-worker` (recommended only as test/dev/POC environments).

## Is k0s really open source?

Yes, k0s is 100% open source. The source code is licensed under the Apache 2.0
License, and the documentation under a Creative Commons License. The project is
part of the [CNCF Sandbox]. While Mirantis, Inc. remains a principal contributor
and sponsor, k0s adheres to CNCF's open governance and IP policies to ensure
transparency and community-driven development under a vendor-neutral umbrella.

[CNCF Sandbox]: https://www.cncf.io/sandbox-projects/

## A kubeconfig created via [`k0s kubeconfig`](../cli/k0s_kubeconfig.md) has been leaked, what can I do?

Kubernetes does not support certificate revocation (see [k/k/18982]). This means
that you cannot disable the leaked credentials. The only way to effectively
revoke them is to [replace the Kubernetes CA] for your cluster.

[k/k/18982]: https://github.com/kubernetes/kubernetes/issues/18982
[replace the Kubernetes CA]: certificate-authorities.md#replacing-the-kubernetes-ca-and-sa-key-pair

---

### Troubleshooting Certificate Authorities (`docs/troubleshooting/certificate-authorities.md`)

# Certificate Authorities (CAs)

## Overview of CAs managed by k0s

k0s maintains two Certificate Authorities and one public/private key pair:

* The **Kubernetes CA** is used to secure the Kubernetes cluster and manage
  client and server certificates for API communication.
* The **etcd CA** is used only when managed etcd is enabled, for securing etcd
  communications.
* The **Kubernetes Service Account (SA) key pair** is used for signing
  Kubernetes [service account tokens].

These CAs are automatically created during cluster initialization and have a
default expiration period of 10 years. They are distributed once to all k0s
controllers as part of k0s's [join process]. Replacing them is a manual process,
as k0s currently lacks automation for CA renewal.

[service account tokens]: https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/
[join process]: ../k0s-multi-node.md#5-add-controllers-to-the-cluster

## Replacing the Kubernetes CA and SA key pair

The following steps describe a way how to manually replace the Kubernetes CA and
SA key pair by taking a cluster down, regenerating those and redistributing them
to all nodes, and then bringing the cluster back online:

1. Take a [backup]! Things might go wrong at any level.

2. Stop k0s on all worker and controller nodes. All the instructions below
   assume that all k0s nodes are using the default data directory
   `/var/lib/k0s`. Adjust the path accordingly if a different data directory is
   used.

3. Delete the Kubernetes CA and SA key pair files from the all the controller
   data directories:

   * `/var/lib/k0s/pki/ca.crt`
   * `/var/lib/k0s/pki/ca.key`
   * `/var/lib/k0s/pki/sa.pub`
   * `/var/lib/k0s/pki/sa.key`

   Delete the kubelet's kubeconfig file and the kubelet's PKI directory from all
   worker data directories. Note that this includes controllers that have been
   started with the `--enable-worker` flag:

   * `/var/lib/k0s/kubelet.conf`
   * `/var/lib/k0s/kubelet/pki`

4. Choose one controller as the "first" one. Restart k0s on the first
   controller. If this controller is running with the `--enable-worker` flag,
   **reboot the machine** instead. This ensures that all processes and pods will
   be cleanly restarted. After the restart, k0s will have regenerated a new
   Kubernetes CA and SA key pair.

5. Distribute the new CA and SA key pair to the other controllers: Copy over the
   following files from the first controller to each of the remaining
   controllers:

   * `/var/lib/k0s/pki/ca.crt`
   * `/var/lib/k0s/pki/ca.key`
   * `/var/lib/k0s/pki/sa.pub`
   * `/var/lib/k0s/pki/sa.key`

   After copying the files, the new CA and SA key pair are in place. Restart k0s
   on the other controllers. For controllers running with the `--enable-worker`
   flag, **reboot the machines** instead.

6. Rejoin all workers. The easiest way to do this is to use a
   `kubelet-bootstrap.conf` file. You can [generate](../cli/k0s_token_create.md)
   such a file on a controller like this (see the section on [join tokens] for
   details):

   ```sh
   touch /tmp/rejoin-token &&
     chmod 0600 /tmp/rejoin-token &&
     k0s token create --expiry 1h |
     base64 -d |
     gunzip >/tmp/rejoin-token
   ```

   Copy that token to each worker node and place it at
   `/var/lib/k0s/kubelet-bootstrap.conf`. Then reboot the machine.

7. When all worker nodes are back online, the `kubelet-bootstrap.conf` files can
   be safely removed from the worker nodes. The token can be invalidated without
   waiting for its expiration: Use [`k0s token list --role
   worker`](../cli/k0s_token_list.md) to list all tokens and [`k0s token
   invalidate <token-id>`](../cli/k0s_token_invalidate.md) to invalidate them
   immediately.

[backup]: ../backup.md
[join tokens]: ../k0s-multi-node.md#about-join-tokens

## See also

* [Install using custom CAs](../custom-ca.md)

---

### Troubleshooting Logs (`docs/troubleshooting/logs.md`)

# Logs

k0s runs most of the Kubernetes and other system components as plain Linux child processes. As it acts as a watchdog for the child processes it also combines the logs for all such processes into its own log stream.

k0s adds a "selector" to each components log stream so it is easier to distinguish logs from various components. For example the log stream from kubelet is enhanced with a selector `component=kubelet`:

```text
Jul 08 08:46:25 worker-kppzr-lls2q k0s[1766]: time="2024-07-08 08:46:25" level=info msg="I0708 08:46:25.876821    1814 operation_generator.go:721] \"MountVolume.SetUp succeeded for volume \\\"kube-api-access-7tfxw\\\" (UniqueName: \\\"kubernetes.io/projected/ca514728-a1de-4408-9be5-8b36ee896752-kube-api-access-7tfxw\\\") pod \\\"node-shell-a16894ee-eb67-4865-8964-44ca5c87e18d\\\" (UID: \\\"ca514728-a1de-4408-9be5-8b36ee896752\\\") \" pod=\"kube-system/node-shell-a16894ee-eb67-4865-8964-44ca5c87e18d\"" component=kubelet stream=stderr
Jul 08 08:46:26 worker-kppzr-lls2q k0s[1766]: time="2024-07-08 08:46:26" level=info msg="I0708 08:46:26.112550    1814 kuberuntime_container_linux.go:167] \"No swap cgroup controller present\" swapBehavior=\"\" pod=\"kube-system/node-shell-a16894ee-eb67-4865-8964-44ca5c87e18d\" containerName=\"shell\"" component=kubelet stream=stderr
```

## Where are the logs?

### systemd based setups

systemd stores logs in its journal. Access k0s and all component logs using
`journalctl`. For example, to view kubelet logs, run `journalctl -u k0sworker |
grep component=kubelet`.

### OpenRC based setups

openRC stores service logs in `/var/log/k0sworker.log`. To view a specific component log, run `grep component=kubelet /var/log/k0s.log`.

---

### Troubleshooting Support Dump (`docs/troubleshooting/support-dump.md`)

# Support Insight

In many cases, especially when looking for [commercial support], there's a need
to share the cluster state with others. While one could always grant access to
the live cluster, this is not always desirable or possible. For these
situations, use the work provided by [Troubleshoot].

The Troubleshoot tool can produce a dump of the cluster state for sharing. The
[`sbctl`] tool can expose the dump tarball as a Kubernetes API.

The following example shows how this works with k0s.

[commercial support]: ../commercial-support.md
[Troubleshoot]: https://troubleshoot.sh
[`sbctl`]: https://github.com/replicatedhq/sbctl

## Setting up

To gather all the needed data, install another tool called [`support-bundle`].
Download it from the [Troubleshoot releases page]. Make sure to select the
executable for the correct architecture.

[`support-bundle`]: https://troubleshoot.sh/docs/support-bundle/introduction/
[Troubleshoot releases page]: https://github.com/replicatedhq/troubleshoot/releases

## Creating support bundle

A Support Bundle needs to know what to collect and optionally, what to analyze. This is defined in a YAML file.

While data collection can be customized to meet specific needs, the following
reference configuration for k0s covers core elements such as:

- collecting info on the host
- collecting system component statuses from `kube-system` namespace
- checking health of Kubernetes API, Etcd etc. components
- collecting k0s logs
- checking status of firewalls, anti-virus etc. services which are known to interfere with Kubernetes

Because host-level information is required, the commands must be run directly on
the k0s nodes.

After setting up the [tooling](#setting-up), run the following command to
generate a support bundle:

```shell
support-bundle --kubeconfig /var/lib/k0s/pki/admin.conf https://docs.k0sproject.io/stable/support-bundle-<role>.yaml
```

Above `<role>` refers to either `controller` or `worker`. Different roles
require different information. When running a controller with `--enable-worker`
or `--single`, which also makes it a worker, capture a combined dump:

```shell
support-bundle --kubeconfig /var/lib/k0s/pki/admin.conf https://docs.k0sproject.io/stable/support-bundle-controller.yaml https://docs.k0sproject.io/stable/support-bundle-worker.yaml
```

Once the data has been collected, a file named
`support-bundle-<timestamp>.tar.gz` will be created. Share this file as needed.

---

### Troubleshooting Troubleshooting (`docs/troubleshooting/troubleshooting.md`)

# Common Pitfalls

There are few common cases we've seen where k0s fails to run properly.

## CoreDNS in crash-loop

The most common case we've encountered so far has been CoreDNS getting into crash-loop on the node(s).

With kubectl you see something like this:

```console
$ kubectl get pod --all-namespaces
NAMESPACE     NAME                                       READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-5f6546844f-25px6   1/1     Running   0          167m
kube-system   calico-node-fwjx5                          1/1     Running   0          164m
kube-system   calico-node-t4tx5                          1/1     Running   0          164m
kube-system   calico-node-whwsg                          1/1     Running   0          164m
kube-system   coredns-5c98d7d4d8-tfs4q                   1/1     Error     17         167m
kube-system   konnectivity-agent-9jkfd                   1/1     Running   0          164m
kube-system   konnectivity-agent-bvhdb                   1/1     Running   0          164m
kube-system   konnectivity-agent-r6mzj                   1/1     Running   0          164m
kube-system   kube-proxy-kr2r9                           1/1     Running   0          164m
kube-system   kube-proxy-tbljr                           1/1     Running   0          164m
kube-system   kube-proxy-xbw7p                           1/1     Running   0          164m
kube-system   metrics-server-7d4bcb75dd-pqkrs            1/1     Running   0          167m
```

When you check the logs, it'll show something like this:

```shell
kubectl -n kube-system logs coredns-5c98d7d4d8-tfs4q
```

```shell
plugin/loop: Loop (127.0.0.1:55953 -> :1053) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 4547991504243258144.3688648895315093531."
```

CoreDNS forwards queries to the nameservers listed in the resolv.conf file that
the kubelet hands to pods. If that file points to a stub resolver on the host's
loopback interface, the queries come back to CoreDNS, which detects the loop.

K0s handles systemd-resolved's stub automatically: if `/etc/resolv.conf` points
to it, the kubelet uses `/run/systemd/resolve/resolv.conf` for pods instead.
Don't disable the stub or empty `/etc/resolv.conf` to work around the loop. K0s
itself relies on `/etc/resolv.conf` for name resolution, see the section about
[NSS integration]. If another resolver on the loopback interface causes the
loop, point the kubelet at a resolv.conf file that lists the upstream
nameservers via the `resolvConf` field of the [kubelet configuration].

Read more in the [CoreDNS loop troubleshooting] and the [Kubernetes DNS
debugging] docs.

[NSS integration]: ../external-runtime-deps.md#no-integration-with-name-service-switch-nss-apis
[kubelet configuration]: ../worker-node-config.md#kubelet-configuration
[CoreDNS loop troubleshooting]: https://coredns.io/plugins/loop/#troubleshooting-loops-in-kubernetes-clusters
[Kubernetes DNS debugging]: https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues

## `k0s controller` fails on ARM boxes

In the logs you probably see etcd not starting up properly.

Etcd is [not fully supported][etcd-platforms] on ARM architecture, thus you need
to run `k0s controller` and thus also etcd process with the environment variable
`ETCD_UNSUPPORTED_ARCH=arm`.

As etcd is not fully supported on ARM, it also means that the k0s control plane
with etcd itself is not fully supported on ARM either.

[etcd-platforms]: https://etcd.io/docs/v3.5/op-guide/supported-platform/#current-support

## `k0s` will not start on ZFS-based systems

On ZFS-based systems k0s will fail to start because containerd runs by default in overlayfs mode to manage image layers. This is not compatible with ZFS and requires a custom config of containerd. The following steps should get k0s working on ZFS-based systems:

- check with `$ ctr -a /run/k0s/containerd.sock plugins ls` that the containerd ZFS snapshotter plugin is in `ok` state (should be the case if ZFS kernel modules and ZFS userspace utils are correctly configured):

```console
TYPE                            ID                       PLATFORMS      STATUS    
...
io.containerd.snapshotter.v1    zfs                      linux/amd64    ok
...
```

- create a containerd config according to the [documentation](../runtime.md): `$ containerd config default > /etc/k0s/containerd.toml`
- modify the line in `/etc/k0s/containerd.toml`:

```toml
...
    [plugins.'io.containerd.cri.v1.images']
      snapshotter = "overlayfs"
...
```

to

```toml
...
    [plugins.'io.containerd.cri.v1.images']
      snapshotter = "zfs"
...
```

- create a ZFS dataset to be used as snapshot storage at your desired location, e.g. `$ zfs create -o mountpoint=/var/lib/k0s/containerd/io.containerd.snapshotter.v1.zfs rpool/containerd`
- install k0s as usual, e.g `$ k0s install controller --single -c /etc/k0s/k0s.yaml`
- containerd should be launched with ZFS support and k0s should initialize the cluster correctly

## Pods pending when using cloud providers

Once we enable [cloud provider support](../cloud-providers.md) on kubelet on worker nodes, kubelet will automatically add a taint `node.cloudprovider.kubernetes.io/uninitialized` for the node. This taint will prevent normal workloads to be scheduled on the node until the cloud provider controller actually runs second initialization on the node and removes the taint. This means that these nodes are not available for scheduling until the cloud provider controller is actually successfully running on the cluster.

For troubleshooting your specific cloud provider see its documentation.

## k0s not working with read only `/usr`

By default k0s does not run on nodes where `/usr` is read only.

This can be fixed by changing the default path for `volumePluginDir` in your k0s config. You will need to change to values, one for the kubelet itself, and one for Calico.

Here is a snippet of an example config with the default values changed:

```yaml
spec:
  controllerManager:
    extraArgs:
      flex-volume-plugin-dir: "/etc/kubernetes/kubelet-plugins/volume/exec"
  network:
    calico:
      flexVolumeDriverPath: /etc/k0s/kubelet-plugins/volume/exec/nodeagent~uds
  workerProfiles:
    - name: coreos
      values:
        volumePluginDir: /etc/k0s/kubelet-plugins/volume/exec/
```

With this config you can start your controller as usual. Any workers will need to be started with

```shell
k0s worker --profile coreos [TOKEN]
```

## iptables version mismatch

If the running kernel has firewall rules created using an iptables version
newer than the version shipped with k0s, the Kubernetes network will [not
work](https://www.mirantis.com/blog/networking-problems-after-installing-kubernetes-1-25-or-after-upgrading-your-host-os-this-might-be-your-problem).

To solve this, make sure that the iptables version used on host is same version
as bundled with k0s. The k0s version is found in `/var/lib/k0s/bin`.

## Profiling

We drop any debug related information and symbols from the compiled binary by utilizing `-w -s` linker flags.

To keep those symbols use `DEBUG` environment variable:

```shell
DEBUG=true make k0s
```

Any value not equal to the "false" would work.

To add custom linker flags use `LDFLAGS` variable.

```shell
LD_FLAGS="--custom-flag=value" make k0s
```

## Using a custom container runtime and missing labels in Prometheus metrics

With Kubernetes' shift to CRI, Kubelet's method of obtaining container metrics
through its embedded [cAdvisor] no longer works as it used to. This process
doesn't go via CRI but directly interacts with the container runtime, which is
only natively supported for containerd. K0s automatically manages this for its
built-in containerd runtime. For custom containerd runtimes, you can use the
flag `--kubelet-extra-flags=--containerd=/path/to/containerd.sock` when starting
k0s.

However, this solution is not applicable to other runtimes such as Docker when
used via the cri-dockerd shim, and there is currently no easy fix. Kubelet is
expected to be refactored in the future to gather container metrics from CRI
instead, as specified in [KEP-2371]. For now, running a standalone cAdvisor, as
explained in Kubernetes' [known issues][dockershim-known-issues] section, is a
recommended workaround.

[cAdvisor]: https://github.com/google/cadvisor
[KEP-2371]: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2371-cri-pod-container-stats/README.md
[dockershim-known-issues]: https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/check-if-dockershim-removal-affects-you/#some-filesystem-metrics-are-missing-and-the-metrics-format-is-different

## Customized configurations

- All data directories reside under `/var/lib/k0s`, for example:
  - `/var/lib/k0s/kubelet`
  - `/var/lib/k0s/etcd`

---

### Examples Oidc Oidc Cluster Configuration (`docs/examples/oidc/oidc-cluster-configuration.md`)

# OpenID Connect integration

Developers use `kubectl` to access Kubernetes clusters. By default `kubectl` uses a certificate to authenticate to the Kubernetes API. This means that when multiple developers need to access a cluster, the certificate needs to be shared. Sharing the credentials to access a Kubernetes cluster presents a significant security problem. Compromise of the certificate is very easy and the consequences can be catastrophic.

In this tutorial, we walk through how to set up your Kubernetes cluster to add Single Sign-On support for kubectl using OpenID Connect (OIDC).

## OpenID Connect based authentication

OpenID Connect can be enabled by modifying k0s configuration (using extraArgs).

### Configuring k0s: overview

There are list of arguments for the kube-api that allows us to manage OIDC based authentication

| Parameter | Description | Example | Required |
| --------- | ----------- | ------- | ------- |
| `--oidc-issuer-url` | URL of the provider which allows the API server to discover public signing keys. Only URLs which use the `https://` scheme are accepted.  This is typically the provider's discovery URL without a path, for example "https://accounts.google.com" or "https://login.salesforce.com".  This URL should point to the level below .well-known/openid-configuration | If the discovery URL is `https://accounts.google.com/.well-known/openid-configuration`, the value should be `https://accounts.google.com` | Yes |
| `--oidc-client-id` | A client id that all tokens must be issued for. | kubernetes | Yes |
| `--oidc-username-claim` | JWT claim to use as the user name. By default `sub`, which is expected to be a unique identifier of the end user. Admins can choose other claims, such as `email` or `name`, depending on their provider. However, claims other than `email` will be prefixed with the issuer URL to prevent naming clashes with other plugins. | sub | No |
| `--oidc-username-prefix` | Prefix prepended to username claims to prevent clashes with existing names (such as `system:` users). For example, the value `oidc:` will create usernames like `oidc:jane.doe`. If this flag isn't provided and `--oidc-username-claim` is a value other than `email` the prefix defaults to `( Issuer URL )#` where `( Issuer URL )` is the value of `--oidc-issuer-url`. The value `-` can be used to disable all prefixing. | `oidc:` | No |
| `--oidc-groups-claim` | JWT claim to use as the user's group. If the claim is present it must be an array of strings. | groups | No |
| `--oidc-groups-prefix` | Prefix prepended to group claims to prevent clashes with existing names (such as `system:` groups). For example, the value `oidc:` will create group names like `oidc:engineering` and `oidc:infra`. | `oidc:` | No |
| `--oidc-required-claim` | A key=value pair that describes a required claim in the ID Token. If set, the claim is verified to be present in the ID Token with a matching value. Repeat this flag to specify multiple claims. | `claim=value` | No |
| `--oidc-ca-file` | The path to the certificate for the CA that signed your identity provider's web certificate.  Defaults to the host's root CAs. | `/etc/kubernetes/ssl/kc-ca.pem` | No |

To set up bare minimum example we need to use:

- oidc-issuer-url
- oidc-client-id
- oidc-username-claim

### Configuring k0s: prerequisites

You will require:

- issuer-url
- client-id
- username-claim

Please, refer to [providers configuration guide](./oidc-provider-configuration.md) or your selected OIDC provider's own documentation (we don't cover all of them in k0s docs).

### Configuration example

```yaml
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  api:
    extraArgs:
      oidc-issuer-url: <issuer-url>
      oidc-client-id: <client-id>
      oidc-username-claim: email # we use email token claim field as a username
```

Use the configuration as a starting point. Continue with [configuration guide](../../configuration.md) for finishing k0s cluster installation.

## OpenID Connect based authorisation

There are two alternative options to implement authorization

### Provider based role mapping

Please refer to the [providers configuration guide](./oidc-provider-configuration.md). Generally speaking, using the `oidc-groups-claim` argument let's you specify which token claim is used a list of RBAC roles for a given user. You still need somehow sync up that data between your OIDC provider and kube-api RBAC system.

### Manual roles management

To use manual role management for each user you will need to create a role and role-binding for each new user within k0s cluster.
The role can be shared for all the users.
Role example:

```yaml
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: default
  name: dev-role
rules:
- apiGroups: ["*"]
  resources: ["*"]
  verbs: ["*"]
```

RoleBinding example:

```yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: dev-role-binding
subjects:
  - kind: User
    name: <provider side user id>
roleRef:
  kind: Role
  name: dev-role
  apiGroup: rbac.authorization.k8s.io
```

The provided Role example is an all-inclusive and comprehensive example and should be tuned up to your actual requirements.

### kubeconfig management

NB: it's not safe to provide full content of the `/var/lib/k0s/pki/admin.conf` to the end-user. Instead, create a user specific kubeconfig with limited permissions.

The authorization side of the kubeconfig management is described in provider specific guides. Use `/var/lib/k0s/pki/admin.conf` as a template for cluster specific kubeconfig.

## References

[OAuth2 spec](https://oauth.net/2/)
[Kubernetes authorization system (RBAC)](https://kubernetes.io/docs/reference/access-authn-authz/rbac/)
[Kubernetes authenticating system](https://kubernetes.io/docs/reference/access-authn-authz/authentication/)

---

### Examples Oidc Oidc Provider Configuration (`docs/examples/oidc/oidc-provider-configuration.md`)

# Providers

We use Google Cloud as a provider for the sake of the example. Check your vendor documentation in case if you use some other vendor.

## Notes on stand-alone providers

If you are using stand-alone OIDC provider, you might need to specify `oidc-ca-file` argument for the kube-api.

## Google Cloud

We use [k8s-oidc-helper](https://github.com/micahhausler/k8s-oidc-helper) tool to create proper kubeconfig user record.

The issuer URL for the Google cloud is `https://accounts.google.com`

### Creating an application

- Go to the [Google Cloud Dashboard](https://console.developers.google.com/apis/dashboard)
- Create a new project in your organization
- Go to the "Credentials" page
- Create "OAuth consent screen"

### Creating a user credentials

- Go to the [Google Cloud Dashboard](https://console.developers.google.com/apis/dashboard)
- Go to the "Credentials" page
- Create new credentials. Select "OAuth client ID" as a type.
- Select "Desktop" app as an application type.
- Save client ID and client secret

### Creating kubeconfig user record

Use the command and follow the instructions:

```bash
k8s-oidc-helper --client-id=<CLIENT_ID> \
  --client-secret=<CLIENT_SECRET> \
  --write=true
```

## Using kubelogin

For other OIDC providers it is possible to use `kubelogin` plugin.
Please refer to the [setup guide](https://github.com/int128/kubelogin/blob/master/docs/setup.md) for details.

### Google Cloud example using `kubelogin`

```bash
kubectl oidc-login setup \
  --oidc-issuer-url=https://accounts.google.com \
  --oidc-client-id=<CLIENT_ID> \
  --oidc-client-secret=<CLIENT_SECRET>

  kubectl config set-credentials oidc \
  --exec-api-version=client.authentication.k8s.io/v1beta1 \
  --exec-command=kubectl \
  --exec-arg=oidc-login \
  --exec-arg=get-token \
  --exec-arg=--oidc-issuer-url=https://accounts.google.com \
  --exec-arg=--oidc-client-id=<CLIENT_ID>  \
  --exec-arg=--oidc-client-secret=<CLIENT_SECRET>
```

You can switch the current context to oidc.

```kubectl config set-context --current --user=oidc```

---

### Governance Cncf Ai Conformance (`docs/governance/cncf/ai-conformance.md`)

# Kubernetes AI conformance

This document summarizes how k0s satisfies the Kubernetes AI Conformance requirements for Kubernetes v1.35.
It is a reviewer-facing evidence page built from captured test runs on k0s v1.35.3+k0s.0.

## Test environment

The evidence below was captured on a two-node k0s cluster running on Azure:

- k0s v1.35.3+k0s.0
- Kubernetes v1.35.3
- One controller node and one NVIDIA T4 worker node
- GPU Operator v26.3.1 for NVIDIA runtime, device plugin, and DCGM Exporter

## DRA support

**Requirement:** Dynamic Resource Allocation APIs must be available and usable for accelerator allocation.

**Status on k0s:** Implemented.

k0s v1.35.3+k0s.0 ships Kubernetes v1.35.3, where DRA is available at `resource.k8s.io/v1` and enabled by default.
We validated the full DRA flow with the upstream `dra-example-driver`: a `ResourceClaimTemplate` was created, the scheduler allocated a simulated GPU device, and the consuming Pod received the allocated device metadata inside the container.
This shows not only that the API surface exists, but that the allocation path is functional end to end on k0s.

Minimal example:

```console
$ kubectl api-resources --api-group=resource.k8s.io
NAME                     APIVERSION           KIND
deviceclasses            resource.k8s.io/v1   DeviceClass
resourceclaims           resource.k8s.io/v1   ResourceClaim
resourceclaimtemplates   resource.k8s.io/v1   ResourceClaimTemplate
resourceslices           resource.k8s.io/v1   ResourceSlice

$ kubectl -n dra-demo get resourceclaim
NAME                     STATE
dra-demo-pod-gpu-nd7zh   allocated,reserved

$ kubectl logs -n dra-demo dra-demo-pod | grep GPU_DEVICE_0
GPU_DEVICE_0="gpu-0"
```

Further reading:

- [Dynamic Resource Allocation in Kubernetes](https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/)
- [kubernetes-sigs/dra-example-driver](https://github.com/kubernetes-sigs/dra-example-driver)

## Driver runtime management

**Requirement:** The platform should support installation and management of accelerator drivers and runtime components.

**Status on k0s:** Implemented.

k0s supports NVIDIA driver and runtime management through the NVIDIA GPU Operator.
On the test cluster, the operator installed the NVIDIA driver, configured the NVIDIA runtime for containerd, and enabled GPU access for workloads that opt into the `nvidia` `RuntimeClass`.
This is the standard upstream integration path for NVIDIA-backed Kubernetes clusters and works on k0s with containerd runtime configuration.

Minimal example:

```console
$ kubectl -n gpu-operator exec ds/nvidia-driver-daemonset -- \
    nvidia-smi --query-gpu=driver_version --format=csv,noheader
580.126.20

$ kubectl logs cuda-smoketest
NVIDIA-SMI 580.126.20             Driver Version: 580.126.20

$ kubectl get runtimeclass
NAME           HANDLER
nvidia         nvidia
nvidia-cdi     nvidia-cdi
```

Further reading:

- [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/)
- [Using `nvidia-container-runtime` with k0s](https://docs.k0sproject.io/stable/runtime/#using-nvidia-container-runtime)

## GPU sharing

**Requirement:** The platform should support mechanisms to share a single physical accelerator among multiple workloads.

**Status on k0s:** Implemented.

k0s supports GPU sharing through NVIDIA device plugin time-slicing.
On the test cluster, a single Tesla T4 was reconfigured to advertise four schedulable `nvidia.com/gpu` replicas, and four GPU-requesting Pods ran concurrently on the same physical GPU.
T4 hardware does not support MIG, so time-slicing is the applicable sharing mechanism on this setup.
This demonstrates a real sharing mode on hardware commonly used for inference and smaller training workloads.

Minimal example:

```console
$ kubectl get node k0s-gpu-0 -o jsonpath='{.status.capacity.nvidia\.com/gpu}{"\n"}'
4

$ kubectl get pods -l app=shared-gpu -o wide
NAME           READY   STATUS    NODE
shared-gpu-1   1/1     Running   k0s-gpu-0
shared-gpu-2   1/1     Running   k0s-gpu-0
shared-gpu-3   1/1     Running   k0s-gpu-0
shared-gpu-4   1/1     Running   k0s-gpu-0

$ kubectl get node k0s-gpu-0 -o jsonpath='{.metadata.labels.nvidia\.com/gpu\.sharing-strategy}{"\n"}'
time-slicing
```

Further reading:

- [NVIDIA k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin)
- [NVIDIA GPU Operator time-slicing documentation](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html)

## Virtualized accelerator

**Requirement:** The platform should support virtualized accelerators where available.

**Status on k0s:** Not Implemented.

k0s does not ship a packaged vGPU integration.
Virtualized GPU support depends on NVIDIA vGPU licensing and hardware-specific enablement outside the scope of the k0s distribution.
Users can integrate upstream NVIDIA vGPU components on suitable infrastructure, but that path was not demonstrated in this submission.

Further reading:

- [NVIDIA vGPU documentation](https://docs.nvidia.com/grid/)
- [NVIDIA GPU Operator](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/)

## AI inference

**Requirement:** A working Kubernetes Gateway API implementation must support inference traffic management.

**Status on k0s:** Implemented.

k0s supports Gateway API resources and standard Gateway implementations.
We installed Gateway API v1.5.1 CRDs with Envoy Gateway v1.7.2, then configured a `GatewayClass`, `Gateway`, and `HTTPRoute` to route `/predict` traffic to a mock inference backend.
Requests matching the configured prefix reached the backend, while non-matching requests were rejected by Envoy.
This provides concrete evidence that k0s can host an inference ingress stack with route attachment and path-based traffic management.

Minimal example:

```console
$ kubectl get gatewayclass eg
NAME   CONTROLLER                                      ACCEPTED
eg     gateway.envoyproxy.io/gatewayclass-controller   True

$ curl -si http://localhost:8888/predict
HTTP/1.1 200 OK
x-app-name: http-echo
{"prediction":0.87}

$ curl -si http://localhost:8888/
HTTP/1.1 404 Not Found
```

Further reading:

- [Gateway API](https://gateway-api.sigs.k8s.io/)
- [Envoy Gateway](https://gateway.envoyproxy.io/)

## Gang scheduling

**Requirement:** The platform must support all-or-nothing scheduling for distributed workloads.

**Status on k0s:** Implemented.

k0s supports gang scheduling with Volcano.
We installed Volcano v1.14.1 and demonstrated both successful atomic placement and full refusal of an oversized gang: a two-task job was bound together, while a five-task job that could not fit was kept pending without partial placement.
That combination matters because it shows both sides of gang semantics on k0s: atomic success when capacity exists and no partial placement when it does not.

Minimal example:

```console
$ kubectl get vcjob gang-demo
NAME        STATUS      MINAVAILABLE
gang-demo   Completed   2

$ kubectl get vcjob gang-overflow
NAME            STATUS    MINAVAILABLE
gang-overflow   Pending   5

$ kubectl get pods -l volcano.sh/job-name=gang-overflow
No resources found in default namespace.
```

Further reading:

- [Volcano](https://volcano.sh/en/)

## Cluster autoscaling

**Requirement:** The platform must support scaling accelerator-capable node groups through Kubernetes cluster autoscaling.

**Status on k0s:** Implemented.

k0s is a Kubernetes distribution and does not bundle its own autoscaler.
Instead, k0s works with the upstream Kubernetes Cluster Autoscaler and the infrastructure-specific provisioning layer behind the cluster, such as Azure VMSS, AWS Auto Scaling Groups, GCP Managed Instance Groups, or Cluster API.
Accelerator-aware scale-up is driven by pending Pods that request `nvidia.com/gpu` and by the GPU resources advertised through the NVIDIA device plugin.
This is the same model used by other conformant distributions: k0s provides the standard Kubernetes substrate, while autoscaling is supplied by the provisioning layer chosen for the cluster.

Reference configuration:

```shell
helm repo add autoscaler https://kubernetes.github.io/autoscaler
helm install cluster-autoscaler autoscaler/cluster-autoscaler \
  --namespace kube-system \
  --set cloudProvider=azure
```

Further reading:

- [Kubernetes Cluster Autoscaler](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler)
- [Cluster Autoscaler GPU support](https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler#gpu-support)

## Pod autoscaling

**Requirement:** Horizontal Pod Autoscaler must function correctly for Pods that use accelerators.

**Status on k0s:** Implemented.

k0s bundles `metrics-server`, so HPA works out of the box on a default cluster.
We deployed a GPU-requesting `Deployment` and an `autoscaling/v2` HPA targeting CPU utilization.
Under load, HPA increased the Deployment replica count from one to two; the second Pod remained pending because the scheduler correctly enforced single-GPU node capacity.
This is the expected and correct interaction: autoscaling decisions are made from workload metrics, while accelerator capacity is still enforced independently by the scheduler.

Minimal example:

```console
$ kubectl get hpa gpu-burner
NAME         REFERENCE               TARGETS         MINPODS   MAXPODS   REPLICAS
gpu-burner   Deployment/gpu-burner   cpu: 497%/50%   1         2         2

$ kubectl get pods -l app=gpu-burner
NAME                          READY   STATUS
gpu-burner-759c8596bf-7gzvr   1/1     Running
gpu-burner-759c8596bf-k8d4x   0/1     Pending
```

Further reading:

- [Horizontal Pod Autoscaling](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/)
- [metrics-server](https://github.com/kubernetes-sigs/metrics-server)

## Accelerator metrics

**Requirement:** The platform must expose fine-grained accelerator metrics through a standard, machine-readable endpoint.

**Status on k0s:** Implemented.

k0s supports accelerator metrics through NVIDIA DCGM Exporter, deployed by the GPU Operator.
We integrated DCGM Exporter with `kube-prometheus-stack` using a `ServiceMonitor` and verified that Prometheus scraped per-GPU metrics with labels such as UUID, model, PCI bus ID, node name, and driver version.
The important part here is not just that metrics exist, but that they are queryable through the standard Prometheus API with enough labels to identify a specific GPU on a specific node.

Minimal example:

```console
$ curl -s 'http://localhost:9090/api/v1/query?query=DCGM_FI_DEV_GPU_UTIL'
"__name__":"DCGM_FI_DEV_GPU_UTIL"
"UUID":"GPU-df2bfa0a-183e-d378-e4ab-1042a1736a51"
"modelName":"Tesla T4"
"Hostname":"k0s-gpu-0"
"pci_bus_id":"00000001:00:00.0"
```

Further reading:

- [NVIDIA DCGM Exporter](https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html)
- [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)

## AI service metrics

**Requirement:** The platform must discover and collect workload metrics in a standard format.

**Status on k0s:** Implemented.

k0s supports workload metrics collection through Prometheus Operator compatible monitoring stacks.
We installed `kube-prometheus-stack`, created a `ServiceMonitor` for a sample inference-shaped workload, and verified that Prometheus discovered the target and stored its metrics with workload-identifying labels.
This demonstrates the standard collection path most AI services use in practice: workload metrics exposed on `/metrics`, discovered by `ServiceMonitor`, and stored in Prometheus.

Minimal example:

```console
$ curl -s 'http://localhost:9090/api/v1/query' \
    --data-urlencode 'query=up{job="sample-inference"}'
"job":"sample-inference"
"namespace":"default"
"service":"sample-inference"
"value":[..., "1"]

$ curl -s 'http://localhost:9090/api/v1/query' \
    --data-urlencode 'query=node_cpu_seconds_total{job="sample-inference"}'
"__name__":"node_cpu_seconds_total"
```

Further reading:

- [Prometheus Operator `ServiceMonitor`](https://prometheus-operator.dev/docs/developer/getting-started/#using-servicemonitors)
- [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)

## Secure accelerator access

**Requirement:** Accelerator access must be isolated so that only authorized workloads can use the device.

**Status on k0s:** Implemented.

k0s relies on the standard Kubernetes device plugin model for GPU isolation.
On the test cluster, a Pod without a GPU request and without the `nvidia` `RuntimeClass` had no access to NVIDIA tooling or device nodes.
When two Pods requested one GPU each on a single-GPU node, the scheduler admitted one and kept the second pending with `Insufficient nvidia.com/gpu`.
Together, these checks show both isolation-by-default and capacity enforcement for accelerator access on k0s.

Minimal example:

```console
$ kubectl logs no-gpu-request
sh: 1: nvidia-smi: not found
NO_GPU_ACCESS

$ kubectl get pods gpu-hog-1 gpu-hog-2
NAME        READY   STATUS
gpu-hog-1   1/1     Running
gpu-hog-2   0/1     Pending
```

Further reading:

- [Kubernetes device plugins](https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/)
- [NVIDIA k8s-device-plugin](https://github.com/NVIDIA/k8s-device-plugin)

## Robust controller

**Requirement:** The platform must be able to run at least one complex AI operator with CRDs and reliable reconciliation behavior.

**Status on k0s:** Implemented.

k0s supports complex AI operators such as KubeRay.
We installed KubeRay v1.6.1, reconciled a `RayCluster` into head and worker Pods plus supporting Services, ran a distributed Ray task successfully, and then deleted the head Pod to verify that the operator recreated it and restored the cluster to a ready state.
This goes beyond a basic install check: it shows CRD registration, reconciliation of child resources, a successful workload, and recovery after disruption.

Minimal example:

```console
$ kubectl get raycluster raycluster-demo -o wide
NAME              DESIRED WORKERS   AVAILABLE WORKERS   STATUS
raycluster-demo   1                 1                   ready

$ kubectl get events --field-selector involvedObject.kind=RayCluster \
    --sort-by=.lastTimestamp
CreatedHeadPod
CreatedWorkerPod
DeletedHeadPod
CreatedHeadPod

$ kubectl exec -i "$HEAD_POD" -- python -
Result: [0, 1, 4, 9, 16]
Cluster resources: {'CPU': 2.0, ...}
```

Further reading:

- [KubeRay](https://docs.ray.io/en/latest/cluster/kubernetes/)

---

### Governance Cncf Gtr (`docs/governance/cncf/gtr.md`)

# General Technical Review (GTR)

This document collects information for the [CNCF General Technical Review]. It is built to support the [k0s CNCF Sandbox application] but will be a living document that we update regularly.

[CNCF General Technical Review]: https://github.com/cncf/toc/blob/main/toc_subprojects/project-reviews-subproject/general-technical-questions.md
[k0s CNCF Sandbox application]: https://github.com/cncf/sandbox/issues/125

As we are currently applying for the sandbox level, the document only covers the GTR's “Day 0” phase.

## Scope

### Roadmap process

The k0s roadmap process is a collaborative effort to align with user needs, community feedback, and emerging trends. We prioritize initiatives that simplify Kubernetes operations and extend its capabilities for far-edge, IoT, and disconnected environments.
The roadmap is a living document, refined iteratively based on technical feasibility, testing, and industry shifts. By emphasizing usability, automation, and feedback-driven iterations, we ensure that k0s continues to lead in lightweight, scalable Kubernetes solutions.

### Target personas

#### DevOps Engineers

* Focused on managing Kubernetes clusters efficiently.
* Seek simplicity in deployment, maintenance, and scaling.
* Prioritize automation and minimize operational overhead.

#### Platform Engineers

* Responsible for building and maintaining internal platforms on top of Kubernetes.
* Interested in integrating k0s into CI/CD pipelines, monitoring systems, and other tooling.

#### IoT and Edge Architects

* Working on deploying Kubernetes at the far edge.
* Require lightweight, resource-efficient solutions for disconnected or resource-constrained environments.

#### Cloud-Native Developers

* Building and deploying applications on Kubernetes.
* Need a developer-friendly experience with minimal setup and maximum portability.

#### System Integrators

* Designing bespoke solutions that leverage Kubernetes in varied environments, including on-premises and hybrid setups.
* Value flexibility and compatibility with existing tools and standards.

#### IT Operations Teams

* Managing infrastructure at scale across data centers, cloud, and edge locations.
* Seek reliability, scalability, and tools to simplify multi-cluster operations.

### Primary use cases

**k0s** is a streamlined, self-contained Kubernetes distribution designed to serve as a portable and lightweight foundational core for Kubernetes environments. It simplifies deployment and management, while maintaining the flexibility to build out additional functionality.
As such, it's tailor-made for a variety of use cases, including:

#### Edge Computing and IoT

Deploying Kubernetes in resource-constrained, far-edge environments where lightweight and efficient solutions are critical.
Ideal for managing IoT devices, disconnected edge nodes, and remote locations.
k0s also support air gapped environments by being able to automatically load container image bundles and having zero dependencies.

#### Cloud-Native Application Hosting

Enabling developers to run containerized applications with minimal operational complexity.
Suitable for both small-scale setups and large-scale production workloads.

#### Dev/Test Environment

Providing a quick, easy-to-set-up Kubernetes environment for development, testing, and CI/CD pipelines.
Allows developers to spin up Kubernetes clusters locally or in isolated environments without significant overhead.

#### Centralized Control for Distributed Clusters

Offering the ability to centrally manage control planes while operating clusters in remote or disconnected locations.
Ensures operational simplicity across distributed systems.

#### Heterogeneous environments

Running Kubernetes efficiently on bare-metal servers or in hybrid cloud/on-premises environments.
Tailored for teams looking to minimize dependency on external cloud services.
Focus on lightweight deployment without the need for heavy dependencies on external cloud services.

#### Isolated control planes

k0s uses [konnectivity] by default and is therefore a good fit for environments where control planes and worker nodes are isolated.
This architecture is good fit for e.g. Edge type use cases or where users want to cenrtalize their control planes in general.

[konnectivity]: https://github.com/kubernetes-sigs/apiserver-network-proxy

## Unsupported use cases

While k0s provides a self-contained core for Kubernetes operations, it does not aim to be a feature-complete "all-in-one" solution. In particular, the following are out of scope:

### Bundling of higher-level ecosystem components

k0s does not include additional tools such as ingress controllers, service meshes, or advanced observability capabilities by default. Cluster administrators are expected to install and configure these components separately using the extension points provided by k0s, according to their unique requirements.

### Being a "Turnkey Solution"

k0s avoids bundling excessive features to remain lightweight and customizable. It is best suited for those who value flexibility over out-of-the-box completeness.

## Intended types of organizations

The intended types of organizations for using **k0s** are those that prioritize simplicity, flexibility, and efficiency in Kubernetes deployments. Some examples:

### Small and Medium-Sized Businesses (SMBs)

* Looking for a straightforward, cost-effective Kubernetes solution.
* Often lack dedicated Kubernetes experts and benefit from k0s's ease of deployment and management.

### Edge Computing and IoT Organizations

* Deploying Kubernetes at the far edge to manage IoT devices, industrial equipment, or remote sites.
* Require lightweight, resource-efficient solutions that can operate in disconnected or low-bandwidth environments.

### Software Development Teams and Startups

* Need fast, simple Kubernetes clusters for development, testing, and CI/CD pipelines.
* Value the minimal overhead of k0s for prototyping and scaling quickly.

### Enterprises with Distributed Operations

* Manage hybrid setups with centralized control planes and distributed worker nodes across on-premises, cloud, and edge environments.
* Use k0s to simplify cluster management across multiple locations.

### Organizations in Regulated Industries

* Operate Kubernetes in on-premises or air-gapped environments due to regulatory requirements.
* 's lean design of reducing unnecessary complexity, making it easier to audit and verify compliance with regulatory standards

### Educational and Research Institutions

* Need Kubernetes clusters for experimentation, training, or academic research.
* Value k0s's simplicity for non-commercial use cases and resource-constrained setups.

### Software Vendors and System Integrators

* Deliver Kubernetes-based solutions to clients across various industries.
* Bundle and package k0s with their applications to create "Kubernetes appliances".
* Ship pre-configured, self-contained solutions for customers without requiring Kubernetes expertise.
* Appreciate k0s's ease of deployment, lightweight architecture, and flexibility to adapt to different environments.
* Benefit from its embedded-friendly design, making it ideal for turnkey application deployments in diverse scenarios.

## Completed end-user research

No comprehensive end-user research has been conducted.

## Usability

k0s simplifies Kubernetes operations by delivering the entire distribution as a single, self-contained binary. This design ensures:

1. **Streamlined Installation**
   * Deploy Kubernetes with a single command, without the need to manage multiple dependencies or configurations.
2. **Portability and Flexibility**
   * The binary can be easily copied and run on various systems, making it ideal for edge devices, bare-metal servers, or cloud environments.
3. **Simplified Upgrades**
   * Upgrading k0s is as easy as replacing the binary, reducing downtime and operational complexity. To help in orchestrating the cluster upgrade k0s comes with a component called [autopilot] that automates the node upgrade orchestration, including the needed node draining and coordination.
4. **Reduced Operational Overhead**
   * No need for additional packaging or complex tooling — everything needed to run Kubernetes is included in one binary.
5. **Developer and Operator Friendly**
   * The single-binary approach removes barriers to entry, enabling faster adoption and simpler workflows for teams of any size.

As such, k0s provides a conformant Kubernetes which means any ecosystem add-on works on k0s.

[autopilot]: https://docs.k0sproject.io/stable/autopilot/

## Design

### Design principles & best practices

#### Design Principles

1. **Simplicity First**
   * k0s is designed to minimize complexity for both users and operators. Packaging Kubernetes as a single binary and automating common tasks eliminates unnecessary configuration overhead.
2. **Lightweight and Efficient**
   * Optimized for resource-constrained environments, k0s ensures minimal system resource usage, making it ideal for edge computing, IoT, and small-scale deployments.
3. **Decoupled and Modular Architecture**
   * k0s separates the control plane and worker components, allowing flexible deployment topologies. It supports running control planes centrally while distributing worker nodes across diverse locations.
4. **Zero Friction, Zero Lock-In**
   * k0s follows a vendor-neutral approach, ensuring users retain full control of their infrastructure. It avoids proprietary tooling, adhering to upstream Kubernetes standards.
5. **Security by Default**
   * Secure configurations are baked into k0s from the start, with features like automatic TLS management, disabling anonymous access etc.
6. **Ease of Maintenance and Upgrades**
   * The project prioritizes operational simplicity, with upgrades streamlined through single-binary replacements, embedded autopilot for upgrades, and minimal manual intervention.
7. **Adaptability**
   * Designed to run in diverse environments, from local development setups to far-edge and production-grade clusters.

### **Best Practices**

1. **Stay Aligned with Upstream Kubernetes**
   * k0s is and will be 100% vanilla upstream Kubernetes.
2. **Automate Wherever Possible**
   * Automation of cluster configuration, control plane management, and other operational tasks ensures reliability and reduces human error.
3. **Test for Real-World Use Cases**
   * Every feature and release is tested in scenarios reflecting actual user environments, such as edge deployments, air-gapped setups, and hybrid clusters.
4. **Community-Driven Development**
   * Open communication with the user community drives prioritization and improvements, ensuring the project meets real-world needs.
5. **Focus on Documentation**
   * Clear and comprehensive documentation ensures users of all skill levels can deploy, manage, and scale k0s effectively.

## Identity and Access Management

k0s being vanilla upstream Kubernetes thus it supports all the same things as [Kubernetes does](https://kubernetes.io/docs/concepts/security/).
By default, k0s only sets up certificate authentication but users can fully configure OIDC or webhook authentication when they need.

## Compliance requirements implemented

## HA Requirements

1. **Control Plane Redundancy**
   * To achieve HA, the control plane (API server, etcd, controller manager, and scheduler) must be deployed in a redundant configuration:
     * **Multiple Control Plane Nodes**: At least three control plane nodes are recommended to ensure quorum and fault tolerance for etcd. In case kine is used with HA SQL database backend, 2 controller nodes are often sufficient.
     * **Load Balancer**: A load balancer is required in front of the control plane nodes to distribute API traffic evenly and provide fail-over.
       * k0s offers an embedded control plane load balancer ([CPLB]) for use cases where the infrastructure/network does not provide easy ways to create load balancers on its own.
       * K0s has a node-local ([NLLB]) for worker nodes to automatically re-route API connections to different nodes in case of failures.
2. **Worker Node Scalability**
   * Worker nodes operate independently of the control plane. In an HA setup:
     * Worker nodes can connect to multiple control plane endpoints for resilience.
     * Horizontal scaling of worker nodes ensures workload availability and supports increased demand.
3. **Distributed etcd**
   * etcd, the data store for Kubernetes, requires three or five instances for HA to maintain quorum.
   * etcd instances are distributed across control plane nodes, ensuring data replication and consistency even if a node fails.
   * k0s supports externally (from k0s point of view) managed etcd.
4. **Kine**
   * k0s also supports [kine] as etcd replacement thus offering users the ability to utilize externally managed HA databases as the control plane state storage.

[CPLB]: https://docs.k0sproject.io/stable/cplb/
[NLLB]: https://docs.k0sproject.io/stable/nllb/
[kine]: https://github.com/k3s-io/kine/

## Resource requirements

**k0s** is designed to be lightweight, making it suitable resource-constrained systems. Below is an overview of the resource requirements, including CPU, memory, storage, and networking considerations.

### Minimum Memory and CPU Requirements

The following table outlines the approximate minimum hardware requirements for different node roles:

| Role | Memory (RAM) | Virtual CPU (vCPU) |
| ----- | ----- | ----- |
| Controller Node | 1 GB | 1 vCPU |
| Worker Node | 0.5 GB | 1 vCPU |
| Controller \+ Worker | 1 GB | 1 vCPU |

These values are approximations; actual requirements may vary based on workload and cluster size.

### Controller Node Recommendations for Larger Clusters

For larger clusters, the recommended resources for controller nodes are:

| Number of Worker Nodes | Number of Pods | Recommended RAM | Recommended vCPU |
| ----- | ----- | ----- | ----- |
| Up to 10 | Up to 1,000 | 1–2 GB | 1–2 vCPU |
| Up to 50 | Up to 5,000 | 2–4 GB | 2–4 vCPU |
| Up to 100 | Up to 10,000 | 4–8 GB | 2–4 vCPU |
| Up to 500 | Up to 50,000 | 8–16 GB | 4–8 vCPU |
| Up to 1,000 | Up to 100,000 | 16–32 GB | 8–16 vCPU |
| Up to 5,000 | Up to 150,000 | 32–64 GB | 16–32 vCPU |

These recommendations help ensure optimal performance and stability for larger deployments.

### Storage Requirements

* **Controller Node**: Approximately 0.5 GB for k0s components; minimum 0.5 GB required.
* **Worker Node**: Approximately 1.3 GB for k0s components; minimum 1.6 GB required.
* **Controller \+ Worker**: Approximately 1.7 GB for k0s components; minimum 2.0 GB required.

It's recommended to use SSDs for optimal storage performance, as cluster latency and throughput are sensitive to storage performance.

### Networking Requirements

k0s requires certain network ports to be open for proper communication between components. Detailed information on the required ports and protocols can be found in the k0s networking documentation.

### Host Operating System and Architecture

* **Operating Systems**:
  * Linux (kernel version 4.3 or later)
    * `x86_64`
    * `aarch64`
    * `armv7l`
  * Windows Server 2019 (experimental)
    * `x86_64`

**Note:** K0s is actively tested on the `armv7l` architecture, which is not
currently done by upstream Kubernetes. This has enabled us to identify and
resolve architecture-related issues that upstream Kubernetes did not detect,
before they were included in a stable Kubernetes release.

These specifications ensure compatibility across a wide range of hardware platforms.

### Additional Considerations

* **Operating System Dependencies**: k0s strives to be as independent from the operating system as possible.
* The necessary kernel configurations and any external runtime dependencies are documented in the [k0s system requirements].

[k0s system requirements]: https://docs.k0sproject.io/stable/system-requirements/

## Storage requirements

### Controller Node Storage

* **Storage for Kubernetes Control Plane**:
  * Approximately **0.5 GB** is required for k0s system components on a controller-only node.
  * Persistent storage is essential for the **etcd data store**, which maintains the cluster’s state.
  * Recommended Storage:
    * **SSD** for improved etcd performance.
    * At least **20 GB** of free space for larger clusters with significant cluster state changes.

### Worker Node Storage

* **Storage for Kubernetes Workloads**:
  * Approximately **1.3 GB** is required for k0s system components.
  * Additional space is needed for container images, temporary files, and any application-specific storage.

## API Design

1. **Kubernetes API Compatibility**
   * k0s runs the upstream Kubernetes API server as-is, providing users with the standard Kubernetes API experience.
   * This ensures seamless interaction with Kubernetes-native tools (e.g., `kubectl`, Helm, and CI/CD systems) and compatibility with Kubernetes custom resources.
2. **Declarative API Model**
   * Like Kubernetes, k0s uses a declarative API model, enabling users to define desired states for resources (e.g., deployments, services, and custom objects).
   * The system continuously reconciles actual states with desired states, ensuring consistent and predictable behavior.
3. **API Evolution and Versioning**
   * The k0s API design follows Kubernetes’ API versioning practices, supporting multiple API versions (e.g., `v1beta1`, `v1`) for gradual transitions and backward compatibility.
     * We currently support only `v1beta1` but are planning for the next version (`v2`)
   * Deprecated APIs are phased out according to Kubernetes release cycles, ensuring compatibility with upstream developments.

## Release process

The k0s project closely follows the upstream Kubernetes release cycle. The only
difference in the Kubernetes release and maintenance schedule is that our
initial release date is always a few weeks after the Kubernetes release date.
This is because we build our version of k0s from the officially released
Kubernetes version and need time to test the final version before shipping.
![][image1]
The k0s version string consists of the Kubernetes version and the k0s version. For example:

```text
{{{ k0s_version }}}
```

The Kubernetes version ({{{ k0s_version.split('+') | first }}}) is the first part, and the last part ({{{ k0s_version.split('+') | last }}}) reflects the k0s version, which is built on top of the certain Kubernetes version.

## Installation

Here’s an example of how to set a single node cluster:

```console
# export K0S_VERSION={{{ k0s_version }}}
# curl -sSfL https://github.com/k0sproject/k0s/releases/download/$K0S_VERSION/k0s-$K0S_VERSION-amd64 -o k0s
# chmod u+x k0s
# ./k0s install controller --single && ./k0s start

```

Naturally, this will spin up k0s with the default configuration. In case the
user needs to configure something, they can create a YAML document to describe
the [configuration](https://docs.k0sproject.io/stable/configuration/):

As part of the startup sequence, k0s performs a series of pre-flight checks. This ensures that the system meets the requirements,
such as all required kernel modules are loaded, enough free disk capacity on the node, and so on.
If the pre-flight checks fail, k0s will refuse to start unless it's explicitly told to, and the logs will clearly indicate why they failed.

Additionally, users have the option to run the CNCF Certified Kubernetes Conformance test suite.
This test suite is executed by the CI for each release (on `x86_64` and `aarch64`) and its results are made available as release artifacts.

## Security

### CNCF Security self-assessment

See separate [document](security-self-assessment.md).

### Security hygiene

#### Code Quality and Development Practices

* **Version Control and Workflow**:
  * All development is managed through Git and a structured branching strategy.
  * Pull Requests (PRs) are mandatory for all changes, requiring reviews and approval by maintainers before merging.
* **Code Reviews**:
  * Every PR undergoes peer review to ensure adherence to coding standards and to identify potential issues early.
* **Automated Testing**:
  * **Unit Tests**: Validate individual components for functionality and correctness.
  * **End-to-End (E2E) Tests**: Assess cluster-level behaviors and workflows under real-world scenarios.
  * **Conformance Tests**: Ensure alignment with upstream Kubernetes standards.
* **Continuous Integration/Continuous Deployment (CI/CD)**:
  * CI pipelines run automated tests, perform builds, and validate compatibility before code is merged.
    * All CI is built on GitHub actions with declarative and open models

#### Security Practices

* **Dependency Management**:
  * **Dependabot** is used to automate security updates for vulnerable libraries.
  * **System Images** shipped with k0s are regurarly scanned by Trivy.
* **Signed Binaries**:
  * All k0s binaries are signed using **Cosign** as part of the release process to prevent tampering.
  * [Documentation](https://docs.k0sproject.io/stable/verifying-signs/) has instructions for users on how to verify the signature
* **Secure Default Configurations**:
  * k0s ships with secure-by-default settings, such as TLS encryption for API communication and RBAC enabled by default.
* **Vulnerability Response**:
  * A defined process for responding to discovered vulnerabilities, including prompt patching and communication with users.
    See [k0s security policy] for more information.

[k0s security policy]: https://github.com/k0sproject/k0s/blob/main/SECURITY.md

#### Community and Governance

* **Community Involvement**:
  * Security and health are bolstered by a transparent development process that invites contributions from the community.
  * Security issues can be reported confidentially to maintainers via a designated vulnerability disclosure program.
* **Governance Model**:
  * Decisions about features, security, and releases are guided by a structured governance framework involving maintainers and key contributors.
* **Documentation Standards**:
  * Comprehensive and updated documentation ensures users follow best practices, reducing misconfigurations and security risks.

#### Release Management

* **Semantic Versioning**:
  * Releases follow semantic versioning to provide clarity on the nature of changes (major, minor, or patch).
* **Testing Before Release**:
  * Each release undergoes rigorous automated testing to ensure stability and security.
* **Timely Security Patches**:
  * Critical vulnerabilities are addressed through prompt patch releases , given that an upstream fix is available.
* **Signed Binaries**:
  * k0s binaries are signed with **Cosign**, ensuring artifact integrity and authenticity.
* **Release Notes**:
  * Detailed notes accompany each release, highlighting new features, bug fixes, and security updates.

#### Compliance with CNCF Best Practices

* **Kubernetes Conformance**:
  * Regular conformance testing ensures compatibility with upstream Kubernetes APIs.
* **TAG Security Alignment**:
  * Security practices align with CNCF TAG Security recommendations, providing a robust baseline for health and security.

#### Supply Chain Security

* **Source Code Integrity**:
  * Code hosted on trusted platform on GitHub.
  * Git used as version control and all changes going through pull request process to ensure integrity.
* **Build Pipeline Security**:
  * All build pipelines are managed as code using GitHub Actions workflows in the k0s git repository.
  * All changes are going through normal review process.
  * Non-maintainer PRs require approval to run the CI workflows allowing a maintainer to verify the PR before allowing it to run.
* **Dependency Transparency**:
  * The project maintains transparency about its dependencies and their versions.
  * Automated dependency updates are handled by [Dependabot] and [Renovate Bot].
  * A signed SBOM report, in SPDX format, is produced for all releases.

[Dependabot]: https://github.com/dependabot
[Renovate Bot]: https://docs.renovatebot.com/

---

--- METRICS ---
- Files Extracted: 82
- Estimated Token Budget: ~130755 tokens
- Canonical Reference: https://codewiki.google/github.com/k0sproject/k0s