k0s Documentation

Technical Documentation: k0sproject/k0s

ℹ️ Provenance: Multi-Tier Ingestion · CodeWiki Reference

Overview (`README.md`) Copy Section

k0s - The Zero Friction Kubernetes

Overview Copy Chapter

k0s is an open source, all-inclusive Kubernetes distribution, which is configured with all of the features needed to build a Kubernetes cluster and packaged as a single binary for ease of use. Due to its simple design, flexible deployment options and modest system requirements, k0s is well suited for

• Any cloud
• Bare metal
• Edge and IoT

k0s drastically reduces the complexity of installing and running a CNCF certified Kubernetes distribution. With k0s new clusters can be bootstrapped in minutes and developer friction is reduced to zero. This allows anyone with no special skills or expertise in Kubernetes to easily get started.

k0s is distributed as a single binary with zero host OS dependencies besides the host OS kernel. It works with any Linux without additional software packages or configuration. Any security vulnerabilities or performance issues can be fixed directly in the k0s distribution that makes it extremely straightforward to keep the clusters up-to-date and secure.

Key Features Copy Chapter

• Certified and 100% upstream Kubernetes
• Multiple installation methods: single-node, multi-node, airgap and Docker
• Automatic lifecycle management with k0sctl: upgrade, backup and restore
• Modest system requirements (1 vCPU, 1 GB RAM)
• Available as a single binary with no external runtime dependencies besides the kernel
• Flexible deployment options with control plane isolation as default
• Scalable from a single node to large, high-available clusters
• Supports custom Container Network Interface (CNI) plugins (Kube-Router is the default, Calico is offered as a preconfigured alternative)
• Supports custom Container Runtime Interface (CRI) plugins (containerd is the default)
• Supports all Kubernetes storage options with Container Storage Interface (CSI)
• Supports a variety of datastore backends: etcd (default for multi-node clusters), SQLite (default for single node clusters), MySQL, and PostgreSQL
• Supports x86-64, ARM64, ARMv7 and RISC-V
• Includes Konnectivity service, CoreDNS and Metrics Server

Getting Started Copy Chapter

If you'd like to try k0s, please jump in to our:

• Quick Start Guide - Create a full Kubernetes cluster with a single node that includes both the controller and the worker.
• Install using k0sctl - Deploy and upgrade multi-node clusters with one command.
• NanoDemo - Watch a .gif recording on how to create a k0s instance.
• Run k0s in Docker - Run k0s controllers and workers in containers.
• For docs, tutorials, and other k0s resources, see docs main page.

Join the Community Copy Chapter

The main community communication channel is #k0s-users on Kubernetes Slack.
There you'll find like-minded people and reach out for assistance and
discussions.

We welcome your help in building k0s! If you are interested, we invite you to
join the #k0s-dev channel on Kubernetes Slack and check out the contributing
guidelines.

We also run a community blog that's open to posts from anyone. Subscribe to
its RSS feed to stay up to date.

Community calls Copy Section

We hold regular community calls. Everyone is welcome to drop by to ask
questions, discuss ideas or planned contributions, or just chat:

• Community office hours, Eastern Hemisphere (EMEA-friendly time)
• Community office hours, Western Hemisphere (Americas-friendly time)
• Maintainers call, open to everyone but meant for maintainers to sync up on
  plans and work

All events are listed in the k0sproject meeting calendar, and the
meeting minutes are public.

Adopters Copy Section

k0s is used across diverse environments, from small-scale far-edge deployments
to large data centers. Share your use case and add yourself to the list of
adopters.

Motivation Copy Chapter

We have seen a gap between the host OS and Kubernetes that runs on top of it: How to ensure they work together as they are upgraded independent from each other? Who is responsible for vulnerabilities or performance issues originating from the host OS that affect the K8S on top?

→ k0s is fully self contained. It’s distributed as a single binary with no host OS deps besides the kernel. Any vulnerability or perf issues may be fixed in k0s Kubernetes.

We have seen Kubernetes with cumbersome lifecycle management, high minimum system requirements, weird host OS and infra restrictions, and/or need to use different distros to meet different use cases.

→ k0s is designed to be lightweight at its core. It comes with a tool to automate cluster lifecycle management. It works on any host OS and infrastructure, and may be extended to work with any use cases such as edge, IoT, telco, public clouds, private data centers, and hybrid & hyper converged cloud applications without sacrificing the pure Kubernetes compliance or amazing developer experience.

Status Copy Chapter

k0s is ready for production (starting from v1.21.0+k0s.0). Since the initial release of k0s back in November 2020, we have made numerous releases, improved stability, added new features, and most importantly, listened to our users and community in an effort to create the most modern Kubernetes product out there. The active development continues to make k0s even better.

Scope Copy Chapter

While some Kubernetes distros package everything and the kitchen sink, k0s tries to minimize the amount of "add-ons" to bundle in. Instead, we aim to provide a robust and versatile "base" for running Kubernetes in various setups. Of course we will provide some ways to easily control and setup various "add-ons", but we will not bundle many of those into k0s itself. There are a couple of reasons why we think this is the correct way:

• Many of the addons such as ingresses, service meshes, storage etc. are VERY opinionated. We try to build this base with fewer opinions. :D
• Keeping up with the upstream releases with many external addons is very maintenance heavy. Shipping with old versions does not make much sense either.

With strong enough arguments we might take in new addons, but in general those should be something that are essential for the "core" of k0s.

Build Copy Chapter

The requirements for building k0s from source are as follows:

• GNU Make (v3.81 or newer)
• A POSIX shell
• coreutils
• findutils
• Docker

All of the compilation steps are performed inside Docker containers, no
installation of Go is required.

The k0s binary can be built in different ways:

The "k0s" way, self-contained, all binaries compiled from source, statically
linked, including embedded binaries:

make

The "package maintainer" way, without building and embedding the required
binaries. This assumes necessary binaries are provided separately at runtime:

make EMBEDDED_BINS_BUILDMODE=none

Docker build integration is enabled by default. However, in environments without
Docker, you can use the Go toolchain installed on the host system to build k0s
without embedding binaries. Note that static linking is not possible with
glibc-based toolchains:

make DOCKER='' EMBEDDED_BINS_BUILDMODE=none BUILD_GO_LDFLAGS_EXTRA=''

Note that the k0s build system does not currently support building the embedded
binaries without Docker. However, the embedded binaries can be built
independently using Docker:

make -C embedded-bins

Builds can be done in parallel:

make -j$(nproc)

Smoke test Copy Chapter

Additionally to the requirements for building k0s, the smoke tests do require
a local Go installation. you can run ./vars.sh go_version in a terminal to
find out the version that's being used to build k0s. It will print the
corresponding Go version to stdout.

To run a basic smoke test after build:

make check-basic

Overview (`docs/README.md`) Copy Section

k0s - The Zero Friction Kubernetes

{%
include-markdown "../README.md"
start=""
end=""
%}

{%
include-markdown "../README.md"
start=""
end=""
%}

Getting Started Copy Chapter

• Quick Start Guide for creating a full Kubernetes cluster with a single node.
• Install using k0sctl - Deploy and upgrade multi-node clusters with one command.
• Run k0s in Docker - Run k0s controllers and workers in containers.

Demo Copy Chapter

{%
include-markdown "../README.md"
start=""
end=""
%}

Adopters Copy Section

k0s is used across diverse environments, from small-scale far-edge deployments
to large data centers. Share your use case and add yourself to the list of
adopters.

Commercial Support Copy Chapter

Mirantis offers technical support, professional services and training for k0s. The support subscriptions include, for example, prioritized support (Phone, Web, Email) and access to verified extensions on top of your k0s cluster.

For any k0s inquiries, please contact us via email [email protected].

{%
include-markdown "../README.md"
start=""
end=""
%}

{%
include-markdown "../README.md"
start=""
end=""
%}

Overview (`inttest/README.md`) Copy Section

Integration tests a.k.a e2e testing

This folder is the root of k0s integration tests. These tests are such that run the actual k0s clusters, currently using bootloose as the target environment.

Running the tests Copy Chapter

There's a Makefile defining the tests as a set of make targets.

Test design Copy Chapter

We're currently building the tests as Golang tests with the help of testify test suite concept. The suite concept allows us to have suite level setup and teardown functionality so we can bootstrap and delete the test environment properly during testing. The suite setup phase creates the "infrastructure" for the tests and the teardown, as the name implies, deletes the infra.

Keeping the test env after tests Copy Chapter

Sometimes, especially when debugging some test failures, it's good to leave the environment running after the tests have ran. To control that behavior there's an env variable called K0S_KEEP_AFTER_TESTS. The value given to that has the following logic:

• no value or K0S_KEEP_AFTER_TESTS="never": The test env is NOT left running regardless of the test results
• K0S_KEEP_AFTER_TESTS="always": The test env is left running regardless of the test results
• K0S_KEEP_AFTER_TESTS="failure": The test env is left running only if the tests have failed

The test output show how to run manual cleanup for the environment, something like:

TestNetworkSuite: bootloosesuite.go:138: bootloose cluster left intact for debugging. Needs to be manually cleaned with: bootloose delete --config /tmp/afghzzvp-bootloose.yaml

This allows you to run manual cleanup after you've done the needed debugging.

Which tests to run when? Copy Chapter

The plan is to run only the basic (and quick) smoke tests on each PR commit. We should build some bot like functionality to run longer and more expensive tests using some trigger before the final merge of a PR. Naturally we should be able to run any of the tests locally, or at least triggered locally, to ensure we can actually debug what is happening.

Overview (`hack/ostests/README.md`) Copy Section

OpenTofu modules for k0s OS testing

Provisioning of k0s test clusters using different operating system stacks and
network providers based on OpenTofu and k0sctl.

By default, a test cluster will have the following properties:

• 3 controller nodes
• 2 worker nodes
• podCIDR is set to 10.244.0.0/16
• Node-local load balancing is enabled.
• All the rest of the configuration is left at its defaults.

Two workers are a minimum requirement for the Kubernetes conformance tests.

Requirements Copy Chapter

• OpenTofu >= 1.8

For the local plumbing:

• A POSIXish environment (env, sh, echo, printf)
• k0sctl (tested with 0.17.1)
• jq (tested with ~= 1.6)

For the AWS infra:

• Have the CLI credentials setup, in the usual AWS CLI way.
• Have a configured default region. That region will be targeted by OpenTofu.

Deploying a cluster Copy Chapter

Be sure to meet the requisites listed above. Configure AWS credentials and
region. This is just an example, there are other ways to do it.

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_SESSION_TOKEN=...
export AWS_REGION=...

Select the desired cluster configuration for OpenTofu. This is just one example;
other methods of passing configuration options are covered in the official
OpenTofu documentation.

export TF_VAR_os=alpine_3_24
export TF_VAR_arch=x86_64
export TF_VAR_k0s_version=stable
export TF_VAR_k0s_network_provider=calico
export TF_VAR_k0s_kube_proxy_mode=ipvs

Apply the configuration:

tofu apply

OpenTofu configuration variables Copy Chapter

`os`: Operating system stack Copy Section

• al2023: Amazon Linux 2023
• alpine_3_21: Alpine Linux 3.21
• alpine_3_24: Alpine Linux 3.24
• centos_9: CentOS Stream 9
• centos_10: CentOS Stream 10 (Coughlan)
• debian_11: Debian GNU/Linux 11 (bullseye) (supported until 2026-08-31)
• debian_12: Debian GNU/Linux 12 (bookworm) (supported until 2028-06-30)
• fcos_stable: Fedora CoreOS stable stream
• fedora_41: Fedora Linux 41 (Cloud Edition)
• flatcar: Flatcar Container Linux by Kinvolk
• oracle_8_9: Oracle Linux Server 8.9
• oracle_9_3: Oracle Linux Server 9.3
• rhel_7: Red Hat Enterprise Linux Server 7.9 (Maipo)
• rhel_8: Red Hat Enterprise Linux 8.10 (Ootpa)
• rhel_9: Red Hat Enterprise Linux 9.7 (Plow)
• rocky_8: Rocky Linux 8.10 (Green Obsidian)
• rocky_9: Rocky Linux 9.5 (Blue Onyx)
• sles_15: SUSE Linux Enterprise Server 15 SP6
• ubuntu_2004: Ubuntu 20.04 LTS
• ubuntu_2204: Ubuntu 22.04 LTS
• ubuntu_2404: Ubuntu 24.04
• windows_server_2022: Windows Server 2022 (runs the control plane on Alpine 3.23)

`arch`: Node architecture Copy Section

The underlying processor architecture for the to-be-provisioned cluster. Note
that not all operating systems support all architectures.

• x86_64
• arm64

`k0sctl_skip`: Skip k0s provisioning altogether Copy Section

Just provision the infrastructure, but no k0s cluster. This may be used for
development and testing purposes.

Assuming the AWS credentials are available, it can be used like this:

tofu init
export TF_VAR_os=alpine_3_24
export TF_VAR_k0sctl_skip=true
tofu apply
tofu output -json | jq -r '
  (.ssh_private_key_filename.value | @sh) as $keyFile
    | .hosts.value[]
    | select(.connection.type = "ssh")
    | . as {name: $host, ipv4: $ip, connection: {username: $user}}
    | "ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i \($keyFile) \($user)@\($ip) # \($host)"
'

`k0s_version`: The k0s version to deploy Copy Section

This may be a fixed version number, "stable" or "latest".

`k0s_executable_path`: The k0s version to deploy Copy Section

Path to the k0s executable to use, or null if it should be downloaded. Note that
for Windows, the .exe suffix is appended automatically.

`k0s_network_provider`: Network provider Copy Section

• kuberouter
• calico

`k0s_kube_proxy_mode`: Mode of operation for kube-proxy Copy Section

• iptables
• ipvs

See Kubernetes's IPVS README for details.

Adding a new operating system Copy Section

• Navigate to modules/os/ and add a new file os_.tf.
  Have a look at the other os_*.tf files for how it should look like.
• Add a new OS entry to modules/os/main.tf.
• Update this README.
• Test it: Be sure to have the requisites ready, as described at the top of this
  README, then do tofu apply -var=os=. When done, don't
  forget to clean up: tofu destroy -var=os=.
• Update the nightly trigger and matrix workflow with the new OS ID.

Updating the existing OS versions Copy Section

New OS version usually mean just updating the OS AMI filtering criteria.
To test the AMI filtering locally, you can do the following:

tofu apply -refresh-only -target=module.os.data.aws_ami.<the-os-id>
tofu state show "module.os.data.aws_ami.<the-os-id>[0]"

GitHub Actions workflows Copy Chapter

There's a reusable GitHub Actions workflow available in ostests-e2e.yaml. It
will deploy the OpenTofu resources and perform Kubernetes conformance tests
against the provisioned test cluster.

Launch a workflow run Copy Section

There's a nightly trigger for the OS testing workflow. It will select and run
a single testing parameter combination each day. There's also a matrix
workflow that exposes more knobs and can be triggered manually, e.g. via gh:

$ gh workflow run ostests-matrix.yaml --ref some/experimental/branch \
  -f oses='["alpine_3_24"]' \
  -f network-providers='["calico"]'
✓ Created workflow_dispatch event for ostests-matrix.yaml at some/experimental/branch

To see runs for this workflow, try: gh run list --workflow=ostests-matrix.yaml

Add -f fail-fast=true to the above command line to stop the e2e test suite
after the first test failure has been recorded.

TODO Copy Chapter

• Figure out the best/canonical way to change host names of the AWS instances

Architecture Index (`docs/architecture/index.md`) Copy Section

Architecture

Note: As k0s is a dynamic project, the product architecture may occasionally outpace the documentation. The high level concepts and patterns, however, should always apply.

Packaging Copy Chapter

The k0s package is a single, self-extracting binary that embeds Kubernetes binaries, the benefits of which include:

• Statically compiled
• No OS-level dependencies
• Requires no RPMs, dependencies, snaps, or any other OS-specific packaging
• Provides a single package for all operating systems
• Allows full version control for each dependency

Control plane Copy Chapter

As a single binary, k0s acts as the process supervisor for all other control plane components. As such, there is no container engine or kubelet running on controllers by default, which thus means that a cluster user cannot schedule workloads onto controller nodes.

Using k0s you can create, manage, and configure each of the components, running each as a "naked" process. Thus, there is no container engine running on the controller node.

Storage Copy Chapter

Kubernetes control plane typically supports only etcd as the datastore. k0s, however, supports many other datastore options in addition to etcd, which it achieves by including kine. Kine allows the use of a wide variety of backend data stores, such as MySQL, PostgreSQL, SQLite, and dqlite (refer to the spec.storage documentation).

In the case of k0s managed etcd, k0s manages the full lifecycle of the etcd cluster. For example, by joining a new controller node with k0s controller "long-join-token" k0s automatically adjusts the etcd cluster membership info to allow the new member to join the cluster.

Note: k0s cannot shrink the etcd cluster. As such, to shut down the k0s controller on a node that node must first be manually removed from the etcd cluster.

Worker node Copy Chapter

As with the control plane, with k0s you can create and manage the core worker components as naked processes on the worker node.

By default, k0s workers use containerd as a high-level runtime and runc as a low-level runtime. Custom runtimes are also supported, refer to Using custom CRI runtimes.

Contributors Index (`docs/contributors/index.md`) Copy Section

Contributing to k0s

Thank you for taking the time to contribute to k0s! This page gives you an
overview of how to contribute, and where to find the details.

Please follow the k0sproject community's code of conduct and
the k0sproject AI policy in all your interactions with the project.

If you have found a security vulnerability, don't disclose it publicly. Report
it confidentially, as described in the security policy.

Contributing to the repository Copy Chapter

K0s is more than its code. Contributions to the documentation are as welcome and
as valuable as contributions to the code base.

Seek early feedback Copy Section

If there's more than one way to approach your change, reach out via GitHub
issues or other community channels before
you start building it. Alternatives to a feature and different implementation
paths are easier to compare in the abstract than in a finished diff. Discussing
early also shows whether a change fits the project's direction, and whether
someone is already working on it.

Code Copy Section

The code guidelines collect the conventions of the k0s
code base beyond what the linters enforce, such as comments, error handling,
logging, and how to maintain compatibility across versions.

If your change affects how k0s is used or configured, update the documentation
along with it.

The testing page explains how to run the linters, unit tests, and
integration tests locally, how to debug CI failures, and how to run CI on your
own fork.

Documentation Copy Section

The documentation lives under the docs directory and is published to
https://docs.k0sproject.io. Fixing a typo, clarifying a confusing section, or
writing a guide for something that isn't covered yet are all great
contributions.

The documentation page explains how the docs are built, how to add
a page to the navigation, and how to preview your changes locally.

Pull requests Copy Section

All changes, to the documentation and to the code alike, arrive as pull
requests. The GitHub workflow page walks you through the
process of

• forking the repository,
• writing commits that serve one purpose each and are signed off to certify
  the Developer Certificate of Origin,
• opening the pull request,
• and reworking your commits after a review round.

Before opening a pull request, run the linters and related tests locally, as
described on the testing page. CI runs on a pull request only
after a maintainer has approved it, so catching problems locally saves a round
trip.

Contributing beyond the repository Copy Chapter

Not every contribution is a change to this repository. There are many ways to
help k0s and the k0sproject community as a whole:

• Report bugs via GitHub issues.
  A clear report with steps to reproduce, the k0s version, and relevant logs is
  a valuable contribution on its own.
• Help others in GitHub issues and on #k0s-users.
  Answering a question, sharing how you solved a problem, or adding what you
  know to an issue someone else reported helps the community.
• Help the maintainers by testing and reviewing pull requests.
  Trying out a change in your environment, or reading it with a fresh pair of
  eyes, is valuable feedback that speeds up the review.
• Share your experience.
  Write a blog post, give a talk, or tell us about your use case.
  Add yourself to the list of adopters, and let us know on Slack, so we can
  spread the word.
• Join the community calls.
  Discuss ideas, ask questions, or just get to know the people behind k0s.

License Copy Chapter

By contributing, you agree that your contributions will be licensed as follows:

• All content residing under the "docs/" directory of this repository is licensed under "Creative Commons Attribution Share Alike 4.0 International" (CC-BY-SA-4.0). See docs/LICENSE for details.
• Content outside of the above mentioned directories or restrictions above is available under the "Apache License 2.0".

Airgap Install (`docs/airgap-install.md`) Copy Section

Air gapped Installation

You can install k0s in environments without Internet access. Air gapped
installations require an image bundle that contains all the container images
that would normally be pulled over the network. K0s uses so-called OCI archives
for this: Tarball representations of an OCI Image Layout. They allow for
multiple images to be packed into a single file. K0s will watch for image
bundles in the <data-dir>/images folder will automatically import them into
the container runtime.

There are several ways to obtain an image bundle:

• Use the pre-built image bundles for different target platforms that are
  created for each k0s release. They contain all the images for the default k0s
  image configuration and can be downloaded from
  the [GitHub releases page].
• Create your own image bundle. In this case, you can easily customize the
  bundle to include container images that are not used by default in k0s.

Note: When importing image bundles, k0s uses "loose" platform
matching. For
example, on arm/v8, k0s will also import arm/v7, arm/v6, and arm/v5 images. This
means that your bundle can contain multi-arch images, and the import will be
done using platform compatibility.

[GitHub releases page]: https://github.com/k0sproject/k0s/releases/{{{ k0s_version }}}

Creating image bundles Copy Chapter

Using k0s built-in tooling Copy Section

k0s ships with the k0s airgap sub-command, which is
dedicated for tooling for airgapped environments. It allows for listing the
required images for a given configuration, as well as bundling them into an OCI
Image Layout archive.

1. Create the list of images required by k0s.

   shell Copy

   k0s airgap list-images --all >airgap-images.txt

2. Review this list and edit it according to your needs.

3. Create the image bundle.

   shell Copy

   k0s airgap bundle-artifacts -v -o image-bundle.tar <airgap-images.txt

From a running worker node Copy Section

As containerd pulls all the images during the k0s worker normal bootstrap, you
can use it to build the OCI bundle with images.

Use the following commands on a machine with an installed k0s worker:

k0s ctr images export image-bundle.tar $(k0s airgap list-images | xargs)

Using third-party tools Copy Section

There are several CLI tools that can help you fetch OCI artifacts and manage OCI
Image Layouts, such as skopeo, oras, or crane. The following is an example
uses Docker:

Docker

1. Create the list of images required by k0s.

   shell Copy

   k0s airgap list-images --all >airgap-images.txt

2. Review this list and edit it according to your needs.

3. Pull the images.

   shell Copy

   xargs -I{} docker pull {} <airgap-images.txt

4. Create the bundle.

   shell Copy

   docker image save -o image-bundle.tar $(xargs <airgap-images.txt)

Placing image bundles on worker nodes Copy Chapter

By hand Copy Section

Copy the image-bundle.tar to the target machine into the images directory in
the k0s data directory. Copy the bundle only to the worker nodes. Controller
nodes don't use it.

# mkdir -p /var/lib/k0s/images
# cp image-bundle.tar /var/lib/k0s/images/image-bundle.tar

Via k0sctl Copy Section

As an alternative to the previous step, you can use k0sctl to upload image
bundles to worker nodes. k0sctl can also be used to upload the k0s binary file
to all nodes. Take a look at this example configuration with one controller and
one worker node to upload k0s binary and an image bundle:

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  k0s:
    version: {{{ k0s_version }}}
  hosts:
    - role: controller
      ssh:
        address: <controller-ip-address>
        user: ubuntu
        keyPath: /path/to/.ssh/id_rsa

      #  uploadBinary: <boolean>
      #    When true the k0s binaries are cached and uploaded
      #    from the host running k0sctl instead of downloading
      #    directly to the target host.
      uploadBinary: true

      #  k0sBinaryPath: <local filepath>
      #    Upload a custom or manually downloaded k0s binary
      #    from a local path on the host running k0sctl to the
      #    target host.
      # k0sBinaryPath: path/to/k0s_binary/k0s

    - role: worker
      ssh:
        address: <worker-ip-address>
        user: ubuntu
        keyPath: /path/to/.ssh/id_rsa
      uploadBinary: true
      files:
        # This airgap bundle file will be uploaded from the k0sctl
        # host to the specified directory on the target host
        - src: /path/to/airgap-bundle-amd64.tar
          dstDir: /var/lib/k0s/images
          perm: 0755

Disable image pulling (optional) Copy Chapter

Use the following k0s configuration to ensure that all pods and pod templates
managed by k0s contain an imagePullPolicy of Never, ensuring that no images
are pulled from the Internet at any time.

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  name: k0s
spec:
  images:
    default_pull_policy: Never

Autopilot Multicommand (`docs/autopilot-multicommand.md`) Copy Section

**Multi-Command Plans**

Autopilot relies on a Plan for defining the Commands that should be
executed, the Signal Nodes that each should be run on, and the status of
each Command.

A Plan:

• Defines one or many Commands that specify what actions should be performed.
• Specifies how Signal Nodes should be discovered per-Command.
• Saves the status of the Plan execution by resolved Signal Nodes

A Command:

• An instructional step inside of a Plan that is applied against a Signal Node

A Signal Node:

• Any node (controller or worker) that can receive updates with Autopilot.

**Execution** Copy Chapter

The execution of a Plan is the result of processing Commands through
a number of Processing States.

When a Plan is executed, each of the Commands are executed in the order
of their appearance in the Plan.

• A Plan transitions to the next Command only when the current Command posts
  a state of Completed.
• Any Command that posts one of the recognized Error States will result in the
  current Command and Plan to abort processing.
  • The status of the Command and Plan will reflect this.
• A Plan is considered finished only when all of its defined Commands post a
  Completed state.

**Status** Copy Chapter

The progress and state of each Command is recorded in the Plan status.

• Every Command in the Plan has an associated status entry with the same index
  as the Command
  • i.e. The second Command in a Plan has an index of 1, and so does its status.
• The status of all Commands is taken into consideration when determining if the
  Plan is finished.

**Example** Copy Chapter

The following is an example of a Plan that has been applied as is currently being
processed by autopilot.

(line numbers added for commentary below)

/* Detailed source-code truncated for AI context efficiency. */

• Lines 7-33 are the two Commands that make up this plan - an airgapupdate and k0supdate.
• Lines 38-55 are the associated status entries for the two Commands.

The state of this Plan excerpt is that autopilot has successfully processed the Plan, and
has begun processing the airgapupdate Command. Its status indicates SignalSent which means
that the Signal Node has been sent signaling information to perform an air gapped update.

**Processing States** Copy Chapter

The following are the various states that both Plans and Commands
adhere to.

stateDiagram-v2
    [*]-->NewPlan
    NewPlan-->SchedulableWait
    NewPlan-->Errors***
    
    SchedulableWait-->Schedulable
    SchedulableWait-->Completed
    Schedulable-->SchedulableWait

    Errors***-->[*]
    Completed-->[*]

Note that the Errors* state is elaborated in detail below in Error States.

**NewPlan** Copy Section

When a Plan is created with the name autopilot, the NewPlan state
processing takes effect.

It is the responsibility of NewPlan to ensure that the status of all
the Commands are represented in the Plan status. This Plan status
is needed at later points in Plan processing to determine if the entire
Plan is completed.

The main difference between NewPlan and all the other states is that
NewPlan will iterate over all commands; the other states deal with the
active command.

**SchedulableWait** Copy Section

Used to evaluate a Command to determine if it can be scheduled for processing.
If the Command is determined that it can be processed, the state is set to
Schedulable.

**Schedulable** Copy Section

The Schedulable state is set by SchedulableWait to indicate that this
command should execute. The execution of a Command in this state will be
whichever logic is defined by the Command.

The ending of this state should either transition to SchedulableWait for
further processing + completion detection, or transition to an error.

**Completed** Copy Section

The Completed state indicates that the command has finished processing.
Once a plan/command are in the Completed state, no further processing
will occur on this plan/command.

**Error States** Copy Chapter

When a plan or command processing goes into one of the designated error
states, this is considered fatal and the plan/command processing will
terminate.

Error states are generally defined by the Command implementation.
The core autopilot functionality is only interested when in the
4 core states (NewPlan, SchedulableWait, Schedulable, Completed),
and treats all other states as an error.

flowchart TD
    Errors --> IncompleteTargets
    Errors --> Restricted
    Errors -. no longer in use .-> MissingSignalNode

**Sequence: Example** Copy Chapter

Using the example above as a reference, this outlines the basic sequence of events of
state transitions to the operations performed on each object.

/* Detailed source-code truncated for AI context efficiency. */

Autopilot (`docs/autopilot.md`) Copy Section

Autopilot

A tool for updating your k0s controller and worker nodes using specialized plans.
There is a public update-server hosted on the same domain as the documentation site. See the example below on how to use it. There is only a single channel edge_release available. The channel exposes the latest released version.

How it works Copy Chapter

• You create a Plan YAML
  • Defining the update payload (new version of k0s, URLs for platforms, etc.)
  • Add definitions for all the nodes that should receive the update.
    • Either statically, or dynamically using label/field selectors
• Apply the Plan
  • Applying a Plan is a simple kubectl apply operation.
• Monitor the progress
  • The applied Plan provides a status that details the progress.

Automatic updates Copy Chapter

To enable automatic updates, create an UpdateConfig object:

apiVersion: autopilot.k0sproject.io/v1beta2
kind: UpdateConfig
metadata:
  name: example
  namespace: default
spec:
  channel: edge_release
  updateServer: https://updates.k0sproject.io/
  upgradeStrategy:
    type: periodic
    periodic:
      # The following fields configures updates to happen only on Tue or Wed at 13:00-15:00
      days: [Tuesdays,Wednesday]
      startTime: "13:00"
      length: 2h
  planSpec: # This defines the plan to be created IF there are updates available
    ...

Safeguards Copy Chapter

There are a number of safeguards in place to avoid breaking a cluster.

Stateless Component Copy Section

• The autopilot component were designed to not require any heavy state, or
  massive synchronization. Controllers can disappear, and backup controllers can
  resume the autopilot operations.

Workers Update Only After Controllers Copy Section

• The versioning that Kubelet and the Kubernetes API server adhere to requires that Kubelets
  should not be of a newer version than the API server.

• How autopilot handles this is that when a Plan is applied that has both controller
  and worker nodes, all the controller nodes will be updated first. It is only when
  all controllers have updated successfully that worker nodes will receive their
  update instructions.

Plans are Immutable Copy Section

• When you apply a Plan, autopilot evaluates all of the controllers and workers that
  should be included into the Plan, and tracks them in the status. After this point, no
  additional changes to the plan (other than status) will be recognized.
  • This helps in largely dynamic worker node environments where nodes that may have been
    matched by the selector discovery method no longer exist by the time the update
    is ready to be scheduled.

Controller Quorum Safety Copy Section

• Prior to scheduling a controller update, autopilot queries the API server of all
  controllers to ensure that they report a successful /ready.
• Only once all controllers are /ready will the current controller get sent update signaling.
• If a previously discovered controller target can no longer be resolved, the Plan transitions
  into an IncompleteTargets state and execution ends. Transient readiness failures requeue.

Controllers Update Sequentially Copy Section

• Despite having the configuration options for controllers to set concurrency, only one
  controller will be updated at a time.

Update Payload Verification Copy Section

• Each update object payload can provide an optional sha256 hash of the update content
  (specified in url), which is compared against the update content after it downloads.

Configuration Copy Chapter

Autopilot relies on a Plan object on its instructions on what to update.

Here is an arbitrary Autopilot plan:

apiVersion: autopilot.k0sproject.io/v1beta2
kind: Plan
metadata:
  name: autopilot

spec:
  id: id1234
  timestamp: now

  commands:
    - k0supdate:
        version: {{{ k0s_version }}}
        platforms:
          linux-amd64:
            url: https://github.com/k0sproject/k0s/releases/download/{{{ k0s_version }}}/k0s-{{{ k0s_version }}}-amd64
            sha256: '0000000000000000000000000000000000000000000000000000000000000000'
        targets:
          controllers:
            discovery:
              static:
                nodes:
                  - ip-172-31-44-131
                  - ip-172-31-42-134
                  - ip-172-31-39-65
          workers:
            limits:
              concurrent: 5
            discovery:
              selector:
                labels: environment=staging
                fields: metadata.name=worker2

Core Fields Copy Section

`apiVersion (required)`

• The current version of the Autopilot API is v1beta2, with a full group-version of autopilot.k0sproject.io/v1beta2

`metadata.name (required)`

• The name of the plan should always be autopilot
  • Note: Plans will not execute if they don't follow this convention.

Spec Fields Copy Section

`spec.id (optional)`

• An identifier that can be provided by the creator for informational and tracking purposes.

`spec.timestamp (optional)`

• A timestamp value that can be provided by the creator for informational purposes. Autopilot
  does nothing with this information.

`spec.commands[] (required)`

• The commands contains the commands that should be performed as a part of the plan.

**`k0supdate`** Command Copy Section

`spec.commands[].k0supdate.version (required)`

• The version of the binary being updated. This version is used to compare against the installed
  version before and after update to ensure success.

`spec.commands[].k0supdate.platforms.*.url (required)`

• A URL providing where the updated binary should be downloaded from, for this specific platform.
  • The naming of platforms is a combination of $GOOS and $GOARCH, separated by a hyphen (-)
    • e.g.: linux-amd64, linux-arm64, linux-arm
  • Note: The main supported platform is linux. Autopilot may work on other platforms, however
    this has not been tested.

`spec.commands[].k0supdate.platforms.*.sha256 (optional)`

• If a SHA256 hash is provided for the binary, the completed downloaded will be verified against it.

`spec.commands[].k0supdate.targets.controllers