k3s Documentation

1. Project Overview & Quickstart (k3s-io/k3s) Copy Chapter

File: README.md Copy Chapter

K3s - Lightweight Kubernetes

Lightweight Kubernetes. Production ready, easy to install, half the memory, all in a binary less than 100 MB.

Great for:

• Edge
• IoT
• CI
• Development
• ARM
• Embedding k8s
• Situations where a PhD in k8s clusterology is infeasible

What is this? Copy Chapter

K3s is a fully conformant production-ready Kubernetes distribution with the following changes:

1. It is packaged as a single binary.
2. It adds support for sqlite3 as the default storage backend. Etcd3, MariaDB, MySQL, and Postgres are also supported.
3. It wraps Kubernetes and other components in a single, simple launcher.
4. It is secure by default with reasonable defaults for lightweight environments.
5. It has minimal to no OS dependencies (just a sane kernel and cgroup mounts needed).
6. It eliminates the need to expose a port on Kubernetes worker nodes for the kubelet API by exposing this API to the Kubernetes control plane nodes over a websocket tunnel.

K3s bundles the following technologies together into a single cohesive distribution:

• Containerd & runc
• Flannel for CNI
• CoreDNS
• Metrics Server
• Traefik for ingress
• Klipper-lb as an embedded service load balancer provider
• Kube-router netpol controller for network policy
• Helm-controller to allow for CRD-driven deployment of helm manifests
• Kine as a datastore shim that allows etcd to be replaced with other databases
• Local-path-provisioner for provisioning volumes using local storage
• Host utilities such as iptables/nftables, ebtables, ethtool, & socat

These technologies can be disabled or swapped out for technologies of your choice.

Additionally, K3s simplifies Kubernetes operations by maintaining functionality for:

• Managing the TLS certificates of Kubernetes components
• Managing the connection between worker and server nodes
• Auto-deploying Kubernetes resources from local manifests in realtime as they are changed.
• Managing an embedded etcd cluster

What's with the name? Copy Chapter

We wanted an installation of Kubernetes that was half the size in terms of memory footprint. Kubernetes is a
10 letter word stylized as k8s. So something half as big as Kubernetes would be a 5 letter word stylized as
K3s. A '3' is also an '8' cut in half vertically. There is neither a long-form of K3s nor official pronunciation.

Is this a fork? Copy Chapter

No, it's a distribution. A fork implies continued divergence from the original. This is not K3s's goal or practice. K3s explicitly intends not to change any core Kubernetes functionality. We seek to remain as close to upstream Kubernetes as possible. However, we maintain a small set of patches (well under 1000 lines) important to K3s's use case and deployment model. We maintain patches for other components as well. When possible, we contribute these changes back to the upstream projects, for example, with SELinux support in containerd. This is a common practice amongst software distributions.

K3s is a distribution because it packages additional components and services necessary for a fully functional cluster that go beyond vanilla Kubernetes. These are opinionated choices on technologies for components like ingress, storage class, network policy, service load balancer, and even container runtime. These choices and technologies are touched on in more detail in the What is this? section.

How is this lightweight or smaller than upstream Kubernetes? Copy Chapter

There are two major ways that K3s is lighter weight than upstream Kubernetes:

1. The memory footprint to run it is smaller
2. The binary, which contains all the non-containerized components needed to run a cluster, is smaller

The memory footprint is reduced primarily by running many components inside of a single process. This eliminates significant overhead that would otherwise be duplicated for each component.

The binary is smaller by removing third-party storage drivers and cloud providers, explained in more detail below.

What have you removed from upstream Kubernetes? Copy Chapter

This is a common point of confusion because it has changed over time. Early versions of K3s had much more removed than the current version. K3s currently removes two things:

1. In-tree storage drivers
2. In-tree cloud provider

Both of these have out-of-tree alternatives in the form of CSI and CCM, which work in K3s and which upstream is moving towards.

We remove these to achieve a smaller binary size. They can be removed while remaining conformant because neither affects core Kubernetes functionality. They are also dependent on third-party cloud or data center technologies/services, which may not be available in many K3s' use cases.

Getting Started Copy Chapter

• Quick Install
• Architecture
• FAQ
• Contribute

Community Copy Chapter

• Slack Copy Section

Join Slack to chat with K3s developers and other K3s users. Great place to learn and ask questions: #k3s and #k3s-contributor and #k3s channel in CNCF Slack

• Getting involved Copy Section

GitHub Issues - Submit your issues and feature requests via GitHub.

• Community Meetings and Office hours Copy Section

The K3s developer community hangs out on Zoom to chat. Everybody is welcome.

Add the Linux Foundation iCal to your calendar:

• AMS/EMEA TZ 10:00 am PST - every second Tuesday of the month
• EMEA/APAC TimeZone friendly - every third Tuesday of the month

Meeting notes and agenda: https://hackmd.io/@k3s/meet-notes/

Meeting recordings: K3s Channel

You can check also the full details on the website: https://k3s.io/community

What's next? Copy Chapter

Check out our roadmap to see what we have planned moving forward.

Release cadence Copy Chapter

K3s maintains pace with upstream Kubernetes releases. Our goal is to release patch releases within one week, and new minors within 30 days.

Our release versioning reflects the version of upstream Kubernetes that is being released. For example, the K3s release v1.27.4+k3s1 maps to the v1.27.4 Kubernetes release. We add a postfix in the form of +k3s<number> to allow us to make additional releases using the same version of upstream Kubernetes while remaining semver compliant. For example, if we discovered a high severity bug in v1.27.4+k3s1 and needed to release an immediate fix for it, we would release v1.27.4+k3s2.

Documentation Copy Chapter

Please see the official docs site for complete documentation.

Quick-Start - Install Script Copy Chapter

The install.sh script provides a convenient way to download K3s and add a service to systemd or openrc.

To install k3s as a service, run:

curl -sfL https://get.k3s.io | sh -

A kubeconfig file is written to /etc/rancher/k3s/k3s.yaml and the service is automatically started or restarted.
The install script will install K3s and additional utilities, such as kubectl, crictl, k3s-killall.sh, and k3s-uninstall.sh, for example:

sudo kubectl get nodes

K3S_TOKEN is created at /var/lib/rancher/k3s/server/node-token on your server.
To install on worker nodes, pass K3S_URL along with
K3S_TOKEN environment variables, for example:

curl -sfL https://get.k3s.io | K3S_URL=https://myserver:6443 K3S_TOKEN=XXX sh -

Manual Download Copy Chapter

1. Download k3s from latest release, x86_64, armhf, arm64 and s390x are supported.
2. Run the server.

sudo k3s server &
# Kubeconfig is written to /etc/rancher/k3s/k3s.yaml
sudo k3s kubectl get nodes

# On a different node run the below. NODE_TOKEN comes from
# /var/lib/rancher/k3s/server/node-token on your server
sudo k3s agent --server https://myserver:6443 --token ${NODE_TOKEN}

Contributing Copy Chapter

Please check out our contributing guide if you're interested in contributing to K3s.

Security Copy Chapter

Security issues in K3s can be reported by sending an email to [email protected].
Please do not file issues about security issues.

File: contrib/ansible/README.md Copy Chapter

Build a Kubernetes cluster using k3s via Ansible

The ansible playbook was moved to https://github.com/k3s-io/k3s-ansible

File: docs/release/expanded/build_container.md Copy Chapter

Generate Build Container

1. set env variable PATH_TO_KUBERNETES_REPO to the path to your local kubernetes/kubernetes copy:
   text Copy

   export GHUSER="mtrachier"
   export PATH_TO_KUBERNETES_REPO="/Users/$GHUSER/go/src/github.com/kubernetes/kubernetes"

2. set env variable GOVERSION to the expected version of go for the kubernetes/kubernetes version checked out:
   text Copy

   export GOVERSION=$(yq -e '.dependencies[] | select(.name == "golang: upstream version").version' $PATH_TO_KUBERNETES_REPO/build/dependencies.yaml)

3. set env variable GOIMAGE to the expected container image to base our custom build image on:
   text Copy

   export GOIMAGE="golang:${GOVERSION}-alpine"

4. set env variable BUILD_CONTAINER to the contents of a dockerfile for the build container:
   text Copy

   export BUILD_CONTAINER="FROM ${GOIMAGE}\nRUN apk add --no-cache bash gnupg git make tar gzip curl git coreutils rsync alpine-sdk"

5. use Docker to create the build container:
   text Copy

   echo -e $BUILD_CONTAINER | docker build -t ${GOIMAGE}-dev -

File: docs/release/expanded/channel_server.md Copy Chapter

Update Channel Server

Once the release is verified, the channel server config needs to be updated to reflect the new version for “stable”.  

1. Channel.yaml can be found at the root of the K3s repo.
   1. When updating the channel server a single-line change will need to be performed.  
   2. Release Captains responsible for this change will need to update the following stanza to reflect the new stable version of kubernetes relative to the release in progress.  
   3. Example:
      text Copy

      channels:
        name: stable
        latest: v1.22.12+k3s1

File: docs/release/expanded/cut_release.md Copy Chapter

Cut Release

1. Verify that the merge CI has successfully completed before cutting the RC
2. After the merge CI has completed, cut an RC by creating a release in the GitHub interface
   1. the title is the version of k3s you are releasing with the rc1 subversion eg. "v1.25.0-rc1+k3s1"
   2. In the case of a update to k3s, it should be incremented from k3s1 to k3s2, for example, meaning the k3s version is being incremented.
   3. the target should match the release branch, remember that the latest version is attached to "main"
   4. no description
   5. the tag should match the title
3. After the RC is cut validate that the CI for the RC passes
4. After the RC CI passes notify the release SLACK channel about the new RC

Example Full Command List (this is not a script!):

/* Detailed source-code truncated for AI context efficiency. */

File: docs/release/expanded/milestones.md Copy Chapter

Generate Milestones

If no milestones exist in the k3s repo for the releases, generate them.
No due date or description necessary, we can update them as necessary afterwards.
Make sure to post the new milestones in the SLACK channel if generated.

File: docs/release/expanded/pr.md Copy Chapter

Generate Pull Request

We update the go.mod in k3s to point to the new modules, and submit the change for review.

1. make sure git is clean before making changes
2. make sure your origin is up to date before making changes
3. checkout a new branch for the new k3s version in the local copy using the formal semantic name eg. "v1.25.1-k3s1"
4. replace any instances of the old k3s version eg. "v1.25.0-k3s1" with the new k3s version eg. "v1.25.1-k3s1" in k3,s-io module links
5. replace any instances of the old Kubernetes version eg. "v1.25.0" with the new Kubernetes version eg. "v1.25.1"
6. replace any instances of the old Kubernetes client-go version eg. "v0.25.0" with the new version eg. "v0.25.1"
7. sed commands make this process easier (this is not a script):
   1. Linux example:
      text Copy

      sed -Ei "\|github.com/k3s-io/kubernetes| s|${OLD_K3S_VER}|${NEW_K3S_VER}|" go.mod
      sed -Ei "s/k8s.io\/kubernetes v\S+/k8s.io\/kubernetes ${NEW_K8S}/" go.mod
      sed -Ei "s/$OLD_K8S_CLIENT/$NEW_K8S_CLIENT/g" go.mod

   2. Mac example:
      text Copy

      # note that sed has different parameters on MacOS than Linux
      # also note that zsh is the default MacOS shell and is not bash/dash (the default Linux shells)
      sed -Ei '' "\|github.com/k3s-io/kubernetes| s|${OLD_K3S_VER}|${NEW_K3S_VER}|" go.mod
      git diff
      
      sed -Ei '' "s/k8s.io\/kubernetes v.*$/k8s.io\/kubernetes ${NEW_K8S}/" go.mod
      git diff
      
      sed -Ei '' "s/${OLD_K8S_CLIENT}/${NEW_K8S_CLIENT}/g" go.mod
      git diff
      
      go mod tidy
      git diff

8. update extra places to make sure the go version is correct
   1. .github/workflows/integration.yaml
   2. .github/workflows/unitcoverage.yaml
   3. Dockerfile.dapper
   4. Dockerfile.manifest
   5. Dockerfile.test
9. commit the changes and push to your origin
   1. make sure to sign your commits
   2. make sure to push to "origin" not "upstream", be explicit in your push commands
   3. example: 'git push -u origin v1.25.1-k3s1'
10. the git output will include a link to generate a pull request, use it
    1. make sure the PR is against the proper release branch
11. generating the PR starts several CI processes, most are in GitHub actions, but some one is in Drone, post the link to the drone CI run in the PR
    1. this keeps everyone on the same page
    2. if there is an error in the CI, make sure to note that and what the errors are for reviewers
    3. finding error messages:
       1. example: https://drone-pr.k3s.io/k3s-io/k3s/4744
       2. click the "show all logs" to see all of the logs
       3. search for " failed." this will find a line like "Test bEaiAq failed."
       4. search for "err=" and look for a log with the id "bEaiAq" in it
       5. example error:
          text Copy

          #- Tail: /tmp/bEaiAq/agents/1/logs/system.log
          [LATEST-SERVER] E0921 19:16:55.430977      57 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs
          [LATEST-SERVER] I0921 19:16:55.431186      57 proxier.go:667] "Failed to load kernel module with modprobe, you can ignore this message when kube-proxy is running inside container without mounting /lib/modules" moduleName="ip_vs_rr"

       6. the first part of the log gives a hint to the log level: "E0921" is an error log "I0921" is an info log
       7. you can also look for "Summarizing \d Failure" (I installed a plugin on my browser to get regex search: "Chrome Regex Search")
       8. example error:
          text Copy

          [Fail] [sig-network] DNS [It] should support configurable pod DNS nameservers [Conformance]

    4. example PR: https://github.com/k3s-io/k3s/pull/6164
    5. many errors are flakey/transitive, it is usually a good idea to simply retry the CI on the first failure
    6. if the same error occurs multiple times then it is a good idea to escalate to the team
12. After the CI passes (or the team dismisses the CI as "flakey"), and you have at least 2 approvals you can merge it
    1. make sure you have 2 approvals on the latest changes
    2. make sure the CI passes or the team approves merging without it passing
    3. make sure the use the "squash and merge" option in GutHub
    4. make sure to update the SLACK channel with the new Publish/Merge CI

• Help! My memory usage is off the charts and everything has slowed to a crawl!
  • I found rebooting after running tag.sh was the only way to solve this problem, seems like a memory leak in VSCode on Mac or maybe some weird behavior between all of the added/removed files along with VSCode's file parser, the Crowdstrike virus scanner, and Docker (my top memory users)

File: docs/release/expanded/rebase.md Copy Chapter

Rebase

1. clear out any cached or old files: git add -A; git reset --hard HEAD
2. clear out any cached or older outputs: rm -rf _output
3. rebase your local copy to move the old k3s tag from the old k8s tag to the new k8s tag
   1. so there are three copies of the code involved in this process:
      1. the upstream kubernetes/kubernets copy on GitHub
      2. the k3s-io/kubernetes copy on GitHub
      3. and the local copy on your laptop which is a merge of those
   2. the local copy has every branch and every tag from the remotes you have added
   3. there are custom/proprietary commits in the k3s-io copy that are not in the kubernetes copy
   4. there are commits in the kubernetes copy do not exist in the k3s-io copy
   5. we want the new commits added to the kubernetes copy to be in the k3s-io copy
   6. we want the custom/proprietary commits from the k3s-io copy on top of the new kubernetes commits
   7. before rebase our local copy has all of the commits, but the custom/proprietary k3s-io commits are between the old kubernetes version and the new kubernetes version
   8. after the rebase our local copy will have the k3s-io custom/proprietary commits after the latest kubernetes commits
   9. git rebase --onto $NEW_K8S $OLD_K8S $OLD_K3S_VER~1
   10. After rebase you will be in a detached head state, this is normal

File: docs/release/expanded/release_images.md Copy Chapter

Create Release Images

Check System Agent Installer Images Copy Chapter

The k3s-io/k3s Release CI should dispatch the rancher/system-agent-installer-k3s repo, generating a tag there and triggering the CI to build images.
The system-agent-installer-k3s repository is used with Rancher v2prov system.

After RCs are cut you need to check the system agent installer k3s, this along with KDM PR allows QA to fully test RCs.
This should happen directly after the KDM PR is generated, within a few hours of the release candidate being cut.
These images depend on the release artifact and can not be generated until after the k3s-io/k3s release CI completes.

Build progress can be tracked here.

This often fails! Check the CI and if it was not triggered do the following:

1. Create a release in the system-agent-installer-k3s repo
   1. it should exactly match the release title in the k3s repo
   2. the target is "main" for all releases (no branches)
   3. no description
   4. make sure to check the "pre-release" checkbox
2. Watch the Drone Publish CI, it should be very quick
3. Verify that the new images appear in Docker hub

Check K3S Upgrade Images Copy Chapter

The k3s-io/k3s Release CI should dispatch the k3s-io/k3s-upgrade repo, generating a tag there and triggering the CI to build images.
These images depend on the release artifact and can not be generated until after the k3s-io/k3s release CI completes.

This process will take some time but upon completion, the images will be listed here.

The k3s images will be published here.
The upgrade images will be published here.

This sometimes fails! Check the CI and if it was not triggered do the following:

1. Create a release in the system-agent-installer-k3s repo
   1. it should exactly match the release title in the k3s repo
   2. the target is "main" for all releases (no branches)
   3. no description
   4. make sure to check the "pre-release" checkbox
2. Watch the Drone Publish CI, it should be very quick
3. Verify that the new images appear in Docker hub

Make sure you are in constant communication with QA during this time so that you can cut more RCs if necessary,
update KDM if necessary, radiate information to the rest of the team and help them in any way possible.

File: docs/release/expanded/release_notes.md Copy Chapter

Create Release Notes PR

1. Use the release notes tool to generate the release notes
   1. the release notes tool is located in ecm distro tools
   2. you will need a valid GitHub token to use the tool
   3. call the tools as follows (example using v1.23.13-rc1+k3s1):
      text Copy

      # this outputs to stdout
      export GHT=$GITHUB_TOKEN
      export PREVIOUS_RELEASE='v1.23.12+k3s1'
      export LAST_RELEASE='v1.23.13-rc2+k3s1'
      docker run --rm -e GITHUB_TOKEN=$GHT rancher/ecm-distro-tools:latest gen_release_notes -r k3s -m $LAST_RELEASE -p $PREVIOUS_RELEASE

2. Update the first line to include the semver of the released version
   • example: ``
3. Make sure the title has the new k8s version, and the "changes since" line has the old version number
   • example title: This release updates Kubernetes to v1.25.3, and fixes a number of issues.
   • example "changes since": ## Changes since v1.25.2+k3s1
4. Verify changes
   1. go to releases
   2. find the previous release
   3. calculate the actual date from "XX days ago"
   4. search for pull requests which merged after that date
   5. go to the GitHub issue search UI
   6. search for PRs for the proper branch, merged after the last release
      • release branch is release-1.23
      • previous release was v1.23.12
      • date of the release was "Sept 28th 2022"
      • example search is:pr base:release-1.23 merged:>2022-09-28 sort:created-asc
   7. for each PR, validate the title of the pr and the commit message comments
      • each PR title (or 'release note' section of the first comment) should get an entry
      • the entry with this item should have a link at the end to the PR
      • the commit messages for the PR should follow, until the next PR
   8. if you suspect there is a missing/extra commit, compare the tags
      • use the github compare tool to compare the older tag to the newer one
      • example: https://github.com/k3s-io/k3s/compare/v1.25.2+k3s1...v1.25.3-rc2+k3s1
        • this will show all of the commit differences between the two, exposing all of the new commits
      • on the commit page you should see the merge issue associated with it
      • validate that the merge issue is listed in the release notes
      • if the commit is not in the comparison, try comparing the previous release tags
        • example: https://github.com/k3s-io/k3s/compare/v1.25.0+k3s1...v1.25.2+k3s1
        • the commit's merge issue should be listed in the release notes
   9. if you are adding backports, make sure you are using the backport issues, not the one for main
5. Verify component release versions
   • the list of components is completely static, someone should say something in the PR if we need to add to the list
     • Kubernetes, Kine, SQLite, Etcd, Containerd, Runc, Flannel, Metrics-server, Traefik, CoreDNS, Helm-controller, Local-path-provisioner
   • the version.sh script found in the k3s repo at scripts/version.sh is the source of truth for version information
   1. go to the k3s repo and browse the release tag for the notes you are verifying, example
   2. start by searching the version.sh file for the component
   3. if you do not find anything, search the build script found in ./scripts/build example
   4. if you still do not find anything, search the go.mod found in the root of the k3s repo example
   5. some things are in the k3s repo's manifests directory, see ./manifests example
   • example info for v1.23.13-rc2+k3s1
     text Copy

/* Detailed source-code truncated for AI context efficiency. */

## Understanding Release Notes

Here are the major sections in the release notes:
- changes since
  - this relates all changes since the previous release
  - more specifically every merge issue (PR) generated should have an entry
  - developers may add a special "User-Facing Change" section to their PR to give custom notes
    - these notes will appear as sub entries on the issue title
- released components
  - this relates all kubernetes 'components' in the release
  - components are generally non-core kubernetes options that we install using Helm charts

## 2. Official Technical Reference & Guides (k3s-io/docs)

## File: README.md

# K3s Website and Docs

This repo contains the content of the K3s documentation website found at https://docs.k3s.io. Please open an issue if you have suggestions for new content or edits. We also gladly accept community PRs. 

The website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator.

### Installation

$ yarn

### Local Development

$ yarn start

This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server.

You can also use Docker to launch the website without needing to install and configure yarn:

$ docker run --rm -d --name k3s-docs -e HOME=$PWD -v $PWD:$PWD -w $PWD -u $(id -u) -p 3000:3000 node:22 sh -c 'corepack yarn install --immutable && corepack yarn start -h 0.0.0.0'

### Build

$ yarn build

This command generates static content into the `build` directory and can be served using any static contents hosting service.

### Deployment

An automated GH action will deploy the website to GitHub Pages once a PR has been merged to `main`.

---

## File: docs/cli/agent.md

---
title: agent
---

# k3s agent

In this section, you'll learn how to configure the K3s agent.

Note that servers also run an agent, so all flags listed on this page are also valid for use on servers.

Options are documented on this page as CLI flags, but can also be passed as configuration file options. See the [Configuration File](../installation/configuration.md#configuration-file) documentation for more information on using YAML configuration files.

### Logging

| Flag                    | Default | Description                                                          |
| ----------------------- | ------- | -------------------------------------------------------------------- |
| `-v` value              | 0       | Number for the log level verbosity                                   |
| `--vmodule` value       | N/A     | Comma-separated list of FILE_PATTERN=LOG_LEVEL settings for file-filtered logging |
| `--log value, -l` value | N/A     | Log to file                                                          |
| `--alsologtostderr`     | N/A     | Log to standard error as well as file (if set)                       |

### Cluster Options

| Flag                       | Environment Variable | Description                          |
| -------------------------- | -------------------- | ------------------------------------ |
| `--token value, -t` value  | `K3S_TOKEN`          | Token to use for authentication      |
| `--token-file` value       | `K3S_TOKEN_FILE`     | Token file to use for authentication |
| `--server value, -s` value | `K3S_URL`            | Server to connect to                 |

### Listener

| Flag |  Default | Description |
| --- |  --- | --- |
| `--bind-address` | 0.0.0.0 | k3s bind address  |

### Data

| Flag                         | Default                | Description          |
| ---------------------------- | ---------------------- | -------------------- |
| `--data-dir value, -d` value | "/var/lib/rancher/k3s" | Folder to hold state |

### Node

| Flag                        | Environment Variable | Description                                                                                   |
| --------------------------- | -------------------- | --------------------------------------------------------------------------------------------- |
| `--node-name` value         | `K3S_NODE_NAME`      | Node name                                                                                     |
| `--with-node-id`            | N/A                  | Append id to node name                                                                        |
| `--node-label` value        | N/A                  | Registering and starting kubelet with set of labels                                           |
| `--node-taint` value        | N/A                  | Registering kubelet with set of taints                                                        |
| `--protect-kernel-defaults` | N/A                  | Kernel tuning behavior. If set, error if kernel tunables are different from kubelet defaults. |
|   `--selinux` | `K3S_SELINUX` | Enable SELinux in containerd |
|   `--lb-server-port` value | `K3S_LB_SERVER_PORT` | Local port for supervisor client load-balancer. If the supervisor and apiserver are not colocated an additional port 1 less than this port will also be used for the apiserver client load-balancer. (default: 6444) |

### Runtime

| Flag                                 | Default                            | Description                                                        |
| ------------------------------------ | ---------------------------------- | ------------------------------------------------------------------ |
| `--container-runtime-endpoint` | N/A | Disable embedded containerd and use the CRI socket at the given path; when used with --docker this sets the docker socket path |
| `--default-runtime` | N/A | Set the default runtime in containerd |
| `--image-service-endpoint` | N/A | Disable embedded containerd image service and use remote image service socket at the given path. If not specified, defaults to --container-runtime-endpoint. |
| `--pause-image` value                | "docker.io/rancher/pause:3.1"      | Customized pause image for containerd or docker sandbox            |
| `--private-registry` value           | "/etc/rancher/k3s/registries.yaml" | Private registry configuration file                                |

### Networking

| Flag                        | Environment Variable | Description                               |
| --------------------------- | -------------------- | ----------------------------------------- |
| `--node-ip value, -i` value | N/A                  | IP address to advertise for node          |
| `--node-external-ip` value  | N/A                  | External IP address to advertise for node |
| `--node-internal-dns` | N/A | internal DNS addresses to advertise for node |
| `--node-external-dns` | N/A | external DNS addresses to advertise for node |
| `--resolv-conf` value       | `K3S_RESOLV_CONF`    | Kubelet resolv.conf file                  |
| `--flannel-iface` value     | N/A                  | Override default flannel interface        |
| `--flannel-conf` value      | N/A                  | Override default flannel config file      |
| `--flannel-cni-conf` value  | N/A                  | Override default flannel cni config file  |

### Customized Flags

| Flag                     | Description                            |
| ------------------------ | -------------------------------------- |
| `--kubelet-arg` value    | Customized flag for kubelet process    |
| `--kube-proxy-arg` value | Customized flag for kube-proxy process |

### Experimental

| Flag         | Description                           |
| ------------ | ------------------------------------- |
| `--rootless` | Run rootless                          |
| `--docker`   | Use cri-dockerd instead of containerd |
| `--enable-pprof` | Enable pprof endpoint on supervisor port |
| `--prefer-bundled-bin` | Prefer bundled userspace binaries over host binaries |
| `--disable-default-registry-endpoint` | See "[Default Endpoint Fallback](../installation/private-registry.md#default-endpoint-fallback)"
| `--vpn-auth` | See "[Integration with the Tailscale VPN provider](../networking/distributed-multicloud.md#integration-with-the-tailscale-vpn-provider-experimental)"  |
| `--vpn-auth-file` | See "[Integration with the Tailscale VPN provider](../networking/distributed-multicloud.md#integration-with-the-tailscale-vpn-provider-experimental)" |

### Deprecated

| Flag                     | Environment Variable | Description                  |
| ------------------------ | -------------------- | ---------------------------- |
| `--no-flannel`           | N/A                  | Use `--flannel-backend=none` |
| `--cluster-secret` value | `K3S_CLUSTER_SECRET` | Use `--token`                |

### Node Labels and Taints for Agents

K3s agents can be configured with the options `--node-label` and `--node-taint` which adds a label and taint to the kubelet. The two options only add labels and/or taints at registration time, so they can only be added once and not changed after that again by running K3s commands.

Below is an example showing how to add labels and a taint:

```bash
     --node-label foo=bar \
     --node-label hello=world \
     --node-taint key1=value1:NoExecute

If you want to change node labels and taints after node registration you should use kubectl. Refer to the official Kubernetes documentation for details on how to add taints and node labels.

K3s Agent CLI Help Copy Section

If an option appears in brackets below, for example [$K3S_URL], it means that the option can be passed in as an environment variable of that name.

/* Detailed source-code truncated for AI context efficiency. */

File: docs/cli/certificate.md Copy Chapter

title: certificate Copy Chapter

k3s certificate

Client and Server Certificates Copy Chapter

K3s client and server certificates are valid for 365 days from their date of issuance.
Any certificates that are expired or within 120 days of expiring are automatically renewed every time K3s starts.
This renewal reuses the existing keys, and extends the lifetime of the existing certificates.
If you want to generate new certificates and keys, instead of extending the validity of the existing certificates, use the rotate subcommand documented below.

When a certificate is within 120 days of expiring a Kubernetes Warning Event with reason: CertificateExpirationWarning is created, with a relation to the Node using the certificate.

Prior to the May 2025 releases (v1.33.1+k3s1, v1.32.5+k3s1, v1.31.9+k3s1, v1.30.13+k3s1), alerts and rotation were triggered at 90 days, instead of 120 days.

Checking expiration dates Copy Section

:::info Version Gate
Output format is configurable as of the January 2025 releases: v1.32.0+k3s1, v1.31.5+k3s1, v1.30.9+k3s1, v1.30.13+k3s1
A new format with more information is available as of the May 2025 releases: v1.33.1+k3s1, v1.32.5+k3s1, v1.31.9+k3s1, v1.29.13+k3s1
:::

To check the node certificates and their expiration date use the k3s certificate check --output table:

FILENAME                    SUBJECT                     USAGES       EXPIRES                  RESIDUAL TIME   STATUS
--------                    -------                     ------       -------                  -------------   ------
client-kube-proxy.crt       system:kube-proxy           ClientAuth   Jun 09, 2026 10:17 UTC   1 year          OK
client-kube-proxy.crt       k3s-client-ca@1749464211    CertSign     Jun 07, 2035 10:16 UTC   10 years        OK
client-kubelet.crt          system:node:ip-10-11-0-14   ClientAuth   Jun 09, 2026 10:17 UTC   1 year          OK
client-kubelet.crt          k3s-client-ca@1749464211    CertSign     Jun 07, 2035 10:16 UTC   10 years        OK
serving-kubelet.crt         ip-10-11-0-14               ServerAuth   Jun 09, 2026 10:17 UTC   1 year          OK
serving-kubelet.crt         k3s-server-ca@1749464211    CertSign     Jun 07, 2035 10:16 UTC   10 years        OK
client-k3s-controller.crt   system:k3s-controller       ClientAuth   Jun 09, 2026 10:17 UTC   1 year          OK
client-k3s-controller.crt   k3s-client-ca@1749464211    CertSign     Jun 07, 2035 10:16 UTC   10 years        OK

:::info SAME CERTIFICATE TWICE
Each certificate file (FILENAME column) contains at least two certificates - the leaf (or end entity) client/server certificate, any intermediate Certificate Authority certificates, and the root Certificate Authority certificate.
:::

In case of unexpected output, please ensure that you are specifying the correct data directory with the --data-dir flag (if using a custom data directory), or use the --debug flag to get additional output from the check process.

Rotating Client and Server Certificates Copy Section

To rotate client and server certificates manually, use the k3s certificate rotate subcommand:

# Stop K3s
systemctl stop k3s

# Rotate certificates
k3s certificate rotate

# Start K3s
systemctl start k3s

Individual or lists of certificates can be rotated by specifying the certificate name:

k3s certificate rotate --service <SERVICE>,<SERVICE>

The following certificates can be rotated: admin, api-server, controller-manager, scheduler, k3s-controller, k3s-server, cloud-controller, etcd, auth-proxy, kubelet, kube-proxy.

Certificate Authority (CA) Certificates Copy Chapter

Kubernetes requires a number of CA certificates for proper operation. For more information on how Kubernetes uses CA certificates, see the Kubernetes PKI Certificates and Requirements documentation.

By default, K3s generates self-signed CA certificates during startup of the first server node. These CA certificates are valid for 10 years from date of issuance, and are not automatically renewed.

The authoritative CA certificates and keys are stored within the datastore's bootstrap key, encrypted using the server token as the PBKDF2 passphrase with AES256-GCM and HMAC-SHA1.
Copies of the CA certificates and keys are extracted to disk during K3s server startup.
Any server may generate leaf certificates for nodes as they join the cluster, and the Kubernetes Certificates API controllers may issue additional certificates at runtime.

To rotate CA certificates and keys, use the k3s certificate rotate-ca command.
The command performs integrity checks to confirm that the updated certificates and keys are usable.
If the updated data is acceptable, the datastore's encrypted bootstrap key is updated, and the new certificates and keys will be used the next time K3s starts.
If problems are encountered while validating the certificates and keys, an error is reported to the system log and the operation is cancelled without changes.

Using Custom CA Certificates Copy Section

Cautions Against CA Reuse

:::warning
It is not recommended to share Root or Intermediate CAs across multiple clusters, or to use an existing private CA as your cluster CA.
:::

When multiple clusters share a common root of trust, any client or server certificates issued by one cluster will also be trusted by all other clusters, as all certificates chain up to the same trust anchor - the common Root Certificate Authority.
This means that any user with a valid client certificate (or kubeconfig) for one cluster will also be able to authenticate to other clusters, and any RBAC that matches a client's user or group will also be applied in those other clusters.
Similarly, server certificates issued by one cluster will also be trusted in all other clusters, and by all other infrastructure or clients that trust that root CA.

Kubernetes does not support use of Certificate Revocation Lists.
If you ever need to revoke a certificate for any reason (for example, due to a compromised admin kubeconfig), you must perform a full replacement of the cluster Certificate Authority in order to invalidate the certificate's trust.

Using Custom CA Certificates

If CA certificates and keys are found the correct location during initial startup of the first server in the cluster, automatic generation of CA certificates will be bypassed.

An example script to pre-create the appropriate certificates and keys is available in the K3s repo at contrib/util/generate-custom-ca-certs.sh.
This script should be run prior to starting K3s for the first time, and will create a full set of leaf CA certificates signed by common Root and Intermediate CA certificates.
If you have an existing Root or Intermediate CA, this script can be used (or used as a starting point) to create the correct CA certificates to provision a K3s cluster with PKI rooted in an existing authority.

Custom Certificate Authority files must be placed in /var/lib/rancher/k3s/server/tls. The following files are required:

• server-ca.crt
• server-ca.key
• client-ca.crt
• client-ca.key
• request-header-ca.crt
• request-header-ca.key
  // note: etcd files are required even if embedded etcd is not in use.
• etcd/peer-ca.crt
• etcd/peer-ca.key
• etcd/server-ca.crt
• etcd/server-ca.key
  // note: This is the private key used to sign service-account tokens. It does not have a corresponding certificate.
• service.key

Custom CA Topology

Custom CA Certificates generated by the example script will have the following topology:

graph TD
  root("Root CA")
  intermediate("Intermediate CA")
  server-ca("Server CA")
  client-ca("Client CA")
  request-header-ca("API Aggregation CA")
  etcd-peer-ca("etcd Peer CA")
  etcd-server-ca("etcd Server CA")

  root-hash>"Join token CA hash"]

  kube-server-certs[["Kubernetes servers(control-plane and kubelet listeners)"]]
  kube-client-certs[["Kubernetes clients(apiserver and kubelet clients)"]]
  request-header-certs[["Kubernetes API aggregation(apiserver proxy client)"]]
  etcd-peer-certs[["etcd peer client/server(etcd replication)"]]
  etcd-server-certs[["etcd client/server certificates(Kubernetes <-> etcd)"]]

  root -.-|SHA256| root-hash
  root ---> intermediate
  intermediate --> server-ca ==> kube-server-certs
  intermediate --> client-ca ==> kube-client-certs
  intermediate --> request-header-ca ==> request-header-certs
  intermediate --> etcd-peer-ca ==> etcd-peer-certs
  intermediate --> etcd-server-ca ==> etcd-server-certs

Using the Example Script

:::info Important
If you want to sign the cluster CA certificates with an existing root CA using the example script, you must place the root and intermediate files in the target directory prior to running the script.
If the files do not exist, the script will create new root and intermediate CA certificates.
:::

If you want to use only an existing root CA certificate, provide the following files:

• root-ca.pem
• root-ca.key

If you want to use existing root and intermediate CA certificates, provide the following files:

• root-ca.pem
• intermediate-ca.pem
• intermediate-ca.key

The example script uses the provided CAs as the root for all cluster CA bundles - Server, Client, API Aggregation, and etcd Peer/Server.
For advanced use cases, such as using the provided CAs for only select bundles, or using different CAs for different bundles, the example script should be modified to meet the specific needs of your organization.

To use the example script to generate custom certs and keys before starting K3s, run the following commands:

# Create the target directory for cert generation.
mkdir -p /var/lib/rancher/k3s/server/tls

# Copy your root CA cert and intermediate CA cert+key into the correct location for the script.
# For the purposes of this example, we assume you have existing root and intermediate CA files in /etc/ssl.
# If you do not have an existing root and/or intermediate CA, the script will generate them for you.
cp /etc/ssl/certs/root-ca.pem /etc/ssl/certs/intermediate-ca.pem /etc/ssl/private/intermediate-ca.key /var/lib/rancher/k3s/server/tls

# Generate custom CA certs and keys.
curl -sL https://github.com/k3s-io/k3s/raw/main/contrib/util/generate-custom-ca-certs.sh | bash -

If the command completes successfully, you may install and/or start K3s for the first time.
If the script generated root and/or intermediate CA files, you should back up these files so that they can be reused if it is necessary to rotate the CA certificates at a later date.

Rotating Custom CA Certificates Copy Section

To rotate custom CA certificates, use the k3s certificate rotate-ca subcommand.
Updated files must be staged into a temporary directory, loaded into the datastore, and k3s must be restarted on all nodes to use the updated certificates.

:::warning
You must not overwrite the currently in-use data in /var/lib/rancher/k3s/server/tls.
Stage the updated certificates and keys into a separate directory.
:::

A cluster that has been started with custom CA certificates can renew or rotate the CA certificates and keys non-disruptively, as long as the same root CA is used.

If a new root CA is required, the rotation will be disruptive. The k3s certificate rotate-ca --force option must be used, all nodes that were joined with a secure token (including servers) will need to be reconfigured to use the new token value, and pods will need to be restarted to trust the new root CA.

Using the Example Script

The example generate-custom-ca-certs.sh script linked above can also be used to generate updated certs in a new temporary directory, by copying files into the correct location and setting the DATA_DIR environment variable.
To use the example script to generate updated certs and keys, run the following commands:

# Create a temporary directory for cert generation.
mkdir -p /opt/k3s/server/tls

# Copy your root CA cert and intermediate CA cert+key into the correct location for the script.
# Non-disruptive rotation requires the same root CA that was used to generate the original certificates.
# If the original files are still in the data directory, you can just run:
cp /var/lib/rancher/k3s/server/tls/root-ca.* /var/lib/rancher/k3s/server/tls/intermediate-ca.* /opt/k3s/server/tls

# Copy the current service-account signing key, so that existing service-account tokens are not invalidated.
cp /var/lib/rancher/k3s/server/tls/service.key /opt/k3s/server/tls

# Generate updated custom CA certs and keys.
curl -sL https://github.com/k3s-io/k3s/raw/main/contrib/util/generate-custom-ca-certs.sh | DATA_DIR=/opt/k3s bash -

# Load the updated CA certs and keys into the datastore.
k3s certificate rotate-ca --path=/opt/k3s/server

If the rotate-ca command returns an error, check the service log for errors.
If the command completes successfully, restart K3s on all nodes in the cluster - servers first, then agents.

If you used the --force option or changed the root CA, ensure that any nodes that were joined with a secure token are reconfigured to use the new token value, prior to being restarted.
The token may be stored in a .env file, systemd unit, or config.yaml, depending on how the node was configured during initial installation.

Rotating Self-Signed CA Certificates Copy Section

To rotate the K3s-generated self-signed CA certificates, use the k3s certificate rotate-ca subcommand.
Updated files must be staged into a temporary directory, loaded into the datastore, and k3s must be restarted on all nodes to use the updated certificates.

:::warning
You must not overwrite the currently in-use data in /var/lib/rancher/k3s/server/tls.
Stage the updated certificates and keys into a separate directory.
:::

If the cluster has been started with default self-signed CA certificates, rotation will be disruptive. All nodes that were joined with a secure token will need to be reconfigured to trust the new CA hash.
If the new CA certificates are not cross-signed by the old CA certificates, you will need to use the --force option to bypass integrity checks, and pods will need to be restarted to trust the new root CA.

Default CA Topology

The default self-signed CA certificates have the following topology:

graph TD
  server-ca("Server CA")
  client-ca("Client CA")
  request-header-ca("API Aggregation CA")
  etcd-peer-ca("etcd Peer CA")
  etcd-server-ca("etcd Server CA")

  root-hash>"Join token CA hash"]

  kube-server-certs[["Kubernetes servers(control-plane and kubelet listeners)"]]
  kube-client-certs[["Kubernetes clients(apiserver and kubelet clients)"]]
  request-header-certs[["Kubernetes API aggregation(apiserver proxy client)"]]
  etcd-peer-certs[["etcd peer client/server(etcd replication)"]]
  etcd-server-certs[["etcd client/server certificates(Kubernetes <-> etcd)"]]

  server-ca -.-|SHA256| root-hash
  server-ca ===> kube-server-certs
  client-ca ===> kube-client-certs
  request-header-ca ===> request-header-certs
  etcd-peer-ca ===> etcd-peer-certs
  etcd-server-ca ===> etcd-server-certs

When rotating the default self-signed CAs, a modified certificate topology with intermediate CAs and a new root CA cross-signed by the old CA can be used so that there is a continuous chain of trust between the old and new CAs:

/* Detailed source-code truncated for AI context efficiency. */

Using The Example Script

An example script to create updated CA certificates and keys cross-signed by the existing CAs is available in the K3s repo at contrib/util/rotate-default-ca-certs.sh.

To use the example script to generate updated self-signed certificates that are cross-signed by the existing CAs, run the following commands:

# Create updated CA certs and keys, cross-signed by the current CAs.
# This script will create a new temporary directory containing the updated certs, and output the new token values.
curl -sL https://github.com/k3s-io/k3s/raw/main/contrib/util/rotate-default-ca-certs.sh | bash -

# Load the updated certs into the datastore; see the script output for the updated token values.
k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca

If the rotate-ca command returns an error, check the service log for errors.
If the command completes successfully, restart K3s on all nodes in the cluster - servers first, then agents.

Ensure that any nodes that were joined with a secure token, including other server nodes, are reconfigured to use the new token value prior to being restarted.
The token may be stored in a .env file, systemd unit, or config.yaml, depending on how the node was configured during initial installation.

Service-Account Issuer Key Rotation Copy Chapter

The service-account issuer key is an RSA private key used to sign service-account tokens.
When rotating the service-account issuer key, at least one old key should be retained in the file so that existing service-account tokens are not invalidated.
It can be rotated independent of the cluster CAs by using the k3s certificate rotate-ca to install only an updated service.key file that includes both the new and old keys.

:::warning
You must not overwrite the currently in-use data in /var/lib/rancher/k3s/server/tls.
Stage the updated key into a separate directory.
:::

For example, to rotate only the service-account issuer key, run the following commands:

# Create a temporary directory for cert generation
mkdir -p /opt/k3s/server/tls

# Check OpenSSL version
openssl version | grep -qF 'OpenSSL 3' && OPENSSL_GENRSA_FLAGS=-traditional

# Generate a new key
openssl genrsa ${OPENSSL_GENRSA_FLAGS:-} -out /opt/k3s/server/tls/service.key 2048

# Append the existing key to avoid invalidating current tokens
cat /var/lib/rancher/k3s/server/tls/service.key >> /opt/k3s/server/tls/service.key

# Load the updated key into the datastore
k3s certificate rotate-ca --path=/opt/k3s/server

It is normal to see warnings for files that are not being updated. If the rotate-ca command returns an error, check the service log for errors.
If the command completes successfully, restart K3s on all servers in the cluster. It is not necessary to restart agents or restart any pods.

File: docs/cli/cli.md Copy Chapter

title: CLI Tools Copy Chapter

The K3s binary contains a number of additional tools the help you manage your cluster.

File: docs/cli/etcd-snapshot.md Copy Chapter

title: etcd-snapshot Copy Chapter

k3s etcd-snapshot

This page describes how to use the k3s etcd-snapshot CLI tool to manage etcd snapshots and how to restore from an etcd snapshot.

K3s etcd snapshots are stored on the node file system, and may optionally be uploaded to an S3 compatible object store for disaster recovery scenarios. Snapshots can be both automated on a reoccurring schedule, and taken manually on-demand. The k3s etcd-snapshot CLI tool offers a set of subcommands that can be used to create, delete, and manage snapshots.

For additional information on the etcd snapshot subcommands, run k3s etcd-snapshot --help.

Creating Snapshots Copy Chapter

Deleting Snapshots Copy Chapter

Scheduled snapshots are deleted automatically when the number of snapshots exceeds the configured retention count (5 by default). The oldest snapshots are removed first.

To manually delete scheduled snapshot(s) or on-demand snapshot(s), you can use the k3s etcd-snapshot delete command:

k3s etcd-snapshot delete <SNAPSHOT-NAME-1> <SNAPSHOT-NAME-2> ...

The prune subcommand removes snapshots that match the name prefix (on-demand by default) and exceed the configured retention count. It includes the flag --snapshot-retention to set the retention count. For scheduled snapshots, it overrides the default retention policy. On-demand snapshots have no retention policy and hence this flag is required.

Prune "on-demand" snapshots down to a smaller amount:

k3s etcd-snapshot prune --snapshot-retention  <NUM-OF-SNAPSHOTS-TO-RETAIN>

Prune "scheduled" snapshots down to a smaller amount:

k3s etcd-snapshot prune --name etcd-snapshot --etcd-snapshot-retention <NUM-OF-SNAPSHOTS-TO-RETAIN>

S3 Compatible Object Store Support Copy Chapter

K3s supports replicating etcd snapshots to and restoring etcd snapshots from S3-compatible object stores. S3 support is available for both on-demand and scheduled snapshots.

:::note
--etcd-snapshot-retention sets --etcd-s3-retention, if it is not explicitly set in the config.
:::

For example, this is how the creation and deletion of on-demand etcd snapshots in S3 would work:

$ k3s etcd-snapshot --s3 --s3-bucket=test-bucket --s3-access-key=test --s3-secret-key=secret save
INFO[0155] Snapshot on-demand-server-0-1753178523 saved. 
INFO[0155] Snapshot on-demand-server-0-1753178523 saved. 

$ k3s etcd-snapshot --s3 --s3-bucket=test-bucket --s3-access-key=test --s3-secret-key=secret ls
Name                              Location                                                                          Size    Created
on-demand-server-0-1753178523     s3://test-bucket/test-folder/on-demand-server-0-1753178523                        5062688 2025-07-22T10:02:03Z
on-demand-server-0-1753178523     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server-0-1753178523     5062688 2025-07-22T10:02:03Z

$ k3s etcd-snapshot --s3 --s3-bucket=test-bucket --s3-access-key=test --s3-secret-key=secret delete on-demand-server-0-1753178523
INFO[0000] Snapshot on-demand-server-0-1753178523 deleted.

$ k3s etcd-snapshot --s3 --s3-bucket=test-bucket --s3-access-key=test --s3-secret-key=secret ls
Name                              Location                                                                          Size    Created

S3 Configuration Secret Support Copy Section

K3s supports reading etcd S3 snapshot configuration from a Kubernetes Secret.
This may be preferred to hardcoding credentials in K3s CLI flags or config files for security reasons, or if credentials need to be rotated without restarting K3s.
To pass S3 snapshot configuration via a Secret, start K3s with --etcd-s3 and --etcd-s3-config-secret=<SECRET-NAME>.
The Secret does not need to exist when K3s is started, but it will be checked for every time a snapshot save/list/delete/prune operation is performed.

The S3 config Secret cannot be used when restoring a snapshot, as the apiserver is not available to provide the secret during a restore.
S3 configuration must be passed via the CLI when restoring a snapshot stored on S3.

:::note
Pass only the --etcd-s3 and --etcd-s3-config-secret flags to enable the Secret.
If any other S3 configuration flags are set, the Secret will be ignored.
:::

Keys in the Secret correspond to the --etcd-s3-* CLI flags listed above.
The etcd-s3-endpoint-ca key accepts a PEM-encoded CA bundle, or the etcd-s3-endpoint-ca-name key may be used to specify the name of a ConfigMap in the kube-system namespace containing one or more PEM-encoded CA bundles.

apiVersion: v1
kind: Secret
metadata:
  name: k3s-etcd-snapshot-s3-config
  namespace: kube-system
type: etcd.k3s.cattle.io/s3-config-secret
stringData:
  etcd-s3-endpoint: ""
  etcd-s3-endpoint-ca: ""
  etcd-s3-endpoint-ca-name: ""
  etcd-s3-skip-ssl-verify: "false"
  etcd-s3-access-key: "AWS_ACCESS_KEY_ID"
  etcd-s3-secret-key: "AWS_SECRET_ACCESS_KEY"
  etcd-s3-bucket: "bucket"
  etcd-s3-folder: "folder"
  etcd-s3-region: "us-east-1"
  etcd-s3-insecure: "false"
  etcd-s3-timeout: "5m"
  etcd-s3-proxy: ""

Restoring Snapshots Copy Chapter

K3s runs through several steps when restoring a snapshot:

1. If the snapshot is stored on S3, the file is downloaded into the snapshot directory.
2. If the snapshot is compressed, it is decompressed.
3. If present, the current etcd database files are moved to ${data-dir}/server/db/etcd-old-$TIMESTAMP/.
4. The snapshot's contents are extracted out to disk, and the checksum is verified.
5. Etcd is started, and all etcd cluster members except the current node are removed from the cluster.
6. CA Certificates and other confidential data are extracted from the datastore and written to disk, for later use.
7. The restore is complete, and K3s can be restarted and used normally on the server where the restore was performed.
8. (optional) Agents and control-plane servers can be started normally.
9. (optional) Etcd servers can be restarted to rejoin to the cluster after removing old database files.

When restoring a snapshot, you don't need to use the same K3s version that created it; a higher minor version is also acceptable.

Snapshot Restore Steps Copy Section

Select the tab below that matches your cluster configuration.

Restoring To New Hosts

It is possible to restore an etcd snapshot to a different host than it was taken on. When doing so, you must pass the server token that was originally used when taking the snapshot, as it is used to decrypt the bootstrap data inside the snapshot. The process is the same as above but changing step 2 by:

1. In the node that took the snapshot save the value of: /var/lib/rancher/k3s/server/token. This is `` in step 3.

2. Copy the snapshot to the new node. The path in the node is `` in step 3

3. Initiate the restore from snapshot on the first server node with the following commands:

k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=
  --token=

The token value can also be set in the K3s config file.

:::warning

1. Node resources are also included in the etcd snapshot. If restoring to a new set of nodes, you will need to manually delete any old nodes that are no longer present in the cluster.
2. If there is a token set in the K3s config file, make sure it is the same as the ``, otherwise k3s will fail to start.
   :::

Security Copy Section

Snapshots contain a complete copy of the etcd datastore, along with cluster certificate authority (CA) certificates and private keys.
If secrets encryption is enabled, snapshots also include the secrets encryption configuration and keys.
This information is encrypted with an AES-256 key derived from the server's join token using PBKDF2.

Administrators should be aware of the following concerns when backing up or restoring snapshots:

• Any attacker who can convince you to restore a snapshot under their control can gain complete control of the cluster and nodes via multiple attack vectors - it is now their cluster, not yours.
• Any attacker who possesses a copy of the etcd snapshot file can extract all unencrypted Kubernetes resources.
• Any attacker who possesses a copy of the etcd snapshot file AND the server join token can extract encrypted resources, along with private keys to the cluster CAs.

Snapshots stored locally on the node are protected via normal Unix filesystem permissions and access controls.
It is recommended that administrators safely store snapshots to a S3-compatible service that supports bucket access controls, server-side encryption, and retention controls.
Validating the integrity and chain of custody of datastore snapshots is the responsibility of the administrator.

ETCDSnapshotFile Custom Resources Copy Chapter

Snapshots can be viewed remotely using any Kubernetes client by listing or describing cluster-scoped ETCDSnapshotFile resources.
Unlike the k3s etcd-snapshot list command, which only shows snapshots visible to that node, ETCDSnapshotFile resources track all snapshots present on cluster members.

$ kubectl get etcdsnapshotfile
NAME                                             SNAPSHOTNAME                        NODE           LOCATION                                                                            SIZE      CREATIONTIME
local-on-demand-k3s-server-1-1730308816-3e9290   on-demand-k3s-server-1-1730308816   k3s-server-1   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-k3s-server-1-1730308816   2891808   2024-10-30T17:20:16Z
s3-on-demand-k3s-server-1-1730308816-79b15c      on-demand-k3s-server-1-1730308816   s3             s3://etcd/k3s-test/on-demand-k3s-server-1-1730308816                                2891808   2024-10-30T17:20:16Z

$ kubectl describe etcdsnapshotfile s3-on-demand-k3s-server-1-1730308816-79b15c
Name:         s3-on-demand-k3s-server-1-1730308816-79b15c
Namespace:
Labels:       etcd.k3s.cattle.io/snapshot-storage-node=s3
Annotations:  etcd.k3s.cattle.io/snapshot-token-hash: b4b83cda3099
API Version:  k3s.cattle.io/v1
Kind:         ETCDSnapshotFile
Metadata:
  Creation Timestamp:  2024-10-30T17:20:16Z
  Finalizers:
    wrangler.cattle.io/managed-etcd-snapshots-controller
  Generation:        1
  Resource Version:  790
  UID:               bec9a51c-dbbe-4746-922e-a5136bef53fc
Spec:
  Location:   s3://etcd/k3s-test/on-demand-k3s-server-1-1730308816
  Node Name:  s3
  s3:
    Bucket:           etcd
    Endpoint:         s3.example.com
    Prefix:           k3s-test
    Region:           us-east-1
    Skip SSL Verify:  true
  Snapshot Name:      on-demand-k3s-server-1-1730308816
Status:
  Creation Time:  2024-10-30T17:20:16Z
  Ready To Use:   true
  Size:           2891808
Events:
  Type    Reason               Age   From            Message
  ----    ------               ----  ----            -------
  Normal  ETCDSnapshotCreated  113s  k3s-supervisor  Snapshot on-demand-k3s-server-1-1730308816 saved on S3

File: docs/cli/secrets-encrypt.md Copy Chapter

title: secrets-encrypt Copy Chapter

k3s secrets-encrypt

K3s supports enabling secrets encryption at rest. For more information, see Secrets Encryption.

Secrets Encryption Tool Copy Chapter

K3s contains a CLI tool secrets-encrypt, which enables automatic control over the following:

• Disabling/Enabling secrets encryption
• Adding new encryption keys
• Rotating and deleting encryption keys
• Reencrypting secrets

:::warning
Failure to follow proper procedure for rotating encryption keys can leave your cluster permanently corrupted. Proceed with caution.
:::

Encryption Key Rotation Copy Section

Enable Secrets Encryption on an Existing Cluster Copy Section

:::info Version Gate
Available as of the March 2026 releases: v1.33.10+k3s1, v1.34.6+k3s1, v1.35.3+k3s1
:::

Use this procedure when the cluster was started without --secrets-encryption and you want to enable secrets encryption at a later time.

Legacy Encryption Key Rotation Copy Section

:::tip New Procedure
If using K3s versions v1.30+, we recommend using the Encryption Key Rotation instead.
:::

Secrets Encryption Disable/Re-enable Copy Section

Secrets Encryption Status Copy Section

The secrets-encrypt tool includes a status command that displays information about the current status of secrets encryption on the node.

An example of the command on a single-server node:

$ k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: start
Server Encryption Hashes: All hashes match

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey

Another example on HA cluster, after rotating the keys, but before restarting the servers:

$ k3s secrets-encrypt status
Encryption Status: Enabled
Current Rotation Stage: rotate
Server Encryption Hashes: hash does not match between node-1 and node-2

Active  Key Type  Name
------  --------  ----
 *      AES-CBC   aescbckey-2021-12-10T22:54:38Z
        AES-CBC   aescbckey

Details on each section are as follows:

• Encryption Status: Displayed whether secrets encryption is disabled or enabled on the node
• Current Rotation Stage: Indicates the current rotation stage on the node.
  Stages are: start, prepare, rotate, reencrypt_request, reencrypt_active, reencrypt_finished
• Server Encryption Hashes: Useful for HA clusters, this indicates whether all servers are on the same stage with their local files. This can be used to identify whether a restart of servers is required before proceeding to the next stage. In the HA example above, node-1 and node-2 have different hashes, indicating that they currently do not have the same encryption configuration. Restarting the servers will sync up their configuration.
• Key Table: Summarizes information about the secrets encryption keys found on the node.
  • Active: The "*" indicates which, if any, of the keys are currently used for secrets encryption. An active key is used by Kubernetes to encrypt any new secrets.
  • Key Type: Keys using this tool support AES-CBC and secretbox types. See more info here.
  • Name: Name of the encryption key.

File: docs/cli/server.md Copy Chapter

title: server Copy Chapter

k3s server

In this section, you'll learn how to configure the K3s server.

Note that servers also run an agent, so all of the configuration options listed in the k3s agent documentation are also supported on servers.

Options are documented on this page as CLI flags, but can also be passed as configuration file options. See the Configuration File documentation for more information on using YAML configuration files.

Critical Configuration Values Copy Chapter

The following options must be set to the same value on all servers in the cluster. Failure to do so will cause new servers to fail to join the cluster when using embedded etcd, or incorrect operation of the cluster when using an external datastore.

• --agent-token
• --cluster-cidr
• --cluster-dns
• --cluster-domain
• --disable-cloud-controller
• --disable-helm-controller
• --disable-network-policy
• --disable=servicelb note: other packaged components may be disabled on a per-server basis
• --egress-selector-mode
• --embedded-registry
• --flannel-backend
• --flannel-external-ip
• --flannel-ipv6-masq
• --secrets-encryption
• --secrets-encryption-provider
• --service-cidr

Commonly Used Options Copy Chapter

Database Copy Section

S3 etcd Snapshot Storage Copy Section

Cluster Options Copy Section

Admin Kubeconfig Options Copy Section

Advanced Options Copy Chapter

Logging Copy Section

Listeners Copy Section

Data Copy Section

Secrets Encryption Copy Section

Networking Copy Section

Storage Class Copy Section

Kubernetes Components Copy Section

Customized Flags for Kubernetes Processes Copy Section

Experimental Options Copy Section

Deprecated Options Copy Section

K3s Server CLI Help Copy Chapter

If an option appears in brackets below, for example [$K3S_TOKEN], it means that the option can be passed in as an environment variable of that name.

/* Detailed source-code truncated for AI context efficiency. */

File: docs/cli/token.md Copy Chapter

title: token Copy Chapter

k3s token

K3s uses tokens to secure the node join process and to encrypt confidential information that is persisted to the datastore. Tokens authenticate the cluster to the joining node, and the node to the cluster.

Token Format Copy Chapter

K3s tokens can be specified in either secure or short format. The secure format is preferred, as it enables the client to authenticate the identity of the cluster it is joining, before sending credentials.

Secure Copy Section

The secure token format (occasionally referred to as a "full" token) contains the following parts:

<cluster CA hash>::<credentials>

• prefix: a fixed K10 prefix that identifies the token format
• cluster CA hash: The hash of the cluster's server CA certificate, used to authenticate the server to the joining node.
  • For self-signed CA certificates, this is the SHA256 sum of the PEM-formatted certificate, as stored on disk.
  • For custom CA certificates, this is the SHA256 sum of the DER encoding of the root certificate; commonly known as the certificate fingerprint.
• credentials: The username and password, or bearer token, used to authenticate the joining node to the cluster.

TLS Bootstrapping

When a secure token is specified, the joining node performs the following steps to validate the identity of the server it has connected to, before transmitting credentials:

1. With TLS verification disabled, download the CA bundle from /cacerts on the server it is joining.
2. Calculate the SHA256 hash of the CA certificate, as described above.
3. Compare the calculated SHA256 hash to the hash from the token.
4. If the hash matches, validate that the certificate presented by the server can be validated by the server's CA bundle.
5. If the server certificate is valid, present credentials to join the cluster using either basic or bearer token authentication, depending on the token type.

Short Copy Section

The short token format includes only the password or bearer token used to authenticate the joining node to the cluster.

If a short token is used, the joining node implicitly trusts the CA bundle presented by the server; steps 2-4 in the TLS Bootstrapping process are skipped. The initial connection may be vulnerable to man-in-the-middle attack.

Token Types Copy Chapter

K3s supports three types of tokens. Only the server token is available by default; additional token types must be configured or created by the administrator.

Server Copy Section

If no token is provided when starting the first server in the cluster, one is created with a random password. The server token is always written to /var/lib/rancher/k3s/server/token, in secure format.

The server token can be used to join both server and agent nodes to the cluster. Anyone with access to the server token essentially has full administrator access to the cluster. This token should be guarded carefully.

The server token is also used as the PBKDF2 passphrase to encrypt confidential information that is persisted to the datastore known as bootstrap data. Bootstrap data is essential to set up new server nodes or restore from a snapshot. For this reason, the token must be backed up alongside the cluster datastore itself.

:::warning
Unless custom CA certificates are in use, only the short (password-only) token format can be used when starting the first server in the cluster. This is because the cluster CA hash cannot be known until after the server has generated the self-signed cluster CA certificates.
:::

For more information on using custom CA certificates, see the k3s certificate documentation.
For more information on backing up your cluster, see the Backup and Restore documentation.

Agent Copy Section

By default, the agent token is the same as the server token. The agent token can be set before or after the cluster has been started, by changing the CLI option or environment variable on all servers in the cluster. The agent token is similar to the server token in that is it statically configured, and does not expire.

The agent token is written to /var/lib/rancher/k3s/server/agent-token, in secure format. If no agent token is specified, this file is a link to the server token.

Bootstrap Copy Section

K3s supports dynamically generated, automatically expiring agent bootstrap tokens.

k3s token Copy Chapter

The k3s token CLI tool handles:

• The life cycle of bootstrap tokens, using the same generation and validation code as kubeadm token bootstrap tokens. Note that both CLIs are similar.
• The rotation of the server token

NAME:
   k3s token - Manage tokens

USAGE:
   k3s token command [command options] [arguments...]

COMMANDS:
   create    Create bootstrap tokens on the server
   delete    Delete bootstrap tokens on the server
   generate  Generate and print a bootstrap token, but do not create it on the server
   list      List bootstrap tokens on the server
   rotate    Rotate original server token with a new server token

OPTIONS:
   --help, -h  show help

`k3s token create [token]`

Create a new token. The [token] is the actual token to write, as generated by k3s token generate. If no token is given, a random one will be generated.

A token in secure format, including the cluster CA hash, will be written to stdout. The output of this command should be saved, as the secret portion of the token cannot be shown again.

`k3s token delete`

Delete one or more tokens. The full token can be provided, or just the token ID.

`k3s token generate`

Generate a randomly-generated bootstrap token.

You don't have to use this command in order to generate a token. You can do so yourself as long as it is in the format [a-z0-9]{6}.[a-z0-9]{16}, where the first portion is the token ID, and the second portion is the secret.

`k3s token list`

List bootstrap tokens, showing their ID, description, and remaining time-to-live.

`k3s token rotate`

Rotate original server token with a new server token. After running this command, all servers and any agents that originally joined with the old token must be restarted with the new token.

If you do not specify a new token, one will be generated for you.

:::warning
Snapshots taken before the rotation will require the old server token when restoring the cluster
:::

File: docs/datastore/backup-restore.md Copy Chapter

title: Backup and Restore Copy Chapter

The way K3s is backed up and restored depends on which type of datastore is used.

:::warning
In addition to backing up the datastore itself, you must also back up the server token file at /var/lib/rancher/k3s/server/token.
You must restore this file, or pass its value into the --token option, when restoring from backup.
If you do not use the same token value when restoring, the snapshot will be unusable, as the token is used to encrypt confidential data within the datastore itself.
:::

Backup and Restore with SQLite Copy Chapter

No special commands are required to back up or restore the SQLite datastore.

• To back up the SQLite datastore, take a copy of /var/lib/rancher/k3s/server/db/.
• To restore the SQLite datastore, restore the contents of /var/lib/rancher/k3s/server/db (and the token, as discussed above).

Backup and Restore with External Datastore Copy Chapter

When an external datastore is used, backup and restore operations are handled outside of K3s. The database administrator will need to back up the external database, or restore it from a snapshot or dump.

We recommend configuring the database to take recurring snapshots.

For details on taking database snapshots and restoring your database from them, refer to the official database documentation:

• Official MySQL documentation
• Official PostgreSQL documentation
• Official etcd documentation

Backup and Restore with Embedded etcd Datastore Copy Chapter

See the k3s etcd-snapshot command documentation for information on performing backup and restore operations on the embedded etcd datastore.

File: docs/datastore/cluster-loadbalancer.md Copy Chapter

title: Cluster Load Balancer Copy Chapter

This section describes how to install an external load balancer in front of a High Availability (HA) K3s cluster's server nodes. Two examples are provided: Nginx and HAProxy.

:::tip
External load-balancers should not be confused with the embedded ServiceLB, which is an embedded controller that allows for use of Kubernetes LoadBalancer Services without deploying a third-party load-balancer controller. For more details, see Service Load Balancer.

External load-balancers can be used to provide a fixed registration address for registering nodes, or for external access to the Kubernetes API Server. For exposing LoadBalancer Services, external load-balancers can be used alongside or instead of ServiceLB, but in most cases, replacement load-balancer controllers such as MetalLB or Kube-VIP are a better choice.
:::

Prerequisites Copy Chapter

All nodes in this example are running Ubuntu 20.04.

For both examples, assume that a HA K3s cluster with embedded etcd has been installed on 3 nodes.

Each k3s server is configured with:

# /etc/rancher/k3s/config.yaml
token: lb-cluster-gd
tls-san: 10.10.10.100

The nodes have hostnames and IPs of:

• server-1: 10.10.10.50
• server-2: 10.10.10.51
• server-3: 10.10.10.52

Two additional nodes for load balancing are configured with hostnames and IPs of:

• lb-1: 10.10.10.98
• lb-2: 10.10.10.99

Three additional nodes exist with hostnames and IPs of:

• agent-1: 10.10.10.101
• agent-2: 10.10.10.102
• agent-3: 10.10.10.103

Setup Load Balancer Copy Chapter

sudo apt-get install haproxy keepalived

frontend k3s-frontend
    bind *:6443
    mode tcp
    option tcplog
    default_backend k3s-backend

backend k3s-backend
    mode tcp
    option tcp-check
    balance roundrobin
    default-server inter 10s downinter 5s
    server server-1 10.10.10.50:6443 check
    server server-2 10.10.10.51:6443 check
    server server-3 10.10.10.52:6443 check

global_defs {
  enable_script_security
  script_user root
}

vrrp_script chk_haproxy {
    script 'killall -0 haproxy' # faster than pidof
    interval 2
}

vrrp_instance haproxy-vip {
    interface eth1
    state <STATE> # MASTER on lb-1, BACKUP on lb-2
    priority  # 200 on lb-1, 100 on lb-2

    virtual_router_id 51

    virtual_ipaddress {
        10.10.10.100/24
    }

    track_script {
        chk_haproxy
    }
}

systemctl restart haproxy
systemctl restart keepalived

curl -sfL https://get.k3s.io | K3S_TOKEN=lb-cluster-gd sh -s - agent --server https://10.10.10.100:6443

root@server-1 $ k3s kubectl get nodes -A
NAME       STATUS   ROLES                       AGE     VERSION
agent-1    Ready    <none>                      32s     v1.27.3+k3s1
agent-2    Ready    <none>                      20s     v1.27.3+k3s1
agent-3    Ready    <none>                      9s      v1.27.3+k3s1
server-1   Ready    control-plane,etcd,master   4m22s   v1.27.3+k3s1
server-2   Ready    control-plane,etcd,master   3m58s   v1.27.3+k3s1
server-3   Ready    control-plane,etcd,master   3m12s   v1.27.3+k3s1

events {}

stream {
  upstream k3s_servers {
    server 10.10.10.50:6443;
    server 10.10.10.51:6443;
    server 10.10.10.52:6443;
  }

  server {
    listen 6443;
    proxy_pass k3s_servers;
  }
}

docker run -d --restart unless-stopped \
    -v ${PWD}/nginx.conf:/etc/nginx/nginx.conf \
    -p 6443:6443 \
    nginx:stable

cp nginx.conf /etc/nginx/nginx.conf
systemctl start nginx

curl -sfL https://get.k3s.io | K3S_TOKEN=lb-cluster-gd sh -s - agent --server https://10.10.10.98:6443

root@server1 $ k3s kubectl get nodes -A
NAME       STATUS   ROLES                       AGE     VERSION
agent-1    Ready    <none>                      30s     v1.27.3+k3s1
agent-2    Ready    <none>                      22s     v1.27.3+k3s1
agent-3    Ready    <none>                      13s     v1.27.3+k3s1
server-1   Ready    control-plane,etcd,master   4m49s   v1.27.3+k3s1
server-2   Ready    control-plane,etcd,master   3m58s   v1.27.3+k3s1
server-3   Ready    control-plane,etcd,master   3m16s   v1.27.3+k3s1

curl -fsSL https://kube-vip.io/manifests/rbac.yaml -o /var/lib/rancher/k3s/server/manifests/kube-vip-rbac.yaml

kubectl apply -f https://kube-vip.io/manifests/rbac.yaml

/* Detailed source-code truncated for AI context efficiency. */

# check pods
kubectl -n kube-system get pods -l app.kubernetes.io/name=kube-vip-ds

# on a control-plane host, confirm the VIP is in the ARP/neighbor table
ip neigh show | grep 10.10.10.100

# Stop K3s service
systemctl stop k3s

# Rotate server certificates to include the configured tls-san/VIP
k3s certificate rotate

# Start K3s service
systemctl start k3s