## 1. Project Overview & Quickstart (k3s-io/token) # token Open-source repository k3s-io/token ### Repository Details - **Repository:** [k3s-io/token](https://github.com/k3s-io/token) - **Primary Language:** Code *Note: High-volume repository documentation is actively indexed and synchronized by YakaAI.* ## 2. Official Technical Reference & Guides (k3s-io/docs) ## File: README.md # K3s Website and Docs This repo contains the content of the K3s documentation website found at https://docs.k3s.io. Please open an issue if you have suggestions for new content or edits. We also gladly accept community PRs. The website is built using [Docusaurus 2](https://docusaurus.io/), a modern static website generator. ### Installation ``` $ yarn ``` ### Local Development ``` $ yarn start ``` This command starts a local development server and opens up a browser window. Most changes are reflected live without having to restart the server. You can also use Docker to launch the website without needing to install and configure yarn: ``` $ docker run --rm -d --name k3s-docs -e HOME=$PWD -v $PWD:$PWD -w $PWD -u $(id -u) -p 3000:3000 node:22 sh -c 'corepack yarn install --immutable && corepack yarn start -h 0.0.0.0' ``` ### Build ``` $ yarn build ``` This command generates static content into the `build` directory and can be served using any static contents hosting service. ### Deployment An automated GH action will deploy the website to GitHub Pages once a PR has been merged to `main`. --- ## File: docs/cli/agent.md --- title: agent --- # k3s agent In this section, you'll learn how to configure the K3s agent. Note that servers also run an agent, so all flags listed on this page are also valid for use on servers. Options are documented on this page as CLI flags, but can also be passed as configuration file options. See the [Configuration File](../installation/configuration.md#configuration-file) documentation for more information on using YAML configuration files. ### Logging | Flag | Default | Description | | ----------------------- | ------- | -------------------------------------------------------------------- | | `-v` value | 0 | Number for the log level verbosity | | `--vmodule` value | N/A | Comma-separated list of FILE_PATTERN=LOG_LEVEL settings for file-filtered logging | | `--log value, -l` value | N/A | Log to file | | `--alsologtostderr` | N/A | Log to standard error as well as file (if set) | ### Cluster Options | Flag | Environment Variable | Description | | -------------------------- | -------------------- | ------------------------------------ | | `--token value, -t` value | `K3S_TOKEN` | Token to use for authentication | | `--token-file` value | `K3S_TOKEN_FILE` | Token file to use for authentication | | `--server value, -s` value | `K3S_URL` | Server to connect to | ### Listener | Flag | Default | Description | | --- | --- | --- | | `--bind-address` | 0.0.0.0 | k3s bind address | ### Data | Flag | Default | Description | | ---------------------------- | ---------------------- | -------------------- | | `--data-dir value, -d` value | "/var/lib/rancher/k3s" | Folder to hold state | ### Node | Flag | Environment Variable | Description | | --------------------------- | -------------------- | --------------------------------------------------------------------------------------------- | | `--node-name` value | `K3S_NODE_NAME` | Node name | | `--with-node-id` | N/A | Append id to node name | | `--node-label` value | N/A | Registering and starting kubelet with set of labels | | `--node-taint` value | N/A | Registering kubelet with set of taints | | `--protect-kernel-defaults` | N/A | Kernel tuning behavior. If set, error if kernel tunables are different from kubelet defaults. | | `--selinux` | `K3S_SELINUX` | Enable SELinux in containerd | | `--lb-server-port` value | `K3S_LB_SERVER_PORT` | Local port for supervisor client load-balancer. If the supervisor and apiserver are not colocated an additional port 1 less than this port will also be used for the apiserver client load-balancer. (default: 6444) | ### Runtime | Flag | Default | Description | | ------------------------------------ | ---------------------------------- | ------------------------------------------------------------------ | | `--container-runtime-endpoint` | N/A | Disable embedded containerd and use the CRI socket at the given path; when used with --docker this sets the docker socket path | | `--default-runtime` | N/A | Set the default runtime in containerd | | `--image-service-endpoint` | N/A | Disable embedded containerd image service and use remote image service socket at the given path. If not specified, defaults to --container-runtime-endpoint. | | `--pause-image` value | "docker.io/rancher/pause:3.1" | Customized pause image for containerd or docker sandbox | | `--private-registry` value | "/etc/rancher/k3s/registries.yaml" | Private registry configuration file | ### Networking | Flag | Environment Variable | Description | | --------------------------- | -------------------- | ----------------------------------------- | | `--node-ip value, -i` value | N/A | IP address to advertise for node | | `--node-external-ip` value | N/A | External IP address to advertise for node | | `--node-internal-dns` | N/A | internal DNS addresses to advertise for node | | `--node-external-dns` | N/A | external DNS addresses to advertise for node | | `--resolv-conf` value | `K3S_RESOLV_CONF` | Kubelet resolv.conf file | | `--flannel-iface` value | N/A | Override default flannel interface | | `--flannel-conf` value | N/A | Override default flannel config file | | `--flannel-cni-conf` value | N/A | Override default flannel cni config file | ### Customized Flags | Flag | Description | | ------------------------ | -------------------------------------- | | `--kubelet-arg` value | Customized flag for kubelet process | | `--kube-proxy-arg` value | Customized flag for kube-proxy process | ### Experimental | Flag | Description | | ------------ | ------------------------------------- | | `--rootless` | Run rootless | | `--docker` | Use cri-dockerd instead of containerd | | `--enable-pprof` | Enable pprof endpoint on supervisor port | | `--prefer-bundled-bin` | Prefer bundled userspace binaries over host binaries | | `--disable-default-registry-endpoint` | See "[Default Endpoint Fallback](../installation/private-registry.md#default-endpoint-fallback)" | `--vpn-auth` | See "[Integration with the Tailscale VPN provider](../networking/distributed-multicloud.md#integration-with-the-tailscale-vpn-provider-experimental)" | | `--vpn-auth-file` | See "[Integration with the Tailscale VPN provider](../networking/distributed-multicloud.md#integration-with-the-tailscale-vpn-provider-experimental)" | ### Deprecated | Flag | Environment Variable | Description | | ------------------------ | -------------------- | ---------------------------- | | `--no-flannel` | N/A | Use `--flannel-backend=none` | | `--cluster-secret` value | `K3S_CLUSTER_SECRET` | Use `--token` | ### Node Labels and Taints for Agents K3s agents can be configured with the options `--node-label` and `--node-taint` which adds a label and taint to the kubelet. The two options only add labels and/or taints at registration time, so they can only be added once and not changed after that again by running K3s commands. Below is an example showing how to add labels and a taint: ```bash --node-label foo=bar \ --node-label hello=world \ --node-taint key1=value1:NoExecute ``` If you want to change node labels and taints after node registration you should use `kubectl`. Refer to the official Kubernetes documentation for details on how to add [taints](https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/) and [node labels.](https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/#add-a-label-to-a-node) ### K3s Agent CLI Help > If an option appears in brackets below, for example `[$K3S_URL]`, it means that the option can be passed in as an environment variable of that name. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ## File: docs/cli/certificate.md --- title: certificate --- # k3s certificate ## Client and Server Certificates K3s client and server certificates are valid for 365 days from their date of issuance. Any certificates that are expired or within 120 days of expiring are automatically renewed every time K3s starts. This renewal reuses the existing keys, and extends the lifetime of the existing certificates. If you want to generate new certificates and keys, instead of extending the validity of the existing certificates, use the [`rotate`](#rotating-client-and-server-certificates) subcommand documented below. When a certificate is within 120 days of expiring a Kubernetes Warning Event with `reason: CertificateExpirationWarning` is created, with a relation to the Node using the certificate. Prior to the May 2025 releases (v1.33.1+k3s1, v1.32.5+k3s1, v1.31.9+k3s1, v1.30.13+k3s1), alerts and rotation were triggered at 90 days, instead of 120 days. ### Checking expiration dates :::info Version Gate Output format is configurable as of the January 2025 releases: v1.32.0+k3s1, v1.31.5+k3s1, v1.30.9+k3s1, v1.30.13+k3s1 A new format with more information is available as of the May 2025 releases: v1.33.1+k3s1, v1.32.5+k3s1, v1.31.9+k3s1, v1.29.13+k3s1 ::: To check the node certificates and their expiration date use the `k3s certificate check --output table`: ```bash FILENAME SUBJECT USAGES EXPIRES RESIDUAL TIME STATUS -------- ------- ------ ------- ------------- ------ client-kube-proxy.crt system:kube-proxy ClientAuth Jun 09, 2026 10:17 UTC 1 year OK client-kube-proxy.crt k3s-client-ca@1749464211 CertSign Jun 07, 2035 10:16 UTC 10 years OK client-kubelet.crt system:node:ip-10-11-0-14 ClientAuth Jun 09, 2026 10:17 UTC 1 year OK client-kubelet.crt k3s-client-ca@1749464211 CertSign Jun 07, 2035 10:16 UTC 10 years OK serving-kubelet.crt ip-10-11-0-14 ServerAuth Jun 09, 2026 10:17 UTC 1 year OK serving-kubelet.crt k3s-server-ca@1749464211 CertSign Jun 07, 2035 10:16 UTC 10 years OK client-k3s-controller.crt system:k3s-controller ClientAuth Jun 09, 2026 10:17 UTC 1 year OK client-k3s-controller.crt k3s-client-ca@1749464211 CertSign Jun 07, 2035 10:16 UTC 10 years OK ``` :::info SAME CERTIFICATE TWICE Each certificate file (FILENAME column) contains at least two certificates - the leaf (or end entity) client/server certificate, any intermediate Certificate Authority certificates, and the root Certificate Authority certificate. ::: In case of unexpected output, please ensure that you are specifying the correct data directory with the `--data-dir` flag (if using a custom data directory), or use the `--debug` flag to get additional output from the check process. ### Rotating Client and Server Certificates To rotate client and server certificates manually, use the `k3s certificate rotate` subcommand: ```bash # Stop K3s systemctl stop k3s # Rotate certificates k3s certificate rotate # Start K3s systemctl start k3s ``` Individual or lists of certificates can be rotated by specifying the certificate name: ```bash k3s certificate rotate --service , ``` The following certificates can be rotated: `admin`, `api-server`, `controller-manager`, `scheduler`, `k3s-controller`, `k3s-server`, `cloud-controller`, `etcd`, `auth-proxy`, `kubelet`, `kube-proxy`. ## Certificate Authority (CA) Certificates Kubernetes requires a number of CA certificates for proper operation. For more information on how Kubernetes uses CA certificates, see the Kubernetes [PKI Certificates and Requirements](https://kubernetes.io/docs/setup/best-practices/certificates/#all-certificates) documentation. By default, K3s generates self-signed CA certificates during startup of the first server node. These CA certificates are valid for 10 years from date of issuance, and are not automatically renewed. The authoritative CA certificates and keys are stored within the datastore's bootstrap key, encrypted using the [server token](token.md#server) as the PBKDF2 passphrase with AES256-GCM and HMAC-SHA1. Copies of the CA certificates and keys are extracted to disk during K3s server startup. Any server may generate leaf certificates for nodes as they join the cluster, and the Kubernetes [Certificates API](https://kubernetes.io/docs/reference/access-authn-authz/certificate-signing-requests/) controllers may issue additional certificates at runtime. To rotate CA certificates and keys, use the `k3s certificate rotate-ca` command. The command performs integrity checks to confirm that the updated certificates and keys are usable. If the updated data is acceptable, the datastore's encrypted bootstrap key is updated, and the new certificates and keys will be used the next time K3s starts. If problems are encountered while validating the certificates and keys, an error is reported to the system log and the operation is cancelled without changes. ### Using Custom CA Certificates #### Cautions Against CA Reuse :::warning It is not recommended to share Root or Intermediate CAs across multiple clusters, or to use an existing private CA as your cluster CA. ::: When multiple clusters share a common root of trust, any client or server certificates issued by one cluster will also be trusted by all other clusters, as all certificates chain up to the same trust anchor - the common Root Certificate Authority. This means that any user with a valid client certificate (or kubeconfig) for one cluster will also be able to authenticate to other clusters, and any RBAC that matches a client's user or group will also be applied in those other clusters. Similarly, server certificates issued by one cluster will also be trusted in all other clusters, and by all other infrastructure or clients that trust that root CA. Kubernetes does not support use of Certificate Revocation Lists. If you ever need to revoke a certificate for any reason (for example, due to a compromised admin kubeconfig), you must perform a full replacement of the cluster Certificate Authority in order to invalidate the certificate's trust. #### Using Custom CA Certificates If CA certificates and keys are found the correct location during initial startup of the first server in the cluster, automatic generation of CA certificates will be bypassed. An example script to pre-create the appropriate certificates and keys is available [in the K3s repo at `contrib/util/generate-custom-ca-certs.sh`](https://github.com/k3s-io/k3s/blob/main/contrib/util/generate-custom-ca-certs.sh). This script should be run prior to starting K3s for the first time, and will create a full set of leaf CA certificates signed by common Root and Intermediate CA certificates. If you have an existing Root or Intermediate CA, this script can be used (or used as a starting point) to create the correct CA certificates to provision a K3s cluster with PKI rooted in an existing authority. Custom Certificate Authority files must be placed in `/var/lib/rancher/k3s/server/tls`. The following files are required: * `server-ca.crt` * `server-ca.key` * `client-ca.crt` * `client-ca.key` * `request-header-ca.crt` * `request-header-ca.key` *// note: etcd files are required even if embedded etcd is not in use.* * `etcd/peer-ca.crt` * `etcd/peer-ca.key` * `etcd/server-ca.crt` * `etcd/server-ca.key` *// note: This is the private key used to sign service-account tokens. It does not have a corresponding certificate.* * `service.key` #### Custom CA Topology Custom CA Certificates generated by the example script will have the following topology: ```mermaid graph TD root("Root CA") intermediate("Intermediate CA") server-ca("Server CA") client-ca("Client CA") request-header-ca("API Aggregation CA") etcd-peer-ca("etcd Peer CA") etcd-server-ca("etcd Server CA") root-hash>"Join token CA hash"] kube-server-certs[["Kubernetes servers(control-plane and kubelet listeners)"]] kube-client-certs[["Kubernetes clients(apiserver and kubelet clients)"]] request-header-certs[["Kubernetes API aggregation(apiserver proxy client)"]] etcd-peer-certs[["etcd peer client/server(etcd replication)"]] etcd-server-certs[["etcd client/server certificates(Kubernetes <-> etcd)"]] root -.-|SHA256| root-hash root ---> intermediate intermediate --> server-ca ==> kube-server-certs intermediate --> client-ca ==> kube-client-certs intermediate --> request-header-ca ==> request-header-certs intermediate --> etcd-peer-ca ==> etcd-peer-certs intermediate --> etcd-server-ca ==> etcd-server-certs ``` #### Using the Example Script :::info Important If you want to sign the cluster CA certificates with an existing root CA using the example script, you must place the root and intermediate files in the target directory prior to running the script. If the files do not exist, the script will create new root and intermediate CA certificates. ::: If you want to use only an existing root CA certificate, provide the following files: * `root-ca.pem` * `root-ca.key` If you want to use existing root and intermediate CA certificates, provide the following files: * `root-ca.pem` * `intermediate-ca.pem` * `intermediate-ca.key` The example script uses the provided CAs as the root for all cluster CA bundles - Server, Client, API Aggregation, and etcd Peer/Server. For advanced use cases, such as using the provided CAs for only select bundles, or using different CAs for different bundles, the example script should be modified to meet the specific needs of your organization. To use the example script to generate custom certs and keys before starting K3s, run the following commands: ```bash # Create the target directory for cert generation. mkdir -p /var/lib/rancher/k3s/server/tls # Copy your root CA cert and intermediate CA cert+key into the correct location for the script. # For the purposes of this example, we assume you have existing root and intermediate CA files in /etc/ssl. # If you do not have an existing root and/or intermediate CA, the script will generate them for you. cp /etc/ssl/certs/root-ca.pem /etc/ssl/certs/intermediate-ca.pem /etc/ssl/private/intermediate-ca.key /var/lib/rancher/k3s/server/tls # Generate custom CA certs and keys. curl -sL https://github.com/k3s-io/k3s/raw/main/contrib/util/generate-custom-ca-certs.sh | bash - ``` If the command completes successfully, you may install and/or start K3s for the first time. If the script generated root and/or intermediate CA files, you should back up these files so that they can be reused if it is necessary to rotate the CA certificates at a later date. ### Rotating Custom CA Certificates To rotate custom CA certificates, use the `k3s certificate rotate-ca` subcommand. Updated files must be staged into a temporary directory, loaded into the datastore, and k3s must be restarted on all nodes to use the updated certificates. :::warning You must not overwrite the currently in-use data in `/var/lib/rancher/k3s/server/tls`. Stage the updated certificates and keys into a separate directory. ::: A cluster that has been started with custom CA certificates can renew or rotate the CA certificates and keys non-disruptively, as long as the same root CA is used. If a new root CA is required, the rotation will be disruptive. The `k3s certificate rotate-ca --force` option must be used, all nodes that were joined with a [secure token](token.md#secure) (including servers) will need to be reconfigured to use the new token value, and pods will need to be restarted to trust the new root CA. #### Using the Example Script The example `generate-custom-ca-certs.sh` script linked above can also be used to generate updated certs in a new temporary directory, by copying files into the correct location and setting the `DATA_DIR` environment variable. To use the example script to generate updated certs and keys, run the following commands: ```bash # Create a temporary directory for cert generation. mkdir -p /opt/k3s/server/tls # Copy your root CA cert and intermediate CA cert+key into the correct location for the script. # Non-disruptive rotation requires the same root CA that was used to generate the original certificates. # If the original files are still in the data directory, you can just run: cp /var/lib/rancher/k3s/server/tls/root-ca.* /var/lib/rancher/k3s/server/tls/intermediate-ca.* /opt/k3s/server/tls # Copy the current service-account signing key, so that existing service-account tokens are not invalidated. cp /var/lib/rancher/k3s/server/tls/service.key /opt/k3s/server/tls # Generate updated custom CA certs and keys. curl -sL https://github.com/k3s-io/k3s/raw/main/contrib/util/generate-custom-ca-certs.sh | DATA_DIR=/opt/k3s bash - # Load the updated CA certs and keys into the datastore. k3s certificate rotate-ca --path=/opt/k3s/server ``` If the `rotate-ca` command returns an error, check the service log for errors. If the command completes successfully, restart K3s on all nodes in the cluster - servers first, then agents. If you used the `--force` option or changed the root CA, ensure that any nodes that were joined with a [secure token](token.md#secure) are reconfigured to use the new token value, prior to being restarted. The token may be stored in a `.env` file, systemd unit, or config.yaml, depending on how the node was configured during initial installation. ### Rotating Self-Signed CA Certificates To rotate the K3s-generated self-signed CA certificates, use the `k3s certificate rotate-ca` subcommand. Updated files must be staged into a temporary directory, loaded into the datastore, and k3s must be restarted on all nodes to use the updated certificates. :::warning You must not overwrite the currently in-use data in `/var/lib/rancher/k3s/server/tls`. Stage the updated certificates and keys into a separate directory. ::: If the cluster has been started with default self-signed CA certificates, rotation will be disruptive. All nodes that were joined with a [secure token](token.md#secure) will need to be reconfigured to trust the new CA hash. If the new CA certificates are not cross-signed by the old CA certificates, you will need to use the `--force` option to bypass integrity checks, and pods will need to be restarted to trust the new root CA. #### Default CA Topology The default self-signed CA certificates have the following topology: ```mermaid graph TD server-ca("Server CA") client-ca("Client CA") request-header-ca("API Aggregation CA") etcd-peer-ca("etcd Peer CA") etcd-server-ca("etcd Server CA") root-hash>"Join token CA hash"] kube-server-certs[["Kubernetes servers(control-plane and kubelet listeners)"]] kube-client-certs[["Kubernetes clients(apiserver and kubelet clients)"]] request-header-certs[["Kubernetes API aggregation(apiserver proxy client)"]] etcd-peer-certs[["etcd peer client/server(etcd replication)"]] etcd-server-certs[["etcd client/server certificates(Kubernetes <-> etcd)"]] server-ca -.-|SHA256| root-hash server-ca ===> kube-server-certs client-ca ===> kube-client-certs request-header-ca ===> request-header-certs etcd-peer-ca ===> etcd-peer-certs etcd-server-ca ===> etcd-server-certs ``` When rotating the default self-signed CAs, a modified certificate topology with intermediate CAs and a new root CA cross-signed by the old CA can be used so that there is a continuous chain of trust between the old and new CAs: ``` /* Detailed source-code truncated for AI context efficiency. */ ``` #### Using The Example Script An example script to create updated CA certificates and keys cross-signed by the existing CAs is available [in the K3s repo at `contrib/util/rotate-default-ca-certs.sh`](https://github.com/k3s-io/k3s/blob/main/contrib/util/rotate-default-ca-certs.sh). To use the example script to generate updated self-signed certificates that are cross-signed by the existing CAs, run the following commands: ```bash # Create updated CA certs and keys, cross-signed by the current CAs. # This script will create a new temporary directory containing the updated certs, and output the new token values. curl -sL https://github.com/k3s-io/k3s/raw/main/contrib/util/rotate-default-ca-certs.sh | bash - # Load the updated certs into the datastore; see the script output for the updated token values. k3s certificate rotate-ca --path=/var/lib/rancher/k3s/server/rotate-ca ``` If the `rotate-ca` command returns an error, check the service log for errors. If the command completes successfully, restart K3s on all nodes in the cluster - servers first, then agents. Ensure that any nodes that were joined with a [secure token](token.md#secure), including other server nodes, are reconfigured to use the new token value prior to being restarted. The token may be stored in a `.env` file, systemd unit, or config.yaml, depending on how the node was configured during initial installation. ## Service-Account Issuer Key Rotation The service-account issuer key is an RSA private key used to sign service-account tokens. When rotating the service-account issuer key, at least one old key should be retained in the file so that existing service-account tokens are not invalidated. It can be rotated independent of the cluster CAs by using the `k3s certificate rotate-ca` to install only an updated `service.key` file that includes both the new and old keys. :::warning You must not overwrite the currently in-use data in `/var/lib/rancher/k3s/server/tls`. Stage the updated key into a separate directory. ::: For example, to rotate only the service-account issuer key, run the following commands: ```bash # Create a temporary directory for cert generation mkdir -p /opt/k3s/server/tls # Check OpenSSL version openssl version | grep -qF 'OpenSSL 3' && OPENSSL_GENRSA_FLAGS=-traditional # Generate a new key openssl genrsa ${OPENSSL_GENRSA_FLAGS:-} -out /opt/k3s/server/tls/service.key 2048 # Append the existing key to avoid invalidating current tokens cat /var/lib/rancher/k3s/server/tls/service.key >> /opt/k3s/server/tls/service.key # Load the updated key into the datastore k3s certificate rotate-ca --path=/opt/k3s/server ``` It is normal to see warnings for files that are not being updated. If the `rotate-ca` command returns an error, check the service log for errors. If the command completes successfully, restart K3s on all servers in the cluster. It is not necessary to restart agents or restart any pods. --- ## File: docs/cli/cli.md --- title: CLI Tools --- The K3s binary contains a number of additional tools the help you manage your cluster. Command | Description --------|------------------ `k3s server`| Run a K3s server node, which launches the Kubernetes `apiserver`, `scheduler`, `controller-manager`, and `cloud-controller-manager` components, in addition a datastore and the agent components. See the [`k3s server` command documentation](server.md) for more information. `k3s agent`| Run the K3s agent node, which launches `containerd`, `flannel`, `kube-router` network policy controller, and the Kubernetes `kubelet` and `kube-proxy` components. See the [`k3s agent` command documentation](agent.md) for more information. `k3s kubectl`| Run the embedded [`kubectl` command](https://kubernetes.io/docs/reference/kubectl). This is a CLI for interacting with the Kubernetes apiserver. If the `KUBECONFIG` environment variable is not set, this will automatically attempt to use the kubeconfig at `/etc/rancher/k3s/k3s.yaml`. `k3s crictl`| Run the embedded [`crictl` command](https://github.com/kubernetes-sigs/cri-tools/blob/master/docs/crictl.md). This is a CLI for interacting with Kubernetes's container runtime interface (CRI). Useful for debugging. `k3s ctr`| Run the embedded [`ctr` command](https://github.com/projectatomic/containerd/blob/master/docs/cli.md). This is a CLI for containerd, the container daemon used by K3s. Useful for debugging. `k3s token` | Manage bootstrap tokens. See the [`k3s token` command documentation](token.md) for more information. `k3s etcd-snapshot` | Perform on demand backups of the K3s cluster data and upload to S3. See the [`k3s etcd-snapshot` command documentation](etcd-snapshot.md) for more information. `k3s secrets-encrypt` | Configure K3s to encrypt secrets when storing them in the cluster. See the [`k3s secrets-encrypt` command documentation](secrets-encrypt.md) for more information. `k3s certificate` | Manage K3s certificates. See the [`k3s certificate` command documentation](certificate.md) for more information. `k3s completion` | Generate shell completion scripts for k3s `k3s help`| Shows a list of commands or help for one command --- ## File: docs/cli/etcd-snapshot.md --- title: etcd-snapshot --- # k3s etcd-snapshot This page describes how to use the `k3s etcd-snapshot` CLI tool to manage etcd snapshots and how to restore from an etcd snapshot. K3s etcd snapshots are stored on the node file system, and may optionally be uploaded to an S3 compatible object store for disaster recovery scenarios. Snapshots can be both automated on a reoccurring schedule, and taken manually on-demand. The `k3s etcd-snapshot` CLI tool offers a set of subcommands that can be used to create, delete, and manage snapshots. | Subcommand | Description | | ----------- | --------------- | | delete | Delete given snapshot(s) | | ls, list, l | List snapshots | | prune | Remove snapshots that exceed the configured retention count | | save | Trigger an on-demand etcd snapshot | For additional information on the etcd snapshot subcommands, run `k3s etcd-snapshot --help`. ## Creating Snapshots Scheduled snapshots are enabled by default, at 00:00 and 12:00 system time, with 5 snapshots retained. Scheduled snapshots have a name that starts with `etcd-snapshot`, followed by the node name and timestamp. The following options control the operation of scheduled snapshots: | Flag | Description | | ----------- | --------------- | | `--etcd-disable-snapshots` | Disable scheduled snapshots | | `--etcd-snapshot-name` | Sets the base name of etcd scheduled snapshots. (Default: `etcd-snapshot`) | | `--etcd-snapshot-compress` | Compress etcd snapshots | | `--etcd-snapshot-dir` | Directory to save db snapshots. (Default location: `${data-dir}/db/snapshots`) | | `--etcd-snapshot-retention` | Number of snapshots to retain (default: 5) | | `--etcd-snapshot-schedule-cron` | Snapshot interval time in cron spec. eg. every 5 hours `0 */5 * * *` (default: `0 */12 * * *`) | The data-dir value defaults to `/var/lib/rancher/k3s` and can be changed independently by setting the `--data-dir` flag. Scheduled snapshots are saved to the path set by the server's `--etcd-snapshot-dir` value. If you want them replicated in S3 compatible object stores, refer to [S3 configuration options](#s3-compatible-object-store-support) Snapshots can be saved manually by running the `k3s etcd-snapshot save` command. There is no retention for these on-demand snapshots and the user needs to remove them manually by using `k3s etcd-snapshot delete` or `k3s etcd-snapshot prune` commands. On-demand snapshots have a name that starts with `on-demand`, followed by the node name and timestamp. The following options control the operation of on-demand snapshots: | Flag | Description | | ----------- | --------------- | | `--name` | Sets the base name of etcd on-demand snapshots. (Default: `on-demand`) | | `--etcd-snapshot-compress` | Compress etcd snapshots | | `--etcd-snapshot-dir` | Directory to save db snapshots. (Default location: `${data-dir}/db/snapshots`) | The data-dir value defaults to `/var/lib/rancher/k3s` and can be changed independently by setting the `--data-dir` flag. The `--name` flag can only be set when running the `k3s etcd-snapshot save` command. The other two can also be part of the `k3s server` [configuration file](../installation/configuration.md#configuration-file) On-demand snapshots are saved to the path set by the server's `--etcd-snapshot-dir` value. If you want them replicated in S3 compatible object stores, refer to [S3 configuration options](#s3-compatible-object-store-support) ## Deleting Snapshots Scheduled snapshots are deleted automatically when the number of snapshots exceeds the configured retention count (5 by default). The oldest snapshots are removed first. To manually delete scheduled snapshot(s) or on-demand snapshot(s), you can use the `k3s etcd-snapshot delete` command: ```bash k3s etcd-snapshot delete ... ``` The `prune` subcommand removes snapshots that match the name prefix (`on-demand` by default) and exceed the configured retention count. It includes the flag `--snapshot-retention` to set the retention count. For scheduled snapshots, it overrides the default retention policy. On-demand snapshots have no retention policy and hence this flag is required. Prune "on-demand" snapshots down to a smaller amount: ```bash k3s etcd-snapshot prune --snapshot-retention ``` Prune "scheduled" snapshots down to a smaller amount: ```bash k3s etcd-snapshot prune --name etcd-snapshot --etcd-snapshot-retention ``` ## S3 Compatible Object Store Support K3s supports replicating etcd snapshots to and restoring etcd snapshots from S3-compatible object stores. S3 support is available for both on-demand and scheduled snapshots. | Flag | Description | | ----------- | --------------- | | `--etcd-s3` | Enable backup to S3 | | `--etcd-s3-endpoint` | S3 endpoint url | | `--etcd-s3-endpoint-ca` | S3 custom CA cert to connect to S3 endpoint | | `--etcd-s3-skip-ssl-verify` | Disables S3 SSL certificate validation | | `--etcd-s3-access-key` | S3 access key | | `--etcd-s3-secret-key` | S3 secret key | | `--etcd-s3-session-token` | S3 session token | | `--etcd-s3-bucket` | S3 bucket name | | `--etcd-s3-bucket-lookup-type` | S3 bucket lookup type, one of 'auto', 'dns', 'path'; default is 'auto' if not set | | `--etcd-s3-region` | S3 region / bucket location (optional). defaults to us-east-1 | | `--etcd-s3-folder` | S3 folder | | `--etcd-s3-retention` | S3 retention limit (default: 5) | | `--etcd-s3-proxy` | Proxy server to use when connecting to S3, overriding any proxy-releated environment variables | | `--etcd-s3-insecure` | Disables S3 over HTTPS | | `--etcd-s3-timeout` | S3 timeout (default: `5m0s`) | | `--etcd-s3-config-secret` | Name of secret in the kube-system namespace used to configure S3, if etcd-s3 is enabled and no other etcd-s3 options are set | :::note `--etcd-snapshot-retention` sets `--etcd-s3-retention`, if it is not explicitly set in the config. ::: For example, this is how the creation and deletion of on-demand etcd snapshots in S3 would work: ```shell-session $ k3s etcd-snapshot --s3 --s3-bucket=test-bucket --s3-access-key=test --s3-secret-key=secret save INFO[0155] Snapshot on-demand-server-0-1753178523 saved. INFO[0155] Snapshot on-demand-server-0-1753178523 saved. $ k3s etcd-snapshot --s3 --s3-bucket=test-bucket --s3-access-key=test --s3-secret-key=secret ls Name Location Size Created on-demand-server-0-1753178523 s3://test-bucket/test-folder/on-demand-server-0-1753178523 5062688 2025-07-22T10:02:03Z on-demand-server-0-1753178523 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-server-0-1753178523 5062688 2025-07-22T10:02:03Z $ k3s etcd-snapshot --s3 --s3-bucket=test-bucket --s3-access-key=test --s3-secret-key=secret delete on-demand-server-0-1753178523 INFO[0000] Snapshot on-demand-server-0-1753178523 deleted. $ k3s etcd-snapshot --s3 --s3-bucket=test-bucket --s3-access-key=test --s3-secret-key=secret ls Name Location Size Created ``` ### S3 Configuration Secret Support K3s supports reading etcd S3 snapshot configuration from a Kubernetes Secret. This may be preferred to hardcoding credentials in K3s CLI flags or config files for security reasons, or if credentials need to be rotated without restarting K3s. To pass S3 snapshot configuration via a Secret, start K3s with `--etcd-s3` and `--etcd-s3-config-secret=`. The Secret does not need to exist when K3s is started, but it will be checked for every time a snapshot save/list/delete/prune operation is performed. The S3 config Secret cannot be used when restoring a snapshot, as the apiserver is not available to provide the secret during a restore. S3 configuration must be passed via the CLI when restoring a snapshot stored on S3. :::note Pass only the `--etcd-s3` and `--etcd-s3-config-secret` flags to enable the Secret. If any other S3 configuration flags are set, the Secret will be ignored. ::: Keys in the Secret correspond to the `--etcd-s3-*` CLI flags listed above. The `etcd-s3-endpoint-ca` key accepts a PEM-encoded CA bundle, or the `etcd-s3-endpoint-ca-name` key may be used to specify the name of a ConfigMap in the `kube-system` namespace containing one or more PEM-encoded CA bundles. ```yaml apiVersion: v1 kind: Secret metadata: name: k3s-etcd-snapshot-s3-config namespace: kube-system type: etcd.k3s.cattle.io/s3-config-secret stringData: etcd-s3-endpoint: "" etcd-s3-endpoint-ca: "" etcd-s3-endpoint-ca-name: "" etcd-s3-skip-ssl-verify: "false" etcd-s3-access-key: "AWS_ACCESS_KEY_ID" etcd-s3-secret-key: "AWS_SECRET_ACCESS_KEY" etcd-s3-bucket: "bucket" etcd-s3-folder: "folder" etcd-s3-region: "us-east-1" etcd-s3-insecure: "false" etcd-s3-timeout: "5m" etcd-s3-proxy: "" ``` ## Restoring Snapshots K3s runs through several steps when restoring a snapshot: 1. If the snapshot is stored on S3, the file is downloaded into the snapshot directory. 2. If the snapshot is compressed, it is decompressed. 3. If present, the current etcd database files are moved to `${data-dir}/server/db/etcd-old-$TIMESTAMP/`. 4. The snapshot's contents are extracted out to disk, and the checksum is verified. 5. Etcd is started, and all etcd cluster members except the current node are removed from the cluster. 6. CA Certificates and other confidential data are extracted from the datastore and written to disk, for later use. 7. The restore is complete, and K3s can be restarted and used normally on the server where the restore was performed. 8. (optional) Agents and control-plane servers can be started normally. 8. (optional) Etcd servers can be restarted to rejoin to the cluster after removing old database files. When restoring a snapshot, you don't need to use the same K3s version that created it; a higher minor version is also acceptable. ### Snapshot Restore Steps Select the tab below that matches your cluster configuration. 1. Stop the K3s service: ```bash systemctl stop k3s ``` 2. Run `k3s server` with the `--cluster-reset` flag, and `--cluster-reset-restore-path` indicating the path to the snapshot to restore. If the snapshot is stored on S3, provide S3 configuration flags (`--etcd-s3`, `--etcd-s3-bucket`, and so on), and give only the filename name of the snapshot as the restore path. :::note Using the `--cluster-reset` flag without specifying a snapshot to restore simply resets the etcd cluster to a single member without restoring a snapshot. ::: ```bash k3s server \ --cluster-reset \ --cluster-reset-restore-path= ``` **Result:** K3s restores the snapshot and resets cluster membership, then prints a message indicating that it is ready to be restarted: `Managed etcd cluster membership has been reset, restart without --cluster-reset flag now.` 3. Start K3s again: ```bash systemctl start k3s ``` If an etcd-s3 backup configuration is defined within the K3s config file, the k3s restore will attempt to pull the snapshot file from the configured S3 bucket. In this instance only the snapshot filename should be passed in the argument `--cluster-reset-restore-path`. To restore from a local snapshot file, where an etcd-s3 backup configuration is present, add the argument `--etcd-s3=false` and pass the full path to the local snapshot file in the argument `--cluster-reset-restore-path`. As a safety mechanism, when K3s resets the cluster, it creates an empty file at `/var/lib/rancher/k3s/server/db/reset-flag` that prevents users from accidentally running multiple cluster resets in succession. This file is deleted when K3s starts normally. In this example there are 3 servers, `S1`, `S2`, and `S3`. The snapshot is located on `S1`. 1. Stop K3s on all servers: ```bash systemctl stop k3s ``` 2. On S1, run `k3s server` with the `--cluster-reset` option, and `--cluster-reset-restore-path` indicating the path to the snapshot to restore. If the snapshot is stored on S3, provide S3 configuration flags (`--etcd-s3`, `--etcd-s3-bucket`, and so on), and give only the filename name of the snapshot as the restore path. :::note Using the `--cluster-reset` flag without specifying a snapshot to restore simply resets the etcd cluster to a single member without restoring a snapshot. ::: ```bash k3s server \ --cluster-reset \ --cluster-reset-restore-path= ``` **Result:** K3s restores the snapshot and resets cluster membership, then prints a message indicating that it is ready to be restarted: `Managed etcd cluster membership has been reset, restart without --cluster-reset flag now.` `Backup and delete ${datadir}/server/db on each peer etcd server and rejoin the nodes.` 3. On S1, start K3s again: ```bash systemctl start k3s ``` 4. On S2 and S3, delete the data directory, `/var/lib/rancher/k3s/server/db/`: ```bash rm -rf /var/lib/rancher/k3s/server/db/ ``` 5. On S2 and S3, start K3s again to join the restored cluster: ```bash systemctl start k3s ``` If an etcd-s3 backup configuration is defined within the K3s config file, the k3s restore will attempt to pull the snapshot file from the configured S3 bucket. In this instance only the snapshot filename should be passed in the argument `--cluster-reset-restore-path`. To restore from a local snapshot file, where an etcd-s3 backup configuration is present, add the argument `--etcd-s3=false` and pass the full path to the local snapshot file in the argument `--cluster-reset-restore-path`. As a safety mechanism, when K3s resets the cluster, it creates an empty file at `/var/lib/rancher/k3s/server/db/reset-flag` that prevents users from accidentally running multiple cluster resets in succession. This file is deleted when K3s starts normally. #### Restoring To New Hosts It is possible to restore an etcd snapshot to a different host than it was taken on. When doing so, you must pass the [server token](token.md#server) that was originally used when taking the snapshot, as it is used to decrypt the bootstrap data inside the snapshot. The process is the same as above but changing step 2 by: 1. In the node that took the snapshot save the value of: `/var/lib/rancher/k3s/server/token`. This is `` in step 3. 2. Copy the snapshot to the new node. The path in the node is `` in step 3 3. Initiate the restore from snapshot on the first server node with the following commands: ```bash k3s server \ --cluster-reset \ --cluster-reset-restore-path= --token= ``` The token value can also be set in the K3s config file. :::warning 1. Node resources are also included in the etcd snapshot. If restoring to a new set of nodes, you will need to manually delete any old nodes that are no longer present in the cluster. 2. If there is a token set in the K3s config file, make sure it is the same as the ``, otherwise k3s will fail to start. ::: ### Security Snapshots contain a complete copy of the etcd datastore, along with cluster certificate authority (CA) certificates and private keys. If [secrets encryption](secrets-encrypt.md) is enabled, snapshots also include the secrets encryption configuration and keys. This information is encrypted with an AES-256 key derived from the server's join token using PBKDF2. Administrators should be aware of the following concerns when backing up or restoring snapshots: * Any attacker who can convince you to restore a snapshot under their control can gain complete control of the cluster and nodes via multiple attack vectors - it is now their cluster, not yours. * Any attacker who possesses a copy of the etcd snapshot file can extract all unencrypted Kubernetes resources. * Any attacker who possesses a copy of the etcd snapshot file AND the server join token can extract encrypted resources, along with private keys to the cluster CAs. Snapshots stored locally on the node are protected via normal Unix filesystem permissions and access controls. It is recommended that administrators safely store snapshots to a S3-compatible service that supports bucket access controls, server-side encryption, and retention controls. Validating the integrity and chain of custody of datastore snapshots is the responsibility of the administrator. ## ETCDSnapshotFile Custom Resources Snapshots can be viewed remotely using any Kubernetes client by listing or describing cluster-scoped `ETCDSnapshotFile` resources. Unlike the `k3s etcd-snapshot list` command, which only shows snapshots visible to that node, `ETCDSnapshotFile` resources track all snapshots present on cluster members. ```shell-session $ kubectl get etcdsnapshotfile NAME SNAPSHOTNAME NODE LOCATION SIZE CREATIONTIME local-on-demand-k3s-server-1-1730308816-3e9290 on-demand-k3s-server-1-1730308816 k3s-server-1 file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-k3s-server-1-1730308816 2891808 2024-10-30T17:20:16Z s3-on-demand-k3s-server-1-1730308816-79b15c on-demand-k3s-server-1-1730308816 s3 s3://etcd/k3s-test/on-demand-k3s-server-1-1730308816 2891808 2024-10-30T17:20:16Z ``` ```shell-session $ kubectl describe etcdsnapshotfile s3-on-demand-k3s-server-1-1730308816-79b15c Name: s3-on-demand-k3s-server-1-1730308816-79b15c Namespace: Labels: etcd.k3s.cattle.io/snapshot-storage-node=s3 Annotations: etcd.k3s.cattle.io/snapshot-token-hash: b4b83cda3099 API Version: k3s.cattle.io/v1 Kind: ETCDSnapshotFile Metadata: Creation Timestamp: 2024-10-30T17:20:16Z Finalizers: wrangler.cattle.io/managed-etcd-snapshots-controller Generation: 1 Resource Version: 790 UID: bec9a51c-dbbe-4746-922e-a5136bef53fc Spec: Location: s3://etcd/k3s-test/on-demand-k3s-server-1-1730308816 Node Name: s3 s3: Bucket: etcd Endpoint: s3.example.com Prefix: k3s-test Region: us-east-1 Skip SSL Verify: true Snapshot Name: on-demand-k3s-server-1-1730308816 Status: Creation Time: 2024-10-30T17:20:16Z Ready To Use: true Size: 2891808 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ETCDSnapshotCreated 113s k3s-supervisor Snapshot on-demand-k3s-server-1-1730308816 saved on S3 ``` --- ## File: docs/cli/secrets-encrypt.md --- title: secrets-encrypt --- # k3s secrets-encrypt K3s supports enabling secrets encryption at rest. For more information, see [Secrets Encryption](../security/secrets-encryption.md). ## Secrets Encryption Tool K3s contains a CLI tool `secrets-encrypt`, which enables automatic control over the following: - Disabling/Enabling secrets encryption - Adding new encryption keys - Rotating and deleting encryption keys - Reencrypting secrets :::warning Failure to follow proper procedure for rotating encryption keys can leave your cluster permanently corrupted. Proceed with caution. ::: ### Encryption Key Rotation To rotate secrets encryption keys on a single-server cluster: 1. Start the K3s server with the flag `--secrets-encryption` 2. Rotate secrets encryption keys ``` k3s secrets-encrypt rotate-keys ``` 3. Wait for reencryption to finish. Watch the server logs, or wait for: ```bash $ k3s secrets-encrypt status Encryption Status: Enabled Current Rotation Stage: reencrypt_finished ``` For brevity, the servers will be referred to as S1, S2, S3. To rotate secrets encryption keys on HA setups: 1. Start up all three K3s servers with the `--secrets-encryption` flag. 2. Rotate secrets encryption keys on S1 ```bash k3s secrets-encrypt rotate-keys ``` 3. Wait for reencryption to finish. Watch the server logs, or wait for: ```bash $ k3s secrets-encrypt status Encryption Status: Enabled Current Rotation Stage: reencrypt_finished ``` :::info K3s will reencrypt ~5 secrets per second. Clusters with large # of secrets can take several minutes to reencrypt. You can track progress in the server logs. ::: 4. Restart K3s on S1 with same arguments. If running K3s as a service: ```bash # If using systemd systemctl restart k3s # If using openrc rc-service k3s restart ``` 5. Once S1 is up, restart K3s on S2 and S3 ### Enable Secrets Encryption on an Existing Cluster :::info Version Gate Available as of the March 2026 releases: v1.33.10+k3s1, v1.34.6+k3s1, v1.35.3+k3s1 ::: Use this procedure when the cluster was started without `--secrets-encryption` and you want to enable secrets encryption at a later time. 1. Verify encryption is currently disabled ```bash $ k3s secrets-encrypt status Encryption Status: Disabled, no configuration file found ``` 2. Enable secrets encryption on the server ```bash k3s secrets-encrypt enable ``` 3. Modify the server args/config and add `--secrets-encryption`. Restart the server. 4. Verify the status shows encryption disabled but at the `start` stage ```bash $ k3s secrets-encrypt status Encryption Status: Disabled Current Rotation Stage: start Server Encryption Hashes: All hashes match ``` 5. Rotate keys to enable encryption for new secrets ```bash k3s secrets-encrypt rotate-keys ``` 6. Restart the server with the same arguments 7. Verify encryption is enabled ```bash $ k3s secrets-encrypt status Encryption Status: Enabled Current Rotation Stage: reencrypt_finished Server Encryption Hashes: All hashes match ``` For brevity, the servers will be referred to as S1, S2, S3. 1. Verify encryption is currently disabled ```bash k3s secrets-encrypt status Encryption Status: Disabled, no configuration file found ``` 2. Enable secrets encryption on S1 ```bash k3s secrets-encrypt enable ``` 3. Restart all servers with `--secrets-encryption` 4. Verify the status shows encryption disabled but at the `start` stage, and that all server hashes match ```bash k3s secrets-encrypt status Encryption Status: Disabled Current Rotation Stage: start Server Encryption Hashes: All hashes match ``` 5. Rotate keys on S1 to enable encryption for new secrets ```bash k3s secrets-encrypt rotate-keys ``` 6. Restart all servers with the same arguments 7. Verify encryption is enabled ```bash k3s secrets-encrypt status Encryption Status: Enabled Current Rotation Stage: reencrypt_finished Server Encryption Hashes: All hashes match ``` ### Legacy Encryption Key Rotation :::tip New Procedure If using K3s versions v1.30+, we recommend using the [Encryption Key Rotation](#encryption-key-rotation) instead. ::: To rotate secrets encryption keys on a single-server cluster: 1. Start the K3s server with the flag `--secrets-encryption` 2. Prepare ```bash k3s secrets-encrypt prepare ``` 3. Kill and restart the K3s server with same arguments. If running K3s as a service: ```bash # If using systemd systemctl restart k3s # If using openrc rc-service k3s restart ``` 4. Rotate ```bash k3s secrets-encrypt rotate ``` 5. Kill and restart the K3s server with same arguments 6. Reencrypt :::info K3s will reencrypt ~5 secrets per second. Clusters with large # of secrets can take several minutes to reencrypt. ::: ```bash k3s secrets-encrypt reencrypt ``` The steps are the same for both embedded DB and external DB clusters. or brevity, the servers will be referred to as S1, S2, S3. To rotate secrets encryption keys on HA setups: 1. Start up all three K3s servers with the `--secrets-encryption` flag. F :::note Notes - While not required, it is recommended that you pick one server node from which to run the `secrets-encrypt` commands. ::: 2. Prepare on S1 ```bash k3s secrets-encrypt prepare ``` 3. Kill and restart S1 with same arguments. If running K3s as a service: ```bash # If using systemd systemctl restart k3s # If using openrc rc-service k3s restart ``` 4. Once S1 is up, kill and restart the S2 and S3 5. Rotate on S1 ```bash k3s secrets-encrypt rotate ``` 6. Kill and restart S1 with same arguments 7. Once S1 is up, kill and restart the S2 and S3 8. Reencrypt on S1 :::info K3s will reencrypt ~5 secrets per second. Clusters with large # of secrets can take several minutes to reencrypt. ::: ```bash k3s secrets-encrypt reencrypt ``` 9. Kill and restart S1 with same arguments 10. Once S1 is up, kill and restart the S2 and S3 ### Secrets Encryption Disable/Re-enable To disable secrets encryption on a single-node cluster: 1. Disable ```bash k3s secrets-encrypt disable ``` 2. Kill and restart the K3s server with same arguments. If running K3s as a service: ```bash # If using systemd systemctl restart k3s # If using openrc rc-service k3s restart ``` 3. Reencrypt with flags ```bash k3s secrets-encrypt reencrypt --force --skip ``` To re-enable secrets encryption on a single node cluster: 1. Enable ```bash k3s secrets-encrypt enable ``` 2. Kill and restart the K3s server with same arguments 3. Reencrypt with flags ```bash k3s secrets-encrypt reencrypt --force --skip ``` :::note While not required, it is recommended that you pick one server node from which to run the `secrets-encrypt` commands. ::: For brevity, the servers will be referred to as S1, S2, S3. To disable secrets encryption on a HA cluster: 1. Disable on S1 ```bash k3s secrets-encrypt disable ``` 2. Kill and restart S1 with same arguments. If running K3s as a service: ```bash # If using systemd systemctl restart k3s # If using openrc rc-service k3s restart ``` 3. Once S1 is up, kill and restart the S2 and S3 4. Reencrypt with flags on S1 ```bash k3s secrets-encrypt reencrypt --force --skip ``` To re-enable secrets encryption on a HA cluster: 1. Enable on S1 ```bash k3s secrets-encrypt enable ``` 2. Kill and restart S1 with same arguments 3. Once S1 is up, kill and restart the S2 and S3 4. Reencrypt with flags on S1 ```bash k3s secrets-encrypt reencrypt --force --skip ``` ### Secrets Encryption Status The secrets-encrypt tool includes a `status` command that displays information about the current status of secrets encryption on the node. An example of the command on a single-server node: ```bash $ k3s secrets-encrypt status Encryption Status: Enabled Current Rotation Stage: start Server Encryption Hashes: All hashes match Active Key Type Name ------ -------- ---- * AES-CBC aescbckey ``` Another example on HA cluster, after rotating the keys, but before restarting the servers: ```bash $ k3s secrets-encrypt status Encryption Status: Enabled Current Rotation Stage: rotate Server Encryption Hashes: hash does not match between node-1 and node-2 Active Key Type Name ------ -------- ---- * AES-CBC aescbckey-2021-12-10T22:54:38Z AES-CBC aescbckey ``` Details on each section are as follows: - __Encryption Status__: Displayed whether secrets encryption is disabled or enabled on the node - __Current Rotation Stage__: Indicates the current rotation stage on the node. Stages are: `start`, `prepare`, `rotate`, `reencrypt_request`, `reencrypt_active`, `reencrypt_finished` - __Server Encryption Hashes__: Useful for HA clusters, this indicates whether all servers are on the same stage with their local files. This can be used to identify whether a restart of servers is required before proceeding to the next stage. In the HA example above, node-1 and node-2 have different hashes, indicating that they currently do not have the same encryption configuration. Restarting the servers will sync up their configuration. - __Key Table__: Summarizes information about the secrets encryption keys found on the node. * __Active__: The "*" indicates which, if any, of the keys are currently used for secrets encryption. An active key is used by Kubernetes to encrypt any new secrets. * __Key Type__: Keys using this tool support `AES-CBC` and `secretbox` types. See more info [here.](https://kubernetes.io/docs/tasks/administer-cluster/encrypt-data/#providers) * __Name__: Name of the encryption key. --- ## File: docs/cli/server.md --- title: server --- # k3s server In this section, you'll learn how to configure the K3s server. Note that servers also run an agent, so all of the configuration options listed in the [`k3s agent` documentation](agent.md) are also supported on servers. Options are documented on this page as CLI flags, but can also be passed as configuration file options. See the [Configuration File](../installation/configuration.md#configuration-file) documentation for more information on using YAML configuration files. ## Critical Configuration Values The following options must be set to the same value on all servers in the cluster. Failure to do so will cause new servers to fail to join the cluster when using embedded etcd, or incorrect operation of the cluster when using an external datastore. * `--agent-token` * `--cluster-cidr` * `--cluster-dns` * `--cluster-domain` * `--disable-cloud-controller` * `--disable-helm-controller` * `--disable-network-policy` * `--disable=servicelb` *note: other packaged components may be disabled on a per-server basis* * `--egress-selector-mode` * `--embedded-registry` * `--flannel-backend` * `--flannel-external-ip` * `--flannel-ipv6-masq` * `--secrets-encryption` * `--secrets-encryption-provider` * `--service-cidr` ## Commonly Used Options ### Database | Flag | Environment Variable | Default | Description | |---------------------------------------|--------------------------|----------------------------------------|-------------------------------------------------------------------------------| | `--datastore-endpoint` value | `K3S_DATASTORE_ENDPOINT` | | Specify etcd, NATS, MySQL, Postgres, or SQLite data source name | | `--datastore-cafile` value | `K3S_DATASTORE_CAFILE` | | TLS Certificate Authority file used to secure datastore backend communication | | `--datastore-certfile` value | `K3S_DATASTORE_CERTFILE` | | TLS certification file used to secure datastore backend communication | | `--datastore-keyfile` value | `K3S_DATASTORE_KEYFILE` | | TLS key file used to secure datastore backend communication | | `--etcd-expose-metrics` | | false | Expose etcd metrics to client interface | | `--etcd-disable-snapshots` | | false | Disable automatic etcd snapshots | | `--etcd-snapshot-name` value | | "etcd-snapshot-<unix-timestamp>" | Set the base name of etcd snapshots. | | `--etcd-snapshot-schedule-cron` value | | "0 */12 \* \* \*" | Snapshot interval time in cron spec. eg. every 5 hours '0 */5 _ \* _' | | `--etcd-snapshot-retention` value | | 5 | Number of snapshots to retain | | `--etcd-snapshot-dir` value | | $\{data-dir\}/db/snapshots | Directory to save db snapshots | ### S3 etcd Snapshot Storage | Flag | Environment Variable | Default | Description | |---------------------------------------|--------------------------|----------------------------------------|-------------------------------------------------------------------------------| | `--etcd-s3` | | | Enable backup to S3 | | `--etcd-s3-endpoint` value | | "s3.amazonaws.com" | S3 endpoint url | | `--etcd-s3-endpoint-ca` value | | | S3 custom CA cert to connect to S3 endpoint | | `--etcd-s3-skip-ssl-verify` | | | Disables S3 SSL certificate validation | | `--etcd-s3-access-key` value | `AWS_ACCESS_KEY_ID` | | S3 access key | | `--etcd-s3-secret-key` value | `AWS_SECRET_ACCESS_KEY` | | S3 secret key | | `--etcd-s3-session-token` value | `AWS_SESSION_TOKEN` | | S3 session token | | `--etcd-s3-bucket` value | | | S3 bucket name | | `--etcd-s3-bucket-lookup-type` value | | | S3 bucket lookup type, one of 'auto', 'dns', 'path'; default is 'auto' if not set | | `--etcd-s3-region` value | | "us-east-1" | S3 region / bucket location (optional) | | `--etcd-s3-folder` value | | | S3 folder | | `--etcd-s3-retention` value | | 5 | S3 retention limit | | `--etcd-s3-proxy` value | | | Proxy server to use when connecting to S3, overriding any proxy-releated environment variables | | `--etcd-s3-config-secret` value | | | Name of secret in the kube-system namespace used to configure S3, if etcd-s3 is enabled and no other etcd-s3 options are set | | `--etcd-s3-insecure` | | | Disables S3 over HTTPS | | `--etcd-s3-timeout` value | | 5m0s | S3 timeout (default: 5m0s) | ### Cluster Options | Flag | Environment Variable | Description | |-----------------------------|------------------------|-------------------------------------------------------------------| | `--token` value, `-t` value | `K3S_TOKEN` | Shared secret used to join a server or agent to a cluster | | `--token-file` value | `K3S_TOKEN_FILE` | File containing the cluster-secret/token | | `--agent-token` value | `K3S_AGENT_TOKEN` | Shared secret used to join agents to the cluster, but not servers | | `--agent-token-file` value | `K3S_AGENT_TOKEN_FILE` | File containing the agent secret | | `--server` value | `K3S_URL` | Server to connect to, used to join a cluster | | `--cluster-init` | `K3S_CLUSTER_INIT` | Initialize a new cluster using embedded Etcd | | `--cluster-reset` | `K3S_CLUSTER_RESET` | Forget all peers and become sole member of a new cluster | ### Admin Kubeconfig Options | Flag | Environment Variable | Description | |--------------------------------------|-------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `--write-kubeconfig value, -o` value | `K3S_KUBECONFIG_OUTPUT` | Write kubeconfig for admin client to this file | | `--write-kubeconfig-mode` value | `K3S_KUBECONFIG_MODE` | Write kubeconfig with this [mode.](https://en.wikipedia.org/wiki/Chmod) The kubeconfig file is owned by root, and written with a default mode of 600. Changing the mode to 644 will allow it to be read by other unprivileged users on the host. | | `--write-kubeconfig-group` value | `K3S_KUBECONFIG_GROUP` | Write kubeconfig group. Combining with `--write-kubeconfig-mode`, it will allow your k3s administrators accessing the kubeconfig file but keeping the file owned by root. | ## Advanced Options ### Logging | Flag | Default | Description | |-------------------------|---------|-----------------------------------------------------------------------------------| | `--debug` | N/A | Turn on debug logs | | `-v` value | 0 | Number for the log level verbosity | | `--vmodule` value | N/A | Comma-separated list of FILE_PATTERN=LOG_LEVEL settings for file-filtered logging | | `--log value, -l` value | N/A | Log to file | | `--alsologtostderr` | N/A | Log to standard error as well as file (if set) | ### Listeners | Flag | Default | Description | |-----------------------------|--------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `--bind-address` value | 0.0.0.0 | k3s bind address | | `--https-listen-port` value | 6443 | HTTPS listen port | | `--advertise-address` value | node-external-ip/node-ip | IPv4/IPv6 address that apiserver advertises for its service endpointNote that the primary `service-cidr` IP range must be of the same address family as the advertised address | | `--advertise-port` value | listen-port/0 | Port that apiserver uses to advertise to members of the cluster | | `--tls-san` value | N/A | Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the TLS cert | | `--tls-san-security` | true | Protect the server TLS cert by refusing to add Subject Alternative Names not associated with the kubernetes apiserver service, server nodes, or values of the tls-san option | ### Data | Flag | Default | Description | | ---------------------------- | ------------------------------------------------------------ | -------------------- | | `--data-dir value, -d` value | `/var/lib/rancher/k3s` or `${HOME}/.rancher/k3s` if not root | Folder to hold state | ### Secrets Encryption | Flag | Default | Description | |------------------------|---------|----------------------------------| | `--secrets-encryption` | false | Enable Secret encryption at rest | | `--secrets-encryption-provider` | aescbc | Encryption provider to use | ### Networking | Flag | Default | Description | |-----------------------------------|-----------------|--------------| | `--cluster-cidr` value | "10.42.0.0/16" | IPv4/IPv6 network CIDRs to use for pod IPs | | `--service-cidr` value | "10.43.0.0/16" | IPv4/IPv6 network CIDRs to use for service | | `--service-node-port-range` value | "30000-32767" | Port range to reserve for services with NodePort visibility | | `--cluster-dns` value | "10.43.0.10" | IPv4 Cluster IP for coredns service. Should be in your service-cidr range | | `--cluster-domain` value | "cluster.local" | Cluster Domain | | `--flannel-backend` value | "vxlan" | One of 'none', 'vxlan', 'ipsec'(deprecated), 'host-gw', 'wireguard-native', or 'wireguard'(deprecated) | | `--flannel-ipv6-masq` | "N/A" | Enable IPv6 masquerading for pod | | `--flannel-external-ip` | "N/A" | Use node external IP addresses for Flannel traffic | | `--servicelb-namespace` value | "kube-system" | Namespace of the pods for the servicelb component | | `--egress-selector-mode` value | "agent" | Must be one of the following:
  • disabled: The apiserver does not use agent tunnels to communicate with nodes. Requires that servers run agents, and have direct connectivity to the kubelet on agents, or the apiserver will not be able to function access service endpoints or perform kubectl exec and kubectl logs.
  • agent: The apiserver uses agent tunnels to communicate with nodes. Nodes allow the tunnel connection from loopback addresses. Requires that servers also run agents, or the apiserver will not be able to access service endpoints. The historical default for k3s.
  • pod: The apiserver uses agent tunnels to communicate with nodes and service endpoints, routing endpoint connections to the correct agent by watching Nodes. Nodes allow the tunnel connection from loopback addresses, or a CIDR assigned to their node.
  • cluster: The apiserver uses agent tunnels to communicate with nodes and service endpoints, routing endpoint connections to the correct agent by watching Endpoints. Nodes allow the tunnel connection from loopback addresses, or the configured cluster CIDR range.
    • | ### Storage Class | Flag | Description | |--------------------------------------|----------------------------------------------------------------| | `--default-local-storage-path` value | Default local storage path for local provisioner storage class (default: `/var/lib/rancher/k3s/storage`) | ### Kubernetes Components | Flag | Description | |------------------------------|-----------------------------------------------------------------------------------------------------| | `--disable` value | See "[Using the `--disable` flag](../installation/packaged-components.md#using-the---disable-flag)" | | `--disable-scheduler` | Disable Kubernetes default scheduler | | `--disable-cloud-controller` | Disable k3s default cloud controller manager | | `--disable-kube-proxy` | Disable running kube-proxy | | `--disable-network-policy` | Disable k3s default network policy controller | | `--disable-helm-controller` | Disable Helm controller | ### Customized Flags for Kubernetes Processes | Flag | Description | |---------------------------------------------|-----------------------------------------------------------| | `--etcd-arg` value | Customized flag for etcd process | | `--kube-apiserver-arg` value | Customized flag for kube-apiserver process | | `--kube-scheduler-arg` value | Customized flag for kube-scheduler process | | `--kube-controller-manager-arg` value | Customized flag for kube-controller-manager process | | `--kube-cloud-controller-manager-arg` value | Customized flag for kube-cloud-controller-manager process | | `--kubelet-arg` value | Customized flag for kubelet process | | `--kube-proxy-arg` value | Customized flag for kube-proxy process | ### Experimental Options | Flag | Description | |------------------------|---------------------------------------------------------------------------------------------------------| | `--rootless` | Run rootless | | `--enable-pprof` | Enable pprof endpoint on supervisor port | | `--docker` | Use cri-dockerd instead of containerd | | `--prefer-bundled-bin` | Prefer bundled userspace binaries over host binaries | | `--disable-agent` | See "[Running Agentless Servers](../advanced.md#running-agentless-servers-experimental)" | | `--embedded-registry` | See "[Embedded Registry Mirror](../installation/registry-mirror.md)" | | `--vpn-auth` | See "[Integration with the Tailscale VPN provider](../networking/distributed-multicloud.md#integration-with-the-tailscale-vpn-provider-experimental)" | | `--vpn-auth-file` | See "[Integration with the Tailscale VPN provider](../networking/distributed-multicloud.md#integration-with-the-tailscale-vpn-provider-experimental)" | ### Deprecated Options | Flag | Environment Variable | Description | |-----------------------------------------|----------------------|---------------------------------------------------------------------------------| | `--no-flannel` | N/A | Use `--flannel-backend=none` | | `--no-deploy` value | N/A | Use `--disable` | | `--cluster-secret` value | `K3S_CLUSTER_SECRET` | Use `--token` | | `--flannel-backend` wireguard | N/A | Use `--flannel-backend=wireguard-native` | | `--flannel-backend` value=option1=value | N/A | Use `--flannel-conf` to specify the flannel config file with the backend config | ## K3s Server CLI Help > If an option appears in brackets below, for example `[$K3S_TOKEN]`, it means that the option can be passed in as an environment variable of that name. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` --- ## File: docs/cli/token.md --- title: token --- # k3s token K3s uses tokens to secure the node join process and to encrypt confidential information that is persisted to the datastore. Tokens authenticate the cluster to the joining node, and the node to the cluster. ## Token Format K3s tokens can be specified in either secure or short format. The secure format is preferred, as it enables the client to authenticate the identity of the cluster it is joining, before sending credentials. ### Secure The secure token format (occasionally referred to as a "full" token) contains the following parts: `::` * `prefix`: a fixed `K10` prefix that identifies the token format * `cluster CA hash`: The hash of the cluster's server CA certificate, used to authenticate the server to the joining node. * For self-signed CA certificates, this is the SHA256 sum of the PEM-formatted certificate, as stored on disk. * For custom CA certificates, this is the SHA256 sum of the DER encoding of the root certificate; commonly known as the certificate fingerprint. * `credentials`: The username and password, or bearer token, used to authenticate the joining node to the cluster. #### TLS Bootstrapping When a secure token is specified, the joining node performs the following steps to validate the identity of the server it has connected to, before transmitting credentials: 1. With TLS verification disabled, download the CA bundle from `/cacerts` on the server it is joining. 2. Calculate the SHA256 hash of the CA certificate, as described above. 3. Compare the calculated SHA256 hash to the hash from the token. 4. If the hash matches, validate that the certificate presented by the server can be validated by the server's CA bundle. 5. If the server certificate is valid, present credentials to join the cluster using either basic or bearer token authentication, depending on the token type. ### Short The short token format includes only the password or bearer token used to authenticate the joining node to the cluster. If a short token is used, the joining node implicitly trusts the CA bundle presented by the server; steps 2-4 in the TLS Bootstrapping process are skipped. The initial connection may be vulnerable to [man-in-the-middle](https://en.wikipedia.org/wiki/Man-in-the-middle_attack) attack. ## Token Types K3s supports three types of tokens. Only the server token is available by default; additional token types must be configured or created by the administrator. Type | CLI Option | Environment Variable --------- | --------------- | -------------------- Server | `--token` | `K3S_TOKEN` Agent | `--agent-token` | `K3S_AGENT_TOKEN` Bootstrap | `n/a` | `n/a` ### Server If no token is provided when starting the first server in the cluster, one is created with a random password. The server token is always written to `/var/lib/rancher/k3s/server/token`, in secure format. The server token can be used to join both server and agent nodes to the cluster. Anyone with access to the server token essentially has full administrator access to the cluster. This token should be guarded carefully. The server token is also used as the [PBKDF2](https://en.wikipedia.org/wiki/PBKDF2) passphrase to encrypt confidential information that is persisted to the datastore known as bootstrap data. Bootstrap data is essential to set up new server nodes or restore from a snapshot. For this reason, the token must be backed up alongside the cluster datastore itself. :::warning Unless custom CA certificates are in use, only the short (password-only) token format can be used when starting the first server in the cluster. This is because the cluster CA hash cannot be known until after the server has generated the self-signed cluster CA certificates. ::: For more information on using custom CA certificates, see the [`k3s certificate` documentation](./certificate.md). For more information on backing up your cluster, see the [Backup and Restore](../datastore/backup-restore.md) documentation. ### Agent By default, the agent token is the same as the server token. The agent token can be set before or after the cluster has been started, by changing the CLI option or environment variable on all servers in the cluster. The agent token is similar to the server token in that is it statically configured, and does not expire. The agent token is written to `/var/lib/rancher/k3s/server/agent-token`, in secure format. If no agent token is specified, this file is a link to the server token. ### Bootstrap K3s supports dynamically generated, automatically expiring agent [bootstrap tokens](https://kubernetes.io/docs/reference/access-authn-authz/bootstrap-tokens/). ## k3s token The k3s token CLI tool handles: * The life cycle of bootstrap tokens, using the same generation and validation code as `kubeadm token` bootstrap tokens. Note that both CLIs are similar. * The rotation of the server token ``` NAME: k3s token - Manage tokens USAGE: k3s token command [command options] [arguments...] COMMANDS: create Create bootstrap tokens on the server delete Delete bootstrap tokens on the server generate Generate and print a bootstrap token, but do not create it on the server list List bootstrap tokens on the server rotate Rotate original server token with a new server token OPTIONS: --help, -h show help ``` #### `k3s token create [token]` Create a new token. The `[token]` is the actual token to write, as generated by `k3s token generate`. If no token is given, a random one will be generated. A token in secure format, including the cluster CA hash, will be written to stdout. The output of this command should be saved, as the secret portion of the token cannot be shown again. Flag | Description ---- | ---- `--data-dir` value | Folder to hold state (default: /var/lib/rancher/k3s or $\{HOME\}/.rancher/k3s if not root) `--kubeconfig` value | Server to connect to [$KUBECONFIG] `--description` value | A human friendly description of how this token is used `--groups` value | Extra groups that this token will authenticate as when used for authentication. (default: Default: "system:bootstrappers:k3s:default-node-token") `--ttl` value | The duration before the token is automatically deleted (e.g. 1s, 2m, 3h). If set to '0', the token will never expire (default: 24h0m0s) `--usages` value | Describes the ways in which this token can be used. (default: "signing,authentication") #### `k3s token delete` Delete one or more tokens. The full token can be provided, or just the token ID. Flag | Description ---- | ---- `--data-dir` value | Folder to hold state (default: /var/lib/rancher/k3s or $\{HOME\}/.rancher/k3s if not root) `--kubeconfig` value |Server to connect to [$KUBECONFIG] #### `k3s token generate` Generate a randomly-generated bootstrap token. You don't have to use this command in order to generate a token. You can do so yourself as long as it is in the format `[a-z0-9]{6}.[a-z0-9]{16}`, where the first portion is the token ID, and the second portion is the secret. Flag | Description ---- | ---- `--data-dir` value | Folder to hold state (default: /var/lib/rancher/k3s or $\{HOME\}/.rancher/k3s if not root) `--kubeconfig` value | Server to connect to [$KUBECONFIG] #### `k3s token list` List bootstrap tokens, showing their ID, description, and remaining time-to-live. Flag | Description ---- | ---- `--data-dir` value | Folder to hold state (default: /var/lib/rancher/k3s or $\{HOME\}/.rancher/k3s if not root) `--kubeconfig` value | Server to connect to [$KUBECONFIG] `--output` value | Output format. Valid options: text, json (default: "text") #### `k3s token rotate` Rotate original server token with a new server token. After running this command, all servers and any agents that originally joined with the old token must be restarted with the new token. If you do not specify a new token, one will be generated for you. Flag | Description ---- | ---- `--data-dir` value | Folder to hold state (default: /var/lib/rancher/k3s or $\{HOME\}/.rancher/k3s if not root) `--kubeconfig` value | Server to connect to [$KUBECONFIG] `--server` value | Server to connect to (default: "https://127.0.0.1:6443") [$K3S_URL] `--token` value | Existing token used to join a server or agent to a cluster [$K3S_TOKEN] `--new-token` value | New token that replaces existing token :::warning Snapshots taken before the rotation will require the old server token when restoring the cluster ::: --- ## File: docs/datastore/backup-restore.md --- title: Backup and Restore --- The way K3s is backed up and restored depends on which type of datastore is used. :::warning In addition to backing up the datastore itself, you must also back up the server token file at `/var/lib/rancher/k3s/server/token`. You must restore this file, or pass its value into the `--token` option, when restoring from backup. If you do not use the same token value when restoring, the snapshot will be unusable, as the token is used to encrypt confidential data within the datastore itself. ::: ## Backup and Restore with SQLite No special commands are required to back up or restore the SQLite datastore. * To back up the SQLite datastore, take a copy of `/var/lib/rancher/k3s/server/db/`. * To restore the SQLite datastore, restore the contents of `/var/lib/rancher/k3s/server/db` (and the token, as discussed above). ## Backup and Restore with External Datastore When an external datastore is used, backup and restore operations are handled outside of K3s. The database administrator will need to back up the external database, or restore it from a snapshot or dump. We recommend configuring the database to take recurring snapshots. For details on taking database snapshots and restoring your database from them, refer to the official database documentation: - [Official MySQL documentation](https://dev.mysql.com/doc/refman/8.0/en/replication-snapshot-method.html) - [Official PostgreSQL documentation](https://www.postgresql.org/docs/8.3/backup-dump.html) - [Official etcd documentation](https://etcd.io/docs/latest/op-guide/recovery/) ## Backup and Restore with Embedded etcd Datastore See the [`k3s etcd-snapshot` command documentation](../cli/etcd-snapshot.md) for information on performing backup and restore operations on the embedded etcd datastore. --- ## File: docs/datastore/cluster-loadbalancer.md --- title: Cluster Load Balancer --- This section describes how to install an external load balancer in front of a High Availability (HA) K3s cluster's server nodes. Two examples are provided: Nginx and HAProxy. :::tip External load-balancers should not be confused with the embedded ServiceLB, which is an embedded controller that allows for use of Kubernetes LoadBalancer Services without deploying a third-party load-balancer controller. For more details, see [Service Load Balancer](../networking/networking-services.md#service-load-balancer). External load-balancers can be used to provide a fixed registration address for registering nodes, or for external access to the Kubernetes API Server. For exposing LoadBalancer Services, external load-balancers can be used alongside or instead of ServiceLB, but in most cases, replacement load-balancer controllers such as MetalLB or Kube-VIP are a better choice. ::: ## Prerequisites All nodes in this example are running Ubuntu 20.04. For both examples, assume that a [HA K3s cluster with embedded etcd](../datastore/ha-embedded.md) has been installed on 3 nodes. Each k3s server is configured with: ```yaml # /etc/rancher/k3s/config.yaml token: lb-cluster-gd tls-san: 10.10.10.100 ``` The nodes have hostnames and IPs of: * server-1: `10.10.10.50` * server-2: `10.10.10.51` * server-3: `10.10.10.52` Two additional nodes for load balancing are configured with hostnames and IPs of: * lb-1: `10.10.10.98` * lb-2: `10.10.10.99` Three additional nodes exist with hostnames and IPs of: * agent-1: `10.10.10.101` * agent-2: `10.10.10.102` * agent-3: `10.10.10.103` ## Setup Load Balancer [HAProxy](http://www.haproxy.org/) is an open source option that provides a TCP load balancer. It also supports HA for the load balancer itself, ensuring redundancy at all levels. See [HAProxy Documentation](http://docs.haproxy.org/2.8/intro.html) for more info. Additionally, we will use KeepAlived to generate a virtual IP (VIP) that will be used to access the cluster. See [KeepAlived Documentation](https://www.keepalived.org/documentation/) for more info. 1) Install HAProxy and KeepAlived: ```bash sudo apt-get install haproxy keepalived ``` 2) Add the following to `/etc/haproxy/haproxy.cfg` on lb-1 and lb-2: ``` frontend k3s-frontend bind *:6443 mode tcp option tcplog default_backend k3s-backend backend k3s-backend mode tcp option tcp-check balance roundrobin default-server inter 10s downinter 5s server server-1 10.10.10.50:6443 check server server-2 10.10.10.51:6443 check server server-3 10.10.10.52:6443 check ``` 3) Add the following to `/etc/keepalived/keepalived.conf` on lb-1 and lb-2: ``` global_defs { enable_script_security script_user root } vrrp_script chk_haproxy { script 'killall -0 haproxy' # faster than pidof interval 2 } vrrp_instance haproxy-vip { interface eth1 state # MASTER on lb-1, BACKUP on lb-2 priority # 200 on lb-1, 100 on lb-2 virtual_router_id 51 virtual_ipaddress { 10.10.10.100/24 } track_script { chk_haproxy } } ``` 6) Restart HAProxy and KeepAlived on lb-1 and lb-2: ```bash systemctl restart haproxy systemctl restart keepalived ``` 5) On agent-1, agent-2, and agent-3, run the following command to install k3s and join the cluster: ```bash curl -sfL https://get.k3s.io | K3S_TOKEN=lb-cluster-gd sh -s - agent --server https://10.10.10.100:6443 ``` You can now use `kubectl` from server node to interact with the cluster. ```bash root@server-1 $ k3s kubectl get nodes -A NAME STATUS ROLES AGE VERSION agent-1 Ready 32s v1.27.3+k3s1 agent-2 Ready 20s v1.27.3+k3s1 agent-3 Ready 9s v1.27.3+k3s1 server-1 Ready control-plane,etcd,master 4m22s v1.27.3+k3s1 server-2 Ready control-plane,etcd,master 3m58s v1.27.3+k3s1 server-3 Ready control-plane,etcd,master 3m12s v1.27.3+k3s1 ``` ## Nginx Load Balancer :::danger Nginx does not natively support a High Availability (HA) configuration. If setting up an HA cluster, having a single load balancer in front of K3s will reintroduce a single point of failure. ::: [Nginx Open Source](http://nginx.org/) provides a TCP load balancer. See [Using nginx as HTTP load balancer](https://nginx.org/en/docs/http/load_balancing.html) for more info. 1) Create a `nginx.conf` file on lb-1 with the following contents: ``` events {} stream { upstream k3s_servers { server 10.10.10.50:6443; server 10.10.10.51:6443; server 10.10.10.52:6443; } server { listen 6443; proxy_pass k3s_servers; } } ``` 2) Run the Nginx load balancer on lb-1: Using docker: ```bash docker run -d --restart unless-stopped \ -v ${PWD}/nginx.conf:/etc/nginx/nginx.conf \ -p 6443:6443 \ nginx:stable ``` Or [install nginx](https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-open-source/) and then run: ```bash cp nginx.conf /etc/nginx/nginx.conf systemctl start nginx ``` 3) On agent-1, agent-2, and agent-3, run the following command to install k3s and join the cluster: ```bash curl -sfL https://get.k3s.io | K3S_TOKEN=lb-cluster-gd sh -s - agent --server https://10.10.10.98:6443 ``` You can now use `kubectl` from server node to interact with the cluster. ```bash root@server1 $ k3s kubectl get nodes -A NAME STATUS ROLES AGE VERSION agent-1 Ready 30s v1.27.3+k3s1 agent-2 Ready 22s v1.27.3+k3s1 agent-3 Ready 13s v1.27.3+k3s1 server-1 Ready control-plane,etcd,master 4m49s v1.27.3+k3s1 server-2 Ready control-plane,etcd,master 3m58s v1.27.3+k3s1 server-3 Ready control-plane,etcd,master 3m16s v1.27.3+k3s1 ``` ## Kube-VIP :::info This example configures kube-vip in ARP (layer‑2) mode to provide a Virtual IP (VIP) and a control-plane load balancer. The manifest below deploys kube-vip as a DaemonSet on control-plane nodes and announces the VIP on the node network. Adjust the interface and subnet to match your environment. ::: [Kube-VIP](https://kube-vip.io/) provides a virtual IP and load balancer for the Kubernetes control plane and for Services of type LoadBalancer. The instructions below show how to generate and deploy the daemonset manifest on K3s control-plane nodes. 1) Install the RBAC manifest: ```bash curl -fsSL https://kube-vip.io/manifests/rbac.yaml -o /var/lib/rancher/k3s/server/manifests/kube-vip-rbac.yaml ``` or ```bash kubectl apply -f https://kube-vip.io/manifests/rbac.yaml ``` 2) Deploy the kube-vip daemonset: - Update these values before applying: - vip_interface: the network interface name on each control-plane host (e.g. ens160, eth0). - address: the VIP (example: 10.10.10.100). - node affinity: ensure it matches your control-plane node labels (node-role.kubernetes.io/control-plane vs master). - The list of environment variables is available in the [documentation](https://kube-vip.io/docs/installation/flags/#environment-variables). Apply the following manifest using the `kubectl apply -f` command. ``` /* Detailed source-code truncated for AI context efficiency. */ ``` 3) Verify kube-vip and VIP announcement ```bash # check pods kubectl -n kube-system get pods -l app.kubernetes.io/name=kube-vip-ds # on a control-plane host, confirm the VIP is in the ARP/neighbor table ip neigh show | grep 10.10.10.100 ``` 4) TLS certificate note If K3s was installed before the VIP was added to the API server certificate SANs, kubelets and API clients will not trust the server certificate for the VIP. To include the VIP in server certificates: ```bash # Stop K3s service systemctl stop k3s # Rotate server certificates to include the configured tls-san/VIP k3s certificate rotate # Start K3s service systemctl start k3s ``` - After rotation, verify API access using the VIP: kubectl --server=https://10.10.10.100:6443 get nodes