## File: README.md # flannel [](https://scorecard.dev/viewer/?uri=github.com%2Fflannel-io%2Fflannel) Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes. ## How it works Flannel runs a small, single binary agent called `flanneld` on each host, and is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space. Flannel uses either the Kubernetes API or [etcd][etcd] directly to store the network configuration, the allocated subnets, and any auxiliary data (such as the host's public IP). Packets are forwarded using one of several [backend mechanisms][backends] including VXLAN and various cloud integrations. ### Networking details Platforms like Kubernetes assume that each container (pod) has a unique, routable IP inside the cluster. The advantage of this model is that it removes the port mapping complexities that come from sharing a single host IP. Flannel is responsible for providing a layer 3 IPv4 network between multiple nodes in a cluster. Flannel does not control how containers are networked to the host, only how the traffic is transported between hosts. However, flannel does provide a CNI plugin for Kubernetes and a guidance on integrating with Docker. Flannel is focused on networking. While the `flanneld` binary does not natively enforce Network Policies, the Flannel project provides ways to add policy support to your cluster. ### Network Policy For Network Policy enforcement, options include: * **Integrated controller:** The [Flannel Helm chart](https://github.com/flannel-io/flannel/tree/master/chart/kube-flannel) has a `netpol.enabled` option that deploys the [Kubernetes SIGs network policy controller](https://github.com/kubernetes-sigs/kube-network-policies) alongside Flannel. * **Third-party controllers:** Use Flannel for connectivity and another project for policy, for example [Calico](https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/install-for-flannel) or CNI chaining with [Cilium](https://docs.cilium.io/en/stable/installation/cni-chaining/). See [Documentation/netpol.md](Documentation/netpol.md) for configuration details. ## Getting started on Kubernetes The easiest way to deploy flannel with Kubernetes is to use one of several deployment tools and distributions that network clusters with flannel by default. For example, [K3s][k3s] sets up flannel in the Kubernetes clusters it creates using the open source [K3s Installer][k3s-installer] to drive the setup process. Though not required, it's recommended that flannel uses the Kubernetes API as its backing store which avoids the need to deploy a discrete `etcd` cluster for `flannel`. This `flannel` mode is known as the *kube subnet manager*. ### Deploying flannel manually Flannel can be added to any existing Kubernetes cluster though it's simplest to add `flannel` before any pods using the pod network have been started. For Kubernetes v1.17+ #### Deploying Flannel with kubectl ```bash kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml ``` If you use custom `podCIDR` (not `10.244.0.0/16`) you first need to download the above manifest and modify the network to match your one. #### Deploying Flannel with Helm ```bash # Needs manual creation of namespace to avoid Helm error kubectl create ns kube-flannel kubectl label --overwrite ns kube-flannel pod-security.kubernetes.io/enforce=privileged helm repo add flannel https://flannel-io.github.io/flannel/ helm install flannel --set podCidr="10.244.0.0/16" --namespace kube-flannel flannel/flannel ``` See [Kubernetes](Documentation/kubernetes.md) for more details. In case a firewall is configured ensure to enable the right port used by the configured [backend][backends]. Flannel uses `portmap` as CNI network plugin by default; when deploying Flannel ensure that the [CNI Network plugins][Network-plugins] are installed in `/opt/cni/bin` the latest binaries can be downloaded with the following commands: ```bash ARCH=$(uname -m) case $ARCH in armv7*) ARCH="arm";; aarch64) ARCH="arm64";; x86_64) ARCH="amd64";; esac mkdir -p /opt/cni/bin curl -O -L https://github.com/containernetworking/plugins/releases/download/v1.7.1/cni-plugins-linux-$ARCH-v1.7.1.tgz tar -C /opt/cni/bin -xzf cni-plugins-linux-$ARCH-v1.7.1.tgz ``` Flannel requires the br_netfilter module to start and from version 1.30 kubeadm doesn't check if the module is installed and Flannel will not rightly start in case the module is missing. ## Getting started on Docker flannel is also widely used outside of Kubernetes. When deployed outside of Kubernetes, etcd is always used as the datastore. For more details integrating flannel with Docker see [Running](Documentation/running.md) ## Documentation - [Building (and releasing)](Documentation/building.md) - [Configuration](Documentation/configuration.md) - [Backends](Documentation/backends.md) - [Running](Documentation/running.md) - [Troubleshooting](Documentation/troubleshooting.md) - [Projects integrating with flannel](Documentation/integrations.md) - [Organizations using flannel in production](ADOPTERS.md) ## Contact - Slack: - #k3s on [Rancher Users Slack](https://slack.rancher.io) - #flannel-users on [Calico Users Slack](https://slack.projectcalico.org) - Planning/Roadmap: [ROADMAP.md][roadmap], [milestones][milestones] - Bugs: [issues][flannel-issues] ## Community Meeting The Flannel Maintainer Community runs a meeting on the third Thursday of each month at 8:30 AM PST (16:30 UTC). This meeting is used to discuss issues, open pull requests, and other topics related to Flannel should the need arise. The meeting agenda and Teams link can be found here: [Flannel Community Meeting Agenda](https://docs.google.com/document/d/1kPMMFDhljWL8_CUZajrfL8Q9sdntd9vvUpe-UGhX5z8) ## Contributing See [CONTRIBUTING][contributing] for details on submitting patches and the contribution workflow. ## Reporting bugs & security vulnerabilities See [reporting bugs][reporting] for details about reporting any issues. For security issues, please first check our [security policy](SECURITY.md). ## Licensing Flannel is under the Apache 2.0 license. See the [LICENSE][license] file for details. [calico]: http://www.projectcalico.org [etcd]: https://go.etcd.io/etcd/v3 [contributing]: CONTRIBUTING.md [license]: https://github.com/flannel-io/flannel/blob/master/LICENSE [milestones]: https://github.com/flannel-io/flannel/milestones [flannel-issues]: https://github.com/flannel-io/flannel/issues [backends]: Documentation/backends.md [roadmap]: ROADMAP.md [reporting]: Documentation/reporting_bugs.md [k3s-installer]: https://github.com/k3s-io/k3s/#quick-start---install-script [k3s]: https://k3s.io/ [Network-plugins]: https://github.com/containernetworking/plugins --- ## File: Documentation/adrs/add-nftables-implementation.md # Add nftables implementation to flannel Date: 2024-02-01 ## Status Writing ## Context At the moment, flannel uses iptables to mask and route packets. Our implementation is based on the library from coreos (https://github.com/coreos/go-iptables). There are several issues with using iptables in flannel: * performance: packets are matched using a list so performance is O(n). This isn't very important for flannel because use few iptables rules anyway. * stability: ** rules must be purged then updated every time flannel needs to change a rule to keep the correct order ** there can be interferences with other k8s components using iptables as well (kube-proxy, kube-router...) * deprecation: nftables is pushed as a replacement for iptables in the kernel and in future distros including the future RHEL. References: - https://github.com/kubernetes/enhancements/blob/master/keps/sig-network/3866-nftables-proxy/README.md#motivation ## Current state In flannel code, all references to iptables are wrapped in the `iptables` package. The package provides the type `IPTableRule` to represent an individual rule. This type is almost entirely internal to the package so it would be easy to refactor the code to hide in favor of a more abstract type that would work for both iptables and nftables rules. Unfortunately the package doesn't provide an interface so in order to provide both an iptables-based and an nftables-based implementation this needs to be refactored. This package includes several Go interfaces (`IPTables`, `IPTablesError`) that are used for testing. ## Requirements Ideally, flannel will include both iptables and nftables implementation. These need to coexist in the code but will be mutually exclusive at runtime. The choice of which implementation to use will be triggered by an optional CLI flag. iptables will remain the default for the time being. Using nftables is an opportunity for optimising the rules deployed by flannel but we need to be careful about retro-compatibility with the current backend. Starting flannel in either mode should reset the other mode as best as possible to ensure that users don't need to reboot if they need to change mode. ## Architecture Currently, flannel uses two dedicated tables for its own rules: `FLANNEL-POSTRTG` and `FLANNEL-FWD`. * flannel adds rules to the `FORWARD` and `POSTROUTING` tables to direct traffic to its own tables. * rules in `FLANNEL-POSTRTG` are used to manage masquerading of the traffic to/from the pods * rules in `FLANNEL-FWD` are used to ensure that traffic to and from the flannel network can be forwarded With nftables, flannel would have its own dedicated table (`flannel`) with arbitrary chains and rules as needed. see https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT) ``` # !! untested example table flannel { chain flannel-postrtg { type nat hook postrouting priority 0; # kube-proxy meta mark 0x4000/0x4000 return # don't NAT traffic within overlay network ip saddr $pod_cidr ip daddr $cluster_cidr return ip saddr $cluster_cidr ip daddr $pod_cidr return # Prevent performing Masquerade on external traffic which arrives from a Node that owns the container/pod IP address ip saddr != $pod_cidr ip daddr $cluster_cidr return # NAT if it's not multicast traffic ip saddr $cluster_cidr ip daddr != 224.0.0.0/4 nat # Masquerade anything headed towards flannel from the host ip saddr != $cluster_cidr ip daddr $cluster_cidr nat } chain flannel-fwd { type filter hook input priority 0; policy drop; # allow traffic to be forwarded if it is to or from the flannel network range ip saddr flannelNetwork accept ip daddr flannelNetwork accept } } ``` ## nftables library We can either: * call the `nft` executable directly * use https://github.com/kubernetes-sigs/knftables which is developed for kube-proxy and should cover our use case ## Implementation steps * refactor current iptables code to better encapsulate iptables calls in the dedicated package * implement nftables mode that is the exact equivalent of the current iptables code * add similar unit tests and e2e test coverage * try to optimize the code using nftables-specific feature * integrate the new flag in k3s ## Decision --- ## File: Documentation/backends.md # Backends Flannel may be paired with several different backends. Once set, the backend should not be changed at runtime. VXLAN is the recommended choice. host-gw is recommended for more experienced users who want the performance improvement and whose infrastructure support it (typically it can't be used in cloud environments). UDP is suggested for debugging only or for very old kernels that don't support VXLAN. In case `firewalld` is enabled on the node the port used by the backend needs to be enabled with `firewall-cmd`: ``` firewall-cmd --permanent --zone=public --add-port=[port]/udp ``` For more information on configuration options for Tencent see [TencentCloud VPC Backend for Flannel][tencentcloud-vpc] ## Recommended backends ### VXLAN Use in-kernel VXLAN to encapsulate the packets. Type and options: * `Type` (string): `vxlan` * `VNI` (number): VXLAN Identifier (VNI) to be used. On Linux, defaults to 1. On Windows should be greater than or equal to 4096. * `Port` (number): UDP port to use for sending encapsulated packets. On Linux, defaults to kernel default, currently 8472, but on Windows, must be 4789. * `GBP` (Boolean): Enable [VXLAN Group Based Policy](https://github.com/torvalds/linux/commit/3511494ce2f3d3b77544c79b87511a4ddb61dc89). Defaults to `false`. GBP is not supported on Windows. * `DirectRouting` (Boolean): Enable direct routes (like `host-gw`) when the hosts are on the same subnet. VXLAN will only be used to encapsulate packets to hosts on different subnets. Defaults to `false`. DirectRouting is not supported on Windows. * `Learning` (Boolean): Linux only. Controls whether the VXLAN device uses MAC learning (`learning` on the kernel link). Defaults to `false`. * `MTU` (number): Desired MTU for the outgoing packets. If not defined, the MTU of the external interface is used. Linux only. * `MacPrefix` (String): Windows only. MAC address prefix for the VXLAN interface, format `xx-xx`. Defaults to `0E-2A`. * `Name` (String): Windows only. Name of the VXLAN network interface. Defaults to `flannel.` (for example `flannel.4096`). Starting with Ubuntu 21.10, vxlan support on Raspberry Pi has been moved into a separate kernel module. ``` sudo apt install linux-modules-extra-raspi ``` ### host-gw Use host-gw to create IP routes to subnets via remote machine IPs. Requires direct layer2 connectivity between hosts running flannel. host-gw provides good performance, with few dependencies, and easy set up. Type: * `Type` (string): `host-gw` ### WireGuard Use in-kernel [WireGuard](https://www.wireguard.com) to encapsulate and encrypt the packets. Type: * `Type` (string): `wireguard` * `PSK` (string): Optional. The pre shared key to use. Use `wg genpsk` to generate a key. * `ListenPort` (int): Optional. The udp port to listen on. Default is `51820`. * `ListenPortV6` (int): Optional. The udp port to listen on for ipv6. Default is `51821`. * `MTU` (number): Desired MTU for the outgoing packets if not defined the MTU of the external interface is used. * `Mode` (string): Optional. * separate - Use separate wireguard tunnels for ipv4 and ipv6 (default) * auto - Single wireguard tunnel for both address families; autodetermine the preferred peer address * ipv4 - Single wireguard tunnel for both address families; use ipv4 for the peer addresses * ipv6 - Single wireguard tunnel for both address families; use ipv6 for the peer addresses * `PersistentKeepaliveInterval` (int): Optional. Default is 0 (disabled). If no private key was generated before the private key is written to `/run/flannel/wgkey`. You can use environment `WIREGUARD_KEY_FILE` to change this path. The static names of the interfaces are `flannel-wg` and `flannel-wg-v6`. WireGuard tools like `wg show` can be used to debug interfaces and peers. Users of kernels < 5.6 need to [install](https://www.wireguard.com/install/) an additional Wireguard package. ### UDP Use UDP only for debugging if your network and kernel prevent you from using VXLAN or host-gw. Type and options: * `Type` (string): `udp` * `Port` (number): UDP port to use for sending encapsulated packets. Defaults to 8285. ## Experimental backends The following options are experimental and unsupported at this time. ### Alloc Alloc performs subnet allocation with no forwarding of data packets. Type: * `Type` (string): `alloc` ### TencentCloud VPC Use TencentCloud VPC to create IP routes in a [TencentCloud VPC route table](https://intl.cloud.tencent.com/product/vpc) when running in an TencentCloud VPC. This mitigates the need to create a separate flannel interface. Requirements: * Running on an CVM instance that is in an TencentCloud VPC. * Permission require `accessid` and `keysecret`. * `Type` (string): `tencent-vpc` * `AccessKeyID` (string): API access key ID. Can also be configured with environment ACCESS_KEY_ID. * `AccessKeySecret` (string): API access key secret. Can also be configured with environment ACCESS_KEY_SECRET. Route Limits: TencentCloud VPC limits the number of entries per route table to 50. [tencentcloud-vpc]: https://github.com/flannel-io/flannel/blob/master/Documentation/tencentcloud-vpc-backend.md ### IPIP Use in-kernel IPIP to encapsulate the packets. IPIP kind of tunnels is the simplest one. It has the lowest overhead, but can incapsulate only IPv4 unicast traffic, so you will not be able to setup OSPF, RIP or any other multicast-based protocol. Type: * `Type` (string): `ipip` * `DirectRouting` (Boolean): Enable direct routes (like `host-gw`) when the hosts are on the same subnet. IPIP will only be used to encapsulate packets to hosts on different subnets. Defaults to `false`. Note that there may exist two ipip tunnel device `tunl0` and `flannel.ipip`, this is expected and it's not a bug. `tunl0` is automatically created per network namespace by ipip kernel module on modprobe ipip module. It is the namespace default IPIP device with attributes local=any and remote=any. When receiving IPIP protocol packets, kernel will forward them to tunl0 as a fallback device if it can't find an option whose local/remote attribute matches their src/dst ip address more precisely. `flannel.ipip` is created by flannel to achieve one to many ipip network. ### IPSec Use in-kernel IPSec to encapsulate and encrypt the packets. [Strongswan](https://www.strongswan.org) is used at the IKEv2 daemon. A single pre-shared key is used for the initial key exchange between hosts and then Strongswan ensures that keys are rotated at regular intervals. Type: * `Type` (string): `ipsec` * `PSK` (string): Required. The pre shared key to use. It needs to be at least 96 characters long. One method for generating this key is to run `dd if=/dev/urandom count=48 bs=1 status=none | xxd -p -c 48` * `UDPEncap` (Boolean): Optional, defaults to false. Forces the use UDP encapsulation of packets which can help with some NAT gateways. * `ESPProposal` (string): Optional, defaults to `aes128gcm16-sha256-prfsha256-ecp256`. Change this string to choose another ESP Proposal. Hint: Add rules to your firewall: Open ports 50 (for ESP protocol), UDP 500 (for IKE, to manage encryption keys) and UDP 4500 (for IPSEC NAT-Traversal mode). #### Troubleshooting Logging * When flannel is run from a container, the Strongswan tools are installed. `swanctl` can be used for interacting with the charon and it provides a logs command. * Charon logs are also written to the stdout of the flannel process. Troubleshooting * `ip xfrm state` can be used to interact with the kernel's security association database. This can be used to show the current security associations (SA) and whether a host is successfully establishing ipsec connections to other hosts. * `ip xfrm policy` can be used to show the installed policies. Flannel installs three policies for each host it connects to. Flannel will not restore policies that are manually deleted (unless flannel is restarted). It will also not delete stale policies on startup. They can be removed by rebooting your host or by removing all ipsec state with `ip xfrm state flush && ip xfrm policy flush` and restarting flannel. --- ## File: Documentation/building.md # Building flannel The most reliable way to build flannel is by using Docker. ## Building in a Docker container To build flannel in a container run `make dist/flanneld-amd64`. You will now have a `flanneld-amd64` binary in the `dist` directory. ## Building for other platforms If you're not running `amd64` then you need to manually set `ARCH` before running `make`. For example, to produce a `flanneld-s390x` binary and image, run * ARCH=s390x make image If you want to cross-compile for a different platform (e.g. you're running `amd64` but you want to produce `arm` binaries) then you need the qemu-static binaries to be present in `/usr/bin`. They can be installed on Ubuntu with * `sudo apt-get install qemu-user-static` Then you should be able to set the ARCH as above * ARCH=arm make image ## Building a multi-arch image To build the multi-arch image of flannel locally, you need to install [Docker buildx](https://github.com/docker/buildx). Then you can use the following target: ``` make build-multi-arch ``` If you don't already have a builder running locally, you can this target to start it: ``` make buildx-create-builder ``` See the [buildx documentation](https://docs.docker.com/reference/cli/docker/buildx/) for more details. ## Running the tests locally To run the end-to-end tests locally, you need to installl [Docker compose](https://docs.docker.com/compose/install/). ## Building manually 1. Make sure you have required dependencies installed on your machine. * On Ubuntu, run `sudo apt-get install linux-libc-dev golang gcc`. If the golang version installed is not 1.7 or higher. Download the newest golang and install manually. To build the flannel.exe on windows, mingw-w64 is also needed. Run command `sudo apt-get install mingw-w64` * On Fedora/Redhat, run `sudo yum install kernel-headers golang gcc glibc-static`. 2. Git clone the flannel repo. It MUST be placed in your GOPATH under `github.com/flannel-io/flannel`: `cd $GOPATH/src; git clone https://github.com/flannel-io/flannel.git` 3. Run the build script, ensuring that `CGO_ENABLED=1`: `cd flannel; CGO_ENABLED=1 make dist/flanneld` for linux usage. Run the build script, ensuring that `CGO_ENABLED=1`: `cd flannel; CGO_ENABLED=1 make dist/flanneld.exe` for windows usage. --- ## File: Documentation/configuration.md # Configuration If the --kube-subnet-mgr argument is true, flannel reads its configuration from `/etc/kube-flannel/net-conf.json`. If the --kube-subnet-mgr argument is false, flannel reads its configuration from etcd. By default, it will read the configuration from `/coreos.com/network/config` (which can be overridden using `--etcd-prefix`). Use the `etcdctl` utility to set values in etcd. The value of the config is a JSON dictionary with the following keys: * `Network` (string): IPv4 network in CIDR format to use for the entire flannel network. (Mandatory if EnableIPv4 is true) * `IPv6Network` (string): IPv6 network in CIDR format to use for the entire flannel network. (Mandatory if EnableIPv6 is true) * `EnableIPv4` (bool): Enables ipv4 support Defaults to `true` * `EnableIPv6` (bool): Enables ipv6 support Defaults to `false` * `EnableNFTables` (bool): (EXPERIMENTAL) If set to true, flannel uses nftables instead of iptables to masquerade the traffic. Default to `false` * `SubnetLen` (integer): The size of the subnet allocated to each host. Defaults to 24 (i.e. /24) unless `Network` was configured to be smaller than a /22 in which case it is two less than the network. * `SubnetMin` (string): The beginning of IP range which the subnet allocation should start with. Defaults to the second subnet of `Network`. * `SubnetMax` (string): The end of the IP range at which the subnet allocation should end with. Defaults to the last subnet of `Network`. * `IPv6SubnetLen` (integer): The size of the ipv6 subnet allocated to each host. Defaults to 64 (i.e. /64) unless `Ipv6Network` was configured to be smaller than a /62 in which case it is two less than the network. * `IPv6SubnetMin` (string): The beginning of IPv6 range which the subnet allocation should start with. Defaults to the second subnet of `Ipv6Network`. * `IPv6SubnetMax` (string): The end of the IPv6 range at which the subnet allocation should end with. Defaults to the last subnet of `Ipv6Network`. * `Backend` (dictionary): Type of backend to use and specific configurations for that backend. The list of available backends and the keys that can be put into the this dictionary are listed in [Backends](backends.md). Defaults to `vxlan` backend. Subnet leases have a duration of 24 hours. Leases are renewed within 1 hour of their expiration, unless a different renewal margin is set with the ``--subnet-lease-renew-margin`` option. ## Example configuration JSON The following configuration illustrates the use of most options with `udp` backend. ```json { "Network": "10.0.0.0/8", "SubnetLen": 20, "SubnetMin": "10.10.0.0", "SubnetMax": "10.99.0.0", "Backend": { "Type": "udp", "Port": 7890 } } ``` ## Key command line options ``` /* Detailed source-code truncated for AI context efficiency. */ ``` MTU is calculated and set automatically by flannel. It then reports that value in `subnet.env`. This value can be changed as [backend](backends.md) config. ## Environment variables The command line options outlined above can also be specified via environment variables. For example `--etcd-endpoints=http://10.0.0.2:2379` is equivalent to `FLANNELD_ETCD_ENDPOINTS=http://10.0.0.2:2379` environment variable. Any command line option can be turned into an environment variable by prefixing it with `FLANNELD_`, stripping leading dashes, converting to uppercase and replacing all other dashes to underscores. `EVENT_QUEUE_DEPTH` is another environment variable to indicate the kubernetes scale. Set `EVENT_QUEUE_DEPTH` to adapter your cluster node numbers. If not set, default value is 5000. `CONT_WHEN_CACHE_NOT_READY` is environment variable to indicate if flanneld should continue even when the node informer cache is not fully sync'd yet. This can happen for large clusters (clusters with node capacity higher than `EVENT_QUEUE_DEPTH`). Set `CONT_WHEN_CACHE_NOT_READY` to "true" to let flanneld not fail startup for such large capacity clusters. ## Health Check Flannel provides two HTTP health check endpoints, both served on `--healthz-port` (disabled by default; set to a non-zero value to enable): - **`/healthz`** — liveness probe. Returns HTTP 200 whenever the `flanneld` process is running. - **`/readyz`** — readiness probe. Returns HTTP 200 only after flannel has completed startup: the iptables or nftables traffic rules (masquerade/forward) have been installed **and** the subnet environment file (`subnet.env`) has been written successfully. Returns HTTP 503 until that point. ## Dual-stack Flannel supports dual-stack mode. This means pods and services could use ipv4 and ipv6 at the same time. Currently, dual-stack is only supported for vxlan, wireguard or host-gw(linux) backends. Requirements: * v1.0.1 of flannel binary from [containernetworking/plugins](https://github.com/containernetworking/plugins) * Nodes must have an ipv4 and ipv6 address in the main interface * Nodes must have an ipv4 and ipv6 address default route * vxlan support ipv6 tunnel require kernel version >= 3.12 * When public IPv6 addresses are used, routing of the "IPv6Network" towards the cluster must be configured externally from flannel (see [#2289](https://github.com/flannel-io/flannel/issues/2289) for details) Configuration: * Set "EnableIPv6": true and the "IPv6Network", for example "IPv6Network": * "2001:cafe:42:0::/56" in the net-conf.json of the kube-flannel-cfg ConfigMap or in `/coreos.com/network/config` for etcd If everything works as expected, flanneld should generate a `/run/flannel/subnet.env` file with IPV6 subnet and network. For example: ```bash FLANNEL_NETWORK=10.42.0.0/16 FLANNEL_SUBNET=10.42.0.1/24 FLANNEL_IPV6_NETWORK=2001:cafe:42::/56 FLANNEL_IPV6_SUBNET=2001:cafe:42::1/64 FLANNEL_MTU=1450 FLANNEL_IPMASQ=true ``` ## IPv6 only To use an IPv6-only environment use the same configuration of the Dual-stack section to enable IPv6 and add "EnableIPv4": false in the net-conf.json of the kube-flannel-cfg ConfigMap. In case of IPv6-only setup, please use the docker.io IPv6-only endpoint as described in the following link: https://www.docker.com/blog/beta-ipv6-support-on-docker-hub-registry/ ## nftables mode To enable `nftables` mode in flannel, set `EnableNFTables` to true in flannel configuration. Note: to test with kube-proxy, use kubeadm with the following configuration: ```yaml apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: v1.29.0 controllerManager: extraArgs: feature-gates: NFTablesProxyMode=true --- apiVersion: kubeproxy.config.k8s.io/v1alpha1 kind: KubeProxyConfiguration mode: "nftables" featureGates: NFTablesProxyMode: true ``` --- ## File: Documentation/extension.md # Extension The `extension` backend provides an easy way for prototyping new backend types for flannel. It is _not_ recommended for production use, for example it doesn't have a built in retry mechanism. This backend has the following configuration * `Type` (string): `extension` * `PreStartupCommand` (string): Command to run before allocating a network to this host * The stdout of the process is captured and passed to the stdin of the SubnetAdd/Remove commands. * `PostStartupCommand` (string): Command to run after allocating a network to this host * The following environment variable is set * SUBNET - The subnet of the remote host that was added. * `SubnetAddCommand` (string): Command to run when a subnet is added * stdin - The output from `PreStartupCommand` is passed in. * The following environment variables are set * SUBNET - The ipv4 subnet of the remote host that was added. * IPV6SUBNET - The ipv6 subnet of the remote host that was added. * PUBLIC_IP - The public IP of the remote host. * PUBLIC_IPV6 - The public IPv6 of the remote host. * `SubnetRemoveCommand`(string): Command to run when a subnet is removed * stdin - The output from `PreStartupCommand` is passed in. * The following environment variables are set * SUBNET - The ipv4 subnet of the remote host that was removed. * IPV6SUBNET - The ipv6 subnet of the remote host that was removed. * PUBLIC_IP - The public IP of the remote host. * PUBLIC_IPV6 - The public IPv6 of the remote host. All commands are run through the `sh` shell and are run with the same permissions as the flannel daemon. ## Simple example (host-gw) To replicate the functionality of the host-gw plugin, there's no need for a startup command. The backend just needs to manage the route to subnets when they are added or removed. An example ```json { "Network": "10.0.0.0/16", "Backend": { "Type": "extension", "SubnetAddCommand": "ip route add \"$SUBNET\" via \"$PUBLIC_IP\"", "SubnetRemoveCommand": "ip route del \"$SUBNET\" via \"$PUBLIC_IP\"" } } ``` ## Complex example (vxlan) VXLAN is more complex. It needs to store the MAC address of the vxlan device when it's created and to make it available to the flannel daemon running on other hosts. The address of the vxlan device also needs to be set _after_ the subnet has been allocated. An example ```json { "Network": "10.50.0.0/16", "Backend": { "Type": "extension", "PreStartupCommand": "export VNI=1; export IF_NAME=flannel-vxlan; ip link del \"$IF_NAME\" 2>/dev/null; ip link add \"$IF_NAME\" type vxlan id \"$VNI\" dstport 8472 && cat \"/sys/class/net/${IF_NAME}/address\"", "PostStartupCommand": "export IF_NAME=flannel-vxlan; export SUBNET_IP=$(echo \"$SUBNET\" | cut -d'/' -f 1); ip addr add \"${SUBNET_IP}/32\" dev \"$IF_NAME\" && ip link set \"$IF_NAME\" up", "SubnetAddCommand": "export SUBNET_IP=$(echo \"$SUBNET\" | cut -d'/' -f 1); export IF_NAME=flannel-vxlan; read VTEP; ip route add \"$SUBNET\" nexthop via \"$SUBNET_IP\" dev \"$IF_NAME\" onlink && ip neigh replace \"$SUBNET_IP\" dev \"$IF_NAME\" lladdr \"$VTEP\" && bridge fdb add \"$VTEP\" dev \"$IF_NAME\" self dst \"$PUBLIC_IP\"" } } ``` --- ## File: Documentation/integrations.md # Integrations This document tracks projects that integrate with flannel. [Join the community](https://github.com/flannel-io/flannel/) and help us keep the list current. ## Projects [kube-network-policies](https://github.com/kubernetes-sigs/kube-network-policies): Network policies controller that can be deployed alongside Flannel. [Canal](https://projectcalico.docs.tigera.io/getting-started/kubernetes/flannel/flannel): Kubernetes CNI plugin that uses Calico for network policies and intra-node communications and Flannel for inter-node communications. [K3s](https://k3s.io/): Kubernetes distribution with flannel embedded as CNI. [RKE2](https://docs.rke2.io/): Kubernetes distribution packed with Canal as default CNI and can be configured with Flannel with support for windows. --- ## File: Documentation/kubernetes.md # kubeadm For information on deploying flannel manually, using the Kubernetes installer toolkit kubeadm, see [Installing Kubernetes on Linux with kubeadm][kubeadm]. NOTE: If `kubeadm` is used, then pass `--pod-network-cidr=10.244.0.0/16` to `kubeadm init` to ensure that the `podCIDR` is set. # kube-flannel.yaml The `flannel` manifest defines five things: 1. A `kube-flannel` with PodSecurity level set to *privileged*. 2. A ClusterRole and ClusterRoleBinding for Role Based Access Control (RBAC). 3. A service account for `flannel` to use. 4. A ConfigMap containing both a CNI configuration and a `flannel` configuration. The `network` in the `flannel` configuration should match the pod network CIDR. The choice of `backend` is also made here and defaults to VXLAN. 5. A DaemonSet for every architecture to deploy the `flannel` pod on each Node. The pod has two containers 1) the `flannel` daemon itself, and 2) an initContainer for deploying the CNI configuration to a location that the `kubelet` can read. When you run pods, they will be allocated IP addresses from the pod network CIDR. No matter which node those pods end up on, they will be able to communicate with each other. # Notes on securing flannel deployment As of Kubernetes v1.21, the [PodSecurityPolicy API was deprecated](https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) and it will be removed in v1.25. Thus, the `flannel` manifest does not use PodSecurityPolicy anymore. If you wish to use the [Pod Security Admission Controller](https://kubernetes.io/docs/concepts/security/pod-security-admission/) which was introduced to [replace PodSecurityPolicy](https://kubernetes.io/docs/tasks/configure-pod-container/migrate-from-psp/), you will need to deploy `flannel` in a namespace which allows the deployment of pods with `privileged` level. The `baseline` level is insufficient to deploy `flannel` and you will see the following error message: ``` Error creating: non-default capabilities (container "kube-flannel" must not include "NET_ADMIN", "NET_RAW" in securityContext.capabilities.add), host namespaces (hostNetwork=true), hostPath volumes (volumes "run", "cni-plugin", "cni", "xtables-lock") ``` The `kube-flannel.yaml` manifest deploys `flannel` in the `kube-flannel` namespace and enables the `privileged` level for this namespace. Thus, you will need to restrict access to this namespace if you wish to secure your cluster. If you want to deploy `flannel` securely in a shared namespace or want more fine-grained control over the pods deployed in your cluster, you can use a 3rd-party admission controller like [Kubewarden](https://kubewarden.io). Kubewarden provides policies that can replace features of PodSecurityPolicy like [capabilities-psp-policy](https://github.com/kubewarden/capabilities-psp-policy) and [hostpaths-psp-policy](https://github.com/kubewarden/hostpaths-psp-policy). Other options include [Kyverno](https://kyverno.io/policies/pod-security/) and [OPA Gatekeeper](https://github.com/open-policy-agent/gatekeeper). # Annotations Additional annotations can be configured on a specific node as parameters used when Flannel starts on that specific node * `flannel.alpha.coreos.com/node-public-ip`, `flannel.alpha.coreos.com/node-public-ipv6`: Define the used IP of the node in case the node has multiple interface it selects the interface with the configured IP for the backend tunnel. If configured when Flannel starts it'll be used as the `public-ip` and `public-ipv6` flag. * `flannel.alpha.coreos.com/public-ip-overwrite`, `flannel.alpha.coreos.com/public-ipv6-overwrite`: Allows to overwrite the public IP of a node that IP can be not configured on the node. Useful if the public IP can not determined from the node, e.G. because it is behind a NAT and the other nodes need to use it to create the tunnel. It can be automatically set to a nodes `ExternalIP` using the [flannel-node-annotator](https://github.com/alvaroaleman/flannel-node-annotator). See also the "NAT" section in [troubleshooting](./troubleshooting.md) if UDP checksums seem corrupted. ## Older versions of Kubernetes `kube-flannel.yaml` has some features that aren't compatible with older versions of Kubernetes, though flanneld itself should work with any version of Kubernetes. ### For Kubernetes v1.6~v1.15 If you see errors saying `found invalid field...` when you try to apply `kube-flannel.yaml` then you can try the "legacy" manifest file * `kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-old-manifests/kube-flannel-legacy.yml` This file does not bundle RBAC permissions. If you need those, run * `kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-old-manifests/kube-flannel-rbac.yml` If you didn't apply the `kube-flannel-rbac.yml` manifest and you need to, you'll see errors in your flanneld logs about failing to connect. * `Failed to create SubnetManager: error retrieving pod spec...` ### For Kubernetes v1.16 `kube-flannel.yaml` uses `ClusterRole` & `ClusterRoleBinding` of `rbac.authorization.k8s.io/v1`. When you use Kubernetes v1.16, you should replace `rbac.authorization.k8s.io/v1` to `rbac.authorization.k8s.io/v1beta1` because `rbac.authorization.k8s.io/v1` had become GA from Kubernetes v1.17. ### For Kubernetes <= v1.24 As of Kubernetes v1.21, the [PodSecurityPolicy API was deprecated](https://kubernetes.io/blog/2021/04/06/podsecuritypolicy-deprecation-past-present-and-future/) and it will be removed in v1.25. Thus, the `flannel` manifest does not use PodSecurityPolicy anymore. If you still wish to use it, you can use `kube-flannel-psp.yaml` instead of `kube-flannel.yaml`. Please note that if you use a Kubernetes version >= 1.21, you will see a deprecation warning for the PodSecurityPolicy API. # Troubleshooting See [troubleshooting](troubleshooting.md) [kubeadm]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/