flannel Documentation

Flannel is a network fabric for containers, designed for Kubernetes

9,450 stars Go Go 1 file · ~9,704 tokens #docker#docker-image#flannel#go#kubernetes#network#overlay-network#subnet
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy flannel on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name flannel -p 5201:5201 --restart always flannel-io/flannel:latest

File: README.md

flannel

Flannel is a simple and easy way to configure a layer 3 network fabric designed for Kubernetes.

How it works

Flannel runs a small, single binary agent called flanneld on each host, and is responsible for allocating a subnet lease to each host out of a larger, preconfigured address space.
Flannel uses either the Kubernetes API or etcd directly to store the network configuration, the allocated subnets, and any auxiliary data (such as the host's public IP).
Packets are forwarded using one of several backend mechanisms including VXLAN and various cloud integrations.

Networking details

Platforms like Kubernetes assume that each container (pod) has a unique, routable IP inside the cluster.
The advantage of this model is that it removes the port mapping complexities that come from sharing a single host IP.

Flannel is responsible for providing a layer 3 IPv4 network between multiple nodes in a cluster. Flannel does not control how containers are networked to the host, only how the traffic is transported between hosts. However, flannel does provide a CNI plugin for Kubernetes and a guidance on integrating with Docker.

Flannel is focused on networking. While the flanneld binary does not natively enforce Network Policies, the Flannel project provides ways to add policy support to your cluster.

Network Policy

For Network Policy enforcement, options include:

See Documentation/netpol.md for configuration details.

Getting started on Kubernetes

The easiest way to deploy flannel with Kubernetes is to use one of several deployment tools and distributions that network clusters with flannel by default. For example, K3s sets up flannel in the Kubernetes clusters it creates using the open source K3s Installer to drive the setup process.

Though not required, it's recommended that flannel uses the Kubernetes API as its backing store which avoids the need to deploy a discrete etcd cluster for flannel. This flannel mode is known as the kube subnet manager.

Deploying flannel manually

Flannel can be added to any existing Kubernetes cluster though it's simplest to add flannel before any pods using the pod network have been started.

For Kubernetes v1.17+

Deploying Flannel with kubectl

bash
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

If you use custom podCIDR (not 10.244.0.0/16) you first need to download the above manifest and modify the network to match your one.

Deploying Flannel with Helm

bash
# Needs manual creation of namespace to avoid Helm error
kubectl create ns kube-flannel
kubectl label --overwrite ns kube-flannel pod-security.kubernetes.io/enforce=privileged

helm repo add flannel https://flannel-io.github.io/flannel/
helm install flannel --set podCidr="10.244.0.0/16" --namespace kube-flannel flannel/flannel

See Kubernetes for more details.

In case a firewall is configured ensure to enable the right port used by the configured backend.

Flannel uses portmap as CNI network plugin by default; when deploying Flannel ensure that the CNI Network plugins are installed in /opt/cni/bin the latest binaries can be downloaded with the following commands:

bash
ARCH=$(uname -m)
  case $ARCH in
    armv7*) ARCH="arm";;
    aarch64) ARCH="arm64";;
    x86_64) ARCH="amd64";;
  esac
mkdir -p /opt/cni/bin
curl -O -L https://github.com/containernetworking/plugins/releases/download/v1.7.1/cni-plugins-linux-$ARCH-v1.7.1.tgz
tar -C /opt/cni/bin -xzf cni-plugins-linux-$ARCH-v1.7.1.tgz

Flannel requires the br_netfilter module to start and from version 1.30 kubeadm doesn't check if the module is installed and Flannel will not rightly start in case the module is missing.

Getting started on Docker

flannel is also widely used outside of Kubernetes. When deployed outside of Kubernetes, etcd is always used as the datastore. For more details integrating flannel with Docker see Running

Documentation

Contact

Community Meeting

The Flannel Maintainer Community runs a meeting on the third Thursday of each month at 8:30 AM PST (16:30 UTC). This meeting is used to discuss issues, open pull requests, and other topics related to Flannel should the need arise.

The meeting agenda and Teams link can be found here: Flannel Community Meeting Agenda

Contributing

See CONTRIBUTING for details on submitting patches and the contribution workflow.

Reporting bugs & security vulnerabilities

See reporting bugs for details about reporting any issues.

For security issues, please first check our security policy.

Licensing

Flannel is under the Apache 2.0 license. See the LICENSE file for details.

File: Documentation/adrs/add-nftables-implementation.md

Add nftables implementation to flannel

Date: 2024-02-01

Status

Writing

Context

At the moment, flannel uses iptables to mask and route packets.
Our implementation is based on the library from coreos (https://github.com/coreos/go-iptables).

There are several issues with using iptables in flannel:

  • performance: packets are matched using a list so performance is O(n). This isn't very important for flannel because use few iptables rules anyway.
  • stability:
    ** rules must be purged then updated every time flannel needs to change a rule to keep the correct order
    ** there can be interferences with other k8s components using iptables as well (kube-proxy, kube-router...)
  • deprecation: nftables is pushed as a replacement for iptables in the kernel and in future distros including the future RHEL.

References:

Current state

In flannel code, all references to iptables are wrapped in the iptables package.

The package provides the type IPTableRule to represent an individual rule. This type is almost entirely internal to the package so it would be easy to refactor the code to hide in favor of a more abstract type that would work for both iptables and nftables rules.

Unfortunately the package doesn't provide an interface so in order to provide both an iptables-based and an nftables-based implementation this needs to be refactored.

This package includes several Go interfaces (IPTables, IPTablesError) that are used for testing.

Requirements

Ideally, flannel will include both iptables and nftables implementation. These need to coexist in the code but will be mutually exclusive at runtime.

The choice of which implementation to use will be triggered by an optional CLI flag.
iptables will remain the default for the time being.

Using nftables is an opportunity for optimising the rules deployed by flannel but we need to be careful about retro-compatibility with the current backend.

Starting flannel in either mode should reset the other mode as best as possible to ensure that users don't need to reboot if they need to change mode.

Architecture

Currently, flannel uses two dedicated tables for its own rules: FLANNEL-POSTRTG and FLANNEL-FWD.

  • flannel adds rules to the FORWARD and POSTROUTING tables to direct traffic to its own tables.
  • rules in FLANNEL-POSTRTG are used to manage masquerading of the traffic to/from the pods
  • rules in FLANNEL-FWD are used to ensure that traffic to and from the flannel network can be forwarded

With nftables, flannel would have its own dedicated table (flannel) with arbitrary chains and rules as needed.

see https://wiki.nftables.org/wiki-nftables/index.php/Performing_Network_Address_Translation_(NAT)

text
# !! untested example
table flannel {
    chain flannel-postrtg {
        type nat hook postrouting priority 0; 
        # kube-proxy
        meta mark 0x4000/0x4000 return
        # don't NAT traffic within overlay network 
        ip saddr $pod_cidr ip daddr $cluster_cidr return 
        ip saddr $cluster_cidr ip daddr $pod_cidr return 
        # Prevent performing Masquerade on external traffic which arrives from a Node that owns the container/pod IP address
        ip saddr != $pod_cidr ip daddr $cluster_cidr return 
        # NAT if it's not multicast traffic
        ip saddr $cluster_cidr ip daddr != 224.0.0.0/4 nat 
        # Masquerade anything headed towards flannel from the host
        ip saddr != $cluster_cidr ip daddr $cluster_cidr nat 
    }

    chain flannel-fwd {
        type filter hook input priority 0; policy drop;
        # allow traffic to be forwarded if it is to or from the flannel network range
        ip saddr flannelNetwork accept
        ip daddr flannelNetwork accept
    }
}

nftables library

We can either:

Implementation steps

  • refactor current iptables code to better encapsulate iptables calls in the dedicated package
  • implement nftables mode that is the exact equivalent of the current iptables code
  • add similar unit tests and e2e test coverage
  • try to optimize the code using nftables-specific feature
  • integrate the new flag in k3s

Decision

File: Documentation/backends.md

Backends

Flannel may be paired with several different backends. Once set, the backend should not be changed at runtime.

VXLAN is the recommended choice. host-gw is recommended for more experienced users who want the performance improvement and whose infrastructure support it (typically it can't be used in cloud environments). UDP is suggested for debugging only or for very old kernels that don't support VXLAN.

In case firewalld is enabled on the node the port used by the backend needs to be enabled with firewall-cmd:

text
firewall-cmd --permanent --zone=public --add-port=[port]/udp

For more information on configuration options for Tencent see TencentCloud VPC Backend for Flannel

VXLAN

Use in-kernel VXLAN to encapsulate the packets.

Type and options:

  • Type (string): vxlan
  • VNI (number): VXLAN Identifier (VNI) to be used. On Linux, defaults to 1. On Windows should be greater than or equal to 4096.
  • Port (number): UDP port to use for sending encapsulated packets. On Linux, defaults to kernel default, currently 8472, but on Windows, must be 4789.
  • GBP (Boolean): Enable VXLAN Group Based Policy. Defaults to false. GBP is not supported on Windows.
  • DirectRouting (Boolean): Enable direct routes (like host-gw) when the hosts are on the same subnet. VXLAN will only be used to encapsulate packets to hosts on different subnets. Defaults to false. DirectRouting is not supported on Windows.
  • Learning (Boolean): Linux only. Controls whether the VXLAN device uses MAC learning (learning on the kernel link). Defaults to false.
  • MTU (number): Desired MTU for the outgoing packets. If not defined, the MTU of the external interface is used. Linux only.
  • MacPrefix (String): Windows only. MAC address prefix for the VXLAN interface, format xx-xx. Defaults to 0E-2A.
  • Name (String): Windows only. Name of the VXLAN network interface. Defaults to flannel.<VNI> (for example flannel.4096).

Starting with Ubuntu 21.10, vxlan support on Raspberry Pi has been moved into a separate kernel module.

text
sudo apt install linux-modules-extra-raspi

host-gw

Use host-gw to create IP routes to subnets via remote machine IPs. Requires direct layer2 connectivity between hosts running flannel.

host-gw provides good performance, with few dependencies, and easy set up.

Type:

  • Type (string): host-gw

WireGuard

Use in-kernel WireGuard to encapsulate and encrypt the packets.

Type:

  • Type (string): wireguard
  • PSK (string): Optional. The pre shared key to use. Use wg genpsk to generate a key.
  • ListenPort (int): Optional. The udp port to listen on. Default is 51820.
  • ListenPortV6 (int): Optional. The udp port to listen on for ipv6. Default is 51821.
  • MTU (number): Desired MTU for the outgoing packets if not defined the MTU of the external interface is used.
  • Mode (string): Optional.
    • separate - Use separate wireguard tunnels for ipv4 and ipv6 (default)
    • auto - Single wireguard tunnel for both address families; autodetermine the preferred peer address
    • ipv4 - Single wireguard tunnel for both address families; use ipv4 for
      the peer addresses
    • ipv6 - Single wireguard tunnel for both address families; use ipv6 for
      the peer addresses
  • PersistentKeepaliveInterval (int): Optional. Default is 0 (disabled).

If no private key was generated before the private key is written to /run/flannel/wgkey. You can use environment WIREGUARD_KEY_FILE to change this path.

The static names of the interfaces are flannel-wg and flannel-wg-v6. WireGuard tools like wg show can be used to debug interfaces and peers.

Users of kernels < 5.6 need to install an additional Wireguard package.

UDP

Use UDP only for debugging if your network and kernel prevent you from using VXLAN or host-gw.

Type and options:

  • Type (string): udp
  • Port (number): UDP port to use for sending encapsulated packets. Defaults to 8285.

Experimental backends

The following options are experimental and unsupported at this time.

Alloc

Alloc performs subnet allocation with no forwarding of data packets.

Type:

  • Type (string): alloc

TencentCloud VPC

Use TencentCloud VPC to create IP routes in a TencentCloud VPC route table when running in an TencentCloud VPC. This mitigates the need to create a separate flannel interface.

Requirements:

  • Running on an CVM instance that is in an TencentCloud VPC.
  • Permission require accessid and keysecret.
    • Type (string): tencent-vpc
    • AccessKeyID (string): API access key ID. Can also be configured with environment ACCESS_KEY_ID.
    • AccessKeySecret (string): API access key secret. Can also be configured with environment ACCESS_KEY_SECRET.

Route Limits: TencentCloud VPC limits the number of entries per route table to 50.

IPIP

Use in-kernel IPIP to encapsulate the packets.

IPIP kind of tunnels is the simplest one. It has the lowest overhead, but can incapsulate only IPv4 unicast traffic, so you will not be able to setup OSPF, RIP or any other multicast-based protocol.

Type:

  • Type (string): ipip
  • DirectRouting (Boolean): Enable direct routes (like host-gw) when the hosts are on the same subnet. IPIP will only be used to encapsulate packets to hosts on different subnets. Defaults to false.

Note that there may exist two ipip tunnel device tunl0 and flannel.ipip, this is expected and it's not a bug.
tunl0 is automatically created per network namespace by ipip kernel module on modprobe ipip module. It is the namespace default IPIP device with attributes local=any and remote=any.
When receiving IPIP protocol packets, kernel will forward them to tunl0 as a fallback device if it can't find an option whose local/remote attribute matches their src/dst ip address more precisely.
flannel.ipip is created by flannel to achieve one to many ipip network.

IPSec

Use in-kernel IPSec to encapsulate and encrypt the packets.

Strongswan is used at the IKEv2 daemon. A single pre-shared key is used for the initial key exchange between hosts and then Strongswan ensures that keys are rotated at regular intervals.

Type:

  • Type (string): ipsec
  • PSK (string): Required. The pre shared key to use. It needs to be at least 96 characters long. One method for generating this key is to run dd if=/dev/urandom count=48 bs=1 status=none | xxd -p -c 48
  • UDPEncap (Boolean): Optional, defaults to false. Forces the use UDP encapsulation of packets which can help with some NAT gateways.
  • ESPProposal (string): Optional, defaults to aes128gcm16-sha256-prfsha256-ecp256. Change this string to choose another ESP Proposal.

Hint:
Add rules to your firewall: Open ports 50 (for ESP protocol), UDP 500 (for IKE, to manage encryption keys) and UDP 4500 (for IPSEC NAT-Traversal mode).

Troubleshooting

Logging

  • When flannel is run from a container, the Strongswan tools are installed. swanctl can be used for interacting with the charon and it provides a logs command.
  • Charon logs are also written to the stdout of the flannel process.

Troubleshooting

  • ip xfrm state can be used to interact with the kernel's security association database. This can be used to show the current security associations (SA) and whether a host is successfully establishing ipsec connections to other hosts.
  • ip xfrm policy can be used to show the installed policies. Flannel installs three policies for each host it connects to.

Flannel will not restore policies that are manually deleted (unless flannel is restarted). It will also not delete stale policies on startup. They can be removed by rebooting your host or by removing all ipsec state with ip xfrm state flush && ip xfrm policy flush and restarting flannel.

File: Documentation/building.md

Building flannel

The most reliable way to build flannel is by using Docker.

Building in a Docker container

To build flannel in a container run make dist/flanneld-amd64.
You will now have a flanneld-amd64 binary in the dist directory.

Building for other platforms

If you're not running amd64 then you need to manually set ARCH before running make. For example, to produce a
flanneld-s390x binary and image, run

  • ARCH=s390x make image

If you want to cross-compile for a different platform (e.g. you're running amd64 but you want to produce arm binaries) then you need the qemu-static binaries to be present in /usr/bin. They can be installed on Ubuntu with

  • sudo apt-get install qemu-user-static

Then you should be able to set the ARCH as above

  • ARCH=arm make image

Building a multi-arch image

To build the multi-arch image of flannel locally, you need to install Docker buildx.
Then you can use the following target:

text
make build-multi-arch

If you don't already have a builder running locally, you can this target to start it:

text
make buildx-create-builder

See the buildx documentation for more details.

Running the tests locally

To run the end-to-end tests locally, you need to installl Docker compose.

Building manually

  1. Make sure you have required dependencies installed on your machine.
    • On Ubuntu, run sudo apt-get install linux-libc-dev golang gcc.
      If the golang version installed is not 1.7 or higher. Download the newest golang and install manually.
      To build the flannel.exe on windows, mingw-w64 is also needed. Run command sudo apt-get install mingw-w64
    • On Fedora/Redhat, run sudo yum install kernel-headers golang gcc glibc-static.
  2. Git clone the flannel repo. It MUST be placed in your GOPATH under github.com/flannel-io/flannel: cd $GOPATH/src; git clone https://github.com/flannel-io/flannel.git
  3. Run the build script, ensuring that CGO_ENABLED=1: cd flannel; CGO_ENABLED=1 make dist/flanneld for linux usage.
    Run the build script, ensuring that CGO_ENABLED=1: cd flannel; CGO_ENABLED=1 make dist/flanneld.exe for windows usage.

File: Documentation/configuration.md

Configuration

If the --kube-subnet-mgr argument is true, flannel reads its configuration from /etc/kube-flannel/net-conf.json.

If the --kube-subnet-mgr argument is false, flannel reads its configuration from etcd.
By default, it will read the configuration from /coreos.com/network/config (which can be overridden using --etcd-prefix).

Use the etcdctl utility to set values in etcd.

The value of the config is a JSON dictionary with the following keys:

  • Network (string): IPv4 network in CIDR format to use for the entire flannel network. (Mandatory if EnableIPv4 is true)

  • IPv6Network (string): IPv6 network in CIDR format to use for the entire flannel network. (Mandatory if EnableIPv6 is true)

  • EnableIPv4 (bool): Enables ipv4 support
    Defaults to true

  • EnableIPv6 (bool): Enables ipv6 support
    Defaults to false

  • EnableNFTables (bool): (EXPERIMENTAL) If set to true, flannel uses nftables instead of iptables to masquerade the traffic.
    Default to false

  • SubnetLen (integer): The size of the subnet allocated to each host.
    Defaults to 24 (i.e. /24) unless Network was configured to be smaller than a /22 in which case it is two less than the network.

  • SubnetMin (string): The beginning of IP range which the subnet allocation should start with.
    Defaults to the second subnet of Network.

  • SubnetMax (string): The end of the IP range at which the subnet allocation should end with.
    Defaults to the last subnet of Network.

  • IPv6SubnetLen (integer): The size of the ipv6 subnet allocated to each host.
    Defaults to 64 (i.e. /64) unless Ipv6Network was configured to be smaller than a /62 in which case it is two less than the network.

  • IPv6SubnetMin (string): The beginning of IPv6 range which the subnet allocation should start with.
    Defaults to the second subnet of Ipv6Network.

  • IPv6SubnetMax (string): The end of the IPv6 range at which the subnet allocation should end with.
    Defaults to the last subnet of Ipv6Network.

  • Backend (dictionary): Type of backend to use and specific configurations for that backend.
    The list of available backends and the keys that can be put into the this dictionary are listed in Backends.
    Defaults to vxlan backend.

Subnet leases have a duration of 24 hours. Leases are renewed within 1 hour of their expiration,
unless a different renewal margin is set with the --subnet-lease-renew-margin option.

Example configuration JSON

The following configuration illustrates the use of most options with udp backend.

json
{
	"Network": "10.0.0.0/8",
	"SubnetLen": 20,
	"SubnetMin": "10.10.0.0",
	"SubnetMax": "10.99.0.0",
	"Backend": {
		"Type": "udp",
		"Port": 7890
	}
}

Key command line options

text
/* Detailed source-code truncated for AI context efficiency. */

MTU is calculated and set automatically by flannel. It then reports that value in subnet.env. This value can be changed as backend config.

Environment variables

The command line options outlined above can also be specified via environment variables.
For example --etcd-endpoints=http://10.0.0.2:2379 is equivalent to FLANNELD_ETCD_ENDPOINTS=http://10.0.0.2:2379 environment variable.
Any command line option can be turned into an environment variable by prefixing it with FLANNELD_, stripping leading dashes, converting to uppercase and replacing all other dashes to underscores.

EVENT_QUEUE_DEPTH is another environment variable to indicate the kubernetes scale. Set EVENT_QUEUE_DEPTH to adapter your cluster node numbers. If not set, default value is 5000.

CONT_WHEN_CACHE_NOT_READY is environment variable to indicate if flanneld should continue even when the node informer cache is not fully sync'd yet. This can happen for large clusters (clusters with node capacity higher than EVENT_QUEUE_DEPTH). Set CONT_WHEN_CACHE_NOT_READY to "true" to let flanneld not fail startup for such large capacity clusters.

Health Check

Flannel provides two HTTP health check endpoints, both served on --healthz-port (disabled by default; set to a non-zero value to enable):

  • /healthz — liveness probe. Returns HTTP 200 whenever the flanneld process is running.
  • /readyz — readiness probe. Returns HTTP 200 only after flannel has completed startup: the iptables or nftables traffic rules (masquerade/forward) have been installed and the subnet environment file (subnet.env) has been written successfully. Returns HTTP 503 until that point.

Dual-stack

Flannel supports dual-stack mode. This means pods and services could use ipv4 and ipv6 at the same time. Currently, dual-stack is only supported for vxlan, wireguard or host-gw(linux) backends.

Requirements:

  • v1.0.1 of flannel binary from containernetworking/plugins
  • Nodes must have an ipv4 and ipv6 address in the main interface
  • Nodes must have an ipv4 and ipv6 address default route
  • vxlan support ipv6 tunnel require kernel version >= 3.12
  • When public IPv6 addresses are used, routing of the "IPv6Network" towards the cluster must be configured externally from flannel (see #2289 for details)

Configuration:

  • Set "EnableIPv6": true and the "IPv6Network", for example "IPv6Network": * "2001:cafe:42:0::/56" in the net-conf.json of the kube-flannel-cfg ConfigMap or in /coreos.com/network/config for etcd

If everything works as expected, flanneld should generate a /run/flannel/subnet.env file with IPV6 subnet and network. For example:

bash
FLANNEL_NETWORK=10.42.0.0/16
FLANNEL_SUBNET=10.42.0.1/24
FLANNEL_IPV6_NETWORK=2001:cafe:42::/56
FLANNEL_IPV6_SUBNET=2001:cafe:42::1/64
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

IPv6 only

To use an IPv6-only environment use the same configuration of the Dual-stack section to enable IPv6 and add "EnableIPv4": false in the net-conf.json of the kube-flannel-cfg ConfigMap. In case of IPv6-only setup, please use the docker.io IPv6-only endpoint as described in the following link: https://www.docker.com/blog/beta-ipv6-support-on-docker-hub-registry/

nftables mode

To enable nftables mode in flannel, set EnableNFTables to true in flannel configuration.

Note: to test with kube-proxy, use kubeadm with the following configuration:

yaml
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
kubernetesVersion: v1.29.0
controllerManager:
  extraArgs:
    feature-gates: NFTablesProxyMode=true
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: "nftables"
featureGates:
  NFTablesProxyMode: true

File: Documentation/extension.md

Extension

The extension backend provides an easy way for prototyping new backend types for flannel.

It is not recommended for production use, for example it doesn't have a built in retry mechanism.

This backend has the following configuration

  • Type (string): extension
  • PreStartupCommand (string): Command to run before allocating a network to this host
    • The stdout of the process is captured and passed to the stdin of the SubnetAdd/Remove commands.
  • PostStartupCommand (string): Command to run after allocating a network to this host
    • The following environment variable is set
      * SUBNET - The subnet of the remote host that was added.
  • SubnetAddCommand (string): Command to run when a subnet is added
    • stdin - The output from PreStartupCommand is passed in.
    • The following environment variables are set
      • SUBNET - The ipv4 subnet of the remote host that was added.
      • IPV6SUBNET - The ipv6 subnet of the remote host that was added.
      • PUBLIC_IP - The public IP of the remote host.
      • PUBLIC_IPV6 - The public IPv6 of the remote host.
  • SubnetRemoveCommand(string): Command to run when a subnet is removed
    • stdin - The output from PreStartupCommand is passed in.
      • The following environment variables are set
        • SUBNET - The ipv4 subnet of the remote host that was removed.
        • IPV6SUBNET - The ipv6 subnet of the remote host that was removed.
        • PUBLIC_IP - The public IP of the remote host.
        • PUBLIC_IPV6 - The public IPv6 of the remote host.

All commands are run through the sh shell and are run with the same permissions as the flannel daemon.

Simple example (host-gw)

To replicate the functionality of the host-gw plugin, there's no need for a startup command.

The backend just needs to manage the route to subnets when they are added or removed.

An example

json
{
  "Network": "10.0.0.0/16",
  "Backend": {
    "Type": "extension",
    "SubnetAddCommand": "ip route add \"$SUBNET\" via \"$PUBLIC_IP\"",
    "SubnetRemoveCommand": "ip route del \"$SUBNET\" via \"$PUBLIC_IP\""
  }
}

Complex example (vxlan)

VXLAN is more complex. It needs to store the MAC address of the vxlan device when it's created and to make it available to the flannel daemon running on other hosts.
The address of the vxlan device also needs to be set after the subnet has been allocated.

An example

json
{
  "Network": "10.50.0.0/16",
  "Backend": {
    "Type": "extension",
    "PreStartupCommand": "export VNI=1; export IF_NAME=flannel-vxlan; ip link del \"$IF_NAME\" 2>/dev/null; ip link add \"$IF_NAME\" type vxlan id \"$VNI\" dstport 8472 && cat \"/sys/class/net/${IF_NAME}/address\"",
    "PostStartupCommand": "export IF_NAME=flannel-vxlan; export SUBNET_IP=$(echo \"$SUBNET\" | cut -d'/' -f 1); ip addr add \"${SUBNET_IP}/32\" dev \"$IF_NAME\" && ip link set \"$IF_NAME\" up",
    "SubnetAddCommand": "export SUBNET_IP=$(echo \"$SUBNET\" | cut -d'/' -f 1); export IF_NAME=flannel-vxlan; read VTEP; ip route add \"$SUBNET\" nexthop via \"$SUBNET_IP\" dev \"$IF_NAME\" onlink && ip neigh replace \"$SUBNET_IP\" dev \"$IF_NAME\" lladdr \"$VTEP\" && bridge fdb add \"$VTEP\" dev \"$IF_NAME\" self dst \"$PUBLIC_IP\""
  }
}

File: Documentation/integrations.md

Integrations

This document tracks projects that integrate with flannel. Join the community and help us keep the list current.

Projects

kube-network-policies: Network policies controller that can be deployed alongside Flannel.

Canal: Kubernetes CNI plugin that uses Calico for network policies and intra-node communications and Flannel for inter-node communications.

K3s: Kubernetes distribution with flannel embedded as CNI.

RKE2: Kubernetes distribution packed with Canal as default CNI and can be configured with Flannel with support for windows.

File: Documentation/kubernetes.md

kubeadm

For information on deploying flannel manually, using the Kubernetes installer toolkit kubeadm, see Installing Kubernetes on Linux with kubeadm.

NOTE: If kubeadm is used, then pass --pod-network-cidr=10.244.0.0/16 to kubeadm init to ensure that the podCIDR is set.

kube-flannel.yaml

The flannel manifest defines five things:

  1. A kube-flannel with PodSecurity level set to privileged.
  2. A ClusterRole and ClusterRoleBinding for Role Based Access Control (RBAC).
  3. A service account for flannel to use.
  4. A ConfigMap containing both a CNI configuration and a flannel configuration. The network in the flannel configuration should match the pod network CIDR. The choice of backend is also made here and defaults to VXLAN.
  5. A DaemonSet for every architecture to deploy the flannel pod on each Node. The pod has two containers 1) the flannel daemon itself, and 2) an initContainer for deploying the CNI configuration to a location that the kubelet can read.

When you run pods, they will be allocated IP addresses from the pod network CIDR. No matter which node those pods end up on, they will be able to communicate with each other.

Notes on securing flannel deployment

As of Kubernetes v1.21, the PodSecurityPolicy API was deprecated and it will be removed in v1.25. Thus, the flannel manifest does not use PodSecurityPolicy anymore.

If you wish to use the Pod Security Admission Controller which was introduced to replace PodSecurityPolicy, you will need to deploy flannel in a namespace which allows the deployment of pods with privileged level. The baseline level is insufficient to deploy flannel and you will see the following error message:

text
Error creating: non-default capabilities (container "kube-flannel" must not include "NET_ADMIN", "NET_RAW" in securityContext.capabilities.add), host namespaces (hostNetwork=true), hostPath volumes (volumes "run", "cni-plugin", "cni", "xtables-lock")

The kube-flannel.yaml manifest deploys flannel in the kube-flannel namespace and enables the privileged level for this namespace.
Thus, you will need to restrict access to this namespace if you wish to secure your cluster.

If you want to deploy flannel securely in a shared namespace or want more fine-grained control over the pods deployed in your cluster, you can use a 3rd-party admission controller like Kubewarden. Kubewarden provides policies that can replace features of PodSecurityPolicy like capabilities-psp-policy and hostpaths-psp-policy.

Other options include Kyverno and OPA Gatekeeper.

Annotations

Additional annotations can be configured on a specific node as parameters used when Flannel starts on that specific node

  • flannel.alpha.coreos.com/node-public-ip, flannel.alpha.coreos.com/node-public-ipv6: Define the used IP of the node in case the node has multiple interface it selects the interface with the configured IP for the backend tunnel. If configured when Flannel starts it'll be used as the public-ip and public-ipv6 flag.
  • flannel.alpha.coreos.com/public-ip-overwrite, flannel.alpha.coreos.com/public-ipv6-overwrite: Allows to overwrite the public IP of a node that IP can be not configured on the node. Useful if the public IP can not determined from the node, e.G. because it is behind a NAT and the other nodes need to use it to create the tunnel. It can be automatically set to a nodes ExternalIP using the flannel-node-annotator.
    See also the "NAT" section in troubleshooting if UDP checksums seem corrupted.

Older versions of Kubernetes

kube-flannel.yaml has some features that aren't compatible with older versions of Kubernetes, though flanneld itself should work with any version of Kubernetes.

For Kubernetes v1.6~v1.15

If you see errors saying found invalid field... when you try to apply kube-flannel.yaml then you can try the "legacy" manifest file

  • kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-old-manifests/kube-flannel-legacy.yml

This file does not bundle RBAC permissions. If you need those, run

  • kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-old-manifests/kube-flannel-rbac.yml

If you didn't apply the kube-flannel-rbac.yml manifest and you need to, you'll see errors in your flanneld logs about failing to connect.

  • Failed to create SubnetManager: error retrieving pod spec...

For Kubernetes v1.16

kube-flannel.yaml uses ClusterRole & ClusterRoleBinding of rbac.authorization.k8s.io/v1. When you use Kubernetes v1.16, you should replace rbac.authorization.k8s.io/v1 to rbac.authorization.k8s.io/v1beta1 because rbac.authorization.k8s.io/v1 had become GA from Kubernetes v1.17.

For Kubernetes <= v1.24

As of Kubernetes v1.21, the PodSecurityPolicy API was deprecated and it will be removed in v1.25. Thus, the flannel manifest does not use PodSecurityPolicy anymore.

If you still wish to use it, you can use kube-flannel-psp.yaml instead of kube-flannel.yaml. Please note that if you use a Kubernetes version >= 1.21, you will see a deprecation warning for the PodSecurityPolicy API.

Troubleshooting

See troubleshooting

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#docker Browse docker catalog → #docker-image Browse docker-image catalog → #flannel Browse flannel catalog → #go Browse go catalog → #kubernetes Browse kubernetes catalog → #network Browse network catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →