kubespray Documentation

Deploy a Production Ready Kubernetes Cluster

18,657 stars Jinja 1 file · ~19,951 tokens #ansible#aws#bare-metal#gce#hacktoberfest#high-availability#k8s-sig-cluster-lifecycle#kubernetes
RAW Doc

Docker & Self-Hosted Deployment Verified OCI

Deploy kubespray on any server, homelab, or cloud container platform.

1-Click Terminal Command
bash (docker run) 
docker run -d --name kubespray --restart always kubernetes-sigs/kubespray:latest

File: README.md

Deploy a Production Ready Kubernetes Cluster

If you have questions, check the documentation at kubespray.io and join us on the kubernetes slack, channel #kubespray.
You can get your invite here

  • Can be deployed on AWS, GCE, Azure, OpenStack, vSphere, Equinix Metal (bare metal), Oracle Cloud Infrastructure (Experimental), or Baremetal
  • Highly available cluster
  • Composable (Choice of the network plugin for instance)
  • Supports most popular Linux distributions
  • Continuous integration tests

Quick Start

Below are several ways to use Kubespray to deploy a Kubernetes cluster.

Docker

Ensure you have installed Docker then

shellsession
docker run --rm -it --mount type=bind,source="$(pwd)"/inventory/sample,dst=/inventory \
  --mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa \
  quay.io/kubespray/kubespray:v2.31.0 bash
# Inside the container you may now run the kubespray playbooks:
ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml

Ansible

Usage

See Getting started

Collection

See here if you wish to use this repository as an Ansible collection

Vagrant

For Vagrant we need to install Python dependencies for provisioning tasks.
Check that Python and pip are installed:

shellsession
python -V && pip -V

If this returns the version of the software, you're good to go. If not, download and install Python from here https://www.python.org/downloads/source/

Install Ansible according to Ansible installation guide
then run the following step:

shellsession
vagrant up

Documents

Supported Linux Distributions

  • Flatcar Container Linux by Kinvolk
  • Debian Bookworm, Bullseye, Trixie
  • Ubuntu 22.04, 24.04
  • CentOS Stream / RHEL 9, 10
  • Fedora 39, 40, 41, 42
  • Fedora CoreOS (see fcos Note)
  • openSUSE Leap 15.x/Tumbleweed
  • Oracle Linux 9, 10
  • Alma Linux 9, 10
  • Rocky Linux 9, 10 (experimental in 10: see Rocky Linux 10 notes)
  • Kylin Linux Advanced Server V10 (experimental: see kylin linux notes)
  • Amazon Linux 2 (experimental: see amazon linux notes)
  • UOS Linux (experimental: see uos linux notes)
  • openEuler (experimental: see openEuler notes)

Note:

  • Upstart/SysV init based OS types are not supported.
  • Kernel requirements (please read if the OS kernel version is < 4.19).

Supported Components

Container Runtime Notes

  • The CRI-O minor version should match the Kubernetes minor version.

Requirements

  • Minimum required version of Kubernetes is v1.34.0

  • Ansible v2.14+, Jinja 2.11+ and python-netaddr is installed on the machine that will run Ansible commands

  • The target servers must have access to the Internet in order to pull docker images. Otherwise, additional configuration is required (See Offline Environment)

  • The target servers are configured to allow IPv4 forwarding.

  • If using IPv6 for pods and services, the target servers are configured to allow IPv6 forwarding.

  • The firewalls are not managed, you'll need to implement your own rules the way you used to.
    in order to avoid any issue during deployment you should disable your firewall.

  • If kubespray is run from non-root user account, correct privilege escalation method
    should be configured in the target servers. Then the ansible_become flag
    or command parameters --become or -b should be specified.

Hardware:
These limits are safeguarded by Kubespray. Actual requirements for your workload can differ. For a sizing guide go to the Building Large Clusters guide.

  • Control Plane
    • Memory: 2 GB
  • Worker Node
    • Memory: 1 GB

Network Plugins

You can choose among ten network plugins. (default: calico, except Vagrant uses flannel)

  • flannel: gre/vxlan (layer 2) networking.

  • Calico is a networking and network policy provider. Calico supports a flexible set of networking options
    designed to give you the most efficient networking across a range of situations, including non-overlay
    and overlay networks, with or without BGP. Calico uses the same engine to enforce network policy for hosts,
    pods, and (if using Istio and Envoy) applications at the service mesh layer.

  • cilium: layer 3/4 networking (as well as layer 7 to protect and secure application protocols), supports dynamic insertion of BPF bytecode into the Linux kernel to implement security services, networking and visibility logic.

  • kube-ovn: Kube-OVN integrates the OVN-based Network Virtualization with Kubernetes. It offers an advanced Container Network Fabric for Enterprises.

  • kube-router: Kube-router is a L3 CNI for Kubernetes networking aiming to provide operational
    simplicity and high performance: it uses IPVS to provide Kube Services Proxy (if setup to replace kube-proxy),
    iptables for network policies, and BGP for ods L3 networking (with optionally BGP peering with out-of-cluster BGP peers).
    It can also optionally advertise routes to Kubernetes cluster Pods CIDRs, ClusterIPs, ExternalIPs and LoadBalancerIPs.

  • macvlan: Macvlan is a Linux network driver. Pods have their own unique Mac and Ip address, connected directly the physical (layer 2) network.

  • multus: Multus is a meta CNI plugin that provides multiple network interface support to pods. For each interface Multus delegates CNI calls to secondary CNI plugins such as Calico, macvlan, etc.

  • custom_cni : You can specify some manifests that will be applied to the clusters to bring you own CNI and use non-supported ones by Kubespray.
    See tests/files/custom_cni/README.md and tests/files/custom_cni/values.yamlfor an example with a CNI provided by a Helm Chart.

The network plugin to use is defined by the variable kube_network_plugin. There is also an
option to leverage built-in cloud provider networking instead.
See also Network checker.

Ingress Plugins

  • metallb: the MetalLB bare-metal service LoadBalancer provider.

Community docs and resources

Tools and projects on top of Kubespray

CI Tests

CI/end-to-end tests sponsored by: CNCF, Equinix Metal, OVHcloud, ELASTX.

See the test matrix for details.

File: contrib/azurerm/README.md

Kubernetes on Azure with Azure Resource Group Templates

Provision the base infrastructure for a Kubernetes cluster by using Azure Resource Group Templates

Status

This will provision the base infrastructure (vnet, vms, nics, ips, ...) needed for Kubernetes in Azure into the specified
Resource Group. It will not install Kubernetes itself, this has to be done in a later step by yourself (using kubespray of course).

Requirements

Configuration through group_vars/all

You have to modify at least two variables in group_vars/all. The one is the cluster_name variable, it must be globally
unique due to some restrictions in Azure. The other one is the ssh_public_keys variable, it must be your ssh public
key to access your azure virtual machines. Most other variables should be self explanatory if you have some basic Kubernetes
experience.

Bastion host

You can enable the use of a Bastion Host by changing use_bastion in group_vars/all to true. The generated
templates will then include an additional bastion VM which can then be used to connect to the masters and nodes. The option
also removes all public IPs from all other VMs.

Generating and applying

To generate and apply the templates, call:

shell
./apply-rg.sh <resource_group_name>

If you change something in the configuration (e.g. number of nodes) later, you can call this again and Azure will
take care about creating/modifying whatever is needed.

Clearing a resource group

If you need to delete all resources from a resource group, simply call:

shell
./clear-rg.sh <resource_group_name>

WARNING this really deletes everything from your resource group, including everything that was later created by you!

Installing Ansible and the dependencies

Install Ansible according to Ansible installation guide

Generating an inventory for kubespray

After you have applied the templates, you can generate an inventory with this call:

shell
./generate-inventory.sh <resource_group_name>

It will create the file ./inventory which can then be used with kubespray, e.g.:

shell
cd kubespray-root-dir
ansible-playbook -i contrib/azurerm/inventory -u devops --become -e "@inventory/sample/group_vars/all/all.yml" cluster.yml

File: contrib/offline/README.md

Offline deployment

manage-offline-container-images.sh

Container image collecting script for offline deployment

This script has two features:
(1) Get container images from an environment which is deployed online, or set IMAGES_FROM_FILE
environment variable to get images from a file (e.g. temp/images.list after running the
./generate_list.sh script).
(2) Deploy local container registry and register the container images to the registry.

Step(1) should be done online site as a preparation, then we bring the gotten images
to the target offline environment. if images are from a private registry,
you need to set PRIVATE_REGISTRY environment variable.
Then we will run step(2) for registering the images to local registry, or to an existing
registry set by the DESTINATION_REGISTRY environment variable. By default, the local registry
will run on port 5000. This can be changed with the REGISTRY_PORT environment variable

Step(1) can be operated with:

shell
manage-offline-container-images.sh   create

Step(2) can be operated with:

shell
manage-offline-container-images.sh   register

generate_list.sh

This script generates the list of downloaded files and the list of container images by roles/kubespray_defaults/defaults/main/download.yml file.

Run this script will execute generate_list.yml playbook in kubespray root directory and generate four files,
all downloaded files url in files.list, all container images in images.list, jinja2 templates in *.template.

shell
./generate_list.sh
tree temp
temp
├── files.list
├── files.list.template
├── images.list
└── images.list.template
0 directories, 5 files

In some cases you may want to update some component version, you can declare version variables in ansible inventory file or group_vars,
then run ./generate_list.sh -i [inventory_file] to update file.list and images.list.

manage-offline-files.sh

This script will download all files according to temp/files.list and run nginx container to provide offline file download.

Step(1) generate files.list

shell
./generate_list.sh

Step(2) download files and run nginx container

shell
./manage-offline-files.sh

when nginx container is running, it can be accessed through http://127.0.0.1:8080/.

upload2artifactory.py

After the steps above, this script can recursively upload each file under a directory to a generic repository in Artifactory.

Environment Variables:

  • USERNAME -- At least permissions'Deploy/Cache' and 'Delete/Overwrite'.
  • TOKEN -- Generate this with 'Set Me Up' in your user.
  • BASE_URL -- The URL including the repository name.

Step(3) (optional) upload files to Artifactory

shell
cd kubespray/contrib/offline/offline-files
export USERNAME=admin
export TOKEN=...
export BASE_URL=https://artifactory.example.com/artifactory/a-generic-repo/
./upload2artifactory.py

File: contrib/terraform/openstack/README.md

Kubernetes on OpenStack with Terraform

Provision a Kubernetes cluster with Terraform on
OpenStack.

Status

This will install a Kubernetes cluster on an OpenStack Cloud. It should work on
most modern installs of OpenStack that support the basic services.

OpenStack Support Documentation

For OpenStack cloud provider configuration (Octavia load balancers, Cinder CSI storage,
network plugins like Calico), see the main OpenStack documentation.

Known compatible public clouds

Approach

The terraform configuration inspects variables found in
variables.tf to create resources in your OpenStack cluster.
There is a python script that reads the generated.tfstate
file to generate a dynamic inventory that is consumed by the main ansible script
to actually install kubernetes and stand up the cluster.

Networking

The configuration includes creating a private subnet with a router to the
external net. It will allocate floating IPs from a pool and assign them to the
hosts where that makes sense. You have the option of creating bastion hosts
inside the private subnet to access the nodes there. Alternatively, a node with
a floating IP can be used as a jump host to nodes without.

Using an existing router

It is possible to use an existing router instead of creating one. To use an
existing router set the router_id variable to the uuid of the router you wish
to use.

For example:

shellsession
router_id = "00c542e7-6f46-4535-ae95-984c7f0391a3"

Kubernetes Nodes

You can create many different kubernetes topologies by setting the number of
different classes of hosts. For each class there are options for allocating
floating IP addresses or not.

  • Control plane nodes with etcd
  • Control plane nodes without etcd
  • Standalone etcd hosts
  • Kubernetes worker nodes

Note that the Ansible script will report an invalid configuration if you wind up
with an even number of etcd instances since that is not a valid configuration. This
restriction includes standalone etcd nodes that are deployed in a cluster along with
control plane nodes with etcd replicas. As an example, if you have three control plane
nodes with etcd replicas and three standalone etcd nodes, the script will fail since
there are now six total etcd replicas.

GlusterFS shared file system

The Terraform configuration supports provisioning of an optional GlusterFS
shared file system based on a separate set of VMs. To enable this, you need to
specify:

  • the number of Gluster hosts (minimum 2)
  • Size of the non-ephemeral volumes to be attached to store the GlusterFS bricks
  • Other properties related to provisioning the hosts

Even if you are using Flatcar Container Linux by Kinvolk for your cluster, you will still
need the GlusterFS VMs to be based on either Debian or RedHat based images.
Flatcar Container Linux by Kinvolk cannot serve GlusterFS, but can connect to it through
binaries available on hyperkube v1.4.3_coreos.0 or higher.

Requirements

  • Install OpenTofu 1.9.0 or later
  • Install Ansible
  • you already have a suitable OS image in Glance
  • you already have a floating IP pool created
  • you have security groups enabled
  • you have a pair of keys generated that can be used to secure the new hosts

Module Architecture

The configuration is divided into four modules:

  • Network
  • Loadbalancer
  • IPs
  • Compute

The main reason for splitting the configuration up in this way is to easily
accommodate situations where floating IPs are limited by a quota or if you have
any external references to the floating IP (e.g. DNS) that would otherwise have
to be updated.

You can force your existing IPs by modifying the compute variables in
kubespray.tf as follows:

ini
k8s_master_fips = ["151.101.129.67"]
k8s_node_fips = ["151.101.129.68"]

Terraform

Terraform will be used to provision all of the OpenStack resources with base software as appropriate.

Configuration

Inventory files

Create an inventory directory for your cluster by copying the existing sample and linking the hosts script (used to build the inventory based on Terraform state):

shellsession
CLUSTER=your-cluster-name
cp -LRp contrib/terraform/openstack/sample-inventory inventory/$CLUSTER
cd inventory/$CLUSTER
ln -s ../../contrib/terraform/openstack/hosts
ln -s ../../contrib

This will be the base for subsequent Terraform commands.

OpenStack access and credentials

No provider variables are hardcoded inside variables.tf because Terraform
supports various authentication methods for OpenStack: the older script and
environment method (using openrc) as well as a newer declarative method, and
different OpenStack environments may support Identity API version 2 or 3.

These are examples and may vary depending on your OpenStack cloud provider,
for an exhaustive list on how to authenticate on OpenStack with Terraform
please read the OpenStack provider documentation.

The recommended authentication method is to describe credentials in a YAML file clouds.yaml that can be stored in:

  • the current directory
  • ~/.config/openstack
  • /etc/openstack

clouds.yaml:

yaml
clouds:
  mycloud:
    auth:
      auth_url: https://openstack:5000/v3
      username: "username"
      project_name: "projectname"
      project_id: projectid
      user_domain_name: "Default"
      password: "password"
    region_name: "RegionOne"
    interface: "public"
    identity_api_version: 3

If you have multiple clouds defined in your clouds.yaml file you can choose
the one you want to use with the environment variable OS_CLOUD:

shellsession
export OS_CLOUD=mycloud

Openrc method

When using classic environment variables, Terraform uses default OS_*
environment variables. A script suitable for your environment may be available
from Horizon under Project -> Compute -> Access & Security -> API Access.

With identity v2:

shellsession
source openrc

env | grep OS

OS_AUTH_URL=https://openstack:5000/v2.0
OS_PROJECT_ID=projectid
OS_PROJECT_NAME=projectname
OS_USERNAME=username
OS_PASSWORD=password
OS_REGION_NAME=RegionOne
OS_INTERFACE=public
OS_IDENTITY_API_VERSION=2

With identity v3:

shellsession
source openrc

env | grep OS

OS_AUTH_URL=https://openstack:5000/v3
OS_PROJECT_ID=projectid
OS_PROJECT_NAME=username
OS_PROJECT_DOMAIN_ID=default
OS_USERNAME=username
OS_PASSWORD=password
OS_REGION_NAME=RegionOne
OS_INTERFACE=public
OS_IDENTITY_API_VERSION=3
OS_USER_DOMAIN_NAME=Default

Terraform does not support a mix of DomainName and DomainID, choose one or the other:

  • provider.openstack: You must provide exactly one of DomainID or DomainName to authenticate by Username
shellsession
unset OS_USER_DOMAIN_NAME
export OS_USER_DOMAIN_ID=default

or

shellsession
unset OS_PROJECT_DOMAIN_ID
set OS_PROJECT_DOMAIN_NAME=Default

Cluster variables

The construction of the cluster is driven by values found in
variables.tf.

For your cluster, edit inventory/$CLUSTER/cluster.tfvars.

Variable Description
cluster_name All OpenStack resources will use the Terraform variablecluster_name (defaultexample) in their name to make it easier to track. For example the first compute resource will be namedexample-kubernetes-1.
az_list List of Availability Zones available in your OpenStack cluster.
network_name The name to be given to the internal network that will be generated
use_existing_network Use an existing network with the name of network_name. false by default
network_dns_domain (Optional) The dns_domain for the internal network that will be generated
dns_nameservers An array of DNS name server names to be used by hosts in the internal subnet.
floatingip_pool Name of the pool from which floating IPs will be allocated
k8s_master_fips A list of floating IPs that you have already pre-allocated; they will be attached to master nodes instead of creating new random floating IPs.
bastion_fips A list of floating IPs that you have already pre-allocated; they will be attached to bastion node instead of creating new random floating IPs.
external_net UUID of the external network that will be routed to
flavor_k8s_master,flavor_k8s_node,flavor_etcd, flavor_bastion,flavor_gfs_node Flavor depends on your openstack installation, you can get available flavor IDs through openstack flavor list
image,image_gfs, image_master Name of the image to use in provisioning the compute resources. Should already be loaded into glance.
image_uuid,image_gfs_uuid, image_master_uuid UUID of the image to use in provisioning the compute resources. Should already be loaded into glance.
ssh_user,ssh_user_gfs The username to ssh into the image with. This usually depends on the image you have selected
public_key_path Path on your local workstation to the public key file you wish to use in creating the key pairs
group_vars_path path to the inventory group vars directory, ./group_vars by default
number_of_k8s_masters, number_of_k8s_masters_no_floating_ip Number of nodes that serve as both master and etcd. These can be provisioned with or without floating IP addresses
number_of_k8s_masters_no_etcd, number_of_k8s_masters_no_floating_ip_no_etcd Number of nodes that serve as just master with no etcd. These can be provisioned with or without floating IP addresses
number_of_etcd Number of pure etcd nodes
number_of_k8s_nodes, number_of_k8s_nodes_no_floating_ip Kubernetes worker nodes. These can be provisioned with or without floating ip addresses.
number_of_bastions Number of bastion hosts to create. Scripts assume this is really just zero or one
number_of_gfs_nodes_no_floating_ip Number of gluster servers to provision.
gfs_volume_size_in_gb Size of the non-ephemeral volumes to be attached to store the GlusterFS bricks
supplementary_master_groups To add ansible groups to the masters, such as kube_node for tainting them as nodes, empty by default.
supplementary_node_groups To add ansible groups to the nodes, such as kube_ingress for running ingress controller pods, empty by default.
bastion_allowed_remote_ips List of CIDR allowed to initiate a SSH connection, ["0.0.0.0/0"] by default
bastion_allowed_remote_ipv6_ips List of IPv6 CIDR allowed to initiate a SSH connection, ["::/0"] by default
master_allowed_remote_ips List of CIDR blocks allowed to initiate an API connection, ["0.0.0.0/0"] by default
master_allowed_remote_ipv6_ips List of IPv6 CIDR blocks allowed to initiate an API connection, ["::/0"] by default
bastion_allowed_ports List of ports to open on bastion node, [] by default
bastion_allowed_ports_ipv6 List of ports to open on bastion node for IPv6 CIDR blocks, [] by default
k8s_allowed_remote_ips List of CIDR allowed to initiate a SSH connection, empty by default
k8s_allowed_remote_ips_ipv6 List of IPv6 CIDR allowed to initiate a SSH connection, empty by default
k8s_allowed_egress_ipv6_ips List of IPv6 CIDRs allowed for egress traffic, ["::/0"] by default
worker_allowed_ports List of ports to open on worker nodes, [{ "protocol" = "tcp", "port_range_min" = 30000, "port_range_max" = 32767, "remote_ip_prefix" = "0.0.0.0/0"}] by default
worker_allowed_ports_ipv6 List of ports to open on worker nodes for IPv6 CIDR blocks, [{ "protocol" = "tcp", "port_range_min" = 30000, "port_range_max" = 32767, "remote_ip_prefix" = "::/0"}, { "protocol" = "ipv6-icmp", "port_range_min" = 0, "port_range_max" = 0, "remote_ip_prefix" = "::/0"}] by default
master_allowed_ports List of ports to open on master nodes, expected format is [{ "protocol" = "tcp", "port_range_min" = 443, "port_range_max" = 443, "remote_ip_prefix" = "0.0.0.0/0"}], empty by default
master_allowed_ports_ipv6 List of ports to open on master nodes for IPv6 CIDR blocks, [{ "protocol" = "ipv6-icmp", "port_range_min" = 0, "port_range_max" = 0, "remote_ip_prefix" = "::/0"}] by default
node_root_volume_size_in_gb Size of the root volume for nodes, 0 to use ephemeral storage
master_root_volume_size_in_gb Size of the root volume for masters, 0 to use ephemeral storage
master_volume_type Volume type of the root volume for control_plane, 'Default' by default
node_volume_type Volume type of the root volume for nodes, 'Default' by default
gfs_root_volume_size_in_gb Size of the root volume for gluster, 0 to use ephemeral storage
etcd_root_volume_size_in_gb Size of the root volume for etcd nodes, 0 to use ephemeral storage
bastion_root_volume_size_in_gb Size of the root volume for bastions, 0 to use ephemeral storage
master_server_group_policy Enable and use openstack nova servergroups for masters with set policy, default: "" (disabled)
node_server_group_policy Enable and use openstack nova servergroups for nodes with set policy, default: "" (disabled)
etcd_server_group_policy Enable and use openstack nova servergroups for etcd with set policy, default: "" (disabled)
additional_server_groups Extra server groups to create. Set "policy" to the policy for the group, expected format is {"new-server-group" = {"policy" = "anti-affinity"}}, default: {} (to not create any extra groups)
use_access_ip If 1, nodes with floating IPs will transmit internal cluster traffic via floating IPs; if 0 private IPs will be used instead. Default value is 1.
port_security_enabled Allow to disable port security by setting this to false. true by default
force_null_port_security Set null instead of true or false for port_security. false by default
k8s_nodes Map containing worker node definition, see explanation below
k8s_masters Map containing master node definition, see explanation for k8s_nodes and sample-inventory/cluster.tfvars
k8s_master_loadbalancer_enabled Enable and use an Octavia load balancer for the K8s master nodes
k8s_master_loadbalancer_listener_port Define via which port the K8s Api should be exposed. 6443 by default
k8s_master_loadbalancer_server_port Define via which port the K8S api is available on the master nodes. 6443 by default
k8s_master_loadbalancer_public_ip Specify if an existing floating IP should be used for the load balancer. A new floating IP is assigned by default

k8s_nodes

Allows a custom definition of worker nodes giving the operator full control over individual node flavor and availability zone placement.
To enable the use of this mode set the number_of_k8s_nodes and number_of_k8s_nodes_no_floating_ip variables to 0.
Then define your desired worker node configuration using the k8s_nodes variable.
The az, flavor and floating_ip parameters are mandatory.
The optional parameter extra_groups (a comma-delimited string) can be used to define extra inventory group memberships for specific nodes.

text
/* Detailed source-code truncated for AI context efficiency. */

For example:

ini
k8s_nodes = {
  "1" = {
    "az" = "sto1"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "2" = {
    "az" = "sto2"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "3" = {
    "az" = "sto3"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
    "extra_groups" = "calico_rr"
  }
}

Would result in the same configuration as:

ini
number_of_k8s_nodes = 3
flavor_k8s_node = "83d8b44a-26a0-4f02-a981-079446926445"
az_list = ["sto1", "sto2", "sto3"]

And:

ini
k8s_nodes = {
  "ing-1" = {
    "az" = "sto1"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "ing-2" = {
    "az" = "sto2"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "ing-3" = {
    "az" = "sto3"
    "flavor" = "83d8b44a-26a0-4f02-a981-079446926445"
    "floating_ip" = true
  },
  "big-1" = {
    "az" = "sto1"
    "flavor" = "3f73fc93-ec61-4808-88df-2580d94c1a9b"
    "floating_ip" = false
  },
  "big-2" = {
    "az" = "sto2"
    "flavor" = "3f73fc93-ec61-4808-88df-2580d94c1a9b"
    "floating_ip" = false
  },
  "big-3" = {
    "az" = "sto3"
    "flavor" = "3f73fc93-ec61-4808-88df-2580d94c1a9b"
    "floating_ip" = false
  },
  "small-1" = {
    "az" = "sto1"
    "flavor" = "7a6a998f-ac7f-4fb8-a534-2175b254f75e"
    "floating_ip" = false
  },
  "small-2" = {
    "az" = "sto2"
    "flavor" = "7a6a998f-ac7f-4fb8-a534-2175b254f75e"
    "floating_ip" = false
  },
  "small-3" = {
    "az" = "sto3"
    "flavor" = "7a6a998f-ac7f-4fb8-a534-2175b254f75e"
    "floating_ip" = false
  }
}

Would result in three nodes in each availability zone each with their own separate naming,
flavor and floating ip configuration.

The "schema":

ini
k8s_nodes = {
  "key | node name suffix, must be unique" = {
    "az" = string
    "flavor" = string
    "floating_ip" = bool
  },
}

All values are required.

Terraform state files

In the cluster's inventory folder, the following files might be created (either by Terraform
or manually), to prevent you from pushing them accidentally they are in a
.gitignore file in the terraform/openstack directory :

  • .terraform
  • .tfvars
  • .tfstate
  • .tfstate.backup

You can still add them manually if you want to.

Initialization

Before Terraform can operate on your cluster you need to install the required
plugins. This is accomplished as follows:

shellsession
cd inventory/$CLUSTER
terraform -chdir="../../contrib/terraform/openstack" init

This should finish fairly quickly telling you Terraform has successfully initialized and loaded necessary modules.

Customizing with cloud-init

You can apply cloud-init based customization for the openstack instances before provisioning your cluster.
One common template is used for all instances. Adjust the file shown below:
contrib/terraform/openstack/modules/compute/templates/cloudinit.yaml.tmpl
For example, to enable openstack novnc access and ansible_user=root SSH access:

shellsession
#cloud-config
## in some cases novnc console access is required
## it requires ssh password to be set
ssh_pwauth: yes
chpasswd:
  list: |
    root:secret
  expire: False

## in some cases direct root ssh access via ssh key is required
disable_root: false

Provisioning cluster

You can apply the Terraform configuration to your cluster with the following command
issued from your cluster's inventory directory (inventory/$CLUSTER):

shellsession
terraform -chdir="../../contrib/terraform/openstack" apply -var-file=cluster.tfvars

if you chose to create a bastion host, this script will create
contrib/terraform/openstack/k8s_cluster.yml with an ssh command for Ansible to
be able to access your machines tunneling through the bastion's IP address. If
you want to manually handle the ssh tunneling to these machines, please delete
or move that file. If you want to use this, just leave it there, as ansible will
pick it up automatically.

Destroying cluster

You can destroy your new cluster with the following command issued from the cluster's inventory directory:

shellsession
terraform -chdir="../../contrib/terraform/openstack" destroy -var-file=cluster.tfvars

If you've started the Ansible run, it may also be a good idea to do some manual cleanup:

  • remove SSH keys from the destroyed cluster from your ~/.ssh/known_hosts file
  • clean up any temporary cache files: rm /tmp/$CLUSTER-*

Debugging

You can enable debugging output from Terraform by setting
OS_DEBUG to 1 andTF_LOG toDEBUG before running the Terraform command.

Terraform output

Terraform can output values that are useful for configure Neutron/Octavia LBaaS or Cinder persistent volume provisioning as part of your Kubernetes deployment:

  • private_subnet_id: the subnet where your instances are running is used for openstack_lbaas_subnet_id
  • floating_network_id: the network_id where the floating IP are provisioned is used for openstack_lbaas_floating_network_id

Ansible

Node access

SSH

Ensure your local ssh-agent is running and your ssh key has been added. This
step is required by the terraform provisioner:

shellsession
eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa

If you have deployed and destroyed a previous iteration of your cluster, you will need to clear out any stale keys from your SSH "known hosts" file ( ~/.ssh/known_hosts).

Metadata variables

The python script that reads the
generated.tfstate file to generate a dynamic inventory recognizes
some variables within a "metadata" block, defined in a "resource"
block (example):

ini
resource "openstack_compute_instance_v2" "example" {
    ...
    metadata {
        ssh_user = "ubuntu"
        prefer_ipv6 = true
        python_bin = "/usr/bin/python3"
    }
    ...
}

As the example shows, these let you define the SSH username for
Ansible, a Python binary which is needed by Ansible if
/usr/bin/python doesn't exist, and whether the IPv6 address of the
instance should be preferred over IPv4.

Bastion host

Bastion access will be determined by:

  • Your choice on the amount of bastion hosts (set by number_of_bastions terraform variable).
  • The existence of nodes/masters with floating IPs (set by number_of_k8s_masters, number_of_k8s_nodes, number_of_k8s_masters_no_etcd terraform variables).

If you have a bastion host, your ssh traffic will be directly routed through it. This is regardless of whether you have masters/nodes with a floating IP assigned.
If you don't have a bastion host, but at least one of your masters/nodes have a floating IP, then ssh traffic will be tunneled by one of these machines.

So, either a bastion host, or at least master/node with a floating IP are required.

Test access

Make sure you can connect to the hosts. Note that Flatcar Container Linux by Kinvolk will have a state FAILED due to Python not being present. This is okay, because Python will be installed during bootstrapping, so long as the hosts are not UNREACHABLE.

shellsession
$ ansible -i inventory/$CLUSTER/hosts -m ping all
example-k8s_node-1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
example-etcd-1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}
example-k8s-master-1 | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

If it fails try to connect manually via SSH. It could be something as simple as a stale host key.

Configure cluster variables

Edit inventory/$CLUSTER/group_vars/all/all.yml:

  • bin_dir:
yml
# Directory where the binaries will be installed
# Default:
# bin_dir: /usr/local/bin
# For Flatcar Container Linux by Kinvolk:
bin_dir: /opt/bin
  • external_cloud_provider:
yml
external_cloud_provider: openstack
  • Only if K8s < v1.31 - cloud_provider:
yml
cloud_provider: openstack

Edit inventory/$CLUSTER/group_vars/k8s_cluster/k8s_cluster.yml:

  • Set variable kube_network_plugin to your desired networking plugin.
yml
# Choose network plugin (calico or flannel)
# Can also be set to 'cloud', which lets the cloud provider setup appropriate routing
kube_network_plugin: flannel
  • Set variable resolvconf_mode
yml
# Can be docker_dns, host_resolvconf or none
# Default:
# resolvconf_mode: docker_dns
# For Flatcar Container Linux by Kinvolk:
resolvconf_mode: host_resolvconf
  • Set max amount of attached cinder volume per host (default 256)
yml
node_volume_attach_limit: 26

Deploy Kubernetes

shellsession
ansible-playbook --become -i inventory/$CLUSTER/hosts cluster.yml

This will take some time as there are many tasks to run.

Kubernetes

Set up kubectl

  1. Install kubectl on your workstation
  2. Add a route to the internal IP of a master node (if needed):
shellsession
sudo route add [master-internal-ip] gw [router-ip]

or

shellsession
sudo route add -net [internal-subnet]/24 gw [router-ip]
  1. List Kubernetes certificates & keys:
shellsession
ssh [os-user]@[master-ip] sudo ls /etc/kubernetes/ssl/
  1. Get admin's certificates and keys:
shellsession
ssh [os-user]@[master-ip] sudo cat /etc/kubernetes/ssl/admin-kube-master-1-key.pem > admin-key.pem
ssh [os-user]@[master-ip] sudo cat /etc/kubernetes/ssl/admin-kube-master-1.pem > admin.pem
ssh [os-user]@[master-ip] sudo cat /etc/kubernetes/ssl/ca.pem > ca.pem
  1. Configure kubectl:
shellsession
$ kubectl config set-cluster default-cluster --server=https://[master-internal-ip]:6443 \
    --certificate-authority=ca.pem

$ kubectl config set-credentials default-admin \
    --certificate-authority=ca.pem \
    --client-key=admin-key.pem \
    --client-certificate=admin.pem

$ kubectl config set-context default-system --cluster=default-cluster --user=default-admin
$ kubectl config use-context default-system
  1. Check it:
shellsession
kubectl version

GlusterFS

GlusterFS is not deployed by the standard cluster.yml playbook, see the
GlusterFS playbook documentation
for instructions.

Basically you will install Gluster as

shellsession
ansible-playbook --become -i inventory/$CLUSTER/hosts ./contrib/network-storage/glusterfs/glusterfs.yml

Relevant Resources

What's next

Try out your new Kubernetes cluster with the Hello Kubernetes service.

Appendix

Migration from `number_of_k8s_nodes*` to `k8s_nodes`

If you currently have a cluster defined using the number_of_k8s_nodes* variables and wish
to migrate to the k8s_nodes style you can do it like so:

text
/* Detailed source-code truncated for AI context efficiency. */

Of course for nodes without floating ips those steps can be omitted.

File: contrib/terraform/hetzner/README.md

Kubernetes on Hetzner with Terraform

Provision a Kubernetes cluster on Hetzner using Terraform and Kubespray

Overview

The setup looks like following

text
Kubernetes cluster
+--------------------------+
|      +--------------+    |
|      | +--------------+  |
| -->  | |              |  |
|      | | Master/etcd  |  |
|      | | node(s)      |  |
|      +-+              |  |
|        +--------------+  |
|              ^           |
|              |           |
|              v           |
|      +--------------+    |
|      | +--------------+  |
| -->  | |              |  |
|      | |    Worker    |  |
|      | |    node(s)   |  |
|      +-+              |  |
|        +--------------+  |
+--------------------------+

The nodes uses a private network for node to node communication and a public interface for all external communication.

Requirements

  • Terraform 0.14.0 or newer

Quickstart

NOTE: Assumes you are at the root of the kubespray repo.

For authentication in your cluster you can use the environment variables.

bash
export HCLOUD_TOKEN=api-token

Copy the cluster configuration file.

bash
CLUSTER=my-hetzner-cluster
cp -r inventory/sample inventory/$CLUSTER
cp contrib/terraform/hetzner/default.tfvars inventory/$CLUSTER/
cd inventory/$CLUSTER

Edit default.tfvars to match your requirement.

Flatcar Container Linux instead of the basic Hetzner Images.

bash
cd ../../contrib/terraform/hetzner

Edit main.tf and reactivate the module source = "./modules/kubernetes-cluster-flatcar"and
comment out the #source = "./modules/kubernetes-cluster".

activate ssh_private_key_path = var.ssh_private_key_path. The VM boots into
Rescue-Mode with the selected image of the var.machines but installs Flatcar instead.

Run Terraform to create the infrastructure.

bash
cd ./kubespray
terraform -chdir=./contrib/terraform/hetzner/ init
terraform -chdir=./contrib/terraform/hetzner/ apply --var-file=../../../inventory/$CLUSTER/default.tfvars

You should now have a inventory file named inventory.ini that you can use with kubespray.
You can use the inventory file with kubespray to set up a cluster.

It is a good idea to check that you have basic SSH connectivity to the nodes. You can do that by:

bash
ansible -i inventory.ini -m ping all

You can setup Kubernetes with kubespray using the generated inventory:

bash
ansible-playbook -i inventory.ini ../../cluster.yml -b -v

Cloud controller

For better support with the cloud you can install the hcloud cloud controller and CSI driver.

Please read the instructions in both repos on how to install it.

Teardown

You can teardown your infrastructure using the following Terraform command:

bash
cd ./kubespray
terraform -chdir=./contrib/terraform/hetzner/ destroy --var-file=../../../inventory/$CLUSTER/default.tfvars

Variables

  • prefix: Prefix to add to all resources, if set to "" don't set any prefix
  • ssh_public_keys: List of public SSH keys to install on all machines
  • zone: The zone where to run the cluster
  • network_zone: the network zone where the cluster is running
  • machines: Machines to provision. Key of this object will be used as the name of the machine
    • node_type: The role of this node (master|worker)
    • size: Size of the VM
    • image: The image to use for the VM
  • ssh_whitelist: List of IP ranges (CIDR) that will be allowed to ssh to the nodes
  • api_server_whitelist: List of IP ranges (CIDR) that will be allowed to connect to the API server
  • nodeport_whitelist: List of IP ranges (CIDR) that will be allowed to connect to the kubernetes nodes on port 30000-32767 (kubernetes nodeports)
  • ingress_whitelist: List of IP ranges (CIDR) that will be allowed to connect to kubernetes workers on port 80 and 443

File: contrib/terraform/gcp/README.md

Kubernetes on GCP with Terraform

Provision a Kubernetes cluster on GCP using Terraform and Kubespray

Overview

The setup looks like following

text
Kubernetes cluster
                        +-----------------------+
+---------------+       |   +--------------+    |
|               |       |   | +--------------+  |
| API server LB +---------> | |              |  |
|               |       |   | | Master/etcd  |  |
+---------------+       |   | | node(s)      |  |
                        |   +-+              |  |
                        |     +--------------+  |
                        |           ^           |
                        |           |           |
                        |           v           |
+---------------+       |   +--------------+    |
|               |       |   | +--------------+  |
|  Ingress LB   +---------> | |              |  |
|               |       |   | |    Worker    |  |
+---------------+       |   | |    node(s)   |  |
                        |   +-+              |  |
                        |     +--------------+  |
                        +-----------------------+

Requirements

  • Terraform 0.12.0 or newer

Quickstart

To get a cluster up and running you'll need a JSON keyfile.
Set the path to the file in the tfvars.json file and run the following:

bash
terraform apply -var-file tfvars.json -state dev-cluster.tfstate -var gcp_project_id= -var keyfile_location=<location of the json keyfile>

To generate kubespray inventory based on the terraform state file you can run the following:

bash
./generate-inventory.sh dev-cluster.tfstate > inventory.ini

You should now have a inventory file named inventory.ini that you can use with kubespray, e.g.

bash
ansible-playbook -i contrib/terraform/gcp/inventory.ini cluster.yml -b -v

Variables

Required

  • keyfile_location: Location to the keyfile to use as credentials for the google terraform provider
  • gcp_project_id: ID of the GCP project to deploy the cluster in
  • ssh_pub_key: Path to public ssh key to use for all machines
  • region: The region where to run the cluster
  • machines: Machines to provision. Key of this object will be used as the name of the machine
    • node_type: The role of this node (master|worker)
    • size: The size to use
    • zone: The zone the machine should run in
    • additional_disks: Extra disks to add to the machine. Key of this object will be used as the disk name
      • size: Size of the disk (in GB)
    • boot_disk: The boot disk to use
      • image_name: Name of the image
      • size: Size of the boot disk (in GB)
  • ssh_whitelist: List of IP ranges (CIDR) that will be allowed to ssh to the nodes
  • api_server_whitelist: List of IP ranges (CIDR) that will be allowed to connect to the API server
  • nodeport_whitelist: List of IP ranges (CIDR) that will be allowed to connect to the kubernetes nodes on port 30000-32767 (kubernetes nodeports)
  • ingress_whitelist: List of IP ranges (CIDR) that will be allowed to connect to ingress on ports 80 and 443
  • extra_ingress_firewalls: Additional ingress firewall rules. Key will be used as the name of the rule
    • source_ranges: List of IP ranges (CIDR). Example: ["8.8.8.8"]
    • protocol: Protocol. Example "tcp"
    • ports: List of ports, as string. Example ["53"]
    • target_tags: List of target tag (either the machine name or control-plane or worker). Example: ["control-plane", "worker-0"]

Optional

  • prefix: Prefix to use for all resources, required to be unique for all clusters in the same project (Defaults to default)
  • master_sa_email: Service account email to use for the control plane nodes (Defaults to "", auto generate one)
  • master_sa_scopes: Service account email to use for the control plane nodes (Defaults to ["https://www.googleapis.com/auth/cloud-platform"])
  • master_preemptible: Enable preemptible
    for the control plane nodes (Defaults to false)
  • master_additional_disk_type: Disk type
    for extra disks added on the control plane nodes (Defaults to "pd-ssd")
  • worker_sa_email: Service account email to use for the worker nodes (Defaults to "", auto generate one)
  • worker_sa_scopes: Service account email to use for the worker nodes (Defaults to ["https://www.googleapis.com/auth/cloud-platform"])
  • worker_preemptible: Enable preemptible
    for the worker nodes (Defaults to false)
  • worker_additional_disk_type: Disk type
    for extra disks added on the worker nodes (Defaults to "pd-ssd")

An example variables file can be found tfvars.json

Known limitations

This solution does not provide a solution to use a bastion host. Thus all the nodes must expose a public IP for kubespray to work.

File: contrib/terraform/exoscale/README.md

Kubernetes on Exoscale with Terraform

Provision a Kubernetes cluster on Exoscale using Terraform and Kubespray

Overview

The setup looks like following

text
Kubernetes cluster
                        +-----------------------+
+---------------+       |   +--------------+    |
|               |       |   | +--------------+  |
| API server LB +---------> | |              |  |
|               |       |   | | Master/etcd  |  |
+---------------+       |   | | node(s)      |  |
                        |   +-+              |  |
                        |     +--------------+  |
                        |           ^           |
                        |           |           |
                        |           v           |
+---------------+       |   +--------------+    |
|               |       |   | +--------------+  |
|  Ingress LB   +---------> | |              |  |
|               |       |   | |    Worker    |  |
+---------------+       |   | |    node(s)   |  |
                        |   +-+              |  |
                        |     +--------------+  |
                        +-----------------------+

Requirements

  • Terraform 0.13.0 or newer (0.12 also works if you modify the provider block to include version and remove all versions.tf files)

Quickstart

NOTE: Assumes you are at the root of the kubespray repo

Copy the sample inventory for your cluster and copy the default terraform variables.

bash
CLUSTER=my-exoscale-cluster
cp -r inventory/sample inventory/$CLUSTER
cp contrib/terraform/exoscale/default.tfvars inventory/$CLUSTER/
cd inventory/$CLUSTER

Edit default.tfvars to match your setup. You MUST, at the very least, change ssh_public_keys.

bash
# Ensure $EDITOR points to your favorite editor, e.g., vim, emacs, VS Code, etc.
$EDITOR default.tfvars

For authentication you can use the credentials file ~/.cloudstack.ini or ./cloudstack.ini.
The file should look like something like this:

ini
[cloudstack]
key = <API key>
secret = <API secret>

Follow the Exoscale IAM Quick-start to learn how to generate API keys.

Encrypted credentials

To have the credentials encrypted at rest, you can use sops and only decrypt the credentials at runtime.

bash
cat << EOF > cloudstack.ini
[cloudstack]
key =
secret =
EOF
sops --encrypt --in-place --pgp  cloudstack.ini
sops cloudstack.ini

Run terraform to create the infrastructure

bash
terraform init ../../contrib/terraform/exoscale
terraform apply -var-file default.tfvars ../../contrib/terraform/exoscale

If your cloudstack credentials file is encrypted using sops, run the following:

bash
terraform init ../../contrib/terraform/exoscale
sops exec-file -no-fifo cloudstack.ini 'CLOUDSTACK_CONFIG={} terraform apply -var-file default.tfvars ../../contrib/terraform/exoscale'

You should now have a inventory file named inventory.ini that you can use with kubespray.
You can now copy your inventory file and use it with kubespray to set up a cluster.
You can type terraform output to find out the IP addresses of the nodes, as well as control-plane and data-plane load-balancer.

It is a good idea to check that you have basic SSH connectivity to the nodes. You can do that by:

bash
ansible -i inventory.ini -m ping all

Example to use this with the default sample inventory:

bash
ansible-playbook -i inventory.ini ../../cluster.yml -b -v

Teardown

The Kubernetes cluster cannot create any load-balancers or disks, hence, teardown is as simple as Terraform destroy:

bash
terraform destroy -var-file default.tfvars ../../contrib/terraform/exoscale

Variables

Required

  • ssh_public_keys: List of public SSH keys to install on all machines
  • zone: The zone where to run the cluster
  • machines: Machines to provision. Key of this object will be used as the name of the machine
    • node_type: The role of this node (master|worker)
    • size: The size to use
    • boot_disk: The boot disk to use
      • image_name: Name of the image
      • root_partition_size: Size (in GB) for the root partition
      • ceph_partition_size: Size (in GB) for the partition for rook to use as ceph storage. (Set to 0 to disable)
      • node_local_partition_size: Size (in GB) for the partition for node-local-storage. (Set to 0 to disable)
  • ssh_whitelist: List of IP ranges (CIDR) that will be allowed to ssh to the nodes
  • api_server_whitelist: List of IP ranges (CIDR) that will be allowed to connect to the API server
  • nodeport_whitelist: List of IP ranges (CIDR) that will be allowed to connect to the kubernetes nodes on port 30000-32767 (kubernetes nodeports)

Optional

  • prefix: Prefix to use for all resources, required to be unique for all clusters in the same project (Defaults to default)

An example variables file can be found default.tfvars

Known limitations

Only single disk

Since Exoscale doesn't support additional disks to be mounted onto an instance, this script has the ability to create partitions for Rook and node-local-storage.

No Kubernetes API

The current solution doesn't use the Exoscale Kubernetes cloud controller.
This means that we need to set up a HTTP(S) loadbalancer in front of all workers and set the Ingress controller to DaemonSet mode.

File: contrib/terraform/aws/README.md

Kubernetes on AWS with Terraform

Overview

This project will create:

  • VPC with Public and Private Subnets in # Availability Zones
  • Bastion Hosts and NAT Gateways in the Public Subnet
  • A dynamic number of masters, etcd, and worker nodes in the Private Subnet
    • even distributed over the # of Availability Zones
  • AWS ELB in the Public Subnet for accessing the Kubernetes API from the internet

Requirements

  • Terraform 0.12.0 or newer

How to Use

  • Export the variables for your AWS credentials or edit credentials.tfvars:
commandline
export TF_VAR_AWS_ACCESS_KEY_ID="www"
export TF_VAR_AWS_SECRET_ACCESS_KEY ="xxx"
export TF_VAR_AWS_SSH_KEY_NAME="yyy"
export TF_VAR_AWS_DEFAULT_REGION="zzz"
  • Update contrib/terraform/aws/terraform.tfvars with your data. By default, the Terraform scripts use Ubuntu 18.04 LTS (Bionic) as base image. If you want to change this behaviour, see note "Using other distrib than Ubuntu" below.
  • Create an AWS EC2 SSH Key
  • Run with terraform apply --var-file="credentials.tfvars" or terraform apply depending if you exported your AWS credentials

Example:

commandline
terraform apply -var-file=credentials.tfvars
  • Terraform automatically creates an Ansible Inventory file called hosts with the created infrastructure in the directory inventory
  • Ansible will automatically generate an ssh config file for your bastion hosts. To connect to hosts with ssh using bastion host use generated ssh-bastion.conf. Ansible automatically detects bastion and changes ssh_args
commandline
ssh -F ./ssh-bastion.conf user@$ip
  • Once the infrastructure is created, you can run the kubespray playbooks and supply inventory/hosts with the -i flag.

Example (this one assumes you are using Ubuntu)

commandline
ansible-playbook -i ./inventory/hosts ./cluster.yml -e ansible_user=ubuntu -b --become-user=root --flush-cache

Using other distrib than Ubuntu***

To leverage a Linux distribution other than Ubuntu 18.04 (Bionic) LTS for your Terraform configurations, you can adjust the AMI search filters within the 'data "aws_ami" "distro"' block by utilizing variables in your terraform.tfvars file. This approach ensures a flexible configuration that adapts to various Linux distributions without directly modifying the core Terraform files.

Example Usages

  • Debian Jessie: To configure the usage of Debian Jessie, insert the subsequent lines into your terraform.tfvars:

    hcl
    ami_name_pattern        = "debian-jessie-amd64-hvm-*"
ami_owners              = ["379101102735"]

  • Ubuntu 16.04: To utilize Ubuntu 16.04 instead, apply the following configuration in your terraform.tfvars:

    hcl
    ami_name_pattern        = "ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-*"
ami_owners              = ["099720109477"]

  • Centos 7: For employing Centos 7, incorporate these lines into your terraform.tfvars:

    hcl
    ami_name_pattern        = "dcos-centos7-*"
ami_owners              = ["688023202711"]

Connecting to Kubernetes

You can use the following set of commands to get the kubeconfig file from your newly created cluster. Before running the commands, make sure you are in the project's root folder.

commandline
# Get the controller's IP address.
CONTROLLER_HOST_NAME=$(cat ./inventory/hosts | grep "\[kube_control_plane\]" -A 1 | tail -n 1)
CONTROLLER_IP=$(cat ./inventory/hosts | grep $CONTROLLER_HOST_NAME | grep ansible_host | cut -d'=' -f2)

# Get the hostname of the load balancer.
LB_HOST=$(cat inventory/hosts | grep apiserver_loadbalancer_domain_name | cut -d'"' -f2)

# Get the controller's SSH fingerprint.
ssh-keygen -R $CONTROLLER_IP > /dev/null 2>&1
ssh-keyscan -H $CONTROLLER_IP >> ~/.ssh/known_hosts 2>/dev/null

# Get the kubeconfig from the controller.
mkdir -p ~/.kube
ssh -F ssh-bastion.conf centos@$CONTROLLER_IP "sudo chmod 644 /etc/kubernetes/admin.conf"
scp -F ssh-bastion.conf centos@$CONTROLLER_IP:/etc/kubernetes/admin.conf ~/.kube/config
sed -i "s^server:.*^server: https://$LB_HOST:6443^" ~/.kube/config
kubectl get nodes

Troubleshooting

Remaining AWS IAM Instance Profile

If the cluster was destroyed without using Terraform it is possible that
the AWS IAM Instance Profiles still remain. To delete them you can use
the AWS CLI with the following command:

commandline
aws iam delete-instance-profile --region <region_name> --instance-profile-name

Ansible Inventory doesn't get created

It could happen that Terraform doesn't create an Ansible Inventory file automatically. If this is the case copy the output after inventory= and create a file named hostsin the directory inventory and paste the inventory into the file.

Architecture

Pictured is an AWS Infrastructure created with this Terraform project distributed over two Availability Zones.

File: contrib/terraform/upcloud/README.md

Kubernetes on UpCloud with Terraform

Provision a Kubernetes cluster on UpCloud using Terraform and Kubespray

Requirements

  • Terraform 0.13.0 or newer

Quickstart

NOTE: Assumes you are at the root of the kubespray repo.

For authentication in your cluster you can use the environment variables.

bash
export TF_VAR_UPCLOUD_USERNAME=username
export TF_VAR_UPCLOUD_PASSWORD=password

To allow API access to your UpCloud account, you need to allow API connections by visiting Account-page in your UpCloud Hub.

Copy the cluster configuration file.

bash
CLUSTER=my-upcloud-cluster
cp -r inventory/sample inventory/$CLUSTER
cp contrib/terraform/upcloud/cluster-settings.tfvars inventory/$CLUSTER/
export ANSIBLE_CONFIG=ansible.cfg
cd inventory/$CLUSTER

Edit cluster-settings.tfvars to match your requirement.

Run Terraform to create the infrastructure.

bash
terraform init ../../contrib/terraform/upcloud
terraform apply --var-file cluster-settings.tfvars \
    -state=tfstate-$CLUSTER.tfstate \
     ../../contrib/terraform/upcloud/

You should now have a inventory file named inventory.ini that you can use with kubespray.
You can use the inventory file with kubespray to set up a cluster.

It is a good idea to check that you have basic SSH connectivity to the nodes. You can do that by:

bash
ansible -i inventory.ini -m ping all

You can setup Kubernetes with kubespray using the generated inventory:

bash
ansible-playbook -i inventory.ini ../../cluster.yml -b -v

Teardown

You can teardown your infrastructure using the following Terraform command:

bash
terraform destroy --var-file cluster-settings.tfvars \
      -state=tfstate-$CLUSTER.tfstate \
      ../../contrib/terraform/upcloud/

Variables

  • prefix: Prefix to add to all resources, if set to "" don't set any prefix
  • template_name: The name or UUID of a base image
  • username: a user to access the nodes, defaults to "ubuntu"
  • private_network_cidr: CIDR to use for the private network, defaults to "172.16.0.0/24"
  • dns_servers: DNS servers that will be used by the nodes. Until this is solved this is done using user_data to reconfigure resolved. Defaults to []
  • use_public_ips: If a NIC connencted to the Public network should be attached to all nodes by default. Can be overridden by force_public_ip if this is set to false. Defaults to true
  • ssh_public_keys: List of public SSH keys to install on all machines
  • zone: The zone where to run the cluster
  • machines: Machines to provision. Key of this object will be used as the name of the machine
    • node_type: The role of this node (master|worker)
    • plan: Preconfigured cpu/mem plan to use (disables cpu and mem attributes below)
    • cpu: number of cpu cores
    • mem: memory size in MB
    • disk_size: The size of the storage in GB
    • force_public_ip: If use_public_ips is set to false, this forces a public NIC onto the machine anyway when set to true. Useful if you're migrating from public nodes to only private. Defaults to false
    • dns_servers: This works the same way as the global dns_severs but only applies to a single node. If set to [] while the global dns_servers is set to something else, then it will not add the user_data and thus will not be recreated. Useful if you're migrating from public nodes to only private. Defaults to null
    • additional_disks: Additional disks to attach to the node.
      • size: The size of the additional disk in GB
      • tier: The tier of disk to use (maxiops is the only one you can choose atm)
  • firewall_enabled: Enable firewall rules
  • firewall_default_deny_in: Set the firewall to deny inbound traffic by default. Automatically adds UpCloud DNS server and NTP port allowlisting.
  • firewall_default_deny_out: Set the firewall to deny outbound traffic by default.
  • master_allowed_remote_ips: List of IP ranges that should be allowed to access API of masters
    • start_address: Start of address range to allow
    • end_address: End of address range to allow
  • k8s_allowed_remote_ips: List of IP ranges that should be allowed SSH access to all nodes
    • start_address: Start of address range to allow
    • end_address: End of address range to allow
  • master_allowed_ports: List of port ranges that should be allowed to access the masters
    • protocol: Protocol (tcp|udp|icmp)
    • port_range_min: Start of port range to allow
    • port_range_max: End of port range to allow
    • start_address: Start of address range to allow
    • end_address: End of address range to allow
  • worker_allowed_ports: List of port ranges that should be allowed to access the workers
    • protocol: Protocol (tcp|udp|icmp)
    • port_range_min: Start of port range to allow
    • port_range_max: End of port range to allow
    • start_address: Start of address range to allow
    • end_address: End of address range to allow
  • loadbalancer_enabled: Enable managed load balancer
  • loadbalancer_plan: Plan to use for load balancer (development|production-small)
  • loadbalancer_legacy_network: If the loadbalancer should use the deprecated network field instead of networks blocks. You probably want to have this set to false (default value)
  • loadbalancers: Ports to load balance and which machines to forward to. Key of this object will be used as the name of the load balancer frontends/backends
    • port: Port to load balance.
    • target_port: Port to the backend servers.
    • backend_servers: List of servers that traffic to the port should be forwarded to.
    • proxy_protocol: If the loadbalancer should set up the backend using proxy protocol.
  • router_enable: If a router should be connected to the private network or not
  • gateways: Gateways that should be connected to the router, requires router_enable is set to true
    • features: List of features for the gateway
    • plan: Plan to use for the gateway
    • connections: The connections and tunnel to create for the gateway
      • type: What type of connection
      • local_routes: Map of local routes for the connection
        • type: Type of route
        • static_network: Destination prefix of the route; needs to be a valid IPv4 prefix
      • remote_routes: Map of local routes for the connection
        • type: Type of route
        • static_network: Destination prefix of the route; needs to be a valid IPv4 prefix
      • tunnels: The tunnels to create for this connection
        • remote_address: The remote address for the tunnel
        • ipsec_properties: Set properties of IPSec, if not set, defaults will be used
          • child_rekey_time: IKE child SA rekey time in seconds
          • dpd_delay: Delay before sending Dead Peer Detection packets if no traffic is detected, in seconds
          • dpd_timeout: Timeout period for DPD reply before considering the peer to be dead, in seconds
          • ike_lifetime: Maximum IKE SA lifetime in seconds()
          • rekey_time: IKE SA rekey time in seconds
          • phase1_algorithms: List of Phase 1: Proposal algorithms
          • phase1_dh_group_numbers: List of Phase 1 Diffie-Hellman group numbers
          • phase1_integrity_algorithms: List of Phase 1 integrity algorithms
          • phase2_algorithms: List of Phase 2: Security Association algorithms
          • phase2_dh_group_numbers: List of Phase 2 Diffie-Hellman group numbers
          • phase2_integrity_algorithms: List of Phase 2 integrity algorithms
  • gateway_vpn_psks: Separate variable for providing psks for connection tunnels. Environment variable can be exported in the following format export TF_VAR_gateway_vpn_psks='{"${gateway-name}-${connecton-name}-tunnel":{psk:"..."}}'
  • static_routes: Static routes to apply to the router, requires router_enable is set to true
  • network_peerings: Other UpCloud private networks to peer with, requires router_enable is set to true
  • server_groups: Group servers together
    • servers: The servers that should be included in the group.
    • anti_affinity_policy: Defines if a server group is an anti-affinity group. Setting this to "strict" or yes" will result in all servers in the group being placed on separate compute hosts. The value can be "strict", "yes" or "no". "strict" refers to strict policy doesn't allow servers in the same server group to be on the same host. "yes" refers to best-effort policy and tries to put servers on different hosts, but this is not guaranteed.

Migration

When null_resource.inventories and data.template_file.inventory was changed to local_file.inventory the old state file needs to be cleaned of the old state.
The error messages you'll see if you encounter this is:

text
Error: failed to read schema for null_resource.inventories in registry.terraform.io/hashicorp/null: failed to instantiate provider "registry.terraform.io/hashicorp/null" to obtain schema: unavailable provider "registry.terraform.io/hashicorp/null"
Error: failed to read schema for data.template_file.inventory in registry.terraform.io/hashicorp/template: failed to instantiate provider "registry.terraform.io/hashicorp/template" to obtain schema: unavailable provider "registry.terraform.io/hashicorp/template"

This can be fixed with the following lines

bash
terraform state rm -state=terraform.tfstate null_resource.inventories
terraform state rm -state=terraform.tfstate data.template_file.inventory

Public to Private only migration

Since there's no way to remove the public NIC on a machine without recreating its private NIC it's not possible to inplace change a cluster to only use private IPs.
The way to migrate is to first set use_public_ips to false, dns_servers to some DNS servers and then update all existing servers to have force_public_ip set to true and dns_severs set to [].
After that you can add new nodes without force_public_ip and dns_servers set and create them.
Add the new nodes into the cluster and when all of them are added, remove the old nodes.

File: contrib/terraform/vsphere/README.md

Kubernetes on vSphere with Terraform

Provision a Kubernetes cluster on vSphere using Terraform and Kubespray.

Overview

The setup looks like following.

text
Kubernetes cluster
+-----------------------+
|   +--------------+    |
|   | +--------------+  |
|   | |              |  |
|   | | Master/etcd  |  |
|   | | node(s)      |  |
|   +-+              |  |
|     +--------------+  |
|           ^           |
|           |           |
|           v           |
|   +--------------+    |
|   | +--------------+  |
|   | |              |  |
|   | |    Worker    |  |
|   | |    node(s)   |  |
|   +-+              |  |
|     +--------------+  |
+-----------------------+

Warning

This setup assumes that the DHCP is disabled in the vSphere cluster and IP addresses have to be provided in the configuration file.

Requirements

  • Terraform 0.13.0 or newer (0.12 also works if you modify the provider block to include version and remove all versions.tf files)

Quickstart

NOTE: Assumes you are at the root of the kubespray repo

Copy the sample inventory for your cluster and copy the default terraform variables.

bash
CLUSTER=my-vsphere-cluster
cp -r inventory/sample inventory/$CLUSTER
cp contrib/terraform/vsphere/default.tfvars inventory/$CLUSTER/
cd inventory/$CLUSTER

Edit default.tfvars to match your setup. You MUST set values specific for you network and vSphere cluster.

bash
# Ensure $EDITOR points to your favorite editor, e.g., vim, emacs, VS Code, etc.
$EDITOR default.tfvars

For authentication in your vSphere cluster you can use the environment variables.

bash
export TF_VAR_vsphere_user=username
export TF_VAR_vsphere_password=password

Run Terraform to create the infrastructure.

bash
terraform init ../../contrib/terraform/vsphere
terraform apply \
    -var-file default.tfvars \
    -state=tfstate-$CLUSTER.tfstate \
    ../../contrib/terraform/vsphere

You should now have a inventory file named inventory.ini that you can use with kubespray.
You can now copy your inventory file and use it with kubespray to set up a cluster.
You can type terraform output to find out the IP addresses of the nodes.

It is a good idea to check that you have basic SSH connectivity to the nodes. You can do that by:

bash
ansible -i inventory.ini -m ping all

Example to use this with the default sample inventory:

bash
ansible-playbook -i inventory.ini ../../cluster.yml -b -v

Variables

Required

  • machines: Machines to provision. Key of this object will be used as the name of the machine
    • node_type: The role of this node (master|worker)
    • ip: The IP address of the machine
    • netmask: The netmask to use (to be used on the right hand side in CIDR notation, e.g., 24)
  • network: The name of the network to attach the machines to
  • gateway: The IP address of the network gateway
  • vsphere_datacenter: The identifier of vSphere data center
  • vsphere_compute_cluster: The identifier of vSphere compute cluster
  • vsphere_datastore: The identifier of vSphere data store
  • vsphere_server: This is the vCenter server name or address for vSphere API operations.
  • ssh_public_keys: List of public SSH keys to install on all machines
  • template_name: The name of a base image (the OVF template be defined in vSphere beforehand)

Optional

  • folder: Name of the folder to put all machines in (default: "")
  • prefix: Prefix to use for all resources, required to be unique for all clusters in the same project (default: "k8s")
  • inventory_file: Name of the generated inventory file for Kubespray to use in the Ansible step (default: inventory.ini)
  • dns_primary: The IP address of primary DNS server (default: 8.8.4.4)
  • dns_secondary: The IP address of secondary DNS server (default: 8.8.8.8)
  • firmware: Firmware to use (default: bios)
  • hardware_version: The version of the hardware (default: 15)
  • master_cores: The number of CPU cores for the master nodes (default: 4)
  • master_memory: The amount of RAM for the master nodes in MB (default: 4096)
  • master_disk_size: The amount of disk space for the master nodes in GB (default: 20)
  • worker_cores: The number of CPU cores for the worker nodes (default: 16)
  • worker_memory: The amount of RAM for the worker nodes in MB (default: 8192)
  • worker_disk_size: The amount of disk space for the worker nodes in GB (default: 100)
  • vapp: Boolean to set the template type to vapp. (Default: false)
  • interface_name: Name of the interface to configure. (Default: ens192)

An example variables file can be found default.tfvars

Explore Related Technologies & Topics

View All 9,111+ Repositories →

Discover similar high-velocity repositories, agent skills, and OpenAPI specifications across the ecosystem.

#ansible Browse ansible catalog → #aws Browse aws catalog → #bare-metal Browse bare-metal catalog → #gce Browse gce catalog → #hacktoberfest Browse hacktoberfest catalog → #high-availability Browse high-availability catalog → ⚡ MCP Servers Model Context Protocol → 🧠 Agent Skills Prompt & rule files → 🔌 OpenAPI Specs REST & Swagger schemas →